Bizarre NFS problem

[UbuntuBantu]

I'm having a hell of a time trying to understand and resolve two NFS issues and I'm hoping that the knowlegeable folk on this forum are able to help me understand and resolve the issues (I'm guessing they're related).

Background:
I am running Ubuntu 10.10 on my workstation and have Ubuntu Server 10.10 installed on a HP Proliant MicroServer.

A NFS share is configured on the MicroServer and successfully mounts from workstation PC/ Ubuntu 10.10

The issues:

Copying files to an NFS share on the microserver is very, very slow.
Whilst files copied to the NFS share are visible to the workstation, they don't appear when using 'ls' on the server.
despite the folder owner/ user account on the Microserver and the workstation having the same name and sharing the same UID & GID (1000), copies made from the workstation have a different UID & GID
workstation$ ls -la
total 28
drwxr-xr-x 7 4294967294 4294967294 4096 2011-04-19 22:04 .
drwxr-xr-x 6 root root 4096 2011-04-21 19:54 ..
drwxrwxrwx 4 4294967294 4294967294 4096 2011-04-21 19:55 disk1
drwxrwxrwx 2 4294967294 4294967294 4096 2011-04-19 22:04 disk2
drwxrwxrwx 2 4294967294 4294967294 4096 2011-04-19 22:04 disk3
drwxrwxrwx 2 4294967294 4294967294 4096 2011-04-19 22:04 disk4microserver$ ls -la
total 28
drwxr-xr-x 7 egd egd 4096 2011-04-19 22:04 .
drwxr-xr-x 4 root root 4096 2011-04-19 07:27 ..
drwxr-xr-x 3 egd egd 4096 2011-04-21 08:24 disk1
drwxr-xr-x 3 egd egd 4096 2011-04-19 21:13 disk2
drwxr-xr-x 3 egd egd 4096 2011-04-19 21:16 disk3
drwxr-xr-x 3 egd egd 4096 2011-04-19 21:17 disk4



Configuration:
HP Proliant MicroServer:

Network config:microserver$ ifconfig
eth0 Link encap:Ethernet HWaddr 68:b5:99:79:ff:90
inet addr:192.168.168.252 Bcast:192.168.168.255 Mask:255.255.255.0
inet6 addr: fe80::6ab5:99ff:fe79:ff90/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:170867 errors:0 dropped:0 overruns:0 frame:0
TX packets:86755 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:255720726 (255.7 MB) TX bytes:6477663 (6.4 MB)
Interrupt:18

4 x 2TB drives formatted ext4, mounted to /media/audiolib/disk1...4 via fstab:
/dev/sda1 /media/audiolib/disk1 ext4 user,noatime,noexec,nodev 0 0
/dev/sdb1 /media/audiolib/disk2 ext4 user,noatime,noexec,nodev 0 0
/dev/sdc1 /media/audiolib/disk3 ext4 user,noatime,noexec,nodev 0 0
/dev/sdd1 /media/audiolib/disk4 ext4 user,noatime,noexec,nodev 0 0


folder/ file permissions for /media/audiolib
microserver$ sudo ls -la
drwxr-xr-x 7 egd egd 4096 2011-04-19 22:04 audiolib
folder/ file permissions for /media/audiolib/disk1...4
drwxr-xr-x 3 egd egd 4096 2011-04-21 08:24 disk1
drwxr-xr-x 3 egd egd 4096 2011-04-19 21:13 disk2
drwxr-xr-x 3 egd egd 4096 2011-04-19 21:16 disk3
drwxr-xr-x 3 egd egd 4096 2011-04-19 21:17 disk4

NFS share configuration:microserver$ cat /etc/exports
# /etc/exports: the access control list for filesystems which may be exported
# to NFS clients. See exports(5).
#
# Example for NFSv2 and NFSv3:
# /srv/homes hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check)
#
# Example for NFSv4:
# /srv/nfs4 gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check)
# /srv/nfs4/homes gss/krb5i(rw,sync,no_subtree_check)
#
/media/audiolib 192.168.168.1/24(rw,root_squash,sync,nohide)


Workstation:
Network config:workstation$ ifconfig
eth0 Link encap:Ethernet HWaddr 00:1e:8c:44:b5:06
inet addr:192.168.168.1 Bcast:192.168.168.255 Mask:255.255.255.0
inet6 addr: fe80::21e:8cff:fe44:b506/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:93652 errors:0 dropped:0 overruns:0 frame:0
TX packets:24996 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:12985610 (12.9 MB) TX bytes:245907927 (245.9 MB)
Interrupt:17

NFS share mounted using sudo mount 192.168.168.252:/media/audiolib /media/canonflash/

workstation$ mount
192.168.168.252:/media/audiolib on /media/canonflash type nfs (rw,vers=4,addr=192.168.168.252,clientaddr=192.168 .168.1)

egd@workstation:/media$ ls -la
total 24
drwxr-xr-x 6 root root 4096 2011-04-21 19:54 .
drwxr-xr-x 22 root root 4096 2011-04-02 11:18 ..
drwxr-xr-x 2 egd egd 4096 2010-05-09 10:13 0CDB-0F33
drwxr-xr-x 2 egd egd 4096 2010-05-03 20:05 AudioLIB
drwxr-xr-x 7 4294967294 4294967294 4096 2011-04-19 22:04 canonflash
drwxr-xr-x 7 egd egd 4096 2011-01-25 19:27 egd_backup
-rw-r--r-- 1 root root 0 2011-04-12 07:04 .hal-mtab

[x]

I'll give it a shot - ok, I've had a few glasses of wine tonight (liquid - not digital! :) ) so please don't kill me if I'm wrong!
This seems to me to be an issue with username/uid.
You need to have the same user on client and server - that is, the user must have same uid on both computers, even if it's the same username that really means nothing. Username is only for us humans it is not used by computers, computer uses number=uid!
(Well, actually you can map a user on client to a different uid on server, but I don't think you've done that.)
Look at how it's displayed when you issue "ls -la" on workstation:
drwxr-xr-x 2 egd egd 4096 2010-05-03 20:05 AudioLIB
drwxr-xr-x 7 4294967294 4294967294 4096 2011-04-19 22:04 canonflash
drwxr-xr-x 7 egd egd 4096 2011-01-25 19:27 egd_backup
The user with uid&gid 4294967294 is not recognized - otherwise the name would be displayed instead of the number.

So my suggestion is to make sure you have a user with same uid on both client & server.

[infiniphunk]

I've had a few diet 7Up & vodka

what x said.

[UbuntuBantu]

So how does one resolve it?

[UbuntuBantu]

Changed NFS config as follows:
Microserver:
microserver:/$ cat /etc/exports
# /etc/exports: the access control list for filesystems which may be exported
# to NFS clients. See exports(5).
#
# share all hard drives mounted to /diskpool
/diskpool 192.168.168.0/24(rw,no_subtree_check,async)

microserver:/$ cat /etc/fstab
# /etc/fstab: static file system information.
/dev/sda1 /diskpool/d1 ext4 user,noatime,noexec,nodev 0 0
/dev/sdb1 /diskpool/d2 ext4 user,noatime,noexec,nodev 0 0
/dev/sdc1 /diskpool/d3 ext4 user,noatime,noexec,nodev 0 0
/dev/sdd1 /diskpool/d4 ext4 user,noatime,noexec,nodev 0 0
/dev/sde1 /diskpool/d5 ext4 user,noatime,noexec,nodev 0 0

microserver:/$ ls -la
total 102
drwxr-xr-x 22 root root 4096 2011-04-22 09:58 .
drwxr-xr-x 22 root root 4096 2011-04-22 09:58 ..
drwxr-xr-x 2 root root 4096 2011-04-19 22:50 bin
drwxr-xr-x 4 root root 2048 2011-04-18 22:16 boot
drwxr-xr-x 18 root root 3860 2011-04-22 11:50 dev
drwxr-xr-x 7 egd egd 4096 2011-04-22 09:59 diskpool

microserver:/diskpool$ ls -la
total 28
drwxr-xr-x 7 egd egd 4096 2011-04-22 09:59 .
drwxr-xr-x 22 root root 4096 2011-04-22 09:58 ..
drwxr-xr-x 3 egd egd 4096 2011-04-22 11:28 d1
drwxr-xr-x 3 egd egd 4096 2011-04-22 11:32 d2
drwxr-xr-x 3 egd egd 4096 2011-04-22 11:42 d3
drwxr-xr-x 3 egd egd 4096 2011-04-22 11:42 d4
drwxr-xr-x 3 egd egd 4096 2011-04-22 11:48 d5

Workstatation:
p5e64wspro:/media$ ls -la
total 28
drwxr-xr-x 7 root root 4096 2011-04-22 12:15 .
drwxr-xr-x 22 root root 4096 2011-04-02 11:18 ..
drwxr-xr-x 7 egd egd 4096 2011-04-22 09:59 diskpool

p5e64wspro:/media/diskpool/d1$ sudo mount 192.168.168.252:/diskpool /media/diskpool/
p5e64wspro:/media/diskpool$ ls -la
total 28
drwxr-xr-x 7 egd egd 4096 2011-04-22 09:59 .
drwxr-xr-x 7 root root 4096 2011-04-22 12:15 ..
drwxr-xr-x 3 egd egd 4096 2011-04-22 12:21 d1
drwxr-xr-x 2 egd egd 4096 2011-04-22 09:59 d2
drwxr-xr-x 2 egd egd 4096 2011-04-22 09:59 d3
drwxr-xr-x 2 egd egd 4096 2011-04-22 09:59 d4
drwxr-xr-x 2 egd egd 4096 2011-04-22 09:59 d5

p5e64wspro:/media/diskpool/d1$ ls -la
total 12
drwxr-xr-x 3 egd egd 4096 2011-04-22 12:41 .
drwxr-xr-x 7 egd egd 4096 2011-04-22 09:59 ..
drwxr-xr-x 2 egd egd 4096 2011-04-19 21:22 Fleetwood Mac - Rumours


Now listing same folder via the Microserver:
microserver:/diskpool/d1$ ls -la
total 24
drwxr-xr-x 3 egd egd 4096 2011-04-22 11:28 .
drwxr-xr-x 7 egd egd 4096 2011-04-22 09:59 ..
drwx------ 2 egd egd 16384 2011-04-22 11:28 lost+found


The copy operation is damned slow and why don't the files show up on the damned server?

[x]

Aha, I think I see the problem:
/diskpool/d1 - d5 are all different partitions, they are therefore not accessible from client via nfs.
You have to explicitly export those directories and then mount them on client - maybe there's a config option in nfs to allow partitions to "follow" when mounting parent dir, nothing that I know of though.

[UbuntuBantu]

Aha, I think I see the problem:
/diskpool/d1 - d5 are all different partitions, they are therefore not accessible from client via nfs.
You have to explicitly export those directories and then mount them on client - maybe there's a config option in nfs to allow partitions to "follow" when mounting parent dir, nothing that I know of though.Thx, I solved it a few minutes prior to your post by doing exactly what you'd just said, explicitly exporting each drive/partition and mounting each individually.

So, the lesson for anyone wanting to use NFS to deal with a pool of drives is as follows:

say you have 5 drives on a server you're wanting to share via NFS, and they're mounted on the server as /drivepool/d1...d5.
One would think that you'd only need to export /drivepool on the server side and mount it on the client end as follows:

Server export: /diskpool 192.168.168.0/24(rw,no_subtree_check,async)
Client mount: $ sudo mount 192.168.168.252:/diskpool/ /media/diskpool/

That works (insofar as you can see the drives and access their content from the client), but writing to any of the drives is dog slow and only Linus knows where the files are actually written to, cause they're only visible on the client side, not on the server (somehwere I have at least 10 copies of FM - Rumours hidden in my filesystem -- judging by write speed it's probably on my memory stick).

The only way (I know of) to get this working as you'd think it would is to explicitly export every drive mountpoint as follows:
Server export:
/diskpool/d1 192.168.168.0/24(rw,no_subtree_check,async)
/diskpool/d2 192.168.168.0/24(rw,no_subtree_check,async)
/diskpool/d3 192.168.168.0/24(rw,no_subtree_check,async)
/diskpool/d4 192.168.168.0/24(rw,no_subtree_check,async)
/diskpool/d5 192.168.168.0/24(rw,no_subtree_check,async)

Client mount:
$ sudo mount 192.168.168.252:/diskpool/d1 /media/diskpool/d1
$ sudo mount 192.168.168.252:/diskpool/d2 /media/diskpool/d2
$ sudo mount 192.168.168.252:/diskpool/d3 /media/diskpool/d3
$ sudo mount 192.168.168.252:/diskpool/d4 /media/diskpool/d4
$ sudo mount 192.168.168.252:/diskpool/d5 /media/diskpool/d5

Thank goodness for fstab!

[Pafnoutios]

I use NFS in my network. I don't have seperate partitions, and in fact export different subdirectories in the same encrypted partition for different purposes. However, there might be some options about submounts (though I thought the default was to export the whole tree and an option was needed to not export the submounts). 'man exports' or 'man /etc/exports' should say more, or maybe 'man nfsmount' or something like that.

Regarding uid/gid consistency, I use NIS for network logins to keep users and groups syncronized. In a small enough network (up to 5 workstations?) it could be managed by hand, but I use NIS even with only 2-3 workstations. (I'll take this moment to mention that I like the way Windows networking uses different namespaces for accounts on different machines, and a centralized authorization certifier for access control, but I don't think it has to be implemented nearly as complicatedly as Microsoft has.)

[x]

I believe the files you copied earlier are on the server, if you unmount /diskpool/d1 on server you'll probably find your lost files.