freebsd stable - May 2005 - problems with nfs+TCP - Resource temporarily unavailable

Hi,


I'm getting the following error when dding a big file on an nfs mount
which is mounted using TCP.


root@kartoffel olivleh1> dd if=/usr/tmp.data of=/mnt/files/tmp.data bs=32k
dd: /mnt/files/tmp.data: Resource temporarily unavailable
639+0 records in
638+0 records out
20905984 bytes transferred in 15.066490 secs (1387582 bytes/sec)
Exit 1

root@dill olivleh1> dd if=/usr/tmp.data of=/mnt/files/tmp.data bs=32k
dd: /mnt/files/tmp.data: Resource temporarily unavailable
1035+0 records in
1034+0 records out
33882112 bytes transferred in 14.698220 secs (2305185 bytes/sec)
Exit 1

dmesg gives me "nfs send error 35 for server file:/mnt/files"

fstab entry on kartoffel and dill:
file:/mnt/files /mnt/files nfs tcp,nfsv3,soft,bg,rw,noauto 0 0


 - kartoffel is an amd64 system with an onboard re0
             running CURRENT from May 24th evening.
 - dill      is an alpha system with an xl0
             running 5.4 STABLE from May 20th
 - file      is an i386 SMP system with an xl0
             running 5.4 STABLE from May 20th

 - dill and file are connected directly through an switch
 - kartoffel and file are connected through an router+switches.

 - kartoffel and nudel are running rpc.lockd and rpc.statd
 - dill doesn't run rpc.lockd neither rpc.statd


Switching from nfsv3 to a nfsv2 mount is much slower and breaks sooner or
later with an error too. It doesn't give me an "error 35" on
client side,
but an "nfsd send error 32" on server side and a slightly different
error
message:

root@kartoffel olivleh1> dd if=/usr/tmp.data of=/mnt/files/tmp.data bs=32k
dd: /mnt/files/tmp.data: Operation timed out
67+0 records in
66+0 records out
2162688 bytes transferred in 6.221855 secs (347595 bytes/sec)
Exit 1


Switching from TCP to UDP makes this error gone.

Any ideas how to fix this properly?

(please CC me, I'm not subscribed to stable@)


-- 
 Oliver Lehmann
  http://www.pofo.de/
  http://wishlist.ans-netz.de/

Hi Mohan,

Mohan Srinivasan wrote:

> Is this consistently reproducible ?
it is - everytime

> I tried reproducing this with this morning's
> current,
it also happens with STABLE

> How big was your file that you tried to dd ? I need to reproduce this here
> in order to track it down.
dd if=/dev/urandom of=/usr/tmp.data bs=512k count=200

> Also, can you try the test without using the soft mount option ? I
don't see
> soft causing this, but just to eliminate those code paths.
I removed soft and bb, but still the same results:

root@kartoffel olivleh1> dd if=/usr/tmp.data of=/mnt/files/temp bs=32k
dd: /mnt/files/temp: Resource temporarily unavailable
1797+0 records in
1796+0 records out
58851328 bytes transferred in 33.651500 secs (1748847 bytes/sec)


######


I tried the same with an other nfs server (using dill as nfs server this
time - system description is in my 1st mail, same mount options like /
mnt/files). And guess what? dill rebooted immediate... dd came never
back, gave no output

dill's dmesg shows me:

fatal kernel trap:

    trap entry     = 0x4 (unaligned access fault)
    faulting va    = 0xfffffc0006b6f44d
    opcode         = 0x28
    register       = 0x5
    pc             = 0xfffffc0000541e08
    ra             = 0xfffffc0000541df4
    sp             = 0xfffffe000a0f9b70
    usp            = 0x11ffea80
    curthread      = 0xfffffc000f91ee10
        pid = 343, comm = nfsd

panic: trap
Uptime: 3d14h15m51s
Dumping 253 MB
 16 32 48 64 80 96 112 128 144 160 176 192 208 224 240
Dump complete

unfortunately...

root@dill tmp> kgdb vmcore.1 /usr/obj/alpha-5.4/usr/src/sys/DILL/
kernel.debug kgdb: bad namelist
Exit 1

and.... damn! yes that panic is reproduceable!

2nd try writing on dill:

root@kartoffel olivleh1> dd if=/usr/tmp.data of=/mnt/www/temp bs=32k
dd: /mnt/www/temp: Resource temporarily unavailable
387+0 records in
386+0 records out
12648448 bytes transferred in 11.766768 secs (1074930 bytes/sec)
<dill is gone - dd exited when dill was up again (NFS..)>

So.. using i386 as an tcp nfs-server - the only thing which happens is
that dd gets interruped, using alpha as an tcp nfs-server makes the alpha
panic.


And now it looks I should at first unmount the tcp mount, and then let my
alpha system come back online ;) (which is of course not possible w/o the
nfsd available of course....)


May 26 00:59:44 dill rpcbind: cannot create socket for udp6
Starting mountd.
NFS on reserved port only=YES
Starting nfsd.
Starting local daemons:
fatal kernel trap:

    trap entry     = 0x4 (unaligned access fault)
    faulting va    = 0xfffffc000f908929
    opcode         = 0x28
    register       = 0x5
    pc             = 0xfffffc0000532164
    ra             = 0xfffffc0000532138
    sp             = 0xfffffe000a1118c0
    usp            = 0x11ffea80
    curthread      = 0xfffffc000f91f950
        pid = 409, comm = nfsd

panic: trap
Uptime: 8m17s
Dumping 253 MB
 16 32 48 64 80 96 112 128 144 160 176 192 208 224 240
Dump complete
Automatic reboot in 15 seconds - press a key on the console to abort
Rebooting...
*** keyboard not plugged in...

halted CPU 0

halt code = 5
HALT instruction executed
PC = fffffc00005bfac0
*** no timer interrupts on CPU 0 ***

CPU 0 booting



If someone want me to test sth.. let me know Systems available are 6.0/
i386, 6.0/amd64, 5.4-SMP/i386, 5.4/i386, 5.4/alpha, 4.11/i386


-- 
 Oliver Lehmann
  http://www.pofo.de/
  http://wishlist.ans-netz.de/