zfs discuss - Jun 2006 - slow mkdir

Hello zfs-discuss,

  NFS server on snv_39/SPARC, zfs filesystems exported.
  Solaris 10 x64 clients (zfs-s10-0315), filesystems mounted from nfs
  server using NFSv3 over TCP.

  What I see from NFS clients is that mkdir operations to ZFS filesystems
  could take even 20s! while to UFS exported filesystems I can''t
  see even one with time over 1s
 (there''re also UFS exported filesystems from other NFS servers).


 How do I measure the time? On nfs client I do:
bash-3.00# dtrace -n syscall::mkdir:entry''/execname ==
"our-app"/{self->t=timestamp;self->vt=vtimestamp;self->arg0=arg0}''
-n syscall::mkdir:return''/self->t/{@[stringof
copyin(self->arg0,11)]=max((timestamp-self->t)/1000000000);self->arg0=0;self->t=0;self->vt=0;}''
-n tick-5s''{printa(@);}''
bash-3.00#


What I get is times even 20-30s but only for ZFS exported filesystems.
It''s not that all mkdir are that bad.

On one of those filesystems I tried several time to just mkdir some
dir from command line - for many tries I got new dir created
immediately, but then I hang for ~8s.

-bash-3.00$ truss -ED -v all mkdir www
[...]
 0.0000  0.0000 umask(022)                                      = 0
mkdir("www", 0777)              (sleeping...)
 8.0158  0.0001 mkdir("www", 0777)                              = 0
 0.0002  0.0000 _exit(0)

I tried it locally on ZFS (the same filesystem) and not on NFS - this
time I get very fast mkdir every time I try it.

So probably something on with client<->NFSv3<->ZFS

Looks like if traffic is lighter then I see 3s at most so
it''s much better.

Any idea?


ps. of course there''re no colisions, etc. on network. At least I
can''t
    find anything unusual.




-- 
Best regards,
 Robert                          mailto:rmilkowski at task.gda.pl
                                     http://milek.blogspot.com

It''s  easy  to reproduce seconds long   mkdir (and other ops
also) with a  side load of  dd even to  local  ZFS. ZFS team
have started investigating this:

	6429205 each zpool needs to monitor it''s  throughput and throttle
heavy writers

which  will help bound  the time to such   operations to < 5
seconds. Today we throttle writers too seldom and that means
there is too much I/O to  sync before we release operations;
Basically when the  storage is the  weak link,  we''re better
off throttling apps  in small increments rather than consume
tons of memory then block everyone waiting on I/O for 10s of
seconds.

and  while I hold the floor,  I posted  this entry last week
which could be of interesting to  try in the general purpose
(small file updates) NFS serving: 

	http://blogs.sun.com/roller/page/roch?entry=tuning_zfs_recordsize 


-r