Leon Koll
2007-Feb-27 22:55 UTC
[zfs-discuss] Why number of NFS threads jumps to the max value?
Hello, gurus I need your help. During the benchmark test of NFS-shared ZFS file systems at some moment the number of NFS threads jumps to the maximal value, 1027 (NFSD_SERVERS was set to 1024). The latency also grows and the number of IOPS is going down. I''ve collected the output of echo "::pgrep nfsd | ::walk thread | ::findstack -v" | mdb -k that can be seen here: http://tinyurl.com/yrvn4z Could you please look at it and tell me what''s wrong with my NFS server. Appreciate, -- Leon This message posted from opensolaris.org
Jim Mauro
2007-Feb-27 23:20 UTC
[zfs-discuss] Why number of NFS threads jumps to the max value?
You don''t honestly, really, reasonably, expect someone, anyone, to look at the stack trace of a few hundred threads, and post something along the lines of "This is what is wrong with your NFS server".....Do you? Without any other information at all? We''re here to help, but please reset your expectations around our abilities to root-cause pathological behavior based an almost no information. What size and type of server? What size and type of storage? What release of Solaris? What how may networks, and what type? What is being used to generate the load for the testing? What is the zpool configuration? What do the system stats look like while under load (e.g. mpstat), and how to they change when you see this behavior? What does "zpool iostat <zpool_name> 1" data look like while under load? Are you collecting nfsstat data - what is the rate of incoming NFS ops? Can you characterize the load - read/write data intensive, metadata intensive? Are the client machines Solaris, or something else? Does this last for seconds, minutes, tens-of-minutes? Does the system remain in this state indefinitely until reboot, or does it normalize? Can you consistently reproduce this problem? /jim Leon Koll wrote:> Hello, gurus > I need your help. During the benchmark test of NFS-shared ZFS file systems at some moment the number of NFS threads jumps to the maximal value, 1027 (NFSD_SERVERS was set to 1024). The latency also grows and the number of IOPS is going down. > I''ve collected the output of > echo "::pgrep nfsd | ::walk thread | ::findstack -v" | mdb -k > that can be seen here: > http://tinyurl.com/yrvn4z > > Could you please look at it and tell me what''s wrong with my NFS server. > Appreciate, > -- Leon > > > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >
Dennis Clarke
2007-Feb-27 23:33 UTC
[zfs-discuss] Why number of NFS threads jumps to the max value?
> > You don''t honestly, really, reasonably, expect someone, anyone, to look > at the stackwell of course he does :-) and I looked at it .. all of it and I can tell exactly what the problem is but I''m not gonna say because its a trick question. so there. Dennis
Jim Mauro
2007-Feb-27 23:39 UTC
[zfs-discuss] Why number of NFS threads jumps to the max value?
There are 517 threads in the zfs_write code path, so there seems to be a fair amount of write activity happening. The server iostat and zpool iostat data would be interesting, along with the backend storage configuration information - how many spindles, and what kind of zpool config... /jim Leon Koll wrote:> Hello, gurus > I need your help. During the benchmark test of NFS-shared ZFS file systems at some moment the number of NFS threads jumps to the maximal value, 1027 (NFSD_SERVERS was set to 1024). The latency also grows and the number of IOPS is going down. > I''ve collected the output of > echo "::pgrep nfsd | ::walk thread | ::findstack -v" | mdb -k > that can be seen here: > http://tinyurl.com/yrvn4z > > Could you please look at it and tell me what''s wrong with my NFS server. > Appreciate, > -- Leon > > > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >
Roch - PAE
2007-Feb-28 10:38 UTC
[zfs-discuss] Why number of NFS threads jumps to the max value?
http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6467988 NFSD threads are created on a demand spike (all of them waiting on I/O) but then tend to stick around servicing moderate loads. -r > > Leon Koll wrote: > > Hello, gurus > > I need your help. During the benchmark test of NFS-shared ZFS file systems at some moment the number of NFS threads jumps to the maximal value, 1027 (NFSD_SERVERS was set to 1024). The latency also grows and the number of IOPS is going down. > > I''ve collected the output of > > echo "::pgrep nfsd | ::walk thread | ::findstack -v" | mdb -k > > that can be seen here: > > http://tinyurl.com/yrvn4z > > > > Could you please look at it and tell me what''s wrong with my NFS server. > > Appreciate, > > -- Leon > > > > > > This message posted from opensolaris.org > > _______________________________________________ > > zfs-discuss mailing list > > zfs-discuss at opensolaris.org > > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Leon Koll
2007-Mar-01 17:31 UTC
[zfs-discuss] Re: Why number of NFS threads jumps to the max value?
Hi Jim, here are the answers to your questions :> > What size and type of server?SUNW,Sun-Fire-V240, Memory size: 2048 Megabytes> What size and type of storage?SAN-attached storage array, dual-path 2GB FC connection 4 LUNs 96GB each : # mpathadm list lu /dev/rdsk/c3t001738010140003Dd0s2 Total Path Count: 2 Operational Path Count: 2 /dev/rdsk/c3t001738010140003Cd0s2 Total Path Count: 2 Operational Path Count: 2 /dev/rdsk/c3t001738010140003Bd0s2 Total Path Count: 2 Operational Path Count: 2 /dev/rdsk/c3t001738010140002Dd0s2 Total Path Count: 2 Operational Path Count: 2> What release of Solaris?Solaris 10 11/06 s10s_u3wos_05a SPARC, all patches installed> What how may networks, and what type?2 GBE interfaces point-to-point between NFS client and server: bge1-bge1, bge2-bge2> What is being used to generate the load for the testing?spec.org SFS97_R1 http://www.spec.org/sfs97r1/> What is the zpool configuration?# zpool status tank1 pool: tank1 state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM tank1 ONLINE 0 0 0 c3t001738010140003Bd0 ONLINE 0 0 0 c3t001738010140003Cd0 ONLINE 0 0 0 c3t001738010140002Dd0 ONLINE 0 0 0 c3t001738010140003Dd0 ONLINE 0 0 0 errors: No known data errors I''ve created 160 ZFS''es in this zpool.> What do the system stats look like while under load > (e.g. mpstat), and how > to they change when you see this behavior?normal load CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 0 0 2 29 24 4236 0 589 489 0 38 1 10 0 89 1 0 0 2 1772 1666 4034 3 604 512 0 25 0 11 0 89 CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 0 0 97 2849 2205 4183 23 287 614 0 12 0 75 0 25 1 0 0 639 5406 5206 3877 56 287 638 0 18 0 75 0 25 CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 0 0 504 198 122 4955 0 730 246 0 57 0 15 0 85 1 0 0 6 2154 2049 4512 5 697 345 0 166 0 16 0 84 CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 0 0 5 39 31 4108 2 603 261 0 44 0 10 0 90 1 0 0 4 1899 1791 3967 6 604 327 0 51 1 12 0 87 When I see this behavior: CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 8 0 123 1065 836 3677 12 687 426 2 85 0 35 0 65 1 7 0 276 2444 2266 3602 23 686 464 3 79 0 36 0 64 CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 0 0 160 1400 971 6710 7 1395 608 3 12 0 59 0 41 1 0 0 412 3733 3499 6596 28 1390 637 3 33 0 63 0 37 prstat | grep nfsd 592 daemon 12M 10M sleep 60 -20 0:57:57 24% nfsd/1028 592 daemon 12M 10M sleep 60 -20 0:57:59 24% nfsd/1028 592 daemon 12M 10M sleep 60 -20 0:58:02 25% nfsd/1028> What does "zpool iostat <zpool_name> 1" data look > like while under load?tank1 60.5G 321G 2.29K 675 144M 4.00M tank1 60.5G 321G 496 2.45K 29.0M 9.26M tank1 60.5G 321G 655 2.64K 40.6M 14.9M tank1 60.5G 321G 2.73K 534 174M 6.96M tank1 60.5G 321G 2.42K 719 151M 4.59M tank1 60.5G 321G 32 2.91K 1.99M 14.8M tank1 60.5G 321G 12 2.46K 824K 12.6M tank1 60.5G 321G 63 1.83K 3.84M 13.9M tank1 60.5G 321G 2.50K 903 150M 14.3M tank1 60.5G 321G 2.93K 414 180M 10.1M tank1 60.5G 321G 1.95K 998 124M 5.78M tank1 60.5G 321G 164 2.65K 10.0M 12.1M tank1 60.5G 321G 959 1.98K 58.6M 12.3M tank1 60.5G 321G 2.82K 477 178M 7.56M tank1 60.5G 321G 338 2.46K 20.8M 13.7M tank1 60.5G 321G 166 3.01K 10.6M 17.8M> Are you collecting nfsstat data - what is the rate of > incoming NFS ops?Server nfs: calls badcalls 18673 0 Version 3: (19090 calls) null getattr setattr lookup access readlink 0 0% 2178 11% 209 1% 5214 27% 1343 7% 1368 7% read write create mkdir symlink mknod 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% remove rmdir rename link readdir readdirplus 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% fsstat fsinfo pathconf commit 0 0% 0 0% 0 0% 0 0%> Can you characterize the load - read/write data > intensive, metadata intensive?There is a standard mix of NFS ops defined by spec.org SFS : SFS Aggregate Results for 1 Client(s), Thu Mar 1 18:28:04 2007 NFS Protocol Version 3 ------------------------------------------------------------------------------ NFS Target Actual NFS NFS Mean Std Dev Std Error Pcnt Op NFS NFS Op Op Response Response of Mean,95% of Type Mix Mix Success Error Time Time Confidence Total Pcnt Pcnt Count Count Msec/Op Msec/Op +- Msec/Op Time ------------------------------------------------------------------------------ getattr 11% 11.1% 66693 0 1.18 1.50 0.01 5.9% setattr 1% 1.0% 6097 0 3.54 3.46 0.05 1.6% lookup 27% 27.5% 164643 0 1.98 2.62 0.01 24.5% readlink 7% 7.1% 42300 0 0.94 0.99 0.01 3.0% read 18% 17.6% 105538 0 3.43 3.60 0.01 27.3% write 9% 8.8% 52640 0 2.68 3.03 0.01 10.6% create 1% 1.0% 6194 0 4.50 6.45 0.06 2.1% remove 1% 1.0% 5922 0 5.86 6.22 0.06 2.6% readdir 2% 2.0% 12261 0 2.46 1.85 0.02 2.3% fsstat 1% 1.0% 6031 0 0.83 0.65 0.02 0.4% access 7% 7.2% 42826 0 0.85 0.69 0.01 2.7% commit 5% 4.5% 27218 0 2.73 3.47 0.02 5.6% fsinfo 1% 1.0% 6010 0 0.84 0.72 0.02 0.4% readdirplus 9% 9.1% 54513 0 2.66 2.26 0.01 10.9% ------------------------------------------------------------------------------ [i]( How can I use fixed-width font here or <pre></pre> tags ??? )[/i]> > Are the client machines Solaris, or something else?Yes, the same configuration as on the server> > Does this last for seconds, minutes, tens-of-minutes?When it becomes bad (in my case when the requested number of IOPS is 4000), it never gets back to normal behavior during the load. [b]Just for comparison: 160 UFS''es on the same 4 LUNs concatenated by SVM and didvided into 160 soft partitions works fine under load of 11000 SFS IOPS. And number of NFS threads never jumps so high.[/b]> Does the system > remain in this > state indefinitely until reboot, or does it > normalize?Yes, remains in this state if the load doesn''t stop.> > Can you consistently reproduce this problem?Yes I can. Thanks in advance, -- Leon This message posted from opensolaris.org
Leon Koll
2007-Mar-02 22:40 UTC
[zfs-discuss] Why number of NFS threads jumps to the max value?
On 2/28/07, Roch - PAE <Roch.Bourbonnais at sun.com> wrote:> > > http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6467988 > > NFSD threads are created on a demand spike (all of them > waiting on I/O) but then tend to stick around servicing > moderate loads. > > -rHello Roch, It''s not my case. NFS stops to service after some point. And the reason is in ZFS. It never happens with NFS/UFS. Shortly, my scenario: 1st SFS run, 2000 requested IOPS. NFS is fine, ;low number of threads. 2st SFS run, 4000 requested IOPS. NFS cannot serve all requests, no of threads jumps to max 3rd SFS run, 2000 requested IOPS. NFS cannot serve all requests, no of threads jumps to max. System cannot get back to the same results under equal load (1st and 3rd). Reboot between 2nd and 3rd doesn''t help. The only persistent thing is a directory structure that was created during the 2nd run (in SFS higher requested load -> more directories/files created). I am sure it''s a bug. I need help. I don''t care that ZFS works N times worse than UFS. I really care that after heavy load everything is totally screwed. Thanks, -- Leon
Roch - PAE
2007-Mar-05 12:09 UTC
[zfs-discuss] Why number of NFS threads jumps to the max value?
Leon Koll writes: > On 2/28/07, Roch - PAE <Roch.Bourbonnais at sun.com> wrote: > > > > > > http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6467988 > > > > NFSD threads are created on a demand spike (all of them > > waiting on I/O) but then tend to stick around servicing > > moderate loads. > > > > -r > > Hello Roch, > It''s not my case. NFS stops to service after some point. And the > reason is in ZFS. It never happens with NFS/UFS. > Shortly, my scenario: > 1st SFS run, 2000 requested IOPS. NFS is fine, ;low number of threads. > 2st SFS run, 4000 requested IOPS. NFS cannot serve all requests, no of > threads jumps to max > 3rd SFS run, 2000 requested IOPS. NFS cannot serve all requests, no of > threads jumps to max. > System cannot get back to the same results under equal load (1st and 3rd). > Reboot between 2nd and 3rd doesn''t help. The only persistent thing is > a directory structure that was created during the 2nd run (in SFS > higher requested load -> more directories/files created). > I am sure it''s a bug. I need help. I don''t care that ZFS works N times > worse than UFS. I really care that after heavy load everything is > totally screwed. > > Thanks, > -- Leon Hi Leon, How much is the slowdown between 1st and 3rd ? How filled is the pool at each stage ? What does ''NFS stops to service'' mean ? -r
Leon Koll
2007-Mar-05 16:52 UTC
[zfs-discuss] Why number of NFS threads jumps to the max value?
On 3/5/07, Roch - PAE <Roch.Bourbonnais at sun.com> wrote:> > Leon Koll writes: > > On 2/28/07, Roch - PAE <Roch.Bourbonnais at sun.com> wrote: > > > > > > > > > http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6467988 > > > > > > NFSD threads are created on a demand spike (all of them > > > waiting on I/O) but then tend to stick around servicing > > > moderate loads. > > > > > > -r > > > > Hello Roch, > > It''s not my case. NFS stops to service after some point. And the > > reason is in ZFS. It never happens with NFS/UFS. > > Shortly, my scenario: > > 1st SFS run, 2000 requested IOPS. NFS is fine, ;low number of threads. > > 2st SFS run, 4000 requested IOPS. NFS cannot serve all requests, no of > > threads jumps to max > > 3rd SFS run, 2000 requested IOPS. NFS cannot serve all requests, no of > > threads jumps to max. > > System cannot get back to the same results under equal load (1st and 3rd). > > Reboot between 2nd and 3rd doesn''t help. The only persistent thing is > > a directory structure that was created during the 2nd run (in SFS > > higher requested load -> more directories/files created). > > I am sure it''s a bug. I need help. I don''t care that ZFS works N times > > worse than UFS. I really care that after heavy load everything is > > totally screwed. > > > > Thanks, > > -- Leon > > Hi Leon, > > How much is the slowdown between 1st and 3rd ? How filled isTypical case is: 1st: 1996 IOPS, latency 2.7 3rd: 1375 IOPS, latency 37.9> the pool at each stage ? What does ''NFS stops to service'' > mean ?There is a lot of error messages on the NFS(SFS) client : sfs352: too many failed RPC calls - 416 good 27 bad sfs3132: too many failed RPC calls - 302 good 27 bad sfs3109: too many failed RPC calls - 533 good 31 bad sfs353: too many failed RPC calls - 301 good 28 bad sfs3144: too many failed RPC calls - 305 good 25 bad sfs3121: too many failed RPC calls - 311 good 30 bad sfs370: too many failed RPC calls - 315 good 27 bad Thanks, -- Leon
Roch - PAE
2007-Mar-05 16:59 UTC
[zfs-discuss] Why number of NFS threads jumps to the max value?
Leon Koll writes: > On 3/5/07, Roch - PAE <Roch.Bourbonnais at sun.com> wrote: > > > > Leon Koll writes: > > > On 2/28/07, Roch - PAE <Roch.Bourbonnais at sun.com> wrote: > > > > > > > > > > > > http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6467988 > > > > > > > > NFSD threads are created on a demand spike (all of them > > > > waiting on I/O) but then tend to stick around servicing > > > > moderate loads. > > > > > > > > -r > > > > > > Hello Roch, > > > It''s not my case. NFS stops to service after some point. And the > > > reason is in ZFS. It never happens with NFS/UFS. > > > Shortly, my scenario: > > > 1st SFS run, 2000 requested IOPS. NFS is fine, ;low number of threads. > > > 2st SFS run, 4000 requested IOPS. NFS cannot serve all requests, no of > > > threads jumps to max > > > 3rd SFS run, 2000 requested IOPS. NFS cannot serve all requests, no of > > > threads jumps to max. > > > System cannot get back to the same results under equal load (1st and 3rd). > > > Reboot between 2nd and 3rd doesn''t help. The only persistent thing is > > > a directory structure that was created during the 2nd run (in SFS > > > higher requested load -> more directories/files created). > > > I am sure it''s a bug. I need help. I don''t care that ZFS works N times > > > worse than UFS. I really care that after heavy load everything is > > > totally screwed. > > > > > > Thanks, > > > -- Leon > > > > Hi Leon, > > > > How much is the slowdown between 1st and 3rd ? How filled is > > Typical case is: > 1st: 1996 IOPS, latency 2.7 > 3rd: 1375 IOPS, latency 37.9 > The large latency increase is the side effect of requesting more than what can be delivered. Queue builds up and latency follow. So it should not be the primary focus IMO. The Decrease in IOPS is the primary problem. One hypothesis is that over the life of the FS we''re moving toward spreading access to the full disk platter. We can imagine some fragmentation hitting as well. I''m not sure how I''d test both hypothesis. > > the pool at each stage ? What does ''NFS stops to service'' > > mean ? > > There is a lot of error messages on the NFS(SFS) client : > sfs352: too many failed RPC calls - 416 good 27 bad > sfs3132: too many failed RPC calls - 302 good 27 bad > sfs3109: too many failed RPC calls - 533 good 31 bad > sfs353: too many failed RPC calls - 301 good 28 bad > sfs3144: too many failed RPC calls - 305 good 25 bad > sfs3121: too many failed RPC calls - 311 good 30 bad > sfs370: too many failed RPC calls - 315 good 27 bad > Can this be timing out or queue full drops ? Might be a side effect of SFS requesting more than what can be delivered. > Thanks, > -- Leon
Leon Koll
2007-Mar-05 17:17 UTC
[zfs-discuss] Why number of NFS threads jumps to the max value?
On 3/5/07, Roch - PAE <Roch.Bourbonnais at sun.com> wrote:> > Leon Koll writes: > > > On 3/5/07, Roch - PAE <Roch.Bourbonnais at sun.com> wrote: > > > > > > Leon Koll writes: > > > > On 2/28/07, Roch - PAE <Roch.Bourbonnais at sun.com> wrote: > > > > > > > > > > > > > > > http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6467988 > > > > > > > > > > NFSD threads are created on a demand spike (all of them > > > > > waiting on I/O) but then tend to stick around servicing > > > > > moderate loads. > > > > > > > > > > -r > > > > > > > > Hello Roch, > > > > It''s not my case. NFS stops to service after some point. And the > > > > reason is in ZFS. It never happens with NFS/UFS. > > > > Shortly, my scenario: > > > > 1st SFS run, 2000 requested IOPS. NFS is fine, ;low number of threads. > > > > 2st SFS run, 4000 requested IOPS. NFS cannot serve all requests, no of > > > > threads jumps to max > > > > 3rd SFS run, 2000 requested IOPS. NFS cannot serve all requests, no of > > > > threads jumps to max. > > > > System cannot get back to the same results under equal load (1st and 3rd). > > > > Reboot between 2nd and 3rd doesn''t help. The only persistent thing is > > > > a directory structure that was created during the 2nd run (in SFS > > > > higher requested load -> more directories/files created). > > > > I am sure it''s a bug. I need help. I don''t care that ZFS works N times > > > > worse than UFS. I really care that after heavy load everything is > > > > totally screwed. > > > > > > > > Thanks, > > > > -- Leon > > > > > > Hi Leon, > > > > > > How much is the slowdown between 1st and 3rd ? How filled is > > > > Typical case is: > > 1st: 1996 IOPS, latency 2.7 > > 3rd: 1375 IOPS, latency 37.9 > > > > The large latency increase is the side effect of requesting > more than what can be delivered. Queue builds up and latency > follow. So it should not be the primary focus IMO. The > Decrease in IOPS is the primary problem. > > One hypothesis is that over the life of the FS we''re moving > toward spreading access to the full disk platter. We can > imagine some fragmentation hitting as well. I''m not sure > how I''d test both hypothesis. > > > > the pool at each stage ? What does ''NFS stops to service'' > > > mean ? > > > > There is a lot of error messages on the NFS(SFS) client : > > sfs352: too many failed RPC calls - 416 good 27 bad > > sfs3132: too many failed RPC calls - 302 good 27 bad > > sfs3109: too many failed RPC calls - 533 good 31 bad > > sfs353: too many failed RPC calls - 301 good 28 bad > > sfs3144: too many failed RPC calls - 305 good 25 bad > > sfs3121: too many failed RPC calls - 311 good 30 bad > > sfs370: too many failed RPC calls - 315 good 27 bad > > > > Can this be timing out or queue full drops ? Might be a side > effect of SFS requesting more than what can be delivered.I don''t know was it timeouts or full drops. SFS marked such runs as INVALID. I can run whatever is needed to help to investigate the problem. If you have a D script that will tell us more, please send it to me. I appreciate your help. -- Leon
Spencer Shepler
2007-Mar-05 18:18 UTC
[zfs-discuss] Why number of NFS threads jumps to the max value?
On Mar 5, 2007, at 11:17 AM, Leon Koll wrote:> On 3/5/07, Roch - PAE <Roch.Bourbonnais at sun.com> wrote: >> >> Leon Koll writes: >> >> > On 3/5/07, Roch - PAE <Roch.Bourbonnais at sun.com> wrote: >> > > >> > > Leon Koll writes: >> > > > On 2/28/07, Roch - PAE <Roch.Bourbonnais at sun.com> wrote: >> > > > > >> > > > > >> > > > > http://bugs.opensolaris.org/bugdatabase/view_bug.do? >> bug_id=6467988 >> > > > > >> > > > > NFSD threads are created on a demand spike (all of >> them >> > > > > waiting on I/O) but then tend to stick around >> servicing >> > > > > moderate loads. >> > > > > >> > > > > -r >> > > > >> > > > Hello Roch, >> > > > It''s not my case. NFS stops to service after some point. >> And the >> > > > reason is in ZFS. It never happens with NFS/UFS. >> > > > Shortly, my scenario: >> > > > 1st SFS run, 2000 requested IOPS. NFS is fine, ;low number >> of threads. >> > > > 2st SFS run, 4000 requested IOPS. NFS cannot serve all >> requests, no of >> > > > threads jumps to max >> > > > 3rd SFS run, 2000 requested IOPS. NFS cannot serve all >> requests, no of >> > > > threads jumps to max. >> > > > System cannot get back to the same results under equal >> load (1st and 3rd). >> > > > Reboot between 2nd and 3rd doesn''t help. The only >> persistent thing is >> > > > a directory structure that was created during the 2nd run >> (in SFS >> > > > higher requested load -> more directories/files created). >> > > > I am sure it''s a bug. I need help. I don''t care that ZFS >> works N times >> > > > worse than UFS. I really care that after heavy load >> everything is >> > > > totally screwed. >> > > > >> > > > Thanks, >> > > > -- Leon >> > > >> > > Hi Leon, >> > > >> > > How much is the slowdown between 1st and 3rd ? How filled is >> > >> > Typical case is: >> > 1st: 1996 IOPS, latency 2.7 >> > 3rd: 1375 IOPS, latency 37.9 >> > >> >> The large latency increase is the side effect of requesting >> more than what can be delivered. Queue builds up and latency >> follow. So it should not be the primary focus IMO. The >> Decrease in IOPS is the primary problem. >> >> One hypothesis is that over the life of the FS we''re moving >> toward spreading access to the full disk platter. We can >> imagine some fragmentation hitting as well. I''m not sure >> how I''d test both hypothesis. >> >> > > the pool at each stage ? What does ''NFS stops to service'' >> > > mean ? >> > >> > There is a lot of error messages on the NFS(SFS) client : >> > sfs352: too many failed RPC calls - 416 good 27 bad >> > sfs3132: too many failed RPC calls - 302 good 27 bad >> > sfs3109: too many failed RPC calls - 533 good 31 bad >> > sfs353: too many failed RPC calls - 301 good 28 bad >> > sfs3144: too many failed RPC calls - 305 good 25 bad >> > sfs3121: too many failed RPC calls - 311 good 30 bad >> > sfs370: too many failed RPC calls - 315 good 27 bad >> > >> >> Can this be timing out or queue full drops ? Might be a side >> effect of SFS requesting more than what can be delivered. > > I don''t know was it timeouts or full drops. SFS marked such runs as > INVALID. > I can run whatever is needed to help to investigate the problem. If > you have a D script that will tell us more, please send it to me. > I appreciate your help.The failed RPCs are indeed a result of the SFS client timing out the requests it has made to the server. The server is being overloaded for its capabilities and the benchmark results show that. I agree with Roch that as the SFS benchmark adds more data to the filesystems that additional latency is added and for this particular configuration and the server is being over-driven. The helpful thing would be to run smaller increments in the benchmark to determine where the response time increases beyond what the SFS workload can handle. There have been a number of changes in ZFS recently that should help with SFS performance measurement but fundamentally it all depends on the configuration of the server (number of spindles and CPU available). So there may be a limit that is being reached based on the hardware configuration. What is your real goal here, Leon? Are you trying to gather SFS data to fit into sizing of a particular solution or just trying to gather performance results for other general comparisons? There are certainly better benchmarks than SFS for either sizing and comparison reasons. Spencer
Leon Koll
2007-Mar-05 22:01 UTC
[zfs-discuss] Why number of NFS threads jumps to the max value?
On 3/5/07, Spencer Shepler <Spencer.Shepler at sun.com> wrote:> > On Mar 5, 2007, at 11:17 AM, Leon Koll wrote: > > > On 3/5/07, Roch - PAE <Roch.Bourbonnais at sun.com> wrote: > >> > >> Leon Koll writes: > >> > >> > On 3/5/07, Roch - PAE <Roch.Bourbonnais at sun.com> wrote: > >> > > > >> > > Leon Koll writes: > >> > > > On 2/28/07, Roch - PAE <Roch.Bourbonnais at sun.com> wrote: > >> > > > > > >> > > > > > >> > > > > http://bugs.opensolaris.org/bugdatabase/view_bug.do? > >> bug_id=6467988 > >> > > > > > >> > > > > NFSD threads are created on a demand spike (all of > >> them > >> > > > > waiting on I/O) but then tend to stick around > >> servicing > >> > > > > moderate loads. > >> > > > > > >> > > > > -r > >> > > > > >> > > > Hello Roch, > >> > > > It''s not my case. NFS stops to service after some point. > >> And the > >> > > > reason is in ZFS. It never happens with NFS/UFS. > >> > > > Shortly, my scenario: > >> > > > 1st SFS run, 2000 requested IOPS. NFS is fine, ;low number > >> of threads. > >> > > > 2st SFS run, 4000 requested IOPS. NFS cannot serve all > >> requests, no of > >> > > > threads jumps to max > >> > > > 3rd SFS run, 2000 requested IOPS. NFS cannot serve all > >> requests, no of > >> > > > threads jumps to max. > >> > > > System cannot get back to the same results under equal > >> load (1st and 3rd). > >> > > > Reboot between 2nd and 3rd doesn''t help. The only > >> persistent thing is > >> > > > a directory structure that was created during the 2nd run > >> (in SFS > >> > > > higher requested load -> more directories/files created). > >> > > > I am sure it''s a bug. I need help. I don''t care that ZFS > >> works N times > >> > > > worse than UFS. I really care that after heavy load > >> everything is > >> > > > totally screwed. > >> > > > > >> > > > Thanks, > >> > > > -- Leon > >> > > > >> > > Hi Leon, > >> > > > >> > > How much is the slowdown between 1st and 3rd ? How filled is > >> > > >> > Typical case is: > >> > 1st: 1996 IOPS, latency 2.7 > >> > 3rd: 1375 IOPS, latency 37.9 > >> > > >> > >> The large latency increase is the side effect of requesting > >> more than what can be delivered. Queue builds up and latency > >> follow. So it should not be the primary focus IMO. The > >> Decrease in IOPS is the primary problem. > >> > >> One hypothesis is that over the life of the FS we''re moving > >> toward spreading access to the full disk platter. We can > >> imagine some fragmentation hitting as well. I''m not sure > >> how I''d test both hypothesis. > >> > >> > > the pool at each stage ? What does ''NFS stops to service'' > >> > > mean ? > >> > > >> > There is a lot of error messages on the NFS(SFS) client : > >> > sfs352: too many failed RPC calls - 416 good 27 bad > >> > sfs3132: too many failed RPC calls - 302 good 27 bad > >> > sfs3109: too many failed RPC calls - 533 good 31 bad > >> > sfs353: too many failed RPC calls - 301 good 28 bad > >> > sfs3144: too many failed RPC calls - 305 good 25 bad > >> > sfs3121: too many failed RPC calls - 311 good 30 bad > >> > sfs370: too many failed RPC calls - 315 good 27 bad > >> > > >> > >> Can this be timing out or queue full drops ? Might be a side > >> effect of SFS requesting more than what can be delivered. > > > > I don''t know was it timeouts or full drops. SFS marked such runs as > > INVALID. > > I can run whatever is needed to help to investigate the problem. If > > you have a D script that will tell us more, please send it to me. > > I appreciate your help. > > The failed RPCs are indeed a result of the SFS client timing out > the requests it has made to the server. The server is being > overloaded for its capabilities and the benchmark results > show that. I agree with Roch that as the SFS benchmark adds > more data to the filesystems that additional latency is > added and for this particular configuration and the > server is being over-driven. > > The helpful thing would be to run smaller increments in the > benchmark to determine where the response time increases > beyond what the SFS workload can handle. > > There have been a number of changes in ZFS recently that should > help with SFS performance measurement but fundamentally it > all depends on the configuration of the server (number of spindles > and CPU available). So there may be a limit that is being > reached based on the hardware configuration. > > What is your real goal here, Leon? Are you trying to gather SFS > data to fit into sizing of a particular solution or just trying > to gather performance results for other general comparisons?Spencer, I am using SFS benchmark to emulate the real-world environment for the NAS solution that I''ve built. SFS is able to create as many processes (each one emulating separate client) as one needs plus it''s rich of meta operations. My real goal is: quiet peaceful nights after my solution will start to work in the production :) What I see now: after some step the directory structure/on-disk layout of ZFS? becomes an obstacle that cannot be passed. I don''t want that this will happen in the field, that''s it. And I am sure we have a bug case here.> There are certainly better benchmarks than SFS for either > sizing and comparison reasons.I haven''t found anything better: client-independent, multi-proc./multi-threaded, meta-rich, comparable. An obvious drawback is: $$. I think that a free replacement of it is almost unknown and underestimated fstress ( http://www.cs.duke.edu/ari/fstress/ ). Which ones are your candidates? Thanks, -- Leon