zfs discuss - Feb 2007 - Why number of NFS threads jumps to the max value?

Hello, gurus
I need your help. During the benchmark test of NFS-shared ZFS file systems at
some moment the number of NFS threads jumps to the maximal value, 1027
(NFSD_SERVERS was set to 1024). The latency also grows and the number of IOPS is
going down.
I''ve collected the output of
echo "::pgrep nfsd | ::walk thread | ::findstack -v" | mdb -k
that can be seen here:
http://tinyurl.com/yrvn4z

Could you please look at it and tell me what''s wrong with my NFS
server.
Appreciate,
-- Leon
 
 
This message posted from opensolaris.org

You don''t honestly, really, reasonably, expect someone, anyone, to look
at the stack
trace of a few  hundred threads, and post something along the lines of 
"This is what is
wrong with your NFS server".....Do you? Without any other information at 
all?

We''re here to help, but please reset your expectations around our 
abilities to
root-cause pathological behavior based an almost no information.

What size and type of server?
What size and type of storage?
What release of Solaris?
What how may networks, and what type?
What is being used to generate the load for the testing?
What is the zpool configuration?
What do the system stats look like while under load (e.g. mpstat), and how
to they change when you see this behavior?
What does "zpool iostat <zpool_name> 1" data look like while
under load?
Are you collecting nfsstat data - what is the rate of incoming NFS ops?
Can you characterize the load - read/write data intensive, metadata 
intensive?

Are the client machines Solaris, or something else?

Does this last for seconds, minutes, tens-of-minutes? Does the system 
remain in this
state indefinitely until reboot, or does it normalize?

Can you consistently reproduce this problem?

/jim


Leon Koll wrote:
> Hello, gurus
> I need your help. During the benchmark test of NFS-shared ZFS file systems
at some moment the number of NFS threads jumps to the maximal value, 1027
(NFSD_SERVERS was set to 1024). The latency also grows and the number of IOPS is
going down.
> I''ve collected the output of
> echo "::pgrep nfsd | ::walk thread | ::findstack -v" | mdb -k
> that can be seen here:
> http://tinyurl.com/yrvn4z
>
> Could you please look at it and tell me what''s wrong with my NFS
server.
> Appreciate,
> -- Leon
>  
>  
> This message posted from opensolaris.org
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>

>
> You don''t honestly, really, reasonably, expect someone, anyone, to
look
> at the stack
  well of course he does :-)

  and I looked at it .. all of it and I can tell exactly what the problem is
  but I''m not gonna say because its a trick question.
  so there.

Dennis

There are 517 threads in the zfs_write code path, so there seems to be
a fair amount of write activity happening.

The server iostat and zpool iostat data would be interesting, along with
the backend storage configuration information - how many spindles, and
what kind of zpool config...

/jim


Leon Koll wrote:
> Hello, gurus
> I need your help. During the benchmark test of NFS-shared ZFS file systems
at some moment the number of NFS threads jumps to the maximal value, 1027
(NFSD_SERVERS was set to 1024). The latency also grows and the number of IOPS is
going down.
> I''ve collected the output of
> echo "::pgrep nfsd | ::walk thread | ::findstack -v" | mdb -k
> that can be seen here:
> http://tinyurl.com/yrvn4z
>
> Could you please look at it and tell me what''s wrong with my NFS
server.
> Appreciate,
> -- Leon
>  
>  
> This message posted from opensolaris.org
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>

http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6467988

NFSD  threads are created  on a  demand  spike (all of  them
waiting  on I/O) but then    tend to stick around  servicing
moderate loads.

-r


 > 
 > Leon Koll wrote:
 > > Hello, gurus
 > > I need your help. During the benchmark test of NFS-shared ZFS file
systems at some moment the number of NFS threads jumps to the maximal value,
1027 (NFSD_SERVERS was set to 1024). The latency also grows and the number of
IOPS is going down.
 > > I''ve collected the output of
 > > echo "::pgrep nfsd | ::walk thread | ::findstack -v" | mdb
-k
 > > that can be seen here:
 > > http://tinyurl.com/yrvn4z
 > >
 > > Could you please look at it and tell me what''s wrong with my
NFS server.
 > > Appreciate,
 > > -- Leon
 > >  
 > >  
 > > This message posted from opensolaris.org
 > > _______________________________________________
 > > zfs-discuss mailing list
 > > zfs-discuss at opensolaris.org
 > > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
 > >   
 > _______________________________________________
 > zfs-discuss mailing list
 > zfs-discuss at opensolaris.org
 > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Hi Jim,
here are the answers  to your questions :
> 
> What size and type of server?
SUNW,Sun-Fire-V240, Memory size: 2048 Megabytes
> What size and type of storage?
SAN-attached storage array, dual-path 2GB FC connection
4 LUNs 96GB each :
# mpathadm list lu
        /dev/rdsk/c3t001738010140003Dd0s2
                Total Path Count: 2
                Operational Path Count: 2
        /dev/rdsk/c3t001738010140003Cd0s2
                Total Path Count: 2
                Operational Path Count: 2
        /dev/rdsk/c3t001738010140003Bd0s2
                Total Path Count: 2
                Operational Path Count: 2
        /dev/rdsk/c3t001738010140002Dd0s2
                Total Path Count: 2
                Operational Path Count: 2
> What release of Solaris?
Solaris 10 11/06 s10s_u3wos_05a SPARC,
all patches installed
> What how may networks, and what type?
2 GBE interfaces point-to-point between NFS client and server:
bge1-bge1, bge2-bge2
> What is being used to generate the load for the testing?
spec.org SFS97_R1  http://www.spec.org/sfs97r1/
> What is the zpool configuration?
# zpool status tank1
  pool: tank1
 state: ONLINE
 scrub: none requested
config:

        NAME                     STATE     READ WRITE CKSUM
        tank1                    ONLINE       0     0     0
          c3t001738010140003Bd0  ONLINE       0     0     0
          c3t001738010140003Cd0  ONLINE       0     0     0
          c3t001738010140002Dd0  ONLINE       0     0     0
          c3t001738010140003Dd0  ONLINE       0     0     0

errors: No known data errors

I''ve created 160 ZFS''es in this zpool.
> What do the system stats look like while under load
> (e.g. mpstat), and how
> to they change when you see this behavior?
normal load
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0    0   0    2    29   24 4236    0  589  489    0    38    1  10   0  89
  1    0   0    2  1772 1666 4034    3  604  512    0    25    0  11   0  89
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0    0   0   97  2849 2205 4183   23  287  614    0    12    0  75   0  25
  1    0   0  639  5406 5206 3877   56  287  638    0    18    0  75   0  25
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0    0   0  504   198  122 4955    0  730  246    0    57    0  15   0  85
  1    0   0    6  2154 2049 4512    5  697  345    0   166    0  16   0  84
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0    0   0    5    39   31 4108    2  603  261    0    44    0  10   0  90
  1    0   0    4  1899 1791 3967    6  604  327    0    51    1  12   0  87

When I see this behavior:
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0    8   0  123  1065  836 3677   12  687  426    2    85    0  35   0  65
  1    7   0  276  2444 2266 3602   23  686  464    3    79    0  36   0  64
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0    0   0  160  1400  971 6710    7 1395  608    3    12    0  59   0  41
  1    0   0  412  3733 3499 6596   28 1390  637    3    33    0  63   0  37

prstat | grep nfsd
   592 daemon     12M   10M sleep   60  -20   0:57:57  24% nfsd/1028
   592 daemon     12M   10M sleep   60  -20   0:57:59  24% nfsd/1028
   592 daemon     12M   10M sleep   60  -20   0:58:02  25% nfsd/1028
> What does "zpool iostat <zpool_name> 1" data look
> like while under load?
tank1       60.5G   321G  2.29K    675   144M  4.00M
tank1       60.5G   321G    496  2.45K  29.0M  9.26M
tank1       60.5G   321G    655  2.64K  40.6M  14.9M
tank1       60.5G   321G  2.73K    534   174M  6.96M
tank1       60.5G   321G  2.42K    719   151M  4.59M
tank1       60.5G   321G     32  2.91K  1.99M  14.8M
tank1       60.5G   321G     12  2.46K   824K  12.6M
tank1       60.5G   321G     63  1.83K  3.84M  13.9M
tank1       60.5G   321G  2.50K    903   150M  14.3M
tank1       60.5G   321G  2.93K    414   180M  10.1M
tank1       60.5G   321G  1.95K    998   124M  5.78M
tank1       60.5G   321G    164  2.65K  10.0M  12.1M
tank1       60.5G   321G    959  1.98K  58.6M  12.3M
tank1       60.5G   321G  2.82K    477   178M  7.56M
tank1       60.5G   321G    338  2.46K  20.8M  13.7M
tank1       60.5G   321G    166  3.01K  10.6M  17.8M
> Are you collecting nfsstat data - what is the rate of
> incoming NFS ops?
Server nfs:
calls     badcalls  
18673     0         
Version 3: (19090 calls)
null        getattr     setattr     lookup      access      readlink    
0 0%        2178 11%    209 1%      5214 27%    1343 7%     1368 7%     
read        write       create      mkdir       symlink     mknod       
0 0%        0 0%        0 0%        0 0%        0 0%        0 0%        
remove      rmdir       rename      link        readdir     readdirplus 
0 0%        0 0%        0 0%        0 0%        0 0%        0 0%        
fsstat      fsinfo      pathconf    commit      
0 0%        0 0%        0 0%        0 0%        
> Can you characterize the load - read/write data
> intensive, metadata intensive?
There is a standard mix of NFS ops defined by spec.org SFS :

SFS Aggregate Results for 1 Client(s), Thu Mar  1 18:28:04 2007
NFS Protocol Version 3
------------------------------------------------------------------------------
NFS         Target Actual     NFS    NFS    Mean    Std Dev  Std Error   Pcnt
Op           NFS    NFS       Op     Op    Response Response of Mean,95%  of
Type         Mix    Mix     Success Error   Time     Time    Confidence  Total
             Pcnt   Pcnt     Count  Count  Msec/Op  Msec/Op  +- Msec/Op  Time
------------------------------------------------------------------------------
getattr      11%   11.1%     66693     0     1.18     1.50      0.01      5.9%
setattr       1%    1.0%      6097     0     3.54     3.46      0.05      1.6%
lookup       27%   27.5%    164643     0     1.98     2.62      0.01     24.5%
readlink      7%    7.1%     42300     0     0.94     0.99      0.01      3.0%
read         18%   17.6%    105538     0     3.43     3.60      0.01     27.3%
write         9%    8.8%     52640     0     2.68     3.03      0.01     10.6%
create        1%    1.0%      6194     0     4.50     6.45      0.06      2.1%
remove        1%    1.0%      5922     0     5.86     6.22      0.06      2.6%
readdir       2%    2.0%     12261     0     2.46     1.85      0.02      2.3%
fsstat        1%    1.0%      6031     0     0.83     0.65      0.02      0.4%
access        7%    7.2%     42826     0     0.85     0.69      0.01      2.7%
commit        5%    4.5%     27218     0     2.73     3.47      0.02      5.6%
fsinfo        1%    1.0%      6010     0     0.84     0.72      0.02      0.4%
readdirplus   9%    9.1%     54513     0     2.66     2.26      0.01     10.9%
------------------------------------------------------------------------------
[i]( How can I use fixed-width font here or <pre></pre> tags ???
)[/i]
> 
> Are the client machines Solaris, or something else?
Yes, the same configuration as on the server
> 
> Does this last for seconds, minutes, tens-of-minutes?
When it becomes bad (in my case when the requested number of IOPS is 4000), it
never gets back to normal behavior during the load.

[b]Just for comparison: 160 UFS''es on the same 4 LUNs concatenated by
SVM and didvided into 160 soft partitions works fine under load of 11000 SFS
IOPS. And number of NFS threads never jumps so high.[/b]
> Does the system 
> remain in this
> state indefinitely until reboot, or does it
> normalize?
Yes, remains in this state if the load doesn''t
stop.
> 
> Can you consistently reproduce this problem?
Yes I can.

Thanks in advance,
-- Leon
 
 
This message posted from opensolaris.org

On 2/28/07, Roch - PAE <Roch.Bourbonnais at sun.com>
wrote:
>
>
> http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6467988
>
> NFSD  threads are created  on a  demand  spike (all of  them
> waiting  on I/O) but then    tend to stick around  servicing
> moderate loads.
>
> -r
Hello Roch,
It''s not my case. NFS stops to service after some point. And the
reason is in ZFS. It never happens with NFS/UFS.
Shortly, my scenario:
1st SFS run, 2000 requested IOPS. NFS is fine, ;low number of threads.
2st SFS run, 4000 requested IOPS. NFS cannot serve all requests, no of
threads jumps to max
3rd SFS run, 2000 requested IOPS. NFS cannot serve all requests, no of
threads jumps to max.
System cannot get back to the same results under equal load (1st and 3rd).
Reboot between 2nd and 3rd doesn''t help. The only persistent thing is
a directory structure that was created during the 2nd run (in SFS
higher requested load -> more directories/files created).
I am sure it''s a bug. I need help. I don''t care that ZFS works
N times
worse than UFS. I really care that after heavy load everything is
totally screwed.

Thanks,
-- Leon

Leon Koll writes:
 > On 2/28/07, Roch - PAE <Roch.Bourbonnais at sun.com> wrote:
 > >
 > >
 > > http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6467988
 > >
 > > NFSD  threads are created  on a  demand  spike (all of  them
 > > waiting  on I/O) but then    tend to stick around  servicing
 > > moderate loads.
 > >
 > > -r
 > 
 > Hello Roch,
 > It''s not my case. NFS stops to service after some point. And the
 > reason is in ZFS. It never happens with NFS/UFS.
 > Shortly, my scenario:
 > 1st SFS run, 2000 requested IOPS. NFS is fine, ;low number of threads.
 > 2st SFS run, 4000 requested IOPS. NFS cannot serve all requests, no of
 > threads jumps to max
 > 3rd SFS run, 2000 requested IOPS. NFS cannot serve all requests, no of
 > threads jumps to max.
 > System cannot get back to the same results under equal load (1st and 3rd).
 > Reboot between 2nd and 3rd doesn''t help. The only persistent
thing is
 > a directory structure that was created during the 2nd run (in SFS
 > higher requested load -> more directories/files created).
 > I am sure it''s a bug. I need help. I don''t care that ZFS
works N times
 > worse than UFS. I really care that after heavy load everything is
 > totally screwed.
 > 
 > Thanks,
 > -- Leon

Hi Leon,

How much is the slowdown between 1st and 3rd ? How filled is 
the pool at each stage ? What does ''NFS stops to service''
mean ?

-r

On 3/5/07, Roch - PAE <Roch.Bourbonnais at sun.com>
wrote:
>
> Leon Koll writes:
>  > On 2/28/07, Roch - PAE <Roch.Bourbonnais at sun.com> wrote:
>  > >
>  > >
>  > >
http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6467988
>  > >
>  > > NFSD  threads are created  on a  demand  spike (all of  them
>  > > waiting  on I/O) but then    tend to stick around  servicing
>  > > moderate loads.
>  > >
>  > > -r
>  >
>  > Hello Roch,
>  > It''s not my case. NFS stops to service after some point. And
the
>  > reason is in ZFS. It never happens with NFS/UFS.
>  > Shortly, my scenario:
>  > 1st SFS run, 2000 requested IOPS. NFS is fine, ;low number of
threads.
>  > 2st SFS run, 4000 requested IOPS. NFS cannot serve all requests, no
of
>  > threads jumps to max
>  > 3rd SFS run, 2000 requested IOPS. NFS cannot serve all requests, no
of
>  > threads jumps to max.
>  > System cannot get back to the same results under equal load (1st and
3rd).
>  > Reboot between 2nd and 3rd doesn''t help. The only persistent
thing is
>  > a directory structure that was created during the 2nd run (in SFS
>  > higher requested load -> more directories/files created).
>  > I am sure it''s a bug. I need help. I don''t care
that ZFS works N times
>  > worse than UFS. I really care that after heavy load everything is
>  > totally screwed.
>  >
>  > Thanks,
>  > -- Leon
>
> Hi Leon,
>
> How much is the slowdown between 1st and 3rd ? How filled is
Typical case is:
1st: 1996 IOPS, latency  2.7
3rd: 1375 IOPS, latency 37.9
> the pool at each stage ? What does ''NFS stops to service''
> mean ?
There is a lot of error messages on the NFS(SFS) client :
sfs352: too many failed RPC calls - 416 good 27 bad
sfs3132: too many failed RPC calls - 302 good 27 bad
sfs3109: too many failed RPC calls - 533 good 31 bad
sfs353: too many failed RPC calls - 301 good 28 bad
sfs3144: too many failed RPC calls - 305 good 25 bad
sfs3121: too many failed RPC calls - 311 good 30 bad
sfs370: too many failed RPC calls - 315 good 27 bad

Thanks,
-- Leon

Leon Koll writes:

 > On 3/5/07, Roch - PAE <Roch.Bourbonnais at sun.com> wrote:
 > >
 > > Leon Koll writes:
 > >  > On 2/28/07, Roch - PAE <Roch.Bourbonnais at sun.com>
wrote:
 > >  > >
 > >  > >
 > >  > >
http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6467988
 > >  > >
 > >  > > NFSD  threads are created  on a  demand  spike (all of 
them
 > >  > > waiting  on I/O) but then    tend to stick around 
servicing
 > >  > > moderate loads.
 > >  > >
 > >  > > -r
 > >  >
 > >  > Hello Roch,
 > >  > It''s not my case. NFS stops to service after some
point. And the
 > >  > reason is in ZFS. It never happens with NFS/UFS.
 > >  > Shortly, my scenario:
 > >  > 1st SFS run, 2000 requested IOPS. NFS is fine, ;low number of
threads.
 > >  > 2st SFS run, 4000 requested IOPS. NFS cannot serve all
requests, no of
 > >  > threads jumps to max
 > >  > 3rd SFS run, 2000 requested IOPS. NFS cannot serve all
requests, no of
 > >  > threads jumps to max.
 > >  > System cannot get back to the same results under equal load
(1st and 3rd).
 > >  > Reboot between 2nd and 3rd doesn''t help. The only
persistent thing is
 > >  > a directory structure that was created during the 2nd run (in
SFS
 > >  > higher requested load -> more directories/files created).
 > >  > I am sure it''s a bug. I need help. I don''t
care that ZFS works N times
 > >  > worse than UFS. I really care that after heavy load everything
is
 > >  > totally screwed.
 > >  >
 > >  > Thanks,
 > >  > -- Leon
 > >
 > > Hi Leon,
 > >
 > > How much is the slowdown between 1st and 3rd ? How filled is
 > 
 > Typical case is:
 > 1st: 1996 IOPS, latency  2.7
 > 3rd: 1375 IOPS, latency 37.9
 > 

The large latency increase is the  side effect of requesting
more than what can be delivered. Queue builds up and latency
follow. So  it  should  not be  the  primary  focus IMO. The
Decrease in IOPS is the primary problem.

One hypothesis is that over the life of the FS we''re moving
toward spreading access to the full disk platter. We can
imagine some fragmentation hitting as well. I''m not sure
how I''d test both hypothesis.

 > > the pool at each stage ? What does ''NFS stops to
service''
 > > mean ?
 > 
 > There is a lot of error messages on the NFS(SFS) client :
 > sfs352: too many failed RPC calls - 416 good 27 bad
 > sfs3132: too many failed RPC calls - 302 good 27 bad
 > sfs3109: too many failed RPC calls - 533 good 31 bad
 > sfs353: too many failed RPC calls - 301 good 28 bad
 > sfs3144: too many failed RPC calls - 305 good 25 bad
 > sfs3121: too many failed RPC calls - 311 good 30 bad
 > sfs370: too many failed RPC calls - 315 good 27 bad
 >

Can this be timing out or queue full drops ? Might be a side 
effect of SFS requesting more than what can be delivered.

 > Thanks,
 > -- Leon

On 3/5/07, Roch - PAE <Roch.Bourbonnais at sun.com>
wrote:
>
> Leon Koll writes:
>
>  > On 3/5/07, Roch - PAE <Roch.Bourbonnais at sun.com> wrote:
>  > >
>  > > Leon Koll writes:
>  > >  > On 2/28/07, Roch - PAE <Roch.Bourbonnais at sun.com>
wrote:
>  > >  > >
>  > >  > >
>  > >  > >
http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6467988
>  > >  > >
>  > >  > > NFSD  threads are created  on a  demand  spike (all
of  them
>  > >  > > waiting  on I/O) but then    tend to stick around 
servicing
>  > >  > > moderate loads.
>  > >  > >
>  > >  > > -r
>  > >  >
>  > >  > Hello Roch,
>  > >  > It''s not my case. NFS stops to service after some
point. And the
>  > >  > reason is in ZFS. It never happens with NFS/UFS.
>  > >  > Shortly, my scenario:
>  > >  > 1st SFS run, 2000 requested IOPS. NFS is fine, ;low number
of threads.
>  > >  > 2st SFS run, 4000 requested IOPS. NFS cannot serve all
requests, no of
>  > >  > threads jumps to max
>  > >  > 3rd SFS run, 2000 requested IOPS. NFS cannot serve all
requests, no of
>  > >  > threads jumps to max.
>  > >  > System cannot get back to the same results under equal
load (1st and 3rd).
>  > >  > Reboot between 2nd and 3rd doesn''t help. The only
persistent thing is
>  > >  > a directory structure that was created during the 2nd run
(in SFS
>  > >  > higher requested load -> more directories/files
created).
>  > >  > I am sure it''s a bug. I need help. I
don''t care that ZFS works N times
>  > >  > worse than UFS. I really care that after heavy load
everything is
>  > >  > totally screwed.
>  > >  >
>  > >  > Thanks,
>  > >  > -- Leon
>  > >
>  > > Hi Leon,
>  > >
>  > > How much is the slowdown between 1st and 3rd ? How filled is
>  >
>  > Typical case is:
>  > 1st: 1996 IOPS, latency  2.7
>  > 3rd: 1375 IOPS, latency 37.9
>  >
>
> The large latency increase is the  side effect of requesting
> more than what can be delivered. Queue builds up and latency
> follow. So  it  should  not be  the  primary  focus IMO. The
> Decrease in IOPS is the primary problem.
>
> One hypothesis is that over the life of the FS we''re moving
> toward spreading access to the full disk platter. We can
> imagine some fragmentation hitting as well. I''m not sure
> how I''d test both hypothesis.
>
>  > > the pool at each stage ? What does ''NFS stops to
service''
>  > > mean ?
>  >
>  > There is a lot of error messages on the NFS(SFS) client :
>  > sfs352: too many failed RPC calls - 416 good 27 bad
>  > sfs3132: too many failed RPC calls - 302 good 27 bad
>  > sfs3109: too many failed RPC calls - 533 good 31 bad
>  > sfs353: too many failed RPC calls - 301 good 28 bad
>  > sfs3144: too many failed RPC calls - 305 good 25 bad
>  > sfs3121: too many failed RPC calls - 311 good 30 bad
>  > sfs370: too many failed RPC calls - 315 good 27 bad
>  >
>
> Can this be timing out or queue full drops ? Might be a side
> effect of SFS requesting more than what can be delivered.
I don''t know was it timeouts or full drops. SFS marked such runs as
INVALID.
I can run whatever is needed to help to investigate the problem. If
you have a D script that will tell us more, please send it to me.
I appreciate your help.

-- Leon

On Mar 5, 2007, at 11:17 AM, Leon Koll wrote:
> On 3/5/07, Roch - PAE <Roch.Bourbonnais at sun.com> wrote:
>>
>> Leon Koll writes:
>>
>>  > On 3/5/07, Roch - PAE <Roch.Bourbonnais at sun.com> wrote:
>>  > >
>>  > > Leon Koll writes:
>>  > >  > On 2/28/07, Roch - PAE <Roch.Bourbonnais at
sun.com> wrote:
>>  > >  > >
>>  > >  > >
>>  > >  > >
http://bugs.opensolaris.org/bugdatabase/view_bug.do?
>> bug_id=6467988
>>  > >  > >
>>  > >  > > NFSD  threads are created  on a  demand  spike
(all of
>> them
>>  > >  > > waiting  on I/O) but then    tend to stick around
>> servicing
>>  > >  > > moderate loads.
>>  > >  > >
>>  > >  > > -r
>>  > >  >
>>  > >  > Hello Roch,
>>  > >  > It''s not my case. NFS stops to service after
some point.
>> And the
>>  > >  > reason is in ZFS. It never happens with NFS/UFS.
>>  > >  > Shortly, my scenario:
>>  > >  > 1st SFS run, 2000 requested IOPS. NFS is fine, ;low
number
>> of threads.
>>  > >  > 2st SFS run, 4000 requested IOPS. NFS cannot serve all
>> requests, no of
>>  > >  > threads jumps to max
>>  > >  > 3rd SFS run, 2000 requested IOPS. NFS cannot serve all
>> requests, no of
>>  > >  > threads jumps to max.
>>  > >  > System cannot get back to the same results under equal
>> load (1st and 3rd).
>>  > >  > Reboot between 2nd and 3rd doesn''t help. The
only
>> persistent thing is
>>  > >  > a directory structure that was created during the 2nd
run
>> (in SFS
>>  > >  > higher requested load -> more directories/files
created).
>>  > >  > I am sure it''s a bug. I need help. I
don''t care that ZFS
>> works N times
>>  > >  > worse than UFS. I really care that after heavy load  
>> everything is
>>  > >  > totally screwed.
>>  > >  >
>>  > >  > Thanks,
>>  > >  > -- Leon
>>  > >
>>  > > Hi Leon,
>>  > >
>>  > > How much is the slowdown between 1st and 3rd ? How filled is
>>  >
>>  > Typical case is:
>>  > 1st: 1996 IOPS, latency  2.7
>>  > 3rd: 1375 IOPS, latency 37.9
>>  >
>>
>> The large latency increase is the  side effect of requesting
>> more than what can be delivered. Queue builds up and latency
>> follow. So  it  should  not be  the  primary  focus IMO. The
>> Decrease in IOPS is the primary problem.
>>
>> One hypothesis is that over the life of the FS we''re moving
>> toward spreading access to the full disk platter. We can
>> imagine some fragmentation hitting as well. I''m not sure
>> how I''d test both hypothesis.
>>
>>  > > the pool at each stage ? What does ''NFS stops to
service''
>>  > > mean ?
>>  >
>>  > There is a lot of error messages on the NFS(SFS) client :
>>  > sfs352: too many failed RPC calls - 416 good 27 bad
>>  > sfs3132: too many failed RPC calls - 302 good 27 bad
>>  > sfs3109: too many failed RPC calls - 533 good 31 bad
>>  > sfs353: too many failed RPC calls - 301 good 28 bad
>>  > sfs3144: too many failed RPC calls - 305 good 25 bad
>>  > sfs3121: too many failed RPC calls - 311 good 30 bad
>>  > sfs370: too many failed RPC calls - 315 good 27 bad
>>  >
>>
>> Can this be timing out or queue full drops ? Might be a side
>> effect of SFS requesting more than what can be delivered.
>
> I don''t know was it timeouts or full drops. SFS marked such runs
as
> INVALID.
> I can run whatever is needed to help to investigate the problem. If
> you have a D script that will tell us more, please send it to me.
> I appreciate your help.
The failed RPCs are indeed a result of the SFS client timing out
the requests it has made to the server.  The server is being
overloaded for its capabilities and the benchmark results
show that.  I agree with Roch that as the SFS benchmark adds
more data to the filesystems that additional latency is
added and for this particular configuration and the
server is being over-driven.

The helpful thing would be to run smaller increments in the
benchmark to determine where the response time increases
beyond what the SFS workload can handle.

There have been a number of changes in ZFS recently that should
help with SFS performance measurement but fundamentally it
all depends on the configuration of the server (number of spindles
and CPU available).  So there may be a limit that is being
reached based on the hardware configuration.

What is your real goal here, Leon?  Are you trying to gather SFS
data to fit into sizing of a particular solution or just trying
to gather performance results for other general comparisons?
There are certainly better benchmarks than SFS for either
sizing and comparison reasons.

Spencer

On 3/5/07, Spencer Shepler <Spencer.Shepler at sun.com>
wrote:
>
> On Mar 5, 2007, at 11:17 AM, Leon Koll wrote:
>
> > On 3/5/07, Roch - PAE <Roch.Bourbonnais at sun.com> wrote:
> >>
> >> Leon Koll writes:
> >>
> >>  > On 3/5/07, Roch - PAE <Roch.Bourbonnais at sun.com>
wrote:
> >>  > >
> >>  > > Leon Koll writes:
> >>  > >  > On 2/28/07, Roch - PAE <Roch.Bourbonnais at
sun.com> wrote:
> >>  > >  > >
> >>  > >  > >
> >>  > >  > >
http://bugs.opensolaris.org/bugdatabase/view_bug.do?
> >> bug_id=6467988
> >>  > >  > >
> >>  > >  > > NFSD  threads are created  on a  demand 
spike (all of
> >> them
> >>  > >  > > waiting  on I/O) but then    tend to stick
around
> >> servicing
> >>  > >  > > moderate loads.
> >>  > >  > >
> >>  > >  > > -r
> >>  > >  >
> >>  > >  > Hello Roch,
> >>  > >  > It''s not my case. NFS stops to service
after some point.
> >> And the
> >>  > >  > reason is in ZFS. It never happens with NFS/UFS.
> >>  > >  > Shortly, my scenario:
> >>  > >  > 1st SFS run, 2000 requested IOPS. NFS is fine,
;low number
> >> of threads.
> >>  > >  > 2st SFS run, 4000 requested IOPS. NFS cannot
serve all
> >> requests, no of
> >>  > >  > threads jumps to max
> >>  > >  > 3rd SFS run, 2000 requested IOPS. NFS cannot
serve all
> >> requests, no of
> >>  > >  > threads jumps to max.
> >>  > >  > System cannot get back to the same results under
equal
> >> load (1st and 3rd).
> >>  > >  > Reboot between 2nd and 3rd doesn''t help.
The only
> >> persistent thing is
> >>  > >  > a directory structure that was created during the
2nd run
> >> (in SFS
> >>  > >  > higher requested load -> more
directories/files created).
> >>  > >  > I am sure it''s a bug. I need help. I
don''t care that ZFS
> >> works N times
> >>  > >  > worse than UFS. I really care that after heavy
load
> >> everything is
> >>  > >  > totally screwed.
> >>  > >  >
> >>  > >  > Thanks,
> >>  > >  > -- Leon
> >>  > >
> >>  > > Hi Leon,
> >>  > >
> >>  > > How much is the slowdown between 1st and 3rd ? How
filled is
> >>  >
> >>  > Typical case is:
> >>  > 1st: 1996 IOPS, latency  2.7
> >>  > 3rd: 1375 IOPS, latency 37.9
> >>  >
> >>
> >> The large latency increase is the  side effect of requesting
> >> more than what can be delivered. Queue builds up and latency
> >> follow. So  it  should  not be  the  primary  focus IMO. The
> >> Decrease in IOPS is the primary problem.
> >>
> >> One hypothesis is that over the life of the FS we''re
moving
> >> toward spreading access to the full disk platter. We can
> >> imagine some fragmentation hitting as well. I''m not sure
> >> how I''d test both hypothesis.
> >>
> >>  > > the pool at each stage ? What does ''NFS stops
to service''
> >>  > > mean ?
> >>  >
> >>  > There is a lot of error messages on the NFS(SFS) client :
> >>  > sfs352: too many failed RPC calls - 416 good 27 bad
> >>  > sfs3132: too many failed RPC calls - 302 good 27 bad
> >>  > sfs3109: too many failed RPC calls - 533 good 31 bad
> >>  > sfs353: too many failed RPC calls - 301 good 28 bad
> >>  > sfs3144: too many failed RPC calls - 305 good 25 bad
> >>  > sfs3121: too many failed RPC calls - 311 good 30 bad
> >>  > sfs370: too many failed RPC calls - 315 good 27 bad
> >>  >
> >>
> >> Can this be timing out or queue full drops ? Might be a side
> >> effect of SFS requesting more than what can be delivered.
> >
> > I don''t know was it timeouts or full drops. SFS marked such
runs as
> > INVALID.
> > I can run whatever is needed to help to investigate the problem. If
> > you have a D script that will tell us more, please send it to me.
> > I appreciate your help.
>
> The failed RPCs are indeed a result of the SFS client timing out
> the requests it has made to the server.  The server is being
> overloaded for its capabilities and the benchmark results
> show that.  I agree with Roch that as the SFS benchmark adds
> more data to the filesystems that additional latency is
> added and for this particular configuration and the
> server is being over-driven.
>
> The helpful thing would be to run smaller increments in the
> benchmark to determine where the response time increases
> beyond what the SFS workload can handle.
>
> There have been a number of changes in ZFS recently that should
> help with SFS performance measurement but fundamentally it
> all depends on the configuration of the server (number of spindles
> and CPU available).  So there may be a limit that is being
> reached based on the hardware configuration.
>
> What is your real goal here, Leon?  Are you trying to gather SFS
> data to fit into sizing of a particular solution or just trying
> to gather performance results for other general comparisons?
Spencer,
I am using SFS benchmark to emulate the real-world environment for the
NAS solution that I''ve built. SFS is able to create as many processes
(each one emulating separate client) as one needs plus it''s rich of
meta operations.
My real goal is: quiet peaceful nights after my solution will start to
work in the production :)
What I see now: after some step the directory structure/on-disk layout
of ZFS? becomes an obstacle that cannot be passed. I don''t want that
this will happen in the field, that''s it.
And I am sure we have a bug case here.
> There are certainly better benchmarks than SFS for either
> sizing and comparison reasons.
I haven''t found anything better: client-independent,
multi-proc./multi-threaded, meta-rich, comparable. An obvious drawback
is: $$. I think that a free replacement of it is almost unknown and
underestimated fstress ( http://www.cs.duke.edu/ari/fstress/ ).
Which ones are your candidates?

Thanks,
-- Leon