We are having slow performance with the UFS volumes on the x4500. They
are slow even on the local server. Which makes me think it is (for once)
not NFS related.
Current settings:
SunOS x4500-01.unix 5.11 snv_70b i86pc i386 i86pc
# cat /etc/release
Solaris Express Developer Edition 9/07 snv_70b X86
Copyright 2007 Sun Microsystems, Inc. All Rights Reserved.
Use is subject to license terms.
Assembled 30 August 2007
NFSD_SERVERS=1024
LOCKD_SERVERS=128
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
12249 daemon 7204K 6748K sleep 60 -20 54:16:26 14% nfsd/731
load averages: 2.22, 2.32, 2.42 12:31:35
63 processes: 62 sleeping, 1 on cpu
CPU states: 68.7% idle, 0.0% user, 31.3% kernel, 0.0% iowait, 0.0% swap
Memory: 16G real, 1366M free, 118M swap in use, 16G swap free
/etc/system:
set ndquot=5048000
We have a setup like:
/export/zfs1
/export/zfs2
/export/zfs3
/export/zfs4
/export/zfs5
/export/zdev/vol1/ufs1
/export/zdev/vol2/ufs2
/export/zdev/vol3/ufs3
What is interesting is that if I run "df", it will display everything
at
normal speed, but pause before "vol1/ufs1" file system. truss confirms
that statvfs64() is slow (5 seconds usually). All other ZFS and UFS
filesystems behave normally. vol1/ufs1 is the most heavily used UFS
filesystem.
Disk:
/dev/zvol/dsk/zpool1/ufs1
991G 224G 758G 23% /export/ufs1
Inodes:
/dev/zvol/dsk/zpool1/ufs1
37698475 25044053 60% /export/ufs1
Possible problems:
# vmstat -s
866193018 total name lookups (cache hits 57%)
# kstat -n inode_cache
module: ufs instance: 0
name: inode_cache class: ufs
maxsize 129797
maxsize reached 269060
thread idles 319098740
vget idles 62136
This leads me to think we should consider setting;
set ncsize=259594 (doubled... are there better values?)
set ufs_ninode=259594
in /etc/system, and reboot. But it is costly to reboot based only on my
guess. Do you have any other suggestions to explore? Will this help?
Sincerely,
Jorgen Lundman
--
Jorgen Lundman | <lundman at lundman.net>
Unix Administrator | +81 (0)3 -5456-2687 ext 1017 (work)
Shibuya-ku, Tokyo | +81 (0)90-5578-8500 (cell)
Japan | +81 (0)3 -3375-1767 (home)
Jorgen Lundman writes:> > We are having slow performance with the UFS volumes on the x4500. They > are slow even on the local server. Which makes me think it is (for once) > not NFS related. > > > Current settings: > > SunOS x4500-01.unix 5.11 snv_70b i86pc i386 i86pc >That''s a very old release, have you considered upgrading? Ian.
>> SunOS x4500-01.unix 5.11 snv_70b i86pc i386 i86pc > That''s a very old release, have you considered upgrading? > Ian. >It was the absolute latest version available when we received the x4500, and now it is live and supporting a large number of customers. However, the 2nd unit will arrive next week (Will be Sol10 508, as that is the only/newest OS version the vendor will support). So yes, in a way we will move to a newer version if we can work out a good way to migrate from one x4500 to another x4500:) But in the meanwhile, we were hoping we could do some kernel tweaking, reboot (3 minute downtime) and it would perform a little better. It would be nice to have someone who knows more than me, give their opinion as to if my guesses has any chances of succeeding. For example, Postfix delivering mail, system calls like open() and fdsync() is currently taking upwards of 7 seconds to complete. Lund -- Jorgen Lundman | <lundman at lundman.net> Unix Administrator | +81 (0)3 -5456-2687 ext 1017 (work) Shibuya-ku, Tokyo | +81 (0)90-5578-8500 (cell) Japan | +81 (0)3 -3375-1767 (home)
Hi Jorgen, This isn''t an answer to your problem I''m afraid, but a request for you to do a test when you get your new x4500. Could you try pulling a SATA drive to see if the system hangs? I''m finding Solaris just locks up if I pull a drive connected to the Supermicro AOC-SAT2-MV8 card, and I was under the belief that uses the same chipset as the Thumper. I''m hoping this is just a driver problem, or a problem specific to the Supermicro card, but since our loan x4500 went back to Sun I''m unable to test this myself, and if the x4500''s do lock up I''m a bit concerned about how they handle hardware failures. thanks, Ross This message posted from opensolaris.org
We have had a disk fail in the the existing x4500 and it sure froze the whole server. I believe it is an OS problem which (should have) been fixed in a version newer than we have. If you want me to test it on the new x4500 because it runs Sol10 508 I can do. Ross wrote:> Hi Jorgen, > > This isn''t an answer to your problem I''m afraid, but a request for you to do a test when you get your new x4500. > > Could you try pulling a SATA drive to see if the system hangs? I''m finding Solaris just locks up if I pull a drive connected to the Supermicro AOC-SAT2-MV8 card, and I was under the belief that uses the same chipset as the Thumper. > > I''m hoping this is just a driver problem, or a problem specific to the Supermicro card, but since our loan x4500 went back to Sun I''m unable to test this myself, and if the x4500''s do lock up I''m a bit concerned about how they handle hardware failures. > > thanks, > > Ross > > > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >-- Jorgen Lundman | <lundman at lundman.net> Unix Administrator | +81 (0)3 -5456-2687 ext 1017 (work) Shibuya-ku, Tokyo | +81 (0)90-5578-8500 (cell) Japan | +81 (0)3 -3375-1767 (home)
We had the same problem, at least a good chunk of the zfs volumes died when the drive failed. Granted, I don''t think the drive actually failed, but a driver issue/lockup. A reboot 2 weeks ago brought the machine back up and the drive hasn''t had a problem since. I was behind on two patches that were released earlier in the month. I put them on the machine (a kernel patch and a marvell driver patch) and the problem hasn''t arisen again. Dave -----Original Message----- From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-bounces at opensolaris.org] On Behalf Of Jorgen Lundman Sent: Thursday, July 24, 2008 7:40 AM To: zfs-discuss at opensolaris.org Subject: Re: [zfs-discuss] x4500 performance tuning. We have had a disk fail in the the existing x4500 and it sure froze the whole server. I believe it is an OS problem which (should have) been fixed in a version newer than we have. If you want me to test it on the new x4500 because it runs Sol10 508 I can do. Ross wrote:> Hi Jorgen, > > This isn''t an answer to your problem I''m afraid, but a request for you to do a test when you get your new x4500. > > Could you try pulling a SATA drive to see if the system hangs? I''m finding Solaris just locks up if I pull a drive connected to the Supermicro AOC-SAT2-MV8 card, and I was under the belief that uses the same chipset as the Thumper. > > I''m hoping this is just a driver problem, or a problem specific to the Supermicro card, but since our loan x4500 went back to Sun I''m unable to test this myself, and if the x4500''s do lock up I''m a bit concerned about how they handle hardware failures. > > thanks, > > Ross > > > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >-- Jorgen Lundman | <lundman at lundman.net> Unix Administrator | +81 (0)3 -5456-2687 ext 1017 (work) Shibuya-ku, Tokyo | +81 (0)90-5578-8500 (cell) Japan | +81 (0)3 -3375-1767 (home) _______________________________________________ zfs-discuss mailing list zfs-discuss at opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Have any of you guys reported this to Sun? A quick search of the bug database doesn''t bring up anything that appears related to sata drives and hanging or hot swapping. This message posted from opensolaris.org
There are known issues with the Marvell drivers in X4500s. You will want to pay attention to the release notes, SRDBs, InfoDocs, and SunAlerts for the platform. http://sunsolve.sun.com/handbook_pub/validateUser.do?target=Systems/SunFireX4500/SunFireX4500 You will want to especially pay attention to SunAlert 201289 http://sunsolve.sun.com/search/document.do?assetkey=1-66-201289-1 If you run into these or other problems which are not already described in the above documents, please log a service call which will get you into the folks who track the platform problems specifically and know about patches in the pipeline. -- richard
Ross, The X4500 uses 6x Marvell 88SX SATA controllers for its internal disks. They are not Supermicro controllers. The new X4540 uses an LSI chipset instead of the Marvell chipset. --Brett This message posted from opensolaris.org
> Ross, > > The X4500 uses 6x Marvell 88SX SATA controllers for > its internal disks. They are not Supermicro > controllers. The new X4540 uses an LSI chipset > instead of the Marvell chipset. > > --BrettYup, and the Supermicro card uses the "Marvell Hercules-2 88SX6081 (Rev. C0) SATA Host Controller", which is part of the series supported by the same driver: http://docs.sun.com/app/docs/doc/816-5177/marvell88sx-7d?a=view. I''ve seen the Supermicro card mentioned in connection with the Thumpers many times on the forums. And Supermicro have a card using the LSI chipset now too:http://www.supermicro.com/products/accessories/addon/AOC-USAS-L8i.cfm I don''t know if that''s the same one as the new Thumper, but posts here have implied it is. Either Sun & Supermicro have very similar taste in controllers, or Supermicro are looking at ZFS and actively copying Sun :) This message posted from opensolaris.org
> Yup, and the Supermicro card uses the "Marvell > Hercules-2 88SX6081 (Rev. C0) SATA Host Controller", > which is part of the series supported by the same > driver: > http://docs.sun.com/app/docs/doc/816-5177/marvell88sx > 7d?a=view. I''ve seen the Supermicro card mentioned > in connection with the Thumpers many times on the > forums.Ahh, I was unfamiliar with Supermicro''s products...I''ll shut up now. :) --Brett This message posted from opensolaris.org
Since we were drowning, we decided to go ahead and reboot with my
guesses, even though I have not heard and expert opinions on the
changes. (Also, 3 mins was way under estimated. It takes 12 minutes to
reboot our x4500).
The new values are: (original)
set bufhwm_pct=10 (2%)
set maxusers=4096 (2048)
set ndquot=5048000 (50480)
set ncsize=1038376 (129797)
set ufs_ninode=1038376 (129797)
It does appear to run more better, but it hard to tell. 7 out of 10
tries, statvfs64 takes less than 2seconds, but I did get as high as 14s.
However, 2 hours later the x4500 hung. Pingable, but no console, nor NFS
response. The "LOM" was fine, and I performed a remote reboot.
Since then it has stayed up 5 hours.
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
521 daemon 7404K 6896K sleep 60 -20 0:25:03 3.1% nfsd/754
Total: 1 processes, 754 lwps, load averages: 0.82, 0.79, 0.79
CPU states: 90.6% idle, 0.0% user, 9.4% kernel, 0.0% iowait, 0.0% swap
Memory: 16G real, 829M free, 275M swap in use, 16G swap free
10191915 total name lookups (cache hits 82%)
maxsize 1038376
maxsize reached 993770
(Increased it by nearly x10 and it still gets a high
''reached'').
Lund
Jorgen Lundman wrote:> We are having slow performance with the UFS volumes on the x4500. They
> are slow even on the local server. Which makes me think it is (for once)
> not NFS related.
>
>
> Current settings:
>
> SunOS x4500-01.unix 5.11 snv_70b i86pc i386 i86pc
>
> # cat /etc/release
> Solaris Express Developer Edition 9/07 snv_70b X86
> Copyright 2007 Sun Microsystems, Inc. All Rights Reserved.
> Use is subject to license terms.
> Assembled 30 August 2007
>
> NFSD_SERVERS=1024
> LOCKD_SERVERS=128
>
> PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
>
> 12249 daemon 7204K 6748K sleep 60 -20 54:16:26 14% nfsd/731
>
> load averages: 2.22, 2.32, 2.42 12:31:35
> 63 processes: 62 sleeping, 1 on cpu
> CPU states: 68.7% idle, 0.0% user, 31.3% kernel, 0.0% iowait, 0.0% swap
> Memory: 16G real, 1366M free, 118M swap in use, 16G swap free
>
>
> /etc/system:
>
> set ndquot=5048000
>
>
> We have a setup like:
>
> /export/zfs1
> /export/zfs2
> /export/zfs3
> /export/zfs4
> /export/zfs5
> /export/zdev/vol1/ufs1
> /export/zdev/vol2/ufs2
> /export/zdev/vol3/ufs3
>
> What is interesting is that if I run "df", it will display
everything at
> normal speed, but pause before "vol1/ufs1" file system. truss
confirms
> that statvfs64() is slow (5 seconds usually). All other ZFS and UFS
> filesystems behave normally. vol1/ufs1 is the most heavily used UFS
> filesystem.
>
> Disk:
> /dev/zvol/dsk/zpool1/ufs1
> 991G 224G 758G 23% /export/ufs1
>
> Inodes:
> /dev/zvol/dsk/zpool1/ufs1
> 37698475 25044053 60% /export/ufs1
>
>
>
>
> Possible problems:
>
> # vmstat -s
> 866193018 total name lookups (cache hits 57%)
>
> # kstat -n inode_cache
> module: ufs instance: 0
> name: inode_cache class: ufs
> maxsize 129797
> maxsize reached 269060
> thread idles 319098740
> vget idles 62136
>
>
> This leads me to think we should consider setting;
>
> set ncsize=259594 (doubled... are there better values?)
> set ufs_ninode=259594
>
> in /etc/system, and reboot. But it is costly to reboot based only on my
> guess. Do you have any other suggestions to explore? Will this help?
>
>
> Sincerely,
>
> Jorgen Lundman
>
>
--
Jorgen Lundman | <lundman at lundman.net>
Unix Administrator | +81 (0)3 -5456-2687 ext 1017 (work)
Shibuya-ku, Tokyo | +81 (0)90-5578-8500 (cell)
Japan | +81 (0)3 -3375-1767 (home)
Richard Elling wrote:> There are known issues with the Marvell drivers in X4500s. You will > want to pay attention to the release notes, SRDBs, InfoDocs, and SunAlerts > for the platform. > http://sunsolve.sun.com/handbook_pub/validateUser.do?target=Systems/SunFireX4500/SunFireX4500 > > You will want to especially pay attention to SunAlert 201289 > http://sunsolve.sun.com/search/document.do?assetkey=1-66-201289-1 > > If you run into these or other problems which are not already described > in the above documents, please log a service call which will get you > into the folks who track the platform problems specifically and know > about patches in the pipeline. > -- richard > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >Although I am not in the SATA group any longer, I have in the past tested hot plugging and failures of SATA disks with x4500s, Marvell plug in cards and SuperMicro plug in cards. It has worked in the past on all of these platforms. Having said that there are things that you might be hitting or might try. 1) The default behavior when a disk is removed and then re-inserted is to leave the disk unconfigured. The operator must issue a cfgadm -c configure sata<x>/<y> to bring the newly plugged in disk on-line. There was some work being done to make this automatic, but I am not currently aware of the state of that work. 2) There were bugs related to disk drive errors that have been addressed (several months ago). If you have old code you could be hitting one or more of those issues. 3) I think there was a change in the sata generic module with respect to when it declares a failed disk as "off-line". You might want to check if you are hitting a problem with that. 4) There are a significant number of bugs in ZFS that can cause hangs. Most have been addressed with recent patches. Make sure you have all the patches. If you use the raw disk (i.e. no ZFS involvement) doing something like dd bs=128k if=/dev/rdsk/c<x>t<y>d0p0 of=/dev/null and then try pulling out the disk. The dd should return with an I/O error virtually immediately. If it doesn''t then ZFS is probably not the issue. You can also issue the command "cfgadm" and see what it lists as the state(s) of the various disks. Hope that helps, Lida Horn
Lida Horn wrote:> Richard Elling wrote: > >> There are known issues with the Marvell drivers in X4500s. You will >> want to pay attention to the release notes, SRDBs, InfoDocs, and SunAlerts >> for the platform. >> http://sunsolve.sun.com/handbook_pub/validateUser.do?target=Systems/SunFireX4500/SunFireX4500 >> >> You will want to especially pay attention to SunAlert 201289 >> http://sunsolve.sun.com/search/document.do?assetkey=1-66-201289-1 >> >> If you run into these or other problems which are not already described >> in the above documents, please log a service call which will get you >> into the folks who track the platform problems specifically and know >> about patches in the pipeline. >> -- richard >> >> _______________________________________________ >> zfs-discuss mailing list >> zfs-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >> >> > Although I am not in the SATA group any longer, I have in the past > tested hot plugging and failures > of SATA disks with x4500s, Marvell plug in cards and SuperMicro plug in > cards. It has worked > in the past on all of these platforms. Having said that there are > things that you might be hitting > or might try. > > 1) The default behavior when a disk is removed and then re-inserted is > to leave the disk unconfigured. > The operator must issue a cfgadm -c configure sata<x>/<y> to bring > the newly plugged in disk on-line. > There was some work being done to make this automatic, but I am not > currently aware of the state of > that work. >As of build 94, it does not automatically bring the disk online. I replaced a failed disk on an x4500 today running Nevada build 94, and still had to manually issue # cfgadm -c configure sata1/3 # zpool replace tank cxt2d0 then wait 7 hours for resilver. But the above is correct and expected. They simply have not automated that yet. Apparently. Neal> 2) There were bugs related to disk drive errors that have been addressed > (several months ago). If you have old > code you could be hitting one or more of those issues. > > 3) I think there was a change in the sata generic module with respect to > when it declares a failed disk as "off-line". > You might want to check if you are hitting a problem with that. > > 4) There are a significant number of bugs in ZFS that can cause hangs. > Most have been addressed with recent patches. > Make sure you have all the patches. > > If you use the raw disk (i.e. no ZFS involvement) doing something like > dd bs=128k if=/dev/rdsk/c<x>t<y>d0p0 of=/dev/null > and then try pulling out the disk. The dd should return with an I/O > error virtually immediately. If it doesn''t then > ZFS is probably not the issue. You can also issue the command "cfgadm" > and see what it lists as the state(s) of the > various disks. > > Hope that helps, > Lida Horn > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080724/e0ec967f/attachment.html>
On Thu, Jul 24, 2008 at 08:38:21PM -0700, Neal Pollack wrote:> > As of build 94, it does not automatically bring the disk online. > I replaced a failed disk on an x4500 today running Nevada build 94, and > still > had to manually issue > > # cfgadm -c configure sata1/3 > # zpool replace tank cxt2d0 > > then wait 7 hours for resilver. > But the above is correct and expected. They simply have not automated > that yet. Apparently.You can set ''sata:sata_auto_online=1'' in /etc/system to enable this behavior. It should be the default, as it is with other drivers (like mpt), but there has been resistance from the SATA team in the past. Anyone wanting to do the PSARC legwork could probably get this changed in nevada. You can also set ''autoreplace=on'' for the ZFS pool, and the replace will happen automatically upon insertion if you are using whole disks and a driver with static device paths (such as sata). - Eric -- Eric Schrock, Fishworks http://blogs.sun.com/eschrock
Richard Elling wrote:> There are known issues with the Marvell drivers in X4500s. You will > want to pay attention to the release notes, SRDBs, InfoDocs, > and SunAlerts > for the platform. > http://sunsolve.sun.com/handbook_pub/validateUser.do?target=Sy > stems/SunFireX4500/SunFireX4500 > > You will want to especially pay attention to SunAlert 201289 > http://sunsolve.sun.com/search/document.do?assetkey=1-66-201289-1 > > If you run into these or other problems which are not already > described > in the above documents, please log a service call which will get you > into the folks who track the platform problems specifically and know > about patches in the pipeline.Richard, while these are very useful links about the X4500 in general, there is really nothing here about what is required in the OpenSolaris world. The original poster is concerned with an X4500 running Nevada (build70). I have two X4500s running Nevada (also build70). At what Nevada build were the (Sun) bugids referenced in SunAlert 201289 fixed? Personally, I''ve been quite reluctant to open a service call with Sun when I''m running an "unsupported" operating system, even on supported hardware. I''ve had too many experiences with first-level support (and second-level support -- not necessarily Sun''s) whose first suggestion is to install (the latest) supported OS, before even investigating the hardware aspects, or in fact even hearing what the problem is in the first place. --Joe
Moore, Joe wrote:> Richard Elling wrote: > >> There are known issues with the Marvell drivers in X4500s. You will >> want to pay attention to the release notes, SRDBs, InfoDocs, >> and SunAlerts >> for the platform. >> http://sunsolve.sun.com/handbook_pub/validateUser.do?target=Sy >> stems/SunFireX4500/SunFireX4500 >> >> You will want to especially pay attention to SunAlert 201289 >> http://sunsolve.sun.com/search/document.do?assetkey=1-66-201289-1 >> >> If you run into these or other problems which are not already >> described >> in the above documents, please log a service call which will get you >> into the folks who track the platform problems specifically and know >> about patches in the pipeline. >> > > Richard, while these are very useful links about the X4500 in general, > there is really nothing here about what is required in the OpenSolaris > world. > > The original poster is concerned with an X4500 running Nevada (build70). > I have two X4500s running Nevada (also build70). At what Nevada build > were the (Sun) bugids referenced in SunAlert 201289 fixed? >You will have to look at each bug report and see where the fix was integrated. http://bugs.opensolaris.org Sometimes you can look at the change log reports, too, but I find the bug database searches to be faster. For example see the b94 change log: http://dlc.sun.com/osol/on/downloads/b94/on-changelog-b94.html This is a tedious task, but is the only way to really track such details. It is generally safe to assume that many changes are made in the latest SXCE or OpenSolaris builds before they arrive in previous releases as patches. Since patches are not created for SXCE/OpenSolaris, or at least the patch process you know and love from S10- is undergoing dramatic changes with IPS, the method instead will be to upgrade to a later release. b70 is approximately 2*(current build (95) - 70) weeks old. Sidebar: significant RAS improvements arrived in b72, so I would suggest scheduling an upgrade anyway.> Personally, I''ve been quite reluctant to open a service call with Sun > when I''m running an "unsupported" operating system, even on supported > hardware. I''ve had too many experiences with first-level support (and > second-level support -- not necessarily Sun''s) whose first suggestion is > to install (the latest) supported OS, before even investigating the > hardware aspects, or in fact even hearing what the problem is in the > first place. >Support contracts are available for OpenSolaris. http://www.sun.com/service/opensolaris/index.jsp But even if you don''t want to go that route, please log bugs you find at http://bugs.opensolaris.org In the best case, the bug will be closed as a duplicate. -- richard