We''re using some X4540s, with OpenSolaris 2008.11. According to my testing, to optimize our systems for our specific workload, I''ve determined that we get the best performance with the write cache disabled on every disk, and with zfs:zfs_nocacheflush=1 set in /etc/system. The only issue is setting the write cache permanently, or at least quickly. Right now, as it is, I''ve scripted up format to run on boot, disabling the write cache of all disks. This takes around two minutes. I''d like to avoid needing to take this time on every bootup (which is more often than you''d think, we''ve got quite a bit of construction happening, which necessitates bringing everything down periodically). This would also be painful in the event of unplanned downtime for one of our Thors. so, basically, my question is: Is there a way to quickly or permanently disable the write cache on every disk in an X4540? Thanks, -Greg
> so, basically, my question is: Is there a way to quickly or permanently > disable the write cache on every disk in an X4540?Hmmm... the only idea I have is to see how format(1M) does it and steal the code to write a small disable-cache tool. :-) Have a look at uscsi(7I) and specifically the ca_write_disable() routine in http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/cmd/format/menu_cache.c HTH -- Volker -- ------------------------------------------------------------------------ Volker A. Brandt Consulting and Support for Sun Solaris Brandt & Brandt Computer GmbH WWW: http://www.bb-c.de/ Am Wiesenpfad 6, 53340 Meckenheim Email: vab at bb-c.de Handelsregister: Amtsgericht Bonn, HRB 10513 Schuhgr??e: 45 Gesch?ftsf?hrer: Rainer J. H. Brandt und Volker A. Brandt
Hello Greg,
Wednesday, February 11, 2009, 10:13:39 PM, you wrote:
GM> We''re using some X4540s, with OpenSolaris 2008.11.
GM> According to my testing, to optimize our systems for our specific
GM> workload, I''ve determined that we get the best performance with
the
GM> write cache disabled on every disk, and with zfs:zfs_nocacheflush=1 set
GM> in /etc/system.
GM> The only issue is setting the write cache permanently, or at least
quickly.
GM> Right now, as it is, I''ve scripted up format to run on boot,
disabling
GM> the write cache of all disks. This takes around two minutes. I''d
like to
GM> avoid needing to take this time on every bootup (which is more often
GM> than you''d think, we''ve got quite a bit of construction
happening, which
GM> necessitates bringing everything down periodically). This would also be
GM> painful in the event of unplanned downtime for one of our Thors.
GM> so, basically, my question is: Is there a way to quickly or permanently
GM> disable the write cache on every disk in an X4540?
Are you sure thar write cache is back on after restart?
--
Best regards,
Robert
http://milek.blogspot.com
On Wed, Feb 11, 2009 at 2:13 PM, Greg Mason <gmason at msu.edu> wrote:> We''re using some X4540s, with OpenSolaris 2008.11. > > According to my testing, to optimize our systems for our specific workload, > I''ve determined that we get the best performance with the write cache > disabled on every disk, and with zfs:zfs_nocacheflush=1 set in /etc/system. > > The only issue is setting the write cache permanently, or at least quickly. > > Right now, as it is, I''ve scripted up format to run on boot, disabling the > write cache of all disks. This takes around two minutes. I''d like to avoid > needing to take this time on every bootup (which is more often than you''d > think, we''ve got quite a bit of construction happening, which necessitates > bringing everything down periodically). This would also be painful in the > event of unplanned downtime for one of our Thors. > > so, basically, my question is: Is there a way to quickly or permanently > disable the write cache on every disk in an X4540? > > Thanks, > > -Greg > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >We use several X4540''s over here as well, what type of workload do you have, and how much performance increase did you see by disabling the write caches? -- Brent Jones brent at servuhome.net
> We use several X4540''s over here as well, what type of workload do you > have, and how much performance increase did you see by disabling the > write caches? >We see the difference between our tests completing in around 2.5 minutes (with write caches) to around a minute an and a half without them, in one instance. I''m trying to optimize our machines for a write-heavy environment, as our users will undoubtedly hit this limitation of the machines. -Greg
> > Are you sure thar write cache is back on after restart? >Yes, I''ve checked with format -e, on each drive. When disabling the write cache with format, it also gives a warning stating this is the case. What I''m looking for is a faster way to do this than format -e -d <disk> -f <script>, for all 48 disks. From format, after a reboot: selecting c10t7d0 [disk formatted] /dev/dsk/c10t7d0s0 is part of active ZFS pool export. Please see zpool(1M). FORMAT MENU: disk - select a disk type - select (define) a disk type partition - select (define) a partition table current - describe the current disk format - format and analyze the disk fdisk - run the fdisk program repair - repair a defective sector label - write label to the disk analyze - surface analysis defect - defect list management backup - search for backup labels verify - read and display labels inquiry - show vendor, product and revision scsi - independent SCSI mode selects cache - enable, disable or query SCSI disk cache volname - set 8-character volume name !<cmd> - execute <cmd>, then return quit format> cache CACHE MENU: write_cache - display or modify write cache settings read_cache - display or modify read cache settings !<cmd> - execute <cmd>, then return quit cache> write_cache WRITE_CACHE MENU: display - display current setting of write cache enable - enable write cache disable - disable write cache !<cmd> - execute <cmd>, then return quit write_cache> display Write Cache is enabled write_cache> disable This setting is valid until next reset only. It is not saved permanently. write_cache> display Write Cache is disabled
On Thu, Feb 12, 2009 at 10:33:40AM -0500, Greg Mason wrote:> What I''m looking for is a faster way to do this than format -e -d <disk> > -f <script>, for all 48 disks.Is the speed critical? I mean, do you have to pause startup while the script runs, or does it interfere with data transfer? -- Darren
well, since the write cache flush command is disabled, I would like this to happen as early as practically possible in the bootup process, as ZFS will not be issuing the cache flush commands to the disks. I''m not really sure what happens in the case where the write flush command is disabled, something makes its way into the write cache, then the cache is disabled. Does this mean the write cache is flushed to disk when the cache is disabled? If so, then I guess it''s less critical when it happens in the bootup process or if it''s permanent... -Greg A Darren Dunham wrote:> On Thu, Feb 12, 2009 at 10:33:40AM -0500, Greg Mason wrote: >> What I''m looking for is a faster way to do this than format -e -d <disk> >> -f <script>, for all 48 disks. > > Is the speed critical? I mean, do you have to pause startup while the > script runs, or does it interfere with data transfer? >
Hello Greg,
Thursday, February 12, 2009, 8:24:38 PM, you wrote:
GM> well, since the write cache flush command is disabled, I would like this
GM> to happen as early as practically possible in the bootup process, as ZFS
GM> will not be issuing the cache flush commands to the disks.
GM> I''m not really sure what happens in the case where the write
flush
GM> command is disabled, something makes its way into the write cache, then
GM> the cache is disabled. Does this mean the write cache is flushed to disk
GM> when the cache is disabled? If so, then I guess it''s less
critical when
GM> it happens in the bootup process or if it''s permanent...
Then do not disable cache flushes via /etc/system but rather via mdb
in your script. So for a moment after a reboot you will have a pool
with some caches still enabled but zfs will send cache flushes so it''s
fine. After your script disabled all caches then you disable cache
flushes in zfs.
The other question actually is - what exaclty is turning your caches
back on after a reboot? It''s not ZFS so it is probably a driver - I
would check if there is a tunable to the driver so it won''t enable
write caches on reboot. I believe it would be a better solution.
--
Best regards,
Robert Milkowski
http://milek.blogspot.com