I''ve run into this problem twice now, before I had 10x500GB drives in a ZFS+ setup and now again in a 12x500GB ZFS+ setup. The problem is when the pool reaches ~85% capacity I get random read failures and around ~90% capacity I get read failures AND zpool corruption. For example: -I open a directory that I know for a fact has files and folders in it but it either shows 0 items or hangs on a directory listing -I try to copy a file from the zpool volume to another volume and it hangs then fails In both these situations if I do a ''zpool status'' after the fact it claims that the volume has experienced an unrecoverable error and I should find the faulty drive and replace it blah blah. If I do a ''zpool scrub'' it eventually reports 0 faults or error, also if I restart the machine it will usually work jut fine again (ie I can read the directory and copy files again). Is this a systemic problem at 90% capacity or do I perhaps have a faulty drive in the array that only gets hit at 90%? If it is a faulty drive why does ''zpool status'' report fully good health, that makes it hard to find the problem drive? Thanks, Sam -- This message posted from opensolaris.org
It is not recommended to store more than 90% on any file system, I think. For instance, NTFS can behave very badly when it runs out of space. Similar to if you fill up your RAM and you have no swap space. Then the computer starts to thrash badly. Not recommended. Avoid 90% and above, and you have eliminated a possible source of problems. -- This message posted from opensolaris.org
I was hoping that this was the problem (because just buying more discs is the cheapest solution given time=$$) but running it by somebody at work they said going over 90% can cause decreased performance but is unlikely to cause the strange errors I''m seeing. However, I think I''ll stick a 1TB drive in as a new volume and pull some data onto it to bring the zpool down to <75% capacity and see if that helps though anyway. Probably update the OS to 2008.11 as well. -- This message posted from opensolaris.org
On 1/6/2009 4:19 PM, Sam wrote:> I was hoping that this was the problem (because just buying more > discs is the cheapest solution given time=$$) but running it by > somebody at work they said going over 90% can cause decreased > performance but is unlikely to cause the strange errors I''m seeing. > However, I think I''ll stick a 1TB drive in as a new volume and pull > some data onto it to bring the zpool down to<75% capacity and see if > that helps though anyway. Probably update the OS to 2008.11 as > well.Pool corruption is _always_ a bug. It may be ZFS, or your block devices, but something is broken -- Carson
On Tue, Jan 6, 2009 at 6:19 PM, Sam <sam at smugmug.com> wrote:> I was hoping that this was the problem (because just buying more discs is > the cheapest solution given time=$$) but running it by somebody at work they > said going over 90% can cause decreased performance but is unlikely to cause > the strange errors I''m seeing. However, I think I''ll stick a 1TB drive in > as a new volume and pull some data onto it to bring the zpool down to <75% > capacity and see if that helps though anyway. Probably update the OS to > 2008.11 as well. > -- >Uhh, I would never accept that one as a solution. 90% full or not a READ should never, ever, ever corrupt a pool. Heck, a write shouldn''t either. I could see the system falling over and puking on itself performance wise, but corruption? No way. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090106/1ad9bcf7/attachment.html>
Since zfs is so smart is other areas is there a particular reason why a high water mark is not calculated and the available space not reset to this? I''d far rather have a zpool of 1000GB that said it only had 900GB but did not have corruption as it ran out of space. Nicholas -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090107/e91def69/attachment.html>
On Tue, Jan 6, 2009 at 10:25 PM, Nicholas Lee <emptysands at gmail.com> wrote:> Since zfs is so smart is other areas is there a particular reason why a > high water mark is not calculated and the available space not reset to this? > I''d far rather have a zpool of 1000GB that said it only had 900GB but did > not have corruption as it ran out of space. > > Nicholas >WHAT??!? Put artificial limits in place to prevent users from killing themselves? How did that go Jeff? "I suggest that you retire to the safety of the rubber room while the rest of us enjoy these zfs features. By the same measures, you would advocate that people should never be allowed to go outside due to the wide open spaces. Perhaps people will wander outside their homes and forget how to make it back. Or perhaps there will be gravity failure and some of the people outside will be lost in space." It''s NEVER a good idea to put a default limitation in place to protect a *regular user*. If they can''t RTFM from front cover to back they don''t deserve to use a computer. --Tim -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090106/1f9c5202/attachment.html>
BTW, high water mark method is not perfect, here is some for Novell support of water mark... best, z http://www.novell.com/coolsolutions/tools/16991.html Based on my own belief that there had to be a "better way" and the number of issues I''d seen reported in the Support Forums, I spent a lot of time researching how different memory settings affect the memory management and stability of the server. Based on that research I''ve made memory tuning recommendations to a large number of forum posters who were having memory tuning issues, and most of them have found their servers to be significantly more stable since applying the changes I recommended. What follows are the formulas I developed for recommending memory tuning changes to a server. The formulas take a number of the values available from SEG.NLM (available from: http://www.novell.com/coolsolutions/tools/14445.html). To get the required values, load SEG.NLM, then from the main screen do ''/'', then ''Info'', then ''Write SEGSTATS.TXT''. The SEGSTATS.TXT file will be created in SYS:SYSTEM. SEG monitors the server and records a number of key memory statistics, my formulae take those statistics and recommend manual memory tuning parameters. Note that as these are manual settings, auto tuning is disabled, and if the memory usage of the server changes significantly, then the server will need to be retuned to reflect the change in memory usage. Also, after making the changes to use manual rather than auto tuning, the server may still recommend that the FCMS and "-u" memory settings be changed. These recommendations can be ignored. Following them will have the same effect as auto tuning, except you''re doing it rather than the server doing it automatically - the same problems will still occur. ----- Original Message ----- From: Tim To: Nicholas Lee Cc: zfs-discuss at opensolaris.org ; Sam Sent: Wednesday, January 07, 2009 12:02 AM Subject: Re: [zfs-discuss] Problems at 90% zpool capacity 2008.05 On Tue, Jan 6, 2009 at 10:25 PM, Nicholas Lee <emptysands at gmail.com> wrote: Since zfs is so smart is other areas is there a particular reason why a high water mark is not calculated and the available space not reset to this? I''d far rather have a zpool of 1000GB that said it only had 900GB but did not have corruption as it ran out of space. Nicholas WHAT??!? Put artificial limits in place to prevent users from killing themselves? How did that go Jeff? "I suggest that you retire to the safety of the rubber room while the rest of us enjoy these zfs features. By the same measures, you would advocate that people should never be allowed to go outside due to the wide open spaces. Perhaps people will wander outside their homes and forget how to make it back. Or perhaps there will be gravity failure and some of the people outside will be lost in space." It''s NEVER a good idea to put a default limitation in place to protect a *regular user*. If they can''t RTFM from front cover to back they don''t deserve to use a computer. --Tim ------------------------------------------------------------------------------ _______________________________________________ zfs-discuss mailing list zfs-discuss at opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090107/a99155b9/attachment.html>
On 01/06/09 21:25, Nicholas Lee wrote:> Since zfs is so smart is other areas is there a particular reason why a > high water mark is not calculated and the available space not reset to this? > > I''d far rather have a zpool of 1000GB that said it only had 900GB but > did not have corruption as it ran out of space. > > NicholasIs there any evidence of corruption at high capacity or just a lack of performance? All file systems will slow down when near capacity, as they struggle to find space and then have to spread writes over the disk. Our priorities are integrity first, followed somewhere by performance. I vaguely remember a time when UFS had limits to prevent ordinary users from consuming past a certain limit, allowing only the super-user to use it. Not that I''m advocating that approach for ZFS. Neil.
--On 06 January 2009 16:37 -0800 Carson Gaspar <carson at taltos.org> wrote:> On 1/6/2009 4:19 PM, Sam wrote: >> I was hoping that this was the problem (because just buying more >> discs is the cheapest solution given time=$$) but running it by >> somebody at work they said going over 90% can cause decreased >> performance but is unlikely to cause the strange errors I''m seeing. >> However, I think I''ll stick a 1TB drive in as a new volume and pull >> some data onto it to bring the zpool down to<75% capacity and see if >> that helps though anyway. Probably update the OS to 2008.11 as >> well. > > Pool corruption is _always_ a bug. It may be ZFS, or your block devices, > but something is brokenAgreed - it shouldn''t break just because you''re using over 90% - checking on one of my systems here I have: " Filesystem 1K-blocks Used Avail Capacity Mounted on vol 2567606528 2403849728 163756800 94% /vol " Been running like that for months without issue... Whilst it may not be ''ideal'' to run it over 90% (I suspect it''s worse for pools made up of different sized devices / redundancy) - it''s not broken in any shape or form with gb''s of reads/writes going to that file system. -Kp
Ok so the capacity is ruled out, it still bothers me that after experiencing the error if I do a ''zpool status'' it just hangs (forever) but if I reboot the system everything comes back up fine (for a little while). Last night I installed the latest SXDE and I''m going to see if that fixes it, if that doesn''t work I''m going to put in a new disc controller, after that I''m replacing the drives. -- This message posted from opensolaris.org
OMG, open folks are really budget concened. In enterprises, a 90% policy as a safety feature is ok... alerts will be sent and POs will be issued... :-) z ----- Original Message ----- From: "Sam" <sam at smugmug.com> To: <zfs-discuss at opensolaris.org> Sent: Wednesday, January 07, 2009 1:33 PM Subject: Re: [zfs-discuss] Problems at 90% zpool capacity 2008.05> Ok so the capacity is ruled out, it still bothers me that after > experiencing the error if I do a ''zpool status'' it just hangs (forever) > but if I reboot the system everything comes back up fine (for a little > while). > > Last night I installed the latest SXDE and I''m going to see if that fixes > it, if that doesn''t work I''m going to put in a new disc controller, after > that I''m replacing the drives. > -- > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
On Wed, Jan 7, 2009 at 12:33 PM, Sam <sam at smugmug.com> wrote:> Ok so the capacity is ruled out, it still bothers me that after > experiencing the error if I do a ''zpool status'' it just hangs (forever) but > if I reboot the system everything comes back up fine (for a little while). > > Last night I installed the latest SXDE and I''m going to see if that fixes > it, if that doesn''t work I''m going to put in a new disc controller, after > that I''m replacing the drives. >I wouldn''t be so quick to rule out capacity because one other user doesn''t have the problem. Things like change rate, how the data was laid down in the first place, etc play a pretty big role. Fragmentation anyone? 90% full completely fragmented is a world of difference from 90% full optimally laid out. --Tim -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090107/2220ab65/attachment.html>
On Tue, 06 Jan 2009 22:18:40 -0700, Neil Perrin <Neil.Perrin at Sun.COM> wrote:>I vaguely remember a time when UFS had limits to prevent >ordinary users from consuming past a certain limit, allowing >only the super-user to use it. Not that I''m advocating that >approach for ZFS.I know that approach from other operating systems, like Fujitsu Siemens BS2000/OSD. There it is called "zip space" and specified in number of 2 kB disk pages. It isn''t meant to prevent corruption (user processes were simple frozen until the requested disk space allocation could be completed), but only to allow the super-user to log in on an otherwise completely stuck system, and very useful as such. Typically, 24 pages in the super-users'' home file system (AKA public volume set) would be enough. -- ( Kees Nuyt ) c[_]
Bill Sommerfeld
2009-Jan-08 01:11 UTC
[zfs-discuss] Problems at 90% zpool capacity 2008.05
On Tue, 2009-01-06 at 22:18 -0700, Neil Perrin wrote:> I vaguely remember a time when UFS had limits to prevent > ordinary users from consuming past a certain limit, allowing > only the super-user to use it. Not that I''m advocating that > approach for ZFS.looks to me like zfs already provides a mechanism for this (quotas and reservations); it''s up to the sysadmin to decide on policy. Don''t want the last 10% of the pool used? Create a "ballast" zvol or filesystem with a big reservation, and don''t put anything in it.. Of course, some degree of experimentation may be necessary before you figure out what policy makes sense for your system or site. - Bill
On 1/8/09, Bill Sommerfeld <sommerfeld at sun.com> wrote:> > > On Tue, 2009-01-06 at 22:18 -0700, Neil Perrin wrote: > > I vaguely remember a time when UFS had limits to prevent > > ordinary users from consuming past a certain limit, allowing > > only the super-user to use it. Not that I''m advocating that > > approach for ZFS.man page of newfs, on Solaris 8 (5.8), gives the option: -m free The minimum percentage of free space to maintain in the file system (between 1% and 99%, inclusively). This space is off-limits to normal users. Once the file system is filled to this threshold, only the super-user can continue writing to the file system. This parameter can be subsequently changed using the tunefs(1M) command. The default is ((64 Mbytes/partition size) * 100), rounded down to the nearest integer and limited between 1% and 10%, inclusively. We always kept it to 1 % but were very glad to have it when, for any reason, the users had nothing left... I should add that we were running most of the time above 90 % (it is just thermodynamic, gas occupy all available space!) and could not see any real slowdown between 40 % and 99 % full (ufs+logging on sparc Solaris 8). Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090108/b8d427aa/attachment.html>