It seems when a zfs filesystem with reserv/quota is 100% full users can no longer even delete files to fix the situation getting errors like these: $ rm rh.pm6895.medial.V2.tif rm: cannot remove `rh.pm6895.medial.V2.tif'': Disk quota exceeded (this is over NFS from a RHEL4 Linux box) I can log in as root on the Sun server and delete the file as root. After doing that, the user can then delete files okay. Is there anyway to workaround this does not involve root intervention? Users are filling up their volumes all the time which is the reason they must have reserv/quota set. -- --------------------------------------------------------------- Paul Raines email: raines at nmr.mgh.harvard.edu MGH/MIT/HMS Athinoula A. Martinos Center for Biomedical Imaging 149 (2301) 13th Street Charlestown, MA 02129 USA
On Fri, May 30, 2008 at 7:43 AM, Paul Raines <raines at nmr.mgh.harvard.edu> wrote:> > It seems when a zfs filesystem with reserv/quota is 100% full users can no > longer even delete files to fix the situation getting errors like these: > > $ rm rh.pm6895.medial.V2.tif > rm: cannot remove `rh.pm6895.medial.V2.tif'': Disk quota exceeded > > (this is over NFS from a RHEL4 Linux box) > > I can log in as root on the Sun server and delete the file as root. > After doing that, the user can then delete files okay. > > Is there anyway to workaround this does not involve root intervention? > Users are filling up their volumes all the time which is the > reason they must have reserv/quota set.Well, with the Copy-on-right filesystem a delete actually requires a write. That said, there have been certain religious arguments on the list about whether the "quota" support presented by ZFS is sufficient. In a nutshell, per user quotas are not implemented, and the suggested workaround is the per-user filesystem with quota/reservations. Its inelegant at best since the auto-mount definitions become their own pain to maintain. The other unimplemented feature is the soft and hard quota limits. Most people have gotten around this by actually presenting only UFS volumes held inside ZFS zvols to end users, but that defeats the purpose of providing snapshots directly to end users, etc. However, since snapshots are only available at the filesystem level, you still are restricted to one filesystem per user to use snapshots well, but I would argue hard/soft limits on the quota are the unanswered problem that doesn''t have a known workaround.> > -- > --------------------------------------------------------------- > Paul Raines email: raines at nmr.mgh.harvard.edu > MGH/MIT/HMS Athinoula A. Martinos Center for Biomedical Imaging > 149 (2301) 13th Street Charlestown, MA 02129 USA > > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >
> It seems when a zfs filesystem with reserv/quota is > 100% full users can no > longer even delete files to fix the situation getting > errors like these: > > $ rm rh.pm6895.medial.V2.tif > rm: cannot remove `rh.pm6895.medial.V2.tif'': Disk > quota exceededWe''ve run into the same problem here. It''s a known problem first mentioned on the ZFS forum in July 2006 and remains unfixed even in Solaris 10 Update 5. The only workaround we found is to truncate the file using something like "cat /dev/null >! file" (for tcsh) since that doesn''t trigger copy-on-write. Unfortunately, it may not be easy to train all users to do that. Note: we have one large zpool on an X4500 Thumper and discovered that even truncation won''t work if the top-level file system 1) has a manually declared quota and 2) is full. So we had to leave the quota turned off at the top level and slightly undercommit the total disk space in user file systems so truncation shell syntax continued to work for NFS users. This message posted from opensolaris.org
This problem is becoming a real pain to us again and I was wondering if there has been in the past few month any known fix or workaround. I normally create zfs fs''s like this: zfs create -o quota=131G -o reserv=131G -o recsize=8K zpool1/newvol and then just nfs export through /etc/dfs/dfstab. We deal with lots of small image files in our MRI data which is the reason for small recsize. On Fri, 30 May 2008, Joe Little wrote:> On Fri, May 30, 2008 at 7:43 AM, Paul Raines <raines at nmr.mgh.harvard.edu> wrote: >> >> It seems when a zfs filesystem with reserv/quota is 100% full users can no >> longer even delete files to fix the situation getting errors like these: >> >> $ rm rh.pm6895.medial.V2.tif >> rm: cannot remove `rh.pm6895.medial.V2.tif'': Disk quota exceeded >> >> (this is over NFS from a RHEL4 Linux box) >> >> I can log in as root on the Sun server and delete the file as root. >> After doing that, the user can then delete files okay. >> >> Is there anyway to workaround this does not involve root intervention? >> Users are filling up their volumes all the time which is the >> reason they must have reserv/quota set. > > Well, with the Copy-on-right filesystem a delete actually requires a > write. That said, there have been certain religious arguments on the > list about whether the "quota" support presented by ZFS is sufficient. > In a nutshell, per user quotas are not implemented, and the suggested > workaround is the per-user filesystem with quota/reservations. Its > inelegant at best since the auto-mount definitions become their own > pain to maintain. > > The other unimplemented feature is the soft and hard quota limits. > Most people have gotten around this by actually presenting only UFS > volumes held inside ZFS zvols to end users, but that defeats the > purpose of providing snapshots directly to end users, etc. > > However, since snapshots are only available at the filesystem level, > you still are restricted to one filesystem per user to use snapshots > well, but I would argue hard/soft limits on the quota are the > unanswered problem that doesn''t have a known workaround. > > > >> >> -- >> --------------------------------------------------------------- >> Paul Raines email: raines at nmr.mgh.harvard.edu >> MGH/MIT/HMS Athinoula A. Martinos Center for Biomedical Imaging >> 149 (2301) 13th Street Charlestown, MA 02129 USA >> >> >> _______________________________________________ >> zfs-discuss mailing list >> zfs-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >> > > >-- --------------------------------------------------------------- Paul Raines email: raines at nmr.mgh.harvard.edu MGH/MIT/HMS Athinoula A. Martinos Center for Biomedical Imaging 149 (2301) 13th Street Charlestown, MA 02129 USA
On 14 August, 2008 - Paul Raines sent me these 2,9K bytes:> This problem is becoming a real pain to us again and I was wondering > if there has been in the past few month any known fix or workaround.Sun is sending me an IDR this/next week regarding this bug.. /Tomas -- Tomas ?gren, stric at acc.umu.se, http://www.acc.umu.se/~stric/ |- Student at Computing Science, University of Ume? `- Sysadmin at {cs,acc}.umu.se
Hello Paul,
Thursday, August 14, 2008, 9:25:45 PM, you wrote:
PR> This problem is becoming a real pain to us again and I was wondering
PR> if there has been in the past few month any known fix or workaround.
PR> I normally create zfs fs''s like this:
PR> zfs create -o quota=131G -o reserv=131G -o recsize=8K zpool1/newvol
PR> and then just nfs export through /etc/dfs/dfstab. We deal with lots
PR> of small image files in our MRI data which is the reason for small
PR> recsize.
If you are creating lots of small files and you are not worried about
accessing only small parts of them with a smaller block, then it
doesn''t really make sense to reduce record size. With standard 128K
recsize if you create a small file, lets say 8K in size, the block
size used by zfs will be actually 8K.
Unless you are doing this to work-around space maps issue.
--
Best regards,
Robert Milkowski mailto:milek at task.gda.pl
http://milek.blogspot.com
I read that that should be case but did not see such in practice. I created one volume without the recsize setting and one with. Than copied the same data to both (lots of small files). The ''du'' report on the one without the recsize was significantly bigger than the one where I made it and in fact filled up before the other volume. On Mon, 18 Aug 2008, Robert Milkowski wrote:> Hello Paul, > > Thursday, August 14, 2008, 9:25:45 PM, you wrote: > > PR> This problem is becoming a real pain to us again and I was wondering > PR> if there has been in the past few month any known fix or workaround. > > PR> I normally create zfs fs''s like this: > > PR> zfs create -o quota=131G -o reserv=131G -o recsize=8K zpool1/newvol > > PR> and then just nfs export through /etc/dfs/dfstab. We deal with lots > PR> of small image files in our MRI data which is the reason for small > PR> recsize. > > If you are creating lots of small files and you are not worried about > accessing only small parts of them with a smaller block, then it > doesn''t really make sense to reduce record size. With standard 128K > recsize if you create a small file, lets say 8K in size, the block > size used by zfs will be actually 8K. > > Unless you are doing this to work-around space maps issue. > > >-- --------------------------------------------------------------- Paul Raines email: raines at nmr.mgh.harvard.edu MGH/MIT/HMS Athinoula A. Martinos Center for Biomedical Imaging 149 (2301) 13th Street Charlestown, MA 02129 USA
On 15 August, 2008 - Tomas ?gren sent me these 0,4K bytes:> On 14 August, 2008 - Paul Raines sent me these 2,9K bytes: > > > This problem is becoming a real pain to us again and I was wondering > > if there has been in the past few month any known fix or workaround. > > Sun is sending me an IDR this/next week regarding this bug..It seems to work, but I am unfortunately not allowed to pass this IDR on. Temporary patch (redistributable) will surface soon and a real patch in 3-6 weeks (Sun Eng estimate). http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6664765 /Tomas -- Tomas ?gren, stric at acc.umu.se, http://www.acc.umu.se/~stric/ |- Student at Computing Science, University of Ume? `- Sysadmin at {cs,acc}.umu.se
On 08/29/08 04:09, Tomas ?gren wrote:> On 15 August, 2008 - Tomas ?gren sent me these 0,4K bytes: > >> On 14 August, 2008 - Paul Raines sent me these 2,9K bytes: >> >>> This problem is becoming a real pain to us again and I was wondering >>> if there has been in the past few month any known fix or workaround. >> Sun is sending me an IDR this/next week regarding this bug.. > > It seems to work, but I am unfortunately not allowed to pass this IDRIDR are "point patches", built against specific kernel builds (IIRC) and as such not intended for a wider distribution. Therefore they need to be tracked so they can be replaced with the proper patch once that is available. If you believe you need the IDR, you need to get in touch with your local services organisation and ask them to get it to you - they know the proper procedures to make sure you get one that works on your machine(s) and that you also get the patch once it''s available. HTH Michael -- Michael Schuster http://blogs.sun.com/recursion Recursion, n.: see ''Recursion''
Thanks Michael for the clarification about the IDR ! :-) I was planing to give this explaination myself. The fix I have in there is a temporary fix. I am currently looking at a better way of accounting the fatzap blocks to make sure we cover all the cases. I have got some pointers from Mark Maybee and am looking into it right now. Thanks and regards, Sanjeev. On Fri, Aug 29, 2008 at 06:59:07AM -0700, Michael Schuster wrote:> On 08/29/08 04:09, Tomas ?gren wrote: > > On 15 August, 2008 - Tomas ?gren sent me these 0,4K bytes: > > > >> On 14 August, 2008 - Paul Raines sent me these 2,9K bytes: > >> > >>> This problem is becoming a real pain to us again and I was wondering > >>> if there has been in the past few month any known fix or workaround. > >> Sun is sending me an IDR this/next week regarding this bug.. > > > > It seems to work, but I am unfortunately not allowed to pass this IDR > > IDR are "point patches", built against specific kernel builds (IIRC) and as > such not intended for a wider distribution. Therefore they need to be > tracked so they can be replaced with the proper patch once that is available. > If you believe you need the IDR, you need to get in touch with your local > services organisation and ask them to get it to you - they know the proper > procedures to make sure you get one that works on your machine(s) and that > you also get the patch once it''s available. > > HTH > Michael > -- > Michael Schuster http://blogs.sun.com/recursion > Recursion, n.: see ''Recursion'' > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
On Aug 29, 2008, at 7:09 AM, Tomas ?gren wrote:> On 15 August, 2008 - Tomas ?gren sent me these 0,4K bytes: > >> On 14 August, 2008 - Paul Raines sent me these 2,9K bytes: >> >>> This problem is becoming a real pain to us again and I was wondering >>> if there has been in the past few month any known fix or workaround.I had this problem in the past. Fortunately I was able to recover by removing an old snapshot which gave me enough room to deal with my problems. Now, I create a fs called reserved and set a small reservation to ensure that there is a small amount of space available. [sferry<@>noroute(0) 12:59 s001] </Users/sferry> [6] zfs get reservation,mountpoint,canmount,type noroute/reserved NAME PROPERTY VALUE SOURCE noroute/reserved reservation 50M local noroute/reserved mountpoint none inherited from noroute noroute/reserved canmount off local noroute/reserved type filesystem - If I fill the pool now, I reduce the reservation (reduce instead of remove in case I have something writing uncontrollably to the pool) and clean up. Shawn -- Shawn Ferry shawn.ferry at sun.com Senior Primary Systems Engineer Sun Managed Operations
On Fri, 29 Aug 2008, Shawn Ferry wrote:> > On Aug 29, 2008, at 7:09 AM, Tomas ?gren wrote: > >> On 15 August, 2008 - Tomas ?gren sent me these 0,4K bytes: >> >>> On 14 August, 2008 - Paul Raines sent me these 2,9K bytes: >>> >>>> This problem is becoming a real pain to us again and I was wondering >>>> if there has been in the past few month any known fix or workaround. > > I had this problem in the past. Fortunately I was able to recover by removing > an old snapshot which gave me enough room to deal with my problems. > > Now, I create a fs called reserved and set a small reservation to ensure that > there is a small amount of space available. > > [sferry<@>noroute(0) 12:59 s001] > </Users/sferry> > [6] zfs get reservation,mountpoint,canmount,type noroute/reserved > NAME PROPERTY VALUE SOURCE > noroute/reserved reservation 50M local > noroute/reserved mountpoint none inherited from noroute > noroute/reserved canmount off local > noroute/reserved type filesystem - > > If I fill the pool now, I reduce the reservation (reduce instead of remove in > case I have something writing uncontrollably to the pool) and clean up.When this problem happens to us, I have no problem deleting a file as root to get things back on track. It is just that normal users cannot delete (who are accessing only over NFS). As soon as I delete a file as root, then normal users can start deleting things themselves.
Hi. I runned into that damn problem too. And after days of searching I finally found this software: Delete Long Path File Tool. It''s GREAT. You can find it here: <a href="http://www.deletelongfile.com">www.deletelongfile.com</a> -- This message posted from opensolaris.org