Is it possible to deplete inodes on zfs?? snv_27 We are testing library call behavior under inode depletion conditions, however our setup scripts appear to run the (very small, 10M) legacy zfs filesystem out of space and not out of inodes. The test-setup method is to create 0 block files in a directory until it gets an error. We also use reservations on other filesystems to limit the space available to our test filesystem. When I try to rm -rf the "inode" directory, it makes this fine complaint. rm: Unable to remove directory /mnt/inodes: No space left on device I end up having to destroy the zfs filesystem. Anyway I am curious about this inode depletion issue, if it is just not possible, we''ll need to re-calculate our testing accordingly, they are based on ufs expectations.
Eric Ahlberg wrote On 11/29/05 19:24,:> Is it possible to deplete inodes on zfs??There is no separate inode pool in ZFS. The inodes (actually dnodes) are just allocated on demand from available pool space, which of course is possible to deplete.> > snv_27 > > We are testing library call behavior under > inode depletion conditions, however our setup > scripts appear to run the (very small, 10M) legacy zfs filesystem > out of space and not out of inodes.Right. See above.> The test-setup method is to create 0 block files > in a directory until it gets an error. > We also use reservations on other filesystems to > limit the space available to our test filesystem. > > When I try to rm -rf the "inode" directory, it makes this fine > complaint. > rm: Unable to remove directory /mnt/inodes: No space left on deviceWe have seen this before, but I thought it was now fixed. I don''t know if there''s a bug filed on it. I expect MarkM will know and if not file one.> > I end up having to destroy the zfs filesystem. > > Anyway I am curious about this inode depletion issue, > if it is just not possible, we''ll need to re-calculate our > testing accordingly, they are based on ufs expectations. > > > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://opensolaris.org/mailman/listinfo/zfs-discuss-- Neil
What I suspect is happening here is that the inode directory itself ends up being the consumer of most of the space in the file system. The removal of directories has not been tuned for the edge-condition when a file system is out-of-space (as normal file deletion has been). Please, as Neil suggests, file a bug on this. Note, you could have worked around this problem by removing one of the reservations you gave to another file system, so that there would be enough space available to complete the delete (delete operations consume a small amount of space before they release space in zfs). -Mark Neil Perrin wrote:>>The test-setup method is to create 0 block files >>in a directory until it gets an error. >>We also use reservations on other filesystems to >>limit the space available to our test filesystem. >> >>When I try to rm -rf the "inode" directory, it makes this fine >>complaint. >>rm: Unable to remove directory /mnt/inodes: No space left on device > > > We have seen this before, but I thought it was now fixed. > I don''t know if there''s a bug filed on it. I expect MarkM > will know and if not file one.
Make sure you don''t have a snapshot in the fs; if that''s the case, rm can and will run the system out of space because there''s no space left for new directory blocks. - Bart -- Bart Smaalders Solaris Kernel Performance barts at cyber.eng.sun.com http://blogs.sun.com/barts
On Tue, 2005-11-29 at 18:50, Neil Perrin wrote:> Eric Ahlberg wrote On 11/29/05 19:24,: > > Is it possible to deplete inodes on zfs?? > > There is no separate inode pool in ZFS. The inodes (actually dnodes) > are just allocated on demand from available pool space, which > of course is possible to deplete.OK possible...how painful? Is there a rule for dnodes per zpool container size? Is there a rule for dnodes per filesystem size? What happens when an evil program goes about making 0 block files forever until something breaks? Our current script, which just sits in a loop making 0 block files, makes about 4000/hour (painful slowness due to zfs performance issues in snv_27), and exits only on ENOSPC. We force it below 1M free space before we start. This approach could take an unreasonably long time, unless we start with a really small zpool container. Hopefully, performance improvements will help us break it sooner. If making 0 block files will always deplete space before depleting dnodes, dnode depletion would never be reached!?
On Wed, Nov 30, 2005 at 10:52:13AM -0800, Eric Ahlberg wrote:> On Tue, 2005-11-29 at 18:50, Neil Perrin wrote: > > Eric Ahlberg wrote On 11/29/05 19:24,: > > > Is it possible to deplete inodes on zfs?? > > > > There is no separate inode pool in ZFS. The inodes (actually dnodes) > > are just allocated on demand from available pool space, which > > of course is possible to deplete. > > OK possible...how painful? > > Is there a rule for dnodes per zpool container size? > Is there a rule for dnodes per filesystem size? > What happens when an evil program goes about making 0 block files > forever until something breaks?Any evil program can use up space, by either creating lots of small files, or making huge jumbo files. With ZFS, there is no space pre-allocation for dnodes; space for them comes out of the same place as space for data blocks. Nothing "breaks", you just run out of space.> Our current script, which just sits in a loop making 0 block files, > makes about 4000/hour (painful slowness due to zfs performance issues in > snv_27), and exits only on ENOSPC. We force it below 1M free spaceThat seems extremely slow; How are you creating the files? Doing: % mkdir /aux1/foo % cd /aux1/foo % time touch `/home/dp/bin/range 1 100000` touch `/home/dp/bin/range 1 100000` 1.64s user 43.66s system 93% cpu 48.232 total (range is a script that generates a full range of numbers) 48 seconds for 100,000 files; that''s 2000/sec. And this is on DEBUG bits.> before we start. This approach could take an unreasonably long > time, unless we start with a really small zpool container. > Hopefully, performance improvements will help us break it sooner. > > If making 0 block files will always deplete space > before depleting dnodes, dnode depletion would > never be reached!?The storage comes from the same place; there''s no such thing as "dnode depletion"; there''s only "no space available". Period. And that''s a Good Thing. Cheers, - jonathan -- Jonathan Adams, Solaris Kernel Development
As Neil said: there is no fixed number of file nodes in a ZFS file system. You can continue to create new files as long as there is space. Note that the space used to create the file node is part of the file. There is no such thing as a "0 block file" in ZFS. So the notion of some "evil program making files until something breaks" is no different that any other ill-behaved application running the file system out of space. Eric Ahlberg wrote:> On Tue, 2005-11-29 at 18:50, Neil Perrin wrote: > >>Eric Ahlberg wrote On 11/29/05 19:24,: >> >>>Is it possible to deplete inodes on zfs?? >> >>There is no separate inode pool in ZFS. The inodes (actually dnodes) >>are just allocated on demand from available pool space, which >>of course is possible to deplete. > > > OK possible...how painful? > > Is there a rule for dnodes per zpool container size? > Is there a rule for dnodes per filesystem size? > What happens when an evil program goes about making 0 block files > forever until something breaks? > > Our current script, which just sits in a loop making 0 block files, > makes about 4000/hour (painful slowness due to zfs performance issues in > snv_27), and exits only on ENOSPC. We force it below 1M free space > before we start. This approach could take an unreasonably long > time, unless we start with a really small zpool container. > Hopefully, performance improvements will help us break it sooner. > > If making 0 block files will always deplete space > before depleting dnodes, dnode depletion would > never be reached!? > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://opensolaris.org/mailman/listinfo/zfs-discuss