zfs discuss - Jul 2008 - Largest (in number of files) ZFS instance tested

I need to find out what is the largest ZFS file system - in numbers of 
files, NOT CAPACITY that has been tested.

Looking to scale to billions of files and would like to know if anyone 
has tested anything close and what the performance ramifications are.

Has anyone tested a ZFS file system with at least 100 million + files?
What were the performance characteristics?

Thanks!

Sean

-- 
<http://www.sun.com> 	* Sean Cochrane *
Global Storage Architect
*Sun Microsystems, Inc.*
525 South 1100 East
Salt Lake City, UT 84102 US
Phone +1877 255 5756
Mobile +1801-949-4799
Fax +1877.255.5756
Email Sean.Cochrane at Sun.COM

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080711/6e8f1af5/attachment.html>

On Fri, 11 Jul 2008, Sean Cochrane - Storage Architect wrote:
> I need to find out what is the largest ZFS file system - in numbers of
files,
> NOT CAPACITY that has been tested.
> 
> Looking to scale to billions of files and would like to know if anyone has
> tested anything close and what the performance ramifications are.
Wow.  Just curious, what sort of application is this?

-- 
Rich Teer, SCSA, SCNA, SCSECA

CEO,
My Online Home Inventory

URLs: http://www.rite-group.com/rich
      http://www.linkedin.com/in/richteer
      http://www.myonlinehomeinventory.com

On Fri, 11 Jul 2008, Sean Cochrane - Storage Architect wrote:
> I need to find out what is the largest ZFS file system - in numbers of
files,
> NOT CAPACITY that has been tested.
In response to an earlier such question (from you?) I created a 
directory with a million files.  I forgot about it since then so the 
million files are still there without impacting anything for a month 
now.

The same simple script (with a small enhancement) could be used to 
create a million directories containing a million files but it might 
take a while to complete.  It seems that a Storage Architect should be 
willing to test this for himself and see what happens.
> Looking to scale to billions of files and would like to know if anyone has 
> tested anything close and what the performance ramifications are.
There are definitely issues with programs like ''ls'' when
listing a
directory with a million files since ''ls'' sorts its output by
default.
My Windows system didn''t like it at all when accessing it with CIFS 
and the file browser since it wants to obtain all file information 
before doing anything else.  System backup with hundreds of millions 
of files sounds like fun.
> Has anyone tested a ZFS file system with at least 100 million + files?
> What were the performance characteristics?
I think that there are more issues with file fragmentation over a long 
period of time than the sheer number of files.

Bob
=====================================Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

On Jul 11, 2008, at 4:59 PM, Bob Friesenhahn wrote:
>>
>> Has anyone tested a ZFS file system with at least 100 million +  
>> files?
>> What were the performance characteristics?
>
> I think that there are more issues with file fragmentation over a long
> period of time than the sheer number of files.
actually it''s a similar problem .. with a maximum blocksize of 128KB  
and the COW nature of the filesytem you get indirect block pointers  
pretty quickly on a large ZFS filesystem as the size of your tree  
grows .. in this case a large constantly modified file (eg: /u01/data/ 
*.dbf) is going to behave over time like a lot of random access to  
files spread across the filesystem .. the only real difference is that  
you won''t walk it every time someone does a getdirent() or an lstat64()

so ultimately the question could be framed as what''s the maximum  
manageable tree size you can get to with ZFS while keeping in mind  
that there''s no real re-layout tool (by design) .. the number
i''m
working with until i hear otherwise is probably about 20M, but in the  
relativistic sense - it *really* does depend on how balanced your tree  
is and what your churn rate is .. we know on QFS we can go up to 100M,  
but i trust the tree layout a little better there, can separate the  
metadata out if i need to and have planned on it, and know that we''ve  
got some tools to relayout the metadata or dump/restore for a tape  
backed archive

jonathan

(oh and btw - i believe this question is a query for field data ..  
architect != crash test dummy .. but some days it does feel like it)

On Fri, Jul 11, 2008 at 5:33 PM, Sean Cochrane - Storage Architect
<Sean.Cochrane at sun.com> wrote:
>
> I need to find out what is the largest ZFS file system - in numbers of
> files, NOT CAPACITY that has been tested.
>
> Looking to scale to billions of files and would like to know if anyone has
> tested anything close and what the performance ramifications are.
>
> Has anyone tested a ZFS file system with at least 100 million + files?
I''ve got a thumper with a pool that has over a hundred million files. I
think
the most in a single filesystem is currently just under 30 million
(we''ve got
plenty of those). It just works, although it''s going to get a lot
bigger before
we''re done.
> What were the performance characteristics?
Not brilliant...

Although I suspect raid-z isn''t exactly the ideal choice. Still,
performance
generally is adequate for our needs, although backup performance isn''t.

(The backup problem is the real stumbling block. And backup is an area ripe
for disruptive innovation.)

-- 
-Peter Tribble
http://www.petertribble.co.uk/ - http://ptribble.blogspot.com/

Peter Tribble wrote:
> On Fri, Jul 11, 2008 at 5:33 PM, Sean Cochrane - Storage Architect
> <Sean.Cochrane at sun.com> wrote:
>> What were the performance characteristics?
>>     
>
> Not brilliant...
>
> Although I suspect raid-z isn''t exactly the ideal choice. Still,
performance
> generally is adequate for our needs, although backup performance
isn''t.
>
> (The backup problem is the real stumbling block. And backup is an area ripe
> for disruptive innovation.)
>
>   
Is down to volume of data, or many small files? 

I''m look into a problem with slow backup of a filesystem with many
thousands for small files.  We see high CPU load and miserable
performance on restores and I''ve been wondering if we can tune the
filesystem, or just zip the files.

I guess working with many small files and tape is more of an issue with
filesystem aware backups than block device ones (ufsdump).

Ian.

On Fri, Jul 11, 2008 at 3:59 PM, Bob Friesenhahn
<bfriesen at simple.dallas.tx.us> wrote:
> There are definitely issues with programs like ''ls'' when
listing a
> directory with a million files since ''ls'' sorts its
output by default.
> My Windows system didn''t like it at all when accessing it with
CIFS
> and the file browser since it wants to obtain all file information
> before doing anything else.  System backup with hundreds of millions
> of files sounds like fun.
Millions of files in a directory has historically been the path to big
performance problems.  Even if zfs can handle millions, other tools
(ls, backup programs, etc.) will choke.  Create a hierarchy and you
will be much happier.

FWIW, I created 10+ million files and the necessary directories to
make it so that no directory had more than 10 entries (dirs or files)
in it.  I found the creation time to be quite steady at about 2500
file/directory creations per second over the entire exercise.  I saw
the kernel memory usage (kstat -p unix:0:system_pages:pp_kernel)
slowly and steadily increase while arc_c slowly decreased.  Out of
curiosity I crashed the box then ran ::findleaks to find that there
was just over 32KB leaked.  I''ve not dug in further to see where the
rest of the memory was used.

In the past when I was observing file creations on UFS, VxFS, and NFS
with millions of files in a single directory, the file operation time
was measured in seconds per operation, rather than operations per
second.  This was with several (<100) processes contending for reading
directory contents, file creations, and file deletions.  This is where
I found the script that though that "touch $dir/test.$$" (followed by
rm) was the right way to check to see if a directory is writable.

-- 
Mike Gerdts
http://mgerdts.blogspot.com/

On Sat, Jul 12, 2008 at 12:23 AM, Ian Collins <ian at ianshome.com>
wrote:
> Peter Tribble wrote:
>>
>> (The backup problem is the real stumbling block. And backup is an area
ripe
>> for disruptive innovation.)
>>
>>
> Is down to volume of data, or many small files?
Many small files. We could handle many more bytes.
> I''m look into a problem with slow backup of a filesystem with many
> thousands for small files.  We see high CPU load and miserable
> performance on restores and I''ve been wondering if we can tune the
> filesystem, or just zip the files.
>
> I guess working with many small files and tape is more of an issue with
> filesystem aware backups than block device ones (ufsdump).
Err, no. Ever tried to ufsrestore a filesystem with an enormous number of files?
The last time I tried this ufsrestore got to 4G and then collapsed as
it was only
a 32-bit application. (Basically, reading in the directory information took way
more than 4G.)

-- 
-Peter Tribble
http://www.petertribble.co.uk/ - http://ptribble.blogspot.com/