Hi. I have many small - mostly jpg - files where the original file is approx. 1 MB and the thumbnail generated is approx. 4 KB. The files are currently on vxfs. I have copied all files from one partition onto a zfs-ditto. The vxfs-partition occupies 401 GB and zfs 449 GB. Most files uploaded are in jpg and all thumbnails are always jpg. Will a different volblocksize (during creation of the partition) make better use of the available diskspace? Will (meta)data require less space if compression is enabled? I read http://www.opensolaris.org/jive/thread.jspa?threadID=37673&tstart=105 which is very similar to my case except for the file type. But no clear pointers otherwise. This is a fresh install of opensolaris nevada b64a on a Dell PE840 (test-machine) and three 750 GB sata disks dedicated to zfs as a raidz pool. -- regards Claus When lenity and cruelty play for a kingdom, the gentlest gamester is the soonest winner. Shakespeare
> Will a different volblocksize (during creation of the partition) make > better use of the available diskspace? Will (meta)data require less > space if compression is enabled?Just re-read the evil-tuning-guide and metadata is allready compressed (http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Disabling_Metadata_Compression). -- regards Claus When lenity and cruelty play for a kingdom, the gentlest gamester is the soonest winner. Shakespeare
Claus Guttesen wrote:> I have many small - mostly jpg - files where the original file is > approx. 1 MB and the thumbnail generated is approx. 4 KB. The files > are currently on vxfs. I have copied all files from one partition onto > a zfs-ditto. The vxfs-partition occupies 401 GB and zfs 449 GB. Most > files uploaded are in jpg and all thumbnails are always jpg.Is there a problem? Also, how are you measuring this (what commands)?> Will a different volblocksize (during creation of the partition) make > better use of the available diskspace? Will (meta)data require less > space if compression is enabled?volblocksize won''t have any affect on file systems, it is for zvols. Perhaps you mean recordsize? But recall that recordsize is the maximum limit, not the actual limit, which is decided dynamically.> I read http://www.opensolaris.org/jive/thread.jspa?threadID=37673&tstart=105 > which is very similar to my case except for the file type. But no > clear pointers otherwise.A good start would be to find the distribution of file sizes. -- richard
> > I have many small - mostly jpg - files where the original file is > > approx. 1 MB and the thumbnail generated is approx. 4 KB. The files > > are currently on vxfs. I have copied all files from one partition onto > > a zfs-ditto. The vxfs-partition occupies 401 GB and zfs 449 GB. Most > > files uploaded are in jpg and all thumbnails are always jpg. > > Is there a problem?Not by the diskusage itself. But if zfs takes up more space than vxfs (12 %) 80 TB will become 89 TB instead (our current storage) and add cost.> Also, how are you measuring this (what commands)?I did a ''df -h''.> > Will a different volblocksize (during creation of the partition) make > > better use of the available diskspace? Will (meta)data require less > > space if compression is enabled? > > volblocksize won''t have any affect on file systems, it is for zvols. > Perhaps you mean recordsize? But recall that recordsize is the maximum limit, > not the actual limit, which is decided dynamically. > > > I read http://www.opensolaris.org/jive/thread.jspa?threadID=37673&tstart=105 > > which is very similar to my case except for the file type. But no > > clear pointers otherwise. > > A good start would be to find the distribution of file sizes.The files are approx. 1 MB with an thumbnail of approx. 4 KB. -- regards Claus When lenity and cruelty play for a kingdom, the gentlest gamester is the soonest winner. Shakespeare
Claus Guttesen writes: > > > I have many small - mostly jpg - files where the original file is > > > approx. 1 MB and the thumbnail generated is approx. 4 KB. The files > > > are currently on vxfs. I have copied all files from one partition onto > > > a zfs-ditto. The vxfs-partition occupies 401 GB and zfs 449 GB. Most > > > files uploaded are in jpg and all thumbnails are always jpg. > > > > Is there a problem? > > Not by the diskusage itself. But if zfs takes up more space than vxfs > (12 %) 80 TB will become 89 TB instead (our current storage) and add > cost. > > > Also, how are you measuring this (what commands)? > > I did a ''df -h''. > > > > Will a different volblocksize (during creation of the partition) make > > > better use of the available diskspace? Will (meta)data require less > > > space if compression is enabled? > > > > volblocksize won''t have any affect on file systems, it is for zvols. > > Perhaps you mean recordsize? But recall that recordsize is the maximum limit, > > not the actual limit, which is decided dynamically. > > > > > I read http://www.opensolaris.org/jive/thread.jspa?threadID=37673&tstart=105 > > > which is very similar to my case except for the file type. But no > > > clear pointers otherwise. > > > > A good start would be to find the distribution of file sizes. > > The files are approx. 1 MB with an thumbnail of approx. 4 KB. > So the 1 MB files are stored as ~8 x 128K recordsize. Because of 5003563 use smaller "tail block" for last block of object The last block of you file is partially used. It will depend on your filesize distribution by without that info we can only guess that we''re wasting an avg of 64K per file. Or 6%. If your distribution is such that most files are slightly more than 1M, then we''d have 12% overhead from this effect. So using 16K/32K recordsize would quite possibly help as files would be stored using ~ 64 x 16K blocks with an overhead of < 1-2% (0.5 blocks wasted every 64). -r > -- > regards > Claus > > When lenity and cruelty play for a kingdom, > the gentlest gamester is the soonest winner. > > Shakespeare > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
> So the 1 MB files are stored as ~8 x 128K recordsize. > > Because of > 5003563 use smaller "tail block" for last block of object > > The last block of you file is partially used. It will depend > on your filesize distribution by without that info we can > only guess that we''re wasting an avg of 64K per file. Or 6%. > > If your distribution is such that most files are slightly > more than 1M, then we''d have 12% overhead from this effect. > > So using 16K/32K recordsize would quite possibly help as > files would be stored using ~ 64 x 16K blocks with an > overhead of < 1-2% (0.5 blocks wasted every 64).I will (re)create the partition and modify the recordsize. I was unwilling to do so when I read the man page which discourages modifying this setting unless a database was used. Does zfs use suballocation if a file does not use an entire recordsize? If not the thumbnails probably wastes most space. They are approx. 4 KB. I''ll be testing recordsizes from 1K and upwards. Actually 1K made zfs very slow but 2K seems fine. I''ll report back when the entire partition has been copied. When I find the sweet spot I''ll try to enable (default) compression. Thank you. -- regards Claus When lenity and cruelty play for a kingdom, the gentlest gamester is the soonest winner. Shakespeare
Claus Guttesen writes: > > So the 1 MB files are stored as ~8 x 128K recordsize. > > > > Because of > > 5003563 use smaller "tail block" for last block of object > > > > The last block of you file is partially used. It will depend > > on your filesize distribution by without that info we can > > only guess that we''re wasting an avg of 64K per file. Or 6%. > > > > If your distribution is such that most files are slightly > > more than 1M, then we''d have 12% overhead from this effect. > > > > So using 16K/32K recordsize would quite possibly help as > > files would be stored using ~ 64 x 16K blocks with an > > overhead of < 1-2% (0.5 blocks wasted every 64). > > I will (re)create the partition and modify the recordsize. I was > unwilling to do so when I read the man page which discourages > modifying this setting unless a database was used. > > Does zfs use suballocation if a file does not use an entire > recordsize? If not the thumbnails probably wastes most space. They are > approx. 4 KB. > Files smaller than ''recordsize'' are stored using a multiple of the sector size. So small files should not factor in this equation. > I''ll be testing recordsizes from 1K and upwards. Actually 1K made zfs > very slow but 2K seems fine. I''ll report back when the entire > partition has been copied. When I find the sweet spot I''ll try to > enable (default) compression. Beware because at 2K you might be generating more indirect blocks. For 1MB files the gains from using a recordsize smaller than 16K start to be quite small. -r > > Thank you. > > -- > regards > Claus > > When lenity and cruelty play for a kingdom, > the gentlest gamester is the soonest winner. > > Shakespeare
> > The files are approx. 1 MB with an thumbnail of approx. 4 KB. > > > > So the 1 MB files are stored as ~8 x 128K recordsize. > > Because of > 5003563 use smaller "tail block" for last block of objectI tried to search for this but only saw references to this and similar threads. Is there a database where I can search?> The last block of you file is partially used. It will depend > on your filesize distribution by without that info we can > only guess that we''re wasting an avg of 64K per file. Or 6%. > > If your distribution is such that most files are slightly > more than 1M, then we''d have 12% overhead from this effect. > > So using 16K/32K recordsize would quite possibly help as > files would be stored using ~ 64 x 16K blocks with an > overhead of < 1-2% (0.5 blocks wasted every 64).Variying recordsizes and no compression: Recordsize=2k: 426G Recordsize=4k: 417G Recordsize=8k: 411G Recordsize=16k: 412G Recordsize=32k: 417G Default recordsize of 128k and compression: Standard-compression: 410G Gzip-1: 402G Gzip-6: 402G Recordsize=16k and standard compression: 406G Recordsize 8- and 16k comes closest to 401 GB on vxfs. Should I try with a single hardrive and verify 16k recordsize disk usage? -- regards Claus When lenity and cruelty play for a kingdom, the gentlest gamester is the soonest winner. Shakespeare
On 24 September, 2007 - Claus Guttesen sent me these 1,5K bytes:> > > The files are approx. 1 MB with an thumbnail of approx. 4 KB. > > > > > > > So the 1 MB files are stored as ~8 x 128K recordsize. > > > > Because of > > 5003563 use smaller "tail block" for last block of object > > I tried to search for this but only saw references to this and similar > threads. Is there a database where I can search?http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=5003563 /Tomas -- Tomas ?gren, stric at acc.umu.se, http://www.acc.umu.se/~stric/ |- Student at Computing Science, University of Ume? `- Sysadmin at {cs,acc}.umu.se