Robert Milkowski
2006-Mar-07 10:15 UTC
[zfs-discuss] ZFS filesystem with a lot of small files consumes more that 2x disk space
Hi.
I created a raid 10 zfs pool using 12 disks. Then I created SVM stripe from 6
disks (the same type). First I copied production data using rsync on zfs
filesystem. Then I copied from zfs filesystem to UFS. While zfs filesystem is
full (400GB) it consumed only 171GB on UFS. So in this case zfs took 2.3X space
than UFS.
I wanted to copy some production data (the same disk space size) and was caught
by surprise that I run out of space.
Some more details:
This is SPARC, zfs based on b32.
bash-3.00# zpool status p-16-32
pool: p-16-32
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
p-16-32 ONLINE 0 0 0
mirror ONLINE 0 0 0
c5t500000E011909320d0 ONLINE 0 0 0
c5t500000E011902FB0d0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c5t500000E011909300d0 ONLINE 0 0 0
c5t500000E0119030F0d0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c5t500000E011903030d0 ONLINE 0 0 0
c5t500000E01190E730d0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c5t500000E011903300d0 ONLINE 0 0 0
c5t500000E01190E7F0d0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c5t500000E011903120d0 ONLINE 0 0 0
c5t500000E0119091E0d0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c5t500000E01190E750d0 ONLINE 0 0 0
c5t500000E0119032D0d0 ONLINE 0 0 0
bash-3.00#
bash-3.00# zfs list p-16-32/d606
NAME USED AVAIL REFER MOUNTPOINT
p-16-32/d606 405G 0 405G /opt/d606
bash-3.00#
bash-3.00# zfs list | grep p-16-32
p-16-32 405G 0 98.5K /p-16-32
p-16-32/d606 405G 0 405G /opt/d606
bash-3.00#
bash-3.00# metastat -p d60
d60 -m d61 1
d61 1 6 /dev/dsk/c5t500000E01192B2F0d0s0 /dev/dsk/c5t500000E01194A760d0s0
/dev/dsk/c5t500000E01192B290d0s0 /dev/dsk/c5t500000E0118B0390d0s0
/dev/dsk/c5t500000E0118C3220d0s0 /dev/dsk/c5t500000E0118ABA70d0s0 -i 64b
bash-3.00#
bash-3.00# df -h /mnt/svm-d606
Filesystem size used avail capacity Mounted on
/dev/md/dsk/d60 404G 171G 229G 43% /mnt/svm-d606
bash-3.00# df -h /opt/d606
Filesystem size used avail capacity Mounted on
p-16-32/d606 405G 405G 0K 100% /opt/d606
bash-3.00#
bash-3.00# df -g /mnt/svm-d606
/mnt/svm-d606 (/dev/md/dsk/d60 ): 8192 block size 1024
frag size
847106926 total blocks 487993124 free blocks 479522056 available 50960000
total files
20203679 free files 22282300 filesys id
ufs fstype 0x00000004 flag 255 filename length
bash-3.00# df -g /opt/d606
/opt/d606 (p-16-32/d606 ): 131072 block size 512
frag size
848954130 total blocks 0 free blocks 0 available 30756321
total files
0 free files 66650134 filesys id
zfs fstype 0x00000004 flag 255 filename length
bash-3.00#
This message posted from opensolaris.org
Matthew Ahrens
2006-Mar-07 20:10 UTC
[zfs-discuss] ZFS filesystem with a lot of small files consumes more that 2x disk space
On Tue, Mar 07, 2006 at 02:15:57AM -0800, Robert Milkowski wrote:> Hi. > > I created a raid 10 zfs pool using 12 disks. Then I created SVM > stripe from 6 disks (the same type). First I copied production data > using rsync on zfs filesystem. Then I copied from zfs filesystem to > UFS. While zfs filesystem is full (400GB) it consumed only 171GB on > UFS. So in this case zfs took 2.3X space than UFS. > > I wanted to copy some production data (the same disk space size) and > was caught by surprise that I run out of space. > > bash-3.00# df -h /mnt/svm-d606 > Filesystem size used avail capacity Mounted on > /dev/md/dsk/d60 404G 171G 229G 43% /mnt/svm-d606 > bash-3.00# df -h /opt/d606 > Filesystem size used avail capacity Mounted on > p-16-32/d606 405G 405G 0K 100% /opt/d606 > bash-3.00#Can you tell us: 1. Most importantly: the output of ''zdb -bb p-16-32'' 2. What version of ZFS you are running? 3. What size files are you creating? FYI, you may be running into 6391873 metadata compression should be turned back on which has been fixed in build 36, or 5003563 use smaller "tail block" for last block of object which is still open. --matt
Bill Sommerfeld
2006-Mar-07 21:10 UTC
[zfs-discuss] ZFS filesystem with a lot of small files consumes more that 2x disk space
On Tue, 2006-03-07 at 15:10, Matthew Ahrens wrote:> , or > 5003563 use smaller "tail block" for last block of object > which is still open.possibly stupid question: is 5003563 only an issue if you''re not using compression on user data? I''d assume trailing zeros at the end of the last block would compress really well... - Bill
Matthew Ahrens
2006-Mar-07 21:20 UTC
[zfs-discuss] ZFS filesystem with a lot of small files consumes more that 2x disk space
On Tue, Mar 07, 2006 at 04:10:07PM -0500, Bill Sommerfeld wrote:> On Tue, 2006-03-07 at 15:10, Matthew Ahrens wrote: > > , or > > 5003563 use smaller "tail block" for last block of object > > which is still open. > > possibly stupid question: is 5003563 only an issue if you''re not using > compression on user data? I''d assume trailing zeros at the end of the > last block would compress really well...Yes, if you have turned on compression of user data (''zfs set compression=on''), 5003563 will not be a problem. It is primarily an issue if you have a lot of 129kb-200kb files, since those will be stored as two 128k blocks, thus resulting in up to 2x expansion. If it turns out that this is a problem, an easy solution might be to simply always compress the last block of a file. --matt
Joe Little
2006-Mar-08 01:30 UTC
[zfs-discuss] ZFS filesystem with a lot of small files consumes more that 2x disk space
In my various tests, I have seen large expansion as well, and wasn''t sure it was simply RAIDZ using up more disk space over time, as it originally shows me full use of all drive capacity when originally formatted. In the case of our primary mirror, which I tested hosting on zfs/raidz/iscsi, my 1TB full of various linux distros, OS projects, etc took 1.4TB on zfs. On 3/7/06, Matthew Ahrens <ahrens at sun.com> wrote:> On Tue, Mar 07, 2006 at 04:10:07PM -0500, Bill Sommerfeld wrote: > > On Tue, 2006-03-07 at 15:10, Matthew Ahrens wrote: > > > , or > > > 5003563 use smaller "tail block" for last block of object > > > which is still open. > > > > possibly stupid question: is 5003563 only an issue if you''re not using > > compression on user data? I''d assume trailing zeros at the end of the > > last block would compress really well... > > Yes, if you have turned on compression of user data (''zfs set compression=on''), > 5003563 will not be a problem. It is primarily an issue if you have a > lot of 129kb-200kb files, since those will be stored as two 128k blocks, > thus resulting in up to 2x expansion. If it turns out that this is a > problem, an easy solution might be to simply always compress the last > block of a file. > > --matt > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >
Matthew Ahrens
2006-Mar-08 01:40 UTC
[zfs-discuss] ZFS filesystem with a lot of small files consumes more that 2x disk space
On Tue, Mar 07, 2006 at 05:30:42PM -0800, Joe Little wrote:> In my various tests, I have seen large expansion as well, and wasn''t > sure it was simply RAIDZ using up more disk space over time, as it > originally shows me full use of all drive capacity when originally > formatted. In the case of our primary mirror, which I tested hosting > on zfs/raidz/iscsi, my 1TB full of various linux distros, OS projects, > etc took 1.4TB on zfs.I''d be happy to try and diagnose this, if you can send the same info (either to the list, or directly to me if you): 1. Most importantly: the output of ''zdb -bb <poolname>'' 2. What version of ZFS you are running? 3. What size files are you creating (approximately)? --matt
Joe Little
2006-Mar-08 04:12 UTC
[zfs-discuss] ZFS filesystem with a lot of small files consumes more that 2x disk space
Well, the data in question has seen been migrated to local disk on linux. It was a full test I did to see the viability of zfs/iscsi-based NAS solutions. At the time, I think I was running B32. I still have a ZFS pool that''s disabled. I''ll try and run it with B33 (I only use non-debug SXCR builds where possible, as I got half that MB/sec throughput on BFUed builds -- whether ZFS or iscsi related, unsure). As for file characterization, it is all over the map. thousand of <128K files (this debian repository with description files), thousands of 128k-16M files (think RPMs, debs), and hundreds of ISO images around 650MB, as well as DVD images. On 3/7/06, Matthew Ahrens <ahrens at sun.com> wrote:> On Tue, Mar 07, 2006 at 05:30:42PM -0800, Joe Little wrote: > > In my various tests, I have seen large expansion as well, and wasn''t > > sure it was simply RAIDZ using up more disk space over time, as it > > originally shows me full use of all drive capacity when originally > > formatted. In the case of our primary mirror, which I tested hosting > > on zfs/raidz/iscsi, my 1TB full of various linux distros, OS projects, > > etc took 1.4TB on zfs. > > I''d be happy to try and diagnose this, if you can send the same info > (either to the list, or directly to me if you): > > 1. Most importantly: the output of ''zdb -bb <poolname>'' > 2. What version of ZFS you are running? > 3. What size files are you creating (approximately)? > > --matt >
Akhilesh Mritunjai
2006-Mar-08 19:05 UTC
[zfs-discuss] Re: ZFS filesystem with a lot of small files consumes more that 2x disk space
Joe, A utility called JDiskReport (http://www.jgoodies.com/freeware/jdiskreport/) can help analyze the sizes of files. This utility can output a number of diagnostics on file sizes and numbers. Might be helpful in pinpointing the issue if its related to file sizes. It can export the data as CSV (please see the FAQ on the site). - Akhilesh This message posted from opensolaris.org
Richard Elling
2006-Mar-08 19:50 UTC
[zfs-discuss] Re: ZFS filesystem with a lot of small files consumes more that 2x disk s
JDiskReport is also in your gnome menu as "Disk Analyzer" under the "System Tools" launch menu (JDS4). I don''t recall offhand which menu it is under for other JDS Solaris/Linux releases. But it has been there for a few years now. Its pretty cool. -- richard This message posted from opensolaris.org
Robert Milkowski
2006-Mar-09 02:29 UTC
[zfs-discuss] Re: ZFS filesystem with a lot of small files consumes more that 2x disk s
Hi.
1.
bash-3.00# zdb -bb p-16-32
Traversing all blocks to verify nothing leaked ...
No leaks (block sum matches space maps exactly)
bp count: 32678637
bp logical: 434608403968 avg: 13299
bp physical: 434608403968 avg: 13299 compression: 1.00
bp allocated: 434665083904 avg: 13301 compression: 1.00
SPA allocated: 434665083904 used: 99.22%
Blocks LSIZE PSIZE ASIZE avg comp %Total Type
49 556K 556K 556K 11.3K 1.00 0.00 deferred free
1 512 512 512 512 1.00 0.00 object directory
18 9.00K 9.00K 9.00K 512 1.00 0.00 object array
1 16K 16K 16K 16K 1.00 0.00 packed nvlist
- - - - - - - packed nvlist size
1 16K 16K 16K 16K 1.00 0.00 bplist
- - - - - - - bplist header
- - - - - - - SPA space map header
9.31K 46.8M 46.8M 46.8M 5.03K 1.00 0.01 SPA space map
- - - - - - - ZIL intent log
947K 14.8G 14.8G 14.8G 16K 1.00 3.65 DMU dnode
3 3.00K 3.00K 3.00K 1K 1.00 0.00 DMU objset
- - - - - - - DSL directory
3 1.50K 1.50K 1.50K 512 1.00 0.00 DSL directory child map
2 1K 1K 1K 512 1.00 0.00 DSL dataset snap map
4 257K 257K 257K 64.2K 1.00 0.00 DSL props
- - - - - - - DSL dataset
- - - - - - - ZFS znode
- - - - - - - ZFS ACL
18.7M 156G 156G 156G 8.38K 1.00 38.62 ZFS plain file
11.6M 234G 234G 234G 20.2K 1.00 57.72 ZFS directory
2 1K 1K 1K 512 1.00 0.00 ZFS master node
2 1K 1K 1K 512 1.00 0.00 ZFS delete queue
- - - - - - - zvol object
- - - - - - - zvol prop
- - - - - - - other uint8[]
- - - - - - - other uint64[]
- - - - - - - other ZAP
31.2M 405G 405G 405G 13.0K 1.00 100.00 Total
bash-3.00#
2. ZFS based on b32 (zfs-s10-child)
3. There''s over 9mln files on that filesystem, and over 11mln symbolink
links. Most of the files should be small - few KBs. Some of the files could be
larger, even few MBs.
4.
I copied all data from the same filesystem to compressed one and still ZFS
consumes more than 2x space than UFS.
bash-3.00# df -h /mnt/d606
Filesystem size used avail capacity Mounted on
10.0.3.99:/opt/d606 402G 187G 195G 49% /mnt/d606
bash-3.00# df -h /opt/d606
Filesystem size used avail capacity Mounted on
p-16-32/d606 405G 405G 0K 100% /opt/d606
bash-3.00# df -h /p-22-38/d606-compressed
Filesystem size used avail capacity Mounted on
p-22-38/d606-compressed
405G 386G 19G 96% /p-22-38/d606-compressed
bash-3.00#
This message posted from opensolaris.org
Robert Milkowski
2006-Mar-09 02:30 UTC
[zfs-discuss] Re: ZFS filesystem with a lot of small files consumes more that 2x disk s
I copied all data from the same filesystem to compressed one and still ZFS consumes more than 2x space than UFS. bash-3.00# df -h /mnt/d606 Filesystem size used avail capacity Mounted on 10.0.3.99:/opt/d606 402G 187G 195G 49% /mnt/d606 bash-3.00# df -h /opt/d606 Filesystem size used avail capacity Mounted on p-16-32/d606 405G 405G 0K 100% /opt/d606 bash-3.00# df -h /p-22-38/d606-compressed Filesystem size used avail capacity Mounted on p-22-38/d606-compressed 405G 386G 19G 96% /p-22-38/d606-compressed bash-3.00# This message posted from opensolaris.org
Joe Little
2006-Mar-09 02:58 UTC
[zfs-discuss] Re: ZFS filesystem with a lot of small files consumes more that 2x disk space
Sadly, I have now JDS installed (no X, headless, and not X11 accessible with the network topology). I just synced 15Gb of files from linux/xfs to B33 and ZFS. Waiting on the zdb results, but its now 19GB in size. On 3/8/06, Akhilesh Mritunjai <mritun+opensolaris at gmail.com> wrote:> Joe, > > A utility called JDiskReport (http://www.jgoodies.com/freeware/jdiskreport/) can help analyze the sizes of files. > > This utility can output a number of diagnostics on file sizes and numbers. Might be helpful in pinpointing the issue if its related to file sizes. It can export the data as CSV (please see the FAQ on the site). > > - Akhilesh > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >
Joe Little
2006-Mar-09 04:14 UTC
[zfs-discuss] Re: ZFS filesystem with a lot of small files consumes more that 2x disk space
This took over an hour.. (again, 19GB usage of 15GB of XFS originally
stored files; some links/rpm/isos/etc)
Traversing all blocks to verify nothing leaked ...
No leaks (block sum matches space maps exactly)
bp count: 409008
bp logical: 33416410112 avg: 81701
bp physical: 33416410112 avg: 81701 compression: 1.00
bp allocated: 41775882240 avg: 102139 compression: 0.80
SPA allocated: 41775882240 used: 3.35%
Blocks LSIZE PSIZE ASIZE avg comp %Total Type
17 176K 176K 220K 12.9K 1.00 0.00 deferred free
1 512 512 1K 1K 1.00 0.00 object directory
3 1.50K 1.50K 3.00K 1K 1.00 0.00 object array
1 16K 16K 20.0K 20.0K 1.00 0.00 packed nvlist
- - - - - - - packed nvlist size
1 16K 16K 20.0K 20.0K 1.00 0.00 bplist
- - - - - - - bplist header
- - - - - - - SPA space map header
25.1K 103M 103M 129M 5.15K 1.00 0.32 SPA space map
2 8K 8K 10.0K 5.00K 1.00 0.00 ZIL intent log
128K 1.99G 1.99G 2.49G 20.0K 1.00 6.41 DMU dnode
5 5.00K 5.00K 10.0K 2K 1.00 0.00 DMU objset
- - - - - - - DSL directory
5 2.50K 2.50K 5.00K 1K 1.00 0.00 DSL directory child map
4 2K 2K 4K 1K 1.00 0.00 DSL dataset snap map
7 514K 514K 643K 91.9K 1.00 0.00 DSL props
- - - - - - - DSL dataset
- - - - - - - ZFS znode
- - - - - - - ZFS ACL
246K 29.0G 29.0G 36.3G 151K 1.00 93.25 ZFS plain file
294 4.50M 4.50M 5.73M 20.0K 1.00 0.01 ZFS directory
4 2K 2K 4K 1K 1.00 0.00 ZFS master node
4 2K 2K 4K 1K 1.00 0.00 ZFS delete queue
- - - - - - - zvol object
- - - - - - - zvol prop
- - - - - - - other uint8[]
- - - - - - - other uint64[]
- - - - - - - other ZAP
399K 31.1G 31.1G 38.9G 100K 1.00 100.00 Total
On 3/8/06, Joe Little <jmlittle at gmail.com>
wrote:> Sadly, I have now JDS installed (no X, headless, and not X11
> accessible with the network topology). I just synced 15Gb of files
> from linux/xfs to B33 and ZFS. Waiting on the zdb results, but its now
> 19GB in size.
>
> On 3/8/06, Akhilesh Mritunjai <mritun+opensolaris at gmail.com>
wrote:
> > Joe,
> >
> > A utility called JDiskReport
(http://www.jgoodies.com/freeware/jdiskreport/) can help analyze the sizes of
files.
> >
> > This utility can output a number of diagnostics on file sizes and
numbers. Might be helpful in pinpointing the issue if its related to file sizes.
It can export the data as CSV (please see the FAQ on the site).
> >
> > - Akhilesh
> > This message posted from opensolaris.org
> > _______________________________________________
> > zfs-discuss mailing list
> > zfs-discuss at opensolaris.org
> > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
> >
>
Joe Little
2006-Mar-09 04:27 UTC
[zfs-discuss] ZFS filesystem with a lot of small files consumes more that 2x disk space
Ok. Here''s some real numbers.
As I said, I moved 15GB to ZFS, and its now 19GB.
distributions are:
0 files over 1GB
17 files 256MB-1GB
6 files 64MB-256MB
43 files 16MB-64MB
196 files 4MB-16MB
346 files 1MB-4MB
260 files 256KB-1MB
295 files 64KB-256KB
515 files 16-64K
2404 files 4-16K
2145 files 1-4K
7478 files 0-1K (links?)
zdb output:
Traversing all blocks to verify nothing leaked ...
No leaks (block sum matches space maps exactly)
bp count: 409008
bp logical: 33416410112 avg: 81701
bp physical: 33416410112 avg: 81701 compression: 1.00
bp allocated: 41775882240 avg: 102139 compression: 0.80
SPA allocated: 41775882240 used: 3.35%
Blocks LSIZE PSIZE ASIZE avg comp %Total Type
17 176K 176K 220K 12.9K 1.00 0.00 deferred free
1 512 512 1K 1K 1.00 0.00 object directory
3 1.50K 1.50K 3.00K 1K 1.00 0.00 object array
1 16K 16K 20.0K 20.0K 1.00 0.00 packed nvlist
- - - - - - - packed nvlist size
1 16K 16K 20.0K 20.0K 1.00 0.00 bplist
- - - - - - - bplist header
- - - - - - - SPA space map header
25.1K 103M 103M 129M 5.15K 1.00 0.32 SPA space map
2 8K 8K 10.0K 5.00K 1.00 0.00 ZIL intent log
128K 1.99G 1.99G 2.49G 20.0K 1.00 6.41 DMU dnode
5 5.00K 5.00K 10.0K 2K 1.00 0.00 DMU objset
- - - - - - - DSL directory
5 2.50K 2.50K 5.00K 1K 1.00 0.00 DSL directory child map
4 2K 2K 4K 1K 1.00 0.00 DSL dataset snap map
7 514K 514K 643K 91.9K 1.00 0.00 DSL props
- - - - - - - DSL dataset
- - - - - - - ZFS znode
- - - - - - - ZFS ACL
246K 29.0G 29.0G 36.3G 151K 1.00 93.25 ZFS plain file
294 4.50M 4.50M 5.73M 20.0K 1.00 0.01 ZFS directory
4 2K 2K 4K 1K 1.00 0.00 ZFS master node
4 2K 2K 4K 1K 1.00 0.00 ZFS delete queue
- - - - - - - zvol object
- - - - - - - zvol prop
- - - - - - - other uint8[]
- - - - - - - other uint64[]
- - - - - - - other ZAP
399K 31.1G 31.1G 38.9G 100K 1.00 100.00 Total
Solaris B33
On 3/7/06, Matthew Ahrens <ahrens at sun.com>
wrote:> On Tue, Mar 07, 2006 at 05:30:42PM -0800, Joe Little wrote:
> > In my various tests, I have seen large expansion as well, and
wasn''t
> > sure it was simply RAIDZ using up more disk space over time, as it
> > originally shows me full use of all drive capacity when originally
> > formatted. In the case of our primary mirror, which I tested hosting
> > on zfs/raidz/iscsi, my 1TB full of various linux distros, OS projects,
> > etc took 1.4TB on zfs.
>
> I''d be happy to try and diagnose this, if you can send the same
info
> (either to the list, or directly to me if you):
>
> 1. Most importantly: the output of ''zdb -bb
<poolname>''
> 2. What version of ZFS you are running?
> 3. What size files are you creating (approximately)?
>
> --matt
>
Matthew Ahrens
2006-Mar-09 08:48 UTC
[zfs-discuss] Re: ZFS filesystem with a lot of small files consumes more that 2x disk s
On Wed, Mar 08, 2006 at 06:29:19PM -0800, Robert Milkowski wrote:> Blocks LSIZE PSIZE ASIZE avg comp %Total Type > 947K 14.8G 14.8G 14.8G 16K 1.00 3.65 DMU dnode > 18.7M 156G 156G 156G 8.38K 1.00 38.62 ZFS plain file > 11.6M 234G 234G 234G 20.2K 1.00 57.72 ZFS directory > 31.2M 405G 405G 405G 13.0K 1.00 100.00 TotalOK, that''s awesome. You are definitely seeing 6391873 metadata compression should be turned back on I fixed this in build 36. Using build 36, all that directory and dnode data (61% of your total) will be compressed, and average compression ratios are better than 4x. So you should see that your overall space usage here goes from 405G to less than 218G, or around 16% more than UFS. Not awesome, but a lot more reasonable. Note, turning on the compression property (''zfs set compression=on'') does not affect compression of metadata. I would be curious to see what your usage looks like (''zdb -bb'') once you have have build 36. But it''s just curiosity, if it''s inconvenient for you then no worries. Thanks for playing with ZFS and reporting bugs! It''s always nice to see that bugs I''ve fixed actually help people. --matt
Matthew Ahrens
2006-Mar-09 09:28 UTC
[zfs-discuss] Re: ZFS filesystem with a lot of small files consumes more that 2x disk space
On Wed, Mar 08, 2006 at 08:14:37PM -0800, Joe Little wrote:> This took over an hour..Yeah, sorry about that... as this is a diagnostic tool we haven''t put much effort into improving its performance yet.> (again, 19GB usage of 15GB of XFS originally > stored files; some links/rpm/isos/etc)Interesting, it appears from the zdb output below that you should see 38.9G reported from ''zfs list <pool>'', ''zpool list'', etc. Perhaps there are some other filesystems in your pool? In any case the ratio is approximately the same as what we see from ''zdb -bb''.> Blocks LSIZE PSIZE ASIZE avg comp %Total Type > 128K 1.99G 1.99G 2.49G 20.0K 1.00 6.41 DMU dnode > 246K 29.0G 29.0G 36.3G 151K 1.00 93.25 ZFS plain file > 399K 31.1G 31.1G 38.9G 100K 1.00 100.00 TotalThis looks amazingly like 6288488 du reports misleading size on RAID-Z Is this on a 5-way raid-z? Your expansion ratio is exactly 1.250, the same as you should see if you were using such a raid-z. If this is the case, then the issue is not really that your files are consuming more space under ZFS, simply that we are reporting the space "differently" (I would say wrongly, hence the bug report). This bug is not targeted for Solaris 10 Update 2, which means that it probably won''t be fixed for several months (at least). If this is a problem, please let us know. I will add a call record to the bug for you. --matt
Robert Milkowski
2006-Mar-09 11:21 UTC
[zfs-discuss] Re: ZFS filesystem with a lot of small files consumes more that 2x disk s
Hello Matthew, Thursday, March 9, 2006, 9:48:46 AM, you wrote: MA> On Wed, Mar 08, 2006 at 06:29:19PM -0800, Robert Milkowski wrote:>> Blocks LSIZE PSIZE ASIZE avg comp %Total Type >> 947K 14.8G 14.8G 14.8G 16K 1.00 3.65 DMU dnode >> 18.7M 156G 156G 156G 8.38K 1.00 38.62 ZFS plain file >> 11.6M 234G 234G 234G 20.2K 1.00 57.72 ZFS directory >> 31.2M 405G 405G 405G 13.0K 1.00 100.00 TotalMA> OK, that''s awesome. You are definitely seeing MA> 6391873 metadata compression should be turned back on MA> I fixed this in build 36. Using build 36, all that directory and dnode MA> data (61% of your total) will be compressed, and average compression MA> ratios are better than 4x. So you should see that your overall space MA> usage here goes from 405G to less than 218G, or around 16% more than MA> UFS. Not awesome, but a lot more reasonable. Note, turning on the MA> compression property (''zfs set compression=on'') does not affect MA> compression of metadata. MA> I would be curious to see what your usage looks like (''zdb -bb'') once MA> you have have build 36. But it''s just curiosity, if it''s inconvenient MA> for you then no worries. You can bet I''ll make a test once I get my hands on b36. MA> Thanks for playing with ZFS and reporting bugs! It''s always nice to see MA> that bugs I''ve fixed actually help people. And it''s always nice to see that someone is listening :) Thank you for your help. Without this corrected at least one production would definitely not move to ZFS. -- Best regards, Robert mailto:rmilkowski at task.gda.pl http://milek.blogspot.com
Joe Little
2006-Mar-09 15:05 UTC
[zfs-discuss] Re: ZFS filesystem with a lot of small files consumes more that 2x disk space
Sorry, I initially got a failure doing a zdb to the underlying zfs filesystem.. I thought you were supposed to point it at a pool. (I did "zdb -bb poolname" and not "zdb -bb poolname/zfsname" This is RAIDZ, and again, this is what the original poster and I commented on about using a lot of extra storage in this config. I originally remarked that I thought it was just RAIDZ, since you seem to get "all" the disks up front when in fact you should generally lose a disk worth for parity. As for Solaris10 U2. I keep hearing about this. Is ZFS planned to make that release, and is there a ball park (which month) release time frame for that? On 3/9/06, Matthew Ahrens <ahrens at eng.sun.com> wrote:> On Wed, Mar 08, 2006 at 08:14:37PM -0800, Joe Little wrote: > > This took over an hour.. > > Yeah, sorry about that... as this is a diagnostic tool we haven''t put > much effort into improving its performance yet. > > > (again, 19GB usage of 15GB of XFS originally > > stored files; some links/rpm/isos/etc) > > Interesting, it appears from the zdb output below that you should see > 38.9G reported from ''zfs list <pool>'', ''zpool list'', etc. Perhaps there > are some other filesystems in your pool? In any case the ratio is > approximately the same as what we see from ''zdb -bb''. > > > Blocks LSIZE PSIZE ASIZE avg comp %Total Type > > 128K 1.99G 1.99G 2.49G 20.0K 1.00 6.41 DMU dnode > > 246K 29.0G 29.0G 36.3G 151K 1.00 93.25 ZFS plain file > > 399K 31.1G 31.1G 38.9G 100K 1.00 100.00 Total > > This looks amazingly like > 6288488 du reports misleading size on RAID-Z > Is this on a 5-way raid-z? Your expansion ratio is exactly 1.250, the > same as you should see if you were using such a raid-z. If this is the > case, then the issue is not really that your files are consuming more > space under ZFS, simply that we are reporting the space "differently" (I > would say wrongly, hence the bug report). > > This bug is not targeted for Solaris 10 Update 2, which means that it > probably won''t be fixed for several months (at least). If this is a > problem, please let us know. I will add a call record to the bug for > you. > > --matt >
George Wilson
2006-Mar-09 16:17 UTC
[zfs-discuss] Re: ZFS filesystem with a lot of small files consumes more that 2x disk space
Joe, Yes, ZFS is planned for Solaris 10u2. Look for it sometime in the June-July timeframe. Thanks, George Joe Little wrote:> Sorry, I initially got a failure doing a zdb to the underlying zfs > filesystem.. I thought you were supposed to point it at a pool. (I did > "zdb -bb poolname" and not "zdb -bb poolname/zfsname" > > This is RAIDZ, and again, this is what the original poster and I > commented on about using a lot of extra storage in this config. I > originally remarked that I thought it was just RAIDZ, since you seem > to get "all" the disks up front when in fact you should generally lose > a disk worth for parity. > > As for Solaris10 U2. I keep hearing about this. Is ZFS planned to make > that release, and is there a ball park (which month) release time > frame for that? > > On 3/9/06, Matthew Ahrens <ahrens at eng.sun.com> wrote: >> On Wed, Mar 08, 2006 at 08:14:37PM -0800, Joe Little wrote: >>> This took over an hour.. >> Yeah, sorry about that... as this is a diagnostic tool we haven''t put >> much effort into improving its performance yet. >> >>> (again, 19GB usage of 15GB of XFS originally >>> stored files; some links/rpm/isos/etc) >> Interesting, it appears from the zdb output below that you should see >> 38.9G reported from ''zfs list <pool>'', ''zpool list'', etc. Perhaps there >> are some other filesystems in your pool? In any case the ratio is >> approximately the same as what we see from ''zdb -bb''. >> >>> Blocks LSIZE PSIZE ASIZE avg comp %Total Type >>> 128K 1.99G 1.99G 2.49G 20.0K 1.00 6.41 DMU dnode >>> 246K 29.0G 29.0G 36.3G 151K 1.00 93.25 ZFS plain file >>> 399K 31.1G 31.1G 38.9G 100K 1.00 100.00 Total >> This looks amazingly like >> 6288488 du reports misleading size on RAID-Z >> Is this on a 5-way raid-z? Your expansion ratio is exactly 1.250, the >> same as you should see if you were using such a raid-z. If this is the >> case, then the issue is not really that your files are consuming more >> space under ZFS, simply that we are reporting the space "differently" (I >> would say wrongly, hence the bug report). >> >> This bug is not targeted for Solaris 10 Update 2, which means that it >> probably won''t be fixed for several months (at least). If this is a >> problem, please let us know. I will add a call record to the bug for >> you. >> >> --matt >> > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Matthew Ahrens
2006-Mar-09 17:44 UTC
[zfs-discuss] Re: ZFS filesystem with a lot of small files consumes more that 2x disk space
On Thu, Mar 09, 2006 at 07:05:23AM -0800, Joe Little wrote:> Sorry, I initially got a failure doing a zdb to the underlying zfs > filesystem.. I thought you were supposed to point it at a pool. (I did > "zdb -bb poolname" and not "zdb -bb poolname/zfsname"You actually can do zdb on a filesystem, but it doesn''t gather the block statistics (and there is still the same caveat about not being able to run while the pool is changing).> As for Solaris10 U2. I keep hearing about this. Is ZFS planned to make > that release, and is there a ball park (which month) release time > frame for that?Yep, ZFS will ship with s10u2 this summer. You can find the answer to this question and more in the FAQ: http://opensolaris.org/os/community/zfs/faq/#whenavailable --matt
Joe Little
2006-Mar-09 23:59 UTC
[zfs-discuss] Re: ZFS filesystem with a lot of small files consumes more that 2x disk space
BTW, I setup a separate non-RAIDZ pool and copied to there, and I get a 1-to-1 ratio of disk usage for that, so your explanation regarding RAIDZ seems to hold. On 3/9/06, Matthew Ahrens <ahrens at sun.com> wrote:> On Wed, Mar 08, 2006 at 08:14:37PM -0800, Joe Little wrote: > > This took over an hour.. > > Yeah, sorry about that... as this is a diagnostic tool we haven''t put > much effort into improving its performance yet. > > > (again, 19GB usage of 15GB of XFS originally > > stored files; some links/rpm/isos/etc) > > Interesting, it appears from the zdb output below that you should see > 38.9G reported from ''zfs list <pool>'', ''zpool list'', etc. Perhaps there > are some other filesystems in your pool? In any case the ratio is > approximately the same as what we see from ''zdb -bb''. > > > Blocks LSIZE PSIZE ASIZE avg comp %Total Type > > 128K 1.99G 1.99G 2.49G 20.0K 1.00 6.41 DMU dnode > > 246K 29.0G 29.0G 36.3G 151K 1.00 93.25 ZFS plain file > > 399K 31.1G 31.1G 38.9G 100K 1.00 100.00 Total > > This looks amazingly like > 6288488 du reports misleading size on RAID-Z > Is this on a 5-way raid-z? Your expansion ratio is exactly 1.250, the > same as you should see if you were using such a raid-z. If this is the > case, then the issue is not really that your files are consuming more > space under ZFS, simply that we are reporting the space "differently" (I > would say wrongly, hence the bug report). > > This bug is not targeted for Solaris 10 Update 2, which means that it > probably won''t be fixed for several months (at least). If this is a > problem, please let us know. I will add a call record to the bug for > you. > > --matt > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >
grant beattie
2006-Mar-14 05:47 UTC
[zfs-discuss] Re: ZFS filesystem with a lot of small files consumes more that 2x disk s
On Thu, Mar 09, 2006 at 12:21:12PM +0100, Robert Milkowski wrote:> MA> I would be curious to see what your usage looks like (''zdb -bb'') once > MA> you have have build 36. But it''s just curiosity, if it''s inconvenient > MA> for you then no worries. > > > You can bet I''ll make a test once I get my hands on b36.I''ve just bfu''d to 20060307 which includes a stack of ZFS fixes, including 6391873 metadata compression should be turned back on. top 3 space consumers in ''zdb -bb'' are: Blocks LSIZE PSIZE ASIZE avg comp %Total Type 17.7K 284M 209M 209M 11.8K 1.36 5.18 DMU dnode 306K 3.92G 3.69G 3.69G 12.3K 1.06 93.76 ZFS plain file 38.7K 156M 38.0M 38.0M 1005 4.11 0.94 ZFS directory this is ~220000 maildir email files/directories, some metadata is left over from pre-bfu''ing which hasn''t been compressed, but this is a definite improvement. thanks guys! :) grant.
Robert Milkowski
2006-Apr-05 08:51 UTC
[zfs-discuss] Re: ZFS filesystem with a lot of small files consumes more that 2x disk s
Hello Matthew, Thursday, March 9, 2006, 10:48:46 AM, you wrote: MA> On Wed, Mar 08, 2006 at 06:29:19PM -0800, Robert Milkowski wrote:>> Blocks LSIZE PSIZE ASIZE avg comp %Total Type >> 947K 14.8G 14.8G 14.8G 16K 1.00 3.65 DMU dnode >> 18.7M 156G 156G 156G 8.38K 1.00 38.62 ZFS plain file >> 11.6M 234G 234G 234G 20.2K 1.00 57.72 ZFS directory >> 31.2M 405G 405G 405G 13.0K 1.00 100.00 TotalMA> I would be curious to see what your usage looks like (''zdb -bb'') once MA> you have have build 36. But it''s just curiosity, if it''s inconvenient MA> for you then no worries. bash-3.00# df -h | egrep "p-16-32|p-54-70" p-16-32/d606 405G 404G 276M 100% /opt/d606 p-16-32 405G 98K 276M 1% /p-16-32 p-54-70 405G 169G 236G 42% /p-54-70 bash-3.00# p-16-32 is the old one, and p-54-70 is zfs filesystem created on build based on b36. Then data were copied from p-16-32/d606 to p-54-70 Well, that''s an improvement! bash-3.00# zdb -bb p-16-32 Traversing all blocks to verify nothing leaked ... No leaks (block sum matches space maps exactly) bp count: 32659636 bp logical: 434319826432 avg: 13298 bp physical: 434319109632 avg: 13298 compression: 1.00 bp allocated: 434375188480 avg: 13300 compression: 1.00 SPA allocated: 434375188480 used: 99.15% Blocks LSIZE PSIZE ASIZE avg comp %Total Type 43 520K 252K 252K 5.85K 2.07 0.00 deferred free 1 512 512 512 512 1.00 0.00 object directory 18 9.00K 9.00K 9.00K 512 1.00 0.00 object array 1 16K 2K 2K 2K 8.00 0.00 packed nvlist - - - - - - - packed nvlist size 1 16K 16K 16K 16K 1.00 0.00 bplist - - - - - - - bplist header - - - - - - - SPA space map header 9.33K 46.9M 46.8M 46.8M 5.01K 1.00 0.01 SPA space map - - - - - - - ZIL intent log 947K 14.8G 14.8G 14.8G 16.0K 1.00 3.66 DMU dnode 3 3.00K 2.50K 2.50K 853 1.20 0.00 DMU objset - - - - - - - DSL directory 3 1.50K 1.50K 1.50K 512 1.00 0.00 DSL directory child map 2 1K 1K 1K 512 1.00 0.00 DSL dataset snap map 4 257K 23.5K 23.5K 5.88K 10.94 0.00 DSL props - - - - - - - DSL dataset - - - - - - - ZFS znode - - - - - - - ZFS ACL 18.6M 156G 156G 156G 8.38K 1.00 38.61 ZFS plain file 11.6M 233G 233G 234G 20.2K 1.00 57.72 ZFS directory 2 1K 1K 1K 512 1.00 0.00 ZFS master node 2 1K 1K 1K 512 1.00 0.00 ZFS delete queue - - - - - - - zvol object - - - - - - - zvol prop - - - - - - - other uint8[] - - - - - - - other uint64[] - - - - - - - other ZAP - - - - - - - persistent error log 31.1M 404G 404G 405G 13.0K 1.00 100.00 Total bash-3.00# bash-3.00# zdb -bb p-54-70 Traversing all blocks to verify nothing leaked ... No leaks (block sum matches space maps exactly) bp count: 32899206 bp logical: 235948411392 avg: 7171 bp physical: 181050077184 avg: 5503 compression: 1.30 bp allocated: 181050077184 avg: 5503 compression: 1.30 SPA allocated: 181050077184 used: 41.33% Blocks LSIZE PSIZE ASIZE avg comp %Total Type 107 1.16M 200K 200K 1.87K 5.92 0.00 deferred free 1 512 512 512 512 1.00 0.00 object directory 18 9.00K 9.00K 9.00K 512 1.00 0.00 object array 1 16K 2K 2K 2K 8.00 0.00 packed nvlist - - - - - - - packed nvlist size 1 16K 16K 16K 16K 1.00 0.00 bplist - - - - - - - bplist header - - - - - - - SPA space map header 9.9K 49.1M 25.7M 25.7M 2.59K 1.91 0.01 SPA space map - - - - - - - ZIL intent log 1.09M 17.4G 4.01G 4.01G 3.69K 4.34 2.38 DMU dnode 2 2K 1K 1K 512 2.00 0.00 DMU objset - - - - - - - DSL directory 2 1K 1K 1K 512 1.00 0.00 DSL directory child map 1 512 512 512 512 1.00 0.00 DSL dataset snap map 2 1K 1K 1K 512 1.00 0.00 DSL props - - - - - - - DSL dataset - - - - - - - ZFS znode - - - - - - - ZFS ACL 18.6M 156G 155G 155G 8.33K 1.01 92.12 ZFS plain file 11.6M 46.1G 9.25G 9.25G 814 4.99 5.49 ZFS directory 1 512 512 512 512 1.00 0.00 ZFS master node 1 512 512 512 512 1.00 0.00 ZFS delete queue - - - - - - - zvol object - - - - - - - zvol prop - - - - - - - other uint8[] - - - - - - - other uint64[] - - - - - - - other ZAP - - - - - - - persistent error log 31.4M 220G 169G 169G 5.37K 1.30 100.00 Total bash-3.00# -- Best regards, Robert mailto:rmilkowski at task.gda.pl http://milek.blogspot.com
Robert Milkowski
2006-Apr-06 08:03 UTC
[zfs-discuss] Re: ZFS filesystem with a lot of small files consumes more that 2x disk s
Hi.
And with new zfs bits (based on b36) in the same case ZFS filesystem
actually consumes (it''s mirror so there''s no raidz issue with
reported
free space) a little bit less space than UFS! This is just awesome!
ps. of course without data compression on zfs
--
Best regards,
Robert mailto:rmilkowski at task.gda.pl
http://milek.blogspot.com