How space efficient are packed small files? For instance, if I have a 10,000 16 byte files with 16 byte file names, how much disk space will they use? Are there any caveats or gotchas around using btrfs to store millions of small (size<64bytes) files? -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, Jan 31, 2012 at 08:21:30AM -0800, Hans Stimer wrote:> How space efficient are packed small files? For instance, if I have a > 10,000 16 byte files with 16 byte file names, how much disk space will > they use? Are there any caveats or gotchas around using btrfs to store > millions of small (size<64bytes) files?For small files, the file content may be kept inline in the btree blocks of the extent tree. The data is packed in alongside the associated keys and inline extent metadata, and the Btree implementation can be somewhat variable in its usage as well, so it''s hard to put fixed values on these things. There''s metadata overhead on storing files as well -- each file will be represented by a number of keys (17 bytes each) and metadata structures to go with them. In the FS tree: DIR_ITEM 13 DIR_INDEX 13 INODE_ITEM 162 INODE_REF 10 + size(name) EXTENT_DATA 52 Keys 85 --- 335 + size(name) And in the extent tree: EXTENT_ITEM 24 + 17 for key extent_inline_ref 9 extent_data_ref 28 --- 78 So you''re looking at a minimum of 413 bytes of metadata overhead for an inline file, plus the length of the filename. Also note that the file is stored in the metadata, so by default it''s stored with DUP or RAID-1 replication (even if data is set to be "single"). This means that you''ll actually use up twice this amount of space on the disks, unless you create the FS with metadata set to "single". I don''t know how these figures compare with other filesystems. My entirely uneducated guess is that they''re probably comparable, with the exception of the DUP effect. Hugo. -- === Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk == PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk --- I know of three kinds: hot, --- cool, and what-time-does-the-tune-start?
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 1/31/2012 11:46 AM, Hugo Mills wrote:> So you''re looking at a minimum of 413 bytes of metadata overhead > for an inline file, plus the length of the filename. > > Also note that the file is stored in the metadata, so by default > it''s stored with DUP or RAID-1 replication (even if data is set to > be "single"). This means that you''ll actually use up twice this > amount of space on the disks, unless you create the FS with > metadata set to "single". > > I don''t know how these figures compare with other filesystems. My > entirely uneducated guess is that they''re probably comparable, > with the exception of the DUP effect.On ext4 you are looking at 256 bytes for the inode, name length + a few bytes for the directory entry, another few bytes for the hashed directory entry, and a whole 4k block to hold the data, so ~4300 bytes ( + name length ) of overhead to store a 64 byte file. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJPNWnBAAoJEJrBOlT6nu75lVkIAIO2mjYeVK5BbMNfw5HJ7jZO WfIBv5xR8V06e0VLgv4FQqPlWcm+ZQHorYDM7h15q4cIgoZ3x0P3n3bSCurFRLfF lSRjn/fsX1Y9isPEB6/monPm+08U6qh7jXGldEMOLKaA7VG/QOVR01k3W2a3FkJ4 kWBjEbK/xE013WaQnfR26PydRT8ILRzGUE4uEKGsdV39JkcEorQ1lDg+XWz5Hvy7 VmelT21272PssIUbRub1QkZXj6p0SUu1zeU1IwOdt6X1uXFcWqFbBFRGJk4f2+ZM 5MquuVC+YrzfDIBnS0ZBZ4UqNmxYuPCSzTLlpPiJJiY/AwR7916H/CoF5k38k/M=8YmX -----END PGP SIGNATURE----- -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html