On 2/8/2005 17:09, Jeffrey W. Baker wrote:> I''ve gone back to testing with the lustre found in SuSE on my toy > installation. What I''ve just noticed is the MDS burns 4K per file, > meaning for decreasing average file size the MDS will need storage > approaching that of all the OSTs put together. Not a happy thought! > Can the MDS use a filesystem with 1K block size?I think this is an artifact of the older version of Lustre that you''re using. When Lustre is formatted with a more modern version, it should create inodes that are between 256 and 4096 bytes (depending on your default stripe count). These large inodes provide enough space for the LOV extended attributes to fit right in the inode, instead of burning an extra block (and incurring an extra seek) for each inode. To find out, run dumpe2fs on your MDS disk partition, as in: [root@b3 root]# dumpe2fs -h /dev/hda1 | grep "Inode size" dumpe2fs 1.35 (28-Feb-2004) Inode size: 128 If you already have large inodes, but still think that it''s taking up 4k per file, let us know. -Phil
On 2005-02-09T12:49:09, smee <snotmee@gmail.com> wrote:> What version of Lustre does the latest Suse 9.2 come with? And is it > only Suse Pro or Enterprise that has Lustre bundled in?Lustre is only bundled with the SUSE Linux Enterprise Server 9. SP1 comes with a Lustre 1.2.x based release. Sincerely, Lars Marowsky-Brée <lmb@suse.de> -- High Availability & Clustering SUSE Labs, Research and Development SUSE LINUX Products GmbH - A Novell Business
On Tue, 2005-02-08 at 19:10 -0500, Phil Schwan wrote:> On 2/8/2005 17:09, Jeffrey W. Baker wrote: > > > I''ve gone back to testing with the lustre found in SuSE on my toy > > installation. What I''ve just noticed is the MDS burns 4K per file, > > meaning for decreasing average file size the MDS will need storage > > approaching that of all the OSTs put together. Not a happy thought! > > Can the MDS use a filesystem with 1K block size? > > I think this is an artifact of the older version of Lustre that you''re > using. > > When Lustre is formatted with a more modern version, it should create inodes > that are between 256 and 4096 bytes (depending on your default stripe > count). These large inodes provide enough space for the LOV extended > attributes to fit right in the inode, instead of burning an extra block (and > incurring an extra seek) for each inode. > > To find out, run dumpe2fs on your MDS disk partition, as in: > > [root@b3 root]# dumpe2fs -h /dev/hda1 | grep "Inode size" > dumpe2fs 1.35 (28-Feb-2004) > Inode size: 128 > > If you already have large inodes, but still think that it''s taking up 4k per > file, let us know.Ok, I was just looking at this empirically. Since you asked, # dumpe2fs -h /dev/test_vg/test_lv dumpe2fs 1.36 (05-Feb-2005) Inode size: 256 That appears to be twice the normal size, so I guess they are large inodes. What I had been watching was how much space was used on the MDS per (empty) file created. In the beginning it looked this this was running near 4K, but now it looks more like 2K. With a bit over 200k files there''s 405MiB used, which works out to just under 2000 bytes per file. What I had seen earlier which lead me to the 4K conclusion was directories. I created 50,000 directories in the same filesystem as above and now 603MiB are used, meaning each directory used 4K. But hey, at least there''s no 32,000 directory limit. Regards, jwb - who just took delivery of rather a lot of disks.
What version of Lustre does the latest Suse 9.2 come with? And is it only Suse Pro or Enterprise that has Lustre bundled in? Cheers. On Tue, 08 Feb 2005 16:32:07 -0800, Jeffrey W. Baker <jwbaker@acm.org> wrote:> On Tue, 2005-02-08 at 19:10 -0500, Phil Schwan wrote: > > On 2/8/2005 17:09, Jeffrey W. Baker wrote: > > > > > I''ve gone back to testing with the lustre found in SuSE on my toy > > > installation. What I''ve just noticed is the MDS burns 4K per file, > > > meaning for decreasing average file size the MDS will need storage > > > approaching that of all the OSTs put together. Not a happy thought! > > > Can the MDS use a filesystem with 1K block size? > > > > I think this is an artifact of the older version of Lustre that you''re > > using. > > > > When Lustre is formatted with a more modern version, it should create inodes > > that are between 256 and 4096 bytes (depending on your default stripe > > count). These large inodes provide enough space for the LOV extended > > attributes to fit right in the inode, instead of burning an extra block (and > > incurring an extra seek) for each inode. > > > > To find out, run dumpe2fs on your MDS disk partition, as in: > > > > [root@b3 root]# dumpe2fs -h /dev/hda1 | grep "Inode size" > > dumpe2fs 1.35 (28-Feb-2004) > > Inode size: 128 > > > > If you already have large inodes, but still think that it''s taking up 4k per > > file, let us know. > > Ok, I was just looking at this empirically. Since you asked, > > # dumpe2fs -h /dev/test_vg/test_lv > dumpe2fs 1.36 (05-Feb-2005) > Inode size: 256 > > That appears to be twice the normal size, so I guess they are large > inodes. What I had been watching was how much space was used on the MDS > per (empty) file created. In the beginning it looked this this was > running near 4K, but now it looks more like 2K. With a bit over 200k > files there''s 405MiB used, which works out to just under 2000 bytes per > file. > > What I had seen earlier which lead me to the 4K conclusion was > directories. I created 50,000 directories in the same filesystem as > above and now 603MiB are used, meaning each directory used 4K. But hey, > at least there''s no 32,000 directory limit. > > Regards, > jwb - who just took delivery of rather a lot of disks. > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss@lists.clusterfs.com > https://lists.clusterfs.com/mailman/listinfo/lustre-discuss >
I''ve gone back to testing with the lustre found in SuSE on my toy installation. What I''ve just noticed is the MDS burns 4K per file, meaning for decreasing average file size the MDS will need storage approaching that of all the OSTs put together. Not a happy thought! Can the MDS use a filesystem with 1K block size? -jwb