My departments Lustre''s lfs df -i looks like this UUID Inodes IUsed IFree IUse% Mounted on xfs001-MDT0000_UUID 82565322 19073586 63491736 23% /xfs/engine1/xfs001[MDT:0] xfs001-OST0000_UUID 41943040 5076445 36866595 12% /xfs/engine1/xfs001[OST:0] xfs001-OST0001_UUID 41943040 3735715 38207325 8% /xfs/engine1/xfs001[OST:1] xfs001-OST0002_UUID 41943040 4784447 37158593 11% /xfs/engine1/xfs001[OST:2] xfs001-OST0003_UUID 41943040 4556627 37386413 10% /xfs/engine1/xfs001[OST:3] filesystem summary: 82565322 19073586 63491736 23% /xfs/engine1/xfs001 Is it safe to say, " My MDT has "19073586" 19 million inodes free? or should I consider. However, when I check my inode count on my mds using tune2fs it reports this. Inode count: 524288000 Free inodes: 505214414 Am I missing something? TIA
On Sun, 2008-08-17 at 15:50 -0700, Mag Gam wrote:> My departments Lustre''s lfs df -i looks like this > > UUID Inodes IUsed IFree IUse% Mounted on > xfs001-MDT0000_UUID 82565322 19073586 63491736 23% > /xfs/engine1/xfs001[MDT:0] > xfs001-OST0000_UUID 41943040 5076445 36866595 12% > /xfs/engine1/xfs001[OST:0] > xfs001-OST0001_UUID 41943040 3735715 38207325 8% > /xfs/engine1/xfs001[OST:1] > xfs001-OST0002_UUID 41943040 4784447 37158593 11% > /xfs/engine1/xfs001[OST:2] > xfs001-OST0003_UUID 41943040 4556627 37386413 10% > /xfs/engine1/xfs001[OST:3] > > filesystem summary: 82565322 19073586 63491736 23% /xfs/engine1/xfs001 > > > Is it safe to say, " > My MDT has "19073586" 19 million inodes free? or should I consider. > > However, when I check my inode count on my mds using tune2fs it reports this. > Inode count: 524288000 > Free inodes: 505214414 > > Am I missing something?Yes. lfs df does not simply report the free inodes on the MDT. It''s more complicated than that. This has been discussed either here or in a bugzilla bug (don''t recall which). I don''t recall the bug or when the lustre-discussion was, offhand, but a search should not be terribly difficult to find the answer. b. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20080819/ddcc00d7/attachment-0001.bin
Brian: Yes, I have looked thru the lists but I could not really get this question answered. tune2fs is a ext2/3 setting, and I though each file I create on a Lustre filesystem an inode gets created on the MDS and the OST as an object. But, I don''t understand how there is that much of a space discrepancy. Also, how does striping account for in inodes? Lets say I stipe a file on 4 OSTs, I suppose 5 (1 for MDS, 4 for OSTs) inodes are occupied, correct? Sorry for a newbie question. TIA On Tue, Aug 19, 2008 at 4:30 PM, Brian J. Murrell <Brian.Murrell at sun.com> wrote:> On Sun, 2008-08-17 at 15:50 -0700, Mag Gam wrote: >> My departments Lustre''s lfs df -i looks like this >> >> UUID Inodes IUsed IFree IUse% Mounted on >> xfs001-MDT0000_UUID 82565322 19073586 63491736 23% >> /xfs/engine1/xfs001[MDT:0] >> xfs001-OST0000_UUID 41943040 5076445 36866595 12% >> /xfs/engine1/xfs001[OST:0] >> xfs001-OST0001_UUID 41943040 3735715 38207325 8% >> /xfs/engine1/xfs001[OST:1] >> xfs001-OST0002_UUID 41943040 4784447 37158593 11% >> /xfs/engine1/xfs001[OST:2] >> xfs001-OST0003_UUID 41943040 4556627 37386413 10% >> /xfs/engine1/xfs001[OST:3] >> >> filesystem summary: 82565322 19073586 63491736 23% /xfs/engine1/xfs001 >> >> >> Is it safe to say, " >> My MDT has "19073586" 19 million inodes free? or should I consider. >> >> However, when I check my inode count on my mds using tune2fs it reports this. >> Inode count: 524288000 >> Free inodes: 505214414 >> >> Am I missing something? > > Yes. lfs df does not simply report the free inodes on the MDT. It''s > more complicated than that. This has been discussed either here or in a > bugzilla bug (don''t recall which). I don''t recall the bug or when the > lustre-discussion was, offhand, but a search should not be terribly > difficult to find the answer. > > b. > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss > >
On Aug 19, 2008 20:27 -0400, Mag Gam wrote:> Yes, I have looked thru the lists but I could not really get this > question answered. tune2fs is a ext2/3 setting, and I though each file > I create on a Lustre filesystem an inode gets created on the MDS and > the OST as an object. > > But, I don''t understand how there is that much of a space discrepancy.The reason for this discrepancy is because of "safety". It is possible that the striping is too large to fit into the inode, so it needs an external block. If that is needed for every remaining inode on the MDS then the limit on the number of files created is equal to the number of free blocks on the MDS. In order to avoid over-stating the number of possible files that can be created (sometimes important to e.g. RPM or regression tests), "df" and "lfs df" will return min(free inodes, free blocks) for the inode count. If (as usually happens) inodes are created that do not need external blocks to store the striping information then more inodes "appear" (i.e. as "IUsed" increases so does "Inodes" and "IFree" stays constant).> Also, how does striping account for in inodes? Lets say I stipe a file > on 4 OSTs, I suppose 5 (1 for MDS, 4 for OSTs) inodes are occupied, > correct?Yes, though each of the inodes are in different filesystems.> On Tue, Aug 19, 2008 at 4:30 PM, Brian J. Murrell <Brian.Murrell at sun.com> wrote: > > On Sun, 2008-08-17 at 15:50 -0700, Mag Gam wrote: > >> My departments Lustre''s lfs df -i looks like this > >> > >> UUID Inodes IUsed IFree IUse% Mounted on > >> xfs001-MDT0000_UUID 82565322 19073586 63491736 23% > >> /xfs/engine1/xfs001[MDT:0] > >> xfs001-OST0000_UUID 41943040 5076445 36866595 12% > >> /xfs/engine1/xfs001[OST:0] > >> xfs001-OST0001_UUID 41943040 3735715 38207325 8% > >> /xfs/engine1/xfs001[OST:1] > >> xfs001-OST0002_UUID 41943040 4784447 37158593 11% > >> /xfs/engine1/xfs001[OST:2] > >> xfs001-OST0003_UUID 41943040 4556627 37386413 10% > >> /xfs/engine1/xfs001[OST:3] > >> > >> filesystem summary: 82565322 19073586 63491736 23% /xfs/engine1/xfs001 > >> > >> > >> Is it safe to say, " > >> My MDT has "19073586" 19 million inodes free? or should I consider. > >> > >> However, when I check my inode count on my mds using tune2fs it reports this. > >> Inode count: 524288000 > >> Free inodes: 505214414 > >> > >> Am I missing something? > > > > Yes. lfs df does not simply report the free inodes on the MDT. It''s > > more complicated than that. This has been discussed either here or in a > > bugzilla bug (don''t recall which). I don''t recall the bug or when the > > lustre-discussion was, offhand, but a search should not be terribly > > difficult to find the answer. > > > > b. > > > > > > _______________________________________________ > > Lustre-discuss mailing list > > Lustre-discuss at lists.lustre.org > > http://lists.lustre.org/mailman/listinfo/lustre-discuss > > > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discussCheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.
Thanks Andreas. Is this documented anywhere? We are facing inode shortage problems on our filesystem and I would like to show this to my professors. The reason why this is occurring is.. I checked the operations manual and Lustre "inter galactic" file system but no luck of this issue. TIA On Wed, Aug 20, 2008 at 1:55 AM, Andreas Dilger <adilger at sun.com> wrote:> On Aug 19, 2008 20:27 -0400, Mag Gam wrote: >> Yes, I have looked thru the lists but I could not really get this >> question answered. tune2fs is a ext2/3 setting, and I though each file >> I create on a Lustre filesystem an inode gets created on the MDS and >> the OST as an object. >> >> But, I don''t understand how there is that much of a space discrepancy. > > The reason for this discrepancy is because of "safety". It is possible > that the striping is too large to fit into the inode, so it needs an > external block. If that is needed for every remaining inode on the MDS > then the limit on the number of files created is equal to the number of > free blocks on the MDS. In order to avoid over-stating the number of > possible files that can be created (sometimes important to e.g. RPM > or regression tests), "df" and "lfs df" will return min(free inodes, > free blocks) for the inode count. > > If (as usually happens) inodes are created that do not need external blocks > to store the striping information then more inodes "appear" (i.e. as > "IUsed" increases so does "Inodes" and "IFree" stays constant). > >> Also, how does striping account for in inodes? Lets say I stipe a file >> on 4 OSTs, I suppose 5 (1 for MDS, 4 for OSTs) inodes are occupied, >> correct? > > Yes, though each of the inodes are in different filesystems. > >> On Tue, Aug 19, 2008 at 4:30 PM, Brian J. Murrell <Brian.Murrell at sun.com> wrote: >> > On Sun, 2008-08-17 at 15:50 -0700, Mag Gam wrote: >> >> My departments Lustre''s lfs df -i looks like this >> >> >> >> UUID Inodes IUsed IFree IUse% Mounted on >> >> xfs001-MDT0000_UUID 82565322 19073586 63491736 23% >> >> /xfs/engine1/xfs001[MDT:0] >> >> xfs001-OST0000_UUID 41943040 5076445 36866595 12% >> >> /xfs/engine1/xfs001[OST:0] >> >> xfs001-OST0001_UUID 41943040 3735715 38207325 8% >> >> /xfs/engine1/xfs001[OST:1] >> >> xfs001-OST0002_UUID 41943040 4784447 37158593 11% >> >> /xfs/engine1/xfs001[OST:2] >> >> xfs001-OST0003_UUID 41943040 4556627 37386413 10% >> >> /xfs/engine1/xfs001[OST:3] >> >> >> >> filesystem summary: 82565322 19073586 63491736 23% /xfs/engine1/xfs001 >> >> >> >> >> >> Is it safe to say, " >> >> My MDT has "19073586" 19 million inodes free? or should I consider. >> >> >> >> However, when I check my inode count on my mds using tune2fs it reports this. >> >> Inode count: 524288000 >> >> Free inodes: 505214414 >> >> >> >> Am I missing something? >> > >> > Yes. lfs df does not simply report the free inodes on the MDT. It''s >> > more complicated than that. This has been discussed either here or in a >> > bugzilla bug (don''t recall which). I don''t recall the bug or when the >> > lustre-discussion was, offhand, but a search should not be terribly >> > difficult to find the answer. >> > >> > b. >> > >> > >> > _______________________________________________ >> > Lustre-discuss mailing list >> > Lustre-discuss at lists.lustre.org >> > http://lists.lustre.org/mailman/listinfo/lustre-discuss >> > >> > >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss > > Cheers, Andreas > -- > Andreas Dilger > Sr. Staff Engineer, Lustre Group > Sun Microsystems of Canada, Inc. > >
On Aug 20, 2008 06:45 -0400, Mag Gam wrote:> Is this documented anywhere? We are facing inode shortage problems on > our filesystem and I would like to show this to my professors. The > reason why this is occurring is..Well, according to the "lfs df -i" output below there are lots of inodes free. Depending on the striping you should be able to create at least 63M more files in the filesystem, possibly as many as 144M if you use only a single stripe per file at which point you run out of inodes for the OST objects.> I checked the operations manual and Lustre "inter galactic" file > system but no luck of this issue.I thought it was in the "Knowledge Base" or "FAQ" section, maybe not. Our trusty Documentation team should be able to add this if it isn''t there already.> On Wed, Aug 20, 2008 at 1:55 AM, Andreas Dilger <adilger at sun.com> wrote: > > On Aug 19, 2008 20:27 -0400, Mag Gam wrote: > >> Yes, I have looked thru the lists but I could not really get this > >> question answered. tune2fs is a ext2/3 setting, and I though each file > >> I create on a Lustre filesystem an inode gets created on the MDS and > >> the OST as an object. > >> > >> But, I don''t understand how there is that much of a space discrepancy. > > > > The reason for this discrepancy is because of "safety". It is possible > > that the striping is too large to fit into the inode, so it needs an > > external block. If that is needed for every remaining inode on the MDS > > then the limit on the number of files created is equal to the number of > > free blocks on the MDS. In order to avoid over-stating the number of > > possible files that can be created (sometimes important to e.g. RPM > > or regression tests), "df" and "lfs df" will return min(free inodes, > > free blocks) for the inode count. > > > > If (as usually happens) inodes are created that do not need external blocks > > to store the striping information then more inodes "appear" (i.e. as > > "IUsed" increases so does "Inodes" and "IFree" stays constant). > > > >> Also, how does striping account for in inodes? Lets say I stipe a file > >> on 4 OSTs, I suppose 5 (1 for MDS, 4 for OSTs) inodes are occupied, > >> correct? > > > > Yes, though each of the inodes are in different filesystems. > > > >> On Tue, Aug 19, 2008 at 4:30 PM, Brian J. Murrell <Brian.Murrell at sun.com> wrote: > >> > On Sun, 2008-08-17 at 15:50 -0700, Mag Gam wrote: > >> >> My departments Lustre''s lfs df -i looks like this > >> >> > >> >> UUID Inodes IUsed IFree IUse% Mounted on > >> >> xfs001-MDT0000_UUID 82565322 19073586 63491736 23% > >> >> /xfs/engine1/xfs001[MDT:0] > >> >> xfs001-OST0000_UUID 41943040 5076445 36866595 12% > >> >> /xfs/engine1/xfs001[OST:0] > >> >> xfs001-OST0001_UUID 41943040 3735715 38207325 8% > >> >> /xfs/engine1/xfs001[OST:1] > >> >> xfs001-OST0002_UUID 41943040 4784447 37158593 11% > >> >> /xfs/engine1/xfs001[OST:2] > >> >> xfs001-OST0003_UUID 41943040 4556627 37386413 10% > >> >> /xfs/engine1/xfs001[OST:3] > >> >> > >> >> filesystem summary: 82565322 19073586 63491736 23% /xfs/engine1/xfs001 > >> >> > >> >> > >> >> Is it safe to say, " > >> >> My MDT has "19073586" 19 million inodes free? or should I consider. > >> >> > >> >> However, when I check my inode count on my mds using tune2fs it reports this. > >> >> Inode count: 524288000 > >> >> Free inodes: 505214414 > >> >> > >> >> Am I missing something? > >> > > >> > Yes. lfs df does not simply report the free inodes on the MDT. It''s > >> > more complicated than that. This has been discussed either here or in a > >> > bugzilla bug (don''t recall which). I don''t recall the bug or when the > >> > lustre-discussion was, offhand, but a search should not be terribly > >> > difficult to find the answer. > >> > > >> > b. > >> > > >> > > >> > _______________________________________________ > >> > Lustre-discuss mailing list > >> > Lustre-discuss at lists.lustre.org > >> > http://lists.lustre.org/mailman/listinfo/lustre-discuss > >> > > >> > > >> _______________________________________________ > >> Lustre-discuss mailing list > >> Lustre-discuss at lists.lustre.org > >> http://lists.lustre.org/mailman/listinfo/lustre-discuss > > > > Cheers, Andreas > > -- > > Andreas Dilger > > Sr. Staff Engineer, Lustre Group > > Sun Microsystems of Canada, Inc. > > > >Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.