Does anybody have experience with Lustre filesystems in the 5-10B inode range? What kind of gotchas do you find at that scale? What are your stripe settings like? daniel --------------------------------------------------------------- This email, along with any attachments, is confidential. If you believe you received this message in error, please contact the sender immediately and delete all copies of the message. Thank you.
I doubt it. The largest lustre systems are a little over 300 million files that are really in existence. However, the limits, in terms of just maximum inodes allowed, using 4K per inode and a 2TB max MDT size (mirrored 2 copies), only allows for a little over 500 million files max. -Sam Whamcloud -----Original Message----- From: lustre-discuss-bounces at lists.lustre.org [mailto:lustre-discuss-bounces at lists.lustre.org] On Behalf Of Daniel Mayfield Sent: Friday, May 20, 2011 9:53 AM To: lustre-discuss at lists.lustre.org Subject: [Lustre-discuss] Huge numbers of inodes Does anybody have experience with Lustre filesystems in the 5-10B inode range? What kind of gotchas do you find at that scale? What are your stripe settings like? daniel --------------------------------------------------------------- This email, along with any attachments, is confidential. If you believe you received this message in error, please contact the sender immediately and delete all copies of the message. Thank you. _______________________________________________ Lustre-discuss mailing list Lustre-discuss at lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
You can adjust the defaults to force 1K spacing rather than the default 4K spacing Sam mentions, (assuming you''re not striping like crazy) and gain ~1.95 billion inodes against 2TB of space. I''ve done this successfully a few times. -cf On 05/20/2011 10:17 AM, Sam Bigger wrote:> I doubt it. The largest lustre systems are a little over 300 million files > that are really in existence. > > However, the limits, in terms of just maximum inodes allowed, using 4K per > inode and a 2TB max MDT size (mirrored 2 copies), only allows for a little > over 500 million files max. > > -Sam > Whamcloud > > -----Original Message----- > From: lustre-discuss-bounces at lists.lustre.org > [mailto:lustre-discuss-bounces at lists.lustre.org] On Behalf Of Daniel > Mayfield > Sent: Friday, May 20, 2011 9:53 AM > To: lustre-discuss at lists.lustre.org > Subject: [Lustre-discuss] Huge numbers of inodes > > Does anybody have experience with Lustre filesystems in the 5-10B inode > range? What kind of gotchas do you find at that scale? What are your > stripe settings like? > > daniel > > --------------------------------------------------------------- > This email, along with any attachments, is confidential. If you > believe you received this message in error, please contact the > sender immediately and delete all copies of the message. > Thank you. > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss
I''ve created 1 billion files (or more) in my MD tests, and one major thing that you do run into is the OST allocation issue I mentioned in my presentation at LUG. You can alleviate this by increasing the number of OSTs you have in your FS (I wouldn''t exceed 32M files per OST), or by changing the OST object layout. -----Original Message----- From: lustre-discuss-bounces at lists.lustre.org [mailto:lustre-discuss-bounces at lists.lustre.org] On Behalf Of Sam Bigger Sent: Friday, May 20, 2011 12:17 PM To: ''Daniel Mayfield''; lustre-discuss at lists.lustre.org Subject: Re: [Lustre-discuss] Huge numbers of inodes I doubt it. The largest lustre systems are a little over 300 million files that are really in existence. However, the limits, in terms of just maximum inodes allowed, using 4K per inode and a 2TB max MDT size (mirrored 2 copies), only allows for a little over 500 million files max. -Sam Whamcloud -----Original Message----- From: lustre-discuss-bounces at lists.lustre.org [mailto:lustre-discuss-bounces at lists.lustre.org] On Behalf Of Daniel Mayfield Sent: Friday, May 20, 2011 9:53 AM To: lustre-discuss at lists.lustre.org Subject: [Lustre-discuss] Huge numbers of inodes Does anybody have experience with Lustre filesystems in the 5-10B inode range? What kind of gotchas do you find at that scale? What are your stripe settings like? daniel --------------------------------------------------------------- This email, along with any attachments, is confidential. If you believe you received this message in error, please contact the sender immediately and delete all copies of the message. Thank you. _______________________________________________ Lustre-discuss mailing list Lustre-discuss at lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss _______________________________________________ Lustre-discuss mailing list Lustre-discuss at lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
On 2011-05-20, at 10:17 AM, "Sam Bigger" wrote:> I doubt it. The largest lustre systems are a little over 300 million files > that are really in existence. > > However, the limits, in terms of just maximum inodes allowed, using 4K per > inode and a 2TB max MDT size (mirrored 2 copies), only allows for a little > over 500 million files max.The theoretical limit is actually 4B inodes for ext3/4. I don''t think anyone has ever tested that. In Lustre 2.1 the default inode ratio has been changed to 1 inode per 2048 bytes (formerly 1 per 4096 bytes). That would allow 4B inodes on an 8TB MDT. This can be used for any version of mkfs.lustre with "--mkfsoptions ''-i 2048''". You probably also want to use the ext4 ldiskfs and add ''-t ext4'' to mkfsoptions so that it enables flex_bg to speed up mkfs and fsck time.> -----Original Message----- > From: lustre-discuss-bounces at lists.lustre.org > [mailto:lustre-discuss-bounces at lists.lustre.org] On Behalf Of Daniel > Mayfield > Sent: Friday, May 20, 2011 9:53 AM > To: lustre-discuss at lists.lustre.org > Subject: [Lustre-discuss] Huge numbers of inodes > > Does anybody have experience with Lustre filesystems in the 5-10B inode > range? What kind of gotchas do you find at that scale? What are your > stripe settings like? > > daniel > > --------------------------------------------------------------- > This email, along with any attachments, is confidential. If you > believe you received this message in error, please contact the > sender immediately and delete all copies of the message. > Thank you. > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss