I read once in the alias that running MDS requires a minimum disk space of 400MB + 4kB * number of files. Is this still the case with 1.6beta? lin
On Jan 29, 2007 12:25 -0800, Lin Shen (lshen) wrote:> I read once in the alias that running MDS requires a minimum disk space > of 400MB + 4kB * number of files. Is this still the case with 1.6beta?Yes. This relates more to ext3 than to Lustre. Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc.
David Ramsthaler (dramstha)
2007-Jan-30 09:02 UTC
[Lustre-discuss] disk space needed to run MDS
Andreas, I created an ext3 file system on a 16 Meg Partition, and it required ~1 Meg (8%) overhead. Mostly for the journal. On a 1 Gig disk, ext3 grabbed 17 Meg (2%). A few weeks ago I used that 16 Meg for mdt and mgs along with a 1 Gig OST. With a couple tests it seemed to work just fine. I was able to create 64K files on the 1 Gig disk. I am trying to understand that 400MB number. Can you explain a bit more detail? FYI - we are still evaluating Lustre for our project. Some of the platforms would be very small systems, and we are trying to make sure Lustre will scale down well. Thanks, -David -----Original Message----- From: lustre-discuss-bounces@clusterfs.com [mailto:lustre-discuss-bounces@clusterfs.com] On Behalf Of Andreas Dilger Sent: Monday, January 29, 2007 7:04 PM To: Lin Shen (lshen) Cc: lustre-discuss@clusterfs.com Subject: Re: [Lustre-discuss] disk space needed to run MDS On Jan 29, 2007 12:25 -0800, Lin Shen (lshen) wrote:> I read once in the alias that running MDS requires a minimum diskspace> of 400MB + 4kB * number of files. Is this still the case with 1.6beta?Yes. This relates more to ext3 than to Lustre. Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc. _______________________________________________ Lustre-discuss mailing list Lustre-discuss@clusterfs.com https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
Hi Andreas, Could you explain a bit more how the number 400M was decided? Ext3 itself won''t need that much disk space, right? Also, is this number related to the total capacity of the Luster filesystem? In other words, will the MDS always need 400M to work properly for a filesystem of size 1T or just 1G? By switching to a different underline filesystem (i.e., from ext3 to ext2), can this number be lowered? This is important to us. Since we may have a use case that runs Luster on a 1G Compact Flash (to be shared among a few nodes). Does this mean that only a little more than half of the capacity can be used for data? Lin> -----Original Message----- > From: Andreas Dilger [mailto:adilger@clusterfs.com] > Sent: Monday, January 29, 2007 7:04 PM > To: Lin Shen (lshen) > Cc: lustre-discuss@clusterfs.com > Subject: Re: [Lustre-discuss] disk space needed to run MDS > > On Jan 29, 2007 12:25 -0800, Lin Shen (lshen) wrote: > > I read once in the alias that running MDS requires a minimum disk > > space of 400MB + 4kB * number of files. Is this still the > case with 1.6beta? > > Yes. This relates more to ext3 than to Lustre. > > Cheers, Andreas > -- > Andreas Dilger > Principal Software Engineer > Cluster File Systems, Inc. >
On Feb 02, 2007 10:25 -0800, Lin Shen (lshen) wrote:> Could you explain a bit more how the number 400M was decided? Ext3 > itself won''t need that much disk space, right? Also, is this number > related to the total capacity of the Luster filesystem? In other words, > will the MDS always need 400M to work properly for a filesystem of size > 1T or just 1G? By switching to a different underline filesystem (i.e., > from ext3 to ext2), can this number be lowered?In fact you do NOT need the full 400MB of journal space, and lconf will not create a journal that large if you have a smaller filesystem. The default size of the journal is: - device size < 1GB => use default mke2fs, 16MB - 1GB < device size < 10GB => use device size / 25 - device size > 10GB => use 400MB> This is important to us. Since we may have a use case that runs Luster > on a 1G Compact Flash (to be shared among a few nodes). Does this mean > that only a little more than half of the capacity can be used for data?No, nothing of the sort. Ext3 can have a journal as small as 4MB, regardless of the device size. The only drawback is a potential reduction in performance when there are many clients. In a small setup like yours it may not make much difference at all.> > -----Original Message----- > > From: Andreas Dilger [mailto:adilger@clusterfs.com] > > Sent: Monday, January 29, 2007 7:04 PM > > To: Lin Shen (lshen) > > Cc: lustre-discuss@clusterfs.com > > Subject: Re: [Lustre-discuss] disk space needed to run MDS > > > > On Jan 29, 2007 12:25 -0800, Lin Shen (lshen) wrote: > > > I read once in the alias that running MDS requires a minimum disk > > > space of 400MB + 4kB * number of files. Is this still the > > case with 1.6beta? > > > > Yes. This relates more to ext3 than to Lustre. > > > > Cheers, Andreas > > -- > > Andreas Dilger > > Principal Software Engineer > > Cluster File Systems, Inc. > >Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc.
How to set the journal size? Don''t see that as part of mkfs.lustre. Lin> -----Original Message----- > From: Andreas Dilger [mailto:adilger@clusterfs.com] > Sent: Monday, February 05, 2007 9:03 AM > To: Lin Shen (lshen) > Cc: lustre-discuss@clusterfs.com > Subject: Re: [Lustre-discuss] disk space needed to run MDS > > On Feb 02, 2007 10:25 -0800, Lin Shen (lshen) wrote: > > Could you explain a bit more how the number 400M was decided? Ext3 > > itself won''t need that much disk space, right? Also, is this number > > related to the total capacity of the Luster filesystem? In other > > words, will the MDS always need 400M to work properly for a > filesystem > > of size 1T or just 1G? By switching to a different underline > > filesystem (i.e., from ext3 to ext2), can this number be lowered? > > In fact you do NOT need the full 400MB of journal space, and > lconf will not create a journal that large if you have a > smaller filesystem. > > The default size of the journal is: > - device size < 1GB => use default mke2fs, 16MB > - 1GB < device size < 10GB => use device size / 25 > - device size > 10GB => use 400MB > > > This is important to us. Since we may have a use case that > runs Luster > > on a 1G Compact Flash (to be shared among a few nodes). > Does this mean > > that only a little more than half of the capacity can be > used for data? > > No, nothing of the sort. Ext3 can have a journal as small as > 4MB, regardless of the device size. The only drawback is a > potential reduction in performance when there are many > clients. In a small setup like yours it may not make much > difference at all. > > > > -----Original Message----- > > > From: Andreas Dilger [mailto:adilger@clusterfs.com] > > > Sent: Monday, January 29, 2007 7:04 PM > > > To: Lin Shen (lshen) > > > Cc: lustre-discuss@clusterfs.com > > > Subject: Re: [Lustre-discuss] disk space needed to run MDS > > > > > > On Jan 29, 2007 12:25 -0800, Lin Shen (lshen) wrote: > > > > I read once in the alias that running MDS requires a > minimum disk > > > > space of 400MB + 4kB * number of files. Is this still the > > > case with 1.6beta? > > > > > > Yes. This relates more to ext3 than to Lustre. > > > > > > Cheers, Andreas > > > -- > > > Andreas Dilger > > > Principal Software Engineer > > > Cluster File Systems, Inc. > > > > > Cheers, Andreas > -- > Andreas Dilger > Principal Software Engineer > Cluster File Systems, Inc. >
> -----Original Message----- > From: Andreas Dilger [mailto:adilger@clusterfs.com] > Sent: Monday, February 05, 2007 9:03 AM > To: Lin Shen (lshen) > Cc: lustre-discuss@clusterfs.com > Subject: Re: [Lustre-discuss] disk space needed to run MDS > > On Feb 02, 2007 10:25 -0800, Lin Shen (lshen) wrote: > > Could you explain a bit more how the number 400M was decided? Ext3 > > itself won''t need that much disk space, right? Also, is this number > > related to the total capacity of the Luster filesystem? In other > > words, will the MDS always need 400M to work properly for a > filesystem > > of size 1T or just 1G? By switching to a different underline > > filesystem (i.e., from ext3 to ext2), can this number be lowered? > > In fact you do NOT need the full 400MB of journal space, and > lconf will not create a journal that large if you have a > smaller filesystem.With 1.6beta, I guess the journal is created by mkfs.lustre --mdt. How does it know the size of the file system?> > The default size of the journal is: > - device size < 1GB => use default mke2fs, 16MB > - 1GB < device size < 10GB => use device size / 25 > - device size > 10GB => use 400MBI think "devide size" at here means the size of the device (eg., partition) that mkfs.lustre --mdt is run on, right? This leads to another question. How the size of the mdt device should be decided? Assuming I have a 512MB disk, how should I partition the disk into 2 (one for mdt, one for ost)? Of course I want as much as possible for OST. Lin> > > This is important to us. Since we may have a use case that > runs Luster > > on a 1G Compact Flash (to be shared among a few nodes). > Does this mean > > that only a little more than half of the capacity can be > used for data? > > No, nothing of the sort. Ext3 can have a journal as small as > 4MB, regardless of the device size. The only drawback is a > potential reduction in performance when there are many > clients. In a small setup like yours it may not make much > difference at all. > > > > -----Original Message----- > > > From: Andreas Dilger [mailto:adilger@clusterfs.com] > > > Sent: Monday, January 29, 2007 7:04 PM > > > To: Lin Shen (lshen) > > > Cc: lustre-discuss@clusterfs.com > > > Subject: Re: [Lustre-discuss] disk space needed to run MDS > > > > > > On Jan 29, 2007 12:25 -0800, Lin Shen (lshen) wrote: > > > > I read once in the alias that running MDS requires a > minimum disk > > > > space of 400MB + 4kB * number of files. Is this still the > > > case with 1.6beta? > > > > > > Yes. This relates more to ext3 than to Lustre. > > > > > > Cheers, Andreas > > > -- > > > Andreas Dilger > > > Principal Software Engineer > > > Cluster File Systems, Inc. > > > > > Cheers, Andreas > -- > Andreas Dilger > Principal Software Engineer > Cluster File Systems, Inc. >
Lin Shen (lshen) wrote:> > > >> -----Original Message----- >> From: Andreas Dilger [mailto:adilger@clusterfs.com] >> Sent: Monday, February 05, 2007 9:03 AM >> To: Lin Shen (lshen) >> Cc: lustre-discuss@clusterfs.com >> Subject: Re: [Lustre-discuss] disk space needed to run MDS >> >> On Feb 02, 2007 10:25 -0800, Lin Shen (lshen) wrote: >> >>> Could you explain a bit more how the number 400M was decided? Ext3 >>> itself won''t need that much disk space, right? Also, is this number >>> related to the total capacity of the Luster filesystem? In other >>> words, will the MDS always need 400M to work properly for a >>> >> filesystem >> >>> of size 1T or just 1G? By switching to a different underline >>> filesystem (i.e., from ext3 to ext2), can this number be lowered? >>> >> In fact you do NOT need the full 400MB of journal space, and >> lconf will not create a journal that large if you have a >> smaller filesystem. >> > > With 1.6beta, I guess the journal is created by mkfs.lustre --mdt. How > does it know the size of the file system? > >mkfs.lustre uses the BLKGETSIZE64 ioctl for the disk size.
> -----Original Message----- > From: Nathaniel Rutman [mailto:nathan@clusterfs.com] > Sent: Wednesday, February 07, 2007 12:22 PM > To: Lin Shen (lshen) > Cc: lustre-discuss@clusterfs.com > Subject: Re: [Lustre-discuss] disk space needed to run MDS > > Lin Shen (lshen) wrote: > > > > > > > >> -----Original Message----- > >> From: Andreas Dilger [mailto:adilger@clusterfs.com] > >> Sent: Monday, February 05, 2007 9:03 AM > >> To: Lin Shen (lshen) > >> Cc: lustre-discuss@clusterfs.com > >> Subject: Re: [Lustre-discuss] disk space needed to run MDS > >> > >> On Feb 02, 2007 10:25 -0800, Lin Shen (lshen) wrote: > >> > >>> Could you explain a bit more how the number 400M was > decided? Ext3 > >>> itself won''t need that much disk space, right? Also, is > this number > >>> related to the total capacity of the Luster filesystem? In other > >>> words, will the MDS always need 400M to work properly for a > >>> > >> filesystem > >> > >>> of size 1T or just 1G? By switching to a different underline > >>> filesystem (i.e., from ext3 to ext2), can this number be lowered? > >>> > >> In fact you do NOT need the full 400MB of journal space, and lconf > >> will not create a journal that large if you have a smaller > >> filesystem. > >> > > > > With 1.6beta, I guess the journal is created by mkfs.lustre > --mdt. How > > does it know the size of the file system? > > > > > mkfs.lustre uses the BLKGETSIZE64 ioctl for the disk size.That''s only the size of the disk for running mdt, right? It''s not the file system size since maybe it''s unknown at the time that mkfs.lustre --mdt is issued (OSTs could be added in the future).>
Lin Shen (lshen) wrote:> > > >> -----Original Message----- >> From: Nathaniel Rutman [mailto:nathan@clusterfs.com] >> Sent: Wednesday, February 07, 2007 12:22 PM >> To: Lin Shen (lshen) >> Cc: lustre-discuss@clusterfs.com >> Subject: Re: [Lustre-discuss] disk space needed to run MDS >> >> Lin Shen (lshen) wrote: >> >>> >>> >>> >>> >>>> -----Original Message----- >>>> From: Andreas Dilger [mailto:adilger@clusterfs.com] >>>> Sent: Monday, February 05, 2007 9:03 AM >>>> To: Lin Shen (lshen) >>>> Cc: lustre-discuss@clusterfs.com >>>> Subject: Re: [Lustre-discuss] disk space needed to run MDS >>>> >>>> On Feb 02, 2007 10:25 -0800, Lin Shen (lshen) wrote: >>>> >>>> >>>>> Could you explain a bit more how the number 400M was >>>>> >> decided? Ext3 >> >>>>> itself won''t need that much disk space, right? Also, is >>>>> >> this number >> >>>>> related to the total capacity of the Luster filesystem? In other >>>>> words, will the MDS always need 400M to work properly for a >>>>> >>>>> >>>> filesystem >>>> >>>> >>>>> of size 1T or just 1G? By switching to a different underline >>>>> filesystem (i.e., from ext3 to ext2), can this number be lowered? >>>>> >>>>> >>>> In fact you do NOT need the full 400MB of journal space, and lconf >>>> will not create a journal that large if you have a smaller >>>> filesystem. >>>> >>>> >>> With 1.6beta, I guess the journal is created by mkfs.lustre >>> >> --mdt. How >> >>> does it know the size of the file system? >>> >>> >>> >> mkfs.lustre uses the BLKGETSIZE64 ioctl for the disk size. >> > > That''s only the size of the disk for running mdt, right? It''s not the > file system size since maybe it''s unknown at the time that mkfs.lustre > --mdt is issued (OSTs could be added in the future). >That''s correct - mkfs only affect the local disk, and the journal is specific to the local disk as well. The other OSTs have their own journals.