" In a Lustre ldiskfs file system, all the inodes are allocated on the MDT and OSTs when the file system is first formatted. The total number of inodes on a formatted MDT or OST cannot be easily changed, although it is possible to add OSTs with additional space and corresponding inodes. Thus, the number of inodes created at format time should be generous enough to anticipate future expansion. " http://build.whamcloud.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml#understandinglustre.fig.cluster I am not sure I understand this 100%, when adding an OST I do or don''t gain inodes?or you basically only get more inodes in the data part of the filesystem but if you don''t have enough inodes on the MDT you still have a problem? Also if lustrefs is really based on ext4 doesn''t ext4 support growing the amount of inodes with resize2fs? Thanks, Eli
*IMO* there are two limiting factors 1. total number of inodes supported by MDT and 2. OSTs i.e. num_ost_inodes = 4 * number_of_mds_inodes * default_stripe_count / *number_of_osts* so your Q I am not sure I understand this 100%, when adding an OST I do or don''t gain inodes?or you basically only get more inodes in the data part of the filesystem but if you don''t have enough inodes on the MDT you still have a problem? A Yes, if there are no enough inodes on MDT adding OSTs wont help much. I don''t think you can change no. of inodes w/o formatting. You might want to take a look at lfs_migrate, on how to increase the no_of_inodes on OST''s. HTH p On 19 June 2013 19:19, E.S. Rosenberg <esr+lustre-ed8gSyzkC85vlA8e7MYcDA@public.gmane.org> wrote:> " > In a Lustre ldiskfs file system, all the inodes are allocated on the > MDT and OSTs when the file system is first formatted. The total number > of inodes on a formatted MDT or OST cannot be easily changed, although > it is possible to add OSTs with additional space and corresponding > inodes. Thus, the number of inodes created at format time should be > generous enough to anticipate future expansion. > " > > http://build.whamcloud.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml#understandinglustre.fig.cluster > > > > Also if lustrefs is really based on ext4 doesn''t ext4 support growing > the amount of inodes with resize2fs? > > Thanks, > Eli > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >_______________________________________________ Lustre-discuss mailing list Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Eli, We probably should clarify the context of inodes: 1) Each OST has "inodes" - these inodes refer to objects on that OST that contain block data for a Lustre file. 2) Each (as of 2.4) MDT has "inodes" - these inodes contain metadata for a single Lustre file. When adding an OST you gain the inodes on that OST for objects (block data) on that OST, but the total amount of Lustre files (MDT inodes) hasn''t changed. -- Brett Lee Sr. Systems Engineer Intel High Performance Data Division From: lustre-discuss-bounces-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org [mailto:lustre-discuss-bounces@lists.lustre.org] On Behalf Of Parinay Kondekar Sent: Wednesday, June 19, 2013 8:23 AM To: E.S. Rosenberg Cc: lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org Subject: Re: [Lustre-discuss] explanation about inodes in MDT/OST IMO there are two limiting factors 1. total number of inodes supported by MDT and 2. OSTs i.e. num_ost_inodes = 4 * number_of_mds_inodes * default_stripe_count / number_of_osts so your Q I am not sure I understand this 100%, when adding an OST I do or don''t gain inodes?or you basically only get more inodes in the data part of the filesystem but if you don''t have enough inodes on the MDT you still have a problem? A Yes, if there are no enough inodes on MDT adding OSTs wont help much. I don''t think you can change no. of inodes w/o formatting. You might want to take a look at lfs_migrate, on how to increase the no_of_inodes on OST''s. HTH p On 19 June 2013 19:19, E.S. Rosenberg <esr+lustre-ed8gSyzkC85vlA8e7MYcDA@public.gmane.org<mailto:esr+lustre-ed8gSyzkC85vlA8e7MYcDA@public.gmane.org>> wrote: " In a Lustre ldiskfs file system, all the inodes are allocated on the MDT and OSTs when the file system is first formatted. The total number of inodes on a formatted MDT or OST cannot be easily changed, although it is possible to add OSTs with additional space and corresponding inodes. Thus, the number of inodes created at format time should be generous enough to anticipate future expansion. " http://build.whamcloud.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml#understandinglustre.fig.cluster Also if lustrefs is really based on ext4 doesn''t ext4 support growing the amount of inodes with resize2fs? Thanks, Eli _______________________________________________ Lustre-discuss mailing list Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org<mailto:Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org> http://lists.lustre.org/mailman/listinfo/lustre-discuss _______________________________________________ Lustre-discuss mailing list Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
On 2013-06-19, at 8:52, "Lee, Brett" <brett.lee-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org<mailto:brett.lee@intel.com>> wrote: We probably should clarify the context of inodes: 1) Each OST has "inodes" – these inodes refer to objects on that OST that contain block data for a Lustre file. I prefer to call these as OST objects, to keep them distinct from the MDT inodes. 2) Each (as of 2.4) MDT has "inodes" – these inodes contain metadata for a single Lustre file. When adding an OST you gain the inodes on that OST for objects (block data) on that OST, but the total amount of Lustre files (MDT inodes) hasn''t changed. Right, though Mr. Rosenberg is correct that since the MDT uses an ext4-based backing storage it is possible to use offline resize2fs to increase the size of the MDT afterward, if it is using LVM or some other form of storage that can be resized. Online resizing is blocked behind LU-137 being finished. Note that resize2fs is not something that we test on any regular basis, because most of our customers have large fixed-size devices for the MDT (over-provisioning on spindles). If someone needs to do this, is encourage them to make a "dd" copy of the MDT to a new device first (could just be a USB SATA disk) and test out the resize2fs first, and verify with e2fsck before and after that the filesystem is still valid. I''ll see whether i can get resize2fs testing added into our normal test suite. Cheers, Andreas From: lustre-discuss-bounces-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org<mailto:lustre-discuss-bounces@lists.lustre.org> [mailto:lustre-discuss-bounces-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org] On Behalf Of Parinay Kondekar Sent: Wednesday, June 19, 2013 8:23 AM To: E.S. Rosenberg Cc: lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org<mailto:lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org> Subject: Re: [Lustre-discuss] explanation about inodes in MDT/OST IMO there are two limiting factors 1. total number of inodes supported by MDT and 2. OSTs i.e. num_ost_inodes = 4 * number_of_mds_inodes * default_stripe_count / number_of_osts so your Q I am not sure I understand this 100%, when adding an OST I do or don''t gain inodes?or you basically only get more inodes in the data part of the filesystem but if you don''t have enough inodes on the MDT you still have a problem? A Yes, if there are no enough inodes on MDT adding OSTs wont help much. I don''t think you can change no. of inodes w/o formatting. You might want to take a look at lfs_migrate, on how to increase the no_of_inodes on OST''s. HTH p On 19 June 2013 19:19, E.S. Rosenberg <esr+lustre-ed8gSyzkC85vlA8e7MYcDA@public.gmane.org<mailto:esr+lustre-ed8gSyzkC85vlA8e7MYcDA@public.gmane.org>> wrote: " In a Lustre ldiskfs file system, all the inodes are allocated on the MDT and OSTs when the file system is first formatted. The total number of inodes on a formatted MDT or OST cannot be easily changed, although it is possible to add OSTs with additional space and corresponding inodes. Thus, the number of inodes created at format time should be generous enough to anticipate future expansion. " http://build.whamcloud.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml#understandinglustre.fig.cluster Also if lustrefs is really based on ext4 doesn''t ext4 support growing the amount of inodes with resize2fs? Thanks, Eli _______________________________________________ Lustre-discuss mailing list Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org<mailto:Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org> http://lists.lustre.org/mailman/listinfo/lustre-discuss _______________________________________________ Lustre-discuss mailing list Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org<mailto:Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org> http://lists.lustre.org/mailman/listinfo/lustre-discuss
Cool, thanks for all the in-depth answers. Eli On Wed, Jun 19, 2013 at 6:33 PM, Dilger, Andreas <andreas.dilger-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:> On 2013-06-19, at 8:52, "Lee, Brett" <brett.lee-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org<mailto:brett.lee@intel.com>> wrote: > We probably should clarify the context of inodes: > > 1) Each OST has "inodes" – these inodes refer to objects on that OST that contain block data for a Lustre file. > > I prefer to call these as OST objects, to keep them distinct from the MDT inodes. > > 2) Each (as of 2.4) MDT has "inodes" – these inodes contain metadata for a single Lustre file. > > When adding an OST you gain the inodes on that OST for objects (block data) on that OST, but the total amount of Lustre files (MDT inodes) hasn''t changed. > > Right, though Mr. Rosenberg is correct that since the MDT uses an ext4-based backing storage it is possible to use offline resize2fs to increase the size of the MDT afterward, if it is using LVM or some other form of storage that can be resized. Online resizing is blocked behind LU-137 being finished. > > Note that resize2fs is not something that we test on any regular basis, because most of our customers have large fixed-size devices for the MDT (over-provisioning on spindles). If someone needs to do this, is encourage them to make a "dd" copy of the MDT to a new device first (could just be a USB SATA disk) and test out the resize2fs first, and verify with e2fsck before and after that the filesystem is still valid. > > I''ll see whether i can get resize2fs testing added into our normal test suite. > > Cheers, Andreas > > > From: lustre-discuss-bounces-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org<mailto:lustre-discuss-bounces-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org> [mailto:lustre-discuss-bounces-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org] On Behalf Of Parinay Kondekar > Sent: Wednesday, June 19, 2013 8:23 AM > To: E.S. Rosenberg > Cc: lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org<mailto:lustre-discuss-aLEFhgZF4x4kHE2CFeF4+A@public.gmane.orgg> > Subject: Re: [Lustre-discuss] explanation about inodes in MDT/OST > > IMO there are two limiting factors > > 1. total number of inodes supported by MDT and > 2. OSTs i.e. num_ost_inodes = 4 * number_of_mds_inodes * default_stripe_count / number_of_osts > > so your Q > I am not sure I understand this 100%, when adding an OST I do or don''t > gain inodes?or you basically only get more inodes in the data part of > the filesystem but if you don''t have enough inodes on the MDT you > still have a problem? > A > Yes, if there are no enough inodes on MDT adding OSTs wont help much. > > I don''t think you can change no. of inodes w/o formatting. You might want to take a look at > lfs_migrate, on how to increase the no_of_inodes on OST''s. > > HTH > p > > On 19 June 2013 19:19, E.S. Rosenberg <esr+lustre-ed8gSyzkC85vlA8e7MYcDA@public.gmane.org<mailto:esr+lustre-ed8gSyzkC85vlA8e7MYcDA@public.gmane.org>> wrote: > " > In a Lustre ldiskfs file system, all the inodes are allocated on the > MDT and OSTs when the file system is first formatted. The total number > of inodes on a formatted MDT or OST cannot be easily changed, although > it is possible to add OSTs with additional space and corresponding > inodes. Thus, the number of inodes created at format time should be > generous enough to anticipate future expansion. > " > http://build.whamcloud.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml#understandinglustre.fig.cluster > > > > Also if lustrefs is really based on ext4 doesn''t ext4 support growing > the amount of inodes with resize2fs? > > Thanks, > Eli > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org<mailto:Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org> > http://lists.lustre.org/mailman/listinfo/lustre-discuss > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org<mailto:Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org> > http://lists.lustre.org/mailman/listinfo/lustre-discuss > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss