I configured a Lustre file system on a collection of storage servers that have 12TB raw devices. I configured a combined MGS/MDS with the default configuration. On the OSTs however I added the force_over_8tb to the mountfsoptions. Two part question: 1- do I need to set that parameter on the MGS/MDS server as well 2- if yes, how do I properly add this parameter on this running Lustre file system (100TB on 9 storage servers) I can''t resolve the ambiguity in the documentation as I can''t find a good explanation of the configuration log mechanism that is being referenced in the man pages. The fact that the doc for --writeconf states "This is very dangerous", I am hesitant to pull the trigger as there is 60TB of data on this file system that I rather not lose.
On Jul 14, 2011, at 1:15 PM, Theodore Omtzigt wrote:> Two part question: > 1- do I need to set that parameter on the MGS/MDS server as wellNo, they are different filesystems. You shouldn''t need to do this on the OSTs either. You must be using an older lustre release.> 2- if yes, how do I properly add this parameter on this running Lustre > file system (100TB on 9 storage servers)covered> I can''t resolve the ambiguity in the documentation as I can''t find a > good explanation of the configuration log mechanism that is being > referenced in the man pages. The fact that the doc for --writeconf > states "This is very dangerous", I am hesitant to pull the trigger as > there is 60TB of data on this file system that I rather not lose.I''ve had no issues with writeconf. Its nice because it shows you the old and new parameters. Make sure that the changes that you made were the what you want, and that the old parameters that you want to keep are still in tact. I don''t remember the exact circumstances, but I''ve found settings were lost when doing a writeconf, and I had to explictly put these settings in tunefs.lustre command to preserve them. -mb -- +----------------------------------------------- | Michael Barnes | | Thomas Jefferson National Accelerator Facility | Scientific Computing Group | 12000 Jefferson Ave. | Newport News, VA 23606 | (757) 269-7634 +-----------------------------------------------
If you are seeing this problem it means you are using the ext3-based ldiskfs. Go back to the download site and get the lustre-ldiskfs and lustre-modules RPMs with ext4 in the name. That is the code that was tested with LUNs over 8TB. We kept these separate for some time to reduce risk for users that did not need larger LUN sizes. This is the default for the recent Whamcloud 1.8.6 release. Cheers, Andreas On 2011-07-14, at 11:15 AM, Theodore Omtzigt <theo at stillwater-sc.com> wrote:> I configured a Lustre file system on a collection of storage servers > that have 12TB raw devices. I configured a combined MGS/MDS with the > default configuration. On the OSTs however I added the force_over_8tb to > the mountfsoptions. > > Two part question: > 1- do I need to set that parameter on the MGS/MDS server as well > 2- if yes, how do I properly add this parameter on this running Lustre > file system (100TB on 9 storage servers) > > I can''t resolve the ambiguity in the documentation as I can''t find a > good explanation of the configuration log mechanism that is being > referenced in the man pages. The fact that the doc for --writeconf > states "This is very dangerous", I am hesitant to pull the trigger as > there is 60TB of data on this file system that I rather not lose. > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss
Andreas: Thanks for taking a look at this. Unfortunately, I don''t quite understand the guidance you present: "If you are seeing ''this'' problem....". I haven''t seen ''any'' problems pertaining to >8tb yet, so I cannot place your guidance in the context of the question I posted. My question was whether or not I need this parameter on the MDS and if so, how to apply it retroactively. The Lustre environment I installed was the 1.8.5 set. Any insight in the issues would be appreciated. Theo On 7/14/2011 1:41 PM, Andreas Dilger wrote:> If you are seeing this problem it means you are using the ext3-based ldiskfs. Go back to the download site and get the lustre-ldiskfs and lustre-modules RPMs with ext4 in the name. > > That is the code that was tested with LUNs over 8TB. We kept these separate for some time to reduce risk for users that did not need larger LUN sizes. This is the default for the recent Whamcloud 1.8.6 release. > > Cheers, Andreas > > On 2011-07-14, at 11:15 AM, Theodore Omtzigt<theo at stillwater-sc.com> wrote: > >> I configured a Lustre file system on a collection of storage servers >> that have 12TB raw devices. I configured a combined MGS/MDS with the >> default configuration. On the OSTs however I added the force_over_8tb to >> the mountfsoptions. >> >> Two part question: >> 1- do I need to set that parameter on the MGS/MDS server as well >> 2- if yes, how do I properly add this parameter on this running Lustre >> file system (100TB on 9 storage servers) >> >> I can''t resolve the ambiguity in the documentation as I can''t find a >> good explanation of the configuration log mechanism that is being >> referenced in the man pages. The fact that the doc for --writeconf >> states "This is very dangerous", I am hesitant to pull the trigger as >> there is 60TB of data on this file system that I rather not lose. >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss
Michael:
The reason I had to do it on the OST''s is because when issuing the
mkfs.lustre command to build the OST it would error out with the message
that I should use the force_over_8tb mount option. I was not able to
create an OST on that device without the force_over_8tb option.
Your insights on the writeconf are excellent: good to know that
writeconf is solid. Thank you.
Theo
On 7/14/2011 1:29 PM, Michael Barnes wrote:> On Jul 14, 2011, at 1:15 PM, Theodore Omtzigt wrote:
>
>> Two part question:
>> 1- do I need to set that parameter on the MGS/MDS server as well
> No, they are different filesystems. You shouldn''t need to do this
on the OSTs either. You must be using an older lustre release.
>
>> 2- if yes, how do I properly add this parameter on this running Lustre
>> file system (100TB on 9 storage servers)
> covered
>
>> I can''t resolve the ambiguity in the documentation as I
can''t find a
>> good explanation of the configuration log mechanism that is being
>> referenced in the man pages. The fact that the doc for --writeconf
>> states "This is very dangerous", I am hesitant to pull the
trigger as
>> there is 60TB of data on this file system that I rather not lose.
> I''ve had no issues with writeconf. Its nice because it shows you
the old and new parameters. Make sure that the changes that you made were the
what you want, and that the old parameters that you want to keep are still in
tact. I don''t remember the exact circumstances, but I''ve
found settings were lost when doing a writeconf, and I had to explictly put
these settings in tunefs.lustre command to preserve them.
>
> -mb
>
> --
> +-----------------------------------------------
> | Michael Barnes
> |
> | Thomas Jefferson National Accelerator Facility
> | Scientific Computing Group
> | 12000 Jefferson Ave.
> | Newport News, VA 23606
> | (757) 269-7634
> +-----------------------------------------------
>
>
>
>
>
--writeconf will erase parameters set via lctl conf_param, and will erase pools definitions. It will also allow you to set rather silly parameters that can prevent your filesystem from starting, such as incorrect server NIDs or incorrect failover NIDs. For this reason (and from a history of customer support) we caveat it''s use in the manual. The --writeconf option never touches data, only server configs, so it will not mess up your data. So, given sensible precautions as mentioned above, it''s safe to do. cliffw On Thu, Jul 14, 2011 at 11:03 AM, Theodore Omtzigt <theo at stillwater-sc.com>wrote:> Andreas: > > Thanks for taking a look at this. Unfortunately, I don''t quite > understand the guidance you present: "If you are seeing ''this'' > problem....". I haven''t seen ''any'' problems pertaining to >8tb yet, so I > cannot place your guidance in the context of the question I posted. > > My question was whether or not I need this parameter on the MDS and if > so, how to apply it retroactively. The Lustre environment I installed > was the 1.8.5 set. Any insight in the issues would be appreciated. > > Theo > > On 7/14/2011 1:41 PM, Andreas Dilger wrote: > > If you are seeing this problem it means you are using the ext3-based > ldiskfs. Go back to the download site and get the lustre-ldiskfs and > lustre-modules RPMs with ext4 in the name. > > > > That is the code that was tested with LUNs over 8TB. We kept these > separate for some time to reduce risk for users that did not need larger LUN > sizes. This is the default for the recent Whamcloud 1.8.6 release. > > > > Cheers, Andreas > > > > On 2011-07-14, at 11:15 AM, Theodore Omtzigt<theo at stillwater-sc.com> > wrote: > > > >> I configured a Lustre file system on a collection of storage servers > >> that have 12TB raw devices. I configured a combined MGS/MDS with the > >> default configuration. On the OSTs however I added the force_over_8tb to > >> the mountfsoptions. > >> > >> Two part question: > >> 1- do I need to set that parameter on the MGS/MDS server as well > >> 2- if yes, how do I properly add this parameter on this running Lustre > >> file system (100TB on 9 storage servers) > >> > >> I can''t resolve the ambiguity in the documentation as I can''t find a > >> good explanation of the configuration log mechanism that is being > >> referenced in the man pages. The fact that the doc for --writeconf > >> states "This is very dangerous", I am hesitant to pull the trigger as > >> there is 60TB of data on this file system that I rather not lose. > >> _______________________________________________ > >> Lustre-discuss mailing list > >> Lustre-discuss at lists.lustre.org > >> http://lists.lustre.org/mailman/listinfo/lustre-discuss > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >-- cliffw Support Guy WhamCloud, Inc. www.whamcloud.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20110714/bfcde4f6/attachment-0001.html
This error message you are seeing is what Andreas was talking about - you must use the ext4-based version, as you will not need any option with your size LUNS. The ''must use force_over_8tb'' error is the key here, you most certainly want/need to *.ext4.rpm versions of stuff. cliffw On Thu, Jul 14, 2011 at 11:10 AM, Theodore Omtzigt <theo at stillwater-sc.com>wrote:> Michael: > > The reason I had to do it on the OST''s is because when issuing the > mkfs.lustre command to build the OST it would error out with the message > that I should use the force_over_8tb mount option. I was not able to > create an OST on that device without the force_over_8tb option. > > Your insights on the writeconf are excellent: good to know that > writeconf is solid. Thank you. > > Theo > > On 7/14/2011 1:29 PM, Michael Barnes wrote: > > On Jul 14, 2011, at 1:15 PM, Theodore Omtzigt wrote: > > > >> Two part question: > >> 1- do I need to set that parameter on the MGS/MDS server as well > > No, they are different filesystems. You shouldn''t need to do this on the > OSTs either. You must be using an older lustre release. > > > >> 2- if yes, how do I properly add this parameter on this running Lustre > >> file system (100TB on 9 storage servers) > > covered > > > >> I can''t resolve the ambiguity in the documentation as I can''t find a > >> good explanation of the configuration log mechanism that is being > >> referenced in the man pages. The fact that the doc for --writeconf > >> states "This is very dangerous", I am hesitant to pull the trigger as > >> there is 60TB of data on this file system that I rather not lose. > > I''ve had no issues with writeconf. Its nice because it shows you the old > and new parameters. Make sure that the changes that you made were the what > you want, and that the old parameters that you want to keep are still in > tact. I don''t remember the exact circumstances, but I''ve found settings > were lost when doing a writeconf, and I had to explictly put these settings > in tunefs.lustre command to preserve them. > > > > -mb > > > > -- > > +----------------------------------------------- > > | Michael Barnes > > | > > | Thomas Jefferson National Accelerator Facility > > | Scientific Computing Group > > | 12000 Jefferson Ave. > > | Newport News, VA 23606 > > | (757) 269-7634 > > +----------------------------------------------- > > > > > > > > > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >-- cliffw Support Guy WhamCloud, Inc. www.whamcloud.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20110714/587c3cc0/attachment.html
With one other note: you should have used "--mkfsoptions=''-t ext4''" when doing mkfs.lustre, and NOT the force option. Given that it is already formatted and you don''t want to use data, at least use the "ext4" Lustre RPMs. Pretty sure you don''t need a --writeconf -- you would either run as-is with ext4-based ldiskfs or reformat. The MDT device should be limited to 8TB; I don''t think anyone has tested a larger MDT. Kevin Cliff White wrote:> This error message you are seeing is what Andreas was talking about - > you must use the > ext4-based version, as you will not need any option with your size > LUNS. The ''must use force_over_8tb'' > error is the key here, you most certainly want/need to *.ext4.rpm > versions of stuff. > cliffw > > > On Thu, Jul 14, 2011 at 11:10 AM, Theodore Omtzigt > <theo at stillwater-sc.com <mailto:theo at stillwater-sc.com>> wrote: > > Michael: > > The reason I had to do it on the OST''s is because when issuing the > mkfs.lustre command to build the OST it would error out with the > message > that I should use the force_over_8tb mount option. I was not able to > create an OST on that device without the force_over_8tb option. > > Your insights on the writeconf are excellent: good to know that > writeconf is solid. Thank you. > > Theo > > On 7/14/2011 1:29 PM, Michael Barnes wrote: > > On Jul 14, 2011, at 1:15 PM, Theodore Omtzigt wrote: > > > >> Two part question: > >> 1- do I need to set that parameter on the MGS/MDS server as well > > No, they are different filesystems. You shouldn''t need to do > this on the OSTs either. You must be using an older lustre release. > > > >> 2- if yes, how do I properly add this parameter on this running > Lustre > >> file system (100TB on 9 storage servers) > > covered > > > >> I can''t resolve the ambiguity in the documentation as I can''t > find a > >> good explanation of the configuration log mechanism that is being > >> referenced in the man pages. The fact that the doc for --writeconf > >> states "This is very dangerous", I am hesitant to pull the > trigger as > >> there is 60TB of data on this file system that I rather not lose. > > I''ve had no issues with writeconf. Its nice because it shows > you the old and new parameters. Make sure that the changes that > you made were the what you want, and that the old parameters that > you want to keep are still in tact. I don''t remember the exact > circumstances, but I''ve found settings were lost when doing a > writeconf, and I had to explictly put these settings in > tunefs.lustre command to preserve them. > > > > -mb > > > > -- > > +----------------------------------------------- > > | Michael Barnes > > | > > | Thomas Jefferson National Accelerator Facility > > | Scientific Computing Group > > | 12000 Jefferson Ave. > > | Newport News, VA 23606 > > | (757) 269-7634 <tel:%28757%29%20269-7634> > > +----------------------------------------------- > > > > > > > > > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > <mailto:Lustre-discuss at lists.lustre.org> > http://lists.lustre.org/mailman/listinfo/lustre-discuss > > > > > -- > cliffw > Support Guy > WhamCloud, Inc. > www.whamcloud.com <http://www.whamcloud.com> > > > ------------------------------------------------------------------------ > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >