We have just implemented lustre at my university and we are trying to get used to some concepts. We have recreated the environment manytimes, but I would like to understand how to do some things. The wiki or the operations manual does not have these directions clearly spelled out. For example, I would like to know what is the correct way to remove remove (OST). The OSTs reside on a Linux LVM. I guess I can do, "unmount on all lustre clients" , on the MDS run "lctl dl" and then "lctl --device OSC_UID deactivate" and then removelv? Does that sound ok? Any ideas? TIA
On Tue, 2008-07-08 at 07:01 -0400, Mag Gam wrote:> We have just implemented lustre at my university and we are trying to > get used to some concepts. We have recreated the environment > manytimes, but I would like to understand how to do some things. The > wiki or the operations manual does not have these directions clearly > spelled out.This is described in the Lustre 1.6 manual under "Removing an OST". I don''t remember the section.> > For example, I would like to know what is the correct way to remove > remove (OST). The OSTs reside on a Linux LVM. I guess I can do, > "unmount on all lustre clients" , on the MDS run "lctl dl" and then > "lctl --device OSC_UID deactivate" and then removelv?Doing a "lctl --device OSC_UUID deactivate" on the MDS will only stop new objects from being allocated on that OST. To completely remove the OST you need to use "lctl conf_param lustre-OST0000.osc.active=0". If later you want to use this OST again you can set the "active" parameter to 1. Or if you wish you can remove this OST permanently. Don''t try to use the same OST index as the old OST, since this will confuse the MDT. Before removing the OST you should copy all the files which have objects on that OST. You can find such files using "lfs find --recursive --obd <OST_UUID> /mnt/lustre" Thanks, Kalpak> > > Does that sound ok? > > Any ideas? > > TIA > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss
See Section 4.2.3.1 Removing an OST: http://manual.lustre.org/manual/LustreManual16_HTML/LustreInstallation.html#50446385_pgfId-1286363 Kalpak Shah wrote:> On Tue, 2008-07-08 at 07:01 -0400, Mag Gam wrote: > >> We have just implemented lustre at my university and we are trying to >> get used to some concepts. We have recreated the environment >> manytimes, but I would like to understand how to do some things. The >> wiki or the operations manual does not have these directions clearly >> spelled out. >> > > This is described in the Lustre 1.6 manual under "Removing an OST". I > don''t remember the section. > > >> For example, I would like to know what is the correct way to remove >> remove (OST). The OSTs reside on a Linux LVM. I guess I can do, >> "unmount on all lustre clients" , on the MDS run "lctl dl" and then >> "lctl --device OSC_UID deactivate" and then removelv? >> > > Doing a "lctl --device OSC_UUID deactivate" on the MDS will only stop > new objects from being allocated on that OST. To completely remove the > OST you need to use "lctl conf_param lustre-OST0000.osc.active=0". If > later you want to use this OST again you can set the "active" parameter > to 1. Or if you wish you can remove this OST permanently. Don''t try to > use the same OST index as the old OST, since this will confuse the MDT. > > Before removing the OST you should copy all the files which have objects > on that OST. You can find such files using "lfs find --recursive --obd > <OST_UUID> /mnt/lustre" > > Thanks, > Kalpak > > >> Does that sound ok? >> >> Any ideas? >> >> TIA >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss >> > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >
Kalpak Shah wrote:> Doing a "lctl --device OSC_UUID deactivate" on the MDS will only stop > new objects from being allocated on that OST. To completely remove the > OST you need to use "lctl conf_param lustre-OST0000.osc.active=0". If >What''s the difference between lctl deactivate and ''completely remove'', is it permanent? What does happen, if I deactivate an OST on the client, connect another new one, create the same object ( from example a file, called test.txt in the same directory), then recover the old OST again? Thank you, tamas
On Tue, 2008-07-08 at 15:54 +0200, Papp Tam?s wrote:> Kalpak Shah wrote: > > Doing a "lctl --device OSC_UUID deactivate" on the MDS will only stop > > new objects from being allocated on that OST. To completely remove the > > OST you need to use "lctl conf_param lustre-OST0000.osc.active=0". If > > > What''s the difference between lctl deactivate and ''completely remove'', > is it permanent?"lctl deactivate" on the _MDS_ means that MDS will not allocate new objects on that OST. If you do "lctl deactivate" on the clients then they will return -EIO when accessing objects on this OST instead of waiting for recovery. When you do "lctl conf_param lustre-OST0000.osc.active=0" this means that when clients are mounted they will set the import related to that OST as inactive thereby not waiting for recovery from that OST. You may choose to use the physical volume for that OST for other purposes. Or you can again activate it using "lctl conf_param lustre-OST0000.osc.active=1".> > > What does happen, if I deactivate an OST on the client, connect another > new one, create the same object ( from example a file, called test.txt > in the same directory), then recover the old OST again? >When you add a new OST, it would get a new index (unless you set the --index property in mkfs.lustre) and hence there should not be any collision. Note that the entry in the MDS for test.txt will now point to the new OST and not the old one. The object for test.txt on the old OST will be a dangling object. Thanks, Kalpak> Thank you, > > tamas
Why can''t you completly remove it from the system instead of disabling it? On Tue, Jul 8, 2008 at 10:18 AM, Kalpak Shah <Kalpak.Shah at sun.com> wrote:> On Tue, 2008-07-08 at 15:54 +0200, Papp Tam?s wrote: >> Kalpak Shah wrote: >> > Doing a "lctl --device OSC_UUID deactivate" on the MDS will only stop >> > new objects from being allocated on that OST. To completely remove the >> > OST you need to use "lctl conf_param lustre-OST0000.osc.active=0". If >> > >> What''s the difference between lctl deactivate and ''completely remove'', >> is it permanent? > > "lctl deactivate" on the _MDS_ means that MDS will not allocate new > objects on that OST. > > If you do "lctl deactivate" on the clients then they will return -EIO > when accessing objects on this OST instead of waiting for recovery. > > When you do "lctl conf_param lustre-OST0000.osc.active=0" this means > that when clients are mounted they will set the import related to that > OST as inactive thereby not waiting for recovery from that OST. You may > choose to use the physical volume for that OST for other purposes. Or > you can again activate it using "lctl conf_param > lustre-OST0000.osc.active=1". > >> >> >> What does happen, if I deactivate an OST on the client, connect another >> new one, create the same object ( from example a file, called test.txt >> in the same directory), then recover the old OST again? >> > > When you add a new OST, it would get a new index (unless you set the > --index property in mkfs.lustre) and hence there should not be any > collision. Note that the entry in the MDS for test.txt will now point to > the new OST and not the old one. The object for test.txt on the old OST > will be a dangling object. > > Thanks, > Kalpak > >> Thank you, >> >> tamas > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >