I do an "lctl dl" and it shows "UP" in the first column for all OST''s... even though I''ve deactivated many disks. "iostat" shows the disks are still in use too. I''m trying to get rid of slow disks... what''s the right way to tell Lustre to quit using a disk? Thanks, Chris
On Mar 25, 2008 01:28 -0600, Chris Worley wrote:> I do an "lctl dl" and it shows "UP" in the first column for all > OST''s... even though I''ve deactivated many disks. "iostat" shows the > disks are still in use too.What does it mean when you say "deactivated many disks"?> I''m trying to get rid of slow disks... what''s the right way to tell > Lustre to quit using a disk?If you deactivate an OST on the MDS node it will stop allocating new files there, and if your files are deleted by some process (e.g. aging or jobs running) the inactive OSTs will eventually become empty. If you want to speed this up you should use "lfs find -o {OST}" to find files on that OST and copy them to a new file. If you deactivate OSTs on the client nodes and stop the OSTs then the clients will return IO errors for any files remaining on those disks. I doubt that is what you want. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.
On Tue, Mar 25, 2008 at 2:13 AM, Andreas Dilger <adilger at sun.com> wrote:> On Mar 25, 2008 01:28 -0600, Chris Worley wrote: > > I do an "lctl dl" and it shows "UP" in the first column for all > > OST''s... even though I''ve deactivated many disks. "iostat" shows the > > disks are still in use too. > > What does it mean when you say "deactivated many disks"? >To deactivate the disk, I use an incantation like: lctl --device ddnlfs-OST001f-osc deactivate ...but new files are still going there, and, if I''m reading it right, the disk is still "up" in Lustre: # lctl dl | grep 1f 36 UP osc ddnlfs-OST001f-osc ddnlfs-mdtlov_UUID 5> > > I''m trying to get rid of slow disks... what''s the right way to tell > > Lustre to quit using a disk? > > If you deactivate an OST on the MDS node it will stop allocating new > files thereFor now, that''s all I want to do... but new files are still going there. ... both a way to deactivate the disk and a way to know which are deactivated would be nice. Thanks, Chris>, and if your files are deleted by some process (e.g. aging > or jobs running) the inactive OSTs will eventually become empty. If > you want to speed this up you should use "lfs find -o {OST}" to find > files on that OST and copy them to a new file. > > If you deactivate OSTs on the client nodes and stop the OSTs then the > clients will return IO errors for any files remaining on those disks. > I doubt that is what you want. > > Cheers, Andreas > -- > Andreas Dilger > Sr. Staff Engineer, Lustre Group > Sun Microsystems of Canada, Inc. > >
On Mar 25, 2008 08:53 -0600, Chris Worley wrote:> On Tue, Mar 25, 2008 at 2:13 AM, Andreas Dilger <adilger at sun.com> wrote: > > On Mar 25, 2008 01:28 -0600, Chris Worley wrote: > > > I do an "lctl dl" and it shows "UP" in the first column for all > > > OST''s... even though I''ve deactivated many disks. "iostat" shows the > > > disks are still in use too. > > > > What does it mean when you say "deactivated many disks"? > > To deactivate the disk, I use an incantation like: > > lctl --device ddnlfs-OST001f-osc deactivateNote that "deactivate" only affects the node on which it is run. The normal place to do this is on the MDS. Note that you also mount the client filesystem on the MDS node you need to deactivate the MDS OSC connection, and not the client filesystem one:> ...but new files are still going there, and, if I''m reading it right, > the disk is still "up" in Lustre: > > # lctl dl | grep 1f > 36 UP osc ddnlfs-OST001f-osc ddnlfs-mdtlov_UUID 5This does look like you have the right device. Using "device_list" only shows which devices are configured. A deactivated device is still configured... The "UP" status is related to the configuration status and not the current connection state. Have a look at the file /proc/fs/lustre/ddnlfs-mdtlov/target_obd to see the device status. # lfs df UUID 1K-blocks Used Available Use% Mounted on mds-myth-0_UUID 9174328 678000 8496328 7% /myth[MDT:0] ost-myth-0_UUID 292223856 286837752 5386104 98% /myth[OST:0] ost-myth-1_UUID 94442984 92833972 1609012 98% /myth[OST:1] ost-myth-2_UUID 487388376 474792788 12595588 97% /myth[OST:2] ost-myth-3_UUID 487865304 472221312 15643992 96% /myth[OST:3] filesystem summary: 1361920520 1326685824 35234696 97% /myth # lctl --device %myth-OST0001-osc deactivate # cat /proc/fs/lustre/lov/myth-mdtlov/target_obd 0: ost-myth-0_UUID ACTIVE 1: ost-myth-1_UUID INACTIVE 2: ost-myth-2_UUID ACTIVE 3: ost-myth-3_UUID ACTIVE # lctl --device %myth-OST0001-osc recover> > > I''m trying to get rid of slow disks... what''s the right way to tell > > > Lustre to quit using a disk? > > > > If you deactivate an OST on the MDS node it will stop allocating new > > files there > > For now, that''s all I want to do... but new files are still going there. > > ... both a way to deactivate the disk and a way to know which are > deactivated would be nice.It was confusing when you say "deactivate the disk" because that could mean a great many things like deactivating a disk from a RAID set, or similar. An OST may reside on many disks (via hardware/software RAID, LVM, etc). What you are trying to do is the right process. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.
On Tue, Mar 25, 2008 at 10:39 AM, Andreas Dilger <adilger at sun.com> wrote:> On Mar 25, 2008 08:53 -0600, Chris Worley wrote: > > On Tue, Mar 25, 2008 at 2:13 AM, Andreas Dilger <adilger at sun.com> wrote: > > > On Mar 25, 2008 01:28 -0600, Chris Worley wrote: > > > > I do an "lctl dl" and it shows "UP" in the first column for all > > > > OST''s... even though I''ve deactivated many disks. "iostat" shows the > > > > disks are still in use too. > > > > > > What does it mean when you say "deactivated many disks"? > > > > To deactivate the disk, I use an incantation like: > > > > lctl --device ddnlfs-OST001f-osc deactivate > > Note that "deactivate" only affects the node on which it is run. > The normal place to do this is on the MDS.That''s what I do.> Note that you also > mount the client filesystem on the MDS node you need to deactivate > the MDS OSC connection, and not the client filesystem one:I''m not sure I understand the above? I think you''re saying that when deactivating, use the block device label with "-osc" appended. That I do too.> > > > ...but new files are still going there, and, if I''m reading it right, > > the disk is still "up" in Lustre: > > > > # lctl dl | grep 1f > > 36 UP osc ddnlfs-OST001f-osc ddnlfs-mdtlov_UUID 5 > > This does look like you have the right device. Using "device_list" > only shows which devices are configured. A deactivated device is > still configured... The "UP" status is related to the configuration > status and not the current connection state. Have a look at the file > /proc/fs/lustre/ddnlfs-mdtlov/target_obd to see the device status. >Ahh, that verifies what''s active/inactive: # cat /proc/fs/lustre/lov/ddnlfs-mdtlov/target_obd | grep " ACTIVE" | wc -l 48 # cat /proc/fs/lustre/lov/ddnlfs-mdtlov/target_obd | grep " INACTIVE" | wc -l 22> # lfs dfThis command returns nothing?> UUID 1K-blocks Used Available Use% Mounted on > mds-myth-0_UUID 9174328 678000 8496328 7% /myth[MDT:0] > ost-myth-0_UUID 292223856 286837752 5386104 98% /myth[OST:0] > ost-myth-1_UUID 94442984 92833972 1609012 98% /myth[OST:1] > ost-myth-2_UUID 487388376 474792788 12595588 97% /myth[OST:2] > ost-myth-3_UUID 487865304 472221312 15643992 96% /myth[OST:3] > > filesystem summary: 1361920520 1326685824 35234696 97% /myth > > # lctl --device %myth-OST0001-osc deactivate > # cat /proc/fs/lustre/lov/myth-mdtlov/target_obd > 0: ost-myth-0_UUID ACTIVE > 1: ost-myth-1_UUID INACTIVE > 2: ost-myth-2_UUID ACTIVE > 3: ost-myth-3_UUID ACTIVE > > # lctl --device %myth-OST0001-osc recover > > > > > > I''m trying to get rid of slow disks... what''s the right way to tell > > > > Lustre to quit using a disk? > > > > > > If you deactivate an OST on the MDS node it will stop allocating new > > > files there > > > > For now, that''s all I want to do... but new files are still going there. > > > > ... both a way to deactivate the disk and a way to know which are > > deactivated would be nice. > > It was confusing when you say "deactivate the disk" because that could > mean a great many things like deactivating a disk from a RAID set, or > similar. An OST may reside on many disks (via hardware/software RAID, > LVM, etc). > > What you are trying to do is the right process. >Thanks for all the help! I think I''ve got it now. Chris> > > Cheers, Andreas > -- > Andreas Dilger > Sr. Staff Engineer, Lustre Group > Sun Microsystems of Canada, Inc. > >
On Mar 25, 2008, at 11:22 AM, Chris Worley wrote:> On Tue, Mar 25, 2008 at 10:39 AM, Andreas Dilger <adilger at sun.com> > wrote: >> On Mar 25, 2008 08:53 -0600, Chris Worley wrote: >>> On Tue, Mar 25, 2008 at 2:13 AM, Andreas Dilger <adilger at sun.com> >>> wrote: >>>> On Mar 25, 2008 01:28 -0600, Chris Worley wrote: >>>>> I do an "lctl dl" and it shows "UP" in the first column for all >>>>> OST''s... even though I''ve deactivated many disks. "iostat" >>>>> shows the >>>>> disks are still in use too. >>>> >>>> What does it mean when you say "deactivated many disks"? >>> >>> To deactivate the disk, I use an incantation like: >>> >>> lctl --device ddnlfs-OST001f-osc deactivate >> >> Note that "deactivate" only affects the node on which it is run. >> The normal place to do this is on the MDS. > > That''s what I do.One thing Andreas did not mention is that when you deactivate a device (OST) on the MDS, you disable NEW file creation. You can still append to existing files. The device is not read-only. So you may still see traffic to those disks as users modify existing files.>> Note that you also >> mount the client filesystem on the MDS node you need to deactivate >> the MDS OSC connection, and not the client filesystem one: > > I''m not sure I understand the above? > > I think you''re saying that when deactivating, use the block device > label with "-osc" appended. That I do too. > >> >> >>> ...but new files are still going there, and, if I''m reading it >>> right, >>> the disk is still "up" in Lustre: >>> >>> # lctl dl | grep 1f >>> 36 UP osc ddnlfs-OST001f-osc ddnlfs-mdtlov_UUID 5 >> >> This does look like you have the right device. Using "device_list" >> only shows which devices are configured. A deactivated device is >> still configured... The "UP" status is related to the >> configuration >> status and not the current connection state. Have a look at the >> file >> /proc/fs/lustre/ddnlfs-mdtlov/target_obd to see the device status. >> > > Ahh, that verifies what''s active/inactive: > > # cat /proc/fs/lustre/lov/ddnlfs-mdtlov/target_obd | grep " ACTIVE" > | wc -l > 48 > # cat /proc/fs/lustre/lov/ddnlfs-mdtlov/target_obd | grep " > INACTIVE" | wc -l > 22 > >> # lfs df > > This command returns nothing?lfs df should be run on a client, not the MDS (unless the MDS is a client as well).>> UUID 1K-blocks Used Available Use% Mounted on >> mds-myth-0_UUID 9174328 678000 8496328 7% /myth >> [MDT:0] >> ost-myth-0_UUID 292223856 286837752 5386104 98% /myth >> [OST:0] >> ost-myth-1_UUID 94442984 92833972 1609012 98% /myth >> [OST:1] >> ost-myth-2_UUID 487388376 474792788 12595588 97% /myth >> [OST:2] >> ost-myth-3_UUID 487865304 472221312 15643992 96% /myth >> [OST:3] >> >> filesystem summary: 1361920520 1326685824 35234696 97% /myth >> >> # lctl --device %myth-OST0001-osc deactivate >> # cat /proc/fs/lustre/lov/myth-mdtlov/target_obd >> 0: ost-myth-0_UUID ACTIVE >> 1: ost-myth-1_UUID INACTIVE >> 2: ost-myth-2_UUID ACTIVE >> 3: ost-myth-3_UUID ACTIVE >> >> # lctl --device %myth-OST0001-osc recover >> >> >>>>> I''m trying to get rid of slow disks... what''s the right way to >>>>> tell >>>>> Lustre to quit using a disk? >>>> >>>> If you deactivate an OST on the MDS node it will stop >>>> allocating new >>>> files there >>> >>> For now, that''s all I want to do... but new files are still going >>> there. >>> >>> ... both a way to deactivate the disk and a way to know which are >>> deactivated would be nice. >> >> It was confusing when you say "deactivate the disk" because that >> could >> mean a great many things like deactivating a disk from a RAID >> set, or >> similar. An OST may reside on many disks (via hardware/software >> RAID, >> LVM, etc). >> >> What you are trying to do is the right process. >> > > Thanks for all the help! I think I''ve got it now. > > Chris >> >> >> Cheers, Andreas >> -- >> Andreas Dilger >> Sr. Staff Engineer, Lustre Group >> Sun Microsystems of Canada, Inc. >> >> > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss-Marc ---- D. Marc Stearman LC Lustre Administration Lead marc at llnl.gov 925.423.9670 Pager: 1.888.203.0641