seth stuttgart
2006-Jul-07 19:45 UTC
[Lustre-discuss] Is there a way to remove an OST and all of its files from an active Lustre filesystem?
I am testing out Lustre 1.5.9.1 and I want to remove an OST and all of its files from a an active lustre filesystem and replace it with a newly created OST. My stripe count is 1, so there is no "raid0" striping across different OST''s. The hard-disk on one of the OST''s has crashed, and I cannot recover the data from that OST. So, I want to remove the OST from the MDS/MGS and remove all references to files stored on that bad OST. After removing the bad OST, I want to install a new hard-disk into the server and format the server as a new OST. I am using mkfs.lustre to create OST''s. How can I go about removing an OST and all of it''s files from a running lustre filesystem? I have tried to use mkfs.reformat to replace the bad OST with a new OST at the same index: mkfs.lustre --fsname=filesystem --ost --mgsnode=mgsnode@tcp1 --reformat --index=1 --param="failover.mode=failout" /dev/sda However, when I try to start the Lustre filesystem after running mkfs.lustre, I get: The target service''s index is already in use. (/dev/sda) I have also tried running mkfs.lustre without "--reformat" and without "--index". However, this creates an OST at a new index and the lustre clients still think the old OST exists (it is listed as "INACTIVE"). When I do an ''ls -al'' on a lustre client for files that used to exist on the bad OST, I get: /fs# ls -al total 4 drwxr-xr-x 2 root root 4096 Jul 8 00:43 . drwxr-xr-x 5 root root 100 Jul 7 21:45 .. ?--------- ? ? ? ? ? 0 ?--------- ? ? ? ? ? foobar ?--------- ? ? ? ? ? testfile I''m trying to figure out how to quickly remove references to all of these nonexistent files. I was thinking that perhaps the MDS keeps a table of filename--OST pairings that is indexed by OST index. If not, I could recursively traverse the lustre filesystem on a Lustre client and run "lfs getstripe <FILENAME>", and then remove the file if "lfs getstripe" matches the bad OST. Is there a quicker way of doing this through lctl? That leaves the problem of removing the OST. No matter what I do, "/proc/fs/lustre/osc" still contains a reference to the bad OST. Any suggestions? Thank you for your help --- Seth Stuttgard "Act in haste and repent at leisure; Code too soon and debug forever."
Andreas Dilger
2006-Jul-07 20:33 UTC
[Lustre-discuss] Is there a way to remove an OST and all of its files from an active Lustre filesystem?
On Jul 07, 2006 19:45 -0600, seth stuttgart wrote:> When I do an ''ls -al'' on a lustre client for files that used to exist > on the bad OST, I get: > /fs# ls -al > total 4 > drwxr-xr-x 2 root root 4096 Jul 8 00:43 . > drwxr-xr-x 5 root root 100 Jul 7 21:45 .. > ?--------- ? ? ? ? ? 0 > ?--------- ? ? ? ? ? foobar > ?--------- ? ? ? ? ? testfile > > I''m trying to figure out how to quickly remove references to all of > these nonexistent files. I was thinking that perhaps the MDS keeps a > table of filename--OST pairings that is indexed by OST index. If not, > I could recursively traverse the lustre filesystem on a Lustre client > and run "lfs getstripe <FILENAME>", and then remove the file if "lfs > getstripe" matches the bad OST. Is there a quicker way of doing this > through lctl?You should use "lfs find --recursive --ost {OST_UUID} /fs" to find the files that reside on that OST. If you had done this before removing the old OST it would have been possible to move those files to other OSTs, like: # deactivate the OST on the MDS only: mds# lctl --device %{OSC_UUID} deactivate # find all of the files with objects on the OST # copy to a new file # replace the old file with the new one client# lfs find --recursive --ost {OST_uuid} --quiet /fs | while read F; do cp $F $F.tmp && mv $F.tmp $F done # should return no files on this OST now client# lfs find --recursive --ost {OST_uuid} --quiet /fs Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc.
Peter J. Braam
2006-Jul-07 21:59 UTC
[Lustre-discuss] Is there a way to remove an OST and all of itsfiles from an active Lustre filesystem?
Hi Seth, You can run "lfs find --uuid .... --recursive" to locate the files that were stored on the OST''s that you have removed; get rid of them by feeding them to "rm". At present you cannot remove the OST, so it will continue to show as inactive. Also, you''ll be left with a log file that has a record of all the deletions, because the MDS considers these incomplete (if the OST were to come back, it would remove the corresponding objects). You could locate this llog file and rm it out of the disk file system under the MDS if you are desperate to clean it up. In Lustre 1.8 we may have an option to do this a bit more cleanly. - peter - -----Original Message----- From: lustre-discuss-bounces@clusterfs.com [mailto:lustre-discuss-bounces@clusterfs.com] On Behalf Of seth stuttgart Sent: Friday, July 07, 2006 7:45 PM To: lustre-discuss@clusterfs.com Subject: [Lustre-discuss] Is there a way to remove an OST and all of itsfiles from an active Lustre filesystem? I am testing out Lustre 1.5.9.1 and I want to remove an OST and all of its files from a an active lustre filesystem and replace it with a newly created OST. My stripe count is 1, so there is no "raid0" striping across different OST''s. The hard-disk on one of the OST''s has crashed, and I cannot recover the data from that OST. So, I want to remove the OST from the MDS/MGS and remove all references to files stored on that bad OST. After removing the bad OST, I want to install a new hard-disk into the server and format the server as a new OST. I am using mkfs.lustre to create OST''s. How can I go about removing an OST and all of it''s files from a running lustre filesystem? I have tried to use mkfs.reformat to replace the bad OST with a new OST at the same index: mkfs.lustre --fsname=filesystem --ost --mgsnode=mgsnode@tcp1 --reformat --index=1 --param="failover.mode=failout" /dev/sda However, when I try to start the Lustre filesystem after running mkfs.lustre, I get: The target service''s index is already in use. (/dev/sda) I have also tried running mkfs.lustre without "--reformat" and without "--index". However, this creates an OST at a new index and the lustre clients still think the old OST exists (it is listed as "INACTIVE"). When I do an ''ls -al'' on a lustre client for files that used to exist on the bad OST, I get: /fs# ls -al total 4 drwxr-xr-x 2 root root 4096 Jul 8 00:43 . drwxr-xr-x 5 root root 100 Jul 7 21:45 .. ?--------- ? ? ? ? ? 0 ?--------- ? ? ? ? ? foobar ?--------- ? ? ? ? ? testfile I''m trying to figure out how to quickly remove references to all of these nonexistent files. I was thinking that perhaps the MDS keeps a table of filename--OST pairings that is indexed by OST index. If not, I could recursively traverse the lustre filesystem on a Lustre client and run "lfs getstripe <FILENAME>", and then remove the file if "lfs getstripe" matches the bad OST. Is there a quicker way of doing this through lctl? That leaves the problem of removing the OST. No matter what I do, "/proc/fs/lustre/osc" still contains a reference to the bad OST. Any suggestions? Thank you for your help --- Seth Stuttgard "Act in haste and repent at leisure; Code too soon and debug forever." _______________________________________________ Lustre-discuss mailing list Lustre-discuss@clusterfs.com https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
Andreas Dilger
2006-Jul-07 23:39 UTC
[Lustre-discuss] Is there a way to remove an OST and all of itsfiles from an active Lustre filesystem?
Andreas Dilger wrote:> On Jul 07, 2006 19:45 -0600, seth stuttgart wrote: > > I''m trying to figure out how to quickly remove references to all of > > these nonexistent files. I was thinking that perhaps the MDS keeps a > > table of filename--OST pairings that is indexed by OST index. If not, > > You should use "lfs find --recursive --ost {OST_UUID} /fs" to find theSorry - this should be "lfs find --recursive --obd {OST_UUID}" here and below (note --obd instead of --ost)...> files that reside on that OST. If you had done this before removing > the old OST it would have been possible to move those files to other > OSTs, like: > > # deactivate the OST on the MDS only: > mds# lctl --device %{OSC_UUID} deactivate > > # find all of the files with objects on the OST > # copy to a new file > # replace the old file with the new one > client# lfs find --recursive --ost {OST_uuid} --quiet /fs | while read > F; do > cp $F $F.tmp && mv $F.tmp $F > done > > # should return no files on this OST now > client# lfs find --recursive --ost {OST_uuid} --quiet /fsCheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc.
seth stuttgart
2006-Jul-08 08:24 UTC
[Lustre-discuss] Is there a way to remove an OST and all of itsfiles from an active Lustre filesystem?
Thank you for the help. "lfs find ..." works great for moving files to a new OST. Problem solved. One more question about the inactive OST. My understanding from the Lustre FAQ is that there is a limit of 512 OST''s per Lustre filesystem. Can I keep deactivating OST''s from the filesystem and adding new ones as long as there are no more than 512 active OST''s, or is the limit for 512 OST''s total (active and inactive)? If there''s no performance/resource problem with having inactive OST''s laying around, then I''m making much ado about nothing. If not, since I cannot remove the OST is there a way to replace the OST? I''ve tried mkfs.lustre with the index of the inactive OST. However, when I try to mount the new OST with the index of the inactive OST, dmesg (on the OST) reports: LustreError: 2027:0:(mgc_request.c:672:mgc_target_register()) register failed. rc=-98 LustreError: 2027:0:(obd_mount.c:932:server_register_target()) registration with the MGS failed (-98) LustreError: 2027:0:(obd_mount.c:1023:server_start_targets()) Required registration failed for fs0-OST0000: -98 LustreError: 2027:0:(obd_mount.c:1482:server_fill_super()) Unable to start targets: -98 LustreError: 2027:0:(mgc_request.c:147:config_log_find()) can''t get log fs0-OST0000 LustreError: 2027:0:(obd_mount.c:1313:server_put_super()) no obd fs0-OST0000 LustreError: 2027:0:(obd_mount.c:119:server_deregister_mount()) fs0-OST0000 not registered LDISKFS-fs: mballoc: 0 blocks 0 reqs (0 success) LDISKFS-fs: mballoc: 0 extents scanned, 0 goal hits, 0 2^N hits, 0 breaks LDISKFS-fs: mballoc: 0 generated and it took 0 Lustre: server umount fs0-OST0000 complete LustreError: 2027:0:(obd_mount.c:1826:lustre_fill_super()) Unable to mount Meanwhile, dmesg on the MDS reports: LustreError: Server fs0-OST0000 requested index 0, but that index is already in use LustreError: 1845:0:(mgs_llog.c:1336:mgs_write_log_target()) Can''t get index (-98) LustreError: 1845:0:(mgs_handler.c:422:mgs_handle_target_reg()) Failed to write fs0-OST0000 log (-98) LustreError: 1845:0:(mgs_handler.c:539:mgs_handle()) MGS handle cmd=253 rc=-98 Would it help if I removed "CONFIGS/fs0-OST0000" from the MDS disk file system before mounting the new OST? Thank you again. On 7/7/06, Andreas Dilger <adilger@clusterfs.com> wrote:> Andreas Dilger wrote: > > On Jul 07, 2006 19:45 -0600, seth stuttgart wrote: > > > I''m trying to figure out how to quickly remove references to all of > > > these nonexistent files. I was thinking that perhaps the MDS keeps a > > > table of filename--OST pairings that is indexed by OST index. If not, > > > > You should use "lfs find --recursive --ost {OST_UUID} /fs" to find the > > Sorry - this should be "lfs find --recursive --obd {OST_UUID}" here and > below (note --obd instead of --ost)... > > > files that reside on that OST. If you had done this before removing > > the old OST it would have been possible to move those files to other > > OSTs, like: > > > > # deactivate the OST on the MDS only: > > mds# lctl --device %{OSC_UUID} deactivate > > > > # find all of the files with objects on the OST > > # copy to a new file > > # replace the old file with the new one > > client# lfs find --recursive --ost {OST_uuid} --quiet /fs | while read > > F; do > > cp $F $F.tmp && mv $F.tmp $F > > done > > > > # should return no files on this OST now > > client# lfs find --recursive --ost {OST_uuid} --quiet /fs > > > Cheers, Andreas > -- > Andreas Dilger > Principal Software Engineer > Cluster File Systems, Inc. > >--- Seth Stuttgard "Act in haste and repent at leisure; Code too soon and debug forever."