OK, so our old MDS had hardware issues so I configured a new MGS / MDS on a VM (this is a backup lustre filesystem and I wanted to separate the MGS / MDS from OSS of the previous), and then did this: For example: mount -t ldiskfs /dev/old /mnt/ost_old mount -t ldiskfs /dev/new /mnt/ost_new rsync -aSv /mnt/ost_old/ /mnt/ost_new # note trailing slash on ost_old/ If you are unable to connect both sets of disk to the same computer, use rsync to copy over the network using rsh (or ssh with -e ssh): rsync -aSvz /mnt/ost_old/ new_ost_node:/mnt/ost_new Use the same procedure for the MDS, with one additional step: cd /mnt/mds_old; getfattr -R -e base64 -d . > /tmp/mdsea; \<copy all MDS files as above>; cd /mnt/mds_new; setfattr \--restore=/tmp/mdsea Run on all nodes (clients and OSS''s & MDS / MGS) to change MGS (where mgs is the name of the server and /dev/sdb1 is the name of the unmounted oat) tunefs.lustre --mgsnode=mgs at tcp0 /dev/sdb1 tunefs.lustre --erase-param --mgsnode=192.168.16.3 at tcp --writeconf /dev/sdc1 After copying with rsync, I had to cd /srv/mdt; rm CATALOGS OBJECTS/* on the new MDT partition. I also upgraded from 1.8.8 to 2. I managed to mount the Lustre filesystem and if I do lfs df -h, I get: NB> I deactivated those two OSTs below. [root at mgs data]# lfs df -h UUID bytes Used Available Use% Mounted on AC3-MDT0000_UUID 37.5G 499.5M 34.5G 1% /data[MDT:0] AC3-OST0000_UUID 16.4T 2.2T 13.3T 14% /data[OST:0] AC3-OST0001_UUID 16.4T 1.8T 13.7T 12% /data[OST:1] AC3-OST0002_UUID 6.4T 6.0T 49.2G 99% /data[OST:2] AC3-OST0003_UUID 6.4T 6.1T 912.9M 100% /data[OST:3] AC3-OST0004_UUID 4.3T 4.1T 17.2G 100% /data[OST:4] AC3-OST0005_UUID 1.9T 1.8T 29.0M 100% /data[OST:5] AC3-OST0006_UUID 1.9T 1.5T 282.1G 85% /data[OST:6] AC3-OST0007_UUID 1.9T 1.8T 434.3M 100% /data[OST:7] AC3-OST0008_UUID 1.9T 1.8T 12.9G 99% /data[OST:8] OST0009 : Resource temporarily unavailable AC3-OST000a_UUID 2.0T 1.9T 971.7M 100% /data[OST:10] AC3-OST000b_UUID 2.0T 1.9T 537.4M 100% /data[OST:11] AC3-OST000c_UUID 2.0T 1.9T 1.4G 100% /data[OST:12] AC3-OST000d_UUID 2.0T 1.6T 234.5G 88% /data[OST:13] AC3-OST000e_UUID 2.0T 1.9T 3.1G 100% /data[OST:14] OST000f : Resource temporarily unavailable AC3-OST0010_UUID 2.0T 1.9T 3.6G 100% /data[OST:16] AC3-OST0011_UUID 2.0T 1.6T 248.3G 87% /data[OST:17] AC3-OST0012_UUID 2.0T 1.9T 3.0G 100% /data[OST:18] AC3-OST0013_UUID 2.0T 1.9T 1.4G 100% /data[OST:19] AC3-OST0014_UUID 2.0T 1.9T 2.8G 100% /data[OST:20] AC3-OST0015_UUID 2.0T 1.9T 2.7G 100% /data[OST:21] AC3-OST0016_UUID 2.0T 1.4T 473.3G 75% /data[OST:22] AC3-OST0017_UUID 1.2T 917.9G 230.1G 80% /data[OST:23] AC3-OST0018_UUID 2.0T 1.3T 616.7G 68% /data[OST:24] AC3-OST0019_UUID 2.0T 1.2T 671.9G 65% /data[OST:25] AC3-OST001a_UUID 2.0T 1.1T 769.4G 59% /data[OST:26] AC3-OST001b_UUID 2.0T 1.2T 642.2G 66% /data[OST:27] AC3-OST001c_UUID 2.0T 1.2T 633.0G 67% /data[OST:28] filesystem summary: 92.1T 55.6T 31.9T 64% /data However, there are hardly any files or folders there - most of them have disappeared, but they are clearly on the OSTs (according to the above command). Is there any way I can reindex the OST''S / MDS to recognise the files and folders properly? When I do lctl dl I get: [root at mgs data]# lctl dl 1 UP mgc MGC192.168.16.3 at tcp 285232ea-c153-187d-1033-47c23b432296 5 2 UP lov AC3-MDT0000-mdtlov AC3-MDT0000-mdtlov_UUID 4 3 UP mdt AC3-MDT0000 AC3-MDT0000_UUID 5 4 UP mds mdd_obd-AC3-MDT0000 mdd_obd_uuid-AC3-MDT0000 3 5 UP osc AC3-OST0002-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5 6 UP osc AC3-OST0003-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5 7 UP osc AC3-OST0004-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5 8 UP osc AC3-OST000a-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5 9 UP osc AC3-OST000b-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5 10 UP osc AC3-OST000c-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5 11 UP osc AC3-OST000d-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5 12 UP osc AC3-OST000e-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5 13 UP osc AC3-OST0011-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5 14 UP osc AC3-OST0013-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5 15 UP osc AC3-OST0012-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5 16 UP osc AC3-OST0014-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5 17 UP osc AC3-OST0015-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5 18 UP osc AC3-OST0016-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5 19 UP osc AC3-OST0017-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5 20 UP osc AC3-OST0001-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5 21 UP osc AC3-OST0000-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5 22 UP osc AC3-OST0010-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5 23 UP osc AC3-OST0005-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5 24 UP osc AC3-OST0006-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5 25 UP osc AC3-OST0008-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5 26 UP osc AC3-OST0007-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5 27 UP osc AC3-OST0018-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5 28 UP osc AC3-OST0019-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5 29 UP osc AC3-OST001a-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5 30 UP osc AC3-OST001b-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5 31 UP osc AC3-OST001c-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5 32 UP lov AC3-clilov-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 4 33 UP lmv AC3-clilmv-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 4 34 UP mdc AC3-MDT0000-mdc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5 35 UP osc AC3-OST0002-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5 36 UP osc AC3-OST0003-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5 37 UP osc AC3-OST0004-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5 38 UP osc AC3-OST000a-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5 39 UP osc AC3-OST000b-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5 40 UP osc AC3-OST000c-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5 41 UP osc AC3-OST000d-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5 42 UP osc AC3-OST000e-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5 43 UP osc AC3-OST0011-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5 44 UP osc AC3-OST0013-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5 45 UP osc AC3-OST0012-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5 46 UP osc AC3-OST0014-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5 47 UP osc AC3-OST0015-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5 48 UP osc AC3-OST0016-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5 49 UP osc AC3-OST0017-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5 50 UP osc AC3-OST0001-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5 51 UP osc AC3-OST0000-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5 52 UP osc AC3-OST0010-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5 53 UP osc AC3-OST0005-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5 54 UP osc AC3-OST0006-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5 55 UP osc AC3-OST0008-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5 56 UP osc AC3-OST0007-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5 57 UP osc AC3-OST0018-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5 58 UP osc AC3-OST0019-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5 59 UP osc AC3-OST001a-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5 60 UP osc AC3-OST001b-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5 61 UP osc AC3-OST001c-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5 SO it all looks fine and /var/log/messages and dmesg seem to look OK. Thanks in advance! Alastair Ferguson IT Manager Capital Markets CRC Limited (CMCRC) Telephone: +61 2 8088 4222 Mobile: +61 424 235 159 Fax: +61 2 8088 4201 www.cmcrc.com Capital Markets CRC Ltd - Confidential Communication The information contained in this e-mail is confidential. It is intended for the addressee only. If you receive this e-mail by mistake please promptly inform us by reply e-mail and then delete the e-mail and destroy any printed copy. You must not disclose or use in any way the information in the e-mail. There is no warranty that this e-mail is error or virus free. It may be a private communication, and if so, does not represent the views of the CMCRC and its associates. If it is a private communication, care should be taken in opening it to ensure that undue offence is not given. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20130212/a1d7e466/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 13489 bytes Desc: not available Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20130212/a1d7e466/attachment-0001.png
On 2013-02-14, at 9:46, "Alastair Ferguson" <aferguson at cmcrc.com<mailto:aferguson at cmcrc.com>> wrote: OK, so our old MDS had hardware issues so I configured a new MGS / MDS on a VM (this is a backup lustre filesystem and I wanted to separate the MGS / MDS from OSS of the previous), and then did this When you write that your files are missing, do you mean that they do not appear with "ls -l", or that they are present but contain no data? I also upgraded from 1.8.8 to 2. I managed to mount the Lustre filesystem Just as a general sysadmin rule, doing too many things at once makes it difficult to figure out what went wrong. Was it the data migration? Splitting off the MGS? Upgrading to 2.x (you don''t report which version)? However, there are hardly any files or folders there - most of them have disappeared, but they are clearly on the OSTs (according to the above command). Is there any way I can reindex the OST''S / MDS to recognise the files and folders properly? Do you still have the old MDT storage? Have your mounted the new MDT as type ldiskfs to see if your files are there? They would be located below the "ROOT/" directory inside the MDT filesystem. Cheers, Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 13489 bytes Desc: image.png Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20130216/cd36b0f2/attachment.png
Thanks for the email Andreas - I actually sorted this one out. I recopied the MDS database and it worked. I think the unzipping process didn''t finish properly the first time. I now have a fully functional 95TB backup filesystem. Alastair Ferguson IT Manager Capital Markets CRC Limited (CMCRC) Telephone: +61 2 8088 4222 Mobile: +61 424 235 159 Fax: +61 2 8088 4201 www.cmcrc.com Capital Markets CRC Ltd - Confidential Communication The information contained in this e-mail is confidential. It is intended for the addressee only. If you receive this e-mail by mistake please promptly inform us by reply e-mail and then delete the e-mail and destroy any printed copy. You must not disclose or use in any way the information in the e-mail. There is no warranty that this e-mail is error or virus free. It may be a private communication, and if so, does not represent the views of the CMCRC and its associates. If it is a private communication, care should be taken in opening it to ensure that undue offence is not given. On 16/02/2013, at 7:11 PM, "Dilger, Andreas" <andreas.dilger at intel.com> wrote:> On 2013-02-14, at 9:46, "Alastair Ferguson" <aferguson at cmcrc.com<mailto:aferguson at cmcrc.com>> wrote: > > OK, so our old MDS had hardware issues so I configured a new MGS / MDS on a VM (this is a backup lustre filesystem and I wanted to separate the MGS / MDS from OSS of the previous), and then did this > > When you write that your files are missing, do you mean that they do not appear with "ls -l", or that they are present but contain no data? > > I also upgraded from 1.8.8 to 2. I managed to mount the Lustre filesystem > > Just as a general sysadmin rule, doing too many things at once makes it difficult to figure out what went wrong. > Was it the data migration? Splitting off the MGS? Upgrading to 2.x (you don''t report which version)? > > However, there are hardly any files or folders there - most of them have disappeared, but they are clearly on the OSTs (according to the above command). Is there any way I can reindex the OST''S / MDS to recognise the files and folders properly? > > Do you still have the old MDT storage? Have your mounted the new MDT as type ldiskfs to see if your files are there? They would be located below the "ROOT/" directory inside the MDT filesystem. > > Cheers, Andreas > <image.png>-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20130217/3923abd0/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 13489 bytes Desc: not available Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20130217/3923abd0/attachment-0001.png