peter.chiu at stfc.ac.uk
2011-Dec-19 14:45 UTC
[Lustre-discuss] How to re-activate "inactive device" on client - again
Not sure what happened to my post on Friday, but it seems to got tagged under another one. Anyway, here is a resend - my apologies if you have read it already. I shall be grateful for any advice. Regards, Peter Chiu STFC Rutherford Appleton Laboratory RAL Space Department Building R25, Room 2.02 Harwell Oxford Didcot OXON OX11 0QX UK Dear all, We have an odd problem on a lustre client in that a couple of OST elements have become "inactive". However, lctl dl shows all devices are in "UP" state. The "UP" status is also echoed on the mds as well as on another client - please see below. I have tried rebooting this client, but with no joy. I have also searched around the discussion list, and can find a few others had experienced similar problems before. But unfortunatley none of them has reported back with a good recovery procedure, other than a complete reformatting of the lustre storage. So can I trouble you experts to give me some clue as to how to recover from this? Many thanks. Regards, Peter PS. Client: SLES 11 with 2.6.32.29 kernel, Lustre 1.8.5 ================================================================================== Client Capuchin: capuchin:~ # lfs df -h UUID bytes Used Available Use% Mounted on ceda3-MDT0000_UUID 350.0G 689.2M 329.3G 0% /disks/ceda3[MDT:0] ceda3-OST0000_UUID 7.2T 6.8T 6.5G 94% /disks/ceda3[OST:0] ceda3-OST0001_UUID 7.2T 6.8T 6.5G 94% /disks/ceda3[OST:1] ceda3-OST0002_UUID : inactive device ceda3-OST0003_UUID : inactive device ceda3-OST0004_UUID 7.2T 6.8T 6.6G 94% /disks/ceda3[OST:4] ceda3-OST0005_UUID 7.2T 6.8T 6.7G 94% /disks/ceda3[OST:5] ceda3-OST0006_UUID 7.2T 6.8T 6.2G 94% /disks/ceda3[OST:6] filesystem summary: 35.8T 34.0T 32.5G 94% /disks/ceda3 capuchin:~ # lctl dl 0 UP mgc MGC130.246.191.64 at tcp<mailto:MGC130.246.191.64 at tcp> be6c0da2-275f-c59d-fa3a-c792134a0d9d 5 1 UP lov ceda3-clilov-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 4 2 UP mdc ceda3-MDT0000-mdc-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 5 3 UP osc ceda3-OST0000-osc-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 5 4 UP osc ceda3-OST0001-osc-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 5 5 UP osc ceda3-OST0002-osc-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 5 6 UP osc ceda3-OST0003-osc-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 5 7 UP osc ceda3-OST0004-osc-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 5 8 UP osc ceda3-OST0005-osc-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 5 9 UP osc ceda3-OST0006-osc-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 5 capuchin:~ # Client2 cmip-dn1: [root at aspre script]# ssh cmip-dn1 lfs df -h UUID bytes Used Available Use% Mounted on ceda3-MDT0000_UUID 350.0G 689.2M 329.3G 0% /disks/ceda3[MDT:0] ceda3-OST0000_UUID 7.2T 6.8T 6.5G 94% /disks/ceda3[OST:0] ceda3-OST0001_UUID 7.2T 6.8T 6.5G 94% /disks/ceda3[OST:1] ceda3-OST0002_UUID 7.2T 451.6M 6.8T 0% /disks/ceda3[OST:2] ceda3-OST0003_UUID 7.2T 451.8M 6.8T 0% /disks/ceda3[OST:3] ceda3-OST0004_UUID 7.2T 6.8T 6.6G 94% /disks/ceda3[OST:4] ceda3-OST0005_UUID 7.2T 6.8T 6.7G 94% /disks/ceda3[OST:5] ceda3-OST0006_UUID 7.2T 6.8T 6.2G 94% /disks/ceda3[OST:6] filesystem summary: 50.1T 34.0T 13.6T 67% /disks/ceda3 Mds: [root at mds02 ~]# lctl dl 0 UP mgc MGC130.246.191.64 at tcp<mailto:MGC130.246.191.64 at tcp> 8aa29420-11f1-8ca3-a361-ce5135a09be2 5 1 UP mdt MDS MDS_uuid 3 2 UP lov ceda3-mdtlov ceda3-mdtlov_UUID 4 3 UP mds ceda3-MDT0000 ceda3-MDT0000_UUID 9 4 UP osc ceda3-OST0000-osc ceda3-mdtlov_UUID 5 5 UP osc ceda3-OST0001-osc ceda3-mdtlov_UUID 5 6 UP osc ceda3-OST0002-osc ceda3-mdtlov_UUID 5 7 UP osc ceda3-OST0003-osc ceda3-mdtlov_UUID 5 8 UP osc ceda3-OST0004-osc ceda3-mdtlov_UUID 5 9 UP osc ceda3-OST0005-osc ceda3-mdtlov_UUID 5 10 UP osc ceda3-OST0006-osc ceda3-mdtlov_UUID 5 11 UP lov ceda4-mdtlov ceda4-mdtlov_UUID 4 12 UP mds ceda4-MDT0000 ceda4-MDT0000_UUID 9 13 UP osc ceda4-OST0000-osc ceda4-mdtlov_UUID 5 14 UP osc ceda4-OST0001-osc ceda4-mdtlov_UUID 5 15 UP osc ceda4-OST0002-osc ceda4-mdtlov_UUID 5 16 UP osc ceda4-OST0003-osc ceda4-mdtlov_UUID 5 17 UP osc ceda4-OST0004-osc ceda4-mdtlov_UUID 5 18 UP osc ceda4-OST0005-osc ceda4-mdtlov_UUID 5 [root at mds02 ~]# -- Scanned by iCritical. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20111219/0f734fd8/attachment.html