Hi,
We run four-node Lustre 2.3, and I needed to both change hardware
under MGS/MDS and reassign an OSS ip. Just the same, I added a brand
new 10GE network to the system, which was the reason for MDS hardware
change.
I ran tunefs.lustre --writeconf as per chapter 14.4 in Lustre Manual,
and everything mounts fine. Log regeneration apparently works, since
it seems to do something, but exceedingly slowly. Disks show all but
no activity, CPU utilization is zero across the board, and memory
should be no issue. I believe it works, but currently it seems the
1,5*10^9 files (some 55 TiB of data) won''t be indexed in a week. My
boss isn''t happy when I can''t even predict how long this will
take, or
even say for sure that it really works.
Two questions: is there a way to know how fast it is progressing
and/or where it is at, or even that it really works, and is there a
way to speed up whatever is slowing it down? Seems all diagnostic
/proc entries have been removed from 2.3. I have tried mounting the
Lustre partitions with -o nobarrier (yes, I know it''s dangerous, but
I''d really need to speed things up) but I don''t know if that
does
anything at all.
We run Centos 6.x in Lustre servers, where Lustre has been installed
from rpm''s from Whamcloud/Intel build bot, and Ubuntu 10.04 in clients
with hand compiled kernel and Lustre. One MGC/MGS with twelve 15k-RPM
SAS disks in RAID-10 as MDT that is all but empty, and six variously
build RAID-6''s in SAS-attached shelves in three OSS''s.
ATdhvaannkcse for any help,
--
Olli Lounela
IT specialist and administrator
DNA sequencing and genomics
Institute of Biotechnology
University of Helsinki
On 2013/10/17 5:34 AM, "Olli Lounela" <olli.lounela-pxSi+dnQzZMxHbG02/KK1g@public.gmane.org> wrote:>Hi, > >We run four-node Lustre 2.3, and I needed to both change hardware >under MGS/MDS and reassign an OSS ip. Just the same, I added a brand >new 10GE network to the system, which was the reason for MDS hardware >change.Note that in Lustre 2.4 there is a "lctl replace_nids" command that allows you to change the NIDs without running --writeconf. That doesn''t help you now, but possibly in the future.>I ran tunefs.lustre --writeconf as per chapter 14.4 in Lustre Manual, >and everything mounts fine. Log regeneration apparently works, since >it seems to do something, but exceedingly slowly. Disks show all but >no activity, CPU utilization is zero across the board, and memory >should be no issue. I believe it works, but currently it seems the >1,5*10^9 files (some 55 TiB of data) won''t be indexed in a week. My >boss isn''t happy when I can''t even predict how long this will take, or >even say for sure that it really works.The --writeconf information is at most a few kB and should only take seconds to complete. What "reindexing" operation are you referencing? It should be possible to mount the filesystem immediately (MGS first, then MDS and OSSes) after running --writeconf. You didn''t really explain what is preventing you from using the filesystem, since you said it mounted properly?>Two questions: is there a way to know how fast it is progressing >and/or where it is at, or even that it really works, and is there a >way to speed up whatever is slowing it down? Seems all diagnostic >/proc entries have been removed from 2.3. I have tried mounting the >Lustre partitions with -o nobarrier (yes, I know it''s dangerous, but >I''d really need to speed things up) but I don''t know if that does >anything at all.I doubt that the "-o nobarrier" is helping you much.>We run Centos 6.x in Lustre servers, where Lustre has been installed >from rpm''s from Whamcloud/Intel build bot, and Ubuntu 10.04 in clients >with hand compiled kernel and Lustre. One MGC/MGS with twelve 15k-RPM >SAS disks in RAID-10 as MDT that is all but empty, and six variously >build RAID-6''s in SAS-attached shelves in three OSS''s. > >ATdhvaannkcse for any help, > >-- > Olli Lounela > IT specialist and administrator > DNA sequencing and genomics > Institute of Biotechnology > University of Helsinki > >_______________________________________________ >Lustre-discuss mailing list >Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org >http://lists.lustre.org/mailman/listinfo/lustre-discuss >Cheers, Andreas -- Andreas Dilger Lustre Software Architect Intel High Performance Data Division
Wolfgang Baudler
2013-Nov-20 19:37 UTC
OST state not reported correctly with Lustre 2.4.1?
If I deactivate an OST on the MDS like lctl --device 56 deactivate I would expect "lctl dl" on the MDS to show it as "IN" instead of "UP". This is not the case anymore with 2.4.1. However it shows correctly as INACTIVE when I do a cat /proc/fs/lustre/lov/vegas-MDT0000-mdtlov/target_obd: 89: vegas-OST0059_UUID INACTIVE lctl dl: 56 UP osp vegas-OST0059-osc-MDT0000 vegas-MDT0000-mdtlov_UUID 5 Is this exepected behaviour? Wolfgang
Frederik Ferner
2013-Nov-20 19:48 UTC
Re: OST state not reported correctly with Lustre 2.4.1?
Hi Wolfgang, On 20/11/13 19:37, Wolfgang Baudler wrote:> If I deactivate an OST on the MDS like > > lctl --device 56 deactivate > > I would expect "lctl dl" on the MDS to show it as "IN" instead of "UP". > > This is not the case anymore with 2.4.1. However it shows correctly as > INACTIVE when I do a > > cat /proc/fs/lustre/lov/vegas-MDT0000-mdtlov/target_obd: > > 89: vegas-OST0059_UUID INACTIVE > > lctl dl: > 56 UP osp vegas-OST0059-osc-MDT0000 vegas-MDT0000-mdtlov_UUID 5 > > Is this exepected behaviour?I don''t know if it is expected, but it seems you are not alone. I have something similar on one file system where the servers have been upgraded to 2.4.1 but the clients are still on 1.8.9. After mounting the file system lctl dl reports all devices as up, lfs df shows some OSTs as inactive and target_obd also shows them as inactive. Running ''lctl --device XXX activate'' does not change this. [bnh65367@cs04r-sc-serv-07 ~]$ lctl dl |grep play01 91 UP lov play01-clilov-ffff810076ae2000 9186608e-d432-283c-0e6e-47b800427d3e 4 92 UP mdc play01-MDT0000-mdc-ffff810076ae2000 9186608e-d432-283c-0e6e-47b800427d3e 5 93 UP osc play01-OST0000-osc-ffff810076ae2000 9186608e-d432-283c-0e6e-47b800427d3e 5 94 UP osc play01-OST0001-osc-ffff810076ae2000 9186608e-d432-283c-0e6e-47b800427d3e 5 95 UP osc play01-OST0002-osc-ffff810076ae2000 9186608e-d432-283c-0e6e-47b800427d3e 5 96 UP osc play01-OST0003-osc-ffff810076ae2000 9186608e-d432-283c-0e6e-47b800427d3e 5 97 UP osc play01-OST0004-osc-ffff810076ae2000 9186608e-d432-283c-0e6e-47b800427d3e 5 98 UP osc play01-OST0005-osc-ffff810076ae2000 9186608e-d432-283c-0e6e-47b800427d3e 5 [bnh65367@cs04r-sc-serv-07 ~]$ lfs df /mnt/play01 UUID 1K-blocks Used Available Use% Mounted on play01-MDT0000_UUID 78636320 3502948 75133372 4% /mnt/play01[MDT:0] play01-OST0000_UUID 7691221300 4506865920 3184355380 59% /mnt/play01[OST:0] play01-OST0001_UUID 7691221300 3765688064 3925533236 49% /mnt/play01[OST:1] play01-OST0002_UUID : inactive device play01-OST0003_UUID : inactive device play01-OST0004_UUID : inactive device play01-OST0005_UUID : inactive device filesystem summary: 15382442600 8272553984 7109888616 54% /mnt/play01 [bnh65367@cs04r-sc-serv-07 ~]$ cat /proc/fs/lustre/lov/play01-clilov-ffff810076ae2000/target_obd 0: play01-OST0000_UUID ACTIVE 1: play01-OST0001_UUID ACTIVE 2: play01-OST0002_UUID INACTIVE 3: play01-OST0003_UUID INACTIVE 4: play01-OST0004_UUID INACTIVE 5: play01-OST0005_UUID INACTIVE [bnh65367@cs04r-sc-serv-07 ~]$ Note that play01-OST0000, play01-OST0002 and play-OST0004 are on the same OSS, play01-OST0001, play01-OST0003 and play-OST0005 are on a different OSS but again all on the same. So I think we can rule out network problems here, it is also repeatable over reboots. This does not happen on the same clients for any of the other file systems where the servers are still running Lustre 1.8. Kind regards, Frederik -- Frederik Ferner Senior Computer Systems Administrator phone: +44 1235 77 8624 Diamond Light Source Ltd. mob: +44 7917 08 5110 (Apologies in advance for the lines below. Some bits are a legal requirement and I have no control over them.) -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom