Lustre discuss - Feb 2011 - Migrating MDT volume to a new location

Is there a recommended way to migrate an MDT (MGS is separate) volume 
from one location to another on the same server?  This uses iSCSI volumes.

Lustre 1.8.4

Thanks,
bob

On 2011-02-02, at 10:15, Bob Ball wrote:
> Is there a recommended way to migrate an MDT (MGS is separate) volume 
> from one location to another on the same server?  This uses iSCSI volumes.
This is all discussed in the manual.

The easiest way, if the device sizes are the same, is to use "dd" to
just copy the raw data to the new device.  While a lot more data is copied,
there is no seeking and it can go a lot faster.

If the new device is larger, it is possible to resize the new filesystem, though
this isn''t something that is tested.  You will have the old device for
backup, in case something goes wrong.

If you decide to use something like tar, be sure to use the "--sparse"
option.  That said, it may take a long time due to the way that the size is
stored on the MDS.

Cheers, Andreas
--
Andreas Dilger 
Principal Engineer
Whamcloud, Inc.

On 2 Feb 2011, at 17:15, Bob Ball wrote:
> Is there a recommended way to migrate an MDT (MGS is separate) volume 
> from one location to another on the same server?  This uses iSCSI volumes.
> 
> Lustre 1.8.4
If using LVM it''s possible to do this entirely at the LVM layer without
interrupting the filesystem, I wouldn''t recommend it without first
taking a backup though.

The commands used for this are pvcreate, vgextend, pvmove, vgreduce and
pvremove.

If using tar make sure you do a test run first, we''ve found several
bugs with tar recently that lead to errors.

https://bugzilla.redhat.com/show_bug.cgi?id=656834

Ashley.

On 02.02.2011 18:15, Bob Ball wrote:
> Is there a recommended way to migrate an MDT (MGS is separate) volume
> from one location to another on the same server?  This uses iSCSI volumes.
>
> Lustre 1.8.4
>
We''ll try the copy (DRBD) + resize variant soon. I''ve tried
that with a
backup copy of the MDT up to the stage where I had mounted the MDT disk 
and then would need a network connection to the MGS - with the 
production system running, I couldn''t test that.

Copying via tar never succeeded in our system - I think because of 
limits in tar.
But I tried to copy via rsync: takes weeks! Neither the amount of bytes 
to be transfered (45 GB) nor the network bandwidth (10 Gb) is an issues, 
but we have 110 mio inodes in use, so rsync probably takes its time to 
go through all these (sparse) files. Consequently, there can be no 
lengthy first copy from a backup and a quick final sync while Lustre has 
been taken down - that step would last just as long.

Regards,
Thomas

Bob Ball wrote:
> Is there a recommended way to migrate an MDT (MGS is separate) volume 
> from one location to another on the same server?  This uses iSCSI volumes.
> 
> Lustre 1.8.4
We have recently migrated our MDT to an new volume to change the inode 
size. We used tar and getfattr/setfattr for this closely following the 
MDT backup and restore procedure from the manual. We did take our file 
system down completely for over a week over Christmas to do this, though 
  it could have possibly been quicker if we had not taken time off as 
well. We also did very careful checking of everything before bringing up 
the new MDT. This was on a file system with at that time about 90M files.

Be sure to use a tar version that has an efficient way to detect files 
that are completely sparse. We found that the version on RHEL5 did not 
have this and taking backups took very long, I think there is a thread 
on the list about this which has a link to a fixed tar version for RHEL5 
which we used in the end. Also, we did not use any --posix or --xattr 
option for tar as we found that this is broken and creates bogus files. 
We used plain ''tar --sparse''.

If you don''t have to change any mkfs options (like inode size in our 
case), I suspect any block based copy mechanism like dd or pvmove is the 
better option, possibly followed by a resize.

Good luck,
Frederik

-- 
Frederik Ferner
Computer Systems Administrator		phone: +44 1235 77 8624
Diamond Light Source Ltd.		mob:   +44 7917 08 5110
(Apologies in advance for the lines below. Some bits are a legal
requirement and I have no control over them.)

-- 
This e-mail and any attachments may contain confidential, copyright and or
privileged material, and are for the use of the intended addressee only. If you
are not the intended addressee or an authorised recipient of the addressee
please notify us of receipt by returning the e-mail and do not use, copy,
retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not
necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments
are free from viruses and we cannot accept liability for any damage which you
may sustain as a result of software viruses which may be transmitted in or with
the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and
Wales with its registered office at Diamond House, Harwell Science and
Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

We used dd.  It took about 12 hours.  After the dd, we did an e2fsck on 
the new volume, remounted it as the MDT, and Lustre happily began 
serving files again.

Thanks to everyone for their help.

bob

On 2/3/2011 12:21 PM, Frederik Ferner wrote:
> Bob Ball wrote:
>> Is there a recommended way to migrate an MDT (MGS is separate) volume
>> from one location to another on the same server?  This uses iSCSI
volumes.
>>
>> Lustre 1.8.4
> We have recently migrated our MDT to an new volume to change the inode
> size. We used tar and getfattr/setfattr for this closely following the
> MDT backup and restore procedure from the manual. We did take our file
> system down completely for over a week over Christmas to do this, though
>    it could have possibly been quicker if we had not taken time off as
> well. We also did very careful checking of everything before bringing up
> the new MDT. This was on a file system with at that time about 90M files.
>
> Be sure to use a tar version that has an efficient way to detect files
> that are completely sparse. We found that the version on RHEL5 did not
> have this and taking backups took very long, I think there is a thread
> on the list about this which has a link to a fixed tar version for RHEL5
> which we used in the end. Also, we did not use any --posix or --xattr
> option for tar as we found that this is broken and creates bogus files.
> We used plain ''tar --sparse''.
>
> If you don''t have to change any mkfs options (like inode size in
our
> case), I suspect any block based copy mechanism like dd or pvmove is the
> better option, possibly followed by a resize.
>
> Good luck,
> Frederik
>

So your new MDT volume had the same size as the old one? You didn''t
have to resize?

Would be interesting to know if someone has experience with resize2fs applied to
a MDT ...

Cheers,
Thomas

On 02/03/2011 06:35 PM, Bob Ball wrote:
> We used dd.  It took about 12 hours.  After the dd, we did an e2fsck on 
> the new volume, remounted it as the MDT, and Lustre happily began 
> serving files again.
> 
> Thanks to everyone for their help.
> 
> bob
> 
> On 2/3/2011 12:21 PM, Frederik Ferner wrote:
>> Bob Ball wrote:
>>> Is there a recommended way to migrate an MDT (MGS is separate)
volume
>>> from one location to another on the same server?  This uses iSCSI
volumes.
>>>
>>> Lustre 1.8.4
>> We have recently migrated our MDT to an new volume to change the inode
>> size. We used tar and getfattr/setfattr for this closely following the
>> MDT backup and restore procedure from the manual. We did take our file
>> system down completely for over a week over Christmas to do this,
though
>>    it could have possibly been quicker if we had not taken time off as
>> well. We also did very careful checking of everything before bringing
up
>> the new MDT. This was on a file system with at that time about 90M
files.
>>
>> Be sure to use a tar version that has an efficient way to detect files
>> that are completely sparse. We found that the version on RHEL5 did not
>> have this and taking backups took very long, I think there is a thread
>> on the list about this which has a link to a fixed tar version for
RHEL5
>> which we used in the end. Also, we did not use any --posix or --xattr
>> option for tar as we found that this is broken and creates bogus files.
>> We used plain ''tar --sparse''.
>>
>> If you don''t have to change any mkfs options (like inode size
in our
>> case), I suspect any block based copy mechanism like dd or pvmove is
the
>> better option, possibly followed by a resize.
>>
>> Good luck,
>> Frederik
>>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
-- 
--------------------------------------------------------------------
Thomas Roth
Department: Informationstechnologie
Location: SB3 1.262
Phone: +49-6159-71 1453  Fax: +49-6159-71 2986

GSI Helmholtzzentrum f?r Schwerionenforschung GmbH
Planckstra?e 1
64291 Darmstadt
www.gsi.de

Gesellschaft mit beschr?nkter Haftung
Sitz der Gesellschaft: Darmstadt
Handelsregister: Amtsgericht Darmstadt, HRB 1528

Gesch?ftsf?hrung: Professor Dr. Dr. h.c. Horst St?cker,
Dr. Hartmut Eickhoff

Vorsitzende des Aufsichtsrates: Dr. Beatrix Vierkorn-Rudolph
Stellvertreter: Ministerialdirigent Dr. Rolf Bernhardt

Luster User Guide section 15.2

"If hardware replacement is the reason for the backup or if a spare 
storage device is
available, it is possible to do a raw copy of the MDS or OST from one 
block device to
the other, as long as the new device is at least as large as the 
original device. To do
this, run:
dd if=/dev/{original} of=/dev/{new} bs=1M"

so, the new volume was slightly bigger than the old.

bob

On 2/3/2011 6:10 PM, Thomas Roth wrote:
> So your new MDT volume had the same size as the old one? You
didn''t have to resize?
>
> Would be interesting to know if someone has experience with resize2fs
applied to a MDT ...
>
> Cheers,
> Thomas
>
> On 02/03/2011 06:35 PM, Bob Ball wrote:
>> We used dd.  It took about 12 hours.  After the dd, we did an e2fsck on
>> the new volume, remounted it as the MDT, and Lustre happily began
>> serving files again.
>>
>> Thanks to everyone for their help.
>>
>> bob
>>
>> On 2/3/2011 12:21 PM, Frederik Ferner wrote:
>>> Bob Ball wrote:
>>>> Is there a recommended way to migrate an MDT (MGS is separate)
volume
>>>> from one location to another on the same server?  This uses
iSCSI volumes.
>>>>
>>>> Lustre 1.8.4
>>> We have recently migrated our MDT to an new volume to change the
inode
>>> size. We used tar and getfattr/setfattr for this closely following
the
>>> MDT backup and restore procedure from the manual. We did take our
file
>>> system down completely for over a week over Christmas to do this,
though
>>>     it could have possibly been quicker if we had not taken time
off as
>>> well. We also did very careful checking of everything before
bringing up
>>> the new MDT. This was on a file system with at that time about 90M
files.
>>>
>>> Be sure to use a tar version that has an efficient way to detect
files
>>> that are completely sparse. We found that the version on RHEL5 did
not
>>> have this and taking backups took very long, I think there is a
thread
>>> on the list about this which has a link to a fixed tar version for
RHEL5
>>> which we used in the end. Also, we did not use any --posix or
--xattr
>>> option for tar as we found that this is broken and creates bogus
files.
>>> We used plain ''tar --sparse''.
>>>
>>> If you don''t have to change any mkfs options (like inode
size in our
>>> case), I suspect any block based copy mechanism like dd or pvmove
is the
>>> better option, possibly followed by a resize.
>>>
>>> Good luck,
>>> Frederik
>>>
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss

I have done resize on our MDT quite a few times without any problem. I did the
following to resize the MDT:

e2fsck -f /dev/mdtvg/mdt
lvextend --size +40G /dev/mdtvg/mdt
e2fsck -f /dev/mdtvg/mdt
resize2fs /dev/mdtvg/mdt
e2fsck -f /dev/mdtvg/mdt

Nirmal