Hi all, I want to move a MDT from one server to another. After studying some mails concerning MDT backup, I''ve just tried (successfully, it seems) to do that on a small test system with rsync: - Stop Lustre, umount all servers. - Format a suitable disk partition on the new hardware, using the same mkfs-options as for the original MDT. - Mount the original MDT: mount -t ldiskfs /dev/sdb1 /mnt - Mount the target partition: mount -t ldiskfs -O ext_attr /dev/sdb1 /mnt - Copy the data: rsync -Xav oldserver:/mnt/ newserver:/mnt - Umount partitions, restart MGS - Mount new MDT This procedure was described by Jim Garlick on this list. You might note that I used the mount option "-O ext_attr" only on the target machine: my mistake perhaps, but no visible problems. In fact, I haven''t found this option mentioned in any man page or on the net. Nevertheless, my mount command did not complain about it. So I wonder whether it is necessary at all - I seem to have extracted the attributes from the old MDT all right, without this mount option - ? My main question is whether this is a correct procedure for MDT backups, or rather copies. I''m investigating this because our production MDT seems to have a number of problems. In particular the underlying file system is in bad shape, fsck correcting a large number of ext3-errors, incorrect inodes and so forth. We want to verify that it is not a hardware issue - bit-flipping RAID controller, silent "memory corruption", whatever. We have a DRBD-mirror of this MDT running, but of course DRBD just reproduces all errors on the mirror. Copying from one ldiskfs to another should avoid that? The traditional backup method of getting the EAs and tar-ing the MDT doesn''t finish in finite time. It did before, and the filesystem has since grown by a mere 40GB of data, so it shouldn''t take that much longer - certainly another indication that there is something wrong. Of course I have yet to see whether "rsync -Xav" does much better on the full system ;-) Hm, not sure whether this all makes sense. The system runs Debian Etch, kernel 2.6.22, Lustre 1.6.7.1 Regards, Thomas
On Jul 15, 2009 18:35 +0200, Thomas Roth wrote:> I want to move a MDT from one server to another. After studying some > mails concerning MDT backup, I''ve just tried (successfully, it seems) to > do that on a small test system with rsync: > > - Stop Lustre, umount all servers. > - Format a suitable disk partition on the new hardware, using the same > mkfs-options as for the original MDT. > - Mount the original MDT: mount -t ldiskfs /dev/sdb1 /mnt > - Mount the target partition: mount -t ldiskfs -O ext_attr > /dev/sdb1 /mnt > - Copy the data: rsync -Xav oldserver:/mnt/ newserver:/mnt > - Umount partitions, restart MGS > - Mount new MDT > > This procedure was described by Jim Garlick on this list. You might note > that I used the mount option "-O ext_attr" only on the target machine: > my mistake perhaps, but no visible problems. In fact, I haven''t found > this option mentioned in any man page or on the net. Nevertheless, my > mount command did not complain about it. So I wonder whether it is > necessary at all - I seem to have extracted the attributes from the old > MDT all right, without this mount option - ?If you have verified that the file data is actually present, this should work correctly. In particular, the critical Lustre information is in the "trusted.lov" xattr, so you need to ensure that is present. The MDS will "work" without this xattr, but it will assume all of the files have no data.> I''m investigating this because our production MDT seems to have a number > of problems. In particular the underlying file system is in bad shape, > fsck correcting a large number of ext3-errors, incorrect inodes and so > forth. We want to verify that it is not a hardware issue - bit-flipping > RAID controller, silent "memory corruption", whatever. We have a > DRBD-mirror of this MDT running, but of course DRBD just reproduces all > errors on the mirror. Copying from one ldiskfs to another should avoid > that? > > The traditional backup method of getting the EAs and tar-ing the MDT > doesn''t finish in finite time. It did before, and the filesystem has > since grown by a mere 40GB of data, so it shouldn''t take that much > longer - certainly another indication that there is something wrong. > Of course I have yet to see whether "rsync -Xav" does much better on the > full system ;-) > > The system runs Debian Etch, kernel 2.6.22, Lustre 1.6.7.1Direct MDT backup has a problem in 1.6.7.1 due to the addition of the file size on the MDT inodes. If you do "--sparse" backups this should avoid the slowdown. You could also try the "dump" program, which can avoid reading the data from sparse files entirely. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.
Andreas Dilger wrote:> On Jul 15, 2009 18:35 +0200, Thomas Roth wrote: >>... >> The traditional backup method of getting the EAs and tar-ing the MDT >> doesn''t finish in finite time. It did before, and the filesystem has >> since grown by a mere 40GB of data, so it shouldn''t take that much >> longer - certainly another indication that there is something wrong. >> Of course I have yet to see whether "rsync -Xav" does much better on the >> full system ;-) >> >> The system runs Debian Etch, kernel 2.6.22, Lustre 1.6.7.1 > > Direct MDT backup has a problem in 1.6.7.1 due to the addition of the > file size on the MDT inodes. If you do "--sparse" backups this should > avoid the slowdown. You could also try the "dump" program, which can > avoid reading the data from sparse files entirely.Is this behavior changed in 1.6.7.2? But the size information is still there in the inodes, isn''t it? Anyhow our backup script already runs tar with the "--sparse" option. Regards, Thomas
For the record, I should add that I had forgotten one step which proves to be important, also mentioned before on this list: After copying with rsync, I had to cd /srv/mdt; rm CATALOGS OBJECTS/* on the new MDT partition. Otherwise the OSTs are kicked out on remount with "error looking up logfile ...: rc -2" and "OST0000_UUID sync failed -2, deactivating" Regards, Thomas Thomas Roth wrote:> Hi all, > > I want to move a MDT from one server to another. After studying some > mails concerning MDT backup, I''ve just tried (successfully, it seems) to > do that on a small test system with rsync: > > - Stop Lustre, umount all servers. > - Format a suitable disk partition on the new hardware, using the same > mkfs-options as for the original MDT. > - Mount the original MDT: mount -t ldiskfs /dev/sdb1 /mnt > - Mount the target partition: mount -t ldiskfs -O ext_attr > /dev/sdb1 /mnt > - Copy the data: rsync -Xav oldserver:/mnt/ newserver:/mnt > - Umount partitions, restart MGS > - Mount new MDT > > This procedure was described by Jim Garlick on this list. You might note > that I used the mount option "-O ext_attr" only on the target machine: > my mistake perhaps, but no visible problems. In fact, I haven''t found > this option mentioned in any man page or on the net. Nevertheless, my > mount command did not complain about it. So I wonder whether it is > necessary at all - I seem to have extracted the attributes from the old > MDT all right, without this mount option - ? > > My main question is whether this is a correct procedure for MDT backups, > or rather copies. > > I''m investigating this because our production MDT seems to have a number > of problems. In particular the underlying file system is in bad shape, > fsck correcting a large number of ext3-errors, incorrect inodes and so > forth. We want to verify that it is not a hardware issue - bit-flipping > RAID controller, silent "memory corruption", whatever. We have a > DRBD-mirror of this MDT running, but of course DRBD just reproduces all > errors on the mirror. Copying from one ldiskfs to another should avoid > that? > > The traditional backup method of getting the EAs and tar-ing the MDT > doesn''t finish in finite time. It did before, and the filesystem has > since grown by a mere 40GB of data, so it shouldn''t take that much > longer - certainly another indication that there is something wrong. > Of course I have yet to see whether "rsync -Xav" does much better on the > full system ;-) > > Hm, not sure whether this all makes sense. > > The system runs Debian Etch, kernel 2.6.22, Lustre 1.6.7.1 > > Regards, > Thomas > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss-- -------------------------------------------------------------------- Thomas Roth Department: Informationstechnologie Location: SB3 1.262 Phone: +49-6159-71 1453 Fax: +49-6159-71 2986 GSI Helmholtzzentrum f?r Schwerionenforschung GmbH Planckstra?e 1 D-64291 Darmstadt www.gsi.de Gesellschaft mit beschr?nkter Haftung Sitz der Gesellschaft: Darmstadt Handelsregister: Amtsgericht Darmstadt, HRB 1528 Gesch?ftsf?hrer: Professor Dr. Horst St?cker Vorsitzende des Aufsichtsrates: Dr. Beatrix Vierkorn-Rudolph, Stellvertreter: Ministerialdirigent Dr. Rolf Bernhardt