Lustre discuss - Mar 2010 - problems restoring from MDT backup (test file system)

Hi List,

on my test file system I''m currently trying to verify once more that I 
can restore the MDT from my backups but I''m running into some problems.

The MDS is running RHEL5.3 and Lustre 1.6.7.2.

I use the following procedure to backup the MDT:

create LVM snapshot from MDT device
mount LVM snapshot as ldiskfs
extract EAs with getfattr
tar up the whole MDT tree from ldiskfs using ''/bin/tar czSf 
/tmp/${BACKUP_FILE} --acls --numeric-owner .''

In previous tests the restore appeared to work fine even though I was 
not quite sure about the ACLs as I had not recorded them before 
destroying the MDT at that time.

This time I have some problems with the tar file when I want to verify 
the list of files before destroying the MDT:

<snip>
tar tizf test_MDT_Backup.tar.gz
....
./ROOT/tmp/frederik/cs04r-sc-com02-04/
./ROOT/tmp/frederik/cs04r-sc-com02-04/iozone.DUMMY.47
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now
</snip>

Doing the same with older backup files or backups from our production 
file system don''t show this error but for the current test file system,
I can reproduce this easily with any new backup file that I create. The 
list of files that I see when creating a new tar file (adding -v to the 
tar options) from the test file system does include many files after the 
last one in the output above, but the new file has exactly the same 
problem at the same place.

Has anyone seen something like this before?

What could we try to recover the data from the old backup? Or is this 
most likely impossible?

Could this indicate a problem on the files system?

I''ve not tried to run fsck on the MDT as I''d like to extract
the files
from the tar files if possible independent of fixing the existing file 
system as an exercise.

Any suggestions are welcome.

Frederik
-- 
Frederik Ferner
Computer Systems Administrator		phone: +44 1235 77 8624
Diamond Light Source Ltd.		mob:   +44 7917 08 5110
(Apologies in advance for the lines below. Some bits are a legal
requirement and I have no control over them.)

On Thu, 2010-03-04 at 11:21 +0000, Frederik Ferner wrote:
> 
> tar tizf test_MDT_Backup.tar.gz
> ....
> ./ROOT/tmp/frederik/cs04r-sc-com02-04/
> ./ROOT/tmp/frederik/cs04r-sc-com02-04/iozone.DUMMY.47
> tar: Unexpected EOF in archive
> tar: Error is not recoverable: exiting now
> </snip>
Looks to me like either your tar executable is broken or your archive is
broken.  A typical process of elimination should help you discover which
is the case.

b.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100304/9b4d56f2/attachment-0001.bin

Brian,

thanks for your reply.

Brian J. Murrell wrote:
> On Thu, 2010-03-04 at 11:21 +0000, Frederik Ferner wrote: 
>> tar tizf test_MDT_Backup.tar.gz
>> ....
>> ./ROOT/tmp/frederik/cs04r-sc-com02-04/
>> ./ROOT/tmp/frederik/cs04r-sc-com02-04/iozone.DUMMY.47
>> tar: Unexpected EOF in archive
>> tar: Error is not recoverable: exiting now
>> </snip>
> 
> Looks to me like either your tar executable is broken or your archive is
> broken.  A typical process of elimination should help you discover which
> is the case.
It certainly looks like it''s the tar archive that is broken. I get the 
same when I copy it over to a different machine. Unless is the tar 
executable that is broken so that it creates the broken archive as every 
time I create a new archive it seems to be broken at the same place.

Other tar files created on the same machine don''t have that problem,
but
I''ll try creating a new archive with a new executable.

Thanks,
Frederik
-- 
Frederik Ferner
Computer Systems Administrator		phone: +44 1235 77 8624
Diamond Light Source Ltd.		mob:   +44 7917 08 5110
(Apologies in advance for the lines below. Some bits are a legal
requirement and I have no control over them.)

On 2010-03-04, at 05:46, Frederik Ferner wrote:
> Brian J. Murrell wrote:
>> On Thu, 2010-03-04 at 11:21 +0000, Frederik Ferner wrote:
>>> tar tizf test_MDT_Backup.tar.gz
>>> ....
>>> ./ROOT/tmp/frederik/cs04r-sc-com02-04/
>>> ./ROOT/tmp/frederik/cs04r-sc-com02-04/iozone.DUMMY.47
>>> tar: Unexpected EOF in archive
>>> tar: Error is not recoverable: exiting now
>>> </snip>
>>
>> Looks to me like either your tar executable is broken or your  
>> archive is
>> broken.  A typical process of elimination should help you discover  
>> which
>> is the case.
>
> It certainly looks like it''s the tar archive that is broken. I get
the
> same when I copy it over to a different machine. Unless is the tar
> executable that is broken so that it creates the broken archive as  
> every
> time I create a new archive it seems to be broken at the same place.
>
> Other tar files created on the same machine don''t have that
problem,
> but
> I''ll try creating a new archive with a new executable.

Make sure you use "--sparse" so that tar isn''t mistakenly
creating
huge archives full of zeroes.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.