Lustre discuss - Jul 2009 - How to remove ghost files when running on an LVM snapshot of the MDT?

Dear list,

my MDS is running on an LVM snapshot of the MDT. Some files
have been deleted after the snapshot was taken, so these
appear as ''ghost'' files now on the clients and I''m
trying
to get rid of them.

 From what I understood, this requires e2fsck and lfsck. So
I ran e2fsck on the MDT and all the OSTs in order to get the
db files for lfsck, however:

-->
[root at mds tmp]# lfsck -v --mdsdb mdt --ostdb ost.sdb ost.sdc ost.sdd 
ost.sde /lustre/backup-MDT/dev/mdtgroup/MDT
lfsck 1.40.11.sun1 (17-June-2008)
MDSDB: mdt
OSTDB[0]: ost.sdb
OSTDB[1]: ost.sdc
OSTDB[2]: ost.sdd
OSTDB[3]: ost.sde
MOUNTPOINT: /lustre/backup-MDT/dev/mdtgroup/MDT
error: can''t get lov name.: Inappropriate ioctl for device (25)
<--

Anything obvious I am doing wrong here?

Is that the correct approach after all?

TIA,
  Arne
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 6380 bytes
Desc: S/MIME Cryptographic Signature
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090709/1a2c799b/attachment-0001.bin

On Jul 09, 2009  15:44 +0200, Arne Wiebalck wrote:
> my MDS is running on an LVM snapshot of the MDT. Some files
> have been deleted after the snapshot was taken, so these
> appear as ''ghost'' files now on the clients and
I''m trying
> to get rid of them.
Can you please explain more about what you are doing?  Is the
MDS itself running on an old snapshot?
> From what I understood, this requires e2fsck and lfsck. So
Well, you can just delete those files.  lfsck isn''t strictly
necessary.
> I ran e2fsck on the MDT and all the OSTs in order to get the
> db files for lfsck, however:
>
> -->
> [root at mds tmp]# lfsck -v --mdsdb mdt --ostdb ost.sdb ost.sdc ost.sdd  
> ost.sde /lustre/backup-MDT/dev/mdtgroup/MDT
Is this actually where your Lustre filesystem is mounted on the client?


Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

Andreas Dilger wrote:
> On Jul 09, 2009  15:44 +0200, Arne Wiebalck wrote:
>> my MDS is running on an LVM snapshot of the MDT. Some files
>> have been deleted after the snapshot was taken, so these
>> appear as ''ghost'' files now on the clients and
I''m trying
>> to get rid of them.
> 
> Can you please explain more about what you are doing?  Is the
> MDS itself running on an old snapshot?
Sure:

I set up a Lustre FS and created 2 files fileA and fileB.
Then I took an LVM snapshot of the MDT.
I deleted file B and created a third file fileC.

On the client I saw now files fileA and fileC.

Next, I unmounted the MDT and mounted the snapshot.

On the client, I saw now entries for fileA and fileB, as
expected, but fileB is not accessible anymore as it has
been deleted meanwhile.

The goal was now to get the system into a consistent state
again.

I thought I had to do

e2fsck on the MDT and all the OSTs
lfsck on the MDT

If that''s the wrong approach, please let me know.

This is all to understand what has to be done in order to use
an LVM snapshot of the MDT in case that''s necessary.
>> From what I understood, this requires e2fsck and lfsck. So
> 
> Well, you can just delete those files.  lfsck isn''t strictly
> necessary.
On the client you mean? That doesn''t work:

-->
[root]# rm fileB
rm: cannot lstat `fileB'': No such file or directory
<--
> 
>> I ran e2fsck on the MDT and all the OSTs in order to get the
>> db files for lfsck, however:
>>
>> -->
>> [root at mds tmp]# lfsck -v --mdsdb mdt --ostdb ost.sdb ost.sdc ost.sdd
>> ost.sde /lustre/backup-MDT/dev/mdtgroup/MDT
> 
> Is this actually where your Lustre filesystem is mounted on the client?
No, that''s on the MDT. I thought I need to correct the MDT by running
lfsck in order to get it in sync again with what is on the OSTs, no?

Thanks,
  Arne

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 6380 bytes
Desc: S/MIME Cryptographic Signature
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090710/76455f9c/attachment.bin

Andreas Dilger wrote:
> On Jul 09, 2009  15:44 +0200, Arne Wiebalck wrote:
>> my MDS is running on an LVM snapshot of the MDT. Some files
>> have been deleted after the snapshot was taken, so these
>> appear as ''ghost'' files now on the clients and
I''m trying
>> to get rid of them.
> 
> Can you please explain more about what you are doing?  Is the
> MDS itself running on an old snapshot?
Yes. And as you raise it: how do I get back from using a snapshot
for the MDT (to which I switched in case I had a problem with the 
original logical volume) to a snapshot''able MDT device?

Cheers,
  Arne


-- 
-----------------------------------------------------------
Dr. Arne Wiebalck                   Office 31-1-011
IT Department                       T +41 (0)22 767 8614
FIO-LA/TSI                          F +41 (0)22 766 8829
CERN, Geneva                        M arne.wiebalck at cern.ch
-----------------------------------------------------------

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 6380 bytes
Desc: S/MIME Cryptographic Signature
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090710/3cfdce76/attachment.bin

Arne Wiebalck wrote:
> Andreas Dilger wrote:
>> On Jul 09, 2009  15:44 +0200, Arne Wiebalck wrote:
>>> my MDS is running on an LVM snapshot of the MDT. Some files
>>> have been deleted after the snapshot was taken, so these
>>> appear as ''ghost'' files now on the clients and
I''m trying
>>> to get rid of them.
>> Can you please explain more about what you are doing?  Is the
>> MDS itself running on an old snapshot?
> 
> Yes. And as you raise it: how do I get back from using a snapshot
> for the MDT (to which I switched in case I had a problem with the 
> original logical volume) to a snapshot''able MDT device?
> 
Forget that question, I got confused. You don''t wanna run from a 
snapshot. Rather, you want the snapshot to have a consistent copy
from which you can do a file level backup. Remains the questions
of removing the ghost files.

Cheers,
  Arne




-- 
-----------------------------------------------------------
Dr. Arne Wiebalck                   Office 31-1-011
IT Department                       T +41 (0)22 767 8614
FIO-LA/TSI                          F +41 (0)22 766 8829
CERN, Geneva                        M arne.wiebalck at cern.ch
-----------------------------------------------------------

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 6380 bytes
Desc: S/MIME Cryptographic Signature
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090710/66fa8369/attachment.bin

On Jul 10, 2009  09:19 +0200, Arne Wiebalck wrote:
> I set up a Lustre FS and created 2 files fileA and fileB.
> Then I took an LVM snapshot of the MDT.
> I deleted file B and created a third file fileC.
>
> On the client I saw now files fileA and fileC.
>
> Next, I unmounted the MDT and mounted the snapshot.
>
> On the client, I saw now entries for fileA and fileB, as
> expected, but fileB is not accessible anymore as it has
> been deleted meanwhile.
>
> The goal was now to get the system into a consistent state
> again.
>
> I thought I had to do
>
> e2fsck on the MDT and all the OSTs
> lfsck on the MDT
>
> If that''s the wrong approach, please let me know.
This should definitely work.
> This is all to understand what has to be done in order to use
> an LVM snapshot of the MDT in case that''s necessary.
>
>>> From what I understood, this requires e2fsck and lfsck. So
>>
>> Well, you can just delete those files.  lfsck isn''t strictly
>> necessary.
>
> On the client you mean? That doesn''t work:
>
> -->
> [root]# rm fileB
> rm: cannot lstat `fileB'': No such file or directory
> <--
Try using "unlink fileB".  The GNU "rm" likes to stat every
file before
unlinking it, and the "stat" fails because the OST objects are
missing.
>>> I ran e2fsck on the MDT and all the OSTs in order to get the
>>> db files for lfsck, however:
>>>
>>> -->
>>> [root at mds tmp]# lfsck -v --mdsdb mdt --ostdb ost.sdb ost.sdc
ost.sdd
>>> ost.sde /lustre/backup-MDT/dev/mdtgroup/MDT
>>
>> Is this actually where your Lustre filesystem is mounted on the client?
>
> No, that''s on the MDT. I thought I need to correct the MDT by
running
> lfsck in order to get it in sync again with what is on the OSTs, no?
You need to mount the client filesystem on the MDT in this case.


Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.



Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

Andreas Dilger wrote:
>> -->
>> [root]# rm fileB
>> rm: cannot lstat `fileB'': No such file or directory
>> <--
> 
> Try using "unlink fileB".  The GNU "rm" likes to stat
every file before
> unlinking it, and the "stat" fails because the OST objects are
missing.
Yes, that works, thanks.
> You need to mount the client filesystem on the MDT in this case.
I did that now. When running with -c, lfsck created an empty object for
my fileB.

Is there a way to get rid of the missing files? (Users may prefer that
their files are cleanly gone rather than there are ghost files or empty
objects.)

As far as I understood, there is no way to sync the MDT from the OSTs, 
right?

One more thing: from what I see, the clients need to remount Lustre when
the MDT is restored, even if it''s the same MDS mounting it. Is that 
correct or will the client recover after a while (and I was just to 
impatient or did something wrong)?

Thanks,
  Arne

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 6380 bytes
Desc: S/MIME Cryptographic Signature
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090714/e664bb82/attachment.bin