Lustre discuss - Mar 2009 - Delete file entries from Metadata server

Hello everybody

We recently had a hardware defect on one of our storage arrays which  
broke the file system on this device. After running e2fsck on the file  
system, I was able to mount the device again as lustre file system.  
However, a lot of data was written into the lost+found. Now we have a  
problem, that the MDS however didn''t recognize that these files are  
not available anymore and so still lists them. But then they are not  
accessible which results in a I/O-Error.

Is there a possibility to resync the MDS with the actual data on this  
OST? How can I delete these wrong entires?

Thanks a lot of your help,
Reto Gantenbein

On Wed, 2009-03-11 at 19:33 +0100, Reto Gantenbein
wrote:
> Hello everybody
Hi.
> Is there a possibility to resync the MDS with the actual data on this  
> OST? How can I delete these wrong entires?
You want lfsck.  There is a manpage for it, but you probably want to
take a look at the ops manual before using it.

b.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090311/386ed2bb/attachment.bin

On Mar 11, 2009  19:33 +0100, Reto Gantenbein wrote:
> We recently had a hardware defect on one of our storage arrays which  
> broke the file system on this device. After running e2fsck on the file  
> system, I was able to mount the device again as lustre file system.  
> However, a lot of data was written into the lost+found. Now we have a  
> problem, that the MDS however didn''t recognize that these files
are
> not available anymore and so still lists them. But then they are not  
> accessible which results in a I/O-Error.
If the objects are in lost+found on the OST then I''d suggest first
to use the "ll_recover_lost_found_objs" on the filesystem (unmounted
from Lustre, mounted locally as "-t ldiskfs").  This will recover a
majority of your objects (=file data).

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

To do the ll_recover_lost_found_objs, can we have everything mounted
up? Or is it recommended we unmount everything?

Also what to do when we see ????? for the file attributes. Most likely
these files are gone, how can we remove them from the system all
together?

TIA



On Wed, Mar 11, 2009 at 11:59 PM, Andreas Dilger <adilger at sun.com>
wrote:
> On Mar 11, 2009  19:33 +0100, Reto Gantenbein wrote:
>> We recently had a hardware defect on one of our storage arrays which
>> broke the file system on this device. After running e2fsck on the file
>> system, I was able to mount the device again as lustre file system.
>> However, a lot of data was written into the lost+found. Now we have a
>> problem, that the MDS however didn''t recognize that these
files are
>> not available anymore and so still lists them. But then they are not
>> accessible which results in a I/O-Error.
>
> If the objects are in lost+found on the OST then I''d suggest first
> to use the "ll_recover_lost_found_objs" on the filesystem
(unmounted
> from Lustre, mounted locally as "-t ldiskfs").  This will recover
a
> majority of your objects (=file data).
>
> Cheers, Andreas
> --
> Andreas Dilger
> Sr. Staff Engineer, Lustre Group
> Sun Microsystems of Canada, Inc.
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>

Mag,

----- "Mag Gam" <magawake at gmail.com> wrote:
> To do the ll_recover_lost_found_objs, can we have everything mounted
> up? Or is it recommended we unmount everything?
You only need to unmount the "damaged" OST and remount it with
"-t ldiskfs".
 
> Also what to do when we see ????? for the file attributes. Most
> likely
> these files are gone, how can we remove them from the system all
> together?
You should still be able to delete them. This will delete the entry on the
MDS but the deletion of the missing object on the OST will just fail. You
should try recovering the lost+found objects first though.

Daire

> On Wed, Mar 11, 2009 at 11:59 PM, Andreas Dilger <adilger at sun.com>
> wrote:
> > On Mar 11, 2009  19:33 +0100, Reto Gantenbein wrote:
> >> We recently had a hardware defect on one of our storage arrays
> which
> >> broke the file system on this device. After running e2fsck on the
> file
> >> system, I was able to mount the device again as lustre file
> system.
> >> However, a lot of data was written into the lost+found. Now we
have
> a
> >> problem, that the MDS however didn''t recognize that these
files
> are
> >> not available anymore and so still lists them. But then they are
> not
> >> accessible which results in a I/O-Error.
> >
> > If the objects are in lost+found on the OST then I''d suggest
first
> > to use the "ll_recover_lost_found_objs" on the filesystem
> (unmounted
> > from Lustre, mounted locally as "-t ldiskfs").  This will
recover a
> > majority of your objects (=file data).
> >
> > Cheers, Andreas
> > --
> > Andreas Dilger
> > Sr. Staff Engineer, Lustre Group
> > Sun Microsystems of Canada, Inc.
> >
> > _______________________________________________
> > Lustre-discuss mailing list
> > Lustre-discuss at lists.lustre.org
> > http://lists.lustre.org/mailman/listinfo/lustre-discuss
> >
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss

Thankyou.

I will try to recover them. I would have to wait for the weekend to do
that because I can''t afford to down a OST during work hours....

TIA


On Wed, Mar 25, 2009 at 5:25 AM, Daire Byrne <Daire.Byrne at
framestore.com> wrote:
> Mag,
>
> ----- "Mag Gam" <magawake at gmail.com> wrote:
>
>> To do the ll_recover_lost_found_objs, can we have everything mounted
>> up? Or is it recommended we unmount everything?
>
> You only need to unmount the "damaged" OST and remount it with
"-t ldiskfs".
>
>> Also what to do when we see ????? for the file attributes. Most
>> likely
>> these files are gone, how can we remove them from the system all
>> together?
>
> You should still be able to delete them. This will delete the entry on the
> MDS but the deletion of the missing object on the OST will just fail. You
> should try recovering the lost+found objects first though.
>
> Daire
>
>
>> On Wed, Mar 11, 2009 at 11:59 PM, Andreas Dilger <adilger at
sun.com>
>> wrote:
>> > On Mar 11, 2009  19:33 +0100, Reto Gantenbein wrote:
>> >> We recently had a hardware defect on one of our storage arrays
>> which
>> >> broke the file system on this device. After running e2fsck on
the
>> file
>> >> system, I was able to mount the device again as lustre file
>> system.
>> >> However, a lot of data was written into the lost+found. Now we
have
>> a
>> >> problem, that the MDS however didn''t recognize that
these files
>> are
>> >> not available anymore and so still lists them. But then they
are
>> not
>> >> accessible which results in a I/O-Error.
>> >
>> > If the objects are in lost+found on the OST then I''d
suggest first
>> > to use the "ll_recover_lost_found_objs" on the
filesystem
>> (unmounted
>> > from Lustre, mounted locally as "-t ldiskfs").  This
will recover a
>> > majority of your objects (=file data).
>> >
>> > Cheers, Andreas
>> > --
>> > Andreas Dilger
>> > Sr. Staff Engineer, Lustre Group
>> > Sun Microsystems of Canada, Inc.
>> >
>> > _______________________________________________
>> > Lustre-discuss mailing list
>> > Lustre-discuss at lists.lustre.org
>> > http://lists.lustre.org/mailman/listinfo/lustre-discuss
>> >
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>

On Wed, 2009-03-25 at 07:56 -0400, Mag Gam wrote:
> Thankyou.
> 
> I will try to recover them. I would have to wait for the weekend to do
> that because I can''t afford to down a OST during work hours....
Two thing I would mention to this are:

     A. if your OSTs were LVM LVs, you could take a snapshot and at
        least get started on trying to find out where files should be
        moved when you can afford to take the OST down and
     B. you can always interrogate a live OST with debugfs (in read-only
        mode, which is the default mode IIRC)

Both of these methods would allow you to get a head start on the work
prior to needing the down time and should result in a shorter down-time.

Obviously using an LVM snapshot would be more efficient/effective than
using debugfs to dump files out of the filesystem for examination.
Although I have often wondered about a fuse-debugfs filesystem, but I
digress.  I guess my point is simply that creating Lustre targets on LVs
provides future flexibility even if it''s not clear today why you might
want to do that.

Cheers,
b.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090325/71bb6cda/attachment.bin

I thought I had read at some point that there is a performance hit  
associated with putting the OST on a LV; or is that just the case when  
there are snapshots in play?

Do you recommend using LV''s in general when configuring the OST?

--bob

On Mar 25, 2009, at 7:28 AM, Brian J. Murrell wrote:
>     A. if your OSTs were LVM LVs, you could take a snapshot and at
>        least get started on trying to find out where files should be
>        moved when you can afford to take the OST down and
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090325/7a43317a/attachment.html

On Wed, 2009-03-25 at 11:29 -0500, Robert Olson wrote:
> I thought I had read at some point that there is a performance hit
> associated with putting the OST on a LV;
There is nothing particular about an OST on an LV that makes it perform
any worse than anything else on an LV.  That said I don''t know what
overhead there is in using an LV over a raw block device.  I''ve been
using LVs exclusively on all of my (non Lustre) systems for many many
years and I''ve never felt that they were unacceptably slow.

You might want to benchmark just to have hard numbers on your side.
Benchmarking LVs against the raw block device should be trivial.
> or is that just the case when there are snapshots in play?
Yes, there is definitely a penalty when snapshots are in play since
every write to a block on the origin that has not been written since the
snapshot is actually two writes to copy (or move -- I''m not sure) the
block to the snapshot before writing the new data in the origin.
> Do you recommend using LV''s in general when configuring the OST?
We don''t recommend one way or the other.  It''s really a matter
of
personal choice.

b.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090325/8b5dd131/attachment.bin

This data is really not important. When I tried to remove it I keep
getting, "Numerical result out of range".

Is there something else I can do?



On Wed, Mar 25, 2009 at 12:50 PM, Brian J. Murrell
<Brian.Murrell at sun.com> wrote:
> On Wed, 2009-03-25 at 11:29 -0500, Robert Olson wrote:
>> I thought I had read at some point that there is a performance hit
>> associated with putting the OST on a LV;
>
> There is nothing particular about an OST on an LV that makes it perform
> any worse than anything else on an LV.  That said I don''t know
what
> overhead there is in using an LV over a raw block device.  I''ve
been
> using LVs exclusively on all of my (non Lustre) systems for many many
> years and I''ve never felt that they were unacceptably slow.
>
> You might want to benchmark just to have hard numbers on your side.
> Benchmarking LVs against the raw block device should be trivial.
>
>> or is that just the case when there are snapshots in play?
>
> Yes, there is definitely a penalty when snapshots are in play since
> every write to a block on the origin that has not been written since the
> snapshot is actually two writes to copy (or move -- I''m not sure)
the
> block to the snapshot before writing the new data in the origin.
>
>> Do you recommend using LV''s in general when configuring the
OST?
>
> We don''t recommend one way or the other.  It''s really a
matter of
> personal choice.
>
> b.
>
>
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>

The main case to get ERANGE error is file have more stripes then have
ost''s, in this case mds can''t get striping info and return the
error to
client.


On Sat, 2009-03-28 at 08:52 -0400, Mag Gam wrote:
> This data is really not important. When I tried to remove it I keep
> getting, "Numerical result out of range".
> 
> Is there something else I can do?
> 
> 
-- 
Alex Lyashkov <alexey.lyashkov at sun.com>
Lustre Group, Sun Microsystems