Erik Froese
2009-Dec-30 15:44 UTC
[Lustre-discuss] OST crashed after slow journal messages
Hey everyone, I had an OST crash (actually it made the entire OSS unresponsive to the point where I had to shoot it). There were messages in /var/log/messages complaining about slow journal performance (we have separate OSTs and journal disks). Dec 28 20:29:02 oss-0-0 kernel: LustreError: 12638:0:(filter_io_26.c:742:filter_commitrw_write()) scratch-OST000e: slow direct_io 85s Dec 28 20:29:02 oss-0-0 kernel: LustreError: 12638:0:(filter_io_26.c:742:filter_commitrw_write()) Skipped 58 previous similar messages Dec 28 21:50:13 oss-0-0 kernel: LustreError: 16499:0:(lustre_fsfilt.h:320:fsfilt_commit_wait()) scratch-OST000e: slow journal start 51s Dec 28 21:50:13 oss-0-0 kernel: LustreError: 16499:0:(lustre_fsfilt.h:320:fsfilt_commit_wait()) Skipped 66 previous similar messages Lustre and e2fsprogs versions: [root at oss-0-0 ~]# rpm -q kernel-lustre kernel-lustre-2.6.18-128.7.1.el5_lustre.1.8.1.1 [root at oss-0-0 ~]# rpm -q e2fsprogs e2fsprogs-1.41.6.sun1-0redhat Then there''s this interesting message: Dec 29 14:11:32 oss-0-0 kernel: LDISKFS-fs error (device sdz): ldiskfs_lookup: unlinked inode 5384166 in dir #145170469 Dec 29 14:11:32 oss-0-0 kernel: Remounting filesystem read-only Followed by a whole lot of these: Dec 29 14:11:32 oss-0-0 kernel: LustreError: 18559:0:(fsfilt-ldiskfs.c:336:fsfilt_ldiskfs_start()) error starting handle for op 8 (106 credits): rc -30 Dec 29 14:11:33 oss-0-0 kernel: LustreError: 12648:0:(fsfilt-ldiskfs.c:336:fsfilt_ldiskfs_start()) error starting handle for op 8 (106 credits): rc -30 Dec 29 14:11:34 oss-0-0 kernel: LustreError: 16430:0:(fsfilt-ldiskfs.c:336:fsfilt_ldiskfs_start()) error starting handle for op 8 (106 credits): rc -30 Dec 29 14:11:35 oss-0-0 kernel: LustreError: 16446:0:(fsfilt-ldiskfs.c:470:fsfilt_ldiskfs_brw_start()) can''t get handle for 125 credits: rc = -30 Dec 29 14:11:35 oss-0-0 kernel: LustreError: 16446:0:(filter_io_26.c:684:filter_commitrw_write()) error starting transaction: rc = -30 Dec 29 14:11:35 oss-0-0 kernel: LustreError: 12666:0:(filter_io_26.c:684:filter_commitrw_write()) error starting transaction: rc = -30 Dec 29 14:11:36 oss-0-0 kernel: LustreError: 16417:0:(fsfilt-ldiskfs.c:470:fsfilt_ldiskfs_brw_start()) can''t get handle for 471 credits: rc = -30 Dec 29 14:11:36 oss-0-0 kernel: LustreError: 16417:0:(fsfilt-ldiskfs.c:470:fsfilt_ldiskfs_brw_start()) Skipped 1 previous similar message Dec 29 14:11:36 oss-0-0 kernel: LustreError: 16417:0:(filter_io_26.c:684:filter_commitrw_write()) error starting transaction: rc = -30 Whenever I try to mount the ost (known as /dev/dsk/ost24) I get the following messages: Dec 29 19:25:35 oss-0-0 kernel: LDISKFS-fs error (device sdz): ldiskfs_check_descriptors: Checksum for group 16303 failed (64812!=44) Dec 29 19:25:35 oss-0-0 kernel: LDISKFS-fs: group descriptors corrupted! Dec 29 19:25:35 oss-0-0 kernel: LustreError: 15067:0:(obd_mount.c:1278:server_kernel_mount()) premount /dev/dsk/ost24:0x0 ldiskfs failed: -22, ldiskfs2 failed: -19. Is the ldi skfs module available? Dec 29 19:25:35 oss-0-0 kernel: LustreError: 15067:0:(obd_mount.c:1592:server_fill_super()) Unable to mount device /dev/dsk/ost24: -22 Dec 29 19:25:35 oss-0-0 kernel: LustreError: 15067:0:(obd_mount.c:1997:lustre_fill_super()) Unable to mount (-22) Dec 29 19:25:35 oss-0-0 Filesystem[15020]: ERROR: Couldn''t mount filesystem /dev/dsk/ost24 on /mnt/scratch/ost24 Dec 29 19:25:35 oss-0-0 Filesystem[15009]: ERROR: Generic error So it looks like the "group descriptors" are corrupted. I''m not sure what those are but e2fsck -n sure enough complains about them. So I tried running it for real. I ran e2fsck -j /dev/$JOURNAL -v -fy -C 0 /dev/$DEVICE. The first time I ran to what looked like completion. It printed a summary and all but then didn''t exit. I sent it a kill but that didn''t stop it. So I let it run and went back to sleep for 3 hours. When I woke up the process was gone but I still get the same error messages. I found this discussion http://lists.lustre.org/pipermail/lustre-discuss/2009-March/009885.html and tried the tune2fs command followed by the e2fsck but it hasn''t exited yet (its a 2.7 TB OST) The LUN comes from a Sun STK 6140/CSM200 device which isn''t reporting any warning, events, or errors. I deactivated the OST with lctl but it still shows up as active on the clients. Also lfs find /scratch -O scratch-OST000e_UUID HANGS! Are we screwed here? Is there a way to run lfs find with the OST disabled? Shouldn''t that just be a metadata operation? Thanks Erik -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20091230/9929ee13/attachment.html
Andreas Dilger
2009-Dec-31 21:52 UTC
[Lustre-discuss] OST crashed after slow journal messages
On 2009-12-30, at 08:44, Erik Froese wrote:> I had an OST crash (actually it made the entire OSS unresponsive to > the point where I had to shoot it). There were messages in /var/log/ > messages complaining about slow journal performance (we have > separate OSTs and journal disks). > > Dec 28 20:29:02 oss-0-0 kernel: LustreError:filter_commitrw_write()) > scratch-OST000e: slow direct_io 85s > Dec 28 20:29:02 oss-0-0 kernel: LustreError:filter_commitrw_write()) > Skipped 58 previous similar messages > Dec 28 21:50:13 oss-0-0 kernel: LustreError:fsfilt_commit_wait()) > scratch-OST000e: slow journal start 51s > Dec 28 21:50:13 oss-0-0 kernel: LustreError:fsfilt_commit_wait()) > Skipped 66 previous similar messagesThese are usually a sign that the back-end storage is overloaded, or somehow performing very slowly. Maybe there was a RAID rebuild going on?> Lustre and e2fsprogs versions: > > [root at oss-0-0 ~]# rpm -q kernel-lustre > kernel-lustre-2.6.18-128.7.1.el5_lustre.1.8.1.1 > [root at oss-0-0 ~]# rpm -q e2fsprogs > e2fsprogs-1.41.6.sun1-0redhat > > > Then there''s this interesting message: > Dec 29 14:11:32 oss-0-0 kernel: LDISKFS-fs error (device sdz): > ldiskfs_lookup: unlinked inode 5384166 in dir #145170469 > Dec 29 14:11:32 oss-0-0 kernel: Remounting filesystem read-onlyThis means the ldiskfs code found some corruption on disk, and remounted the filesystem read-only to avoid further corruptions on disk.> Whenever I try to mount the ost (known as /dev/dsk/ost24) I get the > following messages: > Dec 29 19:25:35 oss-0-0 kernel: LDISKFS-fs error (device sdz): > ldiskfs_check_descriptors: Checksum for group 16303 failed (64812!=44) > Dec 29 19:25:35 oss-0-0 kernel: LDISKFS-fs: group descriptors > corrupted!> So it looks like the "group descriptors" are corrupted. I''m not sure > what those are but e2fsck -n sure enough complains about them. So I > tried running it for real. > > I ran e2fsck -j /dev/$JOURNAL -v -fy -C 0 /dev/$DEVICE. > > The first time I ran to what looked like completion. It printed a > summary and all but then didn''t exit. I sent it a kill but that > didn''t stop it. So I let it run and went back to sleep for 3 hours. > When I woke up the process was gone but I still get the same error > messages.Having a log of the e2fsck errors would be helpful.> I found this discussion http://lists.lustre.org/pipermail/lustre-discuss/2009-March/009885.html > and tried the tune2fs command followed by the e2fsck but it hasn''t > exited yet (its a 2.7 TB OST)It might take an hour or two, depending on how fast your storage is.> The LUN comes from a Sun STK 6140/CSM200 device which isn''t > reporting any warning, events, or errors. > > I deactivated the OST with lctl but it still shows up as active on > the clients. Also lfs find /scratch -O scratch-OST000e_UUID HANGS!You also need to deactivate it on the clients, at which point they will get an IO error when accessing files on that OST.> Are we screwed here? Is there a way to run lfs find with the OST > disabled? Shouldn''t that just be a metadata operation?The size of a file is stored on the OSTs, so it depends on what you are trying to do. "lfs getstripe" can be run with a deactivated OST. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.
Erik Froese
2010-Jan-02 00:20 UTC
[Lustre-discuss] OST crashed after slow journal messages
On Thu, Dec 31, 2009 at 4:52 PM, Andreas Dilger <adilger at sun.com> wrote:> On 2009-12-30, at 08:44, Erik Froese wrote: > >> I had an OST crash (actually it made the entire OSS unresponsive to the >> point where I had to shoot it). There were messages in /var/log/messages >> complaining about slow journal performance (we have separate OSTs and >> journal disks). >> >> Dec 28 20:29:02 oss-0-0 kernel: LustreError:filter_commitrw_write()) >> scratch-OST000e: slow direct_io 85s >> Dec 28 20:29:02 oss-0-0 kernel: LustreError:filter_commitrw_write()) >> Skipped 58 previous similar messages >> Dec 28 21:50:13 oss-0-0 kernel: LustreError:fsfilt_commit_wait()) >> scratch-OST000e: slow journal start 51s >> Dec 28 21:50:13 oss-0-0 kernel: LustreError:fsfilt_commit_wait()) Skipped >> 66 previous similar messages >> > > These are usually a sign that the back-end storage is overloaded, or > somehow > performing very slowly. Maybe there was a RAID rebuild going on?I don''t see any hw failures or rebuild messages on the RAID. I do see periodic media scans going on.> > > Lustre and e2fsprogs versions: >> >> [root at oss-0-0 ~]# rpm -q kernel-lustre >> kernel-lustre-2.6.18-128.7.1.el5_lustre.1.8.1.1 >> [root at oss-0-0 ~]# rpm -q e2fsprogs >> e2fsprogs-1.41.6.sun1-0redhat >> >> >> Then there''s this interesting message: >> Dec 29 14:11:32 oss-0-0 kernel: LDISKFS-fs error (device sdz): >> ldiskfs_lookup: unlinked inode 5384166 in dir #145170469 >> Dec 29 14:11:32 oss-0-0 kernel: Remounting filesystem read-only >> > > This means the ldiskfs code found some corruption on disk, and remounted > the filesystem read-only to avoid further corruptions on disk. > > > Whenever I try to mount the ost (known as /dev/dsk/ost24) I get the >> following messages: >> Dec 29 19:25:35 oss-0-0 kernel: LDISKFS-fs error (device sdz): >> ldiskfs_check_descriptors: Checksum for group 16303 failed (64812!=44) >> Dec 29 19:25:35 oss-0-0 kernel: LDISKFS-fs: group descriptors corrupted! >> > > So it looks like the "group descriptors" are corrupted. I''m not sure what >> those are but e2fsck -n sure enough complains about them. So I tried running >> it for real. >> >> I ran e2fsck -j /dev/$JOURNAL -v -fy -C 0 /dev/$DEVICE. >> >> The first time I ran to what looked like completion. It printed a summary >> and all but then didn''t exit. I sent it a kill but that didn''t stop it. So I >> let it run and went back to sleep for 3 hours. When I woke up the process >> was gone but I still get the same error messages. >> > > Having a log of the e2fsck errors would be helpful.I don''t have the log for the first fsck. First it logged a bunch of messages about the group descriptors and that it was repairing them. Then messages about inodes. I''m attaching the logs from the second e2fsck.> > > I found this discussion >> http://lists.lustre.org/pipermail/lustre-discuss/2009-March/009885.html >> and tried the tune2fs command followed by the e2fsck but it hasn''t exited >> yet (its a 2.7 TB OST) >> > > It might take an hour or two, depending on how fast your storage is. > > > The LUN comes from a Sun STK 6140/CSM200 device which isn''t reporting any >> warning, events, or errors. >> >> I deactivated the OST with lctl but it still shows up as active on the >> clients. Also lfs find /scratch -O scratch-OST000e_UUID HANGS! >> > > You also need to deactivate it on the clients, at which point they will > get an IO error when accessing files on that OST.I didn''t know that the you had to disable it on the clients as well. Thanks> > > Are we screwed here? Is there a way to run lfs find with the OST disabled? >> Shouldn''t that just be a metadata operation? >> > > > The size of a file is stored on the OSTs, so it depends on what you are > trying to do. "lfs getstripe" can be run with a deactivated OST. > > > Cheers, Andreas > -- > Andreas Dilger > Sr. Staff Engineer, Lustre Group > Sun Microsystems of Canada, Inc. > >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100101/49d7d319/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: e2fsck-fy.sdz.out.post-mmp Type: application/octet-stream Size: 31819 bytes Desc: not available Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100101/49d7d319/attachment-0001.obj
Andreas Dilger
2010-Jan-02 06:13 UTC
[Lustre-discuss] OST crashed after slow journal messages
On 2010-01-01, at 17:20, Erik Froese wrote:> On Thu, Dec 31, 2009 at 4:52 PM, Andreas Dilger <adilger at sun.com> > wrote: >> These are usually a sign that the back-end storage is overloaded, >> or somehow performing very slowly. Maybe there was a RAID rebuild >> going on? > > I don''t see any hw failures or rebuild messages on the RAID. I do > see periodic media scans going on.There definitely was corruption of the directories on the OST, consistent with what the kernel originally reported, but it looks like it was resolved with a minimum of further problems. You probably want to run the "ll_recover_lost_found_objs" on this OST to restore the objects from lost+found back to their correct locations, though that isn''t strictly necessary to bring the OST back online - it just avoids losing access to the objects that were moved to lost+found due to the directory corruption.> Lustre and e2fsprogs versions: > > [root at oss-0-0 ~]# rpm -q kernel-lustre > kernel-lustre-2.6.18-128.7.1.el5_lustre.1.8.1.1 > [root at oss-0-0 ~]# rpm -q e2fsprogs > e2fsprogs-1.41.6.sun1-0redhat > > > Then there''s this interesting message: > Dec 29 14:11:32 oss-0-0 kernel: LDISKFS-fs error (device sdz): > ldiskfs_lookup: unlinked inode 5384166 in dir #145170469 > Dec 29 14:11:32 oss-0-0 kernel: Remounting filesystem read-only > > This means the ldiskfs code found some corruption on disk, and > remounted > the filesystem read-only to avoid further corruptions on disk. > > > Whenever I try to mount the ost (known as /dev/dsk/ost24) I get the > following messages: > Dec 29 19:25:35 oss-0-0 kernel: LDISKFS-fs error (device sdz): > ldiskfs_check_descriptors: Checksum for group 16303 failed (64812!=44) > Dec 29 19:25:35 oss-0-0 kernel: LDISKFS-fs: group descriptors > corrupted! > > So it looks like the "group descriptors" are corrupted. I''m not sure > what those are but e2fsck -n sure enough complains about them. So I > tried running it for real. > > I ran e2fsck -j /dev/$JOURNAL -v -fy -C 0 /dev/$DEVICE. > > The first time I ran to what looked like completion. It printed a > summary and all but then didn''t exit. I sent it a kill but that > didn''t stop it. So I let it run and went back to sleep for 3 hours. > When I woke up the process was gone but I still get the same error > messages. > > Having a log of the e2fsck errors would be helpful. > > I don''t have the log for the first fsck. First it logged a bunch of > messages about the group descriptors and that it was repairing them. > Then messages about inodes. I''m attaching the logs from the second > e2fsck. > > > > > > I found this discussion http://lists.lustre.org/pipermail/lustre-discuss/2009-March/009885.html > and tried the tune2fs command followed by the e2fsck but it hasn''t > exited yet (its a 2.7 TB OST) > > It might take an hour or two, depending on how fast your storage is. > > > The LUN comes from a Sun STK 6140/CSM200 device which isn''t > reporting any warning, events, or errors. > > I deactivated the OST with lctl but it still shows up as active on > the clients. Also lfs find /scratch -O scratch-OST000e_UUID HANGS! > > You also need to deactivate it on the clients, at which point they > will > get an IO error when accessing files on that OST. > > I didn''t know that the you had to disable it on the clients as > well. Thanks > > > Are we screwed here? Is there a way to run lfs find with the OST > disabled? Shouldn''t that just be a metadata operation? > > > The size of a file is stored on the OSTs, so it depends on what you > are trying to do. "lfs getstripe" can be run with a deactivated OST. > > > Cheers, Andreas > -- > Andreas Dilger > Sr. Staff Engineer, Lustre Group > Sun Microsystems of Canada, Inc. > > > <e2fsck-fy.sdz.out.post-mmp>Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.
Erik Froese
2010-Jan-02 17:33 UTC
[Lustre-discuss] OST crashed after slow journal messages
Thanks Andreas, After I e2fsck''d the OST again (and once more to make sure there weren''t any errors) the OST mounted cleanly. I''m not able to run the ll_recover_lost_found_objs. [root at oss-0-0 ~]# ll_recover_lost_found_objs -d /mnt/scratch/ost24/lost+found | tee /tmp/ll_recover_lost_found_obj.ost24 error: creating objects directory /mnt/scratch/ost24/O: Not a directory "lost+found" directory path: /mnt/scratch/ost24/lost+found The lustre FS is mounted at /scratch on the clients and I don''t see a lost+found directory there either. Erik On Sat, Jan 2, 2010 at 1:13 AM, Andreas Dilger <adilger at sun.com> wrote:> On 2010-01-01, at 17:20, Erik Froese wrote: > >> On Thu, Dec 31, 2009 at 4:52 PM, Andreas Dilger <adilger at sun.com> wrote: >> >>> These are usually a sign that the back-end storage is overloaded, or >>> somehow performing very slowly. Maybe there was a RAID rebuild going on? >>> >> >> I don''t see any hw failures or rebuild messages on the RAID. I do see >> periodic media scans going on. >> > > There definitely was corruption of the directories on the OST, consistent > with what the kernel originally reported, but it looks like it was resolved > with a minimum of further problems. You probably want to run the > "ll_recover_lost_found_objs" on this OST to restore the objects from > lost+found back to their correct locations, though that isn''t strictly > necessary to bring the OST back online - it just avoids losing access to the > objects that were moved to lost+found due to the directory corruption. > > Lustre and e2fsprogs versions: >> >> [root at oss-0-0 ~]# rpm -q kernel-lustre >> kernel-lustre-2.6.18-128.7.1.el5_lustre.1.8.1.1 >> [root at oss-0-0 ~]# rpm -q e2fsprogs >> e2fsprogs-1.41.6.sun1-0redhat >> >> >> Then there''s this interesting message: >> Dec 29 14:11:32 oss-0-0 kernel: LDISKFS-fs error (device sdz): >> ldiskfs_lookup: unlinked inode 5384166 in dir #145170469 >> Dec 29 14:11:32 oss-0-0 kernel: Remounting filesystem read-only >> >> This means the ldiskfs code found some corruption on disk, and remounted >> the filesystem read-only to avoid further corruptions on disk. >> >> >> Whenever I try to mount the ost (known as /dev/dsk/ost24) I get the >> following messages: >> Dec 29 19:25:35 oss-0-0 kernel: LDISKFS-fs error (device sdz): >> ldiskfs_check_descriptors: Checksum for group 16303 failed (64812!=44) >> Dec 29 19:25:35 oss-0-0 kernel: LDISKFS-fs: group descriptors corrupted! >> >> So it looks like the "group descriptors" are corrupted. I''m not sure what >> those are but e2fsck -n sure enough complains about them. So I tried running >> it for real. >> >> I ran e2fsck -j /dev/$JOURNAL -v -fy -C 0 /dev/$DEVICE. >> >> The first time I ran to what looked like completion. It printed a summary >> and all but then didn''t exit. I sent it a kill but that didn''t stop it. So I >> let it run and went back to sleep for 3 hours. When I woke up the process >> was gone but I still get the same error messages. >> >> Having a log of the e2fsck errors would be helpful. >> >> I don''t have the log for the first fsck. First it logged a bunch of >> messages about the group descriptors and that it was repairing them. Then >> messages about inodes. I''m attaching the logs from the second e2fsck. >> >> >> >> >> >> I found this discussion >> http://lists.lustre.org/pipermail/lustre-discuss/2009-March/009885.html >> and tried the tune2fs command followed by the e2fsck but it hasn''t exited >> yet (its a 2.7 TB OST) >> >> It might take an hour or two, depending on how fast your storage is. >> >> >> The LUN comes from a Sun STK 6140/CSM200 device which isn''t reporting any >> warning, events, or errors. >> >> I deactivated the OST with lctl but it still shows up as active on the >> clients. Also lfs find /scratch -O scratch-OST000e_UUID HANGS! >> >> You also need to deactivate it on the clients, at which point they will >> get an IO error when accessing files on that OST. >> >> I didn''t know that the you had to disable it on the clients as well. >> Thanks >> >> >> Are we screwed here? Is there a way to run lfs find with the OST disabled? >> Shouldn''t that just be a metadata operation? >> >> >> The size of a file is stored on the OSTs, so it depends on what you are >> trying to do. "lfs getstripe" can be run with a deactivated OST. >> >> >> Cheers, Andreas >> -- >> Andreas Dilger >> Sr. Staff Engineer, Lustre Group >> Sun Microsystems of Canada, Inc. >> >> >> <e2fsck-fy.sdz.out.post-mmp> >> > > > Cheers, Andreas > -- > Andreas Dilger > Sr. Staff Engineer, Lustre Group > Sun Microsystems of Canada, Inc. > >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100102/94fb07ed/attachment.html
Erik Froese
2010-Jan-02 17:43 UTC
[Lustre-discuss] OST crashed after slow journal messages
Never mind I figured it out. First you have to mount -t ldiskfs /dev/OSTDEV /mnt/OST Then you can run ll_recover_lost_found_objs. Erik On Sat, Jan 2, 2010 at 12:33 PM, Erik Froese <erik.froese at gmail.com> wrote:> Thanks Andreas, > > After I e2fsck''d the OST again (and once more to make sure there weren''t > any errors) the OST mounted cleanly. > I''m not able to run the ll_recover_lost_found_objs. > > [root at oss-0-0 ~]# ll_recover_lost_found_objs -d > /mnt/scratch/ost24/lost+found | tee /tmp/ll_recover_lost_found_obj.ost24 > error: creating objects directory /mnt/scratch/ost24/O: Not a directory > "lost+found" directory path: /mnt/scratch/ost24/lost+found > > The lustre FS is mounted at /scratch on the clients and I don''t see a > lost+found directory there either. > > Erik > > On Sat, Jan 2, 2010 at 1:13 AM, Andreas Dilger <adilger at sun.com> wrote: > >> On 2010-01-01, at 17:20, Erik Froese wrote: >> >>> On Thu, Dec 31, 2009 at 4:52 PM, Andreas Dilger <adilger at sun.com> wrote: >>> >>>> These are usually a sign that the back-end storage is overloaded, or >>>> somehow performing very slowly. Maybe there was a RAID rebuild going on? >>>> >>> >>> I don''t see any hw failures or rebuild messages on the RAID. I do see >>> periodic media scans going on. >>> >> >> There definitely was corruption of the directories on the OST, consistent >> with what the kernel originally reported, but it looks like it was resolved >> with a minimum of further problems. You probably want to run the >> "ll_recover_lost_found_objs" on this OST to restore the objects from >> lost+found back to their correct locations, though that isn''t strictly >> necessary to bring the OST back online - it just avoids losing access to the >> objects that were moved to lost+found due to the directory corruption. >> >> Lustre and e2fsprogs versions: >>> >>> [root at oss-0-0 ~]# rpm -q kernel-lustre >>> kernel-lustre-2.6.18-128.7.1.el5_lustre.1.8.1.1 >>> [root at oss-0-0 ~]# rpm -q e2fsprogs >>> e2fsprogs-1.41.6.sun1-0redhat >>> >>> >>> Then there''s this interesting message: >>> Dec 29 14:11:32 oss-0-0 kernel: LDISKFS-fs error (device sdz): >>> ldiskfs_lookup: unlinked inode 5384166 in dir #145170469 >>> Dec 29 14:11:32 oss-0-0 kernel: Remounting filesystem read-only >>> >>> This means the ldiskfs code found some corruption on disk, and remounted >>> the filesystem read-only to avoid further corruptions on disk. >>> >>> >>> Whenever I try to mount the ost (known as /dev/dsk/ost24) I get the >>> following messages: >>> Dec 29 19:25:35 oss-0-0 kernel: LDISKFS-fs error (device sdz): >>> ldiskfs_check_descriptors: Checksum for group 16303 failed (64812!=44) >>> Dec 29 19:25:35 oss-0-0 kernel: LDISKFS-fs: group descriptors corrupted! >>> >>> So it looks like the "group descriptors" are corrupted. I''m not sure what >>> those are but e2fsck -n sure enough complains about them. So I tried running >>> it for real. >>> >>> I ran e2fsck -j /dev/$JOURNAL -v -fy -C 0 /dev/$DEVICE. >>> >>> The first time I ran to what looked like completion. It printed a summary >>> and all but then didn''t exit. I sent it a kill but that didn''t stop it. So I >>> let it run and went back to sleep for 3 hours. When I woke up the process >>> was gone but I still get the same error messages. >>> >>> Having a log of the e2fsck errors would be helpful. >>> >>> I don''t have the log for the first fsck. First it logged a bunch of >>> messages about the group descriptors and that it was repairing them. Then >>> messages about inodes. I''m attaching the logs from the second e2fsck. >>> >>> >>> >>> >>> >>> I found this discussion >>> http://lists.lustre.org/pipermail/lustre-discuss/2009-March/009885.html >>> and tried the tune2fs command followed by the e2fsck but it hasn''t exited >>> yet (its a 2.7 TB OST) >>> >>> It might take an hour or two, depending on how fast your storage is. >>> >>> >>> The LUN comes from a Sun STK 6140/CSM200 device which isn''t reporting any >>> warning, events, or errors. >>> >>> I deactivated the OST with lctl but it still shows up as active on the >>> clients. Also lfs find /scratch -O scratch-OST000e_UUID HANGS! >>> >>> You also need to deactivate it on the clients, at which point they will >>> get an IO error when accessing files on that OST. >>> >>> I didn''t know that the you had to disable it on the clients as well. >>> Thanks >>> >>> >>> Are we screwed here? Is there a way to run lfs find with the OST >>> disabled? Shouldn''t that just be a metadata operation? >>> >>> >>> The size of a file is stored on the OSTs, so it depends on what you are >>> trying to do. "lfs getstripe" can be run with a deactivated OST. >>> >>> >>> Cheers, Andreas >>> -- >>> Andreas Dilger >>> Sr. Staff Engineer, Lustre Group >>> Sun Microsystems of Canada, Inc. >>> >>> >>> <e2fsck-fy.sdz.out.post-mmp> >>> >> >> >> Cheers, Andreas >> -- >> Andreas Dilger >> Sr. Staff Engineer, Lustre Group >> Sun Microsystems of Canada, Inc. >> >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100102/8b5df0c9/attachment.html