For reasons that I can''t recall, our OSTs are not in consecutive order -- we have 35 OSTs, which are numbered consecutively from 0000-0021 and then there''s one last OST at 002a When I try to run lfsck on this array, it works fine for the first 34 OSTs, but it can''t seem to find the last OST db file: lfsck: ost_idx 34: pass3 OK (680045 files total) lfsck: can''t find file for ost_idx 35 Files affected by missing ost info are : - lfsck: can''t find file for ost_idx 36 Files affected by missing ost info are : - lfsck: can''t find file for ost_idx 37 Files affected by missing ost info are : - lfsck: can''t find file for ost_idx 38 Files affected by missing ost info are : - lfsck: can''t find file for ost_idx 39 Files affected by missing ost info are : - lfsck: can''t find file for ost_idx 40 Files affected by missing ost info are : - lfsck: can''t find file for ost_idx 41 Files affected by missing ost info are : - lfsck: can''t find file for ost_idx 42 Files affected by missing ost info are : - /n/scratch/hernquist_lab/tcox/tests/SbSbhs_e_8/P-Gadget3.3.1/IdlSubfind/.svn/text-base/ReadSubhaloFromReshuffledSnapshot.pro.svn-base and then lists all of the files that live on OST 002a. This db file definitely does exist -- it lives in the same directory as all of the other db files, and e2fsck for this OST ran without problems. Is there some way of forcing lfsck to recognize this OST db? Or, failing that, is it dangerous to run lfsck on the first 34 OSTs only? We''re using e2fsck 1.41.6.sun1 (30-May-2009) Thanks very much! Chris
The error message indicates that the UUID of OST #35 does not match between the live filesystem and the ostdb file. Is this ostdb obsolete? ? 2010-11-9???11:45? Christopher Walker ???> > > For reasons that I can''t recall, our OSTs are not in consecutive order > -- we have 35 OSTs, which are numbered consecutively from > 0000-0021 > and then there''s one last OST at > 002a > > When I try to run lfsck on this array, it works fine for the first 34 > OSTs, but it can''t seem to find the last OST db file: > > lfsck: ost_idx 34: pass3 OK (680045 files total) > lfsck: can''t find file for ost_idx 35 > Files affected by missing ost info are : - > lfsck: can''t find file for ost_idx 36 > Files affected by missing ost info are : - > lfsck: can''t find file for ost_idx 37 > Files affected by missing ost info are : - > lfsck: can''t find file for ost_idx 38 > Files affected by missing ost info are : - > lfsck: can''t find file for ost_idx 39 > Files affected by missing ost info are : - > lfsck: can''t find file for ost_idx 40 > Files affected by missing ost info are : - > lfsck: can''t find file for ost_idx 41 > Files affected by missing ost info are : - > lfsck: can''t find file for ost_idx 42 > Files affected by missing ost info are : - > /n/scratch/hernquist_lab/tcox/tests/SbSbhs_e_8/P-Gadget3.3.1/IdlSubfind/.svn/text-base/ReadSubhaloFromReshuffledSnapshot.pro.svn-base > > and then lists all of the files that live on OST 002a. This db file > definitely does exist -- it lives in the same directory as all of the > other db files, and e2fsck for this OST ran without problems. > > Is there some way of forcing lfsck to recognize this OST db? Or, > failing that, is it dangerous to run lfsck on the first 34 OSTs only? > > We''re using e2fsck 1.41.6.sun1 (30-May-2009) > > Thanks very much! > > Chris > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss
Thanks very much for your reply. I''ve tried remaking the mdsdb and all of the ostdb''s, but I still get the same error -- it checks the first 34 osts without a problem, but can''t find the ostdb file for the 35th (which has ost_idx 42): ... lfsck: ost_idx 34: pass3 OK (676803 files total) lfsck: can''t find file for ost_idx 35 Files affected by missing ost info are : - lfsck: can''t find file for ost_idx 36 Files affected by missing ost info are : - lfsck: can''t find file for ost_idx 37 Files affected by missing ost info are : - lfsck: can''t find file for ost_idx 38 Files affected by missing ost info are : - lfsck: can''t find file for ost_idx 39 Files affected by missing ost info are : - lfsck: can''t find file for ost_idx 40 Files affected by missing ost info are : - lfsck: can''t find file for ost_idx 41 Files affected by missing ost info are : - lfsck: can''t find file for ost_idx 42 Files affected by missing ost info are : - ... e2fsck claims to be making the ostdb without a problem: Pass 6: Acquiring information for lfsck OST: ''aegalfs-OST002a_UUID'' ost idx 42: compat 0x2 rocomp 0 incomp 0x2 OST: num files = 676803 OST: last_id = 858163 and with the filesystem up I can see files on this OST: [cwalker at iliadaccess04 P-Gadget3.3.1]$ lfs getstripe predict.c OBDS: 0: aegalfs-OST0000_UUID ACTIVE ... 33: aegalfs-OST0021_UUID ACTIVE 42: aegalfs-OST002a_UUID ACTIVE predict.c obdidx objid objid group 42 10 0xa 0 lfsck identifies several hundred GB of orphan data that we''d like to recover, so we''d really like to run lfsck on this array. We''re willing to forgo the recovery on the 35th ost, but I want to make sure that running lfsck -l with the current configuration won''t make things worse. Thanks again for your reply; any further advice is very much appreciated! Best, Chris On 11/10/10 12:10 AM, Wang Yibin wrote:> The error message indicates that the UUID of OST #35 does not match between the live filesystem and the ostdb file. > Is this ostdb obsolete? > > ? 2010-11-9???11:45? Christopher Walker ??? > >> >> For reasons that I can''t recall, our OSTs are not in consecutive order >> -- we have 35 OSTs, which are numbered consecutively from >> 0000-0021 >> and then there''s one last OST at >> 002a >> >> When I try to run lfsck on this array, it works fine for the first 34 >> OSTs, but it can''t seem to find the last OST db file: >> >> lfsck: ost_idx 34: pass3 OK (680045 files total) >> lfsck: can''t find file for ost_idx 35 >> Files affected by missing ost info are : - >> lfsck: can''t find file for ost_idx 36 >> Files affected by missing ost info are : - >> lfsck: can''t find file for ost_idx 37 >> Files affected by missing ost info are : - >> lfsck: can''t find file for ost_idx 38 >> Files affected by missing ost info are : - >> lfsck: can''t find file for ost_idx 39 >> Files affected by missing ost info are : - >> lfsck: can''t find file for ost_idx 40 >> Files affected by missing ost info are : - >> lfsck: can''t find file for ost_idx 41 >> Files affected by missing ost info are : - >> lfsck: can''t find file for ost_idx 42 >> Files affected by missing ost info are : - >> /n/scratch/hernquist_lab/tcox/tests/SbSbhs_e_8/P-Gadget3.3.1/IdlSubfind/.svn/text-base/ReadSubhaloFromReshuffledSnapshot.pro.svn-base >> >> and then lists all of the files that live on OST 002a. This db file >> definitely does exist -- it lives in the same directory as all of the >> other db files, and e2fsck for this OST ran without problems. >> >> Is there some way of forcing lfsck to recognize this OST db? Or, >> failing that, is it dangerous to run lfsck on the first 34 OSTs only? >> >> We''re using e2fsck 1.41.6.sun1 (30-May-2009) >> >> Thanks very much! >> >> Chris >> >> >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss
On 2010-11-11, at 19:53, Christopher Walker wrote:> Thanks very much for your reply. I''ve tried remaking the mdsdb and all > of the ostdb''s, but I still get the same error -- it checks the first 34 > osts without a problem, but can''t find the ostdb file for the 35th > (which has ost_idx 42): > > with the filesystem up I can see files on this OST: > > [cwalker at iliadaccess04 P-Gadget3.3.1]$ lfs getstripe predict.c > OBDS: > 0: aegalfs-OST0000_UUID ACTIVE > ... > 33: aegalfs-OST0021_UUID ACTIVE > 42: aegalfs-OST002a_UUID ACTIVE > predict.c > obdidx objid objid group > 42 10 0xa 0 > > > lfsck identifies several hundred GB of orphan data that we''d like to > recover, so we''d really like to run lfsck on this array. We''re willing > to forgo the recovery on the 35th ost, but I want to make sure that > running lfsck -l with the current configuration won''t make things worse.I''m not sure that what lfsck is reporting in this case is correct. Is the orphan data all on the same OST, or spread around separate OSTs? My concern is that if lfsck thinks the in-use objects on your estranged OST is actually orphan data it will destroy that data. If there are a small number of very large objects on other OSTs that are making up the bulk of the orphan space usage, you could mount those OSTs as type ldiskfs and delete the objects by hand to free up the space. Cheers, Andreas -- Andreas Dilger Lustre Technical Lead Oracle Corporation Canada Inc.
This is a bug in llapi_lov_get_uuids() which assigns UUID to the wrong OST index
when there are sparse OST(s).
Please file a bug for this.
Before this bug can be fixed, you can apply the following patch to
e2fsprogs(version 1.41.12.2.ora1) lfsck.c as a workaround (not verified though).
--- e2fsprogs/e2fsck/lfsck.c 2010-11-12 11:43:42.000000000 +0800
+++ lfsck.c 2010-11-12 12:14:38.000000000 +0800
@@ -1226,6 +1226,12 @@
__u64 last_id;
int i, rc;
+ /* skip empty UUID OST */
+ if(!strlen(lfsck_uuid[ost_idx].uuid)) {
+ log_write("index %d UUID is empty(sparse OST index?).
Skipping.\n", ost_idx);
+ return(0);
+ }
+
sprintf(dbname, "%s.%d", MDS_OSTDB, ost_idx);
VERBOSE(2, "testing ost_idx %d\n", ost_idx);
@@ -1279,11 +1284,20 @@
ost_hdr->ost_uuid.uuid);
if (obd_uuid_equals(&lfsck_uuid[ost_idx],
&ost_hdr->ost_uuid)) {
+ /* must be sparse ost index */
if (ost_hdr->ost_index != ost_idx) {
log_write("Requested ost_idx %u doesn''t match
"
"index %u found in %s\n", ost_idx,
ost_hdr->ost_index, ost_files[i]);
- continue;
+
+ log_write("Moving the index/uuid to the right
place...\n");
+ /* zero the original uuid entry */
+ memset(&lfsck_uuid[ost_idx], 0, sizeof(struct obd_uuid));
+ /* copy it to the right place */
+ ost_idx = ost_hdr->ost_index;
+
strcpy(lfsck_uuid[ost_hdr->ost_index].uuid,ost_hdr->ost_uuid.uuid);
+ /* skip this round */
+ goto out;
}
break;
? 2010-11-12???10:53? Christopher Walker ???
> Thanks very much for your reply. I''ve tried remaking the mdsdb and
all
> of the ostdb''s, but I still get the same error -- it checks the
first 34
> osts without a problem, but can''t find the ostdb file for the 35th
> (which has ost_idx 42):
>
> ...
> lfsck: ost_idx 34: pass3 OK (676803 files total)
> lfsck: can''t find file for ost_idx 35
> Files affected by missing ost info are : -
> lfsck: can''t find file for ost_idx 36
> Files affected by missing ost info are : -
> lfsck: can''t find file for ost_idx 37
> Files affected by missing ost info are : -
> lfsck: can''t find file for ost_idx 38
> Files affected by missing ost info are : -
> lfsck: can''t find file for ost_idx 39
> Files affected by missing ost info are : -
> lfsck: can''t find file for ost_idx 40
> Files affected by missing ost info are : -
> lfsck: can''t find file for ost_idx 41
> Files affected by missing ost info are : -
> lfsck: can''t find file for ost_idx 42
> Files affected by missing ost info are : -
> ...
>
> e2fsck claims to be making the ostdb without a problem:
>
> Pass 6: Acquiring information for lfsck
> OST: ''aegalfs-OST002a_UUID'' ost idx 42: compat 0x2 rocomp
0 incomp 0x2
> OST: num files = 676803
> OST: last_id = 858163
>
> and with the filesystem up I can see files on this OST:
>
> [cwalker at iliadaccess04 P-Gadget3.3.1]$ lfs getstripe predict.c
> OBDS:
> 0: aegalfs-OST0000_UUID ACTIVE
> ...
> 33: aegalfs-OST0021_UUID ACTIVE
> 42: aegalfs-OST002a_UUID ACTIVE
> predict.c
> obdidx objid objid group
> 42 10 0xa 0
>
>
> lfsck identifies several hundred GB of orphan data that we''d like
to
> recover, so we''d really like to run lfsck on this array.
We''re willing
> to forgo the recovery on the 35th ost, but I want to make sure that
> running lfsck -l with the current configuration won''t make things
worse.
>
> Thanks again for your reply; any further advice is very much appreciated!
>
> Best,
> Chris
>
> On 11/10/10 12:10 AM, Wang Yibin wrote:
>> The error message indicates that the UUID of OST #35 does not match
between the live filesystem and the ostdb file.
>> Is this ostdb obsolete?
>>
>> ? 2010-11-9???11:45? Christopher Walker ???
>>
>>>
>>> For reasons that I can''t recall, our OSTs are not in
consecutive order
>>> -- we have 35 OSTs, which are numbered consecutively from
>>> 0000-0021
>>> and then there''s one last OST at
>>> 002a
>>>
>>> When I try to run lfsck on this array, it works fine for the first
34
>>> OSTs, but it can''t seem to find the last OST db file:
>>>
>>> lfsck: ost_idx 34: pass3 OK (680045 files total)
>>> lfsck: can''t find file for ost_idx 35
>>> Files affected by missing ost info are : -
>>> lfsck: can''t find file for ost_idx 36
>>> Files affected by missing ost info are : -
>>> lfsck: can''t find file for ost_idx 37
>>> Files affected by missing ost info are : -
>>> lfsck: can''t find file for ost_idx 38
>>> Files affected by missing ost info are : -
>>> lfsck: can''t find file for ost_idx 39
>>> Files affected by missing ost info are : -
>>> lfsck: can''t find file for ost_idx 40
>>> Files affected by missing ost info are : -
>>> lfsck: can''t find file for ost_idx 41
>>> Files affected by missing ost info are : -
>>> lfsck: can''t find file for ost_idx 42
>>> Files affected by missing ost info are : -
>>>
/n/scratch/hernquist_lab/tcox/tests/SbSbhs_e_8/P-Gadget3.3.1/IdlSubfind/.svn/text-base/ReadSubhaloFromReshuffledSnapshot.pro.svn-base
>>>
>>> and then lists all of the files that live on OST 002a. This db
file
>>> definitely does exist -- it lives in the same directory as all of
the
>>> other db files, and e2fsck for this OST ran without problems.
>>>
>>> Is there some way of forcing lfsck to recognize this OST db? Or,
>>> failing that, is it dangerous to run lfsck on the first 34 OSTs
only?
>>>
>>> We''re using e2fsck 1.41.6.sun1 (30-May-2009)
>>>
>>> Thanks very much!
>>>
>>> Chris
>>>
>>>
>>> _______________________________________________
>>> Lustre-discuss mailing list
>>> Lustre-discuss at lists.lustre.org
>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
Thanks Andreas. The orphan data is scattered throughout the array, although it''s primarily on one OST (30) which seems to have been hit particularly hard by this outage: [root at iliadaccess04 lfsck2]# grep ERROR lfsck2.out lfsck: ost_idx 5: pass2 ERROR: 3817 dangling inodes found (654297 files total) lfsck: ost_idx 6: pass2 ERROR: 2 dangling inodes found (670416 files total) lfsck: ost_idx 11: pass2 ERROR: 942 dangling inodes found (673425 files total) lfsck: ost_idx 12: pass2 ERROR: 64 dangling inodes found (678878 files total) [lfsck: ost_idx 13: pass3 ERROR: 24.2109MB of orphan data (6 of 776532 files total) lfsck: ost_idx 14: pass3 ERROR: 5.34375MB of orphan data (2 of 725085 files total) lfsck: ost_idx 15: pass2 ERROR: 1 dangling inodes found (672942 files total) lfsck: ost_idx 15: pass3 ERROR: 58.4688MB of orphan data (6 of 739995 files total) lfsck: ost_idx 16: pass2 ERROR: 1 dangling inodes found (671379 files total) lfsck: ost_idx 18: pass2 ERROR: 3371 dangling inodes found (692018 files total) lfsck: ost_idx 18: pass3 ERROR: 5499.86MB of orphan data (620 of 688965 files total) lfsck: ost_idx 19: pass3 ERROR: 21.375MB of orphan data (8 of 775964 files total) [20] lfsck: ost_idx 20: pass3 ERROR: 3433.61MB of orphan data (16 of 843328 files total) [22] zero-length orphan objilfsck: ost_idx 22: pass3 ERROR: 1.21094MB of orphan data (16 of 859527 files total) lfsck: ost_idx 23: pass2 ERROR: 1 dangling inodes found (663492 files total) [23] zero-length orphan oblfsck: ost_idx 23: pass3 ERROR: 8571.68MB of orphan data (20 of 838490 files total) [24] zero-length orphan objid 83735lfsck: ost_idx 24: pass3 ERROR: 4367.45MB of orphan data (16 of 837371 files total) [25] zero-length orphan objid lfsck: ost_idx 25: pass3 ERROR: 121.996MB of orphan data (16 of 858679 files total) lfsck: ost_idx 30: pass2 ERROR: 46700 dangling inodes found (682467 files total) lfsck: ost_idx 30: pass3 ERROR: 45313.4MB of orphan data (7648 of 668343 files total) [root at iliadaccess04 lfsck2]# Thanks again, Chris On 11/12/10 3:05 AM, Andreas Dilger wrote:> On 2010-11-11, at 19:53, Christopher Walker wrote: >> Thanks very much for your reply. I''ve tried remaking the mdsdb and all >> of the ostdb''s, but I still get the same error -- it checks the first 34 >> osts without a problem, but can''t find the ostdb file for the 35th >> (which has ost_idx 42): >> >> with the filesystem up I can see files on this OST: >> >> [cwalker at iliadaccess04 P-Gadget3.3.1]$ lfs getstripe predict.c >> OBDS: >> 0: aegalfs-OST0000_UUID ACTIVE >> ... >> 33: aegalfs-OST0021_UUID ACTIVE >> 42: aegalfs-OST002a_UUID ACTIVE >> predict.c >> obdidx objid objid group >> 42 10 0xa 0 >> >> >> lfsck identifies several hundred GB of orphan data that we''d like to >> recover, so we''d really like to run lfsck on this array. We''re willing >> to forgo the recovery on the 35th ost, but I want to make sure that >> running lfsck -l with the current configuration won''t make things worse. > I''m not sure that what lfsck is reporting in this case is correct. Is the orphan data all on the same OST, or spread around separate OSTs? My concern is that if lfsck thinks the in-use objects on your estranged OST is actually orphan data it will destroy that data. > > If there are a small number of very large objects on other OSTs that are making up the bulk of the orphan space usage, you could mount those OSTs as type ldiskfs and delete the objects by hand to free up the space. > > Cheers, Andreas > -- > Andreas Dilger > Lustre Technical Lead > Oracle Corporation Canada Inc. >
Thanks *very* much -- I''ll give this a shot later today and let you know how it goes. Best, Chris On 11/12/10 3:17 AM, Wang Yibin wrote:> This is a bug in llapi_lov_get_uuids() which assigns UUID to the wrong OST index when there are sparse OST(s). > Please file a bug for this. > > Before this bug can be fixed, you can apply the following patch to e2fsprogs(version 1.41.12.2.ora1) lfsck.c as a workaround (not verified though). > > --- e2fsprogs/e2fsck/lfsck.c 2010-11-12 11:43:42.000000000 +0800 > +++ lfsck.c 2010-11-12 12:14:38.000000000 +0800 > @@ -1226,6 +1226,12 @@ > __u64 last_id; > int i, rc; > > + /* skip empty UUID OST */ > + if(!strlen(lfsck_uuid[ost_idx].uuid)) { > + log_write("index %d UUID is empty(sparse OST index?). Skipping.\n", ost_idx); > + return(0); > + } > + > sprintf(dbname, "%s.%d", MDS_OSTDB, ost_idx); > > VERBOSE(2, "testing ost_idx %d\n", ost_idx); > @@ -1279,11 +1284,20 @@ > ost_hdr->ost_uuid.uuid); > > if (obd_uuid_equals(&lfsck_uuid[ost_idx], &ost_hdr->ost_uuid)) { > + /* must be sparse ost index */ > if (ost_hdr->ost_index != ost_idx) { > log_write("Requested ost_idx %u doesn''t match " > "index %u found in %s\n", ost_idx, > ost_hdr->ost_index, ost_files[i]); > - continue; > + > + log_write("Moving the index/uuid to the right place...\n"); > + /* zero the original uuid entry */ > + memset(&lfsck_uuid[ost_idx], 0, sizeof(struct obd_uuid)); > + /* copy it to the right place */ > + ost_idx = ost_hdr->ost_index; > + strcpy(lfsck_uuid[ost_hdr->ost_index].uuid,ost_hdr->ost_uuid.uuid); > + /* skip this round */ > + goto out; > } > > break; > > > ? 2010-11-12???10:53? Christopher Walker ??? > >> Thanks very much for your reply. I''ve tried remaking the mdsdb and all >> of the ostdb''s, but I still get the same error -- it checks the first 34 >> osts without a problem, but can''t find the ostdb file for the 35th >> (which has ost_idx 42): >> >> ... >> lfsck: ost_idx 34: pass3 OK (676803 files total) >> lfsck: can''t find file for ost_idx 35 >> Files affected by missing ost info are : - >> lfsck: can''t find file for ost_idx 36 >> Files affected by missing ost info are : - >> lfsck: can''t find file for ost_idx 37 >> Files affected by missing ost info are : - >> lfsck: can''t find file for ost_idx 38 >> Files affected by missing ost info are : - >> lfsck: can''t find file for ost_idx 39 >> Files affected by missing ost info are : - >> lfsck: can''t find file for ost_idx 40 >> Files affected by missing ost info are : - >> lfsck: can''t find file for ost_idx 41 >> Files affected by missing ost info are : - >> lfsck: can''t find file for ost_idx 42 >> Files affected by missing ost info are : - >> ... >> >> e2fsck claims to be making the ostdb without a problem: >> >> Pass 6: Acquiring information for lfsck >> OST: ''aegalfs-OST002a_UUID'' ost idx 42: compat 0x2 rocomp 0 incomp 0x2 >> OST: num files = 676803 >> OST: last_id = 858163 >> >> and with the filesystem up I can see files on this OST: >> >> [cwalker at iliadaccess04 P-Gadget3.3.1]$ lfs getstripe predict.c >> OBDS: >> 0: aegalfs-OST0000_UUID ACTIVE >> ... >> 33: aegalfs-OST0021_UUID ACTIVE >> 42: aegalfs-OST002a_UUID ACTIVE >> predict.c >> obdidx objid objid group >> 42 10 0xa 0 >> >> >> lfsck identifies several hundred GB of orphan data that we''d like to >> recover, so we''d really like to run lfsck on this array. We''re willing >> to forgo the recovery on the 35th ost, but I want to make sure that >> running lfsck -l with the current configuration won''t make things worse. >> >> Thanks again for your reply; any further advice is very much appreciated! >> >> Best, >> Chris >> >> On 11/10/10 12:10 AM, Wang Yibin wrote: >>> The error message indicates that the UUID of OST #35 does not match between the live filesystem and the ostdb file. >>> Is this ostdb obsolete? >>> >>> ? 2010-11-9???11:45? Christopher Walker ??? >>> >>>> For reasons that I can''t recall, our OSTs are not in consecutive order >>>> -- we have 35 OSTs, which are numbered consecutively from >>>> 0000-0021 >>>> and then there''s one last OST at >>>> 002a >>>> >>>> When I try to run lfsck on this array, it works fine for the first 34 >>>> OSTs, but it can''t seem to find the last OST db file: >>>> >>>> lfsck: ost_idx 34: pass3 OK (680045 files total) >>>> lfsck: can''t find file for ost_idx 35 >>>> Files affected by missing ost info are : - >>>> lfsck: can''t find file for ost_idx 36 >>>> Files affected by missing ost info are : - >>>> lfsck: can''t find file for ost_idx 37 >>>> Files affected by missing ost info are : - >>>> lfsck: can''t find file for ost_idx 38 >>>> Files affected by missing ost info are : - >>>> lfsck: can''t find file for ost_idx 39 >>>> Files affected by missing ost info are : - >>>> lfsck: can''t find file for ost_idx 40 >>>> Files affected by missing ost info are : - >>>> lfsck: can''t find file for ost_idx 41 >>>> Files affected by missing ost info are : - >>>> lfsck: can''t find file for ost_idx 42 >>>> Files affected by missing ost info are : - >>>> /n/scratch/hernquist_lab/tcox/tests/SbSbhs_e_8/P-Gadget3.3.1/IdlSubfind/.svn/text-base/ReadSubhaloFromReshuffledSnapshot.pro.svn-base >>>> >>>> and then lists all of the files that live on OST 002a. This db file >>>> definitely does exist -- it lives in the same directory as all of the >>>> other db files, and e2fsck for this OST ran without problems. >>>> >>>> Is there some way of forcing lfsck to recognize this OST db? Or, >>>> failing that, is it dangerous to run lfsck on the first 34 OSTs only? >>>> >>>> We''re using e2fsck 1.41.6.sun1 (30-May-2009) >>>> >>>> Thanks very much! >>>> >>>> Chris >>>> >>>> >>>> _______________________________________________ >>>> Lustre-discuss mailing list >>>> Lustre-discuss at lists.lustre.org >>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
Thanks again for this patch. I just have one quick question about this
-- 1.41.12.2.ora1 seems to require lustre_user.h from 1.8.x -- is OK to
use a version of lfsck compiled against 1.8.x on a 1.6.6 filesystem, and
with {mds,ost}db that were created with 1.41.6?
Best,
Chris
On 11/12/10 3:17 AM, Wang Yibin wrote:> This is a bug in llapi_lov_get_uuids() which assigns UUID to the wrong OST
index when there are sparse OST(s).
> Please file a bug for this.
>
> Before this bug can be fixed, you can apply the following patch to
e2fsprogs(version 1.41.12.2.ora1) lfsck.c as a workaround (not verified though).
>
> --- e2fsprogs/e2fsck/lfsck.c 2010-11-12 11:43:42.000000000 +0800
> +++ lfsck.c 2010-11-12 12:14:38.000000000 +0800
> @@ -1226,6 +1226,12 @@
> __u64 last_id;
> int i, rc;
>
> + /* skip empty UUID OST */
> + if(!strlen(lfsck_uuid[ost_idx].uuid)) {
> + log_write("index %d UUID is empty(sparse OST index?).
Skipping.\n", ost_idx);
> + return(0);
> + }
> +
> sprintf(dbname, "%s.%d", MDS_OSTDB, ost_idx);
>
> VERBOSE(2, "testing ost_idx %d\n", ost_idx);
> @@ -1279,11 +1284,20 @@
> ost_hdr->ost_uuid.uuid);
>
> if (obd_uuid_equals(&lfsck_uuid[ost_idx],
&ost_hdr->ost_uuid)) {
> + /* must be sparse ost index */
> if (ost_hdr->ost_index != ost_idx) {
> log_write("Requested ost_idx %u doesn''t match
"
> "index %u found in %s\n", ost_idx,
> ost_hdr->ost_index, ost_files[i]);
> - continue;
> +
> + log_write("Moving the index/uuid to the right
place...\n");
> + /* zero the original uuid entry */
> + memset(&lfsck_uuid[ost_idx], 0, sizeof(struct
obd_uuid));
> + /* copy it to the right place */
> + ost_idx = ost_hdr->ost_index;
> +
strcpy(lfsck_uuid[ost_hdr->ost_index].uuid,ost_hdr->ost_uuid.uuid);
> + /* skip this round */
> + goto out;
> }
>
> break;
>
>
> ? 2010-11-12???10:53? Christopher Walker ???
>
>> Thanks very much for your reply. I''ve tried remaking the mdsdb
and all
>> of the ostdb''s, but I still get the same error -- it checks
the first 34
>> osts without a problem, but can''t find the ostdb file for the
35th
>> (which has ost_idx 42):
>>
>> ...
>> lfsck: ost_idx 34: pass3 OK (676803 files total)
>> lfsck: can''t find file for ost_idx 35
>> Files affected by missing ost info are : -
>> lfsck: can''t find file for ost_idx 36
>> Files affected by missing ost info are : -
>> lfsck: can''t find file for ost_idx 37
>> Files affected by missing ost info are : -
>> lfsck: can''t find file for ost_idx 38
>> Files affected by missing ost info are : -
>> lfsck: can''t find file for ost_idx 39
>> Files affected by missing ost info are : -
>> lfsck: can''t find file for ost_idx 40
>> Files affected by missing ost info are : -
>> lfsck: can''t find file for ost_idx 41
>> Files affected by missing ost info are : -
>> lfsck: can''t find file for ost_idx 42
>> Files affected by missing ost info are : -
>> ...
>>
>> e2fsck claims to be making the ostdb without a problem:
>>
>> Pass 6: Acquiring information for lfsck
>> OST: ''aegalfs-OST002a_UUID'' ost idx 42: compat 0x2
rocomp 0 incomp 0x2
>> OST: num files = 676803
>> OST: last_id = 858163
>>
>> and with the filesystem up I can see files on this OST:
>>
>> [cwalker at iliadaccess04 P-Gadget3.3.1]$ lfs getstripe predict.c
>> OBDS:
>> 0: aegalfs-OST0000_UUID ACTIVE
>> ...
>> 33: aegalfs-OST0021_UUID ACTIVE
>> 42: aegalfs-OST002a_UUID ACTIVE
>> predict.c
>> obdidx objid objid group
>> 42 10 0xa 0
>>
>>
>> lfsck identifies several hundred GB of orphan data that we''d
like to
>> recover, so we''d really like to run lfsck on this array.
We''re willing
>> to forgo the recovery on the 35th ost, but I want to make sure that
>> running lfsck -l with the current configuration won''t make
things worse.
>>
>> Thanks again for your reply; any further advice is very much
appreciated!
>>
>> Best,
>> Chris
>>
>> On 11/10/10 12:10 AM, Wang Yibin wrote:
>>> The error message indicates that the UUID of OST #35 does not match
between the live filesystem and the ostdb file.
>>> Is this ostdb obsolete?
>>>
>>> ? 2010-11-9???11:45? Christopher Walker ???
>>>
>>>> For reasons that I can''t recall, our OSTs are not in
consecutive order
>>>> -- we have 35 OSTs, which are numbered consecutively from
>>>> 0000-0021
>>>> and then there''s one last OST at
>>>> 002a
>>>>
>>>> When I try to run lfsck on this array, it works fine for the
first 34
>>>> OSTs, but it can''t seem to find the last OST db file:
>>>>
>>>> lfsck: ost_idx 34: pass3 OK (680045 files total)
>>>> lfsck: can''t find file for ost_idx 35
>>>> Files affected by missing ost info are : -
>>>> lfsck: can''t find file for ost_idx 36
>>>> Files affected by missing ost info are : -
>>>> lfsck: can''t find file for ost_idx 37
>>>> Files affected by missing ost info are : -
>>>> lfsck: can''t find file for ost_idx 38
>>>> Files affected by missing ost info are : -
>>>> lfsck: can''t find file for ost_idx 39
>>>> Files affected by missing ost info are : -
>>>> lfsck: can''t find file for ost_idx 40
>>>> Files affected by missing ost info are : -
>>>> lfsck: can''t find file for ost_idx 41
>>>> Files affected by missing ost info are : -
>>>> lfsck: can''t find file for ost_idx 42
>>>> Files affected by missing ost info are : -
>>>>
/n/scratch/hernquist_lab/tcox/tests/SbSbhs_e_8/P-Gadget3.3.1/IdlSubfind/.svn/text-base/ReadSubhaloFromReshuffledSnapshot.pro.svn-base
>>>>
>>>> and then lists all of the files that live on OST 002a. This db
file
>>>> definitely does exist -- it lives in the same directory as all
of the
>>>> other db files, and e2fsck for this OST ran without problems.
>>>>
>>>> Is there some way of forcing lfsck to recognize this OST db?
Or,
>>>> failing that, is it dangerous to run lfsck on the first 34 OSTs
only?
>>>>
>>>> We''re using e2fsck 1.41.6.sun1 (30-May-2009)
>>>>
>>>> Thanks very much!
>>>>
>>>> Chris
>>>>
>>>>
>>>> _______________________________________________
>>>> Lustre-discuss mailing list
>>>> Lustre-discuss at lists.lustre.org
>>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
For the moment, without investigation, I am not sure about this - There may or may not be compatibility issue. Please checkout the version of the e2fsprogs which is identical with that on your system and patch against the lfsck.c accordingly. Then you can compile against 1.6.6. ? 2010-11-13???11:20? Christopher Walker ???> > Thanks again for this patch. I just have one quick question about this > -- 1.41.12.2.ora1 seems to require lustre_user.h from 1.8.x -- is OK to > use a version of lfsck compiled against 1.8.x on a 1.6.6 filesystem, and > with {mds,ost}db that were created with 1.41.6? > > Best, > Chris > > On 11/12/10 3:17 AM, Wang Yibin wrote: >> This is a bug in llapi_lov_get_uuids() which assigns UUID to the wrong OST index when there are sparse OST(s). >> Please file a bug for this. >> >> Before this bug can be fixed, you can apply the following patch to e2fsprogs(version 1.41.12.2.ora1) lfsck.c as a workaround (not verified though). >> >> --- e2fsprogs/e2fsck/lfsck.c 2010-11-12 11:43:42.000000000 +0800 >> +++ lfsck.c 2010-11-12 12:14:38.000000000 +0800 >> @@ -1226,6 +1226,12 @@ >> __u64 last_id; >> int i, rc; >> >> + /* skip empty UUID OST */ >> + if(!strlen(lfsck_uuid[ost_idx].uuid)) { >> + log_write("index %d UUID is empty(sparse OST index?). Skipping.\n", ost_idx); >> + return(0); >> + } >> + >> sprintf(dbname, "%s.%d", MDS_OSTDB, ost_idx); >> >> VERBOSE(2, "testing ost_idx %d\n", ost_idx); >> @@ -1279,11 +1284,20 @@ >> ost_hdr->ost_uuid.uuid); >> >> if (obd_uuid_equals(&lfsck_uuid[ost_idx], &ost_hdr->ost_uuid)) { >> + /* must be sparse ost index */ >> if (ost_hdr->ost_index != ost_idx) { >> log_write("Requested ost_idx %u doesn''t match " >> "index %u found in %s\n", ost_idx, >> ost_hdr->ost_index, ost_files[i]); >> - continue; >> + >> + log_write("Moving the index/uuid to the right place...\n"); >> + /* zero the original uuid entry */ >> + memset(&lfsck_uuid[ost_idx], 0, sizeof(struct obd_uuid)); >> + /* copy it to the right place */ >> + ost_idx = ost_hdr->ost_index; >> + strcpy(lfsck_uuid[ost_hdr->ost_index].uuid,ost_hdr->ost_uuid.uuid); >> + /* skip this round */ >> + goto out; >> } >> >> break; >> >> >> ? 2010-11-12???10:53? Christopher Walker ??? >> >>> Thanks very much for your reply. I''ve tried remaking the mdsdb and all >>> of the ostdb''s, but I still get the same error -- it checks the first 34 >>> osts without a problem, but can''t find the ostdb file for the 35th >>> (which has ost_idx 42): >>> >>> ... >>> lfsck: ost_idx 34: pass3 OK (676803 files total) >>> lfsck: can''t find file for ost_idx 35 >>> Files affected by missing ost info are : - >>> lfsck: can''t find file for ost_idx 36 >>> Files affected by missing ost info are : - >>> lfsck: can''t find file for ost_idx 37 >>> Files affected by missing ost info are : - >>> lfsck: can''t find file for ost_idx 38 >>> Files affected by missing ost info are : - >>> lfsck: can''t find file for ost_idx 39 >>> Files affected by missing ost info are : - >>> lfsck: can''t find file for ost_idx 40 >>> Files affected by missing ost info are : - >>> lfsck: can''t find file for ost_idx 41 >>> Files affected by missing ost info are : - >>> lfsck: can''t find file for ost_idx 42 >>> Files affected by missing ost info are : - >>> ... >>> >>> e2fsck claims to be making the ostdb without a problem: >>> >>> Pass 6: Acquiring information for lfsck >>> OST: ''aegalfs-OST002a_UUID'' ost idx 42: compat 0x2 rocomp 0 incomp 0x2 >>> OST: num files = 676803 >>> OST: last_id = 858163 >>> >>> and with the filesystem up I can see files on this OST: >>> >>> [cwalker at iliadaccess04 P-Gadget3.3.1]$ lfs getstripe predict.c >>> OBDS: >>> 0: aegalfs-OST0000_UUID ACTIVE >>> ... >>> 33: aegalfs-OST0021_UUID ACTIVE >>> 42: aegalfs-OST002a_UUID ACTIVE >>> predict.c >>> obdidx objid objid group >>> 42 10 0xa 0 >>> >>> >>> lfsck identifies several hundred GB of orphan data that we''d like to >>> recover, so we''d really like to run lfsck on this array. We''re willing >>> to forgo the recovery on the 35th ost, but I want to make sure that >>> running lfsck -l with the current configuration won''t make things worse. >>> >>> Thanks again for your reply; any further advice is very much appreciated! >>> >>> Best, >>> Chris >>> >>> On 11/10/10 12:10 AM, Wang Yibin wrote: >>>> The error message indicates that the UUID of OST #35 does not match between the live filesystem and the ostdb file. >>>> Is this ostdb obsolete? >>>> >>>> ? 2010-11-9???11:45? Christopher Walker ??? >>>> >>>>> For reasons that I can''t recall, our OSTs are not in consecutive order >>>>> -- we have 35 OSTs, which are numbered consecutively from >>>>> 0000-0021 >>>>> and then there''s one last OST at >>>>> 002a >>>>> >>>>> When I try to run lfsck on this array, it works fine for the first 34 >>>>> OSTs, but it can''t seem to find the last OST db file: >>>>> >>>>> lfsck: ost_idx 34: pass3 OK (680045 files total) >>>>> lfsck: can''t find file for ost_idx 35 >>>>> Files affected by missing ost info are : - >>>>> lfsck: can''t find file for ost_idx 36 >>>>> Files affected by missing ost info are : - >>>>> lfsck: can''t find file for ost_idx 37 >>>>> Files affected by missing ost info are : - >>>>> lfsck: can''t find file for ost_idx 38 >>>>> Files affected by missing ost info are : - >>>>> lfsck: can''t find file for ost_idx 39 >>>>> Files affected by missing ost info are : - >>>>> lfsck: can''t find file for ost_idx 40 >>>>> Files affected by missing ost info are : - >>>>> lfsck: can''t find file for ost_idx 41 >>>>> Files affected by missing ost info are : - >>>>> lfsck: can''t find file for ost_idx 42 >>>>> Files affected by missing ost info are : - >>>>> /n/scratch/hernquist_lab/tcox/tests/SbSbhs_e_8/P-Gadget3.3.1/IdlSubfind/.svn/text-base/ReadSubhaloFromReshuffledSnapshot.pro.svn-base >>>>> >>>>> and then lists all of the files that live on OST 002a. This db file >>>>> definitely does exist -- it lives in the same directory as all of the >>>>> other db files, and e2fsck for this OST ran without problems. >>>>> >>>>> Is there some way of forcing lfsck to recognize this OST db? Or, >>>>> failing that, is it dangerous to run lfsck on the first 34 OSTs only? >>>>> >>>>> We''re using e2fsck 1.41.6.sun1 (30-May-2009) >>>>> >>>>> Thanks very much! >>>>> >>>>> Chris >>>>> >>>>> >>>>> _______________________________________________ >>>>> Lustre-discuss mailing list >>>>> Lustre-discuss at lists.lustre.org >>>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss
Thanks again for the advice, and for putting this patch together so quickly. I put your patch into 1.41.6.sun1 and it works perfectly, at least in read-only mode. Unfortunately, when I run lfsck -l on this array, it gets to ost_idx 30, runs through part of the check, but then hangs while checking for orphan objects: lfsck: ost_idx 30: pass2 ERROR: 46700 dangling inodes found (682467 files total) lfsck: ost_idx 30: pass3: check for orphan objects it hung right around the same time that an LBUG appeared on the MDS: Nov 13 21:58:39 aegamds1 kernel: LustreError: 21864:0:(lov_pack.c:179:lov_packmd()) ASSERTION(loi->loi_id) failed:lmm_oid 78132063 stripe 0/1 idx 30 Nov 13 21:58:39 aegamds1 kernel: LustreError: 21864:0:(lov_pack.c:179:lov_packmd()) LBUG Judging from the number of dangling inodes, there''s looks to be something quite wrong with this OST (although it produces no errors when run through e2fsck -fn). The last few lines of the lustre-log are: 00000004:00080000:1:1289703502.347853:0:21853:0:(obd.h:1191:obd_transno_commit_cb()) aegalfs-MDT0000: transno 4267034979 committed 02000000:00080000:0:1289703519.748478:0:21877:0:(upcall_cache.c:185:refresh_entry()) aegalfs-MDT0000: invoked upcall /usr/sbin/l_getgroups aegalfs-MDT0000 0 00020000:00040000:4:1289703519.876199:0:21864:0:(lov_pack.c:179:lov_packmd()) ASSERTION(loi->loi_id) failed:lmm_oid 78132063 stripe 0/1 idx 30 00000000:00040000:4:1289703519.876652:0:21864:0:(lov_pack.c:179:lov_packmd()) LBUG 00000400:00000400:4:1289703519.876877:0:21864:0:(linux-debug.c:185:libcfs_debug_dumpstack()) showing stack for process 21864 Debug log: 212116 lines, 212116 kept, 0 dropped. [root at aegamds1 ~]# is 78132063 an objid that lfsck doesn''t like? Is deleting the associated file (assuming that I can find it) the best path forward? Thanks again, Chris On 11/12/10 11:02 PM, Wang Yibin wrote:> For the moment, without investigation, I am not sure about this - There may or may not be compatibility issue. > Please checkout the version of the e2fsprogs which is identical with that on your system and patch against the lfsck.c accordingly. > Then you can compile against 1.6.6. > > ? 2010-11-13???11:20? Christopher Walker ??? > >> Thanks again for this patch. I just have one quick question about this >> -- 1.41.12.2.ora1 seems to require lustre_user.h from 1.8.x -- is OK to >> use a version of lfsck compiled against 1.8.x on a 1.6.6 filesystem, and >> with {mds,ost}db that were created with 1.41.6? >> >> Best, >> Chris >> >> On 11/12/10 3:17 AM, Wang Yibin wrote: >>> This is a bug in llapi_lov_get_uuids() which assigns UUID to the wrong OST index when there are sparse OST(s). >>> Please file a bug for this. >>> >>> Before this bug can be fixed, you can apply the following patch to e2fsprogs(version 1.41.12.2.ora1) lfsck.c as a workaround (not verified though). >>> >>> --- e2fsprogs/e2fsck/lfsck.c 2010-11-12 11:43:42.000000000 +0800 >>> +++ lfsck.c 2010-11-12 12:14:38.000000000 +0800 >>> @@ -1226,6 +1226,12 @@ >>> __u64 last_id; >>> int i, rc; >>> >>> + /* skip empty UUID OST */ >>> + if(!strlen(lfsck_uuid[ost_idx].uuid)) { >>> + log_write("index %d UUID is empty(sparse OST index?). Skipping.\n", ost_idx); >>> + return(0); >>> + } >>> + >>> sprintf(dbname, "%s.%d", MDS_OSTDB, ost_idx); >>> >>> VERBOSE(2, "testing ost_idx %d\n", ost_idx); >>> @@ -1279,11 +1284,20 @@ >>> ost_hdr->ost_uuid.uuid); >>> >>> if (obd_uuid_equals(&lfsck_uuid[ost_idx], &ost_hdr->ost_uuid)) { >>> + /* must be sparse ost index */ >>> if (ost_hdr->ost_index != ost_idx) { >>> log_write("Requested ost_idx %u doesn''t match " >>> "index %u found in %s\n", ost_idx, >>> ost_hdr->ost_index, ost_files[i]); >>> - continue; >>> + >>> + log_write("Moving the index/uuid to the right place...\n"); >>> + /* zero the original uuid entry */ >>> + memset(&lfsck_uuid[ost_idx], 0, sizeof(struct obd_uuid)); >>> + /* copy it to the right place */ >>> + ost_idx = ost_hdr->ost_index; >>> + strcpy(lfsck_uuid[ost_hdr->ost_index].uuid,ost_hdr->ost_uuid.uuid); >>> + /* skip this round */ >>> + goto out; >>> } >>> >>> break; >>> >>> >>> ? 2010-11-12???10:53? Christopher Walker ??? >>> >>>> Thanks very much for your reply. I''ve tried remaking the mdsdb and all >>>> of the ostdb''s, but I still get the same error -- it checks the first 34 >>>> osts without a problem, but can''t find the ostdb file for the 35th >>>> (which has ost_idx 42): >>>> >>>> ... >>>> lfsck: ost_idx 34: pass3 OK (676803 files total) >>>> lfsck: can''t find file for ost_idx 35 >>>> Files affected by missing ost info are : - >>>> lfsck: can''t find file for ost_idx 36 >>>> Files affected by missing ost info are : - >>>> lfsck: can''t find file for ost_idx 37 >>>> Files affected by missing ost info are : - >>>> lfsck: can''t find file for ost_idx 38 >>>> Files affected by missing ost info are : - >>>> lfsck: can''t find file for ost_idx 39 >>>> Files affected by missing ost info are : - >>>> lfsck: can''t find file for ost_idx 40 >>>> Files affected by missing ost info are : - >>>> lfsck: can''t find file for ost_idx 41 >>>> Files affected by missing ost info are : - >>>> lfsck: can''t find file for ost_idx 42 >>>> Files affected by missing ost info are : - >>>> ... >>>> >>>> e2fsck claims to be making the ostdb without a problem: >>>> >>>> Pass 6: Acquiring information for lfsck >>>> OST: ''aegalfs-OST002a_UUID'' ost idx 42: compat 0x2 rocomp 0 incomp 0x2 >>>> OST: num files = 676803 >>>> OST: last_id = 858163 >>>> >>>> and with the filesystem up I can see files on this OST: >>>> >>>> [cwalker at iliadaccess04 P-Gadget3.3.1]$ lfs getstripe predict.c >>>> OBDS: >>>> 0: aegalfs-OST0000_UUID ACTIVE >>>> ... >>>> 33: aegalfs-OST0021_UUID ACTIVE >>>> 42: aegalfs-OST002a_UUID ACTIVE >>>> predict.c >>>> obdidx objid objid group >>>> 42 10 0xa 0 >>>> >>>> >>>> lfsck identifies several hundred GB of orphan data that we''d like to >>>> recover, so we''d really like to run lfsck on this array. We''re willing >>>> to forgo the recovery on the 35th ost, but I want to make sure that >>>> running lfsck -l with the current configuration won''t make things worse. >>>> >>>> Thanks again for your reply; any further advice is very much appreciated! >>>> >>>> Best, >>>> Chris >>>> >>>> On 11/10/10 12:10 AM, Wang Yibin wrote: >>>>> The error message indicates that the UUID of OST #35 does not match between the live filesystem and the ostdb file. >>>>> Is this ostdb obsolete? >>>>> >>>>> ? 2010-11-9???11:45? Christopher Walker ??? >>>>> >>>>>> For reasons that I can''t recall, our OSTs are not in consecutive order >>>>>> -- we have 35 OSTs, which are numbered consecutively from >>>>>> 0000-0021 >>>>>> and then there''s one last OST at >>>>>> 002a >>>>>> >>>>>> When I try to run lfsck on this array, it works fine for the first 34 >>>>>> OSTs, but it can''t seem to find the last OST db file: >>>>>> >>>>>> lfsck: ost_idx 34: pass3 OK (680045 files total) >>>>>> lfsck: can''t find file for ost_idx 35 >>>>>> Files affected by missing ost info are : - >>>>>> lfsck: can''t find file for ost_idx 36 >>>>>> Files affected by missing ost info are : - >>>>>> lfsck: can''t find file for ost_idx 37 >>>>>> Files affected by missing ost info are : - >>>>>> lfsck: can''t find file for ost_idx 38 >>>>>> Files affected by missing ost info are : - >>>>>> lfsck: can''t find file for ost_idx 39 >>>>>> Files affected by missing ost info are : - >>>>>> lfsck: can''t find file for ost_idx 40 >>>>>> Files affected by missing ost info are : - >>>>>> lfsck: can''t find file for ost_idx 41 >>>>>> Files affected by missing ost info are : - >>>>>> lfsck: can''t find file for ost_idx 42 >>>>>> Files affected by missing ost info are : - >>>>>> /n/scratch/hernquist_lab/tcox/tests/SbSbhs_e_8/P-Gadget3.3.1/IdlSubfind/.svn/text-base/ReadSubhaloFromReshuffledSnapshot.pro.svn-base >>>>>> >>>>>> and then lists all of the files that live on OST 002a. This db file >>>>>> definitely does exist -- it lives in the same directory as all of the >>>>>> other db files, and e2fsck for this OST ran without problems. >>>>>> >>>>>> Is there some way of forcing lfsck to recognize this OST db? Or, >>>>>> failing that, is it dangerous to run lfsck on the first 34 OSTs only? >>>>>> >>>>>> We''re using e2fsck 1.41.6.sun1 (30-May-2009) >>>>>> >>>>>> Thanks very much! >>>>>> >>>>>> Chris >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Lustre-discuss mailing list >>>>>> Lustre-discuss at lists.lustre.org >>>>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss
This sounds like bug 20183 which was caused by race in object creation. Unfortunately the fix is not in Lustre release 1.6.6. You need to upgrade your filesystem to avoid hitting this LBUG again. To fix dangling inodes, if you are very sure that these objects are not useful, use ''-d'' option to delete them. Else use lfsck with ''-c'' option to create 0-length objects on the OSTs (for this particular case, OST 0x30). ? 2010-11-14???12:21? Christopher Walker ???> Thanks again for the advice, and for putting this patch together so > quickly. I put your patch into 1.41.6.sun1 and it works perfectly, at > least in read-only mode. > > Unfortunately, when I run lfsck -l on this array, it gets to ost_idx 30, > runs through part of the check, but then hangs while checking for orphan > objects: > > lfsck: ost_idx 30: pass2 ERROR: 46700 dangling inodes found (682467 > files total) > lfsck: ost_idx 30: pass3: check for orphan objects > > it hung right around the same time that an LBUG appeared on the MDS: > > Nov 13 21:58:39 aegamds1 kernel: LustreError: > 21864:0:(lov_pack.c:179:lov_packmd()) ASSERTION(loi->loi_id) > failed:lmm_oid 78132063 stripe 0/1 idx 30 > Nov 13 21:58:39 aegamds1 kernel: LustreError: > 21864:0:(lov_pack.c:179:lov_packmd()) LBUG > > Judging from the number of dangling inodes, there''s looks to be > something quite wrong with this OST (although it produces no errors when > run through e2fsck -fn). The last few lines of the lustre-log are: > > 00000004:00080000:1:1289703502.347853:0:21853:0:(obd.h:1191:obd_transno_commit_cb()) > aegalfs-MDT0000: transno 4267034979 committed > 02000000:00080000:0:1289703519.748478:0:21877:0:(upcall_cache.c:185:refresh_entry()) > aegalfs-MDT0000: invoked upcall /usr/sbin/l_getgroups aegalfs-MDT0000 0 > 00020000:00040000:4:1289703519.876199:0:21864:0:(lov_pack.c:179:lov_packmd()) > ASSERTION(loi->loi_id) failed:lmm_oid 78132063 stripe 0/1 idx 30 > 00000000:00040000:4:1289703519.876652:0:21864:0:(lov_pack.c:179:lov_packmd()) > LBUG > 00000400:00000400:4:1289703519.876877:0:21864:0:(linux-debug.c:185:libcfs_debug_dumpstack()) > showing stack for process 21864 > Debug log: 212116 lines, 212116 kept, 0 dropped. > [root at aegamds1 ~]# > > is 78132063 an objid that lfsck doesn''t like? Is deleting the associated > file (assuming that I can find it) the best path forward? > > Thanks again, > Chris > > On 11/12/10 11:02 PM, Wang Yibin wrote: >> For the moment, without investigation, I am not sure about this - There may or may not be compatibility issue. >> Please checkout the version of the e2fsprogs which is identical with that on your system and patch against the lfsck.c accordingly. >> Then you can compile against 1.6.6. >> >> ? 2010-11-13???11:20? Christopher Walker ??? >> >>> Thanks again for this patch. I just have one quick question about this >>> -- 1.41.12.2.ora1 seems to require lustre_user.h from 1.8.x -- is OK to >>> use a version of lfsck compiled against 1.8.x on a 1.6.6 filesystem, and >>> with {mds,ost}db that were created with 1.41.6? >>> >>> Best, >>> Chris >>> >>> On 11/12/10 3:17 AM, Wang Yibin wrote: >>>> This is a bug in llapi_lov_get_uuids() which assigns UUID to the wrong OST index when there are sparse OST(s). >>>> Please file a bug for this. >>>> >>>> Before this bug can be fixed, you can apply the following patch to e2fsprogs(version 1.41.12.2.ora1) lfsck.c as a workaround (not verified though). >>>> >>>> --- e2fsprogs/e2fsck/lfsck.c 2010-11-12 11:43:42.000000000 +0800 >>>> +++ lfsck.c 2010-11-12 12:14:38.000000000 +0800 >>>> @@ -1226,6 +1226,12 @@ >>>> __u64 last_id; >>>> int i, rc; >>>> >>>> + /* skip empty UUID OST */ >>>> + if(!strlen(lfsck_uuid[ost_idx].uuid)) { >>>> + log_write("index %d UUID is empty(sparse OST index?). Skipping.\n", ost_idx); >>>> + return(0); >>>> + } >>>> + >>>> sprintf(dbname, "%s.%d", MDS_OSTDB, ost_idx); >>>> >>>> VERBOSE(2, "testing ost_idx %d\n", ost_idx); >>>> @@ -1279,11 +1284,20 @@ >>>> ost_hdr->ost_uuid.uuid); >>>> >>>> if (obd_uuid_equals(&lfsck_uuid[ost_idx], &ost_hdr->ost_uuid)) { >>>> + /* must be sparse ost index */ >>>> if (ost_hdr->ost_index != ost_idx) { >>>> log_write("Requested ost_idx %u doesn''t match " >>>> "index %u found in %s\n", ost_idx, >>>> ost_hdr->ost_index, ost_files[i]); >>>> - continue; >>>> + >>>> + log_write("Moving the index/uuid to the right place...\n"); >>>> + /* zero the original uuid entry */ >>>> + memset(&lfsck_uuid[ost_idx], 0, sizeof(struct obd_uuid)); >>>> + /* copy it to the right place */ >>>> + ost_idx = ost_hdr->ost_index; >>>> + strcpy(lfsck_uuid[ost_hdr->ost_index].uuid,ost_hdr->ost_uuid.uuid); >>>> + /* skip this round */ >>>> + goto out; >>>> } >>>> >>>> break; >>>> >>>> >>>> ? 2010-11-12???10:53? Christopher Walker ??? >>>> >>>>> Thanks very much for your reply. I''ve tried remaking the mdsdb and all >>>>> of the ostdb''s, but I still get the same error -- it checks the first 34 >>>>> osts without a problem, but can''t find the ostdb file for the 35th >>>>> (which has ost_idx 42): >>>>> >>>>> ... >>>>> lfsck: ost_idx 34: pass3 OK (676803 files total) >>>>> lfsck: can''t find file for ost_idx 35 >>>>> Files affected by missing ost info are : - >>>>> lfsck: can''t find file for ost_idx 36 >>>>> Files affected by missing ost info are : - >>>>> lfsck: can''t find file for ost_idx 37 >>>>> Files affected by missing ost info are : - >>>>> lfsck: can''t find file for ost_idx 38 >>>>> Files affected by missing ost info are : - >>>>> lfsck: can''t find file for ost_idx 39 >>>>> Files affected by missing ost info are : - >>>>> lfsck: can''t find file for ost_idx 40 >>>>> Files affected by missing ost info are : - >>>>> lfsck: can''t find file for ost_idx 41 >>>>> Files affected by missing ost info are : - >>>>> lfsck: can''t find file for ost_idx 42 >>>>> Files affected by missing ost info are : - >>>>> ... >>>>> >>>>> e2fsck claims to be making the ostdb without a problem: >>>>> >>>>> Pass 6: Acquiring information for lfsck >>>>> OST: ''aegalfs-OST002a_UUID'' ost idx 42: compat 0x2 rocomp 0 incomp 0x2 >>>>> OST: num files = 676803 >>>>> OST: last_id = 858163 >>>>> >>>>> and with the filesystem up I can see files on this OST: >>>>> >>>>> [cwalker at iliadaccess04 P-Gadget3.3.1]$ lfs getstripe predict.c >>>>> OBDS: >>>>> 0: aegalfs-OST0000_UUID ACTIVE >>>>> ... >>>>> 33: aegalfs-OST0021_UUID ACTIVE >>>>> 42: aegalfs-OST002a_UUID ACTIVE >>>>> predict.c >>>>> obdidx objid objid group >>>>> 42 10 0xa 0 >>>>> >>>>> >>>>> lfsck identifies several hundred GB of orphan data that we''d like to >>>>> recover, so we''d really like to run lfsck on this array. We''re willing >>>>> to forgo the recovery on the 35th ost, but I want to make sure that >>>>> running lfsck -l with the current configuration won''t make things worse. >>>>> >>>>> Thanks again for your reply; any further advice is very much appreciated! >>>>> >>>>> Best, >>>>> Chris >>>>> >>>>> On 11/10/10 12:10 AM, Wang Yibin wrote: >>>>>> The error message indicates that the UUID of OST #35 does not match between the live filesystem and the ostdb file. >>>>>> Is this ostdb obsolete? >>>>>> >>>>>> ? 2010-11-9???11:45? Christopher Walker ??? >>>>>> >>>>>>> For reasons that I can''t recall, our OSTs are not in consecutive order >>>>>>> -- we have 35 OSTs, which are numbered consecutively from >>>>>>> 0000-0021 >>>>>>> and then there''s one last OST at >>>>>>> 002a >>>>>>> >>>>>>> When I try to run lfsck on this array, it works fine for the first 34 >>>>>>> OSTs, but it can''t seem to find the last OST db file: >>>>>>> >>>>>>> lfsck: ost_idx 34: pass3 OK (680045 files total) >>>>>>> lfsck: can''t find file for ost_idx 35 >>>>>>> Files affected by missing ost info are : - >>>>>>> lfsck: can''t find file for ost_idx 36 >>>>>>> Files affected by missing ost info are : - >>>>>>> lfsck: can''t find file for ost_idx 37 >>>>>>> Files affected by missing ost info are : - >>>>>>> lfsck: can''t find file for ost_idx 38 >>>>>>> Files affected by missing ost info are : - >>>>>>> lfsck: can''t find file for ost_idx 39 >>>>>>> Files affected by missing ost info are : - >>>>>>> lfsck: can''t find file for ost_idx 40 >>>>>>> Files affected by missing ost info are : - >>>>>>> lfsck: can''t find file for ost_idx 41 >>>>>>> Files affected by missing ost info are : - >>>>>>> lfsck: can''t find file for ost_idx 42 >>>>>>> Files affected by missing ost info are : - >>>>>>> /n/scratch/hernquist_lab/tcox/tests/SbSbhs_e_8/P-Gadget3.3.1/IdlSubfind/.svn/text-base/ReadSubhaloFromReshuffledSnapshot.pro.svn-base >>>>>>> >>>>>>> and then lists all of the files that live on OST 002a. This db file >>>>>>> definitely does exist -- it lives in the same directory as all of the >>>>>>> other db files, and e2fsck for this OST ran without problems. >>>>>>> >>>>>>> Is there some way of forcing lfsck to recognize this OST db? Or, >>>>>>> failing that, is it dangerous to run lfsck on the first 34 OSTs only? >>>>>>> >>>>>>> We''re using e2fsck 1.41.6.sun1 (30-May-2009) >>>>>>> >>>>>>> Thanks very much! >>>>>>> >>>>>>> Chris >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Lustre-discuss mailing list >>>>>>> Lustre-discuss at lists.lustre.org >>>>>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss >>> _______________________________________________ >>> Lustre-discuss mailing list >>> Lustre-discuss at lists.lustre.org >>> http://lists.lustre.org/mailman/listinfo/lustre-discuss > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss