We were getting errors thrown by an OST. /var/log/messages contained a lot of these: 2010-11-28T17:05:34-05:00 umfs06.aglt2.org kernel: [2102640.735927] LDISKFS-fs error (device sdk): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 639corrupted: 440 blocks free in bitmap, 439 - in gd So, I turned off (most) access to the disk via lctl (we have a LOT of client machines, some were missed) and got problems. Had to use the alternate superblock to e2fsck the disk. When back online, I still saw similar messages. Updated to e2fsprogs 1.41.12 as suggested elsewhere. Repeated e2fsck. Still seeing these. Users report some files corrupted, coming up with bad md5sum.... Any other thoughts on what to do about this problem? [2440763.879143] LDISKFS-fs error (device sdk): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 35406corrupted: 1318 blocks free in bitmap, 1317 - in gd [2440763.879796] [2440763.882724] LustreError: 1651027:0:(fsfilt-ldiskfs.c:1333:fsfilt_ldiskfs_write_record()) can''t read/create block: -28 [2440763.882736] LustreError: 1651027:0:(llog_lvfs.c:116:llog_lvfs_write_blob()) error writing log record: rc -28 [2440763.882789] LustreError: 1651002:0:(mgc_request.c:1089:mgc_copy_llog()) Failed to copy remote log umt3-OST0019 (-28) Rebooted to make system clean as a whole, and found the same kind of thing repeating. [ 285.834864] LDISKFS-fs (sdk): warning: mounting fs with errors, running e2fsck is recommended [ 285.852559] LDISKFS-fs (sdk): mounted filesystem with ordered data mode [ 286.079065] LDISKFS-fs (sdk): warning: mounting fs with errors, running e2fsck is recommended [ 286.096316] LDISKFS-fs (sdk): mounted filesystem with ordered data mode [ 286.940872] LDISKFS-fs error (device sdk): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 35406corrupted: 1318 blocks free in bitmap, 1317 - in gd [ 286.941693] [ 286.945224] LustreError: 5790:0:(fsfilt-ldiskfs.c:1333:fsfilt_ldiskfs_write_record()) can''t read/create block: -28 [ 286.945233] LustreError: 5790:0:(llog_lvfs.c:116:llog_lvfs_write_blob()) error writing log record: rc -28 [ 286.945448] LustreError: 5763:0:(mgc_request.c:1089:mgc_copy_llog()) Failed to copy remote log umt3-OST0019 (-28) All help appreciated. bob
Hi Bob, If you''re seeing the same errors on the same disk after e2fsck run, and it''s not catching them, it''s possible that you''re hitting an edge case which isn''t handled within e2fsck properly, however if you''re experiencing different errors and e2fsck did catch them before, chances are you''re looking at some hardware failure some place. If this is a single disk, and you have SMART monitoring enabled, check your error counters, if it''s a raid device, verify the error counters on that. -cf On 12/02/2010 02:00 PM, Bob Ball wrote:> We were getting errors thrown by an OST. /var/log/messages contained a > lot of these: > 2010-11-28T17:05:34-05:00 umfs06.aglt2.org kernel: [2102640.735927] > LDISKFS-fs error (device sdk): ldiskfs_mb_check_ondisk_bitmap: on-disk > bitmap for group 639corrupted: 440 blocks free in bitmap, 439 - in gd > > So, I turned off (most) access to the disk via lctl (we have a LOT of > client machines, some were missed) and got problems. Had to use the > alternate superblock to e2fsck the disk. When back online, I still saw > similar messages. Updated to e2fsprogs 1.41.12 as suggested elsewhere. > Repeated e2fsck. > > Still seeing these. Users report some files corrupted, coming up with > bad md5sum.... Any other thoughts on what to do about this problem? > > [2440763.879143] LDISKFS-fs error (device sdk): > ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 35406corrupted: > 1318 blocks free in bitmap, 1317 - in gd > [2440763.879796] > [2440763.882724] LustreError: > 1651027:0:(fsfilt-ldiskfs.c:1333:fsfilt_ldiskfs_write_record()) can''t > read/create block: -28 > [2440763.882736] LustreError: > 1651027:0:(llog_lvfs.c:116:llog_lvfs_write_blob()) error writing log > record: rc -28 > [2440763.882789] LustreError: > 1651002:0:(mgc_request.c:1089:mgc_copy_llog()) Failed to copy remote log > umt3-OST0019 (-28) > > Rebooted to make system clean as a whole, and found the same kind of > thing repeating. > [ 285.834864] LDISKFS-fs (sdk): warning: mounting fs with errors, > running e2fsck is recommended > [ 285.852559] LDISKFS-fs (sdk): mounted filesystem with ordered data mode > [ 286.079065] LDISKFS-fs (sdk): warning: mounting fs with errors, > running e2fsck is recommended > [ 286.096316] LDISKFS-fs (sdk): mounted filesystem with ordered data mode > [ 286.940872] LDISKFS-fs error (device sdk): > ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 35406corrupted: > 1318 blocks free in bitmap, 1317 - in gd > [ 286.941693] > [ 286.945224] LustreError: > 5790:0:(fsfilt-ldiskfs.c:1333:fsfilt_ldiskfs_write_record()) can''t > read/create block: -28 > [ 286.945233] LustreError: > 5790:0:(llog_lvfs.c:116:llog_lvfs_write_blob()) error writing log > record: rc -28 > [ 286.945448] LustreError: 5763:0:(mgc_request.c:1089:mgc_copy_llog()) > Failed to copy remote log umt3-OST0019 (-28) > > All help appreciated. > > bob > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss
It is a Dell PERC6 RAID array. OMSA monitoring is enabled and is not throwing errors. Hmmmm, mptctl is old though, so maybe that is a contributing factor. I guess I need to update that. Shoot, megaraid_sas is also not up to date. dkms.... OK, guess I need some driver updates. Later. bob On 12/2/2010 4:05 PM, Colin Faber wrote:> Hi Bob, > > If you''re seeing the same errors on the same disk after e2fsck run, > and it''s not catching them, it''s possible that you''re hitting an edge > case which isn''t handled within e2fsck properly, however if you''re > experiencing different errors and e2fsck did catch them before, > chances are you''re looking at some hardware failure some place. > > If this is a single disk, and you have SMART monitoring enabled, check > your error counters, if it''s a raid device, verify the error counters > on that. > > -cf > > > On 12/02/2010 02:00 PM, Bob Ball wrote: >> We were getting errors thrown by an OST. /var/log/messages contained a >> lot of these: >> 2010-11-28T17:05:34-05:00 umfs06.aglt2.org kernel: [2102640.735927] >> LDISKFS-fs error (device sdk): ldiskfs_mb_check_ondisk_bitmap: on-disk >> bitmap for group 639corrupted: 440 blocks free in bitmap, 439 - in gd >> >> So, I turned off (most) access to the disk via lctl (we have a LOT of >> client machines, some were missed) and got problems. Had to use the >> alternate superblock to e2fsck the disk. When back online, I still saw >> similar messages. Updated to e2fsprogs 1.41.12 as suggested elsewhere. >> Repeated e2fsck. >> >> Still seeing these. Users report some files corrupted, coming up with >> bad md5sum.... Any other thoughts on what to do about this problem? >> >> [2440763.879143] LDISKFS-fs error (device sdk): >> ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 35406corrupted: >> 1318 blocks free in bitmap, 1317 - in gd >> [2440763.879796] >> [2440763.882724] LustreError: >> 1651027:0:(fsfilt-ldiskfs.c:1333:fsfilt_ldiskfs_write_record()) can''t >> read/create block: -28 >> [2440763.882736] LustreError: >> 1651027:0:(llog_lvfs.c:116:llog_lvfs_write_blob()) error writing log >> record: rc -28 >> [2440763.882789] LustreError: >> 1651002:0:(mgc_request.c:1089:mgc_copy_llog()) Failed to copy remote log >> umt3-OST0019 (-28) >> >> Rebooted to make system clean as a whole, and found the same kind of >> thing repeating. >> [ 285.834864] LDISKFS-fs (sdk): warning: mounting fs with errors, >> running e2fsck is recommended >> [ 285.852559] LDISKFS-fs (sdk): mounted filesystem with ordered data >> mode >> [ 286.079065] LDISKFS-fs (sdk): warning: mounting fs with errors, >> running e2fsck is recommended >> [ 286.096316] LDISKFS-fs (sdk): mounted filesystem with ordered data >> mode >> [ 286.940872] LDISKFS-fs error (device sdk): >> ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 35406corrupted: >> 1318 blocks free in bitmap, 1317 - in gd >> [ 286.941693] >> [ 286.945224] LustreError: >> 5790:0:(fsfilt-ldiskfs.c:1333:fsfilt_ldiskfs_write_record()) can''t >> read/create block: -28 >> [ 286.945233] LustreError: >> 5790:0:(llog_lvfs.c:116:llog_lvfs_write_blob()) error writing log >> record: rc -28 >> [ 286.945448] LustreError: 5763:0:(mgc_request.c:1089:mgc_copy_llog()) >> Failed to copy remote log umt3-OST0019 (-28) >> >> All help appreciated. >> >> bob >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss > >
Just to cleanly end this thread, the mptctl was out of date. We also updated megaraid_sas and perc6 firmware. e2fsck found some Block bitmap differences (fixed) at this point, but the OST mounted cleanly and the errors stopped. Unfortunately, there are now corrupted files in the system, that remain corrupted, and we''ll probably never be able to come up with a complete list of them. bob On 12/2/2010 4:35 PM, Bob Ball wrote:> It is a Dell PERC6 RAID array. OMSA monitoring is enabled and is not > throwing errors. Hmmmm, mptctl is old though, so maybe that is a > contributing factor. I guess I need to update that. Shoot, > megaraid_sas is also not up to date. dkms.... > > OK, guess I need some driver updates. > > Later. > bob > > On 12/2/2010 4:05 PM, Colin Faber wrote: >> Hi Bob, >> >> If you''re seeing the same errors on the same disk after e2fsck run, >> and it''s not catching them, it''s possible that you''re hitting an edge >> case which isn''t handled within e2fsck properly, however if you''re >> experiencing different errors and e2fsck did catch them before, >> chances are you''re looking at some hardware failure some place. >> >> If this is a single disk, and you have SMART monitoring enabled, check >> your error counters, if it''s a raid device, verify the error counters >> on that. >> >> -cf >> >> >> On 12/02/2010 02:00 PM, Bob Ball wrote: >>> We were getting errors thrown by an OST. /var/log/messages contained a >>> lot of these: >>> 2010-11-28T17:05:34-05:00 umfs06.aglt2.org kernel: [2102640.735927] >>> LDISKFS-fs error (device sdk): ldiskfs_mb_check_ondisk_bitmap: on-disk >>> bitmap for group 639corrupted: 440 blocks free in bitmap, 439 - in gd >>> >>> So, I turned off (most) access to the disk via lctl (we have a LOT of >>> client machines, some were missed) and got problems. Had to use the >>> alternate superblock to e2fsck the disk. When back online, I still saw >>> similar messages. Updated to e2fsprogs 1.41.12 as suggested elsewhere. >>> Repeated e2fsck. >>> >>> Still seeing these. Users report some files corrupted, coming up with >>> bad md5sum.... Any other thoughts on what to do about this problem? >>> >>> [2440763.879143] LDISKFS-fs error (device sdk): >>> ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 35406corrupted: >>> 1318 blocks free in bitmap, 1317 - in gd >>> [2440763.879796] >>> [2440763.882724] LustreError: >>> 1651027:0:(fsfilt-ldiskfs.c:1333:fsfilt_ldiskfs_write_record()) can''t >>> read/create block: -28 >>> [2440763.882736] LustreError: >>> 1651027:0:(llog_lvfs.c:116:llog_lvfs_write_blob()) error writing log >>> record: rc -28 >>> [2440763.882789] LustreError: >>> 1651002:0:(mgc_request.c:1089:mgc_copy_llog()) Failed to copy remote log >>> umt3-OST0019 (-28) >>> >>> Rebooted to make system clean as a whole, and found the same kind of >>> thing repeating. >>> [ 285.834864] LDISKFS-fs (sdk): warning: mounting fs with errors, >>> running e2fsck is recommended >>> [ 285.852559] LDISKFS-fs (sdk): mounted filesystem with ordered data >>> mode >>> [ 286.079065] LDISKFS-fs (sdk): warning: mounting fs with errors, >>> running e2fsck is recommended >>> [ 286.096316] LDISKFS-fs (sdk): mounted filesystem with ordered data >>> mode >>> [ 286.940872] LDISKFS-fs error (device sdk): >>> ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 35406corrupted: >>> 1318 blocks free in bitmap, 1317 - in gd >>> [ 286.941693] >>> [ 286.945224] LustreError: >>> 5790:0:(fsfilt-ldiskfs.c:1333:fsfilt_ldiskfs_write_record()) can''t >>> read/create block: -28 >>> [ 286.945233] LustreError: >>> 5790:0:(llog_lvfs.c:116:llog_lvfs_write_blob()) error writing log >>> record: rc -28 >>> [ 286.945448] LustreError: 5763:0:(mgc_request.c:1089:mgc_copy_llog()) >>> Failed to copy remote log umt3-OST0019 (-28) >>> >>> All help appreciated. >>> >>> bob >>> _______________________________________________ >>> Lustre-discuss mailing list >>> Lustre-discuss at lists.lustre.org >>> http://lists.lustre.org/mailman/listinfo/lustre-discuss >> > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss > >
Hi Bob, Good to hear you''ve identified and resolved the issue. Sorry to hear you''ll have to restore from backup though. -cf On 12/03/2010 02:41 PM, Bob Ball wrote:> Just to cleanly end this thread, the mptctl was out of date. We also > updated megaraid_sas and perc6 firmware. e2fsck found some Block bitmap > differences (fixed) at this point, but the OST mounted cleanly and the > errors stopped. > > Unfortunately, there are now corrupted files in the system, that remain > corrupted, and we''ll probably never be able to come up with a complete > list of them. > > bob > > > On 12/2/2010 4:35 PM, Bob Ball wrote: >> It is a Dell PERC6 RAID array. OMSA monitoring is enabled and is not >> throwing errors. Hmmmm, mptctl is old though, so maybe that is a >> contributing factor. I guess I need to update that. Shoot, >> megaraid_sas is also not up to date. dkms.... >> >> OK, guess I need some driver updates. >> >> Later. >> bob >> >> On 12/2/2010 4:05 PM, Colin Faber wrote: >>> Hi Bob, >>> >>> If you''re seeing the same errors on the same disk after e2fsck run, >>> and it''s not catching them, it''s possible that you''re hitting an edge >>> case which isn''t handled within e2fsck properly, however if you''re >>> experiencing different errors and e2fsck did catch them before, >>> chances are you''re looking at some hardware failure some place. >>> >>> If this is a single disk, and you have SMART monitoring enabled, check >>> your error counters, if it''s a raid device, verify the error counters >>> on that. >>> >>> -cf >>> >>> >>> On 12/02/2010 02:00 PM, Bob Ball wrote: >>>> We were getting errors thrown by an OST. /var/log/messages contained a >>>> lot of these: >>>> 2010-11-28T17:05:34-05:00 umfs06.aglt2.org kernel: [2102640.735927] >>>> LDISKFS-fs error (device sdk): ldiskfs_mb_check_ondisk_bitmap: on-disk >>>> bitmap for group 639corrupted: 440 blocks free in bitmap, 439 - in gd >>>> >>>> So, I turned off (most) access to the disk via lctl (we have a LOT of >>>> client machines, some were missed) and got problems. Had to use the >>>> alternate superblock to e2fsck the disk. When back online, I still saw >>>> similar messages. Updated to e2fsprogs 1.41.12 as suggested elsewhere. >>>> Repeated e2fsck. >>>> >>>> Still seeing these. Users report some files corrupted, coming up with >>>> bad md5sum.... Any other thoughts on what to do about this problem? >>>> >>>> [2440763.879143] LDISKFS-fs error (device sdk): >>>> ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 35406corrupted: >>>> 1318 blocks free in bitmap, 1317 - in gd >>>> [2440763.879796] >>>> [2440763.882724] LustreError: >>>> 1651027:0:(fsfilt-ldiskfs.c:1333:fsfilt_ldiskfs_write_record()) can''t >>>> read/create block: -28 >>>> [2440763.882736] LustreError: >>>> 1651027:0:(llog_lvfs.c:116:llog_lvfs_write_blob()) error writing log >>>> record: rc -28 >>>> [2440763.882789] LustreError: >>>> 1651002:0:(mgc_request.c:1089:mgc_copy_llog()) Failed to copy remote log >>>> umt3-OST0019 (-28) >>>> >>>> Rebooted to make system clean as a whole, and found the same kind of >>>> thing repeating. >>>> [ 285.834864] LDISKFS-fs (sdk): warning: mounting fs with errors, >>>> running e2fsck is recommended >>>> [ 285.852559] LDISKFS-fs (sdk): mounted filesystem with ordered data >>>> mode >>>> [ 286.079065] LDISKFS-fs (sdk): warning: mounting fs with errors, >>>> running e2fsck is recommended >>>> [ 286.096316] LDISKFS-fs (sdk): mounted filesystem with ordered data >>>> mode >>>> [ 286.940872] LDISKFS-fs error (device sdk): >>>> ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 35406corrupted: >>>> 1318 blocks free in bitmap, 1317 - in gd >>>> [ 286.941693] >>>> [ 286.945224] LustreError: >>>> 5790:0:(fsfilt-ldiskfs.c:1333:fsfilt_ldiskfs_write_record()) can''t >>>> read/create block: -28 >>>> [ 286.945233] LustreError: >>>> 5790:0:(llog_lvfs.c:116:llog_lvfs_write_blob()) error writing log >>>> record: rc -28 >>>> [ 286.945448] LustreError: 5763:0:(mgc_request.c:1089:mgc_copy_llog()) >>>> Failed to copy remote log umt3-OST0019 (-28) >>>> >>>> All help appreciated. >>>> >>>> bob >>>> _______________________________________________ >>>> Lustre-discuss mailing list >>>> Lustre-discuss at lists.lustre.org >>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss >> >> > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss