Please don''t reply to lustre-devel. Instead, comment in Bugzilla by using the following link: https://bugzilla.lustre.org/show_bug.cgi?id=11520 I guess I have to re-type the bug description again... What''s up with that? The ''new bug'' description field seems to get lost. On the same 320 OST fs referenced in bug 11519, we have a hung OST with the following LBUG: lg_OST_15027_0: Serious error: objid 3683 already exists; is this filesystem corrupt? We cannot complete the orphan cleanup due to the problem in 11519, so we have an unusable 100T fs at the moment, thus the high severity. The stack traces are being extracted.
Please don''t reply to lustre-devel. Instead, comment in Bugzilla by
using the following link:
https://bugzilla.lustre.org/show_bug.cgi?id=11520
What |Removed |Added
----------------------------------------------------------------------------
CC| |alex@clusterfs.com
OtherBugsDependingO|3431 |718
nThis| |
AssignedTo|bugs@clusterfs.com |girish@clusterfs.com
Alex and Girish, this is related to the same circumstances as bug 11519
Please don''t reply to lustre-devel. Instead, comment in Bugzilla by using the following link: https://bugzilla.lustre.org/show_bug.cgi?id=11520 Ruth, We have got some eyes from senior engineering on this issue (and 11519). We''ll get back to you with a solution or a request for more information ASAP. Mike
Please don''t reply to lustre-devel. Instead, comment in Bugzilla by using the following link: https://bugzilla.lustre.org/show_bug.cgi?id=11520 Created an attachment (id=9313) Please don''t reply to lustre-devel. Instead, comment in Bugzilla by using the following link: --> (https://bugzilla.lustre.org/attachment.cgi?id=9313&action=view) lustre log on ost nid15027
Please don''t reply to lustre-devel. Instead, comment in Bugzilla by using the following link: https://bugzilla.lustre.org/show_bug.cgi?id=11520 Created an attachment (id=9314) Please don''t reply to lustre-devel. Instead, comment in Bugzilla by using the following link: --> (https://bugzilla.lustre.org/attachment.cgi?id=9314&action=view) syslog on nid15127
Please don''t reply to lustre-devel. Instead, comment in Bugzilla by using the following link: https://bugzilla.lustre.org/show_bug.cgi?id=11520 (In reply to comment #4)> Created an attachment (id=9314)Please don''t reply to lustre-devel. Instead, comment in Bugzilla by using the following link: --> (https://bugzilla.lustre.org/attachment.cgi?id=9314&action=view)> syslog on nid15127 >The admins completed a full lfsck sequence on this file system, and still got the same LBUG when attempting to use it. What else is there to do?
Please don''t reply to lustre-devel. Instead, comment in Bugzilla by using the following link: https://bugzilla.lustre.org/show_bug.cgi?id=11520 (In reply to comment #6)> (In reply to comment #9) > > The admins completed a full lfsck sequence on this file system, and still got > > the same LBUG when attempting to use it. > > > > What else is there to do? > > can you attach the output of that lfsck run? >It''s possible the databases were created before the LBUG occurred, so I advised the admins to re-create the db for the problem OST, and the one for the MDS, and re-run the lfsck. If the problem continues I''ll ask them for the output.
Please don''t reply to lustre-devel. Instead, comment in Bugzilla by using the following link: https://bugzilla.lustre.org/show_bug.cgi?id=11520 The admin team re-ran the process on the problem oss. e2fsck reported some ''high 16 bit extent block set'' errors and claimed to have fixed them. lfsck did not detect any duplicate objects, and reported nothing about object id 3680. They haven''t gone on to re-create the LBUG yet, people are running on the other file systems. If it does happen again they are planning to reformat. Logs and lfsck output will be moved out probably sometime today. Something was mentioned in the phone conference last week about possibly attempting a manual edit of last_id file. The 2 osts on this node have objects only in dir O/0, and last_id files contain ''+'' and ''p''. Any chance it''s time for this? If so how, and what should be in there. ?? Ruth
Please don''t reply to lustre-devel. Instead, comment in Bugzilla by using the following link: https://bugzilla.lustre.org/show_bug.cgi?id=11520 Here is the current plan, Does this sound like a safe way to restore the FS? Ruth>From email,Status update: We''re planning on trying Lee''s suggestion starting tomorrow morning: 1. Use Steve Monk''s utility to find the names of the files on OST lg_OST_15027_0. This only takes a few minutes. Jason will start on Wednesday. 2. Assuming the total GB isn''t too great, root does cp -a on these file to ???? 3. Do a Linux-level newfs, following by an init OST on just this OST. 4. cp -a the files back into /scratch_grande Outside of step 3 maybe, the rest should be doable while the machine is up, but inaccessible to users. Sue
Please don''t reply to lustre-devel. Instead, comment in Bugzilla by using the following link: https://bugzilla.lustre.org/show_bug.cgi?id=11520 (In reply to comment #11)> Ruth, I think that would work. > how many OSTs you have? > how many stripes you use? > another way could be to recreate objects on other OSTs > and then copy into them from failed OSTThe file system has 320 OSTs, the default stripe is 1, but users modify to suit their needs. Is there is an already existing utility to do the move of objects from one OST to another? I wouldn''t know offhand how to do that. Ruth
Please don''t reply to lustre-devel. Instead, comment in Bugzilla by using the following link: https://bugzilla.lustre.org/show_bug.cgi?id=11520 I''ve forwarded the info to admin team, however they are at the moment planning to go ahead with what was described. Ruth
Please don''t reply to lustre-devel. Instead, comment in Bugzilla by using the following link: https://bugzilla.lustre.org/show_bug.cgi?id=11520 (In reply to comment #15)> (In reply to comment #21) > > I''ve forwarded the info to admin team, however they are at the moment > planning > > to go ahead with what was described. > > Ruth, we''re proposing that because it''s much simpler. >I understand. And it may be the people in charge will decide to try this first, but right now the fs is mounted read only to get some essential user data moved off. I have also asked for the logs again, they have to move through a process before they are available to attach here.
Please don''t reply to lustre-devel. Instead, comment in Bugzilla by using the following link: https://bugzilla.lustre.org/show_bug.cgi?id=11520 Created an attachment (id=9374) Please don''t reply to lustre-devel. Instead, comment in Bugzilla by using the following link: --> (https://bugzilla.lustre.org/attachment.cgi?id=9374&action=view) logs from affected oss
Please don''t reply to lustre-devel. Instead, comment in Bugzilla by using the following link: https://bugzilla.lustre.org/show_bug.cgi?id=11520 (In reply to comment #18)> Ruth, have you had a chance to fix up the LAST_ID value for this OST (per the > process in lustre-discuss)? This can be completed on the order of 20-30 > minutes, and I would hate to find out that your system has been down for > additional days for something that should only take a short time to fix. I''d be > happy to assist you via phone or IRC if there are any questions about the > process or interpreting the state of the filesystem.Another fix attempt is scheduled for tomorrow. I realize that fixing last_id is a short effort, but they aren''t confident that this is the only thing wrong with that OST and their plan is to reformat it. If anything changes I''ll let you know. The information about how to edit last_id was unfortunately about a day late, after the file system was already being used read only for users to copy data off. Thanks for the offer. Ruth
Please don''t reply to lustre-devel. Instead, comment in Bugzilla by using the following link: https://bugzilla.lustre.org/show_bug.cgi?id=11520 (In reply to comment #19)> (In reply to comment #18) > > Ruth, have you had a chance to fix up the LAST_ID value for this OST (per the > > process in lustre-discuss)? This can be completed on the order of 20-30 > > minutes, and I would hate to find out that your system has been down for > > additional days for something that should only take a short time to fix. I''d be > > happy to assist you via phone or IRC if there are any questions about the > > process or interpreting the state of the filesystem. >The last_id fix will be tried out at 11 mountain time this morning. I''ll be there with the folks working on it, there is phone access but no irc. If you forward a number to my email (rklundt@sandia.gov) I''ll call if something doesn''t go as expected. Ruth
Please don''t reply to lustre-devel. Instead, comment in Bugzilla by
using the following link:
https://bugzilla.lustre.org/show_bug.cgi?id=11520
What |Removed |Added
----------------------------------------------------------------------------
Severity|S1 (critical) |S3 (normal)
Priority|P1 |P3
(In reply to comment #21)> Ruth
>
> It would probably be safer to call the support hotline (303) 459-4126 and
> leave a message and we will get someone appropriate to contact you. That
way
> you are not tied to any one individual''s availability.
>
> Regards
>
> Peter
The last_id fix seems to have been successful, there was a test run that
definitely was able to write data to the problem OST. The file system is up and
users are being told to check their data and proceed.
The process described in the lustre-discuss mail was straight-forward, except
for the byte-swapped order of the value in last_id, that took a minute to see.
Thanks!
Ruth