Gustavsson, Mathias
2010-May-25 15:01 UTC
[Lustre-discuss] 1.8.1.1 -> 1.8.3 upgrade questions
Hi, We tried to do a 1.8.1.1 to 1.8.3 version upgrade this weekend, but we got i/o error on all of our old file systems (created ~4 years ago), we have a more recently created filesystem (only a couple of moths old) and that was fine. This was in the log of the active mds, i couldn''t find anything related in the log of the client at the time. May 22 14:16:28 semldxlucky kernel: LustreError: 2664:0:(mds_open.c:441:mds_create_objects()) error creating objects for inode 841541: rc = -5 May 22 14:16:28 semldxlucky kernel: LustreError: 2664:0:(mds_open.c:826:mds_finish_open()) mds_create_objects: rc = -5 May 22 14:16:38 semldxlucky kernel: LustreError: 2648:0:(mds_open.c:441:mds_create_objects()) error creating objects for inode 841541: rc = -5 May 22 14:16:38 semldxlucky kernel: LustreError: 2648:0:(mds_open.c:441:mds_create_objects()) Skipped 1 previous similar message May 22 14:16:38 semldxlucky kernel: LustreError: 2648:0:(mds_open.c:826:mds_finish_open()) mds_create_objects: rc = -5 May 22 14:16:38 semldxlucky kernel: LustreError: 2648:0:(mds_open.c:826:mds_finish_open()) Skipped 1 previous similar message MDS + OSS''s version : CentOS 5.4 and lustre version 1.8.1.1 Clients version : CentOS 4.8 and lustre version 1.6.7.2 Mathias Gustavsson Linux System Manager AstraZeneca R&D M?lndal SE-431 83 M?lndal, Sweden Phone: +46 31 776 12 58 mathias.gustavsson at astrazeneca.com -------------------------------------------------------------------------- Confidentiality Notice: This message is private and may contain confidential and proprietary information. If you have received this message in error, please notify us and remove it from your system and note that you must not copy, distribute or take any action in reliance on it. Any unauthorized use or disclosure of the contents of this message is not permitted and may be unlawful.
On 2010-05-25, at 09:01, Gustavsson, Mathias wrote:> We tried to do a 1.8.1.1 to 1.8.3 version upgrade this weekend, but we got i/o error on all of our old file systems (created ~4 years ago), we have a more recently created filesystem (only a couple of moths old) and that was fine. > This was in the log of the active mds, i couldn''t find anything related in the log of the client at the time. > > May 22 14:16:28 semldxlucky kernel: LustreError: 2664:0:(mds_open.c:441:mds_create_objects()) error creating objects for inode 841541: rc = -5 > May 22 14:16:28 semldxlucky kernel: LustreError: 2664:0:(mds_open.c:826:mds_finish_open()) mds_create_objects: rc = -5 > May 22 14:16:38 semldxlucky kernel: LustreError: 2648:0:(mds_open.c:441:mds_create_objects()) error creating objects for inode 841541: rc = -5 > May 22 14:16:38 semldxlucky kernel: LustreError: 2648:0:(mds_open.c:441:mds_create_objects()) Skipped 1 previous similar message > May 22 14:16:38 semldxlucky kernel: LustreError: 2648:0:(mds_open.c:826:mds_finish_open()) mds_create_objects: rc = -5 > May 22 14:16:38 semldxlucky kernel: LustreError: 2648:0:(mds_open.c:826:mds_finish_open()) Skipped 1 previous similar message > > MDS + OSS''s version : CentOS 5.4 and lustre version 1.8.1.1 > Clients version : CentOS 4.8 and lustre version 1.6.7.2This is really a problem between the MDS and the OSS. Is there anything in the OSS logs? Cheers, Andreas -- Andreas Dilger Lustre Technical Lead Oracle Corporation Canada Inc.
Those look familiar. Have you run fsck on the OSTs and MDTs before upgrade? Best regards, Wojciech On 25 May 2010 16:01, Gustavsson, Mathias <Mathias.Gustavsson at astrazeneca.com> wrote:> Hi, > > We tried to do a 1.8.1.1 to 1.8.3 version upgrade this weekend, but we got i/o error on all of our old file systems (created ~4 years ago), we have a more recently created filesystem (only a couple of moths old) and that was fine. > This was in the log of the active mds, i couldn''t find anything related in the log of the client at the time. > > May 22 14:16:28 semldxlucky kernel: LustreError: 2664:0:(mds_open.c:441:mds_create_objects()) error creating objects for inode 841541: rc = -5 > May 22 14:16:28 semldxlucky kernel: LustreError: 2664:0:(mds_open.c:826:mds_finish_open()) mds_create_objects: rc = -5 > May 22 14:16:38 semldxlucky kernel: LustreError: 2648:0:(mds_open.c:441:mds_create_objects()) error creating objects for inode 841541: rc = -5 > May 22 14:16:38 semldxlucky kernel: LustreError: 2648:0:(mds_open.c:441:mds_create_objects()) Skipped 1 previous similar message > May 22 14:16:38 semldxlucky kernel: LustreError: 2648:0:(mds_open.c:826:mds_finish_open()) mds_create_objects: rc = -5 > May 22 14:16:38 semldxlucky kernel: LustreError: 2648:0:(mds_open.c:826:mds_finish_open()) Skipped 1 previous similar message > > MDS + OSS''s version : CentOS 5.4 and lustre version 1.8.1.1 > Clients version : CentOS 4.8 and lustre version 1.6.7.2 > > Mathias Gustavsson > Linux System Manager > AstraZeneca R&D M?lndal > SE-431 83 M?lndal, Sweden > > Phone: ?+46 31 776 12 58 > mathias.gustavsson at astrazeneca.com > > > -------------------------------------------------------------------------- > Confidentiality Notice: This message is private and may contain confidential and proprietary information. If you have received this message in error, please notify us and remove it from your system and note that you must not copy, distribute or take any action in reliance on it. Any unauthorized use or disclosure of the contents of this message is not permitted and may be unlawful. > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >-- -- Wojciech Turek Assistant System Manager High Performance Computing Service University of Cambridge Email: wjt27 at cam.ac.uk Tel: (+)44 1223 763517
Gustavsson, Mathias
2010-May-26 12:40 UTC
[Lustre-discuss] 1.8.1.1 -> 1.8.3 upgrade questions
These lines is only available when we are on version 1.8.3 May 22 13:19:12 semldxludwig kernel: LustreError: 12412:0:(lib-move.c:2441:LNetPut()) Error sending PUT to 12345-10.0.0.242 at tcp: -113 May 22 13:19:51 semldxluis kernel: Lustre: 13460:0:(fsfilt-ldiskfs.c:1385:fsfilt_ldiskfs_setup()) filesystem doesn''t have dir_index feature enabled And a couple of maximal mount count reached, file extents and mballoc enabled. And i got a LustreError with a lustre-log.1274525880.5458 in /tmp. I can''t really sort what is relevant information from the OSS logs (except from the obvious client recovery traffic), give me a hint of what i should look for and I''ll paste it in here. Thanks, Mathias Gustavsson Linux System Manager AstraZeneca R&D M?lndal SE-431 83 M?lndal, Sweden Phone: +46 31 776 12 58 mathias.gustavsson at astrazeneca.com -------------------------------------------------------------------------- Confidentiality Notice: This message is private and may contain confidential and proprietary information. If you have received this message in error, please notify us and remove it from your system and note that you must not copy, distribute or take any action in reliance on it. Any unauthorized use or disclosure of the contents of this message is not permitted and may be unlawful. -----Original Message----- From: Andreas Dilger [mailto:andreas.dilger at oracle.com] Sent: Tue 5/25/2010 11:04 PM To: Gustavsson, Mathias Cc: lustre-discuss at lists.lustre.org Subject: Re: [Lustre-discuss] 1.8.1.1 -> 1.8.3 upgrade questions On 2010-05-25, at 09:01, Gustavsson, Mathias wrote:> We tried to do a 1.8.1.1 to 1.8.3 version upgrade this weekend, but we got i/o error on all of our old file systems (created ~4 years ago), we have a more recently created filesystem (only a couple of moths old) and that was fine. > This was in the log of the active mds, i couldn''t find anything related in the log of the client at the time. > > May 22 14:16:28 semldxlucky kernel: LustreError: 2664:0:(mds_open.c:441:mds_create_objects()) error creating objects for inode 841541: rc = -5 > May 22 14:16:28 semldxlucky kernel: LustreError: 2664:0:(mds_open.c:826:mds_finish_open()) mds_create_objects: rc = -5 > May 22 14:16:38 semldxlucky kernel: LustreError: 2648:0:(mds_open.c:441:mds_create_objects()) error creating objects for inode 841541: rc = -5 > May 22 14:16:38 semldxlucky kernel: LustreError: 2648:0:(mds_open.c:441:mds_create_objects()) Skipped 1 previous similar message > May 22 14:16:38 semldxlucky kernel: LustreError: 2648:0:(mds_open.c:826:mds_finish_open()) mds_create_objects: rc = -5 > May 22 14:16:38 semldxlucky kernel: LustreError: 2648:0:(mds_open.c:826:mds_finish_open()) Skipped 1 previous similar message > > MDS + OSS''s version : CentOS 5.4 and lustre version 1.8.1.1 > Clients version : CentOS 4.8 and lustre version 1.6.7.2This is really a problem between the MDS and the OSS. Is there anything in the OSS logs? Cheers, Andreas -- Andreas Dilger Lustre Technical Lead Oracle Corporation Canada Inc.
Gustavsson, Mathias
2010-May-26 12:40 UTC
[Lustre-discuss] 1.8.1.1 -> 1.8.3 upgrade questions
Nope, wish i had .. "maximal mount count reached" Thanks, Mathias Gustavsson Linux System Manager AstraZeneca R&D M?lndal SE-431 83 M?lndal, Sweden Phone: +46 31 776 12 58 mathias.gustavsson at astrazeneca.com -------------------------------------------------------------------------- Confidentiality Notice: This message is private and may contain confidential and proprietary information. If you have received this message in error, please notify us and remove it from your system and note that you must not copy, distribute or take any action in reliance on it. Any unauthorized use or disclosure of the contents of this message is not permitted and may be unlawful. -----Original Message----- From: turek.wojciech at googlemail.com on behalf of Wojciech Turek Sent: Tue 5/25/2010 11:33 PM To: Gustavsson, Mathias Cc: lustre-discuss at lists.lustre.org Subject: Re: [Lustre-discuss] 1.8.1.1 -> 1.8.3 upgrade questions Those look familiar. Have you run fsck on the OSTs and MDTs before upgrade? Best regards, Wojciech On 25 May 2010 16:01, Gustavsson, Mathias <Mathias.Gustavsson at astrazeneca.com> wrote:> Hi, > > We tried to do a 1.8.1.1 to 1.8.3 version upgrade this weekend, but we got i/o error on all of our old file systems (created ~4 years ago), we have a more recently created filesystem (only a couple of moths old) and that was fine. > This was in the log of the active mds, i couldn''t find anything related in the log of the client at the time. > > May 22 14:16:28 semldxlucky kernel: LustreError: 2664:0:(mds_open.c:441:mds_create_objects()) error creating objects for inode 841541: rc = -5 > May 22 14:16:28 semldxlucky kernel: LustreError: 2664:0:(mds_open.c:826:mds_finish_open()) mds_create_objects: rc = -5 > May 22 14:16:38 semldxlucky kernel: LustreError: 2648:0:(mds_open.c:441:mds_create_objects()) error creating objects for inode 841541: rc = -5 > May 22 14:16:38 semldxlucky kernel: LustreError: 2648:0:(mds_open.c:441:mds_create_objects()) Skipped 1 previous similar message > May 22 14:16:38 semldxlucky kernel: LustreError: 2648:0:(mds_open.c:826:mds_finish_open()) mds_create_objects: rc = -5 > May 22 14:16:38 semldxlucky kernel: LustreError: 2648:0:(mds_open.c:826:mds_finish_open()) Skipped 1 previous similar message > > MDS + OSS''s version : CentOS 5.4 and lustre version 1.8.1.1 > Clients version : CentOS 4.8 and lustre version 1.6.7.2 > > Mathias Gustavsson > Linux System Manager > AstraZeneca R&D M?lndal > SE-431 83 M?lndal, Sweden > > Phone: +46 31 776 12 58 > mathias.gustavsson at astrazeneca.com > > > -------------------------------------------------------------------------- > Confidentiality Notice: This message is private and may contain confidential and proprietary information. If you have received this message in error, please notify us and remove it from your system and note that you must not copy, distribute or take any action in reliance on it. Any unauthorized use or disclosure of the contents of this message is not permitted and may be unlawful. > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >-- -- Wojciech Turek Assistant System Manager High Performance Computing Service University of Cambridge Email: wjt27 at cam.ac.uk Tel: (+)44 1223 763517