Hello everyone I''ve an installation with Lustre 1.8.2, Centos 5, x86_64 and I encountered this problem: After several months of smooth operation, client begin to write empty files without log error,from their point of view writing was successful. OSS wrote, in their log, several lines like: Sep 8 12:40:31 tgoss-0200 kernel: LustreError: 5816:0:(filter_io.c:183:filter_grant_space_left()) lfs01-OST0000: cli 20d94382-3300-f12e-65d1-c0f1743e1e20/ffff8106a4e30a00 grant 39956230144 > available 39956226048 and pending 0 I checked the availability of space and inodes, but this is not the problem. the problem goes away by rebooting ost. This is the second time I have, first at july 2010, second september 2010. Any ideas?It''s a bug? Thanks -- Gianluca Tresoldi ***SysAdmin*** ***Demon''s Trainer*** Tuttogratis Italia Spa E-mail: gianluca.tresoldi at tuttogratis.com http://www.tuttogratis.it Tel Centralino 02-57313101 Tel Diretto 02-57313136 <http://www.gnu.org>Be open... ************ Confidentiality Notice & Disclaimer ************* This message, together with any attachments, is for the confidential and exclusive use of the addressee(s). If you receive it in error, please delete the message and its attachments from your system immediately and notify us by return e-mail. Do not disclose, copy, circulate or use any information contained in this e-mail. * -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100908/df181fec/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: linux40.jpg Type: image/jpeg Size: 1399 bytes Desc: not available Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100908/df181fec/attachment.jpg
Hi Luca, the error mean: ost grant more space than there is avaiable!!! You could check in the proc filesystem of yours client checking the cur_grant_bytes to check how much grant space those clients take. On 09/08/2010 05:02 PM, Gianluca Tresoldi wrote:> Hello everyone > > I''ve an installation with Lustre 1.8.2, Centos 5, x86_64 and I > encountered this problem: > > After several months of smooth operation, client begin to write empty > files without log error,from their point of view writing was successful. > > OSS wrote, in their log, several lines like: > Sep 8 12:40:31 tgoss-0200 kernel: LustreError: > 5816:0:(filter_io.c:183:filter_grant_space_left()) lfs01-OST0000: cli > 20d94382-3300-f12e-65d1-c0f1743e1e20/ffff8106a4e30a00 grant > 39956230144 > available 39956226048 and pending 0 > > I checked the availability of space and inodes, but this is not the > problem. > > the problem goes away by rebooting ost. > > This is the second time I have, first at july 2010, second september 2010. > > Any ideas?It''s a bug? > > Thanks > -- > Gianluca Tresoldi > ***SysAdmin*** > ***Demon''s Trainer*** > Tuttogratis Italia Spa > E-mail: gianluca.tresoldi at tuttogratis.com > http://www.tuttogratis.it > Tel Centralino 02-57313101 > Tel Diretto 02-57313136 > <http://www.gnu.org>Be open... > ************ Confidentiality Notice & Disclaimer ************* > This message, together with any attachments, is for the confidential > and exclusive use of the addressee(s). If you receive it in error, > please delete the message and its attachments from your system > immediately and notify us by return e-mail. > Do not disclose, copy, circulate or use any information contained in > this e-mail. > * > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >-- _Gabriele Paciucci_ http://www.linkedin.com/in/paciucci Pursuant to legislative Decree n. 196/03 you are hereby informed that this email contains confidential information intended only for use of addressee. If you are not the addressee and have received this email by mistake, please send this email to the sender. You may not copy or disseminate this message to anyone. Thank You.
It might be related to bug 22755, but there the client gets ENOSPC On Sep 8, 2010, at 8:02 AM, Gianluca Tresoldi <gianluca.tresoldi at tuttogratis.com > wrote:> Hello everyone > > I''ve an installation with Lustre 1.8.2, Centos 5, x86_64 and I > encountered this problem: > > After several months of smooth operation, client begin to write > empty files without log error,from their point of view writing was > successful. > > OSS wrote, in their log, several lines like: > Sep 8 12:40:31 tgoss-0200 kernel: LustreError: 5816:0:(filter_io.c: > 183:filter_grant_space_left()) lfs01-OST0000: cli 20d94382-3300- > f12e-65d1-c0f1743e1e20/ffff8106a4e30a00 grant 39956230144 > > available 39956226048 and pending 0 > > I checked the availability of space and inodes, but this is not the > problem. > > the problem goes away by rebooting ost. > > This is the second time I have, first at july 2010, second september > 2010. > > Any ideas?It''s a bug? > > Thanks > -- > Gianluca Tresoldi > ***SysAdmin*** > ***Demon''s Trainer*** > Tuttogratis Italia Spa > E-mail: gianluca.tresoldi at tuttogratis.com > http://www.tuttogratis.it > Tel Centralino 02-57313101 > Tel Diretto 02-57313136 > <linux40.jpg>Be open... > *********** Confidentiality Notice & Disclaimer ************* > This message, together with any attachments, is for the confidential > and exclusive use of the addressee(s). If you receive it in error, > please delete the message and its attachments from your system > immediately and notify us by return e-mail. > Do not disclose, copy, circulate or use any information contained in > this e-mail. > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100908/7b314d22/attachment-0001.html
Yes, client gets ENOSP,I see now. Anyway: ThankYou Very Much for your reply ;) On 09/08/10 17:29, Kevin Van Maren wrote:> It might be related to bug 22755, but there the client gets ENOSPC > > > On Sep 8, 2010, at 8:02 AM, Gianluca Tresoldi > <gianluca.tresoldi at tuttogratis.com > <mailto:gianluca.tresoldi at tuttogratis.com>> wrote: > >> Hello everyone >> >> I''ve an installation with Lustre 1.8.2, Centos 5, x86_64 and I >> encountered this problem: >> >> After several months of smooth operation, client begin to write empty >> files without log error,from their point of view writing was successful. >> >> OSS wrote, in their log, several lines like: >> Sep 8 12:40:31 tgoss-0200 kernel: LustreError: >> 5816:0:(filter_io.c:183:filter_grant_space_left()) lfs01-OST0000: cli >> 20d94382-3300-f12e-65d1-c0f1743e1e20/ffff8106a4e30a00 grant >> 39956230144 > available 39956226048 and pending 0 >> >> I checked the availability of space and inodes, but this is not the >> problem. >> >> the problem goes away by rebooting ost. >> >> This is the second time I have, first at july 2010, second september >> 2010. >> >> Any ideas?It''s a bug? >> >> Thanks >> -- >> Gianluca Tresoldi >> ***SysAdmin*** >> ***Demon''s Trainer*** >> Tuttogratis Italia Spa >> E-mail: gianluca.tresoldi at tuttogratis.com >> <mailto:gianluca.tresoldi at tuttogratis.com> >> http://www.tuttogratis.it >> Tel Centralino 02-57313101 >> Tel Diretto 02-57313136 >> <linux40.jpg> <http://www.gnu.org>Be open... >> ************ Confidentiality Notice & Disclaimer ************* >> This message, together with any attachments, is for the confidential >> and exclusive use of the addressee(s). If you receive it in error, >> please delete the message and its attachments from your system >> immediately and notify us by return e-mail. >> Do not disclose, copy, circulate or use any information contained in >> this e-mail. >> * >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at lists.lustre.org <mailto:Lustre-discuss at lists.lustre.org> >> http://lists.lustre.org/mailman/listinfo/lustre-discuss-- Gianluca Tresoldi ***SysAdmin*** ***Demon''s Trainer*** Tuttogratis Italia Spa E-mail: gianluca.tresoldi at tuttogratis.com http://www.tuttogratis.it Tel Centralino 02-57313101 Tel Diretto 02-57313136 <http://www.gnu.org>Be open... ************ Confidentiality Notice & Disclaimer ************* This message, together with any attachments, is for the confidential and exclusive use of the addressee(s). If you receive it in error, please delete the message and its attachments from your system immediately and notify us by return e-mail. Do not disclose, copy, circulate or use any information contained in this e-mail. * -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100909/9022d6a8/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: linux40.jpg Type: image/jpeg Size: 1399 bytes Desc: not available Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100909/9022d6a8/attachment.jpg
the bug 22755 is fixed in 1.8.4 http://wiki.lustre.org/index.php/Use:Change_Log_1.8 On 09/09/2010 11:55 AM, Gianluca Tresoldi wrote:> Yes, client gets ENOSP,I see now. > > Anyway: ThankYou Very Much for your reply ;) > > > On 09/08/10 17:29, Kevin Van Maren wrote: >> It might be related to bug 22755, but there the client gets ENOSPC >> >> >> On Sep 8, 2010, at 8:02 AM, Gianluca Tresoldi >> <gianluca.tresoldi at tuttogratis.com >> <mailto:gianluca.tresoldi at tuttogratis.com>> wrote: >> >>> Hello everyone >>> >>> I''ve an installation with Lustre 1.8.2, Centos 5, x86_64 and I >>> encountered this problem: >>> >>> After several months of smooth operation, client begin to write >>> empty files without log error,from their point of view writing was >>> successful. >>> >>> OSS wrote, in their log, several lines like: >>> Sep 8 12:40:31 tgoss-0200 kernel: LustreError: >>> 5816:0:(filter_io.c:183:filter_grant_space_left()) lfs01-OST0000: >>> cli 20d94382-3300-f12e-65d1-c0f1743e1e20/ffff8106a4e30a00 grant >>> 39956230144 > available 39956226048 and pending 0 >>> >>> I checked the availability of space and inodes, but this is not the >>> problem. >>> >>> the problem goes away by rebooting ost. >>> >>> This is the second time I have, first at july 2010, second september >>> 2010. >>> >>> Any ideas?It''s a bug? >>> >>> Thanks >>> -- >>> Gianluca Tresoldi >>> ***SysAdmin*** >>> ***Demon''s Trainer*** >>> Tuttogratis Italia Spa >>> E-mail: gianluca.tresoldi at tuttogratis.com >>> <mailto:gianluca.tresoldi at tuttogratis.com> >>> http://www.tuttogratis.it >>> Tel Centralino 02-57313101 >>> Tel Diretto 02-57313136 >>> <linux40.jpg> <http://www.gnu.org>Be open... >>> ************ Confidentiality Notice & Disclaimer ************* >>> This message, together with any attachments, is for the confidential >>> and exclusive use of the addressee(s). If you receive it in error, >>> please delete the message and its attachments from your system >>> immediately and notify us by return e-mail. >>> Do not disclose, copy, circulate or use any information contained in >>> this e-mail. >>> * >>> _______________________________________________ >>> Lustre-discuss mailing list >>> Lustre-discuss at lists.lustre.org <mailto:Lustre-discuss at lists.lustre.org> >>> http://lists.lustre.org/mailman/listinfo/lustre-discuss > > > -- > Gianluca Tresoldi > ***SysAdmin*** > ***Demon''s Trainer*** > Tuttogratis Italia Spa > E-mail: gianluca.tresoldi at tuttogratis.com > http://www.tuttogratis.it > Tel Centralino 02-57313101 > Tel Diretto 02-57313136 > <http://www.gnu.org>Be open... > ************ Confidentiality Notice & Disclaimer ************* > This message, together with any attachments, is for the confidential > and exclusive use of the addressee(s). If you receive it in error, > please delete the message and its attachments from your system > immediately and notify us by return e-mail. > Do not disclose, copy, circulate or use any information contained in > this e-mail. > * > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >-- _Gabriele Paciucci_ http://www.linkedin.com/in/paciucci Pursuant to legislative Decree n. 196/03 you are hereby informed that this email contains confidential information intended only for use of addressee. If you are not the addressee and have received this email by mistake, please send this email to the sender. You may not copy or disseminate this message to anyone. Thank You.
I believe grant leak is still possible with 1.8.4, but many of the holes are plugged. Kevin Gabriele Paciucci wrote:> the bug 22755 is fixed in 1.8.4 > > http://wiki.lustre.org/index.php/Use:Change_Log_1.8 > > > > > On 09/09/2010 11:55 AM, Gianluca Tresoldi wrote: >> Yes, client gets ENOSP,I see now. >> >> Anyway: ThankYou Very Much for your reply ;) >> >> >> On 09/08/10 17:29, Kevin Van Maren wrote: >>> It might be related to bug 22755, but there the client gets ENOSPC >>> >>> >>> On Sep 8, 2010, at 8:02 AM, Gianluca Tresoldi >>> <gianluca.tresoldi at tuttogratis.com >>> <mailto:gianluca.tresoldi at tuttogratis.com>> wrote: >>> >>>> Hello everyone >>>> >>>> I''ve an installation with Lustre 1.8.2, Centos 5, x86_64 and I >>>> encountered this problem: >>>> >>>> After several months of smooth operation, client begin to write >>>> empty files without log error,from their point of view writing was >>>> successful. >>>> >>>> OSS wrote, in their log, several lines like: >>>> Sep 8 12:40:31 tgoss-0200 kernel: LustreError: >>>> 5816:0:(filter_io.c:183:filter_grant_space_left()) lfs01-OST0000: >>>> cli 20d94382-3300-f12e-65d1-c0f1743e1e20/ffff8106a4e30a00 grant >>>> 39956230144 > available 39956226048 and pending 0 >>>> >>>> I checked the availability of space and inodes, but this is not the >>>> problem. >>>> >>>> the problem goes away by rebooting ost. >>>> >>>> This is the second time I have, first at july 2010, second >>>> september 2010. >>>> >>>> Any ideas?It''s a bug? >>>> >>>> Thanks >>>> -- >>>> Gianluca Tresoldi