Lustre discuss - Aug 2008 - LBUG on client: Found existing inode ... in lock

Hello,

we''re seing an LBUG on clients running with Lustre 1.6.5.1 (the servers
are
still under 1.6.4.3). I tried finding this in bugzilla with no success. There
seems to be some data inconsistency, can somebody please tell me whether this
is rather on the server side (the data on disk is inconsistent?) or could it
rather be a bug on the client, only?

Thanks in advance for any hint...

Erich


Aug 19 16:25:31 harper3 kernel: Lustre: lustre-OST0023-osc-ffff810228a56c00:
Connection restored to service lustre-OST0023 using nid 10.3.0.233 at o2ib.
Aug 19 16:25:31 harper3 kernel: LustreError:
1168:0:(osc_request.c:2866:osc_set_data_with_check()) ### inconsistent
l_ast_data found ns: lustre-OST0022-osc-ffff810228a56c00 lock:
ffff81016276b600/0x4509c644cc57c7f2 lrc: 4/0,2 mode: PW/PW res: 17055/0 rrc: 2
type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags:
80120000 remote: 0x29a50977c8f92020 expref: -99 pid: 1154
Aug 19 16:25:31 harper3 kernel: LustreError:
1168:0:(osc_request.c:2872:osc_set_data_with_check())
ASSERTION(old_inode->i_state & I_FREEING) failed:Found existing inode
ffff81018510dd78/10979900/3759376635 state 7 in lock: setting data to
ffff81019cff0f38/10979916/3759376661
Aug 19 16:25:31 harper3 kernel: LustreError:
1168:0:(osc_request.c:2872:osc_set_data_with_check()) LBUG
Aug 19 16:25:31 harper3 kernel: 
Aug 19 16:25:31 harper3 kernel: Call Trace:
Aug 19 16:25:31 harper3 kernel:  [<ffffffff883ffc1a>]
:libcfs:lbug_with_loc+0x7a/0xc0
Aug 19 16:25:31 harper3 kernel:  [<ffffffff88615076>]
:osc:osc_set_data_with_check+0x186/0x1d0
Aug 19 16:25:31 harper3 kernel:  [<ffffffff88620f40>]
:osc:osc_enqueue+0x180/0x590
Aug 19 16:25:31 harper3 kernel:  [<ffffffff886b1455>]
:lov:lov_set_add_req+0x15/0x20
Aug 19 16:25:31 harper3 kernel:  [<ffffffff886b7761>]
:lov:lov_prep_enqueue_set+0x981/0xb60
Aug 19 16:25:31 harper3 kernel:  [<ffffffff886bd24c>]
:lov:lsm_unpackmd_plain+0x1c/0x190
Aug 19 16:25:31 harper3 kernel:  [<ffffffff886a11b2>]
:lov:lov_enqueue+0x612/0x8b0
Aug 19 16:25:31 harper3 kernel:  [<ffffffff88400168>]
:libcfs:cfs_alloc+0x28/0x60
Aug 19 16:25:31 harper3 kernel:  [<ffffffff886ed26c>]
:lustre:ll_glimpse_size+0x62c/0xc20
Aug 19 16:25:31 harper3 kernel:  [<ffffffff8850edb0>]
:ptlrpc:ldlm_lock_add_to_lru_nolock+0x60/0xa0
Aug 19 16:25:31 harper3 kernel:  [<ffffffff8000948d>] __d_lookup+0xb0/0xff
Aug 19 16:25:31 harper3 kernel:  [<ffffffff8851246a>]
:ptlrpc:ldlm_lock_decref+0x9a/0xc0
Aug 19 16:25:31 harper3 kernel:  [<ffffffff88619250>]
:osc:osc_extent_blocking_cb+0x0/0x2b0
Aug 19 16:25:31 harper3 kernel:  [<ffffffff8852bf60>]
:ptlrpc:ldlm_completion_ast+0x0/0x6a0
Aug 19 16:25:31 harper3 kernel:  [<ffffffff886efce0>]
:lustre:ll_glimpse_callback+0x0/0x440
Aug 19 16:25:31 harper3 kernel:  [<ffffffff886db0ae>]
:lustre:ll_intent_drop_lock+0x8e/0xb0
Aug 19 16:25:31 harper3 kernel:  [<ffffffff886ededc>]
:lustre:ll_inode_revalidate_it+0x67c/0x720
Aug 19 16:25:31 harper3 kernel:  [<ffffffff8871da10>]
:lustre:ll_mdc_blocking_ast+0x0/0x510
Aug 19 16:25:31 harper3 kernel:  [<ffffffff886f0367>]
:lustre:ll_file_release+0x247/0x2e0
Aug 19 16:25:31 harper3 kernel:  [<ffffffff886edfa4>]
:lustre:ll_getattr_it+0x24/0x110
Aug 19 16:25:31 harper3 kernel:  [<ffffffff886ee0c4>]
:lustre:ll_getattr+0x34/0x40
Aug 19 16:25:31 harper3 kernel:  [<ffffffff800283a7>]
vfs_stat_fd+0x32/0x4a
Aug 19 16:25:31 harper3 kernel:  [<ffffffff886f0367>]
:lustre:ll_file_release+0x247/0x2e0
Aug 19 16:25:31 harper3 kernel:  [<ffffffff8000c199>]
_atomic_dec_and_lock+0x39/0x57
Aug 19 16:25:31 harper3 kernel:  [<ffffffff800231fb>]
sys_newstat+0x19/0x31
Aug 19 16:25:31 harper3 kernel:  [<ffffffff8005c229>] tracesys+0x71/0xe0
Aug 19 16:25:31 harper3 kernel:  [<ffffffff8005c28d>] tracesys+0xd5/0xe0

On Wed, 2008-08-20 at 16:00 +0200, Erich Focht wrote:
> Hello,
> 
> we''re seing an LBUG on clients running with Lustre 1.6.5.1 (the
servers are
> still under 1.6.4.3). I tried finding this in bugzilla with no success.
Bug 16427.

b.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20080820/2569732c/attachment.bin

On Mittwoch 20 August 2008, Brian J. Murrell wrote:
> On Wed, 2008-08-20 at 16:00 +0200, Erich Focht wrote:
> > Hello,
> > 
> > we''re seing an LBUG on clients running with Lustre 1.6.5.1
(the servers are
> > still under 1.6.4.3). I tried finding this in bugzilla with no
success.
> 
> Bug 16427.
Thanks but... I don''t seem to be authorized to see that bug (?).

Is that bug fixed in 1.6.5.1? Any advice resulting from its content?

Thanks & best regards,
Erich

On Wed, 2008-08-20 at 16:47 +0200, Erich Focht wrote:
> 
> Thanks but... I don''t seem to be authorized to see that bug (?).
Oh, yes.  :-(  I tend to forget to look at the privacy settings on bugs.
> Is that bug fixed in 1.6.5.1?
No.  Reported on 1.6.5 in fact.
> Any advice resulting from its content?
None yet.  It''s been escalated to engineering though.

b.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20080820/d842549d/attachment.bin

Thanks, Brian.

A more general comment: what is the use of invisible bugs, anyway? I
suppose the bug has been set "private" by the reporter.
Wouldn''t it
actually make sense to have all bugs open, such that others are warned
of the issue? Guess if somebody doesn''t want to disclose the company
on behalf of which the bug was reported, a mechanism for anonymizing the
reporter would make more sense. Anyway, I feel like hiding bugs is bad
in an open source project.

Regards,
Erich

On Mittwoch 20 August 2008, Brian J. Murrell wrote:
> On Wed, 2008-08-20 at 16:47 +0200, Erich Focht wrote:
> > 
> > Thanks but... I don''t seem to be authorized to see that bug
(?).
> 
> Oh, yes.  :-(  I tend to forget to look at the privacy settings on bugs.
> 
> > Is that bug fixed in 1.6.5.1?
> 
> No.  Reported on 1.6.5 in fact.
> 
> > Any advice resulting from its content?
> 
> None yet.  It''s been escalated to engineering though.

I agree that hiding bugs is quite bad.

<rant>
I''m going to be an open source curmudgeon for a minute and say that if
Sun/CFS wants to track customer-specific, sensitive data bugs, they need
to have a separate system and pay someone to make sure that all internal
bugs are santized and put into the open source project bug tracker.

Sun/CFS gets a huge mindshare and market acceptance benefit from the
open source project. Hiding bugs WILL kill that mindshare and
acceptance benefit. If Lustre isn''t a full first class public open
source project, all the new and really innovative work will get done
on competing open source filesystems.

</rant>

Now back to real work ;)

On Thu, Aug 21, 2008 at 04:40:54PM +0200, Erich Focht
wrote:
> Thanks, Brian.
> 
> A more general comment: what is the use of invisible bugs, anyway? I
> suppose the bug has been set "private" by the reporter.
Wouldn''t it
> actually make sense to have all bugs open, such that others are warned
> of the issue? Guess if somebody doesn''t want to disclose the
company
> on behalf of which the bug was reported, a mechanism for anonymizing the
> reporter would make more sense. Anyway, I feel like hiding bugs is bad
> in an open source project.
> 
> Regards,
> Erich
> 
> On Mittwoch 20 August 2008, Brian J. Murrell wrote:
> > On Wed, 2008-08-20 at 16:47 +0200, Erich Focht wrote:
> > > 
> > > Thanks but... I don''t seem to be authorized to see that
bug (?).
> > 
> > Oh, yes.  :-(  I tend to forget to look at the privacy settings on
bugs.
> > 
> > > Is that bug fixed in 1.6.5.1?
> > 
> > No.  Reported on 1.6.5 in fact.
> > 
> > > Any advice resulting from its content?
> > 
> > None yet.  It''s been escalated to engineering though.
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
-- 
--------------------------------------------------------------------------
Troy Benjegerdes                ''da hozer''               
hozer at hozed.org

Somone asked me why I work on this free (http://www.gnu.org/philosophy/)
software stuff and not get a real job. Charles Shultz had the best answer:

"Why do musicians compose symphonies and poets write poems? They do it
because life wouldn''t have any meaning for them if they
didn''t. That''s why
I draw cartoons. It''s my life." -- Charles Shultz

On Thu, 2008-08-21 at 16:40 +0200, Erich Focht wrote:
> 
> A more general comment: what is the use of invisible bugs, anyway?
You have to remember that we have customers who have sensitive data.
Sometimes they need to share this data with us and yet not have it
publicly consumable.
> I
> suppose the bug has been set "private" by the reporter.
Likely.
> Wouldn''t it
> actually make sense to have all bugs open, such that others are warned
> of the issue?
Ideally, yes, of course.  But given that there are users that are going
to have sensitive data, providing a means for them to share that with us
in order to help identify and fix a bug is a better choice than having
them refrain from reporting the bug because they have no way to provide
the data we need to debug it.  Wouldn''t you agree?
> Guess if somebody doesn''t want to disclose the company
> on behalf of which the bug was reported, a mechanism for anonymizing the
> reporter would make more sense.
Company may be one thing they want to remain secret, but more likely
it''s the actual data they don''t want publicly viewable -- the
data
without which we can''t fix the bug.  Not all data can simply be
"anonymized".
> Anyway, I feel like hiding bugs is bad
> in an open source project.
As I said, yes, it is not ideal, but is a better situation than just not
having bugs reported.

In your particular case, when you reported your bug, work was already
under way solving the problem in the private bug.  Ultimately that is
good news for you as it will ultimately reduce the time until you will
see a fix from what it would have been would that bug reporter have not
bothered reporting it due to not being able to provide the information
needed for us to debug it.

b.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20080821/f8a03f4e/attachment.bin

All LLNL bug reports to Sun are public.  We are willing to do the extra 
work to sanitize bug reports ourselves.  It''s usually little or no
effort anyway - we rarely need to post an entire crash dump or a stack 
trace that includes application data.

I would encourage sites that may have chosen to keep their bug reports 
private due consideration of the negative side effects to reconsider 
their policy.  It sucks for the rest of us - ever had a bug closed as 
a duplicate of a private bug?

It''s not really Sun''s fault IMHO.  Customers can be so
demanding :-)

Jim

On Thu, Aug 21, 2008 at 11:25:55AM -0400, Brian J. Murrell
wrote:
> On Thu, 2008-08-21 at 16:40 +0200, Erich Focht wrote:
> > 
> > A more general comment: what is the use of invisible bugs, anyway?
> 
> You have to remember that we have customers who have sensitive data.
> Sometimes they need to share this data with us and yet not have it
> publicly consumable.
> 
> > I
> > suppose the bug has been set "private" by the reporter.
> 
> Likely.
> 
> > Wouldn''t it
> > actually make sense to have all bugs open, such that others are warned
> > of the issue?
> 
> Ideally, yes, of course.  But given that there are users that are going
> to have sensitive data, providing a means for them to share that with us
> in order to help identify and fix a bug is a better choice than having
> them refrain from reporting the bug because they have no way to provide
> the data we need to debug it.  Wouldn''t you agree?
> 
> > Guess if somebody doesn''t want to disclose the company
> > on behalf of which the bug was reported, a mechanism for anonymizing
the
> > reporter would make more sense.
> 
> Company may be one thing they want to remain secret, but more likely
> it''s the actual data they don''t want publicly viewable --
the data
> without which we can''t fix the bug.  Not all data can simply be
> "anonymized".
> 
> > Anyway, I feel like hiding bugs is bad
> > in an open source project.
> 
> As I said, yes, it is not ideal, but is a better situation than just not
> having bugs reported.
> 
> In your particular case, when you reported your bug, work was already
> under way solving the problem in the private bug.  Ultimately that is
> good news for you as it will ultimately reduce the time until you will
> see a fix from what it would have been would that bug reporter have not
> bothered reporting it due to not being able to provide the information
> needed for us to debug it.
> 
> b.
> 

> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http:// lists.lustre.org/mailman/listinfo/lustre-discuss

On Donnerstag 21 August 2008, Brian J. Murrell wrote:
> On Thu, 2008-08-21 at 16:40 +0200, Erich Focht wrote:
> > 
> > A more general comment: what is the use of invisible bugs, anyway?
> 
> You have to remember that we have customers who have sensitive data.
> Sometimes they need to share this data with us and yet not have it
> publicly consumable.
Of course I can understand that, and the reasoning. I''m also very happy
that
the bug I reported was already around and probably the fix will come sooner.
Just looking for a way to stay informed on what''s going on with this
bug.

If I file it (as non-private) maybe it will be closed as duplicate, but
actually it would be usefull to keep it around and at least synchronize the
status with that of the private bug. This way people can at least find out
what''s hapenning and find the non-secret error message. Duplication has
the
price of additional effort somewhere, but keeping bugs secret has drawbacks,
too.

Anyway, SUN has clearly shown the will to open up the Lustre development, so
maybe there will be some better way to deal with secret bugs, too.

Regards,
Erich

On Thu, 2008-08-21 at 19:58 +0200, Erich Focht wrote:
> 
> Just looking for a way to stay informed on what''s going on with
this bug.
Indeed.  I can understand that.
> If I file it (as non-private) maybe it will be closed as duplicate, but
> actually it would be usefull to keep it around and at least synchronize the
> status with that of the private bug.
Yes.  If you were to file it and make it public and note that you were
told it was an instance of the private bug, while we don''t have any
official policies on what would happen, we could try to at least provide
some linkage between them.
> This way people can at least find out
> what''s hapenning and find the non-secret error message.
Indeed.
> Duplication has the
> price of additional effort somewhere, but keeping bugs secret has
drawbacks,
> too.
Agreed.

b.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20080821/e940966f/attachment.bin

> > If I file it (as non-private) maybe it will be closed as duplicate,
but
> > actually it would be usefull to keep it around and at least
synchronize the
> > status with that of the private bug.
> 
> Yes.  If you were to file it and make it public and note that you were
> told it was an instance of the private bug, while we don''t have
any
> official policies on what would happen, we could try to at least provide
> some linkage between them.
> 
> > This way people can at least find out
> > what''s hapenning and find the non-secret error message.
> 
> Indeed.
I would like to see a policy that all private bugs have a public
duplicate, even if it''s just a bug number and a status.

On Aug 21, 2008  09:59 -0500, Troy Benjegerdes wrote:
> <rant>
> I''m going to be an open source curmudgeon for a minute and say
that if
> Sun/CFS wants to track customer-specific, sensitive data bugs, they need
> to have a separate system and pay someone to make sure that all internal
> bugs are santized and put into the open source project bug tracker.
> 
> Sun/CFS gets a huge mindshare and market acceptance benefit from the
> open source project. Hiding bugs WILL kill that mindshare and
> acceptance benefit. If Lustre isn''t a full first class public open
> source project, all the new and really innovative work will get done
> on competing open source filesystems.
<anti-rant>
Note that bugs the Lustre group files are public.  The privacy is only
a choice of the customer.  This was true when we were CFS, and in fact
a lot MORE of the CFS bugs were private because we had to worry more
about keeping our plans/designs closely guarded as a form of financial
security.

Since being part of Sun the Lustre designs and design discussions
are available to the public (e.g. lustre-devel, wiki, public bugs for
feature development, and internal debugging discussions, etc).  In truth,
it hasn''t made a huge difference in external contributions to code or
design, because the barrier is more to do with complexity than with
being able to just read the document.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

> Since being part of Sun the Lustre designs and design discussions
> are available to the public (e.g. lustre-devel, wiki, public bugs for
> feature development, and internal debugging discussions, etc).  In truth,
> it hasn''t made a huge difference in external contributions to code
or
> design, because the barrier is more to do with complexity than with
> being able to just read the document.
manual.lustre.org and the other wikis are very nice. Keep up the good
work.