Lustre discuss - Nov 2007 - trying to BRW to non-existent file xyz

Hi,

when an OST reports "trying to BRW to non-existent file xyz", how can
I find
out which file the inode xyz belongs to?


Thanks,
Bernd

-- 
Bernd Schubert
Q-Leap Networks GmbH

Hello!

On Nov 26, 2007, at 12:08 PM, Bernd Schubert wrote:
> when an OST reports "trying to BRW to non-existent file xyz", how
> can I find
> out which file the inode xyz belongs to?

Usually there is none. Can you tell us more about situations where you  
see this?
Were there any evictions?
One common scenario for this kind of errors is this:
Client opens a file. File gets unlinked. Client is evicted from mds,  
mds notices
it held last reference to a file and issue destroy request for file  
objects (effectively
removing file objects on OSTs).
Now if client would continue to access the file (because it was not  
evicted from ost
or if it reconnected), you will get these errors.

You can do e.g. lfs find /mountpoint -v for your fs (which I guess  
would take quite a
while if it''s big) and then grep the output for interesting objectid  
(just pay
attention that ost index should also match).

Bye,
     Oleg

On Monday 26 November 2007 18:33:02 you wrote:
> Hello!
>
> On Nov 26, 2007, at 12:08 PM, Bernd Schubert wrote:
> > when an OST reports "trying to BRW to non-existent file
xyz", how
> > can I find
> > out which file the inode xyz belongs to?
>
> Usually there is none. Can you tell us more about situations where you
> see this?
> Were there any evictions?
I think so, this message just happened on a rather fresh customer system and 
is rather annoying, since it fills the logs...
I can reproduce this rather soon here, I just need to run fsstress for some 
hours. After some time there are evictions.
> One common scenario for this kind of errors is this:
> Client opens a file. File gets unlinked. Client is evicted from mds,
> mds notices
> it held last reference to a file and issue destroy request for file
> objects (effectively
> removing file objects on OSTs).
> Now if client would continue to access the file (because it was not
> evicted from ost
> or if it reconnected), you will get these errors.
>
> You can do e.g. lfs find /mountpoint -v for your fs (which I guess
> would take quite a
> while if it''s big) and then grep the output for interesting
objectid
> (just pay
> attention that ost index should also match).
Thanks, I will try over night, don''t want to disturb the people there
now.



Thanks a lot,
Bernd

-- 
Bernd Schubert
Q-Leap Networks GmbH

Hi Bernd,
> this message just happened on a rather fresh customer system
> and is rather annoying, since it fills the logs...
Your comment that "it fills the logs" suggests to me that the client
is retrying the operation indefinitely, similar to the description in CFS bug
11211 comment 8 https://bugzilla.clusterfs.com/show_bug.cgi?id=11211#c8. At that
time Andreas Dilger said "the fact that it is retrying on ENOENT is
wrong... should probably only happen for -ETIMEDOUT and -EIO and not other
errors" but I don''t see any record of this ever being changed. Bug
11211 records the fixing of a memory leak associated with the message, but no
change to the looping behavior.
> After some time there are evictions
In Lustre versions that do not have the fix for bug 11211 (pre 1.4.9 according
to bugzilla) the client retry loop will quite quickly consume all of the memory
on the OSS node (about 40 minutes on a server with 2GB RAM in our experience)
and the server will go down. Later Lustre versions will not leak memory, so the
server will stay up, but the looping client will place a considerable load on
it. I would not be surprised if this is the cause of your evictions.

Joe.

-----Original Message-----
From: Bernd Schubert [mailto:bs at q-leap.de]
Sent: 26 November 2007 18:01
To: Oleg Drokin; lustre-discuss at clusterfs.com
Subject: Re: [Lustre-discuss] trying to BRW to non-existent file xyz

On Monday 26 November 2007 18:33:02 you wrote:
> Hello!
>
> On Nov 26, 2007, at 12:08 PM, Bernd Schubert wrote:
> > when an OST reports "trying to BRW to non-existent file
xyz", how
> > can I find
> > out which file the inode xyz belongs to?
>
> Usually there is none. Can you tell us more about situations where you
> see this?
> Were there any evictions?
I think so, this message just happened on a rather fresh customer system and
is rather annoying, since it fills the logs...
I can reproduce this rather soon here, I just need to run fsstress for some
hours. After some time there are evictions.
> One common scenario for this kind of errors is this:
> Client opens a file. File gets unlinked. Client is evicted from mds,
> mds notices
> it held last reference to a file and issue destroy request for file
> objects (effectively
> removing file objects on OSTs).
> Now if client would continue to access the file (because it was not
> evicted from ost
> or if it reconnected), you will get these errors.
>
> You can do e.g. lfs find /mountpoint -v for your fs (which I guess
> would take quite a
> while if it''s big) and then grep the output for interesting
objectid
> (just pay
> attention that ost index should also match).
Thanks, I will try over night, don''t want to disturb the people there
now.



Thanks a lot,
Bernd

--
Bernd Schubert
Q-Leap Networks GmbH