Lustre discuss - Jul 2008 - Luster recovery when clients go away

One of our OSS''s died with a panic last night.  Between when it was  
found (no failover) and restarted two clients had died also.  (nodes  
crashed by user OOM).

Because of this the OST''s now are looking for 626  clients to recover  
when only 624 are up.  So the 624 recover in about 15 minutes, but  
the OST''s on that OSS hang waiting for the last two that are dead and  
not coming back.  Note the MDS reports only 624 clients.

Is there a a way to tell the OST''s to go ahead and evict those two  
clients and finish recovering?  Also "time remaining" has been 0  
sense it was booted.  How long will the OST''s wait before it lets  
operations continue?

Is there any rule to speeding up recovery?  The OSS that crashed sees  
very little cpus/disk/network traffic when recovery is going on so  
any way to speed it up even if it results in a higher load would be  
great to know.

status: RECOVERING
recovery_start: 1217509142
time remaining: 0
connected_clients: 624/626
completed_clients: 624/626
replayed_requests: 0/??
queued_requests: 0
next_transno: 175342162
status: RECOVERING
recovery_start: 1217509144
time remaining: 0
connected_clients: 624/626
completed_clients: 624/626
replayed_requests: 0/??
queued_requests: 0
next_transno: 193097794



Brock Palen
www.umich.edu/~brockp
Center for Advanced Computing
brockp at umich.edu
(734)936-1985

Hi Brock,

On Thursday 31 July 2008 07:30:04 am Brock Palen wrote:
> Is there a a way to tell the OST''s to go ahead and evict those two
> clients and finish recovering?  Also "time remaining" has been 0
> sense it was booted.  How long will the OST''s wait before it lets
> operations continue?
Well, there should be a timeout, and recovery should be aborted anyway 
when then "time remaining" counter reaches 0, no matter how many 
clients have been recovered (the remaining ones are evicted, I 
believe).

In case this doesn''t work, you can still avoid the recovery process by 
mounting your OSTs with -o abort_recov.

Cheers,
-- 
Kilian

On Jul 31, 2008  10:30 -0400, Brock Palen wrote:
> One of our OSS''s died with a panic last night.  Between when it
was
> found (no failover) and restarted two clients had died also.  (nodes  
> crashed by user OOM).
> 
> Because of this the OST''s now are looking for 626  clients to
recover
> when only 624 are up.  So the 624 recover in about 15 minutes, but  
> the OST''s on that OSS hang waiting for the last two that are dead
and
> not coming back.  Note the MDS reports only 624 clients.
> 
> Is there a a way to tell the OST''s to go ahead and evict those two
> clients and finish recovering?  Also "time remaining" has been 0
> sense it was booted.  How long will the OST''s wait before it lets
> operations continue?
> 
> Is there any rule to speeding up recovery?  The OSS that crashed sees  
> very little cpus/disk/network traffic when recovery is going on so  
> any way to speed it up even if it results in a higher load would be  
> great to know.
> 
> status: RECOVERING
> recovery_start: 1217509142
> time remaining: 0
> connected_clients: 624/626
> completed_clients: 624/626
> replayed_requests: 0/??
> queued_requests: 0
> next_transno: 175342162
> status: RECOVERING
> recovery_start: 1217509144
> time remaining: 0
> connected_clients: 624/626
> completed_clients: 624/626
> replayed_requests: 0/??
> queued_requests: 0
> next_transno: 193097794
The recovery should time out after about 5 minutes (with default 100s
timeouts).  The recovery goes as fast as clients connect and submit RPCs
for replay.  In the case where all clients connect then recovery is
finished as soon as all clients report completion.

Are you saying the system is still stuck in recovery after more than 5
or 10 minutes?

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.