Lustre discuss - Aug 2007 - Simultaneous MDC operations / latency?

Hi,

I was playing with Lustre in an environment where there is noticeable  
roundtrip latency (10ms) between the client and the servers to figure  
out how lustre performs on a WAN link. I tried creating (empty) files  
(open/close) and directories from a single client with multiple  
processes in parallel - the result looks like this:


# of processes  |  Files/s  | Directories/s
---------------------------------------------
       1         |    25     |      50
       2         |    50     |      100
       3         |    75     |     still 100
       4         |    100    |     still 100
       5         |    100    |      ...
       >5        | still 100 |      ...


Can someone shed some light on why there seems to be a hard limit  
around 100 op/s? ( 1000ms / 10ms RTT?) Is there only one simultaneous  
MDC operation possible? Can this value be tuned? What does  
max_rpcs_in_flight exactly do for the MDC in Lustre 1.6? Does file  
creation (for empty files) need twice the number of RPCs compared to  
directories or is this the close operation?

Thanks a lot,

Christoph


-- 
Leibniz-Rechenzentrum (LRZ)
http://www.lrz.de
Abteilung Hochleistungssysteme
Raum I.3.071 - Boltzmannstra?e 1, 85748 Garching
Tel. +49 89 35831-8853

Hello!

On Mon, Aug 20, 2007 at 12:34:53PM +0200, Christoph Biardzki
wrote:
> Can someone shed some light on why there seems to be a hard limit  
> around 100 op/s? ( 1000ms / 10ms RTT?) Is there only one simultaneous  
> MDC operation possible? Can this value be tuned? What does  
There is only one simultaneous rpc in flight for modifying operations
(like open, create, ..., the only exception is close that can be only one in
flight too). This is hardcoded at the moment for recovvery reasons.
> max_rpcs_in_flight exactly do for the MDC in Lustre 1.6? Does file  
This is for nonmodifying operations like getattr, lookup and so on at the
moment.
> creation (for empty files) need twice the number of RPCs compared to  
> directories or is this the close operation?
This is due to close which is rpc as well.
If you replace your open|creat with mknod - you will get RPC numbers
similar to mkdir (use dev_t 0 to create regular files).

Bye,
    Oleg