Lustre discuss - May 2006 - Tuning Lustre for WAN conditions

On Aug 01, 2005  13:49 -0700, Praveen KJ wrote:
>        I am using Lustre 1.4.2 and expect to test it on a  
> cross-country 100ms RTT network with upto 10 Gbps bandwidth.  
> On a 1ms RTT campus wide network, lustre shows *excellent* performance. 
> With a emulated ( through netem module)  100 ms RTT link,
>  the performance is a order of magnitude slower.  The testing was done 
> with iozone. 
> 
>      Somethings I tried to tune are..
> 
>      i)  Adding --tcpbuf option with --add net in lmc seems to give a 
> unrecognized option error.
>      ii) Changing max_rpcs_in_flight on /proc/fs/lustre/osc/OSC*  to 32
You should also increase the /proc/fs/lustre/osc/OSC*/max_dirty_mb number
correspondingly.
>      iii) Increasing net.core.rmem*/wmem*, net.ipv4.tcp_rmem*/wmem*, 
> net.core.netdev_max_backlog, txqueuelen etc
>          
>      They didn''t change the performance much though.  Is there 
> documentation somewhere of tunable network parameters for Lustre ?
There is a basic tuning wiki at:
https://wiki.clusterfs.com/lustre/LustreProc

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.

Hi,
        I am using Lustre 1.4.2 and expect to test it on a  
cross-country 100ms RTT network with upto 10 Gbps bandwidth.  
 On a 1ms RTT campus wide network, lustre shows *excellent* performance. 
With a emulated ( through netem module)  100 ms RTT link,
  the performance is a order of magnitude slower.  The testing was done 
with iozone. 

      Somethings I tried to tune are..

      i)  Adding --tcpbuf option with --add net in lmc seems to give a 
unrecognized option error.
      ii) Changing max_rpcs_in_flight on /proc/fs/lustre/osc/OSC*  to 32
      iii) Increasing net.core.rmem*/wmem*, net.ipv4.tcp_rmem*/wmem*, 
net.core.netdev_max_backlog, txqueuelen etc
          
      They didn''t change the performance much though.  Is there 
documentation somewhere of tunable network parameters for Lustre ?
   Any suggestions are very much welcome . 

Thanks,
Praveen