Hi, Is the client writeback cache only active for each file written or can it cache multiple files simultaneously? I''m assuming it is the former. So if I write a big single file the cache will help but if I write lots of small files it will be of little use? I''m testing Lustre over a high latency VPN and finding that reading from a home dir on Lustre is very good (thanks to the negative dentry cache and client cache) but writing lots of small files is still very slow for each file. Cheers, Daire
On Feb 27, 2009 12:19 +0000, Daire Byrne wrote:> Is the client writeback cache only active for each file written or can > it cache multiple files simultaneously? I''m assuming it is the former. So > if I write a big single file the cache will help but if I write lots > of small files it will be of little use? I''m testing Lustre over a high > latency VPN and finding that reading from a home dir on Lustre is very > good (thanks to the negative dentry cache and client cache) but writing > lots of small files is still very slow for each file.No, in fact the writeback cache is not related to the file size. If you write many small files it should be able to cache the same total amount of space as the single large file. One problem, however, is that once you fill the cache with small files the cache flushing will likely be much slower (due to less data sent per RPC) so you may experience a sharp drop-off in total throughput. Secondly, with small files there is also the overhead of creating the many small files on the MDS, which will likely take as long as the RPCs being sent to the OSTs with data. There is not currently a metadata writeback cache that allows files to be created only on the client. That is something we are examining, but it is still a ways off. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.
Andreas, ----- "Andreas Dilger" <adilger at sun.com> wrote:> Secondly, with small files there is also the overhead of creating > the many small files on the MDS, which will likely take as long > as the RPCs being sent to the OSTs with data. There is not currently > a metadata writeback cache that allows files to be created only on > the client. That is something we are examining, but it is still > a ways off.Ah yes of course. This is the performance bottleneck I''m seeing over a high latency link. Thanks for the clarification. Daire
On Mar 04, 2009 09:22 +0000, Daire Byrne wrote:> ----- "Andreas Dilger" <adilger at sun.com> wrote: > > Secondly, with small files there is also the overhead of creating > > the many small files on the MDS, which will likely take as long > > as the RPCs being sent to the OSTs with data. There is not currently > > a metadata writeback cache that allows files to be created only on > > the client. That is something we are examining, but it is still > > a ways off. > > Ah yes of course. This is the performance bottleneck I''m seeing over > a high latency link. Thanks for the clarification.You can increase the number of RPCs in flight for this client via /proc/fs/lustre/osc/*/max_rpcs_in_flight, and it should improve small-file IO performance. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.