On Nov 20, 2006 23:47 -0400, Peter Bojanic wrote:> It''s true. I did raise users'' expectations for improved
metadata
> performance for patchless client. There will probably be a number of
> users who are interested in plans about this.
> On 2006-11-20, at 11:33 , Bernadat, Philippe wrote:
> >At last week''s CFS BOF you said that even without the kernel
patches
> >you may be able to get as good or even better performance than before.
> >
> >I remember you telling me that I could get more info from Andreas or
> >someone else. So can we know more ?
The patchless client came about because many customers do not want to
have support issues with a patched client. Also, these patches are in a
tricky part of the kernel and are a source of bugs. So removing them is
beneficial for both CFS and customers. In theory (and practise in some
cases) this hurts metadata performance, and that needs to be addressed.
The patchless client still utilizes the intent open operation (which
was accepted into the upstream kernels in a "not useful for anything
but open" form). Lustre uses normal VFS operations for everything else
(meaning that some operations might take multiple RPCs where they would
previously take only a single one).
The truth is that in many real-world use cases (such as GNU rm, ls as
opposed to focussed benchmarks) the Lustre "raw" operations
weren''t of
much benefit because the user-space code is doing an explicit "stat"
and "access" call on the target file anyways, destroying any advantage
that the raw operations provide.
More significantly, in detailed RPC investigation for these use cases
it is seen that the Lustre network protocol can be optimized to either
reduce the RPC count, or overlap serial operations to reduce total time.
In the "rm -r" case (where it is doing stat+access calls on the target
anyways), we have observed that the MDS unlink and OST destroy operations
are always immediately cancelling the locks the client was just granted
for stat. This suggests that if we bundle the lock cancellation with
the original unlink/destroy RPC request that we save a full round-trip
RPC latency for each during the unlink syscall. This is expected to
improve the single-file "rm -r" performance by 25% (remove 2 of 8
RPCs)
over the existing code, regardless of whether the client is patched or not.
Similar cases exist for rename, open(O_WRONLY), etc where high-level
analysis of the RPCs being done show opportunity to improve the wire
protocol to remove costly RPCs without need to modify the kernel.
There are other similar (though longer-term) optimizations to the protocol
such as size-on-MDS, client side stat-ahead for "ls -l" (readdir+stat)
type operations, etc that can also be implemented. In the "rm -r"
case
above, stat-ahead would hide the latency of the MDS and OST stat/getattr
operations, and size-on-MDS would avoid the OST getattr RPCs for closed
files (major win all around, though very complex to get right).
Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.