Lustre discuss - May 2006 - Heavy O_APPEND traffic ?

On Mar 21, 2005  16:54 -0800, Jeffrey W. Baker wrote:
> I just tried an experiment with four clients appending to the same file.
> The file was open with O_APPEND and each host was adding 32-byte records
> into the file continuously.  I wanted to test atomic append.
> Unfortunately I don''t think the writes were atomic, because in
83,076
> records I did get one error:
> 
> $ perl -ne ''print(length($_), "\n")'' < test
| sort -n | uniq -c
>       1 31
>   83074 33
>       1 35
> $ perl -ne ''print(length($_) . " $_")'' <
test | grep -C2 ^31
> 33 eab1bd7393c9a6910986cfafb9a202e1
> 35 ec86e615ceac9b299d174eb6e6be3287ba <= misplaced newline
> 31 e4469abaa0b3d6a460436402a905a8
> 33 f2eb261c564212c217228a5c777b3231
> 33 0b68d25d22479ca8d42c02980856a156
> 
> I also saw lock timeouts starting on one host and eventually propagating
> to the other four, and I wasn''t able to write to the file from all
four
> hosts for several minutes* after that.
The O_APPEND performance has the potential to be improved quite a lot,
especially for small records like this, but we have never done so because
it has not yet been a critical factor for any of our customers.
> I know you can''t depend on O_APPEND on NFS, but is it supposed to
work
> on Lustre?
It should work properly on Lustre.  Do you know if the clients were
getting timeouts at the time this record was added?

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.

I just tried an experiment with four clients appending to the same file.
The file was open with O_APPEND and each host was adding 32-byte records
into the file continuously.  I wanted to test atomic append.
Unfortunately I don''t think the writes were atomic, because in 83,076
records I did get one error:

$ perl -ne ''print(length($_), "\n")'' < test |
sort -n | uniq -c
      1 31
  83074 33
      1 35
$ perl -ne ''print(length($_) . " $_")'' < test |
grep -C2 ^31
33 eab1bd7393c9a6910986cfafb9a202e1
35 ec86e615ceac9b299d174eb6e6be3287ba <= misplaced newline
31 e4469abaa0b3d6a460436402a905a8
33 f2eb261c564212c217228a5c777b3231
33 0b68d25d22479ca8d42c02980856a156

I also saw lock timeouts starting on one host and eventually propagating
to the other four, and I wasn''t able to write to the file from all four
hosts for several minutes* after that.

I know you can''t depend on O_APPEND on NFS, but is it supposed to work
on Lustre?

-jwb

* Quite a long time, in fact, 15 minutes later some hosts are still
recovering:

enqueue wait took 101467775us
enqueue wait took 152441579us
etc.