llvm dev - Apr 2019 - Can we do atomic write to a file by using raw_fd_ostream?

Hi all,

In a pass I’m using raw_fd_ostream to write a string to a file. Does
raw_fd_ostream guarantee
the write is atomic when I do parallel compilation (currently I’m using -j8)? I
have some errs() to
print information for debugging purposes and I saw that the printed information
gets messed up sometime.
If I’m writing a string with the format of “A:B:C”, is it possible that I got
“A1:B2:C1” and “A2:B1:C2”
instead of the correct one “A1:B1:C1” and “A2:B2:C2”? And if this is possible,
how can I do atomic write to a file?

Thanks,
- Jie
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190406/06042282/attachment.html>

Hi Jie,


On Sat, 6 Apr 2019 at 01:43, Jie Zhou via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
> In a pass I’m using raw_fd_ostream to write a string to a file. Does
raw_fd_ostream guarantee
> the write is atomic when I do parallel compilation (currently I’m using
-j8)?
raw_fd_ostream doesn't really attempt to solve this problem. All
writes get filtered through the write_impl function which calls POSIX
::write in chunks (large ones -- 1GB). Additionally, individual calls
to << operators may or may not be buffered, and if they are it's with
a fixed size buffer rather than a transaction based system.

According to
https://stackoverflow.com/questions/1154446/is-file-append-atomic-in-unix,
those individual calls to ::write are atomic and won't interfere
provided the file has been opened in append mode. So your goal should
be to ensure that each logging event gets implemented as a single call
to ::write.

The most straightforward way would obviously be to do it yourself. In
LLVM you might do your normal writing to a raw_string_ostream and then
call ::write once on that.

Cheers.

Tim.

The other solution would be to use ninja instead of make if you can. Ninja
always buffers command output so that error messages from a parallel build
don't interleave. https://ninja-build.org/manual.html#_comparison_to_make

On Sat, Apr 6, 2019 at 4:02 PM Tim Northover via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hi Jie,
>
>
> On Sat, 6 Apr 2019 at 01:43, Jie Zhou via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
> > In a pass I’m using raw_fd_ostream to write a string to a file. Does
> raw_fd_ostream guarantee
> > the write is atomic when I do parallel compilation (currently I’m
using
> -j8)?
>
> raw_fd_ostream doesn't really attempt to solve this problem. All
> writes get filtered through the write_impl function which calls POSIX
> ::write in chunks (large ones -- 1GB). Additionally, individual calls
> to << operators may or may not be buffered, and if they are it's
with
> a fixed size buffer rather than a transaction based system.
>
> According to
> https://stackoverflow.com/questions/1154446/is-file-append-atomic-in-unix,
> those individual calls to ::write are atomic and won't interfere
> provided the file has been opened in append mode. So your goal should
> be to ensure that each logging event gets implemented as a single call
> to ::write.
>
> The most straightforward way would obviously be to do it yourself. In
> LLVM you might do your normal writing to a raw_string_ostream and then
> call ::write once on that.
>
> Cheers.
>
> Tim.
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190408/e01990a9/attachment.html>