[LLVMdev] MemoryBuffer/raw_ostream hybrid for linker?

Chris Lattner
Fri, 04 May 2012 17:10:31 -0000

On May 3, 2012, at 6:10 PM, Nick Kledzik wrote:

> Existing llvm code tends to use raw_ostream for writing files.  But
raw_ostream is not a good match for a linker for a couple of reasons:
>
> 1) When the linker creates an executable, the file needs the 'x'
bit set.  Currently raw_fd_ostream has no way to set that.

If this were the only problem, I'd suggest just generalizing raw_fd_ostream
to support this use case.  It would be straight-forward to do.

> 2) The Unix conformance suite actually has some test cases where the linker
is run and the output file does exists but is not writable, or is not writable
but is in a writable directory, or with funky umask values.   raw_fd_ostream
interface has no way to match those semantics.

If this were the only problem :), I would suggest using a new raw_ostream
subclass, where you do custom stuff to get the file system stuff happening that
you want, but reuse all the streaming API aspect of raw_ostream.

> 3) On darwin we have found the linker performs better if it opens the
output file, truncates it to the output size, then mmaps in the file, then
writes directly into that memory buffer.  This avoids the memory copy from the
private buffer to the OS file system buffer in the write() syscall.

This should also be possible with raw_ostream.

> 4) In the model we are using for lld, a streaming output interface is not
optimal.

This is a show-stopper for raw_ostream.  I really don't want raw_ostream to
support seeking or other non-stream behavior. :)

> CIs there an existing solution for these issues in llvm I've
overlooked?   I've searched the bug database and did not find any similar
requests.
>
> Should I propose a new llvm/Support/ class?

Yes please.  We have a variety of other places in the codebase that are using
open/close/read etc directly (grep for #include's of unistd.h).  Using these
APIs is annoying because they are really low level (an expose nonsense like
EINTR handling) and that windows make them annoying to use.

Having a better wrapper for doing real low-level file system stuff (including
seeking) makes perfect sense for llvm/Support!

-Chris