llvm dev - Mar 2014 - [LLVMdev] Can WriteBitcodeToFile be parallelized?

This function (understandably) takes quite a long time, because it has 
to go through each function in module and write its binary.
But it probably can be parallelized if different threads would write 
binaries separately, and then merge them together.

Is this implemented or planned?

Yuri

On 31 March 2014 12:35, Yuri <yuri at rawbw.com> wrote:
> This function (understandably) takes quite a long time, because it has to
> go through each function in module and write its binary.
> But it probably can be parallelized if different threads would write
> binaries separately, and then merge them together.
>
> Is this implemented or planned?

I've never heard this proposed before. Are you interested in working on it?

Note that we require the output to be deterministic, so what exactly would
you be doing? Performing the write out to a memory buffer for each
function, then another thread which passes through these memory buffers
sequentially and issues the actual I/O calls?

I don't think there's currently even primitives in LLVM to do thread
management yet, but now that we've moved to C++11, <thread> may be
unlocked?

I'm hopeful the same technique could end up being used for multi-threaded
per-function .s and .o emission.

Nick
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140331/f9f37aba/attachment.html>

On 03/31/2014 15:29, Nick Lewycky wrote:
> Note that we require the output to be deterministic, so what exactly 
> would you be doing? Performing the write out to a memory buffer for 
> each function, then another thread which passes through these memory 
> buffers sequentially and issues the actual I/O calls?
If there are N CPUs, and the list of functions to be processed, one can 
split the list in N roughly equal sublists, and process them separately 
as much as possible. Then in the end merge the resulting code into one 
block, and resolve the remaining inter-sublist relocation items. Same 
with the DWARF part.

This can produce equivalent binaries if implemented correctly.

> I've never heard this proposed before. Are you interested in working 
> on it?
I normally would have been interested, but have no time currently.

Yuri

You should do some measurement and try to find the bottleneck. Without
knowing the bottleneck it's hard to know whether threading will even speed
things up.

-- Sean Silva


On Mon, Mar 31, 2014 at 3:35 PM, Yuri <yuri at rawbw.com> wrote:
> This function (understandably) takes quite a long time, because it has to
> go through each function in module and write its binary.
> But it probably can be parallelized if different threads would write
> binaries separately, and then merge them together.
>
> Is this implemented or planned?
>
> Yuri
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140407/2f7ffcbb/attachment.html>