llvm dev - Oct 2016 - RFC: APIs for bitcode files containing multiple modules

Hi all,

As mentioned in my recent RFC entitled "RFC: a more detailed design for
ThinLTO + vcall CFI" I would like to introduce the ability for bitcode
files to contain multiple modules. In https://reviews.llvm.org/D24786 I
took a step towards that by proposing a change to the module format so that
the block info block is stored at the top level. The next step is to think
about what the API would look like for reading and writing multiple modules.

Here's what I have in mind. To create a multi-module bitcode file, you
would create a BitcodeWriter object and add modules to it:

BitcodeWriter W(OS);
W.addModule(M1);
W.addModule(M2);
W.write();

Reading a multi-module bitcode file would be supported with a BitcodeReader
class. Each of the functional reader APIs in ReaderWriter.h would have a
member function on BitcodeReader. We would also have a next() member
function which would move to the next module in the file. For example:

BitcodeReader R(MBRef);
Expected<bool> B = R.hasGlobalValueSummary();
std::unique_ptr<Module> M1 = R.getLazyModule(Ctx); // lazily load the
first
module
R.next();
std::unique_ptr<Module> M2 = R.parseBitcodeFile(Ctx); // eagerly load the
second module

We'd continue to support the existing functional APIs in ReaderWriter.h for
convenience in the common case where the bitcode file has a single module.

Thanks,
-- 
-- 
Peter
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161025/206249cc/attachment.html>

> On Oct 25, 2016, at 6:28 PM, Peter Collingbourne <peter at pcc.me.uk>
wrote:
> 
> Hi all,
> 
> As mentioned in my recent RFC entitled "RFC: a more detailed design
for ThinLTO + vcall CFI" I would like to introduce the ability for bitcode
files to contain multiple modules. In https://reviews.llvm.org/D24786
<https://reviews.llvm.org/D24786> I took a step towards that by proposing
a change to the module format so that the block info block is stored at the top
level. The next step is to think about what the API would look like for reading
and writing multiple modules.
> 
> Here's what I have in mind. To create a multi-module bitcode file, you
would create a BitcodeWriter object and add modules to it:
> 
> BitcodeWriter W(OS);
> W.addModule(M1);
> W.addModule(M2);
> W.write();
That requires the two modules to lives longer than the bitcode write, the API
could be:

BitcodeWriter W(OS);
W.writeModule(M1);
// delete M1
// ...
// create M2
W.writeModule(M2);

(Maybe you had this in mind, but the API naming didn’t reflect it so I’m not
sure).

> 
> Reading a multi-module bitcode file would be supported with a BitcodeReader
class. Each of the functional reader APIs in ReaderWriter.h would have a member
function on BitcodeReader. We would also have a next() member function which
would move to the next module in the file. For example:
> 
> BitcodeReader R(MBRef);
> Expected<bool> B = R.hasGlobalValueSummary();
> std::unique_ptr<Module> M1 = R.getLazyModule(Ctx); // lazily load the
first module
> R.next();
> std::unique_ptr<Module> M2 = R.parseBitcodeFile(Ctx); // eagerly load
the second module

That makes the API quite stateful, you may have good implementation reason for
this, but they’re not clear to me.
I rather see the bitcode reader as a random access container, iterating over
modules.

— 
Mehdi

> 
> We'd continue to support the existing functional APIs in ReaderWriter.h
for convenience in the common case where the bitcode file has a single module.
> 
> Thanks,
> -- 
> -- 
> Peter
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161025/155a2232/attachment.html>

On Tue, Oct 25, 2016 at 8:36 PM, Mehdi Amini <mehdi.amini at apple.com>
wrote:
>
> On Oct 25, 2016, at 6:28 PM, Peter Collingbourne <peter at pcc.me.uk>
wrote:
>
> Hi all,
>
> As mentioned in my recent RFC entitled "RFC: a more detailed design
for
> ThinLTO + vcall CFI" I would like to introduce the ability for bitcode
> files to contain multiple modules. In https://reviews.llvm.org/D24786 I
> took a step towards that by proposing a change to the module format so that
> the block info block is stored at the top level. The next step is to think
> about what the API would look like for reading and writing multiple
modules.
>
> Here's what I have in mind. To create a multi-module bitcode file, you
> would create a BitcodeWriter object and add modules to it:
>
> BitcodeWriter W(OS);
> W.addModule(M1);
> W.addModule(M2);
> W.write();
>
>
> That requires the two modules to lives longer than the bitcode write, the
> API could be:
>
> BitcodeWriter W(OS);
> W.writeModule(M1);
> // delete M1
> // ...
> // create M2
> W.writeModule(M2);
>
> (Maybe you had this in mind, but the API naming didn’t reflect it so I’m
> not sure).
>
In the API I prototyped, I took the maximum BitsRequiredForTypeIndices
value from all the modules, and used it to produce the abbreviations for
the top level block info block (without this I was seeing "Unexpected
abbrev ordering!" errors in the bitcode writer as a result of emitting the
"same" abbreviation multiple times). That would have required us to
keep
the modules around until the call to write(). However, let me revisit this,
because it does not seem necessary (i.e. we can just continue to emit block
info blocks within the module block except with different abbreviation
numbers for each module).
> Reading a multi-module bitcode file would be supported with a
> BitcodeReader class. Each of the functional reader APIs in ReaderWriter.h
> would have a member function on BitcodeReader. We would also have a next()
> member function which would move to the next module in the file. For
> example:
>
> BitcodeReader R(MBRef);
> Expected<bool> B = R.hasGlobalValueSummary();
> std::unique_ptr<Module> M1 = R.getLazyModule(Ctx); // lazily load the
> first module
> R.next();
> std::unique_ptr<Module> M2 = R.parseBitcodeFile(Ctx); // eagerly load
the
> second module
>
>
>
> That makes the API quite stateful, you may have good implementation reason
> for this, but they’re not clear to me.
> I rather see the bitcode reader as a random access container, iterating
> over modules.
>
Random access seems reasonable to me as well. I will see how feasible that
is.

Thanks,
-- 
-- 
Peter
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161026/a5465bf1/attachment.html>