llvm dev - Feb 2016 - [RFC] Embedding Bitcode in Object Files

> On Feb 5, 2016, at 2:14 PM, James Y Knight <jyknight at google.com>
wrote:
> 
> On Wed, Feb 3, 2016 at 1:25 PM, Steven Wu via llvm-dev <llvm-dev at
lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
> "__LLVM, __cmdline" is used to store the clang command-line
options.  There are
> few options that are not reflected in the bitcode that we would like to
replay in
> the rebuild. For example, '-O0' option makes us run FastISel during
rebuild.
> 
> Without knowing more details of your implementation, I'd be concerned
about how this might impact deterministic/reproducible builds.
> 
> Source paths are recorded in a number of places, but you can typically fix
that by using -fdebug-prefix-map. But if the entire command-line including the
-fdebug-prefix-map argument gets stored in the output too, then you still have a
problem.

I don't think we need any path in the command line section. We only record
the command-line options that will affect CodeGen. See my example in one of the
preview reply:
> $ clang -fembed-bitcode -O0 test.c -c -###
> "clang" "-cc1"  (...lots of options...) "-o"
"test.bc" "-x" "c" "test.c"   <---
First stage
> "clang" "-cc1" "-triple"
"x86_64-apple-macosx10.11.0" "-emit-obj"
"-fembed-bitcode" "-O0" "-disable-llvm-optzns"
"-o" "test.o" "-x" "ir"
"test.bc"  <--- Second stage
I can't think of any source path that can affect CodeGen. There should not
be any paths other than the bitcode input path and binary output path exists in
the second stage and they are excluded from the command line section as well.
-fdebug-prefix-map is consumed by the front-end and prefixed paths are a part of
the debug info in the metadata. You don't need to encode -fdebug-prefix-map
in the bitcode section to reproduce the object file with the same debug info.
Did that answer your concern?

Thanks

Steven
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160205/45de3b9e/attachment.html>

On Fri, Feb 5, 2016 at 6:06 PM, Steven Wu <stevenwu at apple.com> wrote:
> I don't think we need any path in the command line section. We only
record
> the command-line options that will affect CodeGen. See my example in one of
> the preview reply:
>
> $ clang -fembed-bitcode -O0 test.c -c -###
> "clang" "-cc1"  (...lots of options...) "-o"
"test.bc" "-x" "c" "test.c"
>   <--- First stage
> "clang" "-cc1" "-triple"
"x86_64-apple-macosx10.11.0" "-emit-obj"
> "-fembed-bitcode" "-O0"
"-disable-llvm-optzns" "-o" "test.o"
"-x" "ir"
> "test.bc"  <--- Second stage
>
> I can't think of any source path that can affect CodeGen. There should
not
> be any paths other than the bitcode input path and binary output path
> exists in the second stage and they are excluded from the command line
> section as well. -fdebug-prefix-map is consumed by the front-end and
> prefixed paths are a part of the debug info in the metadata. You don't
need
> to encode -fdebug-prefix-map in the bitcode section to reproduce the object
> file with the same debug info. Did that answer your concern?
>
Great -- it wasn't clear from the first message if you were just embedding
the whole command-line as is. If the plan instead to embed only a few
relevant options, I agree there should be no issue as far as paths go.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160205/de6fc416/attachment.html>

I don't know whether this is an issue in the current implementation, but I
wanted to bring up a potential privacy issue.

In embedding the information, care should be taken to avoid embedding any
information that contains personally identifiable information. This can
certainly occur
if paths need to be embedded, as user names, or other private/confidential
information may be present in the naming of directories and paths.  Generally, I
suspect
that it would be desirable to have an opt-in strategy for designating in the
compiler which pieces of information/options need to be saved, and for all
options marked
as needed, determine whether there is the possibility/likelihood that they may
contain personally identifiable information.

Kevin B. Smith

From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of James Y
Knight via llvm-dev
Sent: Friday, February 05, 2016 3:13 PM
To: Steven Wu <stevenwu at apple.com>
Cc: LLVM Developers Mailing List <llvm-dev at lists.llvm.org>; Clang Dev
<cfe-dev at lists.llvm.org>
Subject: Re: [llvm-dev] [RFC] Embedding Bitcode in Object Files



On Fri, Feb 5, 2016 at 6:06 PM, Steven Wu <stevenwu at
apple.com<mailto:stevenwu at apple.com>> wrote:
I don't think we need any path in the command line section. We only record
the command-line options that will affect CodeGen. See my example in one of the
preview reply:
$ clang -fembed-bitcode -O0 test.c -c -###
"clang" "-cc1"  (...lots of options...) "-o"
"test.bc" "-x" "c" "test.c"   <---
First stage
"clang" "-cc1" "-triple"
"x86_64-apple-macosx10.11.0" "-emit-obj"
"-fembed-bitcode" "-O0" "-disable-llvm-optzns"
"-o" "test.o" "-x" "ir"
"test.bc"  <--- Second stage
I can't think of any source path that can affect CodeGen. There should not
be any paths other than the bitcode input path and binary output path exists in
the second stage and they are excluded from the command line section as well.
-fdebug-prefix-map is consumed by the front-end and prefixed paths are a part of
the debug info in the metadata. You don't need to encode -fdebug-prefix-map
in the bitcode section to reproduce the object file with the same debug info.
Did that answer your concern?

Great -- it wasn't clear from the first message if you were just embedding
the whole command-line as is. If the plan instead to embed only a few relevant
options, I agree there should be no issue as far as paths go.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160206/3641b461/attachment.html>