llvm dev - Apr 2021 - [RFC] Generating LLD reproducers on crashes

> I am skeptical that users will want to have this behavior by default.
> If this behavior is guarded by an option, it might be fine.
That's a good point. If the reproducer will be more than a few hundreds
MiBs, it is definitely not suitable to be enabled by default. I agree it's
better to be guarded by an option flag such as `--gen-lld-crash-reproducer`.

On Wed, Apr 14, 2021 at 2:40 PM Fangrui Song <maskray at google.com>
wrote:
>
> On 2021-04-14, Haowei Wu via llvm-dev wrote:
> >*Background / Motivation*
> >
> >Both clang and lld have the ability to generate a reproducer (an
archive
> >with input files and invoker script to reproduce the clang/lld build).
> >While clang will generate a reproducer archive when a crash happens,
lld
> >only generates a reproducer when '--reproduce' flag is
explicitly set
> (this
> >is equivalent to Clang's -gen-reproducer flag). This is not very
helpful
> >for debugging lld bugs, particularly when the crash happens in building
> big
> >projects, since it will be unrealistic to set reproducer flags to
generate
> >reproducer archives for every lld invocation. This design also causes
> >troubles when the crash happens on bots only, as in most cases,
developers
> >do not have access to the file system of these bots. It would be great
to
> >improve the lld reproducer generation for easier debugging in these
> >scenarios.
> >
> >*Proposal*
> >
> >Given the use cases and status of clang and lld. I think there are 2
> >possible solutions.
> >
> >*Extend Clang driver*
> >In most cases, lld is invoked by the clang driver instead of being
invoked
> >by the build system directly. Therefore, the clang driver can be
changed
> to
> >re-invoke lld with '--reproduce' flags when it detects the lld
subprocess
> >is crashed.
> >
> >Advantages:
> >    * It probably does not require any changes to the lld and might be
> >easier than handling the crash directly in lld.
> >
> >Disadvantages:
> >    * In case when there is a racing condition in the build system, the
> >input files might have changed between 1st lld crash and 2nd lld rerun
> with
> >'--reproduce' flag. In this case, the generated lld reproducer
archive
> >might not be able to trigger a crash, makes it less useful.
> >
> >*Improve lld reproducer*
> >Another way would be to make lld generate a reproducer archive when it
> >crashes, just like what clang is doing.
> >
> >Advantages:
> >    * It will work no matter if lld is invoked from Clang or from the
> build
> >system.
> >    * It will catch the input file in case the crash is caused by build
> >races.
> >
> >Disadvantages:
> >    * It might need a lot of work if lld does not already have a
> >sophisticated crash handler. It might still need some plumbing changes
in
> >clang driver so lld can honor the '-fcrash-diagnostic-dir'
flag.
> >
> >*Comments?*
> >Which approach do you prefer? Feel free to share your opinions.
>
> There is a resource difference between clang -gen-reproducer /
> environment variable "FORCE_CLANG_DIAGNOSTICS_CRASH" and ld.lld
> --reproduce.
>
> clang -gen-reproducer produces a source file and a .sh file for one
> single translation unit, the space consumption is low.
> ld.lld --reproduce can potentially pack a large list of files, which may
> take hundreds of megabytes or several gigabytes.
>
> I am skeptical that users will want to have this behavior by default.
> If this behavior is guarded by an option, it might be fine.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210414/d4511cd4/attachment.html>

On Wed, Apr 14, 2021 at 3:27 PM Haowei Wu <haowei at google.com>
wrote:
>
> > I am skeptical that users will want to have this behavior by default.
> > If this behavior is guarded by an option, it might be fine.
>
> That's a good point. If the reproducer will be more than a few hundreds
MiBs, it is definitely not suitable to be enabled by default. I agree it's
better to be guarded by an option flag such as `--gen-lld-crash-reproducer`.
>
> On Wed, Apr 14, 2021 at 2:40 PM Fangrui Song <maskray at google.com>
wrote:
>>
>>
>> On 2021-04-14, Haowei Wu via llvm-dev wrote:
>> >*Background / Motivation*
>> >
>> >Both clang and lld have the ability to generate a reproducer (an
archive
>> >with input files and invoker script to reproduce the clang/lld
build).
>> >While clang will generate a reproducer archive when a crash
happens, lld
>> >only generates a reproducer when '--reproduce' flag is
explicitly set (this
>> >is equivalent to Clang's -gen-reproducer flag). This is not
very helpful
>> >for debugging lld bugs, particularly when the crash happens in
building big
>> >projects, since it will be unrealistic to set reproducer flags to
generate
>> >reproducer archives for every lld invocation. This design also
causes
>> >troubles when the crash happens on bots only, as in most cases,
developers
>> >do not have access to the file system of these bots. It would be
great to
>> >improve the lld reproducer generation for easier debugging in these
>> >scenarios.
>> >
>> >*Proposal*
>> >
>> >Given the use cases and status of clang and lld. I think there are
2
>> >possible solutions.
>> >
>> >*Extend Clang driver*
>> >In most cases, lld is invoked by the clang driver instead of being
invoked
>> >by the build system directly. Therefore, the clang driver can be
changed to
>> >re-invoke lld with '--reproduce' flags when it detects the
lld subprocess
>> >is crashed.
>> >
>> >Advantages:
>> >    * It probably does not require any changes to the lld and might
be
>> >easier than handling the crash directly in lld.
>> >
>> >Disadvantages:
>> >    * In case when there is a racing condition in the build system,
the
>> >input files might have changed between 1st lld crash and 2nd lld
rerun with
>> >'--reproduce' flag. In this case, the generated lld
reproducer archive
>> >might not be able to trigger a crash, makes it less useful.
>> >
>> >*Improve lld reproducer*
>> >Another way would be to make lld generate a reproducer archive when
it
>> >crashes, just like what clang is doing.
>> >
>> >Advantages:
>> >    * It will work no matter if lld is invoked from Clang or from
the build
>> >system.
>> >    * It will catch the input file in case the crash is caused by
build
>> >races.
>> >
>> >Disadvantages:
>> >    * It might need a lot of work if lld does not already have a
>> >sophisticated crash handler. It might still need some plumbing
changes in
>> >clang driver so lld can honor the '-fcrash-diagnostic-dir'
flag.
>> >
>> >*Comments?*
>> >Which approach do you prefer? Feel free to share your opinions.
>>
>> There is a resource difference between clang -gen-reproducer /
>> environment variable "FORCE_CLANG_DIAGNOSTICS_CRASH" and
ld.lld --reproduce.
>>
>> clang -gen-reproducer produces a source file and a .sh file for one
>> single translation unit, the space consumption is low.
>> ld.lld --reproduce can potentially pack a large list of files, which
may
>> take hundreds of megabytes or several gigabytes.
>>
>> I am skeptical that users will want to have this behavior by default.
>> If this behavior is guarded by an option, it might be fine.
I'll retract my words about an option. This behavior looks like it
needs a fair bit of customization and is build system dependent.
You can replace the proposed option with a shell script wrapper, which
is more convenient than implementing the restartable action in the
clang driver.
When dealing with linker problems, (I doubt there are many nowadays;
when there are problems, mostly are LTO problems), I will usually
change compiler/linker options a bit.
If you do this, you may only specify the proposed option when all the
stuff has been done, but then it is only a very small extra step to
invoke the link again with -Wl,--reproduce.