llvm dev - Jan 2017 - Your help needed: List of LLVM Open Projects 2017

On Mon, Jan 16, 2017 at 12:31 PM, Sean Silva <chisophugis at gmail.com>
wrote:
> Do we have any open projects on LLD?
>
> I know we usually try to avoid any big "projects" and mainly
add/fix
> things in response to user needs, but just wondering if somebody has any
> ideas.
>
> Some really generic/simple stuff I can think of:
> 1. trying out LLD on a large program corpus and reporting/reducing/fixing
> bugs (e.g. contributing to the FreeBSD effort or trying to build a bunch of
> packages from a linux distro like Debian or Gentoo)
>
I like that idea. FreeBSD is using a very old version of ld.bfd, so on
Linux systems we may be able to find programs that use recent linker
features which LLD does not support. This may sound like an ambitious goal,
but I want make Linux distributions to use LLD as default linker, so it
needs to be able to link entire Linux distributions.

2. performance analysis and optimization of LLD
> 3. getting LLD to link a bootable Linux kernel and/or GRUB
>
I don't know how hard/easy it is to link Linux kernel with LLD, but that
should be doable if we spend enough effort on that. That should be a good
project.

As to porting LLD to other architectures, you can use emulators like bochs,
can't you? You don't even need any emulator for x86 32-bit.

4. write an input verifier such that LLD can survive intensive fuzzing
with
> no crashes / fatal errors [1] when the verifier says the input is okay.
> This will allow us to measure what the overhead of doing this actually is.
>
>
> [1] As of the latest LLD discussion (in the thread "[llvm-dev] LLD
status
> update and performance chart") it sounds like people are okay with LLD
> treating fatal errors the same way that LLVM uses assertions; for inputs
> from the C++ API, we can document to not pass corrupted object files. For
> inputs read from files, there is still community interest in at least
> having the option to run a "verifier" to validate the inputs. I
think the
> best way to approach the verifier is to essentially follow the approach
> suggested by Peter (in the context of "hardening") in
> https://llvm.org/bugs/show_bug.cgi?id=30540#c5 i.e. getting to the point
> where LLD can survive intensive fuzzing.
>
> -- Sean Silva
>
> On Mon, Jan 16, 2017 at 5:18 AM, Vassil Vassilev via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> Hi folks,
>>
>>   Happy new year!
>>
>>   Last LLVM Developers' Meeting I had a BoF: 'Raising Next
Generation
>> LLVM Developers'. It was suggested that we should update our open
projects
>> page and possibly restructure it a little bit.
>>
>>   I volunteered to do this work and I need your help.
>>
>>
>>   Chandler and I started working on a google doc [1]. We pinged few
code
>> owners asking them to list of work items we should get done in 2017 but
we
>> do not have the manpower. Now we would like to ask for your input, too.
>>
>>   I believe an up to date list can serve as a good entry point for
>> students, interns and new contributors.
>>
>>   Feel free to propose a new item or comment under an existing one. I
>> expect to start gradually updating the page beginning of Feb.
>>
>> -- Vassil
>>
>> [1] https://docs.google.com/document/d/1YLK_xINSg1Ei0w8w39uAMR1n
>> 0dlf6wrzfypiX0YDQBc/edit?usp=sharing
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170116/1c8cc08c/attachment.html>

On 17 January 2017 at 02:02, Rui Ueyama <ruiu at google.com>
wrote:
>
> I like that idea. FreeBSD is using a very old version of ld.bfd, so on
Linux
> systems we may be able to find programs that use recent linker features
> which LLD does not support. This may sound like an ambitious goal, but I
> want make Linux distributions to use LLD as default linker, so it needs to
> be able to link entire Linux distributions.
On FreeBSD the very old ld.bfd is used for linking the base system and
is the default for the ports collection, but ports that require
features from newer versions depend on the binutils package. Currently
they'll link with ld.bfd 2.27. I believe there are on the order of 50
packages with a build dependency on binutils. We should be able to
hack Poudriere to try linking these with lld instead of a contemporary
ld.bfd.

That said, I agree it's incredibly valuable to have Linux distros and
the Linux kernel test linking with LLD.
> As to porting LLD to other architectures, you can use emulators like bochs,
> can't you? You don't even need any emulator for x86 32-bit.
QEMU recipes exist for most architectures of interest to FreeBSD, so
there's a straightforward testing approach. We also have fast
development hardware for some non-x86 architectures, like AArch64.

On 17 January 2017 at 11:05, Ed Maste <emaste at freebsd.org>
wrote:
> I believe there are on the order of 50
> packages with a build dependency on binutils. We should be able to
> hack Poudriere to try linking these with lld instead of a contemporary
> ld.bfd.
I missed the ports that have a GCC build dependency in the above
count; the GCC port is configured to use the binutils port's ld so all
the ports that build with GCC explicitly link with ld.bfd. I count
about 300 such ports now. I'll look at adding an option to have GCC
use the system linker to test these with LLD.

On Mon, Jan 16, 2017 at 11:02 PM, Rui Ueyama <ruiu at google.com> wrote:
> On Mon, Jan 16, 2017 at 12:31 PM, Sean Silva <chisophugis at
gmail.com>
> wrote:
>
>> Do we have any open projects on LLD?
>>
>> I know we usually try to avoid any big "projects" and mainly
add/fix
>> things in response to user needs, but just wondering if somebody has
any
>> ideas.
>>
>> Some really generic/simple stuff I can think of:
>> 1. trying out LLD on a large program corpus and
reporting/reducing/fixing
>> bugs (e.g. contributing to the FreeBSD effort or trying to build a
bunch of
>> packages from a linux distro like Debian or Gentoo)
>>
>
> I like that idea. FreeBSD is using a very old version of ld.bfd, so on
> Linux systems we may be able to find programs that use recent linker
> features which LLD does not support. This may sound like an ambitious goal,
> but I want make Linux distributions to use LLD as default linker, so it
> needs to be able to link entire Linux distributions.
>
That's indeed ambitious, as even Gold has not managed to become the default
on any prominent Linux distro I'm aware of. IIRC gold doesn't link all
of
Gentoo (there was a blog post / article but I can't find it). One
intermediate goal we can strive for is to reach parity with Clang, which
currently builds about 95% of the packages: http://clang.debian.net/


>
> 2. performance analysis and optimization of LLD
>> 3. getting LLD to link a bootable Linux kernel and/or GRUB
>>
>
> I don't know how hard/easy it is to link Linux kernel with LLD, but
that
> should be doable if we spend enough effort on that. That should be a good
> project.
>
> As to porting LLD to other architectures, you can use emulators like
> bochs, can't you? You don't even need any emulator for x86 32-bit.
>
These open projects are largely targeted at new contributors, so ideally we
would provide some instructions on how to set up a test system, what to do
on it, etc. The less effort they have to spend yak-shaving setting things
up, the sooner they will be doing useful work on LLD (or even, contributing
to LLD at all, if they can't figure out how to set things up). We already
have community members with workflows for doing this kind of testing, it's
just a matter of writing it down so it is easy to follow. Ideally it's just
"download this emulator and grab this image for <architecture>, then
boot
it and try building/linking/running some programs"; also, any relevant
cross-compile setup would make things easier too.

Similarly for the Linux kernel; all the setup for getting qemu and stuff
working is annoying and I remember it took Davide, Michael, and quite a bit
of time yak-shaving to find a nice workflow that we could iterate on for
FreeBSD. Honestly the Linux/GRUB project is a bit of a stretch: I think
this "open projects" page is mainly aimed at a student/intern-level
audience, and to attack that project a student/intern-level contributor
would have to be truly extraordinary (though I guess we may want to attract
those types too ;-) ). Filling in architecture support in userland is a
much more tractable for a larger audience.

-- Sean Silva

>
> 4. write an input verifier such that LLD can survive intensive fuzzing
>> with no crashes / fatal errors [1] when the verifier says the input is
>> okay. This will allow us to measure what the overhead of doing this
>> actually is.
>>
>>
>> [1] As of the latest LLD discussion (in the thread "[llvm-dev] LLD
status
>> update and performance chart") it sounds like people are okay with
LLD
>> treating fatal errors the same way that LLVM uses assertions; for
inputs
>> from the C++ API, we can document to not pass corrupted object files.
For
>> inputs read from files, there is still community interest in at least
>> having the option to run a "verifier" to validate the inputs.
I think the
>> best way to approach the verifier is to essentially follow the approach
>> suggested by Peter (in the context of "hardening") in
>> https://llvm.org/bugs/show_bug.cgi?id=30540#c5 i.e. getting to the
point
>> where LLD can survive intensive fuzzing.
>>
>> -- Sean Silva
>>
>> On Mon, Jan 16, 2017 at 5:18 AM, Vassil Vassilev via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>>> Hi folks,
>>>
>>>   Happy new year!
>>>
>>>   Last LLVM Developers' Meeting I had a BoF: 'Raising Next
Generation
>>> LLVM Developers'. It was suggested that we should update our
open projects
>>> page and possibly restructure it a little bit.
>>>
>>>   I volunteered to do this work and I need your help.
>>>
>>>
>>>   Chandler and I started working on a google doc [1]. We pinged few
code
>>> owners asking them to list of work items we should get done in 2017
but we
>>> do not have the manpower. Now we would like to ask for your input,
too.
>>>
>>>   I believe an up to date list can serve as a good entry point for
>>> students, interns and new contributors.
>>>
>>>   Feel free to propose a new item or comment under an existing one.
I
>>> expect to start gradually updating the page beginning of Feb.
>>>
>>> -- Vassil
>>>
>>> [1] https://docs.google.com/document/d/1YLK_xINSg1Ei0w8w39uAMR1n
>>> 0dlf6wrzfypiX0YDQBc/edit?usp=sharing
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170118/3afd2fca/attachment.html>

On Tue, Jan 17, 2017 at 8:05 AM, Ed Maste <emaste at freebsd.org> wrote:
> On 17 January 2017 at 02:02, Rui Ueyama <ruiu at google.com> wrote:
> >
> > I like that idea. FreeBSD is using a very old version of ld.bfd, so on
> Linux
> > systems we may be able to find programs that use recent linker
features
> > which LLD does not support. This may sound like an ambitious goal, but
I
> > want make Linux distributions to use LLD as default linker, so it
needs
> to
> > be able to link entire Linux distributions.
>
> On FreeBSD the very old ld.bfd is used for linking the base system and
> is the default for the ports collection, but ports that require
> features from newer versions depend on the binutils package. Currently
> they'll link with ld.bfd 2.27. I believe there are on the order of 50
> packages with a build dependency on binutils. We should be able to
> hack Poudriere to try linking these with lld instead of a contemporary
> ld.bfd.
>
> That said, I agree it's incredibly valuable to have Linux distros and
> the Linux kernel test linking with LLD.
>
> > As to porting LLD to other architectures, you can use emulators like
> bochs,
> > can't you? You don't even need any emulator for x86 32-bit.
>
> QEMU recipes exist for most architectures of interest to FreeBSD, so
> there's a straightforward testing approach. We also have fast
> development hardware for some non-x86 architectures, like AArch64.
>
If you are setting up a wiki page somewhere for LLD status etc., could you
link to these recipes?

Thanks,

-- Sean Silva
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170118/e50abbe0/attachment.html>

On Wed, Jan 18, 2017 at 9:34 PM, Sean Silva <chisophugis at gmail.com>
wrote:
>
>
> On Mon, Jan 16, 2017 at 11:02 PM, Rui Ueyama <ruiu at google.com>
wrote:
>
>> On Mon, Jan 16, 2017 at 12:31 PM, Sean Silva <chisophugis at
gmail.com>
>> wrote:
>>
>>> Do we have any open projects on LLD?
>>>
>>> I know we usually try to avoid any big "projects" and
mainly add/fix
>>> things in response to user needs, but just wondering if somebody
has any
>>> ideas.
>>>
>>> Some really generic/simple stuff I can think of:
>>> 1. trying out LLD on a large program corpus and
>>> reporting/reducing/fixing bugs (e.g. contributing to the FreeBSD
effort or
>>> trying to build a bunch of packages from a linux distro like Debian
or
>>> Gentoo)
>>>
>>
>> I like that idea. FreeBSD is using a very old version of ld.bfd, so on
>> Linux systems we may be able to find programs that use recent linker
>> features which LLD does not support. This may sound like an ambitious
goal,
>> but I want make Linux distributions to use LLD as default linker, so it
>> needs to be able to link entire Linux distributions.
>>
>
> That's indeed ambitious, as even Gold has not managed to become the
> default on any prominent Linux distro I'm aware of. IIRC gold
doesn't link
> all of Gentoo (there was a blog post / article but I can't find it).
One
> intermediate goal we can strive for is to reach parity with Clang, which
> currently builds about 95% of the packages: http://clang.debian.net/
>
It is ambitious, but I think we have one advantage over gold, which is that
LLD is a part of the LLVM project. When LLD is mature enough, we should
probably start thinking of making Clang prefer LLD over other system
default linkers if LLD is installed to a system (we had this discussion
before.) In that sense we may be able to piggy-back on Clang to make it
default. Of course there is a risk in doing that, so LLD needs to become a
tool that "just work" before doing that, though.

>
>>
>> 2. performance analysis and optimization of LLD
>>> 3. getting LLD to link a bootable Linux kernel and/or GRUB
>>>
>>
>> I don't know how hard/easy it is to link Linux kernel with LLD, but
that
>> should be doable if we spend enough effort on that. That should be a
good
>> project.
>>
>> As to porting LLD to other architectures, you can use emulators like
>> bochs, can't you? You don't even need any emulator for x86
32-bit.
>>
>
> These open projects are largely targeted at new contributors, so ideally
> we would provide some instructions on how to set up a test system, what to
> do on it, etc. The less effort they have to spend yak-shaving setting
> things up, the sooner they will be doing useful work on LLD (or even,
> contributing to LLD at all, if they can't figure out how to set things
up).
> We already have community members with workflows for doing this kind of
> testing, it's just a matter of writing it down so it is easy to follow.
> Ideally it's just "download this emulator and grab this image for
> <architecture>, then boot it and try building/linking/running some
> programs"; also, any relevant cross-compile setup would make things
easier
> too.
>
> Similarly for the Linux kernel; all the setup for getting qemu and stuff
> working is annoying and I remember it took Davide, Michael, and quite a bit
> of time yak-shaving to find a nice workflow that we could iterate on for
> FreeBSD. Honestly the Linux/GRUB project is a bit of a stretch: I think
> this "open projects" page is mainly aimed at a
student/intern-level
> audience, and to attack that project a student/intern-level contributor
> would have to be truly extraordinary (though I guess we may want to attract
> those types too ;-) ). Filling in architecture support in userland is a
> much more tractable for a larger audience.
>
> -- Sean Silva
>
>
>>
>> 4. write an input verifier such that LLD can survive intensive fuzzing
>>> with no crashes / fatal errors [1] when the verifier says the input
is
>>> okay. This will allow us to measure what the overhead of doing this
>>> actually is.
>>>
>>>
>>> [1] As of the latest LLD discussion (in the thread "[llvm-dev]
LLD
>>> status update and performance chart") it sounds like people
are okay with
>>> LLD treating fatal errors the same way that LLVM uses assertions;
for
>>> inputs from the C++ API, we can document to not pass corrupted
object
>>> files. For inputs read from files, there is still community
interest in at
>>> least having the option to run a "verifier" to validate
the inputs. I think
>>> the best way to approach the verifier is to essentially follow the
approach
>>> suggested by Peter (in the context of "hardening") in
>>> https://llvm.org/bugs/show_bug.cgi?id=30540#c5 i.e. getting to the
>>> point where LLD can survive intensive fuzzing.
>>>
>>> -- Sean Silva
>>>
>>> On Mon, Jan 16, 2017 at 5:18 AM, Vassil Vassilev via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>>> Hi folks,
>>>>
>>>>   Happy new year!
>>>>
>>>>   Last LLVM Developers' Meeting I had a BoF: 'Raising
Next Generation
>>>> LLVM Developers'. It was suggested that we should update
our open projects
>>>> page and possibly restructure it a little bit.
>>>>
>>>>   I volunteered to do this work and I need your help.
>>>>
>>>>
>>>>   Chandler and I started working on a google doc [1]. We pinged
few
>>>> code owners asking them to list of work items we should get
done in 2017
>>>> but we do not have the manpower. Now we would like to ask for
your input,
>>>> too.
>>>>
>>>>   I believe an up to date list can serve as a good entry point
for
>>>> students, interns and new contributors.
>>>>
>>>>   Feel free to propose a new item or comment under an existing
one. I
>>>> expect to start gradually updating the page beginning of Feb.
>>>>
>>>> -- Vassil
>>>>
>>>> [1]
https://docs.google.com/document/d/1YLK_xINSg1Ei0w8w39uAMR1n
>>>> 0dlf6wrzfypiX0YDQBc/edit?usp=sharing
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170119/0589883d/attachment.html>