llvm dev - Jan 2016 - lld: ELF/COFF main() interface

> Also, one of the other possible motivations of using LLD directly from
Clang would be to avoid process overhead on operating systems where that is
a much more significant part of the compile time cost. We could today
actually take the fork out of the Clang driver because the Clang frontend
*is* designed in this way. But we would also need LLD to work in this way.

Then go change clang and send a patch for lld once you are done. It will be
interested to see if you can measure a single fork in an entire build.

Even better, please finish the new pass manager before working on clang
forking cc1.

In any case, I have simply wasted too much time on a thread with someone
with no patches on the new elf linker. It is really annoying that you don't
put effort into it and seem entitled to dictate its direction.

If you want to kick us out of the llvm project, please start a thread on
llvm-dev.

If you want lld to be a library, figure out how to do it without
sacrificing lld's productivity,  error reporting and performance (no
error_code spaghetti) and write a patch. Just don't expect it to be
reviewed while we have actual missing features.

I will go back to implementing the linker.

Rafael
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160121/a25c4d4c/attachment.html>

> In any case, I have simply wasted too much time on a thread with someone
with no patches on the new elf linker. It is really annoying that you don't
put effort into it and seem entitled to dictate its direction.

Sorry about that. I was initially planning to work on a patch to enhance
the interface for new lld - hence my questions in the original post. Since
I learned that people writing the code for lld are hostile to the idea of
linker-as-a-library, error_code is treated as spaghetti (which would be
fine if LLVM used exceptions which it does not) and patches, even if
submitted, will not actually be reviewed in a timely manner, I'll try to
adapt my code to either not use lld or use lld-as-a-binary.

I'm disappointed by all of this but obviously it's not my project so I
should not have a say in this.

Thank you for your time.

Arseny

On Thu, Jan 21, 2016 at 8:44 PM, Rafael Espíndola <
rafael.espindola at gmail.com> wrote:
> > Also, one of the other possible motivations of using LLD directly from
> Clang would be to avoid process overhead on operating systems where that is
> a much more significant part of the compile time cost. We could today
> actually take the fork out of the Clang driver because the Clang frontend
> *is* designed in this way. But we would also need LLD to work in this way.
>
> Then go change clang and send a patch for lld once you are done. It will
> be interested to see if you can measure a single fork in an entire build.
>
> Even better, please finish the new pass manager before working on clang
> forking cc1.
>
> In any case, I have simply wasted too much time on a thread with someone
> with no patches on the new elf linker. It is really annoying that you
don't
> put effort into it and seem entitled to dictate its direction.
>
> If you want to kick us out of the llvm project, please start a thread on
> llvm-dev.
>
> If you want lld to be a library, figure out how to do it without
> sacrificing lld's productivity,  error reporting and performance (no
> error_code spaghetti) and write a patch. Just don't expect it to be
> reviewed while we have actual missing features.
>
> I will go back to implementing the linker.
>
> Rafael
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160121/6414fc61/attachment.html>

On Thu, Jan 21, 2016 at 8:44 PM Rafael Espíndola <rafael.espindola at
gmail.com>
wrote:
> > Also, one of the other possible motivations of using LLD directly from
> Clang would be to avoid process overhead on operating systems where that is
> a much more significant part of the compile time cost. We could today
> actually take the fork out of the Clang driver because the Clang frontend
> *is* designed in this way. But we would also need LLD to work in this way.
>
> Then go change clang and send a patch for lld once you are done. It will
> be interested to see if you can measure a single fork in an entire build.
>
> Even better, please finish the new pass manager before working on clang
> forking cc1.
>
Trying to do that.

> In any case, I have simply wasted too much time on a thread with someone
> with no patches on the new elf linker. It is really annoying that you
don't
> put effort into it and seem entitled to dictate its direction.
>
I'm working really hard to dictate as little as possible here. However, I
think it is a mistake for the LLD developers (and I agree that I am *not*
one of them) to ignore specific broad design advice from the rest of the
LLVM community. We have historically been a cohesive community and project,
and I think we gain tremendous strength from that. However, to retain that
cohesion as a community, I think it is actually important to seriously
consider the high level design and architecture feedback from the larger
community.

I'm sorry that I haven't been able to personally contribute to LLD. I
actually would like to do that because I do care about it. But until then,
I clearly can't dictate how you write the code and complain that you should
format things differently, etc. I have actually contributed a few patches
to COFF port of LLD, but that hardly counts, I know.

But I had hoped you would be more interested in my design feedback. I have
to admit I'm pretty disappointed in your response. I think that a deep
design discussion like this is never a waste of time.

As an example, even though Philip and Sanjoy haven't really been sending
lots of patches to the pass manager and some of the crazier call graph
analyses in LLVM, I know that this is because they are working on other
important parts of LLVM. And when they started looking and these parts and
raised very serious concerns about the design of the LazyCallGraph to me
(sorry, it was at a social, so I can't send a link), I think that was
entirely appropriate. I didn't agree with their concerns, even after a very
long debate. But I still don't think it was a waste of time. As it happens,
I now *know* it was not a waste of time, because their points kept coming
back to me over the past couple of months, and surprise surprise, as I dug
deeper into the call graph passes *they were right*. Completely. Having had
that discussion was extremely valuable because I probably would have caught
on a lot less quickly without it. In other cases Duncan provided excellent
feedback that I immediately incorporated. And Duncan Sands was instrumental
in giving me design feedback of SROA even though I was the only one coding
on it.

Anyways, I hope you reconsider whether this is a waste. I don't think it is.

Also, I don't think being dismissive of potential new users of LLVM (and
LLD is still part of LLVM) is the right thing for the project, and so I
don't think you should continue to do that. My two cents as a member of the
larger LLVM community and project. By all means, solicit opinions from
others in the community about the most appropriate approach here.

If you want to kick us out of the llvm project, please start a thread
on
> llvm-dev.
>
Note, this *is* a thread on llvm-dev, because we have worked to keep the
LLD development *part* of the LLVM project and community.

In case there was any confusion, I don't want to kick anyone out of the
community. I'm trying to make LLD more appealing to more people *within*
the community.

> I will go back to implementing the linker.
>
This comes off as really passive aggressive to me, and that makes this
discussion much less productive IMO. It would help me (and I suspect
others) if you could try to change the tone you are using here.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160122/b037f505/attachment.html>

.
>
> In case there was any confusion, I don't want to kick anyone out of the
community. I'm trying to make LLD more appealing to more people *within*
the community.
>
Then code it and show it is not a regression of the current design.
>>
>> I will go back to implementing the linker.
>
> This comes off as really passive aggressive to me, and that makes this
discussion much less productive IMO. It would help me (and I suspect
others) if you could try to change the tone you are using here.

It is not. Having to fight back your push and mischaracterization of the
situation has cost a lot of time, sleep and productivity. In particular, it
is certainly not true when you say:

------------------
During the discussion, there was a *specific* discussion of both the new
COFF port and ELF port continuing to be libraries with a common command
line driver
--------------

Rui's design was explicit enough that this was going to be just a linker to
get me excited about it. If you didn't pick it up you were simply not
paying attention.

I have been tracking llvm since 2007 and I can't remember another situation
as bad as this. Had the project been like this      I would probably have
found something else to do.

In the future if you have something you want but existing developers don't
care about (lld should make coffee), remember the burden is on you to show
that it will not interfere with the rest. It is not OK to push people
saying their work should not be part of llvm if they don't do what you want.

Cheers,
Rafael
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160122/bcd5afcf/attachment.html>

I think I have an idea to cover your need and possibly other people's on
this thread. It provides the "main() as a library function" feature,
input/output files wouldn't go through disks nor file systems, and it
doesn't require any major design changes. Sounds too good?

That is, we can provide a function that takes command line parameters, do
fork, and call the linker's main function.

This may sound too simple, but I think that is fairly powerful. Because the
child process has a copy(-on-write) of the parent's memory, it can read
parent's in-memory object files directly with no overhead. The child can
pass the resulting file back to the parent through a shared memory object
(which we can obtain using shm_open or something like that). In addition to
that, your main process gets a protection from linker's bugs thanks to the
operating system's memory protection. But from the user's point of view,
that is just a linker's main function that you can call and that works as
expected.

Even if we want to call exec for whatever reason, we can copy in-memory
objects to shared memory objects and exec the linker, so the basic design
should work in such case too.

The function signature would be something like:

  bool link(ArrayRef<LinkerArg> CommandLineArgs, MemoryBuffer
&OutputFile,
std::string &ErrorMsg);

where the return value indicates success/failure. LinkerArg is a union type
of StringRef and MemoryBufferRef. The result is returned as OutputFile
memory buffer. If it prints out any message, ErrorMsg will hold it.

(I want to point out that the function "bool link(ArrayRef<const
char*>
args, raw_ostream& diagnostics,
ArrayRef<unique_ptr<MemoryBuffer>> inputs)"
doesn't work because the order of command line parameters matters. Some
command line parameters, such as --whole-archive/--no-whole-archive,
affects how files in between will be interpreted, so you can't separate
command line parameters from a list of files.)

I think this is a practical solution that we can do now. I can implement
this for you.

On Thu, Jan 21, 2016 at 9:28 PM, Arseny Kapoulkine <
arseny.kapoulkine at gmail.com> wrote:
> > In any case, I have simply wasted too much time on a thread with
> someone with no patches on the new elf linker. It is really annoying that
> you don't put effort into it and seem entitled to dictate its
direction.
>
> Sorry about that. I was initially planning to work on a patch to enhance
> the interface for new lld - hence my questions in the original post. Since
> I learned that people writing the code for lld are hostile to the idea of
> linker-as-a-library, error_code is treated as spaghetti (which would be
> fine if LLVM used exceptions which it does not) and patches, even if
> submitted, will not actually be reviewed in a timely manner, I'll try
to
> adapt my code to either not use lld or use lld-as-a-binary.
>
> I'm disappointed by all of this but obviously it's not my project
so I
> should not have a say in this.
>
> Thank you for your time.
>
> Arseny
>
> On Thu, Jan 21, 2016 at 8:44 PM, Rafael Espíndola <
> rafael.espindola at gmail.com> wrote:
>
>> > Also, one of the other possible motivations of using LLD directly
from
>> Clang would be to avoid process overhead on operating systems where
that is
>> a much more significant part of the compile time cost. We could today
>> actually take the fork out of the Clang driver because the Clang
frontend
>> *is* designed in this way. But we would also need LLD to work in this
way.
>>
>> Then go change clang and send a patch for lld once you are done. It
will
>> be interested to see if you can measure a single fork in an entire
build.
>>
>> Even better, please finish the new pass manager before working on clang
>> forking cc1.
>>
>> In any case, I have simply wasted too much time on a thread with
someone
>> with no patches on the new elf linker. It is really annoying that you
don't
>> put effort into it and seem entitled to dictate its direction.
>>
>> If you want to kick us out of the llvm project, please start a thread
on
>> llvm-dev.
>>
>> If you want lld to be a library, figure out how to do it without
>> sacrificing lld's productivity,  error reporting and performance
(no
>> error_code spaghetti) and write a patch. Just don't expect it to be
>> reviewed while we have actual missing features.
>>
>> I will go back to implementing the linker.
>>
>> Rafael
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160122/1b61c8cd/attachment.html>

Rafael,

This is an unreasonable and unacceptable escalation of this thread. You
have been dismissive, derailing, and taking out of context everything
Chandler, and other long term active contributors have been asking and
talking about. Your tone has been inflammatory and unhelpful in what is
both a technical discussion and project discussion.

Irrespective of anything else I believe you owe Chandler an apology.

To further Chandler's arguments as far as the project is concerned, please
do keep in mind that: a) llvm is designed around being a reusable set of
libraries, b) Chandler is articulating in a very effective way what many of
us on the project see as the direction for lld. I, and others, have not
seen it as helpful to merely reply to all of his posts with a "+1" as
that's just adding additional noise - and we all get enough email.

-eric

On Thu, Jan 21, 2016 at 8:44 PM Rafael Espíndola <llvm-dev at
lists.llvm.org>
wrote:
> > Also, one of the other possible motivations of using LLD directly from
> Clang would be to avoid process overhead on operating systems where that is
> a much more significant part of the compile time cost. We could today
> actually take the fork out of the Clang driver because the Clang frontend
> *is* designed in this way. But we would also need LLD to work in this way.
>
> Then go change clang and send a patch for lld once you are done. It will
> be interested to see if you can measure a single fork in an entire build.
>
> Even better, please finish the new pass manager before working on clang
> forking cc1.
>
> In any case, I have simply wasted too much time on a thread with someone
> with no patches on the new elf linker. It is really annoying that you
don't
> put effort into it and seem entitled to dictate its direction.
>
> If you want to kick us out of the llvm project, please start a thread on
> llvm-dev.
>
> If you want lld to be a library, figure out how to do it without
> sacrificing lld's productivity,  error reporting and performance (no
> error_code spaghetti) and write a patch. Just don't expect it to be
> reviewed while we have actual missing features.
>
> I will go back to implementing the linker.
>
> Rafael
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160122/c1cbba20/attachment.html>

----- Original Message ----- 
> From: "Eric Christopher via llvm-dev" <llvm-dev at
lists.llvm.org>
> To: "Rafael Espíndola" <rafael.espindola at gmail.com>,
"Chandler
> Carruth" <chandlerc at gmail.com>
> Cc: "llvm-dev" <llvm-dev at lists.llvm.org>, "Arseny
Kapoulkine"
> <arseny.kapoulkine at gmail.com>
> Sent: Friday, January 22, 2016 5:42:14 PM
> Subject: Re: [llvm-dev] lld: ELF/COFF main() interface
> Rafael,
> This is an unreasonable and unacceptable escalation of this thread.
> You have been dismissive, derailing, and taking out of context
> everything Chandler, and other long term active contributors have
> been asking and talking about. Your tone has been inflammatory and
> unhelpful in what is both a technical discussion and project
> discussion.
> Irrespective of anything else I believe you owe Chandler an apology.
> To further Chandler's arguments as far as the project is concerned,
> please do keep in mind that: a) llvm is designed around being a
> reusable set of libraries, b) Chandler is articulating in a very
> effective way what many of us on the project see as the direction
> for lld. I, and others, have not seen it as helpful to merely reply
> to all of his posts with a "+1" as that's just adding
additional
> noise - and we all get enough email.
Eric, thanks so much for writing this. I do believe this captures how many of us
feel.

 -Hal

> -eric
> On Thu, Jan 21, 2016 at 8:44 PM Rafael Espíndola <
> llvm-dev at lists.llvm.org > wrote:
> > Also, one of the other possible motivations of using LLD directly
> > from Clang would be to avoid process overhead on operating systems
> > where that is a much more significant part of the compile time
> > cost. We could today actually take the fork out of the Clang
> > driver because the Clang frontend *is* designed in this way. But
> > we would also need LLD to work in this way.
> Then go change clang and send a patch for lld once you are done. It
> will be interested to see if you can measure a single fork in an
> entire build.
> Even better, please finish the new pass manager before working on
> clang forking cc1.
> In any case, I have simply wasted too much time on a thread with
> someone with no patches on the new elf linker. It is really annoying
> that you don't put effort into it and seem entitled to dictate its
> direction.
> If you want to kick us out of the llvm project, please start a thread
> on llvm-dev.
> If you want lld to be a library, figure out how to do it without
> sacrificing lld's productivity, error reporting and performance (no
> error_code spaghetti) and write a patch. Just don't expect it to be
> reviewed while we have actual missing features.
> I will go back to implementing the linker.
> Rafael _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> --
Hal Finkel 
Assistant Computational Scientist 
Leadership Computing Facility 
Argonne National Laboratory 
-- 
Hal Finkel Assistant Computational Scientist Leadership Computing Facility
Argonne National Laboratory

-- 
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory

On Sat, Jan 23, 2016 at 6:42 AM, Eric Christopher via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
> Rafael,
>
> This is an unreasonable and unacceptable escalation of this thread. You
have
> been dismissive, derailing, and taking out of context everything Chandler,
> and other long term active contributors have been asking and talking about.
> Your tone has been inflammatory and unhelpful in what is both a technical
> discussion and project discussion.
>
> Irrespective of anything else I believe you owe Chandler an apology.
/*
>From my peanut gallery perspective (I don't want to get involved here)
I'd disagree with your outlook.
*/
----------
I tend to be rude, not by design, but because I don't apply the same
filters which other (normal) people expect in a conversation. Over the
past __ years I've found the best (relatively speaking) projects are
the ones where people aren't hyper sensitive. I love Linus' commentary
from time to time and it just adds a bit of flavor. (I'm not saying
that' is or should be happening here)

If any apologies are due - can we keep this list technical and do it
offlist. Any adults who feel the necessity to "help out" in situations
like this, please do so more tactfully, by this I mean
privately/quietly. There's really nothing worse than getting
flogged/flawed/____ for something publicly when all you were trying to
do is achieve some technical goal.