llvm dev - May 2017 - Add more projects into Git monorepo

On 9 May 2017, at 15:58, Mehdi AMINI <joker.eph at gmail.com> wrote:
> I'd expect any CI system to be able to cache this.
> Also if you're issue is archiving a lot of build artifacts, the
constant cost of the checkout isn't gonna matter that much.
> Finally, the read-only individual repo can still be used by CI, which
address this entirely.
If we want to pull in new libunwind fixes from upstream, we’ll also pull in
irrelevant LLVM, clang, lldb, lld, and so on changes.  This translates to extra
bandwidth and storage requirements for *every* copy of the libunwind repo that
we need.

If we follow the monorepo approach downstream and merge these independent repos,
then we add extra merges for everyone downstream because people committing
improvements to our LLVM and clang trees will require rebase pulls for anyone
working on libc++ or libunwind, even though the changes were to a component that
they’re not needing to build, let alone modify.
> There is another philosophical perspective: encouraging communities to get
closer together. You talking about "libunwind developers", and there
are "lldb developers" as well, I rather get closer to: "we're
working on the same project", with shared practices and goals. And
ultimately, to come back to your software engineering practices, encouraging
code motion and code reuse between sub-projects.
I disagree, as someone who wears hats as a libunwind, libc++, clang and LLVM
developer: I am no more engaged between the different groups by having the repos
combined, but I am inconvenienced by having to carry around clones of unrelated
code when I am working on one component and by having to rebase my libunwind
repo because someone committed to clang.

Combining the clang and LLVM repos is a necessary evil.  If we could have clean
layering and well-defined APIs for the LLVM APIs needed for clang, then I would
be opposed to this as well, but unfortunately this has too high an engineering
cost and so we need to be able to perform atomic commits of LLVM and LLVM-using
projects (this, unfortunately, means that we often don’t see the cost that this
imposes on developers of other front ends).  In contrast, if we need to perform
an atomic commit between libc++ and clang or libunwind and clang then this tells
us that we have a bug: a new version of clang may introduce a feature that
relies on a new libc++ or libunwind, but a new libunwind or libc++ should always
work with an old clang (or an old gcc, or any other compiler that targets it).
>> All of this applies to libc++ and libc++abi as well.
> 
> Ultimately I don't know about libunwind, and if it has to live
separately it is not a big deal. The others (libc++ and libc++abi for instance)
are more tied to the rest of the project though.
> We duplicate the demangler from libc++abi in llvm for instance, and this is
quite an important software engineer issue to me.
The requirements for a libc++abi demangler and a generic LLVM one are very
different.  For libc++abi, the requirements are:

 - Must be small (the binary size of libc++abi is very important)

 - Must be tolerant of out-of-memory conditions (it is used for generating error
messages when an out-of-memory exception is thrown)

 - Must use malloc() / realloc() for providing the demangled string (a
requirement of the Itanium ABI public APIs)

In contrast, the demangler for the rest of LLVM:

 - Must be flexible (e.g. lldb wants to be able to get the base name of a
demangled function, so that it can insert breakpoints on all overloads)

 - Must be fast (e.g. lldb wants to demangle every symbol in a library in a
UI-critical path)

 - Must provide structured information about the demangled symbol, not just a
string as output.

 - Must integrate with other memory allocation mechanisms (e.g. support
std::allocator)

Copying the demangler was a quick way of getting something to work portably, but
it wasn’t a good solution given the different requirements (the libc++abi
demangler doesn’t do a good job of meeting either set of requirements), so this
is a very bad justification for merging the repos.

David

2017-05-09 8:17 GMT-07:00 David Chisnall <David.Chisnall at cl.cam.ac.uk>:
> On 9 May 2017, at 15:58, Mehdi AMINI <joker.eph at gmail.com> wrote:
>
> > I'd expect any CI system to be able to cache this.
> > Also if you're issue is archiving a lot of build artifacts, the
constant
> cost of the checkout isn't gonna matter that much.
> > Finally, the read-only individual repo can still be used by CI, which
> address this entirely.
>
> If we want to pull in new libunwind fixes from upstream, we’ll also pull
> in irrelevant LLVM, clang, lldb, lld, and so on changes.  This translates
> to extra bandwidth and storage requirements for *every* copy of the
> libunwind repo that we need.
>
I'm not sure if you really read the last sentence of what I wrote, or if
you followed the previous discussions on the plan here?
At this point I believe that this concern is non-existent per the read-only
individual repo.

>
> If we follow the monorepo approach downstream and merge these independent
> repos, then we add extra merges for everyone downstream because people
> committing improvements to our LLVM and clang trees will require rebase
> pulls for anyone working on libc++ or libunwind, even though the changes
> were to a component that they’re not needing to build, let alone modify.
>
Every downstream has its own choice, I'm not sure what's the point here?
If
it is not relevant to you, then don't do it...

>
> > There is another philosophical perspective: encouraging communities to
> get closer together. You talking about "libunwind developers",
and there
> are "lldb developers" as well, I rather get closer to:
"we're working on
> the same project", with shared practices and goals. And ultimately, to
come
> back to your software engineering practices, encouraging code motion and
> code reuse between sub-projects.
>
> I disagree,  [...].
>
We can leave it there :)
There have been extensive discussions, a BoF, and documentations, please
refer you to these first (granted we haven't really talked about libunwind,
but I'm not sure many people will be strongly opposed to libunwind having
its separate life).

>
> >> All of this applies to libc++ and libc++abi as well.
> >
> > Ultimately I don't know about libunwind, and if it has to live
> separately it is not a big deal. The others (libc++ and libc++abi for
> instance) are more tied to the rest of the project though.
> > We duplicate the demangler from libc++abi in llvm for instance, and
this
> is quite an important software engineer issue to me.
>
> The requirements for a libc++abi demangler and a generic LLVM one are very
> different.
>

They are so different that we ask anyone who want to change something to
the LLVM demangler to make the change in compiler-rt first and then pull
the patch...
Maintaining two demanglers (or more) is silly IMO (no-one is even
maintaining the existing one...).

-- 
Mehdi
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170509/0f6fb7f9/attachment.html>

On 9 May 2017, at 16:59, Mehdi AMINI <joker.eph at gmail.com>
wrote:
> 
> I'm not sure if you really read the last sentence of what I wrote, or
if you followed the previous discussions on the plan here?
> At this point I believe that this concern is non-existent per the read-only
individual repo.
The read-only repo is only useful if you don’t intend to contribute stuff back
upstream.  There is no convenient workflow for cloning libunwind / libc++ /
libwhatever, hacking on it, and sending pull requests.
> We can leave it there :)
> There have been extensive discussions, a BoF, and documentations, please
refer you to these first (granted we haven't really talked about libunwind,
but I'm not sure many people will be strongly opposed to libunwind having
its separate life).
There have been multiple discussions, and the conclusion from all that I have
participated in was that projects that are tightly version locked to LLVM should
be in the monorepo, everything else should be separate.  Apparently there is now
a plan underway to not do this and to make life harder for people who work on
the projects that are not version locked to LLVM.

I am simply repeating the objections that numerous people have made in each of
these discussions.

David