llvm dev - Sep 2016 - [RFC] One or many git repositories?

> On Sep 3, 2016, at 6:05 PM, Patrice Kouame <pkouame at mac.com>
wrote:
> 
> Someone mentioned llvm in a mono repository below…
Right, we actually have a proposal to take what is in the current SVN repo here:
http://llvm.org/svn/llvm-project/ and migrate this to a single repository.
I was not sure if you were referring to this proposal (monorepo) or to the
recent emails about “external libraries” that GCC uses like  gmp and mpfr.

You can find more details here: https://reviews.llvm.org/D24167

If you have some good reasons why you would think a proposal would be
problematic to you, or one would better fit your workflow, feel free to expose
them now.

Best,

— 
Mehdi
> 
> On Sep 3, 2016, at 7:37 PM, Mehdi Amini <mehdi.amini at apple.com
<mailto:mehdi.amini at apple.com>> wrote:
> 
>> 
>> 
>> On Sep 3, 2016, at 3:04 PM, Patrice Kouame <pkouame at mac.com
<mailto:pkouame at mac.com>> wrote:
>> 
>>> 
>>> +1 for keeping it separate. 
>> 
>> Can you clarify what you referring to specifically?
>> This sub thread (the last 4 messages) started with a mention of GCC
dependencies. It is not clear to me how to relate to llvm now.
>> 
>> Mehdi
>> 
>>> 
>>> One can easily set up a git subproject structure if the need is
pressing...
>>> 
>>> Patrice
>>> 
>>>> On Sep 3, 2016, at 5:39 PM, Mehdi Amini via llvm-dev
<llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>
wrote:
>>>> 
>>>> 
>>>>> On Sep 2, 2016, at 12:33 PM, Dimitry Andric via llvm-dev
<llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>
wrote:
>>>>> 
>>>>> On 16 Aug 2016, at 00:12, Lawrence, Peter via llvm-dev
<llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>
wrote:
>>>>>> 
>>>>>> As someone that has worked with both gcc and llvm,
>>>>>> 
>>>>>> One thing about gcc that drives me bat-guano-crazy is
that
>>>>>> First you check out gcc, try to build it, and find that
you also
>>>>>> Need mpc, so you check that out and try to build it,
and
>>>>>> Find out you also need gmp, so you check that out and
try
>>>>>> To build it, and find out that you also need mpfr, …..
>>>>> 
>>>>> I can understand your frustration, but these are all
completely external libraries, and it does not really make sense to include this
into any mono-repository.
>>>>> 
>>>>> For example, you are also dependent on libc headers, are
you going to include these into your repository?  And kernel headers?  The end
result will possibly include half of GitHub into that mono-repository... :)
>>>>> 
>>>>> So as usual, for any open source project, read the
requirements first, and install those from whatever your local package
management system is.
>>>>> 
>>>>> If LLVM is going to use a mono-repository, it should only
include LLVM components, in my opinion.
>>>> 
>>>> This seems a good principle in general, but note that we
already included external projects in the repo. Out-of-my-head right now I can
think of the google tests library, and ISL.
>>>> 
>>>> — 
>>>> Mehdi
>>>> 
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160903/146eb388/attachment.html>

Ok Mehdi.

I'm obviously coming into this pretty late, but from a totally neutral user
perspective having
recently rebuilt llvm and it's components on macOS Sierra. I think I am
interpreting mono versus
multi repo correctly as "single repository for all llvm
components/modules" vs. "multiple
(separate) repositories". So here's a humble perspective from an
outsider: while building
all of the components separately was a slight (very) inconvenience adding a few
more terminal
commands to my workflow, it did force me to RTFM and digest the project's
architecture and
framework. In the end, I have a much better understanding of how all this
connects.
Again, the impact on day to day development and maintenance is best left to you
guys and
gals, but retaining a multiple logical delineation between modules works for me.

Has the move to native git/github been formally approved yet? I use Sourcetree,
moving away from git svn would be a plus...

It's all good.

Regards - Patrice Kouame

Sent from my iPad
> On Sep 3, 2016, at 10:59 PM, Mehdi Amini <mehdi.amini at apple.com>
wrote:
> 
> 
>> On Sep 3, 2016, at 6:05 PM, Patrice Kouame <pkouame at mac.com>
wrote:
>> 
>> Someone mentioned llvm in a mono repository below…
> 
> Right, we actually have a proposal to take what is in the current SVN repo
here: http://llvm.org/svn/llvm-project/ and migrate this to a single repository.
> I was not sure if you were referring to this proposal (monorepo) or to the
recent emails about “external libraries” that GCC uses like  gmp and mpfr.
> 
> You can find more details here: https://reviews.llvm.org/D24167
> 
> If you have some good reasons why you would think a proposal would be
problematic to you, or one would better fit your workflow, feel free to expose
them now.
> 
> Best,
> 
> — 
> Mehdi
> 
>> 
>>> On Sep 3, 2016, at 7:37 PM, Mehdi Amini <mehdi.amini at
apple.com> wrote:
>>> 
>>> 
>>> 
>>>> On Sep 3, 2016, at 3:04 PM, Patrice Kouame <pkouame at
mac.com> wrote:
>>>> 
>>>> 
>>>> +1 for keeping it separate.
>>> 
>>> Can you clarify what you referring to specifically?
>>> This sub thread (the last 4 messages) started with a mention of GCC
dependencies. It is not clear to me how to relate to llvm now.
>>> 
>>> Mehdi
>>> 
>>>> 
>>>> One can easily set up a git subproject structure if the need is
pressing...
>>>> 
>>>> Patrice
>>>> 
>>>>> On Sep 3, 2016, at 5:39 PM, Mehdi Amini via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
>>>>> 
>>>>> 
>>>>>> On Sep 2, 2016, at 12:33 PM, Dimitry Andric via
llvm-dev <llvm-dev at lists.llvm.org> wrote:
>>>>>> 
>>>>>> On 16 Aug 2016, at 00:12, Lawrence, Peter via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
>>>>>>> 
>>>>>>> As someone that has worked with both gcc and llvm,
>>>>>>> 
>>>>>>> One thing about gcc that drives me bat-guano-crazy
is that
>>>>>>> First you check out gcc, try to build it, and find
that you also
>>>>>>> Need mpc, so you check that out and try to build
it, and
>>>>>>> Find out you also need gmp, so you check that out
and try
>>>>>>> To build it, and find out that you also need mpfr,
…..
>>>>>> 
>>>>>> I can understand your frustration, but these are all
completely external libraries, and it does not really make sense to include this
into any mono-repository.
>>>>>> 
>>>>>> For example, you are also dependent on libc headers,
are you going to include these into your repository?  And kernel headers?  The
end result will possibly include half of GitHub into that mono-repository... :)
>>>>>> 
>>>>>> So as usual, for any open source project, read the
requirements first, and install those from whatever your local package
management system is.
>>>>>> 
>>>>>> If LLVM is going to use a mono-repository, it should
only include LLVM components, in my opinion.
>>>>> 
>>>>> This seems a good principle in general, but note that we
already included external projects in the repo. Out-of-my-head right now I can
think of the google tests library, and ISL.
>>>>> 
>>>>> — 
>>>>> Mehdi
>>>>> 
>>>>> _______________________________________________
>>>>> LLVM Developers mailing list
>>>>> llvm-dev at lists.llvm.org
>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160904/504b20e7/attachment.html>

> On Sep 4, 2016, at 9:34 AM, Patrice Kouame <pkouame at mac.com>
wrote:
> 
> Ok Mehdi.
> 
> I'm obviously coming into this pretty late, but from a totally neutral
user perspective having
> recently rebuilt llvm and it's components on macOS Sierra. I think I am
interpreting mono versus
> multi repo correctly as "single repository for all llvm
components/modules" vs. "multiple
> (separate) repositories". So here's a humble perspective from an
outsider: while building
> all of the components separately was a slight (very) inconvenience adding a
few more terminal
> commands to my workflow, it did force me to RTFM and digest the
project's architecture and
> framework. In the end, I have a much better understanding of how all this
connects.
> Again, the impact on day to day development and maintenance is best left to
you guys and
> gals, but retaining a multiple logical delineation between modules works
for me.

Thanks for sharing your thoughts and experience!

> 
> Has the move to native git/github been formally approved yet? I use
Sourcetree, moving away from git svn would be a plus…
No, we’re still formalizing the proposals, then a survey will be sent out, then
we’ll probably discuss this at the next llvm-dev meeting, then I don’t know :)


—
Mehdi

> 
> It's all good.
> 
> Regards - Patrice Kouame
> 
> Sent from my iPad
> 
> On Sep 3, 2016, at 10:59 PM, Mehdi Amini <mehdi.amini at apple.com
<mailto:mehdi.amini at apple.com>> wrote:
> 
>> 
>>> On Sep 3, 2016, at 6:05 PM, Patrice Kouame <pkouame at mac.com
<mailto:pkouame at mac.com>> wrote:
>>> 
>>> Someone mentioned llvm in a mono repository below…
>> 
>> Right, we actually have a proposal to take what is in the current SVN
repo here: http://llvm.org/svn/llvm-project/
<http://llvm.org/svn/llvm-project/> and migrate this to a single
repository.
>> I was not sure if you were referring to this proposal (monorepo) or to
the recent emails about “external libraries” that GCC uses like  gmp and mpfr.
>> 
>> You can find more details here: https://reviews.llvm.org/D24167
<https://reviews.llvm.org/D24167>
>> 
>> If you have some good reasons why you would think a proposal would be
problematic to you, or one would better fit your workflow, feel free to expose
them now.
>> 
>> Best,
>> 
>> — 
>> Mehdi
>> 
>>> 
>>> On Sep 3, 2016, at 7:37 PM, Mehdi Amini <mehdi.amini at
apple.com <mailto:mehdi.amini at apple.com>> wrote:
>>> 
>>>> 
>>>> 
>>>> On Sep 3, 2016, at 3:04 PM, Patrice Kouame <pkouame at
mac.com <mailto:pkouame at mac.com>> wrote:
>>>> 
>>>>> 
>>>>> +1 for keeping it separate. 
>>>> 
>>>> Can you clarify what you referring to specifically?
>>>> This sub thread (the last 4 messages) started with a mention of
GCC dependencies. It is not clear to me how to relate to llvm now.
>>>> 
>>>> Mehdi
>>>> 
>>>>> 
>>>>> One can easily set up a git subproject structure if the
need is pressing...
>>>>> 
>>>>> Patrice
>>>>> 
>>>>>> On Sep 3, 2016, at 5:39 PM, Mehdi Amini via llvm-dev
<llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>
wrote:
>>>>>> 
>>>>>> 
>>>>>>> On Sep 2, 2016, at 12:33 PM, Dimitry Andric via
llvm-dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>> wrote:
>>>>>>> 
>>>>>>> On 16 Aug 2016, at 00:12, Lawrence, Peter via
llvm-dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>> wrote:
>>>>>>>> 
>>>>>>>> As someone that has worked with both gcc and
llvm,
>>>>>>>> 
>>>>>>>> One thing about gcc that drives me
bat-guano-crazy is that
>>>>>>>> First you check out gcc, try to build it, and
find that you also
>>>>>>>> Need mpc, so you check that out and try to
build it, and
>>>>>>>> Find out you also need gmp, so you check that
out and try
>>>>>>>> To build it, and find out that you also need
mpfr, …..
>>>>>>> 
>>>>>>> I can understand your frustration, but these are
all completely external libraries, and it does not really make sense to include
this into any mono-repository.
>>>>>>> 
>>>>>>> For example, you are also dependent on libc
headers, are you going to include these into your repository?  And kernel
headers?  The end result will possibly include half of GitHub into that
mono-repository... :)
>>>>>>> 
>>>>>>> So as usual, for any open source project, read the
requirements first, and install those from whatever your local package
management system is.
>>>>>>> 
>>>>>>> If LLVM is going to use a mono-repository, it
should only include LLVM components, in my opinion.
>>>>>> 
>>>>>> This seems a good principle in general, but note that
we already included external projects in the repo. Out-of-my-head right now I
can think of the google tests library, and ISL.
>>>>>> 
>>>>>> — 
>>>>>> Mehdi
>>>>>> 
>>>>>> _______________________________________________
>>>>>> LLVM Developers mailing list
>>>>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>
>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
>> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160904/723ef989/attachment.html>

Mehdi Amini via llvm-dev <llvm-dev at lists.llvm.org> writes:
> Right, we actually have a proposal to take what is in the current SVN
> repo here: http://llvm.org/svn/llvm-project/ and migrate this to a
> single repository.
> I was not sure if you were referring to this proposal (monorepo) or to
> the recent emails about “external libraries” that GCC uses like gmp
> and mpfr. 
>
> You can find more details here: https://reviews.llvm.org/D24167
>
> If you have some good reasons why you would think a proposal would be
> problematic to you, or one would better fit your workflow, feel free
> to expose them now.
It could be problematic for us depending on how the monorepository is
structured.  We reference the LLVM git repository directly and use it to
migrate to new versions, pick patches, etc.  If LLVM proper were part of
a larger repository that becomes more difficult to do because the commit
file paths won't match.  We'd be back to essentially manual diff+patch
which is quite a step backward from the smoth git-oriented process we
use now.

The document says that the individual git repositories will remain.
Does that mean the monorepository is using git-submodule to manage the
aggregate repository?  If so that should work for us.  I'm more
concerned about the case where the individual repositories' histories
were interwoven into a single repository and the individual repositories
went away.

I have extensive experience transitioning a very large project from a
set of individual repositories to a single repository where we interwove
the individual histories.  It was the right direction for us but I don't
think it would be for LLVM.

I completely understand the benefits of a monorepository.  One of the
biggest for us was the ability to git-bisect across components.  How
does git-bisect work with submodules?  I have very little experience
with submodules but would like to learn more.

                              -David

Hi,

> On Sep 7, 2016, at 10:30 AM, dag at cray.com wrote:
> 
> Mehdi Amini via llvm-dev <llvm-dev at lists.llvm.org> writes:
> 
>> Right, we actually have a proposal to take what is in the current SVN
>> repo here: http://llvm.org/svn/llvm-project/ and migrate this to a
>> single repository.
>> I was not sure if you were referring to this proposal (monorepo) or to
>> the recent emails about “external libraries” that GCC uses like gmp
>> and mpfr. 
>> 
>> You can find more details here: https://reviews.llvm.org/D24167
>> 
>> If you have some good reasons why you would think a proposal would be
>> problematic to you, or one would better fit your workflow, feel free
>> to expose them now.
> 
> It could be problematic for us depending on how the monorepository is
> structured.  We reference the LLVM git repository directly and use it to
> migrate to new versions, pick patches, etc.  If LLVM proper were part of
> a larger repository that becomes more difficult to do because the commit
> file paths won't match.  We'd be back to essentially manual
diff+patch
> which is quite a step backward from the smoth git-oriented process we
> use now.
Can you clarify what you mean? Which part of the process would quite manual
patch that wouldn’t otherwise?
> 
> The document says that the individual git repositories will remain.
> Does that mean the monorepository is using git-submodule to manage the
> aggregate repository?
First, have you read this document: https://reviews.llvm.org/D24167
<https://reviews.llvm.org/D24167> ?

TLDR: The answer is no: you have to see it as it is today, i.e. a single SVN
repo containing all the sub-projects, and “exports” in individual repositories.
The same thing after: a single git repo containing all the subprojects
side-by-side and the *same* “exports” in individual repositories.

>  If so that should work for us.  I'm more
> concerned about the case where the individual repositories' histories
> were interwoven into a single repository and the individual repositories
> went away.
> 
> I have extensive experience transitioning a very large project from a
> set of individual repositories to a single repository where we interwove
> the individual histories.  It was the right direction for us but I
don't
> think it would be for LLVM.
> 
> I completely understand the benefits of a monorepository.  One of the
> biggest for us was the ability to git-bisect across components.  How
> does git-bisect work with submodules?  I have very little experience
> with submodules but would like to learn more.
Fairly easy, the document mentions it in the examples.



— 
Mehdi
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160907/0602283b/attachment.html>