llvm dev - Nov 2018 - New LLVM git repository conversion prototype

On Fri, Nov 16, 2018 at 4:04 PM NAKAMURA Takumi <geek4civic at gmail.com>
wrote:
> Hello, David.
>
> On Sat, Nov 17, 2018 at 7:46 AM David Jones via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> Hi James,
>>
>> I've started working with the prototype layout in context of
Google's
>> internal infrastructure. With deep apologies, I have a (very late, I
know)
>> pair of requests that have only recently solidified for me.
>>
>> 1. Could you add annotated tags after the cut point of each release? (I
>> think this would probably be easy.)
>>
>
> I suppose the monorepo has annotated tags.
>
> $ git describe RELEASE_701/rc1^
> RELEASE_700/final-20-gd0540f76dd4f
>
> $ git show RELEASE_700/final
> tag RELEASE_700/final
> Tagger: Hans Wennborg <hans at hanshq.net>
> Date:   Mon Sep 17 20:38:10 2018
>
> Creating release candidate final from release_700 branch
>
> ...Are they not what you want?
>
No, I want git-describe to be usable on master. (Your example describes a
change to the release branch.)

>
>
>
>>
>> 2. Could you mark branch/tag operations somehow other than a annotated
>> tag on master? (This seems less trivial, but my reasoning is below.)
>>
>>
>> The overall goal is for `git describe` to give a reasonable default
>> output.
>>
>>
>> Reasoning for request #1:
>>
>> Example: add a tag after the cut point of the 7.0 release:
>>
>> $ git log --oneline $(git merge-base HEAD origin/release_70)..HEAD |
tail
>> -1
>> 63297479398 Bump the trunk version to 8.0.0svn
>> $ git tag -m 'Begin development of 8.0.0.' '8.0.0svn'
63297479398
>>
>> Then, `git describe` can give a useful result that would be stable(-ish
>> [*]):
>>
>> $ git describe --match=\*svn
>> 8.0.0svn-7878-gfdb08034fd8
>>
>> ([*] Subject to the normal caveat for git-describe: the trailing short
>> hash may become non-unique over time. This is mentioned in the git
help.)
>>
>> Again, I think these tags could probably be added later, but it would
be
>> nice to have a single source of truth (especially since the tag
annotation
>> is itself a commit).
>>
>> I think we could also probably come up with better than than
'8.0.0svn';
>> although whatever the choice, they probably need to be suitable for
>> --match, because...
>>
>
> Interesting. It'd be better if release.sh has such a functionality.
> Options I think;
>
> 1) Tag 700/rc0 automatically as merge base.
>   I think it's not smart since 8.0.0svn cannot be shown as 8.0.0 but 7.
> 2) Tag 8svn manually at the point of "bump to" by the release
manager.
>   2-1) Improve release.sh to bump version in trunk and tag it
> automatically.
>
> Thoughts?
>
>
>> Reasoning for request #2:
>>
>> Since many other branch/tag operations are stored as annotated tags,
they
>> are currently used by git-describe:
>>
>> $ git describe
>> google/stable/2018-10-04-3473-gfdb08034fd8
>>
>
> We could just consider to remove such tags in trunk for public version of
> the monorepo.
> (I didn't know tags/google are in trunk)
>
> Thanks,
> Takumi
>
>
>
>> This doesn't seem as generally useful to me as the release-cut
based
>> revisions... describing a commit as "the 7878th of 8.0.0
development" seems
>> more generally helpful than saying "the 3473rd after a
google/stable
>> release."
>>
>> A very similar approach might be to drop the existing annotated tags,
and
>> add the unchanged branches as no-change commits (i.e., the equivalent
of
>> `git commit --allow-empty`), with a lightweight tag.
>>
>> Unfortunately, this seems (to me) to be harder to fix than #1. The
>> "similar approach" seems like it could be added safely
without recreating
>> the entire history, but it seems like this would need to be done before
>> folks start depending on the existing tags.
>>
>>
>>
>> (Of course, it's possible I've completely missed something
obvious or
>> somehow fouled up my clone of the repo, and this is already supposed to
>> work... let me know if I've missed something, or if you have a
better
>> approach.)
>>
>> On Fri, Nov 16, 2018 at 1:11 PM James Y Knight via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>>> Yes, I believe all the known issues are resolved and that we're
ready to
>>> go, whenever the other thread (on whether to abandon this monorepo,
and
>>> zipper-commit the existing gitrepositories) is concluded.
>>>
>>> On Thu, Nov 15, 2018 at 4:09 PM Tom Stellard <tstellar at
redhat.com>
>>> wrote:
>>>
>>>> On 10/11/2018 03:27 PM, James Y Knight via llvm-dev wrote:
>>>> > TLDR: https://github.com/llvm-git-prototype/ exists as a
read-only
>>>> mirror of SVN, and is being updated continuously with a script
running on
>>>> an llvm-project AWS VM.
>>>> >
>>>>
>>>> Hi James,
>>>>
>>>> What is the current status of the monorepo, have you resolved
all the
>>>> known issues with the history?  Are there any other changes
that need
>>>> to be made before it can be finalized?
>>>>
>>>> -Tom
>>>>
>>>> > Let me know what you think.
>>>> >
>>>> > I had meant to get this prototype finalized 6 months ago,
and I must
>>>> apologize for the delay. I hope this is close to final for what
we want our
>>>> git repository to look like, and that we can move forward with
the
>>>> remainder of the work to convert to git.
>>>> >
>>>> > At this point, there's no guarantee that the
repository won't be
>>>> rebuilt from scratch with new hashes, if some problem is
discovered which
>>>> requires changing something way back in history. But I hope
we're now close
>>>> to being able to declare a conversion final -- and let people
start
>>>> depending on the hashes being stable.
>>>> >
>>>> > This conversion uses the "flat monorepo" layout,
like the previous
>>>> existing git monorepo, and as discussed previously. The process
generating
>>>> it is different, which allows a more faithful conversion,
including
>>>> branches. I've also converted a bunch of the auxiliary
repositories.
>>>> >
>>>> > I would request that other people help take charge of the
remainder
>>>> of the work. Most importantly -- making a plan for implementing
the *rest*
>>>> of the migration. We have
>>>> https://llvm.org/docs/Proposals/GitHubMove.html, but I think
it'll
>>>> need significant fleshing out and updating. I'm happy to
assist with the
>>>> rest of the migration, but I'd like to _not_ be primarily
responsible for
>>>> other parts beyond svn->git repository conversion.
>>>> >
>>>> > Some things that could be discussed in such a plan:
>>>> >   * Verifying that this conversion is good, what we want,
and
>>>> declaring it final (at which point the hashes can be relied
upon not to
>>>> change).
>>>> >     * Any particular steps wanted here?
>>>> >   * Converting buildbots to use git.
>>>> >   * Phabricator changes?
>>>> >   * How do email notifications get sent for commits?
>>>> >   * Gathering github accounts for all committers, adding
them to a
>>>> github team.
>>>> >   * Deciding upon and announcing a timeline for switching
over.
>>>> >   * Proposing, implementing, and testing new workflows for
direct git
>>>> usage:
>>>> >     * Github pull requests instead of (or in addition to?)
>>>> phabricator?
>>>> >     * Github Protected Branch configuration options?
>>>> >       * E.g. -- direct pushing to git without any
restriction, or,
>>>> require that pull requests be created first?
>>>> >       * Automated Pre-commit testing? Do we setup CI (e.g.
>>>> travis-ci.org <http://travis-ci.org>) to do some testing
on pull
>>>> requests, to reduce avoidable tree breakages?
>>>> >       * Any other github configuration options that need
to be
>>>> decided upon?
>>>> >   * ....other things I forgot about at the moment...
>>>> >   * Timeline for switchover.
>>>> >
>>>> >
>>>> >
>>>> > Anyways, what's been done _so far_ is a full
SVN->Git repository
>>>> conversion. This conversion:
>>>> >   * Places the SVN revision number into the commit
message, as
>>>> "llvm-svn=1234"
>>>> >
>>>> >   * Automatically preserves all branches from the SVN
repository (it
>>>> merges the branches named /$project/branches/$name into a
single "$name"
>>>> branch, attempting, as much as possible, to make the
branch-creation
>>>> commits not look insane).
>>>> >
>>>> >   * Attempts to convert the svn branches in the
"tags" subdir into
>>>> annotated git tags pointing to the proper commit on the parent
branch,
>>>> where feasible. Sometimes this is impossible, since the
"tags" have had
>>>> modifications after their creation. (They're just branches
in SVN, so you
>>>> can do that, although you shouldn't). If so, they're
preserved as a branch
>>>> named "svntag/$name", instead.
>>>> >
>>>> >   * Preserves the svn id -> email mapping that was
in-use at the time
>>>> of each SVN commit, as far as is known.
>>>> >
>>>> >   * Fixes a bunch of -- but not all -- the CVS->SVN
conversion errors
>>>> (due, e.g., to files being renamed directly in the CVS
repository).
>>>> >
>>>> >
>>>> >
>>>> > Most of the SVN directories are migrated into
sub-directories inside
>>>> the main "llvm" mono-repository:
>>>> >   * cfe (renamed to clang in the conversion)
>>>> >   * clang-tools-extra
>>>> >   * compiler-rt
>>>> >   * debuginfo-tests
>>>> >   * dragonegg (also "gcc-plugin", the original
name)
>>>> >   * libclc
>>>> >   * libcxx
>>>> >   * libcxxabi
>>>> >   * libunwind
>>>> >   * lld
>>>> >   * lldb
>>>> >   * llgo
>>>> >   * llvm
>>>> >   * openmp
>>>> >   * parallel-libs
>>>> >   * polly
>>>> >   * pstl
>>>> >   * stacker (deleted after r40406)
>>>> > (Additionally, files added to the
"monorepo-root/trunk" directory in
>>>> SVN end up at the root of this repository).
>>>> >
>>>> > Some SVN projects are still active, but not part of the
LLVM
>>>> codebase. These get migrated to their own separate git
repositories:
>>>> >   * lnt
>>>> >   * test-suite
>>>> >   * www
>>>> >   * www-pubs
>>>> >   * www-releases ## TODO. Not done yet as it requires the
use of
>>>> git-lfs, due to large files.
>>>> >   * zorg
>>>> >
>>>> > A couple inactive projects which are somewhat related to
the LLVM
>>>> codebase, migrated to separate repos:
>>>> >   * poolalloc
>>>> >   * safecode
>>>> >
>>>> > Legacy projects that are not particularly interesting,
migrated to a
>>>> single separate git repository named "archive":
>>>> >   * clang-tests # Copy of GCC 4.2 testsuite, modified to
work with
>>>> clang
>>>> >   * clang-tests-external # Copy of GDB testsuite
>>>> >   * llvm-gcc-4.0 # GCC 4.0, modified for llvm
>>>> >   * llvm-gcc-4.2 # GCC 4.2, modified for llvm
>>>> >   * llvm-gcc-4-2 # (merge with above)
>>>> >   * java
>>>> >   * vmkit
>>>> >   * nightly-test-server
>>>> >   * llbrowse # An LLVM bitcode GUI browser
>>>> >   * television # A different LLVM GUI browser; shows
effects of
>>>> transforms, etc
>>>> >   * website # 2007-era snapshot of website, not actually
maintained
>>>> here.
>>>> >   * core, llvm-top, sample, support, hlvm # from the
"HLVM"
>>>> refactoring attempt.
>>>> >
>>>> > Projects _not_ migrated from SVN in this conversion, since
they're
>>>> elsewhere already:
>>>> >   * giri # Never actually developed here; actually
>>>> https://github.com/liuml07/giri
>>>> >   * klee # Already migrated to github with history;
>>>> https://github.com/klee/klee
>>>> >
>>>> >
>>>> >
>>>> > _______________________________________________
>>>> > LLVM Developers mailing list
>>>> > llvm-dev at lists.llvm.org
>>>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>> >
>>>>
>>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20181116/5393a936/attachment.html>

On Sat, Nov 17, 2018 at 9:06 AM David Jones <dlj at google.com> wrote:
> On Fri, Nov 16, 2018 at 4:04 PM NAKAMURA Takumi <geek4civic at
gmail.com>
> wrote:
>
>> Hello, David.
>>
>> On Sat, Nov 17, 2018 at 7:46 AM David Jones via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>>> Hi James,
>>>
>>> I've started working with the prototype layout in context of
Google's
>>> internal infrastructure. With deep apologies, I have a (very late,
I know)
>>> pair of requests that have only recently solidified for me.
>>>
>>> 1. Could you add annotated tags after the cut point of each
release? (I
>>> think this would probably be easy.)
>>>
>>
>> I suppose the monorepo has annotated tags.
>>
>> $ git describe RELEASE_701/rc1^
>> RELEASE_700/final-20-gd0540f76dd4f
>>
>> $ git show RELEASE_700/final
>> tag RELEASE_700/final
>> Tagger: Hans Wennborg <hans at hanshq.net>
>> Date:   Mon Sep 17 20:38:10 2018
>>
>> Creating release candidate final from release_700 branch
>>
>> ...Are they not what you want?
>>
>
> No, I want git-describe to be usable on master. (Your example describes a
> change to the release branch.)
>
Excuse me. I misread "the cut point" due to my poor English.
I suppose I understood what you want.

>
>
>>
>>
>>
>>>
>>> 2. Could you mark branch/tag operations somehow other than a
annotated
>>> tag on master? (This seems less trivial, but my reasoning is
below.)
>>>
>>>
>>> The overall goal is for `git describe` to give a reasonable default
>>> output.
>>>
>>>
>>> Reasoning for request #1:
>>>
>>> Example: add a tag after the cut point of the 7.0 release:
>>>
>>> $ git log --oneline $(git merge-base HEAD origin/release_70)..HEAD
|
>>> tail -1
>>> 63297479398 Bump the trunk version to 8.0.0svn
>>> $ git tag -m 'Begin development of 8.0.0.'
'8.0.0svn' 63297479398
>>>
>>> Then, `git describe` can give a useful result that would be
stable(-ish
>>> [*]):
>>>
>>> $ git describe --match=\*svn
>>> 8.0.0svn-7878-gfdb08034fd8
>>>
>>> ([*] Subject to the normal caveat for git-describe: the trailing
short
>>> hash may become non-unique over time. This is mentioned in the git
help.)
>>>
>>> Again, I think these tags could probably be added later, but it
would be
>>> nice to have a single source of truth (especially since the tag
annotation
>>> is itself a commit).
>>>
>>> I think we could also probably come up with better than than
'8.0.0svn';
>>> although whatever the choice, they probably need to be suitable for
>>> --match, because...
>>>
>>
>> Interesting. It'd be better if release.sh has such a functionality.
>> Options I think;
>>
>> 1) Tag 700/rc0 automatically as merge base.
>>   I think it's not smart since 8.0.0svn cannot be shown as 8.0.0
but 7.
>> 2) Tag 8svn manually at the point of "bump to" by the release
manager.
>>   2-1) Improve release.sh to bump version in trunk and tag it
>> automatically.
>>
>> Thoughts?
>>
>>
>>> Reasoning for request #2:
>>>
>>> Since many other branch/tag operations are stored as annotated
tags,
>>> they are currently used by git-describe:
>>>
>>> $ git describe
>>> google/stable/2018-10-04-3473-gfdb08034fd8
>>>
>>
>> We could just consider to remove such tags in trunk for public version
of
>> the monorepo.
>> (I didn't know tags/google are in trunk)
>>
>> Thanks,
>> Takumi
>>
>>
>>
>>> This doesn't seem as generally useful to me as the release-cut
based
>>> revisions... describing a commit as "the 7878th of 8.0.0
development" seems
>>> more generally helpful than saying "the 3473rd after a
google/stable
>>> release."
>>>
>>> A very similar approach might be to drop the existing annotated
tags,
>>> and add the unchanged branches as no-change commits (i.e., the
equivalent
>>> of `git commit --allow-empty`), with a lightweight tag.
>>>
>>> Unfortunately, this seems (to me) to be harder to fix than #1. The
>>> "similar approach" seems like it could be added safely
without recreating
>>> the entire history, but it seems like this would need to be done
before
>>> folks start depending on the existing tags.
>>>
>>>
>>>
>>> (Of course, it's possible I've completely missed something
obvious or
>>> somehow fouled up my clone of the repo, and this is already
supposed to
>>> work... let me know if I've missed something, or if you have a
better
>>> approach.)
>>>
>>> On Fri, Nov 16, 2018 at 1:11 PM James Y Knight via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>>> Yes, I believe all the known issues are resolved and that
we're ready
>>>> to go, whenever the other thread (on whether to abandon this
monorepo, and
>>>> zipper-commit the existing gitrepositories) is concluded.
>>>>
>>>> On Thu, Nov 15, 2018 at 4:09 PM Tom Stellard <tstellar at
redhat.com>
>>>> wrote:
>>>>
>>>>> On 10/11/2018 03:27 PM, James Y Knight via llvm-dev wrote:
>>>>> > TLDR: https://github.com/llvm-git-prototype/ exists as
a read-only
>>>>> mirror of SVN, and is being updated continuously with a
script running on
>>>>> an llvm-project AWS VM.
>>>>> >
>>>>>
>>>>> Hi James,
>>>>>
>>>>> What is the current status of the monorepo, have you
resolved all the
>>>>> known issues with the history?  Are there any other changes
that need
>>>>> to be made before it can be finalized?
>>>>>
>>>>> -Tom
>>>>>
>>>>> > Let me know what you think.
>>>>> >
>>>>> > I had meant to get this prototype finalized 6 months
ago, and I must
>>>>> apologize for the delay. I hope this is close to final for
what we want our
>>>>> git repository to look like, and that we can move forward
with the
>>>>> remainder of the work to convert to git.
>>>>> >
>>>>> > At this point, there's no guarantee that the
repository won't be
>>>>> rebuilt from scratch with new hashes, if some problem is
discovered which
>>>>> requires changing something way back in history. But I hope
we're now close
>>>>> to being able to declare a conversion final -- and let
people start
>>>>> depending on the hashes being stable.
>>>>> >
>>>>> > This conversion uses the "flat monorepo"
layout, like the previous
>>>>> existing git monorepo, and as discussed previously. The
process generating
>>>>> it is different, which allows a more faithful conversion,
including
>>>>> branches. I've also converted a bunch of the auxiliary
repositories.
>>>>> >
>>>>> > I would request that other people help take charge of
the remainder
>>>>> of the work. Most importantly -- making a plan for
implementing the *rest*
>>>>> of the migration. We have
>>>>> https://llvm.org/docs/Proposals/GitHubMove.html, but I
think it'll
>>>>> need significant fleshing out and updating. I'm happy
to assist with the
>>>>> rest of the migration, but I'd like to _not_ be
primarily responsible for
>>>>> other parts beyond svn->git repository conversion.
>>>>> >
>>>>> > Some things that could be discussed in such a plan:
>>>>> >   * Verifying that this conversion is good, what we
want, and
>>>>> declaring it final (at which point the hashes can be relied
upon not to
>>>>> change).
>>>>> >     * Any particular steps wanted here?
>>>>> >   * Converting buildbots to use git.
>>>>> >   * Phabricator changes?
>>>>> >   * How do email notifications get sent for commits?
>>>>> >   * Gathering github accounts for all committers,
adding them to a
>>>>> github team.
>>>>> >   * Deciding upon and announcing a timeline for
switching over.
>>>>> >   * Proposing, implementing, and testing new workflows
for direct
>>>>> git usage:
>>>>> >     * Github pull requests instead of (or in addition
to?)
>>>>> phabricator?
>>>>> >     * Github Protected Branch configuration options?
>>>>> >       * E.g. -- direct pushing to git without any
restriction, or,
>>>>> require that pull requests be created first?
>>>>> >       * Automated Pre-commit testing? Do we setup CI
(e.g.
>>>>> travis-ci.org <http://travis-ci.org>) to do some
testing on pull
>>>>> requests, to reduce avoidable tree breakages?
>>>>> >       * Any other github configuration options that
need to be
>>>>> decided upon?
>>>>> >   * ....other things I forgot about at the moment...
>>>>> >   * Timeline for switchover.
>>>>> >
>>>>> >
>>>>> >
>>>>> > Anyways, what's been done _so far_ is a full
SVN->Git repository
>>>>> conversion. This conversion:
>>>>> >   * Places the SVN revision number into the commit
message, as
>>>>> "llvm-svn=1234"
>>>>> >
>>>>> >   * Automatically preserves all branches from the SVN
repository (it
>>>>> merges the branches named /$project/branches/$name into a
single "$name"
>>>>> branch, attempting, as much as possible, to make the
branch-creation
>>>>> commits not look insane).
>>>>> >
>>>>> >   * Attempts to convert the svn branches in the
"tags" subdir into
>>>>> annotated git tags pointing to the proper commit on the
parent branch,
>>>>> where feasible. Sometimes this is impossible, since the
"tags" have had
>>>>> modifications after their creation. (They're just
branches in SVN, so you
>>>>> can do that, although you shouldn't). If so,
they're preserved as a branch
>>>>> named "svntag/$name", instead.
>>>>> >
>>>>> >   * Preserves the svn id -> email mapping that was
in-use at the
>>>>> time of each SVN commit, as far as is known.
>>>>> >
>>>>> >   * Fixes a bunch of -- but not all -- the CVS->SVN
conversion
>>>>> errors (due, e.g., to files being renamed directly in the
CVS repository).
>>>>> >
>>>>> >
>>>>> >
>>>>> > Most of the SVN directories are migrated into
sub-directories inside
>>>>> the main "llvm" mono-repository:
>>>>> >   * cfe (renamed to clang in the conversion)
>>>>> >   * clang-tools-extra
>>>>> >   * compiler-rt
>>>>> >   * debuginfo-tests
>>>>> >   * dragonegg (also "gcc-plugin", the
original name)
>>>>> >   * libclc
>>>>> >   * libcxx
>>>>> >   * libcxxabi
>>>>> >   * libunwind
>>>>> >   * lld
>>>>> >   * lldb
>>>>> >   * llgo
>>>>> >   * llvm
>>>>> >   * openmp
>>>>> >   * parallel-libs
>>>>> >   * polly
>>>>> >   * pstl
>>>>> >   * stacker (deleted after r40406)
>>>>> > (Additionally, files added to the
"monorepo-root/trunk" directory in
>>>>> SVN end up at the root of this repository).
>>>>> >
>>>>> > Some SVN projects are still active, but not part of
the LLVM
>>>>> codebase. These get migrated to their own separate git
repositories:
>>>>> >   * lnt
>>>>> >   * test-suite
>>>>> >   * www
>>>>> >   * www-pubs
>>>>> >   * www-releases ## TODO. Not done yet as it requires
the use of
>>>>> git-lfs, due to large files.
>>>>> >   * zorg
>>>>> >
>>>>> > A couple inactive projects which are somewhat related
to the LLVM
>>>>> codebase, migrated to separate repos:
>>>>> >   * poolalloc
>>>>> >   * safecode
>>>>> >
>>>>> > Legacy projects that are not particularly interesting,
migrated to a
>>>>> single separate git repository named "archive":
>>>>> >   * clang-tests # Copy of GCC 4.2 testsuite, modified
to work with
>>>>> clang
>>>>> >   * clang-tests-external # Copy of GDB testsuite
>>>>> >   * llvm-gcc-4.0 # GCC 4.0, modified for llvm
>>>>> >   * llvm-gcc-4.2 # GCC 4.2, modified for llvm
>>>>> >   * llvm-gcc-4-2 # (merge with above)
>>>>> >   * java
>>>>> >   * vmkit
>>>>> >   * nightly-test-server
>>>>> >   * llbrowse # An LLVM bitcode GUI browser
>>>>> >   * television # A different LLVM GUI browser; shows
effects of
>>>>> transforms, etc
>>>>> >   * website # 2007-era snapshot of website, not
actually maintained
>>>>> here.
>>>>> >   * core, llvm-top, sample, support, hlvm # from the
"HLVM"
>>>>> refactoring attempt.
>>>>> >
>>>>> > Projects _not_ migrated from SVN in this conversion,
since they're
>>>>> elsewhere already:
>>>>> >   * giri # Never actually developed here; actually
>>>>> https://github.com/liuml07/giri
>>>>> >   * klee # Already migrated to github with history;
>>>>> https://github.com/klee/klee
>>>>> >
>>>>> >
>>>>> >
>>>>> > _______________________________________________
>>>>> > LLVM Developers mailing list
>>>>> > llvm-dev at lists.llvm.org
>>>>> >
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>> >
>>>>>
>>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20181117/2520daea/attachment-0001.html>

On Fri, Nov 16, 2018 at 4:08 PM NAKAMURA Takumi <geek4civic at gmail.com>
wrote:
>
>
> On Sat, Nov 17, 2018 at 9:06 AM David Jones <dlj at google.com>
wrote:
>
>> On Fri, Nov 16, 2018 at 4:04 PM NAKAMURA Takumi <geek4civic at
gmail.com>
>> wrote:
>>
>>> Hello, David.
>>>
>>> On Sat, Nov 17, 2018 at 7:46 AM David Jones via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>>> Hi James,
>>>>
>>>> I've started working with the prototype layout in context
of Google's
>>>> internal infrastructure. With deep apologies, I have a (very
late, I know)
>>>> pair of requests that have only recently solidified for me.
>>>>
>>>> 1. Could you add annotated tags after the cut point of each
release? (I
>>>> think this would probably be easy.)
>>>>
>>>
>>> I suppose the monorepo has annotated tags.
>>>
>>> $ git describe RELEASE_701/rc1^
>>> RELEASE_700/final-20-gd0540f76dd4f
>>>
>>> $ git show RELEASE_700/final
>>> tag RELEASE_700/final
>>> Tagger: Hans Wennborg <hans at hanshq.net>
>>> Date:   Mon Sep 17 20:38:10 2018
>>>
>>> Creating release candidate final from release_700 branch
>>>
>>> ...Are they not what you want?
>>>
>>
>> No, I want git-describe to be usable on master. (Your example describes
a
>> change to the release branch.)
>>
>
> Excuse me. I misread "the cut point" due to my poor English.
> I suppose I understood what you want.
>
I actually don't know the "correct" term... picking consistent tag
names
seems like a win for this purpose. :-)

>
>
>>
>>
>>>
>>>
>>>
>>>>
>>>> 2. Could you mark branch/tag operations somehow other than a
annotated
>>>> tag on master? (This seems less trivial, but my reasoning is
below.)
>>>>
>>>>
>>>> The overall goal is for `git describe` to give a reasonable
default
>>>> output.
>>>>
>>>>
>>>> Reasoning for request #1:
>>>>
>>>> Example: add a tag after the cut point of the 7.0 release:
>>>>
>>>> $ git log --oneline $(git merge-base HEAD
origin/release_70)..HEAD |
>>>> tail -1
>>>> 63297479398 Bump the trunk version to 8.0.0svn
>>>> $ git tag -m 'Begin development of 8.0.0.'
'8.0.0svn' 63297479398
>>>>
>>>> Then, `git describe` can give a useful result that would be
stable(-ish
>>>> [*]):
>>>>
>>>> $ git describe --match=\*svn
>>>> 8.0.0svn-7878-gfdb08034fd8
>>>>
>>>> ([*] Subject to the normal caveat for git-describe: the
trailing short
>>>> hash may become non-unique over time. This is mentioned in the
git help.)
>>>>
>>>> Again, I think these tags could probably be added later, but it
would
>>>> be nice to have a single source of truth (especially since the
tag
>>>> annotation is itself a commit).
>>>>
>>>> I think we could also probably come up with better than than
>>>> '8.0.0svn'; although whatever the choice, they probably
need to be suitable
>>>> for --match, because...
>>>>
>>>
>>> Interesting. It'd be better if release.sh has such a
functionality.
>>> Options I think;
>>>
>>> 1) Tag 700/rc0 automatically as merge base.
>>>   I think it's not smart since 8.0.0svn cannot be shown as
8.0.0 but 7.
>>> 2) Tag 8svn manually at the point of "bump to" by the
release manager.
>>>   2-1) Improve release.sh to bump version in trunk and tag it
>>> automatically.
>>>
>>> Thoughts?
>>>
>>>
>>>> Reasoning for request #2:
>>>>
>>>> Since many other branch/tag operations are stored as annotated
tags,
>>>> they are currently used by git-describe:
>>>>
>>>> $ git describe
>>>> google/stable/2018-10-04-3473-gfdb08034fd8
>>>>
>>>
>>> We could just consider to remove such tags in trunk for public
version
>>> of the monorepo.
>>> (I didn't know tags/google are in trunk)
>>>
>>> Thanks,
>>> Takumi
>>>
>>>
>>>
>>>> This doesn't seem as generally useful to me as the
release-cut based
>>>> revisions... describing a commit as "the 7878th of 8.0.0
development" seems
>>>> more generally helpful than saying "the 3473rd after a
google/stable
>>>> release."
>>>>
>>>> A very similar approach might be to drop the existing annotated
tags,
>>>> and add the unchanged branches as no-change commits (i.e., the
equivalent
>>>> of `git commit --allow-empty`), with a lightweight tag.
>>>>
>>>> Unfortunately, this seems (to me) to be harder to fix than #1.
The
>>>> "similar approach" seems like it could be added
safely without recreating
>>>> the entire history, but it seems like this would need to be
done before
>>>> folks start depending on the existing tags.
>>>>
>>>>
>>>>
>>>> (Of course, it's possible I've completely missed
something obvious or
>>>> somehow fouled up my clone of the repo, and this is already
supposed to
>>>> work... let me know if I've missed something, or if you
have a better
>>>> approach.)
>>>>
>>>> On Fri, Nov 16, 2018 at 1:11 PM James Y Knight via llvm-dev
<
>>>> llvm-dev at lists.llvm.org> wrote:
>>>>
>>>>> Yes, I believe all the known issues are resolved and that
we're ready
>>>>> to go, whenever the other thread (on whether to abandon
this monorepo, and
>>>>> zipper-commit the existing gitrepositories) is concluded.
>>>>>
>>>>> On Thu, Nov 15, 2018 at 4:09 PM Tom Stellard <tstellar
at redhat.com>
>>>>> wrote:
>>>>>
>>>>>> On 10/11/2018 03:27 PM, James Y Knight via llvm-dev
wrote:
>>>>>> > TLDR: https://github.com/llvm-git-prototype/
exists as a read-only
>>>>>> mirror of SVN, and is being updated continuously with a
script running on
>>>>>> an llvm-project AWS VM.
>>>>>> >
>>>>>>
>>>>>> Hi James,
>>>>>>
>>>>>> What is the current status of the monorepo, have you
resolved all the
>>>>>> known issues with the history?  Are there any other
changes that need
>>>>>> to be made before it can be finalized?
>>>>>>
>>>>>> -Tom
>>>>>>
>>>>>> > Let me know what you think.
>>>>>> >
>>>>>> > I had meant to get this prototype finalized 6
months ago, and I
>>>>>> must apologize for the delay. I hope this is close to
final for what we
>>>>>> want our git repository to look like, and that we can
move forward with the
>>>>>> remainder of the work to convert to git.
>>>>>> >
>>>>>> > At this point, there's no guarantee that the
repository won't be
>>>>>> rebuilt from scratch with new hashes, if some problem
is discovered which
>>>>>> requires changing something way back in history. But I
hope we're now close
>>>>>> to being able to declare a conversion final -- and let
people start
>>>>>> depending on the hashes being stable.
>>>>>> >
>>>>>> > This conversion uses the "flat monorepo"
layout, like the previous
>>>>>> existing git monorepo, and as discussed previously. The
process generating
>>>>>> it is different, which allows a more faithful
conversion, including
>>>>>> branches. I've also converted a bunch of the
auxiliary repositories.
>>>>>> >
>>>>>> > I would request that other people help take charge
of the remainder
>>>>>> of the work. Most importantly -- making a plan for
implementing the *rest*
>>>>>> of the migration. We have
>>>>>> https://llvm.org/docs/Proposals/GitHubMove.html, but I
think it'll
>>>>>> need significant fleshing out and updating. I'm
happy to assist with the
>>>>>> rest of the migration, but I'd like to _not_ be
primarily responsible for
>>>>>> other parts beyond svn->git repository conversion.
>>>>>> >
>>>>>> > Some things that could be discussed in such a
plan:
>>>>>> >   * Verifying that this conversion is good, what
we want, and
>>>>>> declaring it final (at which point the hashes can be
relied upon not to
>>>>>> change).
>>>>>> >     * Any particular steps wanted here?
>>>>>> >   * Converting buildbots to use git.
>>>>>> >   * Phabricator changes?
>>>>>> >   * How do email notifications get sent for
commits?
>>>>>> >   * Gathering github accounts for all committers,
adding them to a
>>>>>> github team.
>>>>>> >   * Deciding upon and announcing a timeline for
switching over.
>>>>>> >   * Proposing, implementing, and testing new
workflows for direct
>>>>>> git usage:
>>>>>> >     * Github pull requests instead of (or in
addition to?)
>>>>>> phabricator?
>>>>>> >     * Github Protected Branch configuration
options?
>>>>>> >       * E.g. -- direct pushing to git without any
restriction, or,
>>>>>> require that pull requests be created first?
>>>>>> >       * Automated Pre-commit testing? Do we setup
CI (e.g.
>>>>>> travis-ci.org <http://travis-ci.org>) to do some
testing on pull
>>>>>> requests, to reduce avoidable tree breakages?
>>>>>> >       * Any other github configuration options
that need to be
>>>>>> decided upon?
>>>>>> >   * ....other things I forgot about at the
moment...
>>>>>> >   * Timeline for switchover.
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > Anyways, what's been done _so far_ is a full
SVN->Git repository
>>>>>> conversion. This conversion:
>>>>>> >   * Places the SVN revision number into the commit
message, as
>>>>>> "llvm-svn=1234"
>>>>>> >
>>>>>> >   * Automatically preserves all branches from the
SVN repository
>>>>>> (it merges the branches named /$project/branches/$name
into a single
>>>>>> "$name" branch, attempting, as much as
possible, to make the
>>>>>> branch-creation commits not look insane).
>>>>>> >
>>>>>> >   * Attempts to convert the svn branches in the
"tags" subdir into
>>>>>> annotated git tags pointing to the proper commit on the
parent branch,
>>>>>> where feasible. Sometimes this is impossible, since the
"tags" have had
>>>>>> modifications after their creation. (They're just
branches in SVN, so you
>>>>>> can do that, although you shouldn't). If so,
they're preserved as a branch
>>>>>> named "svntag/$name", instead.
>>>>>> >
>>>>>> >   * Preserves the svn id -> email mapping that
was in-use at the
>>>>>> time of each SVN commit, as far as is known.
>>>>>> >
>>>>>> >   * Fixes a bunch of -- but not all -- the
CVS->SVN conversion
>>>>>> errors (due, e.g., to files being renamed directly in
the CVS repository).
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > Most of the SVN directories are migrated into
sub-directories
>>>>>> inside the main "llvm" mono-repository:
>>>>>> >   * cfe (renamed to clang in the conversion)
>>>>>> >   * clang-tools-extra
>>>>>> >   * compiler-rt
>>>>>> >   * debuginfo-tests
>>>>>> >   * dragonegg (also "gcc-plugin", the
original name)
>>>>>> >   * libclc
>>>>>> >   * libcxx
>>>>>> >   * libcxxabi
>>>>>> >   * libunwind
>>>>>> >   * lld
>>>>>> >   * lldb
>>>>>> >   * llgo
>>>>>> >   * llvm
>>>>>> >   * openmp
>>>>>> >   * parallel-libs
>>>>>> >   * polly
>>>>>> >   * pstl
>>>>>> >   * stacker (deleted after r40406)
>>>>>> > (Additionally, files added to the
"monorepo-root/trunk" directory
>>>>>> in SVN end up at the root of this repository).
>>>>>> >
>>>>>> > Some SVN projects are still active, but not part
of the LLVM
>>>>>> codebase. These get migrated to their own separate git
repositories:
>>>>>> >   * lnt
>>>>>> >   * test-suite
>>>>>> >   * www
>>>>>> >   * www-pubs
>>>>>> >   * www-releases ## TODO. Not done yet as it
requires the use of
>>>>>> git-lfs, due to large files.
>>>>>> >   * zorg
>>>>>> >
>>>>>> > A couple inactive projects which are somewhat
related to the LLVM
>>>>>> codebase, migrated to separate repos:
>>>>>> >   * poolalloc
>>>>>> >   * safecode
>>>>>> >
>>>>>> > Legacy projects that are not particularly
interesting, migrated to
>>>>>> a single separate git repository named
"archive":
>>>>>> >   * clang-tests # Copy of GCC 4.2 testsuite,
modified to work with
>>>>>> clang
>>>>>> >   * clang-tests-external # Copy of GDB testsuite
>>>>>> >   * llvm-gcc-4.0 # GCC 4.0, modified for llvm
>>>>>> >   * llvm-gcc-4.2 # GCC 4.2, modified for llvm
>>>>>> >   * llvm-gcc-4-2 # (merge with above)
>>>>>> >   * java
>>>>>> >   * vmkit
>>>>>> >   * nightly-test-server
>>>>>> >   * llbrowse # An LLVM bitcode GUI browser
>>>>>> >   * television # A different LLVM GUI browser;
shows effects of
>>>>>> transforms, etc
>>>>>> >   * website # 2007-era snapshot of website, not
actually maintained
>>>>>> here.
>>>>>> >   * core, llvm-top, sample, support, hlvm # from
the "HLVM"
>>>>>> refactoring attempt.
>>>>>> >
>>>>>> > Projects _not_ migrated from SVN in this
conversion, since they're
>>>>>> elsewhere already:
>>>>>> >   * giri # Never actually developed here; actually
>>>>>> https://github.com/liuml07/giri
>>>>>> >   * klee # Already migrated to github with
history;
>>>>>> https://github.com/klee/klee
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > _______________________________________________
>>>>>> > LLVM Developers mailing list
>>>>>> > llvm-dev at lists.llvm.org
>>>>>> >
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>> >
>>>>>>
>>>>>> _______________________________________________
>>>>> LLVM Developers mailing list
>>>>> llvm-dev at lists.llvm.org
>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20181116/df126a26/attachment.html>