llvm dev - Nov 2018 - New LLVM git repository conversion prototype

Hi James,

I've started working with the prototype layout in context of Google's
internal infrastructure. With deep apologies, I have a (very late, I know)
pair of requests that have only recently solidified for me.

1. Could you add annotated tags after the cut point of each release? (I
think this would probably be easy.)

2. Could you mark branch/tag operations somehow other than a annotated tag
on master? (This seems less trivial, but my reasoning is below.)


The overall goal is for `git describe` to give a reasonable default output.


Reasoning for request #1:

Example: add a tag after the cut point of the 7.0 release:

$ git log --oneline $(git merge-base HEAD origin/release_70)..HEAD | tail -1
63297479398 Bump the trunk version to 8.0.0svn
$ git tag -m 'Begin development of 8.0.0.' '8.0.0svn'
63297479398

Then, `git describe` can give a useful result that would be stable(-ish
[*]):

$ git describe --match=\*svn
8.0.0svn-7878-gfdb08034fd8

([*] Subject to the normal caveat for git-describe: the trailing short hash
may become non-unique over time. This is mentioned in the git help.)

Again, I think these tags could probably be added later, but it would be
nice to have a single source of truth (especially since the tag annotation
is itself a commit).

I think we could also probably come up with better than than '8.0.0svn';
although whatever the choice, they probably need to be suitable for
--match, because...



Reasoning for request #2:

Since many other branch/tag operations are stored as annotated tags, they
are currently used by git-describe:

$ git describe
google/stable/2018-10-04-3473-gfdb08034fd8

This doesn't seem as generally useful to me as the release-cut based
revisions... describing a commit as "the 7878th of 8.0.0 development"
seems
more generally helpful than saying "the 3473rd after a google/stable
release."

A very similar approach might be to drop the existing annotated tags, and
add the unchanged branches as no-change commits (i.e., the equivalent of
`git commit --allow-empty`), with a lightweight tag.

Unfortunately, this seems (to me) to be harder to fix than #1. The "similar
approach" seems like it could be added safely without recreating the entire
history, but it seems like this would need to be done before folks start
depending on the existing tags.



(Of course, it's possible I've completely missed something obvious or
somehow fouled up my clone of the repo, and this is already supposed to
work... let me know if I've missed something, or if you have a better
approach.)

On Fri, Nov 16, 2018 at 1:11 PM James Y Knight via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Yes, I believe all the known issues are resolved and that we're ready
to
> go, whenever the other thread (on whether to abandon this monorepo, and
> zipper-commit the existing gitrepositories) is concluded.
>
> On Thu, Nov 15, 2018 at 4:09 PM Tom Stellard <tstellar at redhat.com>
wrote:
>
>> On 10/11/2018 03:27 PM, James Y Knight via llvm-dev wrote:
>> > TLDR: https://github.com/llvm-git-prototype/ exists as a read-only
>> mirror of SVN, and is being updated continuously with a script running
on
>> an llvm-project AWS VM.
>> >
>>
>> Hi James,
>>
>> What is the current status of the monorepo, have you resolved all the
>> known issues with the history?  Are there any other changes that need
>> to be made before it can be finalized?
>>
>> -Tom
>>
>> > Let me know what you think.
>> >
>> > I had meant to get this prototype finalized 6 months ago, and I
must
>> apologize for the delay. I hope this is close to final for what we want
our
>> git repository to look like, and that we can move forward with the
>> remainder of the work to convert to git.
>> >
>> > At this point, there's no guarantee that the repository
won't be
>> rebuilt from scratch with new hashes, if some problem is discovered
which
>> requires changing something way back in history. But I hope we're
now close
>> to being able to declare a conversion final -- and let people start
>> depending on the hashes being stable.
>> >
>> > This conversion uses the "flat monorepo" layout, like
the previous
>> existing git monorepo, and as discussed previously. The process
generating
>> it is different, which allows a more faithful conversion, including
>> branches. I've also converted a bunch of the auxiliary
repositories.
>> >
>> > I would request that other people help take charge of the
remainder of
>> the work. Most importantly -- making a plan for implementing the *rest*
of
>> the migration. We have https://llvm.org/docs/Proposals/GitHubMove.html,
>> but I think it'll need significant fleshing out and updating.
I'm happy to
>> assist with the rest of the migration, but I'd like to _not_ be
primarily
>> responsible for other parts beyond svn->git repository conversion.
>> >
>> > Some things that could be discussed in such a plan:
>> >   * Verifying that this conversion is good, what we want, and
declaring
>> it final (at which point the hashes can be relied upon not to change).
>> >     * Any particular steps wanted here?
>> >   * Converting buildbots to use git.
>> >   * Phabricator changes?
>> >   * How do email notifications get sent for commits?
>> >   * Gathering github accounts for all committers, adding them to a
>> github team.
>> >   * Deciding upon and announcing a timeline for switching over.
>> >   * Proposing, implementing, and testing new workflows for direct
git
>> usage:
>> >     * Github pull requests instead of (or in addition to?)
phabricator?
>> >     * Github Protected Branch configuration options?
>> >       * E.g. -- direct pushing to git without any restriction, or,
>> require that pull requests be created first?
>> >       * Automated Pre-commit testing? Do we setup CI (e.g.
>> travis-ci.org <http://travis-ci.org>) to do some testing on pull
>> requests, to reduce avoidable tree breakages?
>> >       * Any other github configuration options that need to be
decided
>> upon?
>> >   * ....other things I forgot about at the moment...
>> >   * Timeline for switchover.
>> >
>> >
>> >
>> > Anyways, what's been done _so far_ is a full SVN->Git
repository
>> conversion. This conversion:
>> >   * Places the SVN revision number into the commit message, as
>> "llvm-svn=1234"
>> >
>> >   * Automatically preserves all branches from the SVN repository
(it
>> merges the branches named /$project/branches/$name into a single
"$name"
>> branch, attempting, as much as possible, to make the branch-creation
>> commits not look insane).
>> >
>> >   * Attempts to convert the svn branches in the "tags"
subdir into
>> annotated git tags pointing to the proper commit on the parent branch,
>> where feasible. Sometimes this is impossible, since the
"tags" have had
>> modifications after their creation. (They're just branches in SVN,
so you
>> can do that, although you shouldn't). If so, they're preserved
as a branch
>> named "svntag/$name", instead.
>> >
>> >   * Preserves the svn id -> email mapping that was in-use at
the time
>> of each SVN commit, as far as is known.
>> >
>> >   * Fixes a bunch of -- but not all -- the CVS->SVN conversion
errors
>> (due, e.g., to files being renamed directly in the CVS repository).
>> >
>> >
>> >
>> > Most of the SVN directories are migrated into sub-directories
inside
>> the main "llvm" mono-repository:
>> >   * cfe (renamed to clang in the conversion)
>> >   * clang-tools-extra
>> >   * compiler-rt
>> >   * debuginfo-tests
>> >   * dragonegg (also "gcc-plugin", the original name)
>> >   * libclc
>> >   * libcxx
>> >   * libcxxabi
>> >   * libunwind
>> >   * lld
>> >   * lldb
>> >   * llgo
>> >   * llvm
>> >   * openmp
>> >   * parallel-libs
>> >   * polly
>> >   * pstl
>> >   * stacker (deleted after r40406)
>> > (Additionally, files added to the "monorepo-root/trunk"
directory in
>> SVN end up at the root of this repository).
>> >
>> > Some SVN projects are still active, but not part of the LLVM
codebase.
>> These get migrated to their own separate git repositories:
>> >   * lnt
>> >   * test-suite
>> >   * www
>> >   * www-pubs
>> >   * www-releases ## TODO. Not done yet as it requires the use of
>> git-lfs, due to large files.
>> >   * zorg
>> >
>> > A couple inactive projects which are somewhat related to the LLVM
>> codebase, migrated to separate repos:
>> >   * poolalloc
>> >   * safecode
>> >
>> > Legacy projects that are not particularly interesting, migrated to
a
>> single separate git repository named "archive":
>> >   * clang-tests # Copy of GCC 4.2 testsuite, modified to work with
clang
>> >   * clang-tests-external # Copy of GDB testsuite
>> >   * llvm-gcc-4.0 # GCC 4.0, modified for llvm
>> >   * llvm-gcc-4.2 # GCC 4.2, modified for llvm
>> >   * llvm-gcc-4-2 # (merge with above)
>> >   * java
>> >   * vmkit
>> >   * nightly-test-server
>> >   * llbrowse # An LLVM bitcode GUI browser
>> >   * television # A different LLVM GUI browser; shows effects of
>> transforms, etc
>> >   * website # 2007-era snapshot of website, not actually
maintained
>> here.
>> >   * core, llvm-top, sample, support, hlvm # from the
"HLVM" refactoring
>> attempt.
>> >
>> > Projects _not_ migrated from SVN in this conversion, since
they're
>> elsewhere already:
>> >   * giri # Never actually developed here; actually
>> https://github.com/liuml07/giri
>> >   * klee # Already migrated to github with history;
>> https://github.com/klee/klee
>> >
>> >
>> >
>> > _______________________________________________
>> > LLVM Developers mailing list
>> > llvm-dev at lists.llvm.org
>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>> >
>>
>> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20181116/f985649c/attachment.html>

On Fri, Nov 16, 2018 at 5:46 PM David Jones <dlj at google.com> wrote:
> Hi James,
>
> I've started working with the prototype layout in context of
Google's
> internal infrastructure. With deep apologies, I have a (very late, I know)
> pair of requests that have only recently solidified for me.
>
Changing the set of, the names of, or in many cases, even the contents of,
the tags/branches in the repository is going to be significantly less
intrusive than changing commit hashes on trunk. I don't think it's at
all
too late to have this discussion. (Although, before engaging too much more
in making changes, I'd _really_ like to finally resolve the zippered
repository alternative thread, which decides whether all this gets entirely
discarded...)

> 1. Could you add annotated tags after the cut point of each release? (I
> think this would probably be easy.)
>
The tags and commits in the repository right now are migrated from
subversion. If we want additional tags around releases, the most
straightforward way to do that will be to add the tags we want in svn (once
we decide what we want).

> 2. Could you mark branch/tag operations somehow other than a annotated tag
> on master? (This seems less trivial, but my reasoning is below.)
>
Only subversion copies, to the "*/tags/*" hierarchy, which don't
have any
changes to their content after the copy (e.g. commits made to the
"tag"),
get converted from a git branch into a git annotated tag. That entire
tag-conversion process could easily be scrapped, leaving everything as a
git branch (named svntag/*). But I think that is not a good idea in general
-- e.g., the RELEASE_701/rc2 tag on the release_70 branch really does
function best as an annotated tag.

That said, I do think most of the branches and tags in the repository
should be moved to an "llvm-historical-branches" repository (or
something
like that), which most people will ignore, but is available for those who
want to go digging through historical artifacts. After the git migration,
people will most likely make their own non-release branches and tags in
their own git forks published in their own github accounts, not in the
primary llvm repository.

We do still need to select which branches/tags to retain in the primary
repository -- I haven't thought much about this, beyond that it should be
less than everything, and should have at least the releases branches and
tags.

> The overall goal is for `git describe` to give a reasonable default output.
>
>
> Reasoning for request #1:
>
> Example: add a tag after the cut point of the 7.0 release:
>
> $ git log --oneline $(git merge-base HEAD origin/release_70)..HEAD | tail
> -1
>
Sidenote: this can be more simply spelled:
 $ git log --oneline origin/release_70..HEAD | tail -1
if you recall that "X..Y" is simply a shortcut for "^X Y"
(that is: all
commits reachable from Y, excluding those reachable from X; so X having
additional branch commits that aren't on master makes no difference).


63297479398 Bump the trunk version to 8.0.0svn
> $ git tag -m 'Begin development of 8.0.0.' '8.0.0svn'
63297479398
>
> Then, `git describe` can give a useful result that would be stable(-ish
> [*]):
>
> $ git describe --match=\*svn
> 8.0.0svn-7878-gfdb08034fd8
>
> ([*] Subject to the normal caveat for git-describe: the trailing short
> hash may become non-unique over time. This is mentioned in the git help.)
>
> Again, I think these tags could probably be added later, but it would be
> nice to have a single source of truth (especially since the tag annotation
> is itself a commit).
>
> I think we could also probably come up with better than than
'8.0.0svn';
> although whatever the choice, they probably need to be suitable for
> --match, because...
>
We could name tags named something like: "v8.0dev" (for trunk tag),
"v8.0.0rc1" (after making a branch and updating the version numbers in
the
branch), "v8.0.0" (for the final release from the branch). And, use
--match="v*" in the describe output. I think this basically obviates
the
need for request #2.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20181116/96f7f1fd/attachment.html>

Hello, David.

On Sat, Nov 17, 2018 at 7:46 AM David Jones via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hi James,
>
> I've started working with the prototype layout in context of
Google's
> internal infrastructure. With deep apologies, I have a (very late, I know)
> pair of requests that have only recently solidified for me.
>
> 1. Could you add annotated tags after the cut point of each release? (I
> think this would probably be easy.)
>
I suppose the monorepo has annotated tags.

$ git describe RELEASE_701/rc1^
RELEASE_700/final-20-gd0540f76dd4f

$ git show RELEASE_700/final
tag RELEASE_700/final
Tagger: Hans Wennborg <hans at hanshq.net>
Date:   Mon Sep 17 20:38:10 2018

Creating release candidate final from release_700 branch

...Are they not what you want?


>
> 2. Could you mark branch/tag operations somehow other than a annotated tag
> on master? (This seems less trivial, but my reasoning is below.)
>
>
> The overall goal is for `git describe` to give a reasonable default output.
>
>
> Reasoning for request #1:
>
> Example: add a tag after the cut point of the 7.0 release:
>
> $ git log --oneline $(git merge-base HEAD origin/release_70)..HEAD | tail
> -1
> 63297479398 Bump the trunk version to 8.0.0svn
> $ git tag -m 'Begin development of 8.0.0.' '8.0.0svn'
63297479398
>
> Then, `git describe` can give a useful result that would be stable(-ish
> [*]):
>
> $ git describe --match=\*svn
> 8.0.0svn-7878-gfdb08034fd8
>
> ([*] Subject to the normal caveat for git-describe: the trailing short
> hash may become non-unique over time. This is mentioned in the git help.)
>
> Again, I think these tags could probably be added later, but it would be
> nice to have a single source of truth (especially since the tag annotation
> is itself a commit).
>
> I think we could also probably come up with better than than
'8.0.0svn';
> although whatever the choice, they probably need to be suitable for
> --match, because...
>
Interesting. It'd be better if release.sh has such a functionality.
Options I think;

1) Tag 700/rc0 automatically as merge base.
  I think it's not smart since 8.0.0svn cannot be shown as 8.0.0 but 7.
2) Tag 8svn manually at the point of "bump to" by the release manager.
  2-1) Improve release.sh to bump version in trunk and tag it automatically.

Thoughts?

> Reasoning for request #2:
>
> Since many other branch/tag operations are stored as annotated tags, they
> are currently used by git-describe:
>
> $ git describe
> google/stable/2018-10-04-3473-gfdb08034fd8
>
We could just consider to remove such tags in trunk for public version of
the monorepo.
(I didn't know tags/google are in trunk)

Thanks,
Takumi


> This doesn't seem as generally useful to me as the release-cut based
> revisions... describing a commit as "the 7878th of 8.0.0
development" seems
> more generally helpful than saying "the 3473rd after a google/stable
> release."
>
> A very similar approach might be to drop the existing annotated tags, and
> add the unchanged branches as no-change commits (i.e., the equivalent of
> `git commit --allow-empty`), with a lightweight tag.
>
> Unfortunately, this seems (to me) to be harder to fix than #1. The
> "similar approach" seems like it could be added safely without
recreating
> the entire history, but it seems like this would need to be done before
> folks start depending on the existing tags.
>
>
>
> (Of course, it's possible I've completely missed something obvious
or
> somehow fouled up my clone of the repo, and this is already supposed to
> work... let me know if I've missed something, or if you have a better
> approach.)
>
> On Fri, Nov 16, 2018 at 1:11 PM James Y Knight via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> Yes, I believe all the known issues are resolved and that we're
ready to
>> go, whenever the other thread (on whether to abandon this monorepo, and
>> zipper-commit the existing gitrepositories) is concluded.
>>
>> On Thu, Nov 15, 2018 at 4:09 PM Tom Stellard <tstellar at
redhat.com> wrote:
>>
>>> On 10/11/2018 03:27 PM, James Y Knight via llvm-dev wrote:
>>> > TLDR: https://github.com/llvm-git-prototype/ exists as a
read-only
>>> mirror of SVN, and is being updated continuously with a script
running on
>>> an llvm-project AWS VM.
>>> >
>>>
>>> Hi James,
>>>
>>> What is the current status of the monorepo, have you resolved all
the
>>> known issues with the history?  Are there any other changes that
need
>>> to be made before it can be finalized?
>>>
>>> -Tom
>>>
>>> > Let me know what you think.
>>> >
>>> > I had meant to get this prototype finalized 6 months ago, and
I must
>>> apologize for the delay. I hope this is close to final for what we
want our
>>> git repository to look like, and that we can move forward with the
>>> remainder of the work to convert to git.
>>> >
>>> > At this point, there's no guarantee that the repository
won't be
>>> rebuilt from scratch with new hashes, if some problem is discovered
which
>>> requires changing something way back in history. But I hope
we're now close
>>> to being able to declare a conversion final -- and let people start
>>> depending on the hashes being stable.
>>> >
>>> > This conversion uses the "flat monorepo" layout,
like the previous
>>> existing git monorepo, and as discussed previously. The process
generating
>>> it is different, which allows a more faithful conversion, including
>>> branches. I've also converted a bunch of the auxiliary
repositories.
>>> >
>>> > I would request that other people help take charge of the
remainder of
>>> the work. Most importantly -- making a plan for implementing the
*rest* of
>>> the migration. We have
https://llvm.org/docs/Proposals/GitHubMove.html,
>>> but I think it'll need significant fleshing out and updating.
I'm happy to
>>> assist with the rest of the migration, but I'd like to _not_ be
primarily
>>> responsible for other parts beyond svn->git repository
conversion.
>>> >
>>> > Some things that could be discussed in such a plan:
>>> >   * Verifying that this conversion is good, what we want, and
>>> declaring it final (at which point the hashes can be relied upon
not to
>>> change).
>>> >     * Any particular steps wanted here?
>>> >   * Converting buildbots to use git.
>>> >   * Phabricator changes?
>>> >   * How do email notifications get sent for commits?
>>> >   * Gathering github accounts for all committers, adding them
to a
>>> github team.
>>> >   * Deciding upon and announcing a timeline for switching
over.
>>> >   * Proposing, implementing, and testing new workflows for
direct git
>>> usage:
>>> >     * Github pull requests instead of (or in addition to?)
phabricator?
>>> >     * Github Protected Branch configuration options?
>>> >       * E.g. -- direct pushing to git without any restriction,
or,
>>> require that pull requests be created first?
>>> >       * Automated Pre-commit testing? Do we setup CI (e.g.
>>> travis-ci.org <http://travis-ci.org>) to do some testing on
pull
>>> requests, to reduce avoidable tree breakages?
>>> >       * Any other github configuration options that need to be
decided
>>> upon?
>>> >   * ....other things I forgot about at the moment...
>>> >   * Timeline for switchover.
>>> >
>>> >
>>> >
>>> > Anyways, what's been done _so far_ is a full SVN->Git
repository
>>> conversion. This conversion:
>>> >   * Places the SVN revision number into the commit message, as
>>> "llvm-svn=1234"
>>> >
>>> >   * Automatically preserves all branches from the SVN
repository (it
>>> merges the branches named /$project/branches/$name into a single
"$name"
>>> branch, attempting, as much as possible, to make the
branch-creation
>>> commits not look insane).
>>> >
>>> >   * Attempts to convert the svn branches in the
"tags" subdir into
>>> annotated git tags pointing to the proper commit on the parent
branch,
>>> where feasible. Sometimes this is impossible, since the
"tags" have had
>>> modifications after their creation. (They're just branches in
SVN, so you
>>> can do that, although you shouldn't). If so, they're
preserved as a branch
>>> named "svntag/$name", instead.
>>> >
>>> >   * Preserves the svn id -> email mapping that was in-use
at the time
>>> of each SVN commit, as far as is known.
>>> >
>>> >   * Fixes a bunch of -- but not all -- the CVS->SVN
conversion errors
>>> (due, e.g., to files being renamed directly in the CVS repository).
>>> >
>>> >
>>> >
>>> > Most of the SVN directories are migrated into sub-directories
inside
>>> the main "llvm" mono-repository:
>>> >   * cfe (renamed to clang in the conversion)
>>> >   * clang-tools-extra
>>> >   * compiler-rt
>>> >   * debuginfo-tests
>>> >   * dragonegg (also "gcc-plugin", the original name)
>>> >   * libclc
>>> >   * libcxx
>>> >   * libcxxabi
>>> >   * libunwind
>>> >   * lld
>>> >   * lldb
>>> >   * llgo
>>> >   * llvm
>>> >   * openmp
>>> >   * parallel-libs
>>> >   * polly
>>> >   * pstl
>>> >   * stacker (deleted after r40406)
>>> > (Additionally, files added to the
"monorepo-root/trunk" directory in
>>> SVN end up at the root of this repository).
>>> >
>>> > Some SVN projects are still active, but not part of the LLVM
codebase.
>>> These get migrated to their own separate git repositories:
>>> >   * lnt
>>> >   * test-suite
>>> >   * www
>>> >   * www-pubs
>>> >   * www-releases ## TODO. Not done yet as it requires the use
of
>>> git-lfs, due to large files.
>>> >   * zorg
>>> >
>>> > A couple inactive projects which are somewhat related to the
LLVM
>>> codebase, migrated to separate repos:
>>> >   * poolalloc
>>> >   * safecode
>>> >
>>> > Legacy projects that are not particularly interesting,
migrated to a
>>> single separate git repository named "archive":
>>> >   * clang-tests # Copy of GCC 4.2 testsuite, modified to work
with
>>> clang
>>> >   * clang-tests-external # Copy of GDB testsuite
>>> >   * llvm-gcc-4.0 # GCC 4.0, modified for llvm
>>> >   * llvm-gcc-4.2 # GCC 4.2, modified for llvm
>>> >   * llvm-gcc-4-2 # (merge with above)
>>> >   * java
>>> >   * vmkit
>>> >   * nightly-test-server
>>> >   * llbrowse # An LLVM bitcode GUI browser
>>> >   * television # A different LLVM GUI browser; shows effects
of
>>> transforms, etc
>>> >   * website # 2007-era snapshot of website, not actually
maintained
>>> here.
>>> >   * core, llvm-top, sample, support, hlvm # from the
"HLVM"
>>> refactoring attempt.
>>> >
>>> > Projects _not_ migrated from SVN in this conversion, since
they're
>>> elsewhere already:
>>> >   * giri # Never actually developed here; actually
>>> https://github.com/liuml07/giri
>>> >   * klee # Already migrated to github with history;
>>> https://github.com/klee/klee
>>> >
>>> >
>>> >
>>> > _______________________________________________
>>> > LLVM Developers mailing list
>>> > llvm-dev at lists.llvm.org
>>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>> >
>>>
>>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20181117/bc8171e0/attachment.html>

On Fri, Nov 16, 2018 at 4:04 PM NAKAMURA Takumi <geek4civic at gmail.com>
wrote:
> Hello, David.
>
> On Sat, Nov 17, 2018 at 7:46 AM David Jones via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> Hi James,
>>
>> I've started working with the prototype layout in context of
Google's
>> internal infrastructure. With deep apologies, I have a (very late, I
know)
>> pair of requests that have only recently solidified for me.
>>
>> 1. Could you add annotated tags after the cut point of each release? (I
>> think this would probably be easy.)
>>
>
> I suppose the monorepo has annotated tags.
>
> $ git describe RELEASE_701/rc1^
> RELEASE_700/final-20-gd0540f76dd4f
>
> $ git show RELEASE_700/final
> tag RELEASE_700/final
> Tagger: Hans Wennborg <hans at hanshq.net>
> Date:   Mon Sep 17 20:38:10 2018
>
> Creating release candidate final from release_700 branch
>
> ...Are they not what you want?
>
No, I want git-describe to be usable on master. (Your example describes a
change to the release branch.)

>
>
>
>>
>> 2. Could you mark branch/tag operations somehow other than a annotated
>> tag on master? (This seems less trivial, but my reasoning is below.)
>>
>>
>> The overall goal is for `git describe` to give a reasonable default
>> output.
>>
>>
>> Reasoning for request #1:
>>
>> Example: add a tag after the cut point of the 7.0 release:
>>
>> $ git log --oneline $(git merge-base HEAD origin/release_70)..HEAD |
tail
>> -1
>> 63297479398 Bump the trunk version to 8.0.0svn
>> $ git tag -m 'Begin development of 8.0.0.' '8.0.0svn'
63297479398
>>
>> Then, `git describe` can give a useful result that would be stable(-ish
>> [*]):
>>
>> $ git describe --match=\*svn
>> 8.0.0svn-7878-gfdb08034fd8
>>
>> ([*] Subject to the normal caveat for git-describe: the trailing short
>> hash may become non-unique over time. This is mentioned in the git
help.)
>>
>> Again, I think these tags could probably be added later, but it would
be
>> nice to have a single source of truth (especially since the tag
annotation
>> is itself a commit).
>>
>> I think we could also probably come up with better than than
'8.0.0svn';
>> although whatever the choice, they probably need to be suitable for
>> --match, because...
>>
>
> Interesting. It'd be better if release.sh has such a functionality.
> Options I think;
>
> 1) Tag 700/rc0 automatically as merge base.
>   I think it's not smart since 8.0.0svn cannot be shown as 8.0.0 but 7.
> 2) Tag 8svn manually at the point of "bump to" by the release
manager.
>   2-1) Improve release.sh to bump version in trunk and tag it
> automatically.
>
> Thoughts?
>
>
>> Reasoning for request #2:
>>
>> Since many other branch/tag operations are stored as annotated tags,
they
>> are currently used by git-describe:
>>
>> $ git describe
>> google/stable/2018-10-04-3473-gfdb08034fd8
>>
>
> We could just consider to remove such tags in trunk for public version of
> the monorepo.
> (I didn't know tags/google are in trunk)
>
> Thanks,
> Takumi
>
>
>
>> This doesn't seem as generally useful to me as the release-cut
based
>> revisions... describing a commit as "the 7878th of 8.0.0
development" seems
>> more generally helpful than saying "the 3473rd after a
google/stable
>> release."
>>
>> A very similar approach might be to drop the existing annotated tags,
and
>> add the unchanged branches as no-change commits (i.e., the equivalent
of
>> `git commit --allow-empty`), with a lightweight tag.
>>
>> Unfortunately, this seems (to me) to be harder to fix than #1. The
>> "similar approach" seems like it could be added safely
without recreating
>> the entire history, but it seems like this would need to be done before
>> folks start depending on the existing tags.
>>
>>
>>
>> (Of course, it's possible I've completely missed something
obvious or
>> somehow fouled up my clone of the repo, and this is already supposed to
>> work... let me know if I've missed something, or if you have a
better
>> approach.)
>>
>> On Fri, Nov 16, 2018 at 1:11 PM James Y Knight via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>>> Yes, I believe all the known issues are resolved and that we're
ready to
>>> go, whenever the other thread (on whether to abandon this monorepo,
and
>>> zipper-commit the existing gitrepositories) is concluded.
>>>
>>> On Thu, Nov 15, 2018 at 4:09 PM Tom Stellard <tstellar at
redhat.com>
>>> wrote:
>>>
>>>> On 10/11/2018 03:27 PM, James Y Knight via llvm-dev wrote:
>>>> > TLDR: https://github.com/llvm-git-prototype/ exists as a
read-only
>>>> mirror of SVN, and is being updated continuously with a script
running on
>>>> an llvm-project AWS VM.
>>>> >
>>>>
>>>> Hi James,
>>>>
>>>> What is the current status of the monorepo, have you resolved
all the
>>>> known issues with the history?  Are there any other changes
that need
>>>> to be made before it can be finalized?
>>>>
>>>> -Tom
>>>>
>>>> > Let me know what you think.
>>>> >
>>>> > I had meant to get this prototype finalized 6 months ago,
and I must
>>>> apologize for the delay. I hope this is close to final for what
we want our
>>>> git repository to look like, and that we can move forward with
the
>>>> remainder of the work to convert to git.
>>>> >
>>>> > At this point, there's no guarantee that the
repository won't be
>>>> rebuilt from scratch with new hashes, if some problem is
discovered which
>>>> requires changing something way back in history. But I hope
we're now close
>>>> to being able to declare a conversion final -- and let people
start
>>>> depending on the hashes being stable.
>>>> >
>>>> > This conversion uses the "flat monorepo" layout,
like the previous
>>>> existing git monorepo, and as discussed previously. The process
generating
>>>> it is different, which allows a more faithful conversion,
including
>>>> branches. I've also converted a bunch of the auxiliary
repositories.
>>>> >
>>>> > I would request that other people help take charge of the
remainder
>>>> of the work. Most importantly -- making a plan for implementing
the *rest*
>>>> of the migration. We have
>>>> https://llvm.org/docs/Proposals/GitHubMove.html, but I think
it'll
>>>> need significant fleshing out and updating. I'm happy to
assist with the
>>>> rest of the migration, but I'd like to _not_ be primarily
responsible for
>>>> other parts beyond svn->git repository conversion.
>>>> >
>>>> > Some things that could be discussed in such a plan:
>>>> >   * Verifying that this conversion is good, what we want,
and
>>>> declaring it final (at which point the hashes can be relied
upon not to
>>>> change).
>>>> >     * Any particular steps wanted here?
>>>> >   * Converting buildbots to use git.
>>>> >   * Phabricator changes?
>>>> >   * How do email notifications get sent for commits?
>>>> >   * Gathering github accounts for all committers, adding
them to a
>>>> github team.
>>>> >   * Deciding upon and announcing a timeline for switching
over.
>>>> >   * Proposing, implementing, and testing new workflows for
direct git
>>>> usage:
>>>> >     * Github pull requests instead of (or in addition to?)
>>>> phabricator?
>>>> >     * Github Protected Branch configuration options?
>>>> >       * E.g. -- direct pushing to git without any
restriction, or,
>>>> require that pull requests be created first?
>>>> >       * Automated Pre-commit testing? Do we setup CI (e.g.
>>>> travis-ci.org <http://travis-ci.org>) to do some testing
on pull
>>>> requests, to reduce avoidable tree breakages?
>>>> >       * Any other github configuration options that need
to be
>>>> decided upon?
>>>> >   * ....other things I forgot about at the moment...
>>>> >   * Timeline for switchover.
>>>> >
>>>> >
>>>> >
>>>> > Anyways, what's been done _so far_ is a full
SVN->Git repository
>>>> conversion. This conversion:
>>>> >   * Places the SVN revision number into the commit
message, as
>>>> "llvm-svn=1234"
>>>> >
>>>> >   * Automatically preserves all branches from the SVN
repository (it
>>>> merges the branches named /$project/branches/$name into a
single "$name"
>>>> branch, attempting, as much as possible, to make the
branch-creation
>>>> commits not look insane).
>>>> >
>>>> >   * Attempts to convert the svn branches in the
"tags" subdir into
>>>> annotated git tags pointing to the proper commit on the parent
branch,
>>>> where feasible. Sometimes this is impossible, since the
"tags" have had
>>>> modifications after their creation. (They're just branches
in SVN, so you
>>>> can do that, although you shouldn't). If so, they're
preserved as a branch
>>>> named "svntag/$name", instead.
>>>> >
>>>> >   * Preserves the svn id -> email mapping that was
in-use at the time
>>>> of each SVN commit, as far as is known.
>>>> >
>>>> >   * Fixes a bunch of -- but not all -- the CVS->SVN
conversion errors
>>>> (due, e.g., to files being renamed directly in the CVS
repository).
>>>> >
>>>> >
>>>> >
>>>> > Most of the SVN directories are migrated into
sub-directories inside
>>>> the main "llvm" mono-repository:
>>>> >   * cfe (renamed to clang in the conversion)
>>>> >   * clang-tools-extra
>>>> >   * compiler-rt
>>>> >   * debuginfo-tests
>>>> >   * dragonegg (also "gcc-plugin", the original
name)
>>>> >   * libclc
>>>> >   * libcxx
>>>> >   * libcxxabi
>>>> >   * libunwind
>>>> >   * lld
>>>> >   * lldb
>>>> >   * llgo
>>>> >   * llvm
>>>> >   * openmp
>>>> >   * parallel-libs
>>>> >   * polly
>>>> >   * pstl
>>>> >   * stacker (deleted after r40406)
>>>> > (Additionally, files added to the
"monorepo-root/trunk" directory in
>>>> SVN end up at the root of this repository).
>>>> >
>>>> > Some SVN projects are still active, but not part of the
LLVM
>>>> codebase. These get migrated to their own separate git
repositories:
>>>> >   * lnt
>>>> >   * test-suite
>>>> >   * www
>>>> >   * www-pubs
>>>> >   * www-releases ## TODO. Not done yet as it requires the
use of
>>>> git-lfs, due to large files.
>>>> >   * zorg
>>>> >
>>>> > A couple inactive projects which are somewhat related to
the LLVM
>>>> codebase, migrated to separate repos:
>>>> >   * poolalloc
>>>> >   * safecode
>>>> >
>>>> > Legacy projects that are not particularly interesting,
migrated to a
>>>> single separate git repository named "archive":
>>>> >   * clang-tests # Copy of GCC 4.2 testsuite, modified to
work with
>>>> clang
>>>> >   * clang-tests-external # Copy of GDB testsuite
>>>> >   * llvm-gcc-4.0 # GCC 4.0, modified for llvm
>>>> >   * llvm-gcc-4.2 # GCC 4.2, modified for llvm
>>>> >   * llvm-gcc-4-2 # (merge with above)
>>>> >   * java
>>>> >   * vmkit
>>>> >   * nightly-test-server
>>>> >   * llbrowse # An LLVM bitcode GUI browser
>>>> >   * television # A different LLVM GUI browser; shows
effects of
>>>> transforms, etc
>>>> >   * website # 2007-era snapshot of website, not actually
maintained
>>>> here.
>>>> >   * core, llvm-top, sample, support, hlvm # from the
"HLVM"
>>>> refactoring attempt.
>>>> >
>>>> > Projects _not_ migrated from SVN in this conversion, since
they're
>>>> elsewhere already:
>>>> >   * giri # Never actually developed here; actually
>>>> https://github.com/liuml07/giri
>>>> >   * klee # Already migrated to github with history;
>>>> https://github.com/klee/klee
>>>> >
>>>> >
>>>> >
>>>> > _______________________________________________
>>>> > LLVM Developers mailing list
>>>> > llvm-dev at lists.llvm.org
>>>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>> >
>>>>
>>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20181116/5393a936/attachment.html>

On Fri, Nov 16, 2018 at 3:54 PM James Y Knight <jyknight at google.com>
wrote:
> On Fri, Nov 16, 2018 at 5:46 PM David Jones <dlj at google.com>
wrote:
>
>> Hi James,
>>
>> I've started working with the prototype layout in context of
Google's
>> internal infrastructure. With deep apologies, I have a (very late, I
know)
>> pair of requests that have only recently solidified for me.
>>
>
> Changing the set of, the names of, or in many cases, even the contents of,
> the tags/branches in the repository is going to be significantly less
> intrusive than changing commit hashes on trunk. I don't think it's
at all
> too late to have this discussion. (Although, before engaging too much more
> in making changes, I'd _really_ like to finally resolve the zippered
> repository alternative thread, which decides whether all this gets entirely
> discarded...)
>
>
>> 1. Could you add annotated tags after the cut point of each release? (I
>> think this would probably be easy.)
>>
>
> The tags and commits in the repository right now are migrated from
> subversion. If we want additional tags around releases, the most
> straightforward way to do that will be to add the tags we want in svn (once
> we decide what we want).
>
>
I can certainly do that... this strategy doesn't really make sense from the
Subversion side (but I suppose that's fine for conversion purposes).

> 2. Could you mark branch/tag operations somehow other than a annotated tag
>> on master? (This seems less trivial, but my reasoning is below.)
>>
>
> Only subversion copies, to the "*/tags/*" hierarchy, which
don't have any
> changes to their content after the copy (e.g. commits made to the
"tag"),
> get converted from a git branch into a git annotated tag. That entire
> tag-conversion process could easily be scrapped, leaving everything as a
> git branch (named svntag/*). But I think that is not a good idea in general
> -- e.g., the RELEASE_701/rc2 tag on the release_70 branch really does
> function best as an annotated tag.
>
Side note: on a release branch, that's fine... it doesn't cause problems
on
master since that tag doesn't point to a commit reachable from master/HEAD.
The problem is when annotated tags are reachable from master.

>
> That said, I do think most of the branches and tags in the repository
> should be moved to an "llvm-historical-branches" repository (or
something
> like that), which most people will ignore, but is available for those who
> want to go digging through historical artifacts. After the git migration,
> people will most likely make their own non-release branches and tags in
> their own git forks published in their own github accounts, not in the
> primary llvm repository.
>
> We do still need to select which branches/tags to retain in the primary
> repository -- I haven't thought much about this, beyond that it should
be
> less than everything, and should have at least the releases branches and
> tags.
>
>
>> The overall goal is for `git describe` to give a reasonable default
>> output.
>>
>
>>
>> Reasoning for request #1:
>>
>> Example: add a tag after the cut point of the 7.0 release:
>>
>> $ git log --oneline $(git merge-base HEAD origin/release_70)..HEAD |
tail
>> -1
>>
>
> Sidenote: this can be more simply spelled:
>  $ git log --oneline origin/release_70..HEAD | tail -1
> if you recall that "X..Y" is simply a shortcut for "^X
Y" (that is: all
> commits reachable from Y, excluding those reachable from X; so X having
> additional branch commits that aren't on master makes no difference).
>
>
> 63297479398 Bump the trunk version to 8.0.0svn
>> $ git tag -m 'Begin development of 8.0.0.' '8.0.0svn'
63297479398
>>
>> Then, `git describe` can give a useful result that would be stable(-ish
>> [*]):
>>
>> $ git describe --match=\*svn
>> 8.0.0svn-7878-gfdb08034fd8
>>
>> ([*] Subject to the normal caveat for git-describe: the trailing short
>> hash may become non-unique over time. This is mentioned in the git
help.)
>>
>> Again, I think these tags could probably be added later, but it would
be
>> nice to have a single source of truth (especially since the tag
annotation
>> is itself a commit).
>>
>> I think we could also probably come up with better than than
'8.0.0svn';
>> although whatever the choice, they probably need to be suitable for
>> --match, because...
>>
>
> We could name tags named something like: "v8.0dev" (for trunk
tag),
> "v8.0.0rc1" (after making a branch and updating the version
numbers in the
> branch), "v8.0.0" (for the final release from the branch). And,
use
> --match="v*" in the describe output. I think this basically
obviates the
> need for request #2.
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20181116/ba45546a/attachment.html>

On Sat, 17 Nov 2018 at 10:25, James Y Knight via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
>
>
> We could name tags named something like: "v8.0dev" (for trunk
tag),
> "v8.0.0rc1" (after making a branch and updating the version
numbers in the
> branch), "v8.0.0" (for the final release from the branch). And,
use
> --match="v*" in the describe output. I think this basically
obviates the
> need for request #2.
>
>
>
Semantic versioning would recommend "v8.0.0-dev",
"v8.0.0-rc1" etc. The
hyphen indicating that this is a pre-release version coming before
"v8.0.0"
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20181117/35fcb585/attachment.html>

James Y Knight via llvm-dev <llvm-dev at lists.llvm.org>
writes:
> On Fri, Nov 16, 2018 at 5:46 PM David Jones <dlj at google.com>
wrote:
>> $ git tag -m 'Begin development of 8.0.0.' '8.0.0svn'
63297479398
>>
>> Then, `git describe` can give a useful result that would be stable(-ish
>> [*]):
>>
>> $ git describe --match=\*svn
>> 8.0.0svn-7878-gfdb08034fd8
>>
>> ([*] Subject to the normal caveat for git-describe: the trailing short
>> hash may become non-unique over time. This is mentioned in the git
help.)
>>
>> Again, I think these tags could probably be added later, but it would
be
>> nice to have a single source of truth (especially since the tag
annotation
>> is itself a commit).
>>
>> I think we could also probably come up with better than than
'8.0.0svn';
>> although whatever the choice, they probably need to be suitable for
>> --match, because...
>>
>
> We could name tags named something like: "v8.0dev" (for trunk
tag),
> "v8.0.0rc1" (after making a branch and updating the version
numbers in the
> branch), "v8.0.0" (for the final release from the branch). And,
use
> --match="v*" in the describe output. I think this basically
obviates the
> need for request #2.
This seems like a very good thing to do, and if it wasn't obvious, this
is also something we should plan to do going forward once we start
branching our releases in git directly.

As a bit of a side note, v8.0.0 is probably too brief - I expect v*
could easily match some arbitrary tag that starts with the letter v too
easily. I don't have strong opinions about the particular name, but
something like llvm-8.0.0 or llvm.org-v8.0.0 would be better.