llvm dev - Jul 2016 - [cfe-dev] [lldb-dev] Sequential ID Git hook

> -----Original Message-----
> From: cfe-dev [mailto:cfe-dev-bounces at lists.llvm.org] On Behalf Of
Renato
> Golin via cfe-dev
> Sent: Thursday, June 30, 2016 9:49 AM
> To: Reid Kleckner
> Cc: LLVM Dev; llvm-foundation at lists.llvm.org; Clang Dev; LLDB Dev
> Subject: Re: [cfe-dev] [lldb-dev] [llvm-dev] Sequential ID Git hook
> 
> On 30 June 2016 at 17:33, Reid Kleckner <rnk at google.com> wrote:
> > Agreed, the llvm-project repository can completely take on the role of
> the
> > SQL database in Renato's proposal.
> 
> Hum, doing it in a separate server was suggested by the GitHub folks,
> so I just assumed they can't do that in the umbrella project for some
> reason.
> 
> I'm all for using the umbrella if we can, I just though we
couldn't... :(
> 
> Can someone try the suggested tag style? Are we sure we can guarantee
> atomicity in there? I know SQL can. :)
> 
> I know that changing the commit message works because of Gerrit and
> our current SVN integration, I don't know how much adding one tag for
> each push will work in Git over time.
We were using tags for a while in our own SVN->git conversion internally.
(git branch is pushed to SVN and the SVN r-number used to create a tag.)
They are convenient for some things, but each tag adds a new (if small)
file to .git/tags and I don't know that it really scales well when you
are talking about (long term) hundreds of thousands of them.  That was
not what tags were designed for.

We've since stopped creating the tags, and gotten used to not having
them.  We do the 'rev-list --count' trick which mainly gets recorded as
one component of the version number, and it has been working for us.

I think having the number in the commit log (even if it's just for the
superproject) would be preferable.  You can use 'git log --grep' to
find a particular rev if you need to.
--paulr
> 
> cheers,
> --renato
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

On Jun 30, 2016, at 2:25 PM, Robinson, Paul via llvm-dev <llvm-dev at
lists.llvm.org> wrote:

(talking about lots of tags)
> I don't know that it really scales well when you
> are talking about (long term) hundreds of thousands of them.
I can say from experience that it does not scale well.    After some time,
everyone would start feeling the pain.


Jim Rowan
jmr at codeaurora.org
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by the
Linux Foundation



-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160630/1bbacda4/attachment.html>

On 6/30/2016 5:20 PM, Robinson, Paul via cfe-dev wrote:
> We were using tags for a while in our own SVN->git conversion
internally.
> (git branch is pushed to SVN and the SVN r-number used to create a tag.)
> They are convenient for some things, but each tag adds a new (if small)
> file to .git/tags and I don't know that it really scales well when you
> are talking about (long term) hundreds of thousands of them.  That was
> not what tags were designed for.
We're using tags in this manner for our internal repos and LLVM/Clang 
mirrors and haven't experienced any problems.  We're at ~50k tags for 
our most used repo, so not quite at hundreds of thousands yet.

When I look in .git/refs/tags of one of my repos, I do *not* see 50k 
files; I see ~400.  I'm not sure what causes some to appear here and 
others not.

I don't see how this use of tags is not representative of what tags were 
designed for.  They are designed to label a commit.  That seems to match 
well what is desired here.
> We've since stopped creating the tags, and gotten used to not having
> them.  We do the 'rev-list --count' trick which mainly gets
recorded as
> one component of the version number, and it has been working for us.
As I understand it, 'git rev-list --count HEAD' requires walking the 
entire commit history.  Perhaps the performance is actually ok in 
practice, but I would be concerned about scaling with this approach as well:

$ time git rev-list --count HEAD
115968

real    0m1.170s
user    0m1.100s
sys     0m0.064s
> I think having the number in the commit log (even if it's just for the
> superproject) would be preferable.  You can use 'git log --grep' to
> find a particular rev if you need to.
Grepping every commit doesn't seem like the most scalable option either. 
  I did a quick test on a large repo.  First a grep for an identifier:

$ time git log --grep <id>
...
real    0m1.450s
user    0m1.340s
sys     0m0.092s

Then I did the same for the associated push tag:

$ time git log -n 1 <tag>
...
real    0m0.048s
user    0m0.024s
sys     0m0.016s

Tom.

On 6/30/2016 6:18 PM, Jim Rowan via cfe-dev wrote:
>
> On Jun 30, 2016, at 2:25 PM, Robinson, Paul via llvm-dev
> <llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>> wrote:
>
> (talking about lots of tags)
>
>> I don't know that it really scales well when you
>> are talking about (long term) hundreds of thousands of them.
>
> I can say from experience that it does not scale well.    After some
> time, everyone would start feeling the pain.
Can you elaborate on this?  As I mentioned in another email, we're at 
~50k tags in one repo and not having any problems.  I can't see why git 
would fundamentally have scaling or performance issues in conjunction 
with lots of tags.  Perhaps some UI interfaces were failing to scale well?

Tom.

On 1 July 2016 at 08:18, Tom Honermann via lldb-dev
<lldb-dev at lists.llvm.org> wrote:
> On 6/30/2016 5:20 PM, Robinson, Paul via cfe-dev wrote:
> We're using tags in this manner for our internal repos and LLVM/Clang
> mirrors and haven't experienced any problems.  We're at ~50k tags
for
> our most used repo, so not quite at hundreds of thousands yet.
My issue with using tags like this is that they pollute the tag
namespace and will quickly swamp what I consider to be the important
ones ("release-X"). OK, so we've got "git tag -l
'release*'" but
that's pretty ugly.

Tim.

On Sat, Jul 2, 2016 at 3:18 AM, Tom Honermann via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> We're using tags in this manner for our internal repos and LLVM/Clang
> mirrors and haven't experienced any problems.  We're at ~50k tags
for
> our most used repo, so not quite at hundreds of thousands yet.
>
> When I look in .git/refs/tags of one of my repos, I do *not* see 50k
> files; I see ~400.  I'm not sure what causes some to appear here and
> others not.
>
You don't see many tags because tags (both heavy and lightweight) are
transferred from the file system into a pack file whenever a "git
repack"
or "git gc" happens.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160706/aa448679/attachment.html>