Chandler Carruth via llvm-dev
2016-Feb-24 20:57 UTC
[llvm-dev] RFC: Move the test-suite LLVM project to GitHub?
Subject kinda says it all. Here is my rationale: The test-suite is really weird relative to the rest of the LLVM project: 1) It contains all manner of crazily licensed code. 2) We don't really care about the history at all. Any concerns around linear history or bisection are pretty much irrelevant. 3) We don't ever plan to have LLVM code move into or out from the test-suite 4) Its already big, and really should be much bigger. We shouldn't have incentives to keep stuff out of the test suite because of size, hosting cost, or anything else. For all of these reasons, and also because I'd like to see how well (or rather, how poorly) a service like GitHub actually works for the project, it seems like splitting the test-suite out of the current subversion repository and moving it there is the right call. When I chatted with folks on the board, this made sense to them as well, and I've made sure we have a reasonable LLVM organization set up on GitHub and all the board members are on it: https://github.com/llvm (I think only my membership is public at the moment). There is still plenty to figure out about how to manage this on github, but before doing anything else I just wanted to shoot an email and see if folks like this idea. Thanks! -Chandler -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160224/2a26a845/attachment.html>
Pete Cooper via llvm-dev
2016-Feb-24 21:12 UTC
[llvm-dev] RFC: Move the test-suite LLVM project to GitHub?
> On Feb 24, 2016, at 12:57 PM, Chandler Carruth via llvm-dev <llvm-dev at lists.llvm.org> wrote: > > Subject kinda says it all. Here is my rationale: > > The test-suite is really weird relative to the rest of the LLVM project: > 1) It contains all manner of crazily licensed code. > 2) We don't really care about the history at all. Any concerns around linear history or bisection are pretty much irrelevant. > 3) We don't ever plan to have LLVM code move into or out from the test-suite > 4) Its already big, and really should be much bigger. We shouldn't have incentives to keep stuff out of the test suite because of size, hosting cost, or anything else. > > For all of these reasons, and also because I'd like to see how well (or rather, how poorly) a service like GitHub actually works for the project, it seems like splitting the test-suite out of the current subversion repository and moving it there is the right call. > > When I chatted with folks on the board, this made sense to them as well, and I've made sure we have a reasonable LLVM organization set up on GitHub and all the board members are on it: https://github.com/llvm <https://github.com/llvm> (I think only my membership is public at the moment). > > There is still plenty to figure out about how to manage this on github,Thank for pointing out this part. We certainly need a discussion on how to handle a git repo, but that can be after others have chimed in on this post.> but before doing anything else I just wanted to shoot an email and see if folks like this idea.Sounds good. +1 from me for all the various reasons you already stated. Cheers, Pete> > Thanks! > -Chandler > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160224/6f23b22e/attachment.html>
David Blaikie via llvm-dev
2016-Feb-24 21:15 UTC
[llvm-dev] RFC: Move the test-suite LLVM project to GitHub?
On Wed, Feb 24, 2016 at 12:57 PM, Chandler Carruth via llvm-dev < llvm-dev at lists.llvm.org> wrote:> Subject kinda says it all. Here is my rationale: > > The test-suite is really weird relative to the rest of the LLVM project: > 1) It contains all manner of crazily licensed code. > 2) We don't really care about the history at all. Any concerns around > linear history or bisection are pretty much irrelevant. > 3) We don't ever plan to have LLVM code move into or out from the > test-suite > 4) Its already big, and really should be much bigger. We shouldn't have > incentives to keep stuff out of the test suite because of size, hosting > cost, or anything else. > > For all of these reasons, and also because I'd like to see how well (or > rather, how poorly) a service like GitHub actually works for the project, > it seems like splitting the test-suite out of the current subversion > repository and moving it there is the right call. > > When I chatted with folks on the board, this made sense to them as well, > and I've made sure we have a reasonable LLVM organization set up on GitHub > and all the board members are on it: https://github.com/llvm (I think > only my membership is public at the moment). > > There is still plenty to figure out about how to manage this on github, > but before doing anything else I just wanted to shoot an email and see if > folks like this idea. >My question's probably somewhere in this "plenty to figure out" but it'll be moderately annoying to have multiple ways of managing LLVM subprojects (yeah, I realize this is sort of an exceptional one - one I don't usually have checked out anyway, so I don't care too much - but it would break my cute little script that trawls subrepositories and syncs them all up in my llvm repo/checkout). Also, I assume there's some amount of version lock between the rest of the project and the test-suite (cleaning it up if we make breaking changes, etc - the idea of having LLVM bitcode in there for Halide would mean that we wouldn't want to run newer versions of the test-suite on older versions of the compiler, etc). So, yeah, just curious about the practical problems, no philosophical objection I suppose. - Dave> > Thanks! > -Chandler > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160224/a9e37588/attachment.html>
Jonathan Roelofs via llvm-dev
2016-Feb-24 21:20 UTC
[llvm-dev] RFC: Move the test-suite LLVM project to GitHub?
On 2/24/16 1:57 PM, Chandler Carruth via llvm-dev wrote:> Subject kinda says it all. Here is my rationale: > > The test-suite is really weird relative to the rest of the LLVM project: > 1) It contains all manner of crazily licensed code. > 2) We don't really care about the history at all. Any concerns around > linear history or bisection are pretty much irrelevant. > 3) We don't ever plan to have LLVM code move into or out from the test-suite > 4) Its already big, and really should be much bigger. We shouldn't have > incentives to keep stuff out of the test suite because of size, hosting > cost, or anything else.There are two size limitations w.r.t. moving to github: 1) They have a prohibition on having individual files over 100M [1], unless the repository is set up with their Git LFS plugin. This plugin hasn't been around all that long, and is not something I would use in production. (Not to mention the fact that it's not supported on public forks anyway [2]) The largest individual files in the test-suite are around 10-12M, so this isn't a problem yet, but it could become one later... something to think about. 2) There is the cap on total repository size, which is in the neighborhood of 1Gb [3]. A fresh checkout of test-suite clocks in at just over 3Gb. This one actually is a problem.> > For all of these reasons, and also because I'd like to see how well (or > rather, how poorly) a service like GitHub actually works for the > project, it seems like splitting the test-suite out of the current > subversion repository and moving it there is the right call. > > When I chatted with folks on the board, this made sense to them as well, > and I've made sure we have a reasonable LLVM organization set up on > GitHub and all the board members are on it: https://github.com/llvm (I > think only my membership is public at the moment). > > There is still plenty to figure out about how to manage this on github, > but before doing anything else I just wanted to shoot an email and see > if folks like this idea.+1, assuming it can be made to work given the other concerns above. Jon 1: https://help.github.com/articles/working-with-large-files/ 2: https://github.com/github/git-lfs/issues/773#issuecomment-150569337 3: https://help.github.com/articles/what-is-my-disk-quota/> > Thanks! > -Chandler > > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >-- Jon Roelofs jonathan at codesourcery.com CodeSourcery / Mentor Embedded
Matthias Braun via llvm-dev
2016-Feb-24 21:25 UTC
[llvm-dev] RFC: Move the test-suite LLVM project to GitHub?
I don't really care where the repository is located, but I do have some comments on the future test-suite directions:> On Feb 24, 2016, at 12:57 PM, Chandler Carruth via llvm-dev <llvm-dev at lists.llvm.org> wrote: > > Subject kinda says it all. Here is my rationale: > > The test-suite is really weird relative to the rest of the LLVM project: > 1) It contains all manner of crazily licensed code.That's indeed a good reason to move the repository away.> 2) We don't really care about the history at all. Any concerns around linear history or bisection are pretty much irrelevant.We do care about the history. Sometimes benchmarks get fixed or tweaked which may change the results, we should be able to dig into the history to see what happened when. In any way retaining the history wouldn't be a problem, would it?> 3) We don't ever plan to have LLVM code move into or out from the test-suiteI could actually see moving llvm code into the test-suite (we already use lit code from llvm) but indeed move code out of the testsuite into llvm I don't foresee happening.> 4) Its already big, and really should be much bigger. We shouldn't have incentives to keep stuff out of the test suite because of size, hosting cost, or anything else.I agree with the goal of having a big test-suite. However I think there is a point where we should rather strive to have a stable base system for building and running tests, etc. and then have the actual benchmarks/tests being modules on top of that. We already have that situation today with External/SPEC* and I think it would be a good idea to have a mode where you just checkout more benchmarks into a test-suite subdirectory and they are automatically recognized and used (in fact that is something on my TODO list though at a very low position). - Matthias> > For all of these reasons, and also because I'd like to see how well (or rather, how poorly) a service like GitHub actually works for the project, it seems like splitting the test-suite out of the current subversion repository and moving it there is the right call. > > When I chatted with folks on the board, this made sense to them as well, and I've made sure we have a reasonable LLVM organization set up on GitHub and all the board members are on it: https://github.com/llvm <https://github.com/llvm> (I think only my membership is public at the moment). > > There is still plenty to figure out about how to manage this on github, but before doing anything else I just wanted to shoot an email and see if folks like this idea. > > Thanks! > -Chandler > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160224/48ea276d/attachment.html>
John Criswell via llvm-dev
2016-Feb-24 21:26 UTC
[llvm-dev] RFC: Move the test-suite LLVM project to GitHub?
Dear Chandler, First, can you articulate why you want to move the test suite to Github? Is it taking up too much space, or is there some other problem that you're trying to solve? I think you clearly explain why moving the revision history isn't necessary, but it's not clear to me what problem you are trying to solve. Second, if we move the revision history to Github, it would be nice to archive the existing Subversion history somewhere (e.g., leave it on llvm.org but disable commit access to it). The test suite has been used in numerous research papers, so keeping the revision history around is good practice. We should only delete the Subversion revision history if keeping it around is just too onerous. Third, I assume your plan is to continue to track changes on Github. Is that correct? As long as there's a good reason to do it and the existing Subversion history isn't deleted, I don't see a problem with the change. Regards, John Criswell On 2/24/16 3:57 PM, Chandler Carruth via llvm-dev wrote:> Subject kinda says it all. Here is my rationale: > > The test-suite is really weird relative to the rest of the LLVM project: > 1) It contains all manner of crazily licensed code. > 2) We don't really care about the history at all. Any concerns around > linear history or bisection are pretty much irrelevant. > 3) We don't ever plan to have LLVM code move into or out from the > test-suite > 4) Its already big, and really should be much bigger. We shouldn't > have incentives to keep stuff out of the test suite because of size, > hosting cost, or anything else. > > For all of these reasons, and also because I'd like to see how well > (or rather, how poorly) a service like GitHub actually works for the > project, it seems like splitting the test-suite out of the current > subversion repository and moving it there is the right call. > > When I chatted with folks on the board, this made sense to them as > well, and I've made sure we have a reasonable LLVM organization set up > on GitHub and all the board members are on it: https://github.com/llvm > (I think only my membership is public at the moment). > > There is still plenty to figure out about how to manage this on > github, but before doing anything else I just wanted to shoot an email > and see if folks like this idea. > > Thanks! > -Chandler > > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev-- John Criswell Assistant Professor Department of Computer Science, University of Rochester http://www.cs.rochester.edu/u/criswell -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160224/eb38182d/attachment.html>
John Criswell via llvm-dev
2016-Feb-24 21:31 UTC
[llvm-dev] RFC: Move the test-suite LLVM project to GitHub?
On 2/24/16 4:25 PM, Matthias Braun via llvm-dev wrote:> I don't really care where the repository is located, but I do have > some comments on the future test-suite directions: > >> On Feb 24, 2016, at 12:57 PM, Chandler Carruth via llvm-dev >> <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote: >> >> Subject kinda says it all. Here is my rationale: >> >> The test-suite is really weird relative to the rest of the LLVM project: >> 1) It contains all manner of crazily licensed code. > That's indeed a good reason to move the repository away.I don't see that as a reason to move the repository. Where a repository lives and what format it uses is an orthogonal issue to what license the software within the repository is allowed to use. There are policies on what licenses can be used for Clang and LLVM code; the policy for the allowed licenses in test-suite is just different.> >> 2) We don't really care about the history at all. Any concerns around >> linear history or bisection are pretty much irrelevant. > We do care about the history. Sometimes benchmarks get fixed or > tweaked which may change the results, we should be able to dig into > the history to see what happened when. In any way retaining the > history wouldn't be a problem, would it?I think Chandler's point (Chandler, please correct me if I'm wrong) is that it's not important to a) match the test suite revision numbers to LLVM source code revision numbers and b) copy the SVN history to the Github repository. Regards, John Criswell -- John Criswell Assistant Professor Department of Computer Science, University of Rochester http://www.cs.rochester.edu/u/criswell -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160224/4d76bc3d/attachment.html>
Joerg Sonnenberger via llvm-dev
2016-Feb-24 21:43 UTC
[llvm-dev] RFC: Move the test-suite LLVM project to GitHub?
On Wed, Feb 24, 2016 at 02:20:24PM -0700, Jonathan Roelofs via llvm-dev wrote:> 2) There is the cap on total repository size, which is in the neighborhood > of 1Gb [3]. A fresh checkout of test-suite clocks in at just over 3Gb. This > one actually is a problem.This is not really true. Individual pack files must be below 1GB, but the total repository can be much larger. That's true even for the free tiers. That said, it might be good idea to split the repository into modules to keep it managable. Joerg
Joachim Durchholz via llvm-dev
2016-Feb-24 21:49 UTC
[llvm-dev] RFC: Move the test-suite LLVM project to GitHub?
Am 24.02.2016 um 22:20 schrieb Jonathan Roelofs via llvm-dev:> > There are two size limitations w.r.t. moving to github: > > 1) They have a prohibition on having individual files over 100M [1], > unless the repository is set up with their Git LFS plugin. This plugin > hasn't been around all that long, and is not something I would use in > production. (Not to mention the fact that it's not supported on public > forks anyway [2]) > > The largest individual files in the test-suite are around 10-12M, so > this isn't a problem yet, but it could become one later... something to > think about.I'd expect the LFS plugin to be mature before this becomes an issue.> 2) There is the cap on total repository size, which is in the > neighborhood of 1Gb [3]. A fresh checkout of test-suite clocks in at > just over 3Gb. This one actually is a problem.Talk to the GitHub folks. They say they will make exceptions if you can explain your needs. I wouldn't be surprised if GitHub went out of their way to get LLVM onto their boat. Regards, Jo.
Joachim Durchholz via llvm-dev
2016-Feb-24 22:01 UTC
[llvm-dev] RFC: Move the test-suite LLVM project to GitHub?
Am 24.02.2016 um 21:57 schrieb Chandler Carruth via llvm-dev:> For all of these reasons, and also because I'd like to see how well (or > rather, how poorly) a service like GitHub actually works for the project, > it seems like splitting the test-suite out of the current subversion > repository and moving it there is the right call.My experience from a few years of contributing and a bit of project setup: #1) GitHub works really well for public discussions of code changes (pull requests) and issues. #2) Labels are too roughly granular to be very useful. Don't expect to be doing request priorization etc. on that route. (This may change in the future.) #3) Contributors without commit rights need to set up a fork (really a "git clone") on GitHub, and commit to that before they can issue a pull request. This sounds easy enough in theory, but for your local work you pull and merge from "origin" and push to "fork", and then you go to the GH site and start the pull request, and once it's in you start cleaning up work branches both locally and on GitHub. It's a lot of clerical work, and some aspects of this all are easy to get wrong for a git newbie. I found that GitLab does #3 much better. People can push their work branches directly to the project repo. You can set up the master branch (or any other branch) to require committer rights, so you can protect the master branch and don't have to force contributors into their own repositories. The rest is roughly on par with GitHub. GitLab isn't as polished in all respects. Oh, and the GitLab server-side code is available as Open Source. So if you want to set up your own servers, you can do that. GitHub did not open source their server code. If Github goes down, the projects need to migrate to a different git hoster.> There is still plenty to figure out about how to manage this on github, but > before doing anything else I just wanted to shoot an email and see if folks > like this idea.Feel free to ask, I have been through some of the setup pain. Regards, Jo
Renato Golin via llvm-dev
2016-Feb-24 22:10 UTC
[llvm-dev] RFC: Move the test-suite LLVM project to GitHub?
On 24 February 2016 at 20:57, Chandler Carruth via llvm-dev <llvm-dev at lists.llvm.org> wrote:> The test-suite is really weird relative to the rest of the LLVM project: > 1) It contains all manner of crazily licensed code. > 2) We don't really care about the history at all. Any concerns around linear > history or bisection are pretty much irrelevant. > 3) We don't ever plan to have LLVM code move into or out from the test-suite > 4) Its already big, and really should be much bigger. We shouldn't have > incentives to keep stuff out of the test suite because of size, hosting > cost, or anything else.5) It could be used by other compilers / projects that are not LLVM related. 6) We could accept pull-requests from a much larger community 7) GitHub (or similar) can scale *A LOT* better than our infrastructure, probably even use CDNs etc.> There is still plenty to figure out about how to manage this on github, but > before doing anything else I just wanted to shoot an email and see if folks > like this idea.Maybe put LNT in there, too? Some downsides: * the separate administration of commit access for new developers, but should be pretty low cost * we'll have a full non-linear Git solution (no SVN behind) for some projects, thus branches, merges, etc will be harder to tag for releases. I'd also avoid going for a full GitHub model (few commit access, need pull request), since this will be different from the current LLVM model. But we can easily apply the current LLVM model to GitHub by any committer accepting pull-requests from the wider community, just like we commit for people without access today. Overall, seems like a good choice for me. +1. --renato
Hans Wennborg via llvm-dev
2016-Feb-24 22:21 UTC
[llvm-dev] RFC: Move the test-suite LLVM project to GitHub?
On Wed, Feb 24, 2016 at 12:57 PM, Chandler Carruth via llvm-dev <llvm-dev at lists.llvm.org> wrote:> Subject kinda says it all. Here is my rationale: > > The test-suite is really weird relative to the rest of the LLVM project: > 1) It contains all manner of crazily licensed code. > 2) We don't really care about the history at all. Any concerns around linear > history or bisection are pretty much irrelevant. > 3) We don't ever plan to have LLVM code move into or out from the test-suite > 4) Its already big, and really should be much bigger. We shouldn't have > incentives to keep stuff out of the test suite because of size, hosting > cost, or anything else.I don't think I've ever actually successfully run the test-suite and fully understood what I was doing. As you say, it is kind of a weird relative to the LLVM project, but wouldn't moving it out of the repository make it even more so? I wish it was instead more integrated with the project, more useful, and better understood. We branch, tag, and release test-suite as we do with the other modules as part of the release process. That's maybe not super important for most folks, but moving it to a separate repository would make that process more complicated. If the main motivation is 4), maybe we should consider moving the whole repository to something that scales better? Sorry if this is coming across as negative, but it just seems that the most natural place for the LLVM test-suite is with the rest of LLVM, so I don't see why we should move it without a good reason. - Hans
Joachim Durchholz via llvm-dev
2016-Feb-24 22:51 UTC
[llvm-dev] RFC: Move the test-suite LLVM project to GitHub?
Am 24.02.2016 um 23:21 schrieb Hans Wennborg via llvm-dev:> If the main motivation is 4), maybe we should consider moving the > whole repository to something that scales better?It might be useful as an experiment. If it goes well, the rest of LLVM could follow suit. If it does not go well, the test suite could be moved back. Sites like GitHub lower the entry barrier into project contribution. That would be *my* motives; I don't know the motives of the LLVM team. Regards, Jo
Chandler Carruth via llvm-dev
2016-Feb-25 02:18 UTC
[llvm-dev] RFC: Move the test-suite LLVM project to GitHub?
On Wed, Feb 24, 2016 at 1:26 PM John Criswell <jtcriswel at gmail.com> wrote:> Dear Chandler, > > First, can you articulate why you want to move the test suite to Github? > Is it taking up too much space, or is there some other problem that you're > trying to solve? I think you clearly explain why moving the revision > history isn't necessary, but it's not clear to me what problem you are > trying to solve. >Well, I tried in my original email, but perhaps I should state the issue more generally. The costs of us managing our own hosting of the test suite seem higher than for the rest of the project (size, scope, license diversity, etc), and yet the benefits of us managing our own hosting (compared to using a managed service like github) seem *much* lower. It will also make checking out the test suite, especially as it grows, substantially faster. And I really do think the test suite should grow, and grow a lot. I don't think we should *always* run all of it, I actually think having good, focused slices of the test suite is really important (this has come up elsewhere on the thread). But I think we should also be in the business of making it easier to get more testing for LLVM. And one way to do that would be to move to a faster and cheaper (in maintenance/support terms) solution such as using well known managed hosting like github. So ultimately, I guess I'm trying to clear a path for growth of the test suite (within reason) and reduce support burden on our common infrastructure. Neither are really pressing problems, but they both seem worth addressing.> Second, if we move the revision history to Github, it would be nice to > archive the existing Subversion history somewhere (e.g., leave it on > llvm.org but disable commit access to it). The test suite has been used > in numerous research papers, so keeping the revision history around is good > practice. We should only delete the Subversion revision history if keeping > it around is just too onerous. >Oh, I wouldn't want to delete it. Your re-interpretation was correct, I just mean that a strict, linear, correlated flow of history common to the test suite and the compiler doesn't seem important. Sorry for confusion, i'll follow up more on the history point on the relevant sub-thread.> > Third, I assume your plan is to continue to track changes on Github. Is > that correct? >Yep. I definitely wouldn't want to see any real changes to process here, just a different "master" so-to-speak. But this also gets to the "there would be a ton of stuff to figure out if this is the right direction" issue. =] So sorry for the hand waving.>-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160225/df3a7884/attachment.html>
Chandler Carruth via llvm-dev
2016-Feb-25 02:22 UTC
[llvm-dev] RFC: Move the test-suite LLVM project to GitHub?
On Wed, Feb 24, 2016 at 1:26 PM Matthias Braun <matze at braunis.de> wrote:> I don't really care where the repository is located, but I do have some > comments on the future test-suite directions: >Just as a meta-point, I don't want to conflate any of this with a specific design direction. I'm really focused on "where is it hosted" as a simplifying thing for the projects infrastructure.> > On Feb 24, 2016, at 12:57 PM, Chandler Carruth via llvm-dev < > llvm-dev at lists.llvm.org> wrote: > > Subject kinda says it all. Here is my rationale: > > The test-suite is really weird relative to the rest of the LLVM project: > 1) It contains all manner of crazily licensed code. > > That's indeed a good reason to move the repository away. > > 2) We don't really care about the history at all. Any concerns around > linear history or bisection are pretty much irrelevant. > > We do care about the history. Sometimes benchmarks get fixed or tweaked > which may change the results, we should be able to dig into the history to > see what happened when. In any way retaining the history wouldn't be a > problem, would it? >See John's response, and sorry for the broad statement. I meant, we don't care about a shared linear monotonic history with the rest of the compiler that we can bisect across simultaneously. Clearly we still want version control!> > 3) We don't ever plan to have LLVM code move into or out from the > test-suite > > I could actually see moving llvm code into the test-suite (we already use > lit code from llvm) but indeed move code out of the testsuite into llvm I > don't foresee happening. >Well, I think it might make sense to separate the LLVM code *used* by the test suite from the test suite itself. I'd be happy to keep that code in the LLVM repository to the extent possible (perhaps with expanded stuff under utils/...).> > 4) Its already big, and really should be much bigger. We shouldn't have > incentives to keep stuff out of the test suite because of size, hosting > cost, or anything else. > > I agree with the goal of having a big test-suite. However I think there is > a point where we should rather strive to have a stable base system for > building and running tests, etc. and then have the actual benchmarks/tests > being modules on top of that. We already have that situation today with > External/SPEC* and I think it would be a good idea to have a mode where you > just checkout more benchmarks into a test-suite subdirectory and they are > automatically recognized and used (in fact that is something on my TODO > list though at a very low position). >No argument to me about needed better organization, modularity and such. And we definitely need to have reasonably small slices that we *really* care about. I think its useful to the extent possible to provide a common repository so that folks don't have to aggregate too many things just so that we can have more productive discussions ("Well, where did you pull benchmark Whizzbang from? Oh, I have a different variant of it, so that's why I don't see that regression"). But none of that should argue against better modularity and extensibility in it -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160225/2d7b24bd/attachment-0001.html>
Chandler Carruth via llvm-dev
2016-Feb-25 02:25 UTC
[llvm-dev] RFC: Move the test-suite LLVM project to GitHub?
On Wed, Feb 24, 2016 at 2:01 PM Joachim Durchholz via llvm-dev < llvm-dev at lists.llvm.org> wrote:> Am 24.02.2016 um 21:57 schrieb Chandler Carruth via llvm-dev: > > For all of these reasons, and also because I'd like to see how well (or > > rather, how poorly) a service like GitHub actually works for the project, > > it seems like splitting the test-suite out of the current subversion > > repository and moving it there is the right call. > > My experience from a few years of contributing and a bit of project setup: >All of the stuff you mentioned is really great to know. I just want to clarify in case anyone is confused: I am in no way suggesting that we would use pull requests, issues, or anything else as part of the workflow for the test suite. I'm fine if folks want to talk about that later, but I really view it as a separate discussion. I'm much more interested in just leveraging the managed *hosting* of the repository.> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160225/6fe2aa8e/attachment.html>
Chandler Carruth via llvm-dev
2016-Feb-25 02:28 UTC
[llvm-dev] RFC: Move the test-suite LLVM project to GitHub?
On Wed, Feb 24, 2016 at 2:10 PM Renato Golin <renato.golin at linaro.org> wrote:> On 24 February 2016 at 20:57, Chandler Carruth via llvm-dev > <llvm-dev at lists.llvm.org> wrote: > > The test-suite is really weird relative to the rest of the LLVM project: > > 1) It contains all manner of crazily licensed code. > > 2) We don't really care about the history at all. Any concerns around > linear > > history or bisection are pretty much irrelevant. > > 3) We don't ever plan to have LLVM code move into or out from the > test-suite > > 4) Its already big, and really should be much bigger. We shouldn't have > > incentives to keep stuff out of the test suite because of size, hosting > > cost, or anything else. > > 5) It could be used by other compilers / projects that are not LLVM > related. > 6) We could accept pull-requests from a much larger community >To both of these, I would say "maybe?" -- I mostly want to wait and see if this is interesting. I feel like we need to sort out the hosting issues first.> 7) GitHub (or similar) can scale *A LOT* better than our > infrastructure, probably even use CDNs etc. >YES! =D> > > > There is still plenty to figure out about how to manage this on github, > but > > before doing anything else I just wanted to shoot an email and see if > folks > > like this idea. > > Maybe put LNT in there, too? >No idea, I'd wait to see how folks are liking github first and whether we have issues with it (process wise). Some downsides:> * the separate administration of commit access for new developers, > but should be pretty low cost > * we'll have a full non-linear Git solution (no SVN behind) for some > projects, thus branches, merges, etc will be harder to tag for > releases. >Yep.> > I'd also avoid going for a full GitHub model (few commit access, need > pull request), since this will be different from the current LLVM > model.I strongly agree. I don't want to see a process change here.> But we can easily apply the current LLVM model to GitHub by any > committer accepting pull-requests from the wider community, just like > we commit for people without access today. >Perhaps. I think this might still impose process changes, which is why I'd like to just look at the test suite right now where a lot of this is, honestly, simpler. Thanks for the great feedback! -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160225/df0326e9/attachment.html>
Chandler Carruth via llvm-dev
2016-Feb-25 02:32 UTC
[llvm-dev] RFC: Move the test-suite LLVM project to GitHub?
On Wed, Feb 24, 2016 at 2:21 PM Hans Wennborg <hans at chromium.org> wrote:> On Wed, Feb 24, 2016 at 12:57 PM, Chandler Carruth via llvm-dev > <llvm-dev at lists.llvm.org> wrote: > > Subject kinda says it all. Here is my rationale: > > > > The test-suite is really weird relative to the rest of the LLVM project: > > 1) It contains all manner of crazily licensed code. > > 2) We don't really care about the history at all. Any concerns around > linear > > history or bisection are pretty much irrelevant. > > 3) We don't ever plan to have LLVM code move into or out from the > test-suite > > 4) Its already big, and really should be much bigger. We shouldn't have > > incentives to keep stuff out of the test suite because of size, hosting > > cost, or anything else. > > I don't think I've ever actually successfully run the test-suite and > fully understood what I was doing. As you say, it is kind of a weird > relative to the LLVM project, but wouldn't moving it out of the > repository make it even more so? I wish it was instead more integrated > with the project, more useful, and better understood. > > We branch, tag, and release test-suite as we do with the other modules > as part of the release process. That's maybe not super important for > most folks, but moving it to a separate repository would make that > process more complicated. > > If the main motivation is 4), maybe we should consider moving the > whole repository to something that scales better? > > Sorry if this is coming across as negative, but it just seems that the > most natural place for the LLVM test-suite is with the rest of LLVM, > so I don't see why we should move it without a good reason. >I mean, I do agree with you about this being awkward in general. I would be pretty adamantly opposed to it for any other part of LLVM right now. I do think we should be looking at more scalable solutions for the whole repository, but: 1) That's a looooong process with lots of huge unknowns in its way. 2) The rest of the LLVM project is unlikely to want to grow sharply. So I would personally like to avoid the test suite forcing the issue. And I feel like while this is awkward, it is perhaps the least awkward for the test suite. It should at least buy us time, and may even give us information about what kinds of more scalable solutions we should be looking toward. -Chandler -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160225/24096642/attachment.html>
Yaron Keren via llvm-dev
2016-Feb-25 07:02 UTC
[llvm-dev] RFC: Move the test-suite LLVM project to GitHub?
If I understand correctly, for contributors without commit rights, the new patches "path" is 1) local work 2) local git 3) contributor github fork 4) official github fork 2016-02-25 0:01 GMT+02:00 Joachim Durchholz via llvm-dev < llvm-dev at lists.llvm.org>:> #3) Contributors without commit rights need to set up a fork (really a > "git clone") on GitHub, and commit to that before they can issue a pull > request. > This sounds easy enough in theory, but for your local work you pull and > merge from "origin" and push to "fork", and then you go to the GH site and > start the pull request, and once it's in you start cleaning up work > branches both locally and on GitHub. It's a lot of clerical work, and some > aspects of this all are easy to get wrong for a git newbie. >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160225/e4303ba3/attachment.html>
Kristof Beyls via llvm-dev
2016-Feb-25 08:48 UTC
[llvm-dev] RFC: Move the test-suite LLVM project to GitHub?
My biggest concerns and care-abouts are largely what Matthias expressed below. Most of them have been addressed already further down the thread. I hope that the move to github in practice would mean that the only difference is that I 'git clone https://github...' instead of 'git clone http://llvm.org/git/test-suite.git' ? Thanks, Kristof On 24/02/2016 22:25, Matthias Braun via llvm-dev wrote:> I don't really care where the repository is located, but I do have > some comments on the future test-suite directions: > >> On Feb 24, 2016, at 12:57 PM, Chandler Carruth via llvm-dev >> <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote: >> >> Subject kinda says it all. Here is my rationale: >> >> The test-suite is really weird relative to the rest of the LLVM project: >> 1) It contains all manner of crazily licensed code. > That's indeed a good reason to move the repository away. > >> 2) We don't really care about the history at all. Any concerns around >> linear history or bisection are pretty much irrelevant. > We do care about the history. Sometimes benchmarks get fixed or > tweaked which may change the results, we should be able to dig into > the history to see what happened when. In any way retaining the > history wouldn't be a problem, would it? > >> 3) We don't ever plan to have LLVM code move into or out from the >> test-suite > I could actually see moving llvm code into the test-suite (we already > use lit code from llvm) but indeed move code out of the testsuite into > llvm I don't foresee happening. > >> 4) Its already big, and really should be much bigger. We shouldn't >> have incentives to keep stuff out of the test suite because of size, >> hosting cost, or anything else. > I agree with the goal of having a big test-suite. However I think > there is a point where we should rather strive to have a stable base > system for building and running tests, etc. and then have the actual > benchmarks/tests being modules on top of that. We already have that > situation today with External/SPEC* and I think it would be a good > idea to have a mode where you just checkout more benchmarks into a > test-suite subdirectory and they are automatically recognized and used > (in fact that is something on my TODO list though at a very low position). > > - Matthias > >> >> For all of these reasons, and also because I'd like to see how well >> (or rather, how poorly) a service like GitHub actually works for the >> project, it seems like splitting the test-suite out of the current >> subversion repository and moving it there is the right call. >> >> When I chatted with folks on the board, this made sense to them as >> well, and I've made sure we have a reasonable LLVM organization set >> up on GitHub and all the board members are on it: >> https://github.com/llvm (I think only my membership is public at the >> moment). >> >> There is still plenty to figure out about how to manage this on >> github, but before doing anything else I just wanted to shoot an >> email and see if folks like this idea. >> >> Thanks! >> -Chandler >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> >> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160225/38d2e450/attachment.html>
Chandler Carruth via llvm-dev
2016-Feb-25 21:24 UTC
[llvm-dev] RFC: Move the test-suite LLVM project to GitHub?
Summing up the thread, It seems pretty clear that there is general interest in at least investigating this path. There are specific (reasonable) concerns about the mechanics of how it will work, which only really makes sense as we don't know how it will work yet! Next steps: 1) Actually push a clone of the test suite onto github and sort out any boring details of getting the hosting of it working[1]. 2) Figure out some initial practices (with enforcement if available) for keeping the github master clean. 3) Propose the mechanical "how will this work" with accompanying documentation update to the LLVM webpages in a new thread, and make sure folks are happy with the concrete mechanics at least as an initial cut 4) If and when everyone is happy with the mechanics, work to stop commits flowing into the svn repo, refresh the github copy to be a pristine mirror of it, and allow commits (using whatever mechanism) there. Naturally, the last step is going to be last, and has an enormous *IF* in front of it. =] No actual patches to it (other than testing this whole system) will happen on github, etc. I'm not going to be doing all of this myself, unsurprisingly. I think I know a few people who would be willing to help, but if other LLVM contributors are interested in helping with the #1 - #3 stuff, just lemme know and I'll try to make sure everyone has access to the github bits. At least up through #3 I'm not really worried about anything as its just an experiment. Sound about right? Any big issues? Thanks to everyone! -Chandler [1]: Just to be clear, my plan is to take existing git mirror and use that as the starting point, so we shouldn't lose any actual history here, etc. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160225/7ccfebb7/attachment.html>
Renato Golin via llvm-dev
2016-Feb-26 15:05 UTC
[llvm-dev] RFC: Move the test-suite LLVM project to GitHub?
On 25 February 2016 at 21:24, Chandler Carruth via llvm-dev <llvm-dev at lists.llvm.org> wrote:> 1) Actually push a clone of the test suite onto github and sort out any > boring details of getting the hosting of it working[1].Is this going to be a move or a copy with auto-update from the current git repo? The former is simpler in the long run, but may disrupt some people and automated processes.> 2) Figure out some initial practices (with enforcement if available) for > keeping the github master clean.Depends on how you doing step 1. Again, moving will make this trivial.> 3) Propose the mechanical "how will this work" with accompanying > documentation update to the LLVM webpages in a new thread, and make sure > folks are happy with the concrete mechanics at least as an initial cutIf you're moving, I'd think you should do this step first. If not, someone will have to bear the cost of updating until we get to a consensus.> Naturally, the last step is going to be last, and has an enormous *IF* in > front of it. =] No actual patches to it (other than testing this whole > system) will happen on github, etc.The test-suite repo is low-traffic enough that this will probably be an easy migration.> I'm not going to be doing all of this myself, unsurprisingly. I think I know > a few people who would be willing to help, but if other LLVM contributors > are interested in helping with the #1 - #3 stuff, just lemme know and I'll > try to make sure everyone has access to the github bits. At least up through > #3 I'm not really worried about anything as its just an experiment.This tells me you're not moving, but maintaining a separate copy. :) I'd be glad to help. My GitHub username is (unsurprisingly) rengolin. cheers, --renato