Hey guys and gals, I suggested that perhaps we should use Integrity (http://github.com/foca/integrity) for the CI for Rails rather than CC.rb. DHH said he was all for it, but I should bring it before you all before he made a decision. So, here''s why I like Integrity: 1. It needs less hacking to do interesting things. For example, multiruby support would be dead simple to setup, which is something we really need. 2. Adding different sorts of notifiers to it is really easy. Being able to dump CI reports into IRC, Campfire, etc. without a lot of hassle would be awesome. 3. It''s really lightweight. There are a number of other reasons I prefer it, but these hit the main technical ones. The others are mostly opinion. :) What do you all think? --Jeremy -- http://jeremymcanally.com/ http://entp.com/ http://omgbloglol.com My books: http://manning.com/mcanally/ http://humblelittlerubybook.com/ (FREE!) --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. To post to this group, send email to rubyonrails-core@googlegroups.com To unsubscribe from this group, send email to rubyonrails-core+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=en -~----------~----~----~----~------~----~------~--~---
> Hey guys and gals, > I suggested that perhaps we should use Integrity > (http://github.com/foca/integrity) for the CI for Rails rather than > CC.rb. DHH said he was all for it, but I should bring it before you > all before he made a decision.Almost all the work on the CI server to dahas been done by chad woolley, so from my perspective the decision is basically completely up to him :).> So, here''s why I like Integrity: > 1. It needs less hacking to do interesting things. For example, > multiruby support would be dead simple to setup, which is something we > really need.I believe the multiruby config stuff is currently underway for our CI setup. Perhaps chad or mike could ump in here and compare the configuration difficulty when switching?> 2. Adding different sorts of notifiers to it is really easy. Being > able to dump CI reports into IRC, Campfire, etc. without a lot of > hassle would be awesome.Indeed, this sounds great.> 3. It''s really lightweight.Does it scale? j/k> There are a number of other reasons I prefer it, but these hit the > main technical ones. The others are mostly opinion. :) > > What do you all think?Basically the guys doing the work should have the most say in what happens. From my perspective the CI box is something that needs to email the core-list when things stop working, and the configuration and build scripts need to live in git alongside the rest of the code. Everything else beyond that is gravy! -- Cheers Koz --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. To post to this group, send email to rubyonrails-core@googlegroups.com To unsubscribe from this group, send email to rubyonrails-core+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=en -~----------~----~----~----~------~----~------~--~---
On Tue, Jan 13, 2009 at 9:05 PM, Jeremy McAnally <jeremymcanally@gmail.com> wrote:> > Hey guys and gals, > I suggested that perhaps we should use Integrity > (http://github.com/foca/integrity) for the CI for Rails rather than > CC.rb. DHH said he was all for it, but I should bring it before you > all before he made a decision.I''m open minded. As Koz said, I''ve done most of the work myself, with some help from Mike Gunderloy. We (my company) use ccrb mostly by default, because it used to be the only mature, fully-featured CI server in Ruby. Apparently that isn''t the case anymore, which is a Good Thing! If using Integrity means that more people will be on board with supporting CI (which primarily means making sure Rails Core applies patches to keep it green), then I''m in favor of it. There are a few things that ccrb does have going for it that I wouldn''t want to lose, though - I''ll mention those later. I also wrote Cinabox (http://github.com/thewoolleyman/cinabox/tree/master) which makes it almost a complete no-brainer to set up ccrb on a clean Ubuntu distro (just two scripts and everything is running, including a working init script)> > So, here''s why I like Integrity: > 1. It needs less hacking to do interesting things. For example, > multiruby support would be dead simple to setup, which is something we > really need.Is this really any easier than ccrb? Once you have the multiple interpreters installed (with multiruby or something like cinabox''s bootstrap_ruby script, that''s another discussion), most of the work is kicking off the build with the proper environment. As Integrity''s homepage says: "as long as your build process can be run from an unix-y environment and it returns a zero status code for success and non-zero for failure, then integrity works for you". For the rails build, this is just ci_build.rb (http://github.com/rails/rails/blob/9bcf01b23c25e640da7908ac8b1b49fbf7d2e51a/ci/ci_build.rb) - it doesn''t matter what CI tool you use to invoke it, as long as you can pass some env or param to indicate the interpreter to use. I''ve hacked ccrb build scripts to dynamically take this from the ccrb project name via env var, which is a slick way to do it. That (plus GemInstaller) is how I handle dynamically building against multiple rails gem versions without modifying any source code or re-freezing.> 2. Adding different sorts of notifiers to it is really easy. Being > able to dump CI reports into IRC, Campfire, etc. without a lot of > hassle would be awesome.CCRB is also fairly extensible, and it''s all Ruby, so it is a wash in terms of feasability. However, if Integrity already has all these working and mostly bug-free, then that is a plus.> 3. It''s really lightweight.This is actually pretty compelling. CCRB is in rails, and it loads the rails env for each builder, which is a significant memory overhead on low-memory servers running several builds.> > There are a number of other reasons I prefer it, but these hit the > main technical ones. The others are mostly opinion. :) > > What do you all think?Here''s the things that I''d like to confirm before I commit to it: * A working debian-based init script that is shipped with the project. By working, I mean it works seamlessly on reboots, on killed servers (no stale pids - e.g. mongrel cluster not mongrel), and standard start-stop-restart. Cinabox + ccrb handles all this automatically - e.g. just run setup_ci.rb and all the init script is working. Full disclosure, the stale pid problem still isn''t fixed in ccrb''s init script if the mongrel is killed. * Updated docs in rails (http://github.com/rails/rails/blob/9bcf01b23c25e640da7908ac8b1b49fbf7d2e51a/ci/ci_setup_notes.txt) that explain exactly how to set up integrity. Preferably, this will be by adding an "integrity" flavor to cinabox - currently the ruby/ccrb server setup in the previous doc is one line: "Use cinabox to perform rest of ruby/ccrb setup:", which consists of naught but downloading cinabox and running two command line scripts. * Git support: Is git support rock solid? How about for multiple non-master branches? Because it took a lot of hacking from various people to get all the known bugs out of ccrb''s git support. * How about git submodules/piston/braid/giternal? Specifically, can it automatically trigger a build based on changes in a submodule/pistonized/braidified/giternal? This is completely unrelated to the Rails build, but something that would make me really want to switch to Integrity for my other CI projects :) * I want to be disciplined about having EVERYTHING repeatable on a clean Ubuntu server via /ci/ci_setup_notes.txt. In other words, I DON''T want to take the approach of hacking and manually installing crap on the CI server and nobody knows what is running or how. I want to be able to completely blow away the server and have it back up and running by following the ci_setup_notes.txt exactly. This lets other people reproduce the rails CI environment on their own, which is critical to being able to reproduce problems that "only" occur on CI (which really means all the edge case tests, gem versions, databases, and libraries that aren''t even running on your dev box but are on CI). It also (by leveraging GemInstaller and having a solid init script) mostly eliminates the need for people to manually log on to the server to debug problems - everything should be reproducible in an ubuntu VM. * all the other mature CI features that ccrb has - reporting of scm failures, changesets, build failures, control over custom build artifacts, history of build output and artifacts, etc... So, I''m totally open to try it given the above caveats. If you want to do it, get everything set up on a fork of rails, and we''ll set them up to run concurrently for a while on ci.rubyonrails.org (following the diff of your forked /ci/ci_setup_notes.txt exactly) at a different nginx virtual host. If Integrity runs smoothly and does everything ccrb does and more, then why not? -- Chad --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. To post to this group, send email to rubyonrails-core@googlegroups.com To unsubscribe from this group, send email to rubyonrails-core+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=en -~----------~----~----~----~------~----~------~--~---
I''m going to chime in mostly with "what Chad said." I got interested in the CI story for Rails itself a couple of weeks ago because I simply could not get all the tests to pass on my own dev box, and thought this was due to my own stupidity. It didn''t take me long to find out "oh, the tests haven''t all passed in ages." This struck me as a ridiculous state of affairs, and (thanks of course to many people) we are now at a point where both the master and 2.2 branches are consistently building green on ci.rubyonrails.org. To my mind, moving forward the most important thing is that we have a solid CI server with a reproducible setup that is as painless as possible. It also needs to build in a way that most Rails developers do not. Some things to think about: - Testing against multiple branches of the source. - There are tests in the Rails source that do not run if you don''t have particular gems installed. Have *you* tested FCGI support lately, or SQLite2? The CI box consistently hits a large number of the test paths (though there''s always room for improvement: for example, we don''t have an Oracle or SQL Server testing strategy yet). This is going to get more tedious as we move forward with JRuby support (and the JRuby guys are committed to quickly fixing any bugs we expose): installed any JDBC-based AR adapters lately? - There are test paths that are mutually incompatible: for example, testing AR changes against PostgreSQL with both the pg library and the postgres library (this just bit us last week). The CI box can handle this, while most developers are unlikely to shuffle gems to get more thorough tests. - Testing against multiple ruby implementations (we really need matrix tests of n branches by m ruby implementations). This is not yet rolled out to ci.rubyonrails.org, but Chad and I have been tossing back and forth ideas about how to roll that in to the current painless setup, and we''re close. (Meanwhile, if you''re curious, I''m doing some experimental builds on my own clone of the CI server: on master, we''ve got 1 test failing against MRI 1.9.1RC1 [and a patch for that is just waiting to be committed] and 40 tests failing against JRuby trunk). As Chad said, some of Integrity''s benefits look interesting, especially the ability to consume less resources. I think multiruby is a 90% solved problem for our current setup; it''s just a matter of finding time to finish rolling it out. And other notifiers do exist for cc.rb, though frankly, its source tree is a bit of a mess. So, bottom line for me: Rails needs a solid CI process, because the current tree is very close to untestable by individual developers. The existing cc.rb solution can serve as that process. Integrity *may* be able to replace cc.rb with superior awesomeness, but I''d like to see someone set up a proof of concept with repro instructions. If anyone goes down that path, I''ll be happy to lend a hand by testing and comparing. Mike --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. To post to this group, send email to rubyonrails-core@googlegroups.com To unsubscribe from this group, send email to rubyonrails-core+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=en -~----------~----~----~----~------~----~------~--~---
On Wed, Jan 14, 2009 at 12:43, Mike Gunderloy <larkware@gmail.com> wrote:> > Integrity *may* be > able to replace cc.rb with superior awesomeness, but I''d like to see > someone set up a proof of concept with repro instructions. If anyone > goes down that path, I''ll be happy to lend a hand by testing and > comparing.I''ve been hacking on Integrity in our company. It''s very nice, but somewhat vanilla in its current state. I''d like to see Rails CI powered by Integrity one day, but from what I''ve seen now is not the time. Setting up everything that Chad and Mike listed would take much more effort than sticking with and improving the current system. But with this many forks and people interested in Integrity, after some time (and our contributions to it) I''d like this idea to be revisited. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. To post to this group, send email to rubyonrails-core@googlegroups.com To unsubscribe from this group, send email to rubyonrails-core+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=en -~----------~----~----~----~------~----~------~--~---
We've been looking at switching to integrity as well. One of the major benefits I see at the moment is that integrity would likely be more willing to accept features. On a separate note, I think it would be beneficial to have an easy, built in way to run YOUR app against edge or latest of a given branch (like 2.2-stable). I think this would do 2 things: 1) Encourage more people to have their apps in CI 2) Cause people to see changes in the core that affect their apps. This would get more, and more immediate feedback to the core team about commits. -Mike Gaffney On Wed, Jan 14, 2009 at 6:19 AM, Mislav Marohnić <mislav.marohnic@gmail.com>wrote:> On Wed, Jan 14, 2009 at 12:43, Mike Gunderloy <larkware@gmail.com> wrote: > >> >> Integrity *may* be >> able to replace cc.rb with superior awesomeness, but I'd like to see >> someone set up a proof of concept with repro instructions. If anyone >> goes down that path, I'll be happy to lend a hand by testing and >> comparing. > > > I've been hacking on Integrity in our company. It's very nice, but somewhat > vanilla in its current state. I'd like to see Rails CI powered by Integrity > one day, but from what I've seen now is not the time. Setting up everything > that Chad and Mike listed would take much more effort than sticking with and > improving the current system. > > But with this many forks and people interested in Integrity, after some > time (and our contributions to it) I'd like this idea to be revisited. > > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. To post to this group, send email to rubyonrails-core@googlegroups.com To unsubscribe from this group, send email to rubyonrails-core+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=en -~----------~----~----~----~------~----~------~--~---
2009/1/14 Mike Gaffney <mr.gaffo@gmail.com>:> We''ve been looking at switching to integrity as well. One of the major > benefits I see at the moment is that integrity would likely be more willing > to accept features.We are running the ci.rubyonrails.org off my ccrb GitHub branch, and it''s possible that ThoughtWorks would add some committers. I don''t think that''s a big concern long term.> > On a separate note, I think it would be beneficial to have an easy, built in > way to run YOUR app against edge or latest of a given branch (like > 2.2-stable). I think this would do 2 things: > 1) Encourage more people to have their apps in CI > 2) Cause people to see changes in the core that affect their apps. This > would get more, and more immediate feedback to the core team about commits.Yep, this would be great, and I''ve put a lot of thought into how to do this. We (my employer) already do this on our internal projects - we have the ability to automatically run against multiple rails versions or edge without modifying source/unfreezing/refreezing/etc. As I mentioned before, this can be as easy as creating an new ccrb project with the desired rails version in the ccrb project name - e.g: ''rails_rel_2.1.0'', ''rails_rel_2.2.0'', ''rails_rel_edge'' (although we don''t do edge yet). You can hack the cruise.rake to incorporate this into the build, and leverage GemInstaller to activate the appropriate rails gem via config/preinitializer.rb. Here''s a live example where I use this approach [1] to test GemInstaller against multiple RubyGems versions [2]. Here is a gist showing the same approach for a rails app [3]. Note again that there would be no editing/freezing/unfreezing/installing/uninstalling involved to continuously test against multiple versions - running CI against a new version is a simple as creating the ccrb project with the appropriate name. Even cooler would be extending this to plugins and gems - ensuring that your app works against the edge of not only rails, but all the plugins and gems you are using. That way, you can be very proactive about dealing with breaking changes (or sending them patches which do things in a different way that doesn''t break you). However, this involves a much more involved infrastructure of automatically building gems whenever source is changed, which is also a great place to leverage CI. I''ve discussed this some with Koz in the past - automatically building a nightly rails gem which would facilitate the approach I mentioned above. You could use git submodules/piston/braid/giternal to accomplish the same thing with frozen edge rails/plugins, which is why I asked how Integrity handled this. [1] http://geminstaller.rubyforge.org/svn/trunk/cruise_config.rb [2] http://ci.thewoolleyweb.com/ [3] http://gist.github.com/47004 Thanks, -- Chad --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. To post to this group, send email to rubyonrails-core@googlegroups.com To unsubscribe from this group, send email to rubyonrails-core+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=en -~----------~----~----~----~------~----~------~--~---
On Wed, Jan 14, 2009 at 12:13 PM, Chad Woolley <thewoolleyman@gmail.com> wrote:> 2009/1/14 Mike Gaffney <mr.gaffo@gmail.com>: >> On a separate note, I think it would be beneficial to have an easy, built in >> way to run YOUR app against edge or latest of a given branch (like >> 2.2-stable). > > As I mentioned before, this can be as easy as creating an new ccrb > project with the desired rails version in the ccrb project nameI''d like to put up an open example of what I''m talking about. Does anyone have suggestions for a Rails app that is: * open source on github * has a fast well-written test suite * currently has green tests against multiple rails versions and edge * would be willing to accept patches to accomplish the goals mentioned above Thanks, -- Chad --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. To post to this group, send email to rubyonrails-core@googlegroups.com To unsubscribe from this group, send email to rubyonrails-core+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=en -~----------~----~----~----~------~----~------~--~---
Hey all, sorry for the late response :) So, there''s a lot of points to cover, let''s go one by one. One note though, I''m currently refactoring a large chunk of the internals (wrt background building, mostly, but also cleaning up a bit the database schema). I wouldn''t propose we use integrity in any "serious" environment until this is done. Basically, I agree with Mislav that the current state it''s probably not something I''d recommend installing--specially if we have a working environment already (even if it''s using CC.rb :P). (Integrity works fine, but it''s just a bit rough around some edges -- like blocking while a project is building) *** Script to setup an ubuntu box. This looks like the biggest roadblock for integrity adoption in ci.rubyonrails.org. However, since I''m not a big fan of ubuntu I don''t use ubuntu much, however, so I''m not sure I''m the correct person to attempt this. Any help is greatly appreciated, though :) *** multiruby As far as I understand, using multiruby is just calling a different command from rake, and not much more than that, this would work out of the box with integrity, without any hacking or modification, since it''s just changing the command you run on the shell to build the project. *** git support Part of the current refactoring is easing out the tracking of multiple branches for a given project. (Right now you can go and create a project in integrity for each branch, which works fine, but is a bit clunky, specially if you have many non-master branches.) Other than that, git support is rock solid. *** Triggering builds on submodule updates Integrity will trigger a build on any commit to your repository. This should include changing submodules, or changing braid config files. I don''t know about piston or giternal, but I guess it should work fine with those too. *** Build artifacts This is something we don''t support ATM, though we''ve been going back and forth about it on #integrity and we''ll support something along the lines of this. *** Building against all versions of rails Has anyone tried http://github.com/ianwhite/garlic/tree/master? It looks interesting. Other than that, integrity doesn''t provide any magic for this, it just shells out and calls a user-provided script, recording the output and the exit code. If you want to go crazy in the script and uninstall gems, install other versions and run tests again (like the issue with both postgres gems), you can, and it''s trivial. (Something like https://gist.github.com/962e6ccd1e0389265e70) One of the ideas for integrity is to have multiple build agents running in external servers, so you could potentially have a box setup to test postgres, a box setup to test mysql, and each of them just builds the necessary parts and integrity will aggregate the reports. This is not the highest priority right now, but if you want/need this more than having a complicated build script and want to help out, then jump into #integrity and let''s discuss how this can be done :) --- So, to sum up. After the current refactoring takes place (which I expect to finish within the next week), I''ll be more than glad to help you set it up. And regarding the debian provisioning script, I would gladly accept any help on that front :) What do you guys think? -foca --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. To post to this group, send email to rubyonrails-core@googlegroups.com To unsubscribe from this group, send email to rubyonrails-core+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=en -~----------~----~----~----~------~----~------~--~---
One note on this: we probably *don''t* want to just run the tests with multiruby. Instead, we want to have multiple versions of ruby installed, and run the tests separately with each version. That makes it much easier to spot whether we have failures on a particular ruby implementation. multiruby may have a place in helping install the versions in the first place, though I suspect it''s overkill for that. Mike On Jan 15, 2009, at 9:49 AM, Nicolas Sanguinetti wrote:> > > *** multiruby > > As far as I understand, using multiruby is just calling a different > command from rake, and not much more than that, this would work out of > the box with integrity, without any hacking or modification, since > it''s just changing the command you run on the shell to build the > project. > > *** git support--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. To post to this group, send email to rubyonrails-core@googlegroups.com To unsubscribe from this group, send email to rubyonrails-core+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=en -~----------~----~----~----~------~----~------~--~---
On Thu, Jan 15, 2009 at 11:29 AM, Mike Gunderloy <larkware@gmail.com> wrote:> > One note on this: we probably *don''t* want to just run the tests with > multiruby. Instead, we want to have multiple versions of ruby > installed, and run the tests separately with each version. That makes > it much easier to spot whether we have failures on a particular ruby > implementation. > > multiruby may have a place in helping install the versions in the > first place, though I suspect it''s overkill for that. > > Mike >Hi Mike Can you expand on this? I''ve been getting multiruby setup and figuring out ways to let developers use it easily on runcoderun.com (our hosted CI product), and it seems like the best option right now for testing against multiple ruby versions. Multiruby does pretty much exactly what you want -- it runs the suite against each version of Ruby, and will tell you which ruby versions pass and which fail. The pain really comes in when you start dealing with gems, and having to reinstall gems (at least those with native dependencies) for each ruby version. For example, you won''t be able to really test Rails against 1.9 until all the database drivers compile and install cleanly against Ruby 1.9. The gem thing gets messy no matter what approach you take, but I think multiruby is a little cleaner then having multiple ruby''s and gem repos installed, and switching things around in the environment or via symlinks for each version. - Rob http://thinkrelevance.com http://runcoderun.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. To post to this group, send email to rubyonrails-core@googlegroups.com To unsubscribe from this group, send email to rubyonrails-core+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=en -~----------~----~----~----~------~----~------~--~---
Here''s the problem: if we use the multiruby command, we end up with one pass-or-fail for "Rails across all installed versions of ruby". Instead, we need separate cc.rb projects (or integrity or runcoderun or whatever) so we can quickly see "rails passed on ruby 1.8.7" in one project and "rails failed on ruby 1.9.1rc1" in another - much more useful, especially when we''ve got build targets (like JRuby) where we know things are going to fail for a while. Red for JRuby shouldn''t mean red for everything. Mike On Jan 20, 2009, at 10:30 AM, Rob Sanheim wrote:> > On Thu, Jan 15, 2009 at 11:29 AM, Mike Gunderloy > <larkware@gmail.com> wrote: >> >> One note on this: we probably *don''t* want to just run the tests with >> multiruby. Instead, we want to have multiple versions of ruby >> installed, and run the tests separately with each version. That makes >> it much easier to spot whether we have failures on a particular ruby >> implementation. >> >> multiruby may have a place in helping install the versions in the >> first place, though I suspect it''s overkill for that. >> >> Mike >> > > Hi Mike > > Can you expand on this? I''ve been getting multiruby setup and > figuring out ways to let developers use it easily on runcoderun.com > (our hosted CI product), and it seems like the best option right now > for testing against multiple ruby versions. Multiruby does pretty > much exactly what you want -- it runs the suite against each version > of Ruby, and will tell you which ruby versions pass and which fail. >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. To post to this group, send email to rubyonrails-core@googlegroups.com To unsubscribe from this group, send email to rubyonrails-core+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=en -~----------~----~----~----~------~----~------~--~---