Hi,
I think this has been discussed previously on the list, so apologies if
I''m covering old ground but I''ve been using puppet to manage a
few Rails
applications which have dependencies on quite a few gems, so it''s
getting to be a bit of an onerous task to manage them. It looks like
the command line interface to gems doesn''t really support automated
installs. At the moment my options seem to be download specific gem
versions (which isn''t ideal for things like mongrel which have a few
dependencies) or run my own gem repository with *exactly* the right
versions of gems and their dependencies - are these my only options ?
There seems to be a recipe for managing mongrel
(http://www.reductivelabs.com/trac/puppet/wiki/RailsMongrelApacheProxybalancer)
but I can''t see how this can work as the mongrel gem prompts the user
for version/platform info ?
So as an example I''m using this to get mongrel :
class www-mongrel {
exec { "wget
http://rubyforge.org/frs/download.php/19878/rake-0.7.3.gem":
cwd => "/tmp/",
creates => "/usr/bin/rake"
}
exec { "gem install --local /tmp/rake-0.7.3.gem":
cwd => "/tmp/",
creates => "/usr/bin/rake"
}
exec { "wget
http://rubyforge.org/frs/download.php/16719/mongrel-1.0.1.gem":
cwd => "/tmp/",
creates => "/usr/bin/mongrel_rails"
}
exec { "wget
http://rubyforge.org/frs/download.php/18636/fastthread-1.0.gem":
cwd => "/tmp/",
creates =>
"/usr/lib/ruby/gems/1.8/gems/fastthread-1.0/ext/fastthread"
}
exec { "wget
http://rubyforge.org/frs/download.php/22543/daemons-1.0.7.gem":
cwd => "/tmp/",
creates =>
"/usr/lib/ruby/gems/1.8/gems/daemons-1.0.7/lib/daemons.rb"
}
exec { "wget
http://rubyforge.org/frs/download.php/16607/gem_plugin-0.2.2.gem":
cwd => "/tmp/",
creates =>
"/usr/lib/ruby/gems/1.8/gems/gem_plugin-0.2.2/bin/gpgen"
}
exec { "wget
http://rubyforge.org/frs/download.php/17197/cgi_multipart_eof_fix-2.1.gem":
cwd => "/tmp/",
creates =>
"/usr/lib/ruby/gems/1.8/gems/cgi_multipart_eof_fix-2.1/lib/cgi_multipart_eof_fix.rb"
}
exec { "gem install --local /tmp/*.gem":
cwd => "/tmp/",
creates => "/usr/bin/mongrel_rails"
}
exec { "gem install mongrel_cluster":
creates =>
"/usr/lib/ruby/gems/1.8/gems/mongrel_cluster-0.2.1/bin/mongrel_cluster_ctl"
}
exec { "wget
http://rubyforge.org/tracker/download.php/1306/5147/11098/2025/mongrel_rails_cleanup_stale_pid_files.diff":
cwd =>
''/usr/lib/ruby/gems/1.8/gems/mongrel-1.0/bin/'',
creates =>
"/usr/lib/ruby/gems/1.8/gems/mongrel-1.0/bin/mongrel_rails_cleanup_stale_pid_files.diff"
}
exec { "patch -t -p1 <
mongrel_rails_cleanup_stale_pid_files.diff":
cwd =>
''/usr/lib/ruby/gems/1.8/gems/mongrel-1.0/bin/'',
unless => "/bin/grep -Fq ''mongrels that crash can
leave stale
PID files behind''
''/usr/lib/ruby/gems/1.8/gems/mongrel-1.0/bin/mongrel_rails''",
}
file { "/etc/mongrel_cluster":
ensure => directory,
}
}
This seems like quite a large recipe to just install mongrel, has
anybody done this a better way (without the need for a local gem
repository) ?
Thanks,
Rob
On Jul 12, 2007, at 5:17 AM, robl wrote:> This seems like quite a large recipe to just install mongrel, has > anybody done this a better way (without the need for a local gem > repository) ?I''ll be building a set of modules to manage Rails this week, so I should be encountering and hopefully fixing this problem then. If you have any recommendations for how to fix it, I''d love to hear them. -- ''Tis better to be silent and be thought a fool, than to speak and remove all doubt. --Abraham Lincoln --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
> I''ll be building a set of modules to manage Rails this week, so I > should be encountering and hopefully fixing this problem then. > > If you have any recommendations for how to fix it, I''d love to hear > them. > >From what I''ve read, until the gem interface is updated, it''s pretty tough. The only thing I thought about doing was writing some form of expect scripting type for puppet that could manage the prompts. If you can fix it, I''d really appreciate it, and if you need a tester for modules then just shout.
On Jul 12, 2007, at 9:46 AM, robl wrote:> From what I''ve read, until the gem interface is updated, it''s pretty > tough. The only thing I thought about doing was writing some form of > expect scripting type for puppet that could manage the prompts. If > you > can fix it, I''d really appreciate it, and if you need a tester for > modules then just shout.Please lord, let me never have to write another expect-style script. Plus I think I''d need extra information from the user to make it work. I''m going to see if I can use internal APIs or something, I guess. -- We either are networking, or we areNT networking... --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
I''m looking forward to seeing the Rails-related recipes you come up
with,
Luke. I expect you already have a good idea of what you''re planning to
doand how, but if you want some feedback from a rails user who has just
started using puppet I''m glad to offer my thoughts/opinions.
Having just finished my initial take at using puppet to configure a fresh
install to run a rails app I''ll share a few things I learned:
1. Using puppet to install rubygems itself seems to be a problem. The
problem is that if rubygems is not already installed, then puppet believes
it cannot use the gem provider for package. My solution was to install
rubygems as part of my bootstrap script that installs puppet.
2. Instead of using config/mongrel_cluster.yml in my rails app (and in its
svn repo), i have capistrano create a symlink to the appropriate file in
/etc/mongrel_cluster. This is not quite ideal. I prefer configuring things
like the number of mongrel processors and the ports in puppet (which will
use this to template the correct values into my webserver config file). So,
for instance, when i have multiple app servers it makes more sense (to me)
to have puppet know how much each server can handle, rather than dealing
with this in capistrano. But, i would have preferred puppet not needing to
know which applications i might install.
3. I wrote a definition to install certain packages (gems and rpms) from my
puppet file store. the definition (i called it package_from_puppet) paired
a file{} and an exec{} to install the correct version.
4. I used this on a slicehost VPS with a minimal centos x86_64 install. I
would prefer to have ALL packages installed from my own local copies so i
can keep strict control over their versions. But, for now i only do this
for a few packages where versions were an issue, or where gem would not work
because it requires user input to select the version. If there were an easy
way to download a package and all its dependencies, and an easy way to
provide my own repository (for gem and yum) that would be great.
5. as for the DMZ issue (recently discussed in a thread on push vs pull), i
created an ssh tunnel from my local puppetmaster server to the client
machine, and manually run puppet there. Thus, my puppetmaster machine can
still be firewalled from the server tobe configured.
best
rich
On 7/12/07, Luke Kanies <luke@madstop.com> wrote:>
> On Jul 12, 2007, at 9:46 AM, robl wrote:
>
> > From what I''ve read, until the gem interface is updated,
it''s pretty
> > tough. The only thing I thought about doing was writing some form of
> > expect scripting type for puppet that could manage the prompts. If
> > you
> > can fix it, I''d really appreciate it, and if you need a
tester for
> > modules then just shout.
>
> Please lord, let me never have to write another expect-style script.
>
> Plus I think I''d need extra information from the user to make it
work.
>
> I''m going to see if I can use internal APIs or something, I guess.
>
> --
> We either are networking, or we areNT networking...
> ---------------------------------------------------------------------
> Luke Kanies | http://reductivelabs.com | http://madstop.com
>
>
> _______________________________________________
> Puppet-users mailing list
> Puppet-users@madstop.com
> https://mail.madstop.com/mailman/listinfo/puppet-users
>
_______________________________________________
Puppet-users mailing list
Puppet-users@madstop.com
https://mail.madstop.com/mailman/listinfo/puppet-users
> Please lord, let me never have to write another expect-style script. > > Plus I think I''d need extra information from the user to make it work. > > I''m going to see if I can use internal APIs or something, I guess. > >You''re right - I thought the expect script would be easier but it''s actually pretty trivial to install gems once you have rubygems installed ... #!/usr/bin/ruby require ''rubygems'' Gem::manage_gems PROPER_SOURCES = %w( http://gems.rubyforge.org ) Gem.clear_paths installer = Gem::RemoteInstaller.new p = installer.install(''kwalify'') puts p.inspect Of course this means we need to install rubygems somehow :)
On Jul 12, 2007, at 11:49 AM, robl wrote:>> > You''re right - I thought the expect script would be easier but it''s > actually pretty trivial to install gems once you have rubygems > installed ... > > #!/usr/bin/ruby > > require ''rubygems'' > Gem::manage_gems > > PROPER_SOURCES = %w( http://gems.rubyforge.org ) > > Gem.clear_paths > installer = Gem::RemoteInstaller.new > p = installer.install(''kwalify'') > > puts p.inspectThis solves your problem, installing the gem without asking you any annoying questions? I was just looking at the Gems API, if you could call it that. If you could get a chunk of code I can use to install gems instead of calling out, I''d apply it in a minute. -- If computers get too powerful, we can organize them into a committee -- that will do them in. -- Bradley''s Bromide --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
puppet-users-bounces@madstop.com wrote on 12/07/2007 16:40:52:> > I''m going to see if I can use internal APIs or something, I guess.Going to try avoiding the issue entirely by packaging the requisite gems as an rpm to install either at build or by puppet. This will be my task for tomorrow. I''d would have assumed that someone has have tried this before. If that''s the case and it''s actually impossible do, please, let me know. :-) Anyway, that would leave having to install a rubygems rpm (already built) just to satisfy puppet''s craving for bling. Puppetz ''n the ''hood, on a screen near you soon! This says it all really: lib/puppet.rb:1:# Try to load rubygems. Hey rubygems, I hate you. Matthew -- . ------------------------------------------------------------------ The Guardian Public Services Awards 2007, in partnership with Hays Public Services, recognise and reward outstanding performance from public, private and voluntary sector teams. To find out more and nominate a deserving team or individual, visit http://society.guardian.co.uk/publicservicesawards ------------------------------------------------------------------ Visit Guardian Unlimited - the UK''s most popular newspaper website http://guardian.co.uk http://observer.co.uk ------------------------------------------------------------------ The Newspaper Marketing Agency Opening Up Newspapers http://www.nmauk.co.uk ------------------------------------------------------------------ This e-mail and all attachments are confidential and may also be privileged. If you are not the named recipient, please notify the sender and delete the e-mail and all attachments immediately. Do not disclose the contents to another person. You may not use the information for any purpose, or store, or copy, it in any way. Guardian News & Media Limited is not liable for any computer viruses or other material transmitted with or as part of this e-mail. You should employ virus checking software. Guardian News & Media Limited A member of Guardian Media Group PLC Registered Office Number 1 Scott Place, Manchester M3 3GG Registered in England Number 908396
> This solves your problem, installing the gem without asking you any > annoying questions? > > I was just looking at the Gems API, if you could call it that. > > If you could get a chunk of code I can use to install gems instead of > calling out, I''d apply it in a minute. > >I thought this code would enable us to do that but .... For reference have a look at http://rubyfurnace.com/docs/rubygems-update-0.9.3/classes/Gem/RemoteInstaller.html and /usr/local/lib/site_ruby/1.8/rubygems/remote_installer.rb. I thought this would be a programmatic way to do it but it turns out that ruby gems really are appalling, when you look at remote_installer.rb around line 508 # only non-binary gems...return latest return specs_n_sources.first if non_binary_gems.empty? list = specs_n_sources.collect { |item| "#{item[0].name} #{item[0].version} (#{item[0].platform.to_s})" } list << "Cancel installation" string, index = choose_from_list( "Select which gem to install for your platform (#{RUBY_PLATFORM})", list) If it''s a binary gem then it *forces* you to choose from a list ... I think it would be pretty easy to patch the library to support autodetection of platform and return the latest gem but I assume there''s a good reason why they haven''t done it. Back to the drawing board ? (or expect ?!)
On Jul 12, 2007, at 12:30 PM, matthew.malthouse@guardian.co.uk wrote:> This says it all really: > > lib/puppet.rb:1:# Try to load rubygems. Hey rubygems, I hate > you.Ayup. -- A bore is a man who deprives you of solitude without providing you with company. -- Gian Vincenzo Gravina --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On Jul 12, 2007, at 12:37 PM, robl wrote:> If it''s a binary gem then it *forces* you to choose from a list ... I > think it would be pretty easy to patch the library to support > autodetection of platform and return the latest gem but I assume > there''s > a good reason why they haven''t done it.Your assumption seems to show that you are somehow immune to cynicism or that you just haven''t interacted enough with RubyGems. "Why is this all so horrible?" "It''s not horrible, it''s exactly how we like it." -- I think that''s how Chicago got started. A bunch of people in New York said, ''Gee, I''m enjoying the crime and the poverty, but it just isn''t cold enough. Let''s go west.'' --Richard Jeni --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
> Your assumption seems to show that you are somehow immune to cynicism > or that you just haven''t interacted enough with RubyGems. > > "Why is this all so horrible?" "It''s not horrible, it''s exactly how > we like it." > >I''ll try sending a patch over to them :)
robl wrote:>> Your assumption seems to show that you are somehow immune to cynicism >> or that you just haven''t interacted enough with RubyGems. >> >> "Why is this all so horrible?" "It''s not horrible, it''s exactly how >> we like it." >> >> >> > I''ll try sending a patch over to them :) > _______________________________________________ >To wrap this one up for now, there are a few patches already submitted to rubygems, specifically around these issues http://rubyforge.org/pipermail/rubygems-developers/2007-June/002858.html http://rubyforge.org/tracker/index.php?func=detail&aid=8643&group_id=126&atid=577 http://rubyforge.org/tracker/index.php?func=detail&aid=10660&group_id=126&atid=577 These combined would probably solve the issue, but for now I think it''s a tricky one to solve.
On Jul 12, 2007, at 1:32 PM, robl wrote:> To wrap this one up for now, there are a few patches already submitted > to rubygems, specifically around these issues > > http://rubyforge.org/pipermail/rubygems-developers/2007-June/ > 002858.html > http://rubyforge.org/tracker/index.php? > func=detail&aid=8643&group_id=126&atid=577 > http://rubyforge.org/tracker/index.php? > func=detail&aid=10660&group_id=126&atid=577 > > These combined would probably solve the issue, but for now I think > it''s > a tricky one to solve.Patched submitted in february, no response from the devs. Yeah, I at least saw that patch, and the lack of action didn''t exactly give me hope. -- I do not feel obliged to believe that the same God who has endowed us with sense, reason, and intellect has intended us to forgo their use. -- Galileo Galilei --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On Thu, 2007-07-12 at 18:30 +0100, matthew.malthouse@guardian.co.uk wrote:> puppet-users-bounces@madstop.com wrote on 12/07/2007 16:40:52: > > > > I''m going to see if I can use internal APIs or something, I guess. > > Going to try avoiding the issue entirely by packaging the requisite gems > as an rpm to install either at build or by puppet. This will be my task > for tomorrow. > > I''d would have assumed that someone has have tried this before. If that''s > the case and it''s actually impossible do, please, let me know. :-)I have .. and it''s not that hard. I wrote a little tool that will spit out a spec file[1] - let me know if that works for you. What distros are you building rpm''s for ? Submitting rails rpm''s to fedora has been on my todo list for quite some time now :( David [1] http://rubyforge.org/projects/gem2rpm/
On Jul 12, 2007, at 11:00 AM, Rich Liebling wrote:> I''m looking forward to seeing the Rails-related recipes you come up > with, Luke. I expect you already have a good idea of what you''re > planning to doand how, but if you want some feedback from a rails > user who has just started using puppet I''m glad to offer my > thoughts/opinions.I''m actually having to teach myself quite a bit about Rails deployment... I should be publishing all of the modules I''m creating as soon as they''re done. My two goals for the weekend are to finish these modules and write a simple Rails app to make it easy to publish modules (and, of course, use the modules to build the app and then publish them using it -- talk about self-reference).> Having just finished my initial take at using puppet to configure a > fresh install to run a rails app I''ll share a few things I learned: > > 1. Using puppet to install rubygems itself seems to be a problem. > The problem is that if rubygems is not already installed, then > puppet believes it cannot use the gem provider for package. My > solution was to install rubygems as part of my bootstrap script > that installs puppet.I hope to soon fix this problem, but in the meantime, yeah, you either have to have two runs or make sure rubygems are installed beforehand.> 2. Instead of using config/mongrel_cluster.yml in my rails app > (and in its svn repo), i have capistrano create a symlink to the > appropriate file in /etc/mongrel_cluster. This is not quite > ideal. I prefer configuring things like the number of mongrel > processors and the ports in puppet (which will use this to template > the correct values into my webserver config file). So, for > instance, when i have multiple app servers it makes more sense (to > me) to have puppet know how much each server can handle, rather > than dealing with this in capistrano. But, i would have preferred > puppet not needing to know which applications i might install.This is where I''m at right now -- trying to figure out which files need to actually exist and what they should contain.> 3. I wrote a definition to install certain packages (gems and > rpms) from my puppet file store. the definition (i called it > package_from_puppet) paired a file{} and an exec{} to install the > correct version.I''ve got a ''brokengem'' definition that does this, but I''m sticking to yum for rpms. At least, for this project.> 4. I used this on a slicehost VPS with a minimal centos x86_64 > install. I would prefer to have ALL packages installed from my own > local copies so i can keep strict control over their versions. > But, for now i only do this for a few packages where versions were > an issue, or where gem would not work because it requires user > input to select the version. If there were an easy way to download > a package and all its dependencies, and an easy way to provide my > own repository (for gem and yum) that would be great.Yeah, I don''t know the best solution here. It''s definitely a question of local policy, but I haven''t seen enough implementations to have a good feel for how to solve these problems.> 5. as for the DMZ issue (recently discussed in a thread on push vs > pull), i created an ssh tunnel from my local puppetmaster server to > the client machine, and manually run puppet there. Thus, my > puppetmaster machine can still be firewalled from the server tobe > configured.Thankfully, that''s not an issue here, but it''s something I expect to be available soon. -- The most incomprehensible thing about the world is that it is at all comprehensible. --Albert Einstein --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com