Puppet users - Jan 2012 - best practices for caching packages on EC2

While Puppet allows us to manage a large number of instances on EC2,
what are the best practices for caching packages on EC2??

Instead of manually configuring yum or apt proxies on each node, it
would save us lots of time if caching could be "somehow" handled by
Puppet. Are there any plans to add the ability to handle caching in
the package providers??

Rayson

================================Open Grid Scheduler / Grid Engine
http://gridscheduler.sourceforge.net/

Scalable Grid Engine Support Program
http://www.scalablelogic.com/

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Mon, Jan 23, 2012 at 1:21 PM, Rayson Ho <raysonlogin@gmail.com>
wrote:
> While Puppet allows us to manage a large number of instances on EC2,
> what are the best practices for caching packages on EC2??
>
> Instead of manually configuring yum or apt proxies on each node, it
> would save us lots of time if caching could be "somehow" handled
by
> Puppet. Are there any plans to add the ability to handle caching in
> the package providers??
How would Puppet handle it?  It would still have to either cache the
package locally (eating lots of disk space) or point to a proxy, which
you can do with Puppet already.  Your simplest solution might be to
look for a mirror hosted in EC2, which I assume there are some.

Another option would be to build up a squid proxy via puppet and then
configure a client-side usage of that cache, but that''s no different
in EC2 vs any other setup.




>
> Rayson
>
> ================================> Open Grid Scheduler / Grid Engine
> http://gridscheduler.sourceforge.net/
>
> Scalable Grid Engine Support Program
> http://www.scalablelogic.com/
>
> --
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Mon, Jan 23, 2012 at 5:14 PM, Michael Stahnke <stahnma@puppetlabs.com>
wrote:
> How would Puppet handle it?  It would still have to either cache the
> package locally (eating lots of disk space) or point to a proxy, which
> you can do with Puppet already.  Your simplest solution might be to
> look for a mirror hosted in EC2, which I assume there are some.
Hi Michael,

I knew that I could point to a proxy - but I am looking for ways to
automate the setup. Hardcoding yum or apt proxies is not a very
"Puppet" way of setting things up. If someone has 100 machines on EC2,
then I am sure it would be way more efficient to pull the package once
into EC2, cache it on the puppet master, and then redistribute it to
all of the 100 machines. And when the master finds that every machine
has the package installed, it can retire the package and free up space
(with an ordinary proxy it is harder to do the retirement part) - of
course if it is needed again it needs to be pulled in again.

If the package provider layer can handle all the work of caching, then
that would save lots of time - not every site wants to reinvent the
wheel of package caching.

Rayson

================================Open Grid Scheduler / Grid Engine
http://gridscheduler.sourceforge.net/

Scalable Grid Engine Support Program
http://www.scalablelogic.com/

>
> Another option would be to build up a squid proxy via puppet and then
> configure a client-side usage of that cache, but that''s no
different
> in EC2 vs any other setup.
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Mon, Jan 23, 2012 at 4:06 PM, Rayson Ho <raysonlogin@gmail.com>
wrote:
> On Mon, Jan 23, 2012 at 5:14 PM, Michael Stahnke
<stahnma@puppetlabs.com> wrote:
>> How would Puppet handle it?  It would still have to either cache the
>> package locally (eating lots of disk space) or point to a proxy, which
>> you can do with Puppet already.  Your simplest solution might be to
>> look for a mirror hosted in EC2, which I assume there are some.
>
> Hi Michael,
>
> I knew that I could point to a proxy - but I am looking for ways to
> automate the setup. Hardcoding yum or apt proxies is not a very
> "Puppet" way of setting things up. If someone has 100 machines on
EC2,
> then I am sure it would be way more efficient to pull the package once
> into EC2, cache it on the puppet master, and then redistribute it to
> all of the 100 machines. And when the master finds that every machine
> has the package installed, it can retire the package and free up space
> (with an ordinary proxy it is harder to do the retirement part) - of
> course if it is needed again it needs to be pulled in again.
I assure you, it is not more efficient to serve it through it puppet
(at least in its file serving capacity). Coding a proxy in a
configuration file is common pattern in puppet.  It can be a variable
set at top scope, via ENC, fact, or other methods.  It''s also
available as an option in the yumrepo type, if I am remembering
correctly.
>
> If the package provider layer can handle all the work of caching, then
> that would save lots of time - not every site wants to reinvent the
> wheel of package caching.
I would think it would be better to have a puppet module that sets up
package repository mirroring and caching rather than modify each
provider.  Packages are managed differently across different types of
systems.  (quite differently in some cases).  However a good squid
module for setting up a cache would reduce reinvention.  Additionally
in many setups the puppet client talks to the master over a different
set of TCP ports than the package management clients do.

Here are some decent looking modules.  I think we internally use a
variation of the first one.

https://github.com/xaque208/puppet-squid

https://github.com/duritong/puppet-squid

https://github.com/example42/puppet-modules/tree/81489cbf7fc79fb0c0e05fde65233d6132b7b5b4/squid

Of course another option is to use a close proximity mirror.





>
> Rayson
>
> ================================> Open Grid Scheduler / Grid Engine
> http://gridscheduler.sourceforge.net/
>
> Scalable Grid Engine Support Program
> http://www.scalablelogic.com/
>
>
>>
>> Another option would be to build up a squid proxy via puppet and then
>> configure a client-side usage of that cache, but that''s no
different
>> in EC2 vs any other setup.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.