It would be neat if puppet could use tar.gz''s as a source, instead of
just bare directory trees. So I''ve lodged a feature request:
https://projects.puppetlabs.com/issues/5786
Many of my manifests for applications need to cover the following
process: 1. Download .tar.gz to host
2. Expand .tar.gz
3. Whatever install process is required
As an example (assuming Feature Request#5783):
file {“/srv/<application>”:
source => “<puppet|http>:///<path>/<application>.tar.gz”,
expand => true,
recurse => true,
ensure => directory,
}
This would create a directory, and populate it with the contents of
.tar.gz It would also keep the contents in sync with .tar.gz.
Expand is a new option for file, but perhaps this could be inferred (and
the option not needed) as ensure => directory and source is a file.
--
Michael Knox
Systems Administrator, DEK Technologies P/L
Email:michael.knox@dektech.com.au, Skype: michael.knox.au
Phone: +61 (0) 3 9302 8940, Mobile: +61 (0) 410 124 816
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
On Jan 5, 2011, at 2:52 PM, Michael Knox wrote:> It would be neat if puppet could use tar.gz''s as a source, instead of just bare directory trees. So I''ve lodged a feature request: https://projects.puppetlabs.com/issues/5786 > > Many of my manifests for applications need to cover the following process: 1. Download .tar.gz to host > 2. Expand .tar.gz > 3. Whatever install process is required > > As an example (assuming Feature Request#5783): > file {“/srv/<application>”: > source => “<puppet|http>:///<path>/<application>.tar.gz”, > expand => true, > recurse => true, > ensure => directory, > } > > This would create a directory, and populate it with the contents of .tar.gz It would also keep the contents in sync with .tar.gz. > > Expand is a new option for file, but perhaps this could be inferred (and the option not needed) as ensure => directory and source is a file.I''m curious what you''re thinking of. Would this be a File? (The directory contents are checked each time.) or would this be a Package (assume that the contents are right if the file on the server hasn''t changed) -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
> It would be neat if puppet could use tar.gz''s as a source, instead of > just bare directory trees. So I''ve lodged a feature request: > https://projects.puppetlabs.com/issues/5786 > > Many of my manifests for applications need to cover the following > process: 1. Download .tar.gz to host > 2. Expand .tar.gz > 3. Whatever install process is required > >The best option is to build a Debian package or RPM. If you must build from source on each agent, I''ve had good success with something like this: file { "/usr/local/sbin/install-thing": content => template("install-thing"), group => root, mode => 0755, owner => root, } exec { "/usr/local/sbin/install-thing": cwd => "/root", refreshonly => true, subscribe => File["/usr/local/sbin/install-thing"], timeout => "-1", } If you use the pattern frequently, define a type that takes care of the boilerplate so you can just do: sourceinstall { "thing": script => template("install-thing") } Your install-thing program may look something like this: VERSION=0.0.0 TMPDIR="$(mktemp -d)" cd "$TMPDIR" trap "cd / && rm -rf \"$TMPDIR\"" EXIT wget "http://thing-server/thing-$VERSION.tar.gz" tar xf "thing-$VERSION.tar.gz" cd "thing-$VERSION" ./configure make make install -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On 6/01/11 10:48 AM, Patrick wrote:> On Jan 5, 2011, at 2:52 PM, Michael Knox wrote: >> It would be neat if puppet could use tar.gz''s as a source, instead of just bare directory trees. So I''ve lodged a feature request: https://projects.puppetlabs.com/issues/5786 >> >> Many of my manifests for applications need to cover the following process: 1. Download .tar.gz to host >> 2. Expand .tar.gz >> 3. Whatever install process is required >> >> As an example (assuming Feature Request#5783): >> file {“/srv/<application>”: >> source => “<puppet|http>:///<path>/<application>.tar.gz”, >> expand => true, >> recurse => true, >> ensure => directory, >> } >> >> This would create a directory, and populate it with the contents of .tar.gz It would also keep the contents in sync with .tar.gz. >> >> Expand is a new option for file, but perhaps this could be inferred (and the option not needed) as ensure => directory and source is a file. > > I''m curious what you''re thinking of. > > Would this be a File? (The directory contents are checked each time.) or would this be a Package (assume that the contents are right if the file on the server hasn''t changed) >I was thinking of FIle, I hadn''t thought of it as a Package. My concern was ensuring that the contents on the disk match the source. Both the exec {"expand_tar_file": onlyif => some test }, and package{"some app": } have the same risk ... what''s on the disk has changed, and won''t be refreshed as our metadata say''s that it''s all good. I guess the onlyif for the exec could be a checksum ... currently I only do a basic check to see if it''s been downloaded and expanded. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On Wed, Jan 5, 2011 at 2:52 PM, Michael Knox <michael.knox.au@gmail.com>wrote:> It would be neat if puppet could use tar.gz''s as a source, instead of just > bare directory trees. So I''ve lodged a feature request: > https://projects.puppetlabs.com/issues/5786 > > Many of my manifests for applications need to cover the following process: > 1. Download .tar.gz to host > 2. Expand .tar.gz > 3. Whatever install process is required >I''m not totally against this, but I am curious why you haven''t decided to build packages for these applications, as that should essentially contain all your desired functionality right?> > As an example (assuming Feature Request#5783): > file {“/srv/<application>”: > source => “<puppet|http>:///<path>/<application>.tar.gz”, > expand => true, > recurse => true, > ensure => directory, > } > > This would create a directory, and populate it with the contents of .tar.gz > It would also keep the contents in sync with .tar.gz. > > Expand is a new option for file, but perhaps this could be inferred (and > the option not needed) as ensure => directory and source is a file. > > > -- > Michael Knox > > Systems Administrator, DEK Technologies P/L > > Email:michael.knox@dektech.com.au <Email%3Amichael.knox@dektech.com.au>, > Skype: michael.knox.au > Phone: +61 (0) 3 9302 8940, Mobile: +61 (0) 410 124 816 > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On Jan 5, 2011, at 5:52 PM, Nigel Kersten wrote:> > > On Wed, Jan 5, 2011 at 2:52 PM, Michael Knox <michael.knox.au@gmail.com> wrote: > It would be neat if puppet could use tar.gz''s as a source, instead of just bare directory trees. So I''ve lodged a feature request: https://projects.puppetlabs.com/issues/5786 > > Many of my manifests for applications need to cover the following process: 1. Download .tar.gz to host > 2. Expand .tar.gz > 3. Whatever install process is required > > I''m not totally against this, but I am curious why you haven''t decided to build packages for these applications, as that should essentially contain all your desired functionality right? >I can''t answer for the original poster, but I can answer for why I''ve done this myself. I had a vendor application that packages dynamically linked libraries with the application. This caused the Debian package builder to have a fit when it tried to figure out what libraries the application depended on because the application bundles 32-bit libraries and the OS is 64-bit. I read the manuals and used Google, but I couldn''t figure out how to disable the check. I tried IRC and couldn''t get a better response than RTFM so I finally gave up and switched to tar.bz2. Since this is in one directory, I name the directory with the application version in the directory name. Then I set the path in the application launch script. Not an elegant solution, but it did finally work. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On 6 January 2011 12:52, Nigel Kersten <nigel@puppetlabs.com> wrote:> > > On Wed, Jan 5, 2011 at 2:52 PM, Michael Knox <michael.knox.au@gmail.com>wrote: > >> It would be neat if puppet could use tar.gz''s as a source, instead of just >> bare directory trees. So I''ve lodged a feature request: >> https://projects.puppetlabs.com/issues/5786 >> >> Many of my manifests for applications need to cover the following process: >> 1. Download .tar.gz to host >> 2. Expand .tar.gz >> 3. Whatever install process is required >> > > I''m not totally against this, but I am curious why you haven''t decided to > build packages for these applications, as that should essentially contain > all your desired functionality right? >Our own situation is that we have developers who build their own applications, and if we packaged them with RPM or pkg, then they would have to be installed as root. We don''t trust them enough for that, so right now we run an exec as the application owner to unpack the tar.gz John -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On Wed, Jan 5, 2011 at 6:39 PM, John Warburton <jwarburton@gmail.com> wrote:> On 6 January 2011 12:52, Nigel Kersten <nigel@puppetlabs.com> wrote: > >> >> >> On Wed, Jan 5, 2011 at 2:52 PM, Michael Knox <michael.knox.au@gmail.com>wrote: >> >>> It would be neat if puppet could use tar.gz''s as a source, instead of >>> just bare directory trees. So I''ve lodged a feature request: >>> https://projects.puppetlabs.com/issues/5786 >>> >>> Many of my manifests for applications need to cover the following >>> process: 1. Download .tar.gz to host >>> 2. Expand .tar.gz >>> 3. Whatever install process is required >>> >> >> I''m not totally against this, but I am curious why you haven''t decided to >> build packages for these applications, as that should essentially contain >> all your desired functionality right? >> > > Our own situation is that we have developers who build their own > applications, and if we packaged them with RPM or pkg, then they would have > to be installed as root. We don''t trust them enough for that, so right now > we run an exec as the application owner to unpack the tar.gz > >Do you just unpack once and leave it alone? Drop a marker file when an install script succeeds and you don''t have to run it again? It sounds like we could maybe do a tar.{gz,bz2,zip} "package" provider, with simple install, uninstall, maybe the ability to specify versions by symlinks. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On 6 January 2011 16:18, Nigel Kersten <nigel@puppetlabs.com> wrote:> On Wed, Jan 5, 2011 at 6:39 PM, John Warburton <jwarburton@gmail.com>wrote: > >> Our own situation is that we have developers who build their own >> applications, and if we packaged them with RPM or pkg, then they would have >> to be installed as root. We don''t trust them enough for that, so right now >> we run an exec as the application owner to unpack the tar.gz >> >> > Do you just unpack once and leave it alone? Drop a marker file when an > install script succeeds and you don''t have to run it again? >Yes - unpack and leave there - which is "messy", and on the "To Do" list, but good enough for now. The marker file is the directory we expect to have been unpacked from the tar ball It sounds like we could maybe do a tar.{gz,bz2,zip} "package" provider, with> simple install, uninstall, maybe the ability to specify versions by > symlinks. >We actually have a define on top of the package provider to use graft ( http://peters.gormand.com.au/Home/tools/graft/graft-html) to use different versions of packages with sym links because package management in Solaris isn''t version aware like rpm is John -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On 01/05/2011 09:39 PM, John Warburton wrote:> On 6 January 2011 12:52, Nigel Kersten <nigel@puppetlabs.com > <mailto:nigel@puppetlabs.com>> wrote: > > > > On Wed, Jan 5, 2011 at 2:52 PM, Michael Knox <michael.knox.au > <http://michael.knox.au>@gmail.com <http://gmail.com>> wrote: > > It would be neat if puppet could use tar.gz''s as a source, instead of > just bare directory trees. So I''ve lodged a feature request: > https://projects.puppetlabs.com/issues/5786 > > Many of my manifests for applications need to cover the following > process: 1. Download .tar.gz to host > 2. Expand .tar.gz > 3. Whatever install process is required > > > I''m not totally against this, but I am curious why you haven''t decided to > build packages for these applications, as that should essentially contain > all your desired functionality right? > > > Our own situation is that we have developers who build their own applications, > and if we packaged them with RPM or pkg, then they would have to be installed > as root. We don''t trust them enough for that, so right now we run an exec as > the application owner to unpack the tar.gz > > JohnAt least in RPM-land, packages can be built as non-root (and is highly recommended you do so). You''re right that packages can''t be installed w/o root, but they can be relocated if you build your package right. If you''re installing the package w/ puppet root privs are already there. -Doug
On Jan 5, 8:39 pm, John Warburton <jwarbur...@gmail.com> wrote: [...]> Our own situation is that we have developers who build their own > applications, and if we packaged them with RPM or pkg, then they would have > to be installed as root. We don''t trust them enough for that, so right now > we run an exec as the application owner to unpack the tar.gzLike Doug, I don''t quite follow that. Perhaps I misunderstand "installed as root", because Puppet is already providing root privileges for the installation. If you mean "installed as owned by root" or "installed in <choose particular location>" then you are mistaken: RPMs can easilly be built so that their files are installed wherever you like and have whatever ownership and permissions you like. If you are concerned about scriptlets in the RPM being run as root then you can easily avoid that. Don''t rely on the developers to package their own software; instead take the tarballs they already provide and package up all the contents in RPM form (without any scriptlets). I do such RPMs by hand when I have to install an application that is delivered in binary-only form (it''s pretty easy), but it should also be relatively easy to script. I try at all costs to avoid installing anything on my systems without packaging it. That way I know what''s (supposed to be) there, I can update or remove it with ease, the package installation system detects conflicts and dependencies for me, and, as a bonus, third-party tools such as Puppet support me better. YMMV. John -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On 01/06/2011 03:33 PM, jcbollinger wrote:> > On Jan 5, 8:39 pm, John Warburton <jwarbur...@gmail.com> wrote: > [...] >> Our own situation is that we have developers who build their own >> applications, and if we packaged them with RPM or pkg, then they would have >> to be installed as root. We don''t trust them enough for that, so right now >> we run an exec as the application owner to unpack the tar.gz > > Like Doug, I don''t quite follow that. Perhaps I misunderstand > "installed as root", because Puppet is already providing root > privileges for the installation. If you mean "installed as owned by > root" or "installed in <choose particular location>" then you are > mistaken: RPMs can easilly be built so that their files are installed > wherever you like and have whatever ownership and permissions you > like. > > If you are concerned about scriptlets in the RPM being run as root > then you can easily avoid that. Don''t rely on the developers to > package their own software; instead take the tarballs they already > provide and package up all the contents in RPM form (without any > scriptlets). I do such RPMs by hand when I have to install an > application that is delivered in binary-only form (it''s pretty easy), > but it should also be relatively easy to script. > > I try at all costs to avoid installing anything on my systems without > packaging it. That way I know what''s (supposed to be) there, I can > update or remove it with ease, the package installation system detects > conflicts and dependencies for me, and, as a bonus, third-party tools > such as Puppet support me better.+1 to that. Both RPM and DEB provide rather simple tools to build "dumb" packages, i.e. glorified tarballs. These are much better suited to puppet use than actual tarballs. That being said, in some places (like tomcat installation) I myself currently have puppet wget a tarball from one of my servers and unpack it, but stuffing a dumb package into my apt repo would be a saner approach there, as well. Regards, Felix -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On 6/01/11 12:52 PM, Nigel Kersten wrote:> > > On Wed, Jan 5, 2011 at 2:52 PM, Michael Knox <michael.knox.au > <http://michael.knox.au>@gmail.com <http://gmail.com>> wrote: > > It would be neat if puppet could use tar.gz''s as a source, instead > of just bare directory trees. So I''ve lodged a feature request: > https://projects.puppetlabs.com/issues/5786 > > Many of my manifests for applications need to cover the following > process: 1. Download .tar.gz to host > 2. Expand .tar.gz > 3. Whatever install process is required > > > I''m not totally against this, but I am curious why you haven''t decided > to build packages for these applications, as that should essentially > contain all your desired functionality right?Generally I''d use packages (rpm, deb etc) as my first preference. With some applications (such as Codestriker etc), they generally aren''t packaged, or easily available as packages, particularly for SLES/OpenSuSE. Yes, I could create packages for them, but I was avoiding that as it''s something else I need to train the team up on, who are already on a steep learning curve with Puppet. My original thought with this was as a file source, and not for package management, even though that''s were I''d be using it.> > As an example (assuming Feature Request#5783): > file {“/srv/<application>”: > source => “<puppet|http>:///<path>/<application>.tar.gz”, > expand => true, > recurse => true, > ensure => directory, > } > > This would create a directory, and populate it with the contents > of .tar.gz It would also keep the contents in sync with .tar.gz. > > Expand is a new option for file, but perhaps this could be > inferred (and the option not needed) as ensure => directory and > source is a file. > > om/group/puppet-users?hl=en. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
John Warburton
2011-Jan-07 05:16 UTC
Re: [Puppet Users] Re: Supporting tar.gz as file source
On 7 January 2011 01:33, jcbollinger <John.Bollinger@stjude.org> wrote:> > On Jan 5, 8:39 pm, John Warburton <jwarbur...@gmail.com> wrote: > [...] > > Our own situation is that we have developers who build their own > > applications, and if we packaged them with RPM or pkg, then they would > have > > to be installed as root. We don''t trust them enough for that, so right > now > > we run an exec as the application owner to unpack the tar.gz > > Like Doug, I don''t quite follow that. Perhaps I misunderstand > "installed as root", because Puppet is already providing root > privileges for the installation. If you mean "installed as owned by > root" or "installed in <choose particular location>" then you are > mistaken: RPMs can easilly be built so that their files are installed > wherever you like and have whatever ownership and permissions you > like. >I probably wasn''t clear, but what I meant was that rpm and pkgadd have to run as root, so we have to trust that the developers didn''t do anything silly / naughty / destructive in the script areas, or overwrote into places like /bin. We don''t have the resources right now to build them ourselves or audit such packages, so the least worst compromise (for us) was installing a tar ball with an exec being run as the application owner. If you are concerned about scriptlets in the RPM being run as root> then you can easily avoid that. Don''t rely on the developers to > package their own software; instead take the tarballs they already > provide and package up all the contents in RPM form (without any >as above, we''d like to, but we just don''t have the resources to do this right now> I try at all costs to avoid installing anything on my systems without > packaging it. That way I know what''s (supposed to be) there, I can >> YMMV. >I agree, but we''re in the early stages of large scale puppet deployment. Proper package management techniques for the outliers hopefully will come in the next iteration BTW for Solaris people, we use pkgbuild (http://pkgbuild.sourceforge.net/) which will automatically build a SVR4 pkg or IPS package from a RPM like SPEC file. It is pretty cool - it enforces that you don''t build packages as root, and if you have your SPEC file created correctly does everything from downloading a tarball, configure, compile and packaging. Very nice, and is what the Open Solaris project uses Regards John -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.