Clay Caviness
2013-Jul-24 21:28 UTC
[Puppet Users] creating a (mac) package from a known good source + local patches
I need to have a way to re-create the installation package we use for facter and puppet, offline, from a known-good source. Often, we have local patches (e.g., backporting certain providers from 3.x to 2.x because our servers are still on 2.x, or patching things not yet released upstream), so our process has been: 1) Download tarball for a given release 2) Update makefile we use to create package to refer to new tarball 3) Verify local patches are still required 4) Build (possibly patched) package using ./etc/osx/createpackage.sh Now that createpackage.sh is really fully deprecated and gone, I''m looking at the rake packaging tasks. As I found in http://projects.puppetlabs.com/issues/21760#note-4 the tarball is useless since the rake tasks assume you''re in a git checkout. So now I''m at a bit of a loss on how to create these packages. Checkout a specific tag, and then make my own tarball? I''m concerned that the reliance on it being a git checkout means the tasks may end up talking to github, making the whole process not hermetic. What''s the best solution for this? Shall I create this as an issue against the packaging tasks? Thanks -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
Moses Mendoza
2013-Jul-24 23:38 UTC
Re: [Puppet Users] creating a (mac) package from a known good source + local patches
Hi Clay, I totally see your use case. We''re currently looking to revisit the way we build apple packages to make things more sane. In the meantime, there is a workaround workflow, which still uses git/github but allows you to keep things offline after you download them. As noted in the ticket, the packaging tasks rely on our projects'' git code info, so we have to keep that around to build. If you don''t want to rsync around git clones, one way I''ve accomplished it is using git bundles[1] (like a git archive, but retains the version control info). This process is the same for facter, puppet, and hiera. Roughly: Clone facter and check out the tag that you want to build, e.g. 1.7.2. The tags we push to github are the canonical artifacts we use to mark a release. # git clone git://github.com/puppetlabs/facter && cd facter && git checkout 1.7.2 We sign our tags, so you can verify the tag if you have our public key installed (see [2] on how to get it) # git verify-tag 1.7.2 Create a git bundle of your clone # git bundle create facter_bundle HEAD --tags Clone the packaging repo and create a git archive of it as well. 0.2.0 is the most recent version # git clone git://github.com/puppetlabs/packaging && cd packaging && git checkout 0.2.0 # git bundle create packaging_bundle HEAD --tags With the two git bundle files, you can now create mac packages offline. Take the git bundles to wherever you want to build packages, and git clone then to extract. # git clone facter_bundle facter # git clone packaging_bundle facter/ext/packaging # apply your patches to facter (even branch and commit them if you want) Now build packages: # cd facter && rake package:apple ; # a dmg will be staged in ./pkg/apple/ I completely understand your concern about the tasks talking to github - some of them do. However, rake package:apple isn''t one of them - it relies entirely on tools present on the system. [1] http://git-scm.com/blog/2010/03/10/bundles.html [2] http://puppetlabs.com/misc/download-options On Wed, Jul 24, 2013 at 2:28 PM, Clay Caviness <ccaviness@gmail.com> wrote:> I need to have a way to re-create the installation package we use for facter > and puppet, offline, from a known-good source. Often, we have local patches > (e.g., backporting certain providers from 3.x to 2.x because our servers are > still on 2.x, or patching things not yet released upstream), so our process > has been: > > 1) Download tarball for a given release > 2) Update makefile we use to create package to refer to new tarball > 3) Verify local patches are still required > 4) Build (possibly patched) package using ./etc/osx/createpackage.sh > > Now that createpackage.sh is really fully deprecated and gone, I''m looking > at the rake packaging tasks. As I found in > http://projects.puppetlabs.com/issues/21760#note-4 the tarball is useless > since the rake tasks assume you''re in a git checkout. > > So now I''m at a bit of a loss on how to create these packages. Checkout a > specific tag, and then make my own tarball? I''m concerned that the reliance > on it being a git checkout means the tasks may end up talking to github, > making the whole process not hermetic. > > What''s the best solution for this? Shall I create this as an issue against > the packaging tasks? > > Thanks > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users. > For more options, visit https://groups.google.com/groups/opt_out. > >-- Moses Mendoza Puppet Labs Join us at PuppetConf 2013, August 22-23 in San Francisco - http://bit.ly/pupconf13 Register now and take advantage of the Final Countdown discount - save 15%! -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
Clay Caviness
2013-Jul-25 22:45 UTC
Re: [Puppet Users] creating a (mac) package from a known good source + local patches
Thanks, Moses. I''ll update our internal build process to use this method. Also, FYI, I created https://github.com/puppetlabs/packaging/pull/171 to update the apple task to use pkgbuild instead of PackageMaker (since that was easier than trying to find and install packagemaker...) On Wed, Jul 24, 2013 at 7:38 PM, Moses Mendoza <moses@puppetlabs.com> wrote:> Hi Clay, > > I totally see your use case. We''re currently looking to revisit the > way we build apple packages to make things more sane. > > In the meantime, there is a workaround workflow, which still uses > git/github but allows you to keep things offline after you download > them. As noted in the ticket, the packaging tasks rely on our > projects'' git code info, so we have to keep that around to build. If > you don''t want to rsync around git clones, one way I''ve accomplished > it is using git bundles[1] (like a git archive, but retains the > version control info). This process is the same for facter, puppet, > and hiera. > > Roughly: > > Clone facter and check out the tag that you want to build, e.g. 1.7.2. > The tags we push to github are the canonical artifacts we use to mark > a release. > # git clone git://github.com/puppetlabs/facter && cd facter && git > checkout 1.7.2 > > We sign our tags, so you can verify the tag if you have our public key > installed (see [2] on how to get it) > # git verify-tag 1.7.2 > > Create a git bundle of your clone > # git bundle create facter_bundle HEAD --tags > > Clone the packaging repo and create a git archive of it as well. 0.2.0 > is the most recent version > # git clone git://github.com/puppetlabs/packaging && cd packaging > && git checkout 0.2.0 > # git bundle create packaging_bundle HEAD --tags > > With the two git bundle files, you can now create mac packages > offline. Take the git bundles to wherever you want to build packages, > and git clone then to extract. > # git clone facter_bundle facter > # git clone packaging_bundle facter/ext/packaging > # apply your patches to facter (even branch and commit them if you want) > > Now build packages: > # cd facter && rake package:apple ; # a dmg will be staged in > ./pkg/apple/ > > I completely understand your concern about the tasks talking to github > - some of them do. However, rake package:apple isn''t one of them - it > relies entirely on tools present on the system. > > [1] http://git-scm.com/blog/2010/03/10/bundles.html > [2] http://puppetlabs.com/misc/download-options > > On Wed, Jul 24, 2013 at 2:28 PM, Clay Caviness <ccaviness@gmail.com> > wrote: > > I need to have a way to re-create the installation package we use for > facter > > and puppet, offline, from a known-good source. Often, we have local > patches > > (e.g., backporting certain providers from 3.x to 2.x because our servers > are > > still on 2.x, or patching things not yet released upstream), so our > process > > has been: > > > > 1) Download tarball for a given release > > 2) Update makefile we use to create package to refer to new tarball > > 3) Verify local patches are still required > > 4) Build (possibly patched) package using ./etc/osx/createpackage.sh > > > > Now that createpackage.sh is really fully deprecated and gone, I''m > looking > > at the rake packaging tasks. As I found in > > http://projects.puppetlabs.com/issues/21760#note-4 the tarball is > useless > > since the rake tasks assume you''re in a git checkout. > > > > So now I''m at a bit of a loss on how to create these packages. Checkout a > > specific tag, and then make my own tarball? I''m concerned that the > reliance > > on it being a git checkout means the tasks may end up talking to github, > > making the whole process not hermetic. > > > > What''s the best solution for this? Shall I create this as an issue > against > > the packaging tasks? > > > > Thanks > > > > -- > > You received this message because you are subscribed to the Google Groups > > "Puppet Users" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to puppet-users+unsubscribe@googlegroups.com. > > To post to this group, send email to puppet-users@googlegroups.com. > > Visit this group at http://groups.google.com/group/puppet-users. > > For more options, visit https://groups.google.com/groups/opt_out. > > > > > > > > -- > Moses Mendoza > Puppet Labs > > Join us at PuppetConf 2013, August 22-23 in San Francisco - > http://bit.ly/pupconf13 > Register now and take advantage of the Final Countdown discount - save 15%! > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users. > For more options, visit https://groups.google.com/groups/opt_out. > > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
Moses Mendoza
2013-Jul-26 00:59 UTC
Re: [Puppet Users] creating a (mac) package from a known good source + local patches
Hi Clay, Thanks for the patch! We''ll take a look. Switching over to pkgbuild has been on the list for awhile. Cheers, Moses On Thu, Jul 25, 2013 at 3:45 PM, Clay Caviness <ccaviness@gmail.com> wrote:> Thanks, Moses. I''ll update our internal build process to use this method. > > Also, FYI, I created https://github.com/puppetlabs/packaging/pull/171 to > update the apple task to use pkgbuild instead of PackageMaker (since that > was easier than trying to find and install packagemaker...) > > > On Wed, Jul 24, 2013 at 7:38 PM, Moses Mendoza <moses@puppetlabs.com> wrote: >> >> Hi Clay, >> >> I totally see your use case. We''re currently looking to revisit the >> way we build apple packages to make things more sane. >> >> In the meantime, there is a workaround workflow, which still uses >> git/github but allows you to keep things offline after you download >> them. As noted in the ticket, the packaging tasks rely on our >> projects'' git code info, so we have to keep that around to build. If >> you don''t want to rsync around git clones, one way I''ve accomplished >> it is using git bundles[1] (like a git archive, but retains the >> version control info). This process is the same for facter, puppet, >> and hiera. >> >> Roughly: >> >> Clone facter and check out the tag that you want to build, e.g. 1.7.2. >> The tags we push to github are the canonical artifacts we use to mark >> a release. >> # git clone git://github.com/puppetlabs/facter && cd facter && git >> checkout 1.7.2 >> >> We sign our tags, so you can verify the tag if you have our public key >> installed (see [2] on how to get it) >> # git verify-tag 1.7.2 >> >> Create a git bundle of your clone >> # git bundle create facter_bundle HEAD --tags >> >> Clone the packaging repo and create a git archive of it as well. 0.2.0 >> is the most recent version >> # git clone git://github.com/puppetlabs/packaging && cd packaging >> && git checkout 0.2.0 >> # git bundle create packaging_bundle HEAD --tags >> >> With the two git bundle files, you can now create mac packages >> offline. Take the git bundles to wherever you want to build packages, >> and git clone then to extract. >> # git clone facter_bundle facter >> # git clone packaging_bundle facter/ext/packaging >> # apply your patches to facter (even branch and commit them if you want) >> >> Now build packages: >> # cd facter && rake package:apple ; # a dmg will be staged in >> ./pkg/apple/ >> >> I completely understand your concern about the tasks talking to github >> - some of them do. However, rake package:apple isn''t one of them - it >> relies entirely on tools present on the system. >> >> [1] http://git-scm.com/blog/2010/03/10/bundles.html >> [2] http://puppetlabs.com/misc/download-options >> >> On Wed, Jul 24, 2013 at 2:28 PM, Clay Caviness <ccaviness@gmail.com> >> wrote: >> > I need to have a way to re-create the installation package we use for >> > facter >> > and puppet, offline, from a known-good source. Often, we have local >> > patches >> > (e.g., backporting certain providers from 3.x to 2.x because our servers >> > are >> > still on 2.x, or patching things not yet released upstream), so our >> > process >> > has been: >> > >> > 1) Download tarball for a given release >> > 2) Update makefile we use to create package to refer to new tarball >> > 3) Verify local patches are still required >> > 4) Build (possibly patched) package using ./etc/osx/createpackage.sh >> > >> > Now that createpackage.sh is really fully deprecated and gone, I''m >> > looking >> > at the rake packaging tasks. As I found in >> > http://projects.puppetlabs.com/issues/21760#note-4 the tarball is >> > useless >> > since the rake tasks assume you''re in a git checkout. >> > >> > So now I''m at a bit of a loss on how to create these packages. Checkout >> > a >> > specific tag, and then make my own tarball? I''m concerned that the >> > reliance >> > on it being a git checkout means the tasks may end up talking to github, >> > making the whole process not hermetic. >> > >> > What''s the best solution for this? Shall I create this as an issue >> > against >> > the packaging tasks? >> > >> > Thanks >> > >> > -- >> > You received this message because you are subscribed to the Google >> > Groups >> > "Puppet Users" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> > an >> > email to puppet-users+unsubscribe@googlegroups.com. >> > To post to this group, send email to puppet-users@googlegroups.com. >> > Visit this group at http://groups.google.com/group/puppet-users. >> > For more options, visit https://groups.google.com/groups/opt_out. >> > >> > >> >> >> >> -- >> Moses Mendoza >> Puppet Labs >> >> Join us at PuppetConf 2013, August 22-23 in San Francisco - >> http://bit.ly/pupconf13 >> Register now and take advantage of the Final Countdown discount - save >> 15%! >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to puppet-users+unsubscribe@googlegroups.com. >> To post to this group, send email to puppet-users@googlegroups.com. >> Visit this group at http://groups.google.com/group/puppet-users. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users. > For more options, visit https://groups.google.com/groups/opt_out. > >-- Moses Mendoza Puppet Labs Join us at PuppetConf 2013, August 22-23 in San Francisco - http://bit.ly/pupconf13 Register now and take advantage of the Final Countdown discount - save 15%! -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.