Puppet users - Dec 2007 - Testing / Coding methodologies?

I''ve been having great "fun" testing the new (Puppetised)
build
end-to-end for several days now.  The majority of the fun has been
coming from vendor packages (no names mentioned), typically with
conflicting permissions on files, or perhaps different home directories,
UIDs for the app users so it matters in which order you do things in.

My current methodology is to go through things linearly.  Install the
packages, check for dependencies, check for directories and (config)
files, users etc and write the manifests manually.

I''m pretty liberal putting dependencies into my manifests.  However you
just can''t spot all of them easily.  These are being flushed out mostly
as I change the external node classification (e.g. dmz/prod/dev) of the
test machines I''m building.  The changes in the tree result in a
slightly different ordering which may throw one of these out in the
wash.

It''s not an easy thing to spot all of these.  I suppose you could query
the RPM database for conflicting file modes (in fact this sounds like a
worthwhile exercise), but it gets more complicated with the scripts
embedded in the RPMs.  Puppet gives you all the tools you need to deal
with the conflicts and generally poor quality packages, but the process
to get there has been maddening at times.

Does anyone have any thoughts on methodologies?  Or as is more likely to
be practical, rules of thumb such as the RPM query idea?

What I''m not keen on is really overdoing the dependencies, so much as
to
create a manifest which is a script which runs in a straight up
top-to-bottom fashion.  It takes away the elegance of the language.
(Unless I''m installing a well known clustering software which has such
appalling dependency omissions in the RPMs you absolutely have to
install them in the order you''re told to).

Regards,
Derek
------------------------------------------------------------------------
For important statutory and regulatory disclosures and more information about
Barclays Capital, please visit our web site at http://www.barcap.com.

Internet communications are not secure and therefore the Barclays Group does not
accept legal responsibility for the contents of this message.  Although the
Barclays Group operates anti-virus programmes, it does not accept responsibility
for any damage whatsoever that is caused by viruses being passed.  Any views or
opinions presented are solely those of the author and do not necessarily
represent those of the Barclays Group.  Replies to this email may be monitored
by the Barclays Group for operational or business reasons.

Barclays Capital is the investment banking division of Barclays Bank PLC, a
company registered in England (number 1026167) with its registered office at 1
Churchill Place, London, E14 5HP. This email may relate to or be sent from other
members of the Barclays Group.
------------------------------------------------------------------------

On Tue, 2007-12-18 at 11:21 +0000, Derek.Whayman@barclayscapital.com
wrote:
> It''s not an easy thing to spot all of these.  I suppose you could
query
> the RPM database for conflicting file modes (in fact this sounds like a
> worthwhile exercise), but it gets more complicated with the scripts
> embedded in the RPMs.  Puppet gives you all the tools you need to deal
> with the conflicts and generally poor quality packages, but the process
> to get there has been maddening at times.
> 
> Does anyone have any thoughts on methodologies?  Or as is more likely to
> be practical, rules of thumb such as the RPM query idea?
I don''t have any advice from the puppet side; though for poorly
packaged
RPM''s[1] there''s a few things you can do from the RPM side,
and I think
the only way to clean up some of those things is from the RPM side, e.g.
broken %post scripts.

      * If the vendor is willing to give you a src.rpm, you can fix the
        specfile directly, rebuild, and stick those new RPM''s in a
        site-specific repo
      * If they don''t give you a src.rpm, maybe they can give you a
        nosrc.rpm (no sources, just binaries) that lets you fix the
        specfile and rebuild
      * Failing that, you can make your own nosrc.rpm from their binary
        RPM using rpm2cpio and bundling their binaries into your own
        nosrc.rpm; that at least gives you control over file
        permissions, %pre/%post scripts, and maybe even file locations
      * If the problems can be fixed after the vendor RPM has been
        installed, you can build your own fake RPM that requires the
        vendor''s RPM and uses scripts to fix the vendor''s
problems after
        the fact. Then always install your fake RPM. (That''s also a way
        to add missing dependencies to vendor RPM''s)

It''s all kinda ugly, and ultimately the vendors need to fix their
packaging; but one of the above might just work. And don''t forget to
complain loudly tothe vendor how their packages are broken :)

David

[1] It still blows my mind how badly multi-billion dollar companies
package their software. You, as a customer, should hit them over the
head as hard as you can, and tell them to read, understand, and apply
sane practcies, as the ones in
http://fedoraproject.org/wiki/Packaging/Guidelines

[2] Duke has some examples: http://linux.duke.edu/projects/nosrc-rpm/

I like the guidelines.  I like them a lot.

But many of these vendor package problems are much more brain dead than
that.  E.g.  Linux package requires perl-Win32-Registry.

And yes, re: [1] we tell them time and time again.  I don''t suppose Red
Hat operate a "Red Hat Certified Package" programme?  This kind of
thing
seems really OTT, my instincts don''t point towards such schemes.  I
dunno.  Something for the strategist-types.

I succeeded in the end in working around the problem within Puppet, but
I suppose the real issue is that if the software really is that
indeterministic, you''re right, you have to fix it outside of Puppet.

I especially liked the seemingly random UIDs and GIDs this software
created app users with ;-)

Many thanks,
Derek


David Lutterkort wrote:

[1] It still blows my mind how badly multi-billion dollar companies
package their software. You, as a customer, should hit them over the
head as hard as you can, and tell them to read, understand, and apply
sane practcies, as the ones in
http://fedoraproject.org/wiki/Packaging/Guidelines

[2] Duke has some examples: http://linux.duke.edu/projects/nosrc-rpm/
------------------------------------------------------------------------
For important statutory and regulatory disclosures and more information about
Barclays Capital, please visit our web site at http://www.barcap.com.

Internet communications are not secure and therefore the Barclays Group does not
accept legal responsibility for the contents of this message.  Although the
Barclays Group operates anti-virus programmes, it does not accept responsibility
for any damage whatsoever that is caused by viruses being passed.  Any views or
opinions presented are solely those of the author and do not necessarily
represent those of the Barclays Group.  Replies to this email may be monitored
by the Barclays Group for operational or business reasons.

Barclays Capital is the investment banking division of Barclays Bank PLC, a
company registered in England (number 1026167) with its registered office at 1
Churchill Place, London, E14 5HP. This email may relate to or be sent from other
members of the Barclays Group.
------------------------------------------------------------------------