Puppet users - Mar 2014 - Changes to PuppetDB to support proper environments

Hey all,

TL;DR: We're adding support to environments to PuppetDB but have a
small migration hassle we wanted some community opinion on. If you're
interested in PuppetDB and environments read on.

So we're looking at adding first class support to PuppetDB for
environments. In the past we would happily accept facts/reports &
catalogs from multiple different environments, but that information
was never stored with the data.

As a consequence people trying to use the same PuppetDB instance for
environments would find they are collecting globally across all
environments thus creating a high chance of resource collision. For
many people this just wouldn't work, but for some they probably found
horrible ways of working around this (if you are one of these people,
I'd probably want to hear from you).

Not to mention we also have no way to represent this data in the PE
console either, basically PuppetDB just has no environment awareness.

So we're fixing this now (at least the PDB part). Our current Epic:
https://tickets.puppetlabs.com/browse/PDB-47 and relevant child
tickets covers this work if you want to follow along at home.

The issue we've hit however is how to cleanly migrate from a world of
no environments, to one where environments are going to be
constraining collections. Let me break it down:

* Currently all submissions to PDB have no environment knowledge, and
we can't store it anyway.
* During an upgrade to this new feature we have to set the environment
of all items in our database to 'something' internally and externally,
however without prior knowledge everything will be labelled the same.
* Once environment awareness is added to the terminus we only want to
collect on environments the agent transaction was run with, this
complies with the semantics of collections going back to whenever
storeconfigs & environments were added to Puppet.

One strategy for upgrades is to set the environment to 'production'
for existing data. This will stay like this until another catalog is
submitted with the true environment for that node.

Now the problem with that solution is related to people who might be
using exported resources (and somehow avoiding collection collisions):

* In a single environment setup, where the name of the environment is
not 'production'.
* In a multi-environment setup where environment names are quite
different (but only 1 is production).

For both of these cases, if we just default the environment to
'production' there will be a short time where collections will return
nothing for environments not called 'production' - until the next
catalog submission occurs. This could be detrimental, and we'd like to
understand how many users this might impact. Please let me know if
this is you.

An alternate solution is to migrate existing data to have the
environment set to something completely different (like 'nil' or
something else that can't possibly collide with a real name). With
this solution, we can put in a collection query that not only collects
for the current environment, but also for 'nil' to pick up the older
global resources ... until such time as all catalogs have submitted
thus putting the catalogs in the correct environment.

This concept of 'nil' is almost transient (although we be longer lived
for old reports obviously) its a temporary marker to say 'we don't
know the environment'. So in most cases, once all catalogs have been
submitted by all nodes the concept disappears. No new data should be
added to this internal special environment.

Anyway - I'm looking for some feedback for these two alternate
solutions (or a third more whacky solution - whatever :-). The first
one is obviously easiest and avoids leaving behind a transient
solution, the second we think solves this concern to some extent (at
least we think so).

Any feedback or opinion on this would be much appreciated.

ken.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAE4bNT%3DwHZtttzvAuL7qYX_6Ao%2BJuPggY%2BosgwP1%2BB5%2B7QhOJA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.