** Release Candidate **
PuppetDB 2.0.0 RC1 - April 30, 2014.
Prerelease: PuppetDB 2.0.0 is not yet released
PuppetDB 2.0.0 RC1 Downloads
------------------------
Available in native package format in the release repositories at:
http://yum.puppetlabs.com and http://apt.puppetlabs.com
For information on how to enable the Puppet Labs pre-release repos, see:
http://docs.puppetlabs.com/guides/puppetlabs_package_repositories.html#enabling-the-prerelease-repos
Binary tarball: http://downloads.puppetlabs.com/puppetdb/
Source: http://github.com/puppetlabs/puppetdb
Please report feedback via the Puppet Labs tickets site, using an
affected PuppetDB version of 2.0.0:
https://tickets.puppetlabs.com/browse/PDB
Documentation: http://docs.puppetlabs.com/puppetdb/2.0
Puppet module:
http://forge.puppetlabs.com/puppetlabs/puppetdb
PuppetDB 2.0.0 RC1 Release Notes
--------------------------------
PuppetDB 2.0.0 RC1 is a release candidate for a feature release focusing on
environments support. Note that *this is a major version bump* and there
are several breaking changes, including dropping support for versions of
PostgreSQL prior to version 8.4 and Java 1.6. See the "Deprecations and
potentially breaking changes" section below for more information.
Things to take note of before upgrading:
* There is a known issue in Puppet 3.4.0, 3.4.1 and 3.4.2 with duplicate
failed events being tracked here:
https://tickets.puppetlabs.com/browse/PUP-1524. This is being fixed in an
upcoming release of Puppet. Since 1.6.0 this error will become more obvious
as we are now raising full stack traces, so you may notice it more readily
after the upgrade.
* Make sure all your PuppetDB instances are shut down and only upgrade one
at a time.
* As usual, don’t forget to upgrade your puppetdb-terminus package also (on
the host where your Puppet Master lives), and restart your master service.
New features:
* (PDB-452,453,454,456,457,526,557) Adding support for storing, querying
and importing/exporting environments
This change forced a new revision of the `replace facts`,
`replace catalog` and `store report` commands. The PuppetDB terminus
also needed to be updated to support new environment information
being sent to PuppetDB. USERS MUST ALSO UPDATE THE PUPPETDB
TERMINUS. Previous versions of those commands (and wire formats) are
now deprecated. See the PuppetDB API docs for more information.
Environments support has been added to the new v4 (currently
experimental) query API. The following query endpoints now include
environment in the response:
- facts
- resources
- nodes
- catalogs
- reports
- events
- event counts
- aggregate event counts
The below query endpoints now allow query/filtering by environment:
- facts
- resources
- nodes
- catalogs
- reports
- events
This release also includes a new environments query endpoint to list
all known environments and allow an easy filtering based on
environment for things like events, facts reports and resources. See
the query API docs for more information. PuppetDB
import/export/anonymization and benchmark tools also now have
environment support. Storeconfigs export does not include
environments as that information is not being stored in the old
storeconfigs module. Environments that are no longer associated with
a fact set, report or catalog will be "garbage collected" by being
removed from the database.
* (PDB-581) Add subqueries to events query endpoint
The events endpoint now supports select-resources and select-facts
* (PDB-234) Add v4 query API, deprecate v2 query API
This patch adds the new code relevant for doing any future v4 work. It
has been
raised as an experimental end-point only so there are no commitments to
its
interface yet. Once stable we will need another patch to declare it as so.
This patch also deprecates the v2 end-point in documentation and by
adding the
same headers we used to use for the v1 end-point.
* (PDB-470) Provide new db setting 'statements-cache-size' with a
default
of 1000
This setting adjusts how many SQL prepared statements get cached via
BoneCP.
By using this setting we've seen an almost 40% decrease in wall clock
time it
takes to store a new catalog.
This patch adds the new configuration item as a user configurable one,
with a
default set to 1000 for now. Documentation has also been added for this
setting.
* (PDB-221) Add facts to import/export
This commit imports/exports facts similar to how we currently
import/export
catalogs and reports. Anonymize doesn't currently work for facts, which is
going to be added separately.
* (PDB-469) - Support Anonymizing Facts
This commit adds support for the anonymization of facts. The levels of
anonymization supported are:
- none - no anonymization
- low - only values with a fact name of secret, password etc
- moderate - recognized "safe" facts are untouched, recognized
facts with sensitive information (i.e. ipaddress)
have their values anonymized
- full - all fact names and values are anonymized
Deprecations and potentially breaking changes:
* (PDB-88, PDB-271) JDK 1.6 no longer supported
* (PDB-308) Drop 2.7.x support
This patch removes support for Puppet 2.7.x in several ways:
* New check for every entry point in terminus will return an error if the
version of Puppet is not supported. This is done in a 'soft' manner
to
avoid Puppet from not working.
* Documentation now only references Puppet 3
* Documentation now states only latest version of Puppet is supported
* Packaging now has hard dependencies on the latest version of Puppet
* Contrib gemspec has been updated
* Gemfile for tests have been updated
* (PDB-552) Pin support for Puppet to 3.5.1 and above
Starting PuppetDB with a Puppet earlier than 3.5.1 will now fail on
startup.
* (PDB-605) Pin facter requirement to 1.7.0
Using prior versions of Facter will cause PuppetDB to fail on startup
* (PDB-592) Removing support for Ubuntu Raring
Raring went EOL in Jan 2014, so we are no longer building packages for it.
* (PDB-79) Drop support for Postgres < 8.4
PuppetDB will now log an error and exit if it connects to an instance of
Postgres
older than 8.4. Users of older versions will need to upgrade (especially
EL 5
users as it defaults to 8.1). The acceptance tests for EL 5 have been
updated
to be explicit about using Postgres 8.4 packages instead.
* (PDB-204) Ensure all commands no longer need a serialized payload
Some previous commands required the payload of the command to be
JSON serialized strings as opposed to the relevant JSON type
directly in the payload. All commands no longer require the payload
to be serialized to a string first.
* (PDB-238) - Remove v1 API
This commit removes the v1 API and builds on the HTTP api refactor.
This commit contains:
- Remove all v1 namespaces and the namespaces calling them
- Remove api tests excercising the v1 routes
- Remove v1 references in the docs
* (PDB-354) Deprecate old versions of commands
This patch drops a warning whenever an old version of the commands API is
used
and updates the documentation to warn the user these old commands are
deprecated.
* (PDB-570) Remove planetarium endpoint
Old endpoint that had significant overlap with the current catalogs
endpoint
* (PDB-113) Remove swank
As swank is now a deprecated project. This patch removes swank support
completely from the code base.
Notable improvements and fixes:
* (PDB-473) Support POST using application/json data in the body
This patch adds to the commands end-point the ability to simply POST using
application/json with the JSON content in the body.
It also switches the terminus to use this mechanism.
We found that the url encode/decode required to support
x-www-form-urlencoded
was actually quite an overhead in a number of ways:
* The urlencode on the terminus added overhead
* The urldecode in the server added overhead
* The interim strings created during this encode/decode process can get
quite
large increasing the amount of garbage collection required
This feature has been implemented by providing a new middleware that will
move
the body into the parameter :body-string of the request when the
content-type
is not set to application/x-www-form-urlencoded. This provides a
convenient
backwards compatible layer so that the old form url encoding can still be
supported for older versions of the API.
* (PDB-567,191) Use hash not config_version for report export files
This fixes a bug related to config_versions containing characts not
safe to be use in file names (such as '/').
* (PDB-518) Fix bug storeconfig export of arrays
For exported Resources with parameters which value is a Array the
storeconfig export fails to collect them. Instead of collecting all
the parameter values into a array it simply override the value with
each value in turn.
* (PDB-228) Use JSON in terminus instead of PSON
The PuppetDB API specifies that it is JSON, so we should parse it as
that and not as PSON.
Some Puppet classes (Puppet::Node and Puppet::Node::Facts) don't
support JSON serialization, so continue to use PSON serialization
for them. In Puppet 3.4.0+ they have methods to do seralization in
other formats than PSON though, so once support for older versions
of Puppet is dropped they can be seralized in JSON as well.
* (PDB-476) Decorate the terminus code with Puppet profiling blocks
This patch adds some select profiling blocks to the PuppetDB terminus
code.
The profiler is provided by puppet core from
Puppet::Util::Puppetdb#profile,
which has recently become public for our use. We provide here in our own
utils
library our own wrapper implementation that can be mixed in.
Key areas of our terminus functionality have now been profiled with this
patch:
* Entry points are profiled and identified by their entry methods (save,
find,
search etc.)
* Remote calls, HTTP gets/posts
* Code that does any form of encoding/decoding that might be potentially
slow
at capacity.
The style of messages I've used follow along with the existing Puppet
profiling
examples already in place so as to be readable together. We have prefixed
our
profile message with "PuppetDB" for easy searchability also.
I have provided a small FAQ entry that explains in brief the process of
debugging, although we lack something to link to in Puppet for a more
detailed
explanation. This will probably need to be fixed if better documentation
comes
available.
* (PDB-472) - Annotate MQ messages without parsing payload
Received time and a UUID are currently added to incoming (via HTTP)
messages
before placing them on the queue. This commit adds those annotations to
the
MQ message header no longer requires parsing the incoming message payload
before placing it on the queue.
* (PDB-87) Port PuppetDB to TrapperKeeper
TrapperKeeper is a new container that PuppetDB will be deployed in.
This is mainly a refactoring of existing code and error handling to
use the centralized TrapperKeeper service. More information on
TrapperKeeper can be found here:
http://puppetlabs.com/blog/new-era-application-services-puppet-labs.
* (PDB-401) Upgrade to TrapperKeeper 0.3.4
This commit updates PuppetDB to use the new trapperkeeper 0.3.4
API. This includes:
* Slightly modified syntax for defining services and service
lifecycle behavior
* Switch from log4j to logback, update documentation and packaging
accordingly
* Switch from jetty7 to jetty9
* Add example of how to use "reloaded" interactive development
pattern
in REPL
* Upgrade to kitchensink 0.5.3, with bouncycastle fix for improved
HTTPS performance
* (PDB-529) Added latest-report? example to the events docs
* (PDB-512) Upgrade to Clojure 1.6.0
* (PDB-521) Switch to using /dev/urandom (using java.security.egd)
* (PDB-564) Added OpenBSD-specific variables to puppetdb.env
Adding OpenBSD specific variables allows the OpenBSD package
maintained downstream in the OpenBSD ports tree to be greatly
simplified.
* (PDB-177) Replace ssl-host default with 0.0.0.0
By trying to use a hostname, the amount of issues people suffer with
during
setup times related to hostname resolution is quite high. This patch
replaces the hostname with 0.0.0.0 which by default listens on all
interfaces.
* (PDB-402) Remove ahead-of-time compilation
This patch removes AOT compilation from our leiningen project and updates
all relevant shell scripts to use the non-AOT methodology for invoking
clojure projects.
* (PDB-576) Update beaker tests to use host.hostname instead of host.name
* (PDB-481) Added Arch Linux build/install support
* (PDB-572) New community packages of puppet install in vendorlibdir
* (PDB-575) Updated install from source docs
* (PDB-254) Change benchmark to mutate catalog resources and edges send a
defined number of messages
Previously benchmark.clj when it would mutate a catalog would only
add a single resource. This change will add a new resource or
mutate a random existing resource. It will also add a new or
change an existing edge. One of these four mutations is picked
at random. This commit also adds a new parameter to benchmark.clj
to allow the syncronous sending of a specified number of commands
(per host) via the -N argument
* (PDB-591) Allow gem source to come from env vars
* (PDB-595) Added docs for the load testing and benchmarking tool
Mostly a developer tool, but documented how to use it in case it's
useful to others. See "Load Testing" in the Usage / Admin section of
the docs site
* (PDB-602) Updated acceptance tests to use a proper release of leiningen
* (DOCUMENT-6) Update config page for PuppetDB module's improved settings
behavior
PuppetDB 2.0.0 Contributors
---------------------------
Alice Nodelman, Aric Gardner, Chris Price, Daniele Sluijters, Deepak
Giridharagopal, Erik Dalén, Jasper Lievisse Adriaanse, Ken Barber, Marius
Rieder, Matthaus Owens, Melissa Stone, Nate Wolfe, Nick Fagerlund, Niels
Abspoel, Riley Shott, Ryan Senior
PuppetDB 2.0.0 Changlog
-----------------------
Alice Nodelman (1):
974a19e (maint) have beaker tests use the host.hostname instead of
host.name
Aric Gardner (3):
d4a3773 added example for latest-report?
e549822 flubbed the initial operator.
6325e42 removed a space
Chris Price (15):
97755c5 Port PuppetDB to run on trapperkeeper
703699b Introduce `error-shutdown!` function for handling fatal errors
324d67a Add environment variable to allow SNAPSHOT dependencies
cac398c Update to released versions of deps, use new cli exceptions
e92f9ab Add clarifying comments to `bootstrap.cfg`
a62b6dd Remove SNAPSHOT dependency from test profile
6d50bbf Change trapperkeeper docs link to point to head of 0.1.x
branch
f2bfd00 Remove unused Jetty configuration
088da45 Remove unused dependency on ring-jetty-adapter
198129d Fix tk version string and test deps
1fadc84 Fix acceptance test race condition related to stop_puppetdb
61c9b2a Port to trapperkeeper 0.3.4
bf6b089 Add back in the placeholder log4j.properties.erb
3fe1191 Add `modify_config_setting` helper method for acceptance
c15ba49 Update CLI tools to use new form of specs for args
Daniele Sluijters (1):
b38ea55 PDB-521: Switch to using /dev/urandom
Deepak Giridharagopal (4):
59fa50a Fixes for Clojure 1.6 compatibility
228d854 (PDB-512) Update to Clojure 1.6.0-RC1
50309b6 Add jdk8 tests to travis-ci
29d0007 Allow gem source to come from env vars
Erik Dalén (3):
660522f (#23422) Use JSON in terminus instead of PSON
e3f3fbe (PDB-581) Add subqueries to events query endpoint
abb8f2f Fix command API Content-Type
Jasper Lievisse Adriaanse (1):
20efae8 [PDB-564] Add OpenBSD-specific variables to puppetdb.env
Ken Barber (51):
9da1078 PDB-177 Replace ssl-host default with 0.0.0.0
ef96659 PDB-177 Remove superflous fqdn lookups
b4b85ee Fix merge up mistake
f96946d PDB-234 Add v4, deprecate v2 query API
40c42e3 PDB-308 Drop 2.7.x support
424d104 PDB-308 Fix missing files from packaging
4acafb0 Fix puppetdb-env.erb so we don't get stripped carriage returns
9e6beab PDB-354 Deprecate old versions of commands
1b42c1d PDB-113 Remove swank
08d849a PDB-402 Remove ahead-of-time compilation
bfc45a0 PDB-476 Decorate the terminus code with Puppet profiling
blocks
295edd7 PDB-470 Provide new db setting 'statements-cache-size'
with a
default of 1000
15d68d8 PDB-473 Support POST using application/json data in the body
c789867 PDB-473 Add a content-type verification middleware
0b604cf PDB-473 Additional tests for verify-content-type and other
cleanups
ef2ea91 PDB-473 Convert submit-command-via-http! to use
application/json POST
1185307 PDB-473 Use (seq foo) on its own
9dd010b PDB-473 Adapt for new queue message format
4aaf0a2 Use Clojure 1.6.0-rc3
e03b4f0 Clojure 1.6.0-RC4
e5c3f72 Clojure 1.6.0 final
0685e8b PDB-453 Add environments to all terminii
619600e PDB-453 Normalize request.environment with to_s to avoid
'stack level too deep'
fa79af6 PDB-554 Switch to using new experimental ec2 beaker driver
63360a0 PDB-554 Remove temp refspec entry
856edea (PDB-526) Garbage collect unreferenced environments
f578757 (PDB-581) Add tests and minor cleanups for event subqueries
89ba090 (PDB-552) Pin support for Puppet to 3.5.1 and above
5c0d6f6 Use ActiveRecord 3.2.x instead of 2.3.x
6c9ad0d Use rubygem-sqlite3 for fedora 19/20
e34a661 Run storeconfig tests regardless
5be8845 Fix Fedora case statement and install sqlite3 from gem
4b98fea (PDB-594) Ensure libsqlite3-dev is installed
52df5c8 (PDB-594) Install builder to help with Ubuntu 10.04 rubygems
failure to follow dependencies
5ea8b27 (PDB-594) Manually install i18n for the sake of Ubuntu 10.04
old rubygems
1619cde (PDB-594) Confine legacy import/export scripts to avoid
Ubuntu 10.04
d4cf33d (PDB-567 & PDB-191) Use hash not config_version for report
export files
4821eeb (PDB-465) Remove Ubuntu 14.04 trust support for master
c334ef4 (PDB-555) Environments for /nodes & hyphenation consistency
fixes
42be0b4 (PDB-204) Ensure all commands no longer need a serialized
payload
05da2c4 (PDB-204) Fix terminus unit tests to avoid seralization for
facts/deactivate
0abbd53 (PDB-204) Fix submit command for terminus so it returns the
commands response
2daf865 (PDB-555) Rename *-last-environment to -environment
4c794b4 (PDB-204) Simplify variable destructuring and other minor
cleanups
c1743e8 (PDB-605) Prepare all documentation headers for 2.0 release
e55b34b (PDB-605) Revert some 1.6->2.0 changes
66d97b0 (PDB-605) Pin facter requirement to 1.7.0
ef8a2ca Fix environment end-point links to be compatible with our
docs site
a9a29b0 Wrap API examples in back-ticks to make the hyperlinks work
ded3e1e Fix example query to be JSON, not clojure
5ea3901 Correct hyperlink to anchor syntax in FAQ
Marius Rieder (1):
6952ac0 fix storeconfig export for array values
Matthaus Owens (1):
5d3e90b (PDB-592) Raring went EOL in Jan 2014, so we should no longer
build packages for it.
Melissa Stone (1):
3f34e3c (PDB-465) Add ubuntu-14.04, trusty tahr
Nate Wolfe (3):
0ea1949 Properly handle cli! errors
30bca1d Properly handle cli! --help
8720ff7 Hit actual endpoint to see if PuppetDB started
Nick Fagerlund (1):
ea700fd (DOCUMENT-6) Update config page for PuppetDB module's
improved settings behavior
Niels Abspoel (2):
35fcfed Aded Archlinux build/install support
5d9a2dc new community packages of puppet install in vendorlibdir
Riley Shott (2):
e16e308 Updated install from source commands
88ec143 Fixed typo.
Ryan Senior (45):
7818a59 (PDB-254) Change benchmark to mutate catalog resources and
edges send a defined number of messages
504431d Fixed report submission part of benchmark to change the
start/end time and config version
455b4fa (PDB-79) Drop support for Postgres < 8.4
3236907 Added Postgres 8.4 declaration to the acceptance test
PuppetDB package installation code
48e8fa1 Split metrics/version/command out of api.clj and into their
own namespaces
c01a774 (PDB-238) - Remove v1 API
a6f42af (PDB-271) Update docs to reflect removal of JDK 1.6 support
d88e89b (PDB-88) Fail on JDK version 1.6
173b9b2 Remove the option of a JDK 1.6 RPM dependency
a850666 Bump the redhat JDK dep for acceptance tests to 1.7.0
4e7ad25 (PDB-221) Add facts to import/export
4ed5aca Added support for exporting facts/reports when no catalog has
been persisted for the given node
6c9e4d3 PDB-469 - Support Anonymizing Facts
e1961d0 Tighten the fact-value anonymization for the moderate profile
f567003 (PDB-472) - Annotate MQ messages without parsing payload
fa8453f Fixed a date related issue causing tests to fail (only at
certain times of the day)
2585409 (PDB-452) Adding support for storing environments
fd6ed60 Minor doc changes
60aea48 Removed unused command attribute "puppetdb-version"
432dd02 (PDB-454) - Updating the catalogs endpoint to return
environments
b571324 Fix for the terminus to stringify version per spec, added
defensive conversion code in the back end.
b2cdeb0 Changed to covert catalog version to a string for wire
formats v1-v3. Also a minor doc update for the terminus.
1939ee3 (PDB-454) Added support for querying/returning environments
for resources and facts
dd75f69 Changed v2/v3 resource/fact API calls to remove environment
from the response
bf26afe Fixed issue with doseq over deftests in the events tests
20959ad (PDB-454) - Added environment support for querying reports,
events, event-counts and aggregate-event-counts
40e1d2d (PDB-574) 404 should be returned when querying for a missing
catalog
6a30c4c (PDB-457) - Update import/export/anonymization for v4 API
f9124ac Added tests around exporting storeconfigs with resources
containing arrays for param values.
bab2401 Only pull in activerecord and sqlite3 on new versions of Ruby
(1.9.3). This will cause the storeconfigs spec tests to only run on newer
Rubies.
2943316 Changed the storeconfig acceptance tests to verify import by
querying
3ea8a68 (PDB-457) More storeconfig acceptance test fixes
789802c (PDB-457) Storconfigs acceptance tests broken on Fedora
e9cd65a (PDB-570) Remove planetarium endpoint
93ef9d1 (PDB-595) Added docs for the load testing and benchmarking
tool
8ceed05 (PDB-589) Add indexes to environment foreign keys
91257df Bump catalog/report wire format versions for the benchmark
tool
ea4e576 (PDB-456) Added environments endpoint and related query
functions
3b320d4 Added a ring middleware for creating a db transaction and a
fixture for diagnosing routes that don't create a connection
dd3a64c Fixed inconsistencies in the deprecation warning messages of
commands
9c58a5b DRY'd up the environments docs a bit
f121f26 Updated acceptance tests to use a proper release of leiningen
3ca240f (PDB-557) Add support for standard query operators and
subqueries
aa95bca Updated the release notes for the 2.0.0 release
ff807b1 (PDB-626) Added some additional fixes/features to the 2.0.0
release notes
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAGDMwd2%2BZnF-QJ0K%3D1UytD3vrmV8CKdoHKkXrDe%2BdNZFruPybQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.