We run our puppet agents as daemons with runinterval set to 30m. This
works fine for the mostpart, but recently we took on three MariaDB
Galera cluster nodes. In order for the cluster to survive, at least one
node must be running at all times, i.e. you must not restart the MariaDB
service on all three nodes at the same time, e.g. by pushing a change to
the config file with puppet.
With the runinterval set to 30m, it is unlikely that all three nodes
will restart their MariaDB service at the same time but not impossible.
It would be a Bad Thing if I pushed a config change, all three nodes did
their puppet run at the same time, and the database cluster disappeared.
Is there a way of incorporating some sort of locking to ensure that only
one out of the three nodes can run puppet agent simultaneously? I have
half an idea based around using PuppetDB and something that checks a
variable in a run stage which is executed before main, which could cause
the run to abort if another node has locked a row.
I suppose it would be possible to disable the agent from running as a
daemon and use cron, and the cron job could easily use a MySQL handle as
a locking device. But it still doesn't stop me from simply sshing to
each of the nodes and forcing a puppet run, and breaking the cluster.
Has anyone done anything like this before? Hope to have some interesting
ideas from you all :)
Cheers,
Jonathan
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/536B87D0.4090402%40bristol.ac.uk.
For more options, visit https://groups.google.com/d/optout.
