Puppet users - Jan 2014 - puppet catalog compilation job queue idea

There's been an idea floating in my mind for quite a while now about using 
a job queue for compiling puppet catalogs. I just mentioned the idea on IRC 
and a few people really liked the idea, so I thought I'd bring it up here 
and get other thoughts on it.

The idea is that instead of puppet masters compiling catalogs on demand, 
they would operate out of a job queue:

   - When a node's cert is signed, the compilation queue gets a job for 
   that node.
   - When a compile job finishes, that node gets added to the queue at the 
   very end. This results in the puppet master constantly compiling catalogs.
   - When a catalog changes, the puppet master notifies the node that it 
   needs to fetch the updated catalog and do a puppet run.
   - When an exported resource changes, any nodes which collect that 
   resource should get their compilation jobs moved to the front of the queue. 
   (This should be optional, as the behavior might not be desired)
   - Nodes would still run puppet out of a cron job, but they would use the 
   cached catalog, not request one from the master.
   - If any of the facts used in the catalog change, the node would notify 
   the master and the compilation job would move to the front of the queue.

In the world of cloud computing, this becomes extremely beneficial, though 
it still is advantageous for traditional datacenter environments:

   - Puppet runs are computationally expensive for the client. With this 
   design, the client could have its cron job set to a very infrequent value 
   (once per hour or such). This way small instances without much available 
   resources wont waste them on puppet runs that don't change anything.
   - The masters could be very beefy instances with a lot of CPU. By 
   constantly generating catalogs, you ensure that the system's resources 
   aren't sitting idle and being wasted.
   - By using a queuing mechanism, you can determine how loaded the masters 
   are. If the oldest job in the queue is X minutes old, you can spin up 
   another master.
   - With the current method of generating catalogs on demand, if a lot of 
   nodes request a catalog at the same time, it can cause compilations to go 
   very slow. If they go too slow, the client will get a timeout. The client 
   will then go to the master again to request another catalog when the first 
   generation completed fine, it just took a while. With queuing, the master 
   can compile exactly X amount of catalogs at the same time. It can even be 
   configured to only start a new compilation job if the system load is less 
   than X.
   - Since puppet has to serve up both files and catalogs, if all the 
   available processes are used compiling catalogs, requests for files end up 
   hanging. By moving catalog compilations to a background worker, file 
   requests will be faster.
   (you can implement a workaround for this: I have 2 puppet master pools, 
   catalog requests get routed to one pool, everything else to the other pool, 
   but this isn't a simple or standard design)

Now most of this could be done on top of puppet. You could create a system 
which reads from a queue, performs a `puppet master compile`, notifies the 
client if it changes, etc. But there are a few sticky points which it 
wouldn't be able to do (just features, none would prevent the system from 
working):

   - There is no way of determining which facts were used in a catalog 
   compilation. A client could issue a catalog compilation request when any 
   fact changes, but facts like `uptime_seconds` always change, so it would 
   always result in a catalog compilation. The proper way to handle this would 
   be to monitor when a fact variable is read. Then add a list of "facts
used"
   and their value to the resulting catalog. Then the puppet agent can use 
   that to see if the facts used have changed.
   This may not be a significant issue though. If the `puppet agent` cron 
   job is set to something very infrequent, it won't be requesting catalog 
   compilations very often.
   - The catalog doesn't indicate whether a resource was collected from 
   another node. So when an exported resource changes, you wouldn't be able
to
   find nodes which collect those exported resources and recompile those 
   catalogs. Now this isn't a big deal since the current method of using
cron
   jobs can't even do this.
   The catalog does indicate that a resource is exported, so you could look 
   for nodes with resources of the same type & title as the exported
resource.
   But it's possible for a resource to have the same type & title of an 
   exported resource as long as the exported resource isn't collected during
   the same catalog compilation, so this method might end up recompiling 
   catalogs which don't use the exported resource.

Puppet could probably be monkey patched to address these and add the data 
to the catalog. The script system could then extract the info, while the 
puppet agent would just ignore the data.


Thoughts? I might be up for coding this, but it's would be low on the 
priority list and I wouldn't get to it for a long time.


-Patrick

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/a40c4cdb-766d-4cb6-957e-31594aed1371%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.