Matthew Nicholson
2012-Jul-09 17:41 UTC
[Puppet Users] puppetdb = rise in exection expired notices?
I''m getting more and more "execution expired" as systems checking and hit puppetdb for the first time (switching from a mysql instance). The command queue isn''t long (1-5, if anything, all the time), and ym master itself seems to be dealing well enough. I have seen the collection time growing higher and higher though. This is a ~2K node deployment, and one of the few things storedconfigs is used for is ssh-key collection/exports (basically collect from 90% of there hosts and export to the same 90%). mysql seemed to deal with this fine... I''m hesitant to tune puppetdb in terms of threads and heap size, as both of those SEEM okay right now...would either affect collection query time? -- Matthew Nicholson -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Walter Heck
2012-Jul-09 18:03 UTC
Re: [Puppet Users] puppetdb = rise in exection expired notices?
I tried out puppetdb a few weeks ago, and I remember reading somewhere that for larger deployments it is recommended to use postgres as the backend for puppetdb for any installations of size. At http://docs.puppetlabs.com/puppetdb/0.9/requirements.html it says: "Deployments with more than 100 nodes should configure a PostgreSQL database for PuppetDB. Smaller deployments may also wish to use the PostgreSQL backend.There are two available backends for PuppetDB storage:PuppetDB’s embedded databasePostgreSQLThe embedded database works well for small deployments (up to approximately 100 hosts). It requires no additional daemons or setup, and as such is very simple to get started with. It supports all PuppetDB features.However, there is a cost: the embedded database requires a fair amount of RAM to operate correctly. We’d recommend allocating 1GB to PuppetDB as a starting point. Additionally, the embedded database is somewhat opaque; unlike more off-the-shelf database daemons, there isn’t much companion tooling for things like interactive SQL consoles, performance analysis, or backups.That said, if you have a small installation and enough RAM, then the embedded database will work just fine.For most “real” use, we recommend running an instance of PostgreSQL. Simply install PostgreSQL using a module from the Puppet Forge or your local package manager, create a new (empty) database for PuppetDB, and verify that you can login via psql to this DB you just created. Then just supply PuppetDB with the DB host, port, name, and credentials you’ve just configured, and we’ll take care of the rest!" Hope this helps! On Jul 10, 2012 1:41 AM, "Matthew Nicholson" <matthew.a.nicholson@gmail.com> wrote:> I''m getting more and more "execution expired" as systems checking and > hit puppetdb for the first time (switching from a mysql instance). The > command queue isn''t long (1-5, if anything, all the time), and ym > master itself seems to be dealing well enough. I have seen the > collection time growing higher and higher though. This is a ~2K node > deployment, and one of the few things storedconfigs is used for is > ssh-key collection/exports (basically collect from 90% of there hosts > and export to the same 90%). > > mysql seemed to deal with this fine... > > I''m hesitant to tune puppetdb in terms of threads and heap size, as > both of those SEEM okay right now...would either affect collection > query time? > > -- > Matthew Nicholson > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Matthew Nicholson
2012-Jul-09 18:08 UTC
Re: [Puppet Users] puppetdb = rise in exection expired notices?
Sorry, I wasn''t clear, this is with Ppostgres as the DB backend. Just upped the db servers kernel.shmmax value and postgres''s shared_buffers values, seems to be helping thus far... On Mon, Jul 9, 2012 at 2:03 PM, Walter Heck <walterheck@gmail.com> wrote:> I tried out puppetdb a few weeks ago, and I remember reading somewhere that > for larger deployments it is recommended to use postgres as the backend for > puppetdb for any installations of size. > > At http://docs.puppetlabs.com/puppetdb/0.9/requirements.html it says: > "Deployments with more than 100 nodes should configure a PostgreSQL database > for PuppetDB. Smaller deployments may also wish to use the PostgreSQL > backend.There are two available backends for PuppetDB storage:PuppetDB’s > embedded databasePostgreSQLThe embedded database works well for small > deployments (up to approximately 100 hosts). It requires no additional > daemons or setup, and as such is very simple to get started with. It > supports all PuppetDB features.However, there is a cost: the embedded > database requires a fair amount of RAM to operate correctly. We’d recommend > allocating 1GB to PuppetDB as a starting point. Additionally, the embedded > database is somewhat opaque; unlike more off-the-shelf database daemons, > there isn’t much companion tooling for things like interactive SQL consoles, > performance analysis, or backups.That said, if you have a small installation > and enough RAM, then the embedded database will work just fine.For most > “real” use, we recommend running an instance of PostgreSQL. Simply install > PostgreSQL using a module from the Puppet Forge or your local package > manager, create a new (empty) database for PuppetDB, and verify that you can > login via psql to this DB you just created. Then just supply PuppetDB with > the DB host, port, name, and credentials you’ve just configured, and we’ll > take care of the rest!" > > Hope this helps! > > On Jul 10, 2012 1:41 AM, "Matthew Nicholson" <matthew.a.nicholson@gmail.com> > wrote: >> >> I''m getting more and more "execution expired" as systems checking and >> hit puppetdb for the first time (switching from a mysql instance). The >> command queue isn''t long (1-5, if anything, all the time), and ym >> master itself seems to be dealing well enough. I have seen the >> collection time growing higher and higher though. This is a ~2K node >> deployment, and one of the few things storedconfigs is used for is >> ssh-key collection/exports (basically collect from 90% of there hosts >> and export to the same 90%). >> >> mysql seemed to deal with this fine... >> >> I''m hesitant to tune puppetdb in terms of threads and heap size, as >> both of those SEEM okay right now...would either affect collection >> query time? >> >> -- >> Matthew Nicholson >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To post to this group, send email to puppet-users@googlegroups.com. >> To unsubscribe from this group, send email to >> puppet-users+unsubscribe@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/puppet-users?hl=en. >> > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en.-- Matthew Nicholson -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Deepak Giridharagopal
2012-Jul-09 20:59 UTC
Re: [Puppet Users] puppetdb = rise in exection expired notices?
FYI, we''ve been debugging this on IRC. We believe the issues were that there were insufficient system resources allocated to the database, and that frequent queries from the puppetdb dashboard itself were causing load spikes. Allocating more shared_buffers to the database and reducing the dashboard''s polling interval (via the "pollingInterval" URL argument) appears sufficient. deepak On Mon, Jul 09, 2012 at 02:08:13PM -0400, Matthew Nicholson <matthew.a.nicholson@gmail.com> wrote:> Sorry, I wasn''t clear, this is with Ppostgres as the DB backend. Just > upped the db servers kernel.shmmax value and postgres''s shared_buffers > values, seems to be helping thus far... > > On Mon, Jul 9, 2012 at 2:03 PM, Walter Heck <walterheck@gmail.com> wrote: > > I tried out puppetdb a few weeks ago, and I remember reading somewhere that > > for larger deployments it is recommended to use postgres as the backend for > > puppetdb for any installations of size. > > > > At http://docs.puppetlabs.com/puppetdb/0.9/requirements.html it says: > > "Deployments with more than 100 nodes should configure a PostgreSQL database > > for PuppetDB. Smaller deployments may also wish to use the PostgreSQL > > backend.There are two available backends for PuppetDB storage:PuppetDB’s > > embedded databasePostgreSQLThe embedded database works well for small > > deployments (up to approximately 100 hosts). It requires no additional > > daemons or setup, and as such is very simple to get started with. It > > supports all PuppetDB features.However, there is a cost: the embedded > > database requires a fair amount of RAM to operate correctly. We’d recommend > > allocating 1GB to PuppetDB as a starting point. Additionally, the embedded > > database is somewhat opaque; unlike more off-the-shelf database daemons, > > there isn’t much companion tooling for things like interactive SQL consoles, > > performance analysis, or backups.That said, if you have a small installation > > and enough RAM, then the embedded database will work just fine.For most > > “real” use, we recommend running an instance of PostgreSQL. Simply install > > PostgreSQL using a module from the Puppet Forge or your local package > > manager, create a new (empty) database for PuppetDB, and verify that you can > > login via psql to this DB you just created. Then just supply PuppetDB with > > the DB host, port, name, and credentials you’ve just configured, and we’ll > > take care of the rest!" > > > > Hope this helps! > > > > On Jul 10, 2012 1:41 AM, "Matthew Nicholson" <matthew.a.nicholson@gmail.com> > > wrote: > >> > >> I''m getting more and more "execution expired" as systems checking and > >> hit puppetdb for the first time (switching from a mysql instance). The > >> command queue isn''t long (1-5, if anything, all the time), and ym > >> master itself seems to be dealing well enough. I have seen the > >> collection time growing higher and higher though. This is a ~2K node > >> deployment, and one of the few things storedconfigs is used for is > >> ssh-key collection/exports (basically collect from 90% of there hosts > >> and export to the same 90%). > >> > >> mysql seemed to deal with this fine... > >> > >> I''m hesitant to tune puppetdb in terms of threads and heap size, as > >> both of those SEEM okay right now...would either affect collection > >> query time? > >> > >> -- > >> Matthew Nicholson > >> > >> -- > >> You received this message because you are subscribed to the Google Groups > >> "Puppet Users" group. > >> To post to this group, send email to puppet-users@googlegroups.com. > >> To unsubscribe from this group, send email to > >> puppet-users+unsubscribe@googlegroups.com. > >> For more options, visit this group at > >> http://groups.google.com/group/puppet-users?hl=en. > >> > > -- > > You received this message because you are subscribed to the Google Groups > > "Puppet Users" group. > > To post to this group, send email to puppet-users@googlegroups.com. > > To unsubscribe from this group, send email to > > puppet-users+unsubscribe@googlegroups.com. > > For more options, visit this group at > > http://groups.google.com/group/puppet-users?hl=en. > > > > -- > Matthew Nicholson > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. >-- Deepak Giridharagopal / Puppet Labs / grim_radical -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.