Max
2007-Apr-10 12:07 UTC
Wtih version 0.22.1 using SQLite storage, only resource -> host foreign keys are present
Hi, I am developing an end user WWW-based front end for Rails for my system integrator and developer colleagues to use to easily view and search our Puppet configs via a Rails app. I set up the SQLite storage option on the server and the puppet tables are being populated in /var/puppet/state/, but foreign keys for everything except host_id in resources are not being set. Is this expected behaviour with version 0.22.1? Makes the data in the DB much less useful without the parent child relationships :p. Using Rails 1.2.1 on the puppet master server. Regards, Max
David Schmitt
2007-Apr-10 14:56 UTC
Re: Wtih version 0.22.1 using SQLite storage, only resource -> host foreign keys are present
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tuesday 10 April 2007 14:07, Max wrote:> I am developing an end user WWW-based front end for Rails for my > system integrator and developer colleagues to use to easily view and > search our Puppet configs via a Rails app. > > I set up the SQLite storage option on the server and the puppet tables > are being populated in /var/puppet/state/, but foreign keys for > everything except host_id in resources are not being set. > > Is this expected behaviour with version 0.22.1? Makes the data in the > DB much less useful without the parent child relationships :p.The Rails stuff is very experimental and currently undergoing a major rewrite by Blake. See mail.madstop.com/pipermail/puppet-dev/2007-March/003371.html There also are quite a few other problems with the schema in 0.22.1, at least some (missing indices) are fixed in current SVN. Regards, David - -- - - hallo... wie gehts heute? - - *hust* gut *rotz* *keuch* - - gott sei dank kommunizieren wir über ein septisches medium ;) -- Matthias Leeb, Uni f. angewandte Kunst, 2005-02-15 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFGG6W6/Pp1N6Uzh0URAhNUAJ9xL3prV+xkXuTk4n9TMGDGy/47LgCfQh0C /nv/7DN1HzkVIAzDnXh/hDU=iDe4 -----END PGP SIGNATURE-----
Blake Barnett
2007-Apr-10 22:10 UTC
Re: Wtih version 0.22.1 using SQLite storage, only resource -> host foreign keys are present
On Apr 10, 2007, at 7:56 AM, David Schmitt wrote:> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On Tuesday 10 April 2007 14:07, Max wrote: >> I am developing an end user WWW-based front end for Rails for my >> system integrator and developer colleagues to use to easily view and >> search our Puppet configs via a Rails app. >> >> I set up the SQLite storage option on the server and the puppet >> tables >> are being populated in /var/puppet/state/, but foreign keys for >> everything except host_id in resources are not being set. >> >> Is this expected behaviour with version 0.22.1? Makes the data in >> the >> DB much less useful without the parent child relationships :p. > > The Rails stuff is very experimental and currently undergoing a > major rewrite > by Blake. See > mail.madstop.com/pipermail/puppet-dev/2007-March/003371.html > > There also are quite a few other problems with the schema in > 0.22.1, at least > some (missing indices) are fixed in current SVN. >This bothered me as well because I''m accessing the database from outside puppet too. This motivated me to add foreign key support to the webservice (which I''m currently calling PuppetStore but that can/ might change.) Thanks, -Blake
Luke Kanies
2007-Apr-10 23:16 UTC
Re: Wtih version 0.22.1 using SQLite storage, only resource -> host foreign keys are present
On Tue, April 10, 2007 9:56 am, David Schmitt wrote:>> The Rails stuff is very experimental and currently undergoing a major > rewrite > by Blake. See > mail.madstop.com/pipermail/puppet-dev/2007-March/003371.html > > There also are quite a few other problems with the schema in 0.22.1, at > least > some (missing indices) are fixed in current SVN.Yeah, the SVN version of the rails stuff is currently much, much faster than the released stuff, but as David says, its speed is still bad enough that I consider it experimental, at least partially because we keep refactoring it to try to fix the speed. -- Luke Kanies reductivelabs.com
Luke Kanies
2007-Apr-10 23:22 UTC
Re: Wtih version 0.22.1 using SQLite storage, only resource -> host foreign keys are present
On Tue, April 10, 2007 7:07 am, Max wrote:> Hi, > > I am developing an end user WWW-based front end for Rails for my > system integrator and developer colleagues to use to easily view and > search our Puppet configs via a Rails app.Have you seen PuppetShow (reductivelabs.com/trac/puppetshow)? It''d be great if you could help on that, rather than creating a new tool.> I set up the SQLite storage option on the server and the puppet tables > are being populated in /var/puppet/state/, but foreign keys for > everything except host_id in resources are not being set. > > Is this expected behaviour with version 0.22.1? Makes the data in the > DB much less useful without the parent child relationships :p. > > Using Rails 1.2.1 on the puppet master server.0.22.2 has significant changes to the Rails stuff, including adding lots of indices and making the Rails interface at least 3x faster (altho it''s still disappointingly slow). -- Luke Kanies reductivelabs.com
Max
2007-Apr-11 13:31 UTC
Re: Wtih version 0.22.1 using SQLite storage, only resource -> host foreign keys are present
Blake, Very cool. Thanks for the response. On 4/10/07, Blake Barnett <shadoi@nanovoid.com> wrote:> This bothered me as well because I''m accessing the database from > outside puppet too. This motivated me to add foreign key support to > the webservice (which I''m currently calling PuppetStore but that can/ > might change.) > > Thanks, > -Blake > _______________________________________________ > Puppet-users mailing list > Puppet-users@madstop.com > mail.madstop.com/mailman/listinfo/puppet-users >
Max
2007-Apr-11 13:37 UTC
Re: Wtih version 0.22.1 using SQLite storage, only resource -> host foreign keys are present
Hi Luke, On 4/10/07, Luke Kanies <luke@madstop.com> wrote:> Have you seen PuppetShow (reductivelabs.com/trac/puppetshow)? It''d > be great if you could help on that, rather than creating a new tool.Yes, happy to help, I had read about PuppetShow before and had misread what it does, I am definitely happy to help out.> 0.22.2 has significant changes to the Rails stuff, including adding lots > of indices and making the Rails interface at least 3x faster (altho it''s > still disappointingly slow).I will update to the .22.2 version today, I assume that using 0.22 on the server doesn''t require updating all cients to 0.22.2 as well. I will also install PuppetShow today and see what if anything I can do to help Blake with PuppetShow. Thank you, Max
Luke Kanies
2007-Apr-11 21:07 UTC
Re: Wtih version 0.22.1 using SQLite storage, only resource -> host foreign keys are present
On Wed, April 11, 2007 8:37 am, Max wrote:> > Yes, happy to help, I had read about PuppetShow before and had misread > what it does, I am definitely happy to help out.Great, glad to hear it.> I will update to the .22.2 version today, I assume that using 0.22 on > the server doesn''t require updating all cients to 0.22.2 as well.Correct, although you should upgrade the server to 0.22.3, as there were minor compatibility issues between 0.22.1 clients and 0.22.2 servers.> I will also install PuppetShow today and see what if anything I can do > to help Blake with PuppetShow.Great, we could really use both help and more eyes looking at the problem and helping us decide appropriate functionality. -- Luke Kanies reductivelabs.com
Blake Barnett
2007-Apr-11 22:28 UTC
Re: Wtih version 0.22.1 using SQLite storage, only resource -> host foreign keys are present
On Apr 11, 2007, at 2:07 PM, Luke Kanies wrote:> On Wed, April 11, 2007 8:37 am, Max wrote: >> >> Yes, happy to help, I had read about PuppetShow before and had >> misread >> what it does, I am definitely happy to help out. > > Great, glad to hear it. > >> I will update to the .22.2 version today, I assume that using 0.22 on >> the server doesn''t require updating all cients to 0.22.2 as well. > > Correct, although you should upgrade the server to 0.22.3, as there > were > minor compatibility issues between 0.22.1 clients and 0.22.2 servers. > >> I will also install PuppetShow today and see what if anything I >> can do >> to help Blake with PuppetShow. > > Great, we could really use both help and more eyes looking at the > problem > and helping us decide appropriate functionality.I just got PuppetStore to a point where I think it''s ready for testing. The way it currently works is this: 1. A client requests it''s config from puppetmaster 2. Puppetmaster checks if puppetstore is configured, if so it spawns a new thread and returns 3. The new thread calls storeconfig (lib/puppet/network/handler/ master.rb) 4. Storeconfig posts a YAML dump of data to puppetstore and includes the client hostname/ip in the URL params. 5. Puppet store chugs through the YAML and puts everything in the DB with pretty associations and junk. New configuration options: puppetstore = <true|false> puppetstorehost = <hostname> puppetstoreport = <port> I was thinking of adding a puppetstoreurl option too for those that wanted to customize it... #5 is currently taking around ~17 seconds to store a host the first time and ~10 seconds for updates roundtrip, but it doesn''t block the client at all. I''m sure We can speed this up significantly to by doing things in a more DB friendly way (ala collection_merge). I''m not sure if it''s best to commit the whole app? Keep it separate? -Blake (This should probably go to the dev list, but since this thread is here....)
Luke Kanies
2007-Apr-16 17:24 UTC
Re: Wtih version 0.22.1 using SQLite storage, only resource -> host foreign keys are present
On Apr 11, 2007, at 5:28 PM, Blake Barnett wrote:> > I just got PuppetStore to a point where I think it''s ready for > testing. The way it currently works is this: > > 1. A client requests it''s config from puppetmaster > 2. Puppetmaster checks if puppetstore is configured, if so it spawns > a new thread and returns > 3. The new thread calls storeconfig (lib/puppet/network/handler/ > master.rb) > 4. Storeconfig posts a YAML dump of data to puppetstore and includes > the client hostname/ip in the URL params. > 5. Puppet store chugs through the YAML and puts everything in the DB > with pretty associations and junk. > > New configuration options: > puppetstore = <true|false> > puppetstorehost = <hostname> > puppetstoreport = <port>Sounds good.> I was thinking of adding a puppetstoreurl option too for those that > wanted to customize it... > > #5 is currently taking around ~17 seconds to store a host the first > time and ~10 seconds for updates roundtrip, but it doesn''t block the > client at all. I''m sure We can speed this up significantly to by > doing things in a more DB friendly way (ala collection_merge).How does that compare to doing it inline as it works right now? That is, is this process now significantly slower overall but doesn''t block the client, or is it about the same speed and now doesn''t block the client? Is there any queuing or anything? We''ll still need to handle the case where 100 clients (or 1000) check in quickly, and if the webservices server is swamped we might start losing connections unless the configs are queued, right?> I''m not sure if it''s best to commit the whole app? Keep it separate?Are all of the tests passing, including the export/collect ones? I''m planning on releasing a 0.22.4 this week with very simple bugfixes, so I''d at least wait until after that. -- Someday I want to be rich. Some people get so rich they lose all respect for humanity. That''s how rich I want to be. --Rita Rudner --------------------------------------------------------------------- Luke Kanies | reductivelabs.com | madstop.com
Blake Barnett
2007-Apr-17 03:46 UTC
Re: Wtih version 0.22.1 using SQLite storage, only resource -> host foreign keys are present
Sorry for the late response, I''ve been in Vegas, on my way home tonight. On Apr 16, 2007, at 10:24 AM, Luke Kanies wrote:> On Apr 11, 2007, at 5:28 PM, Blake Barnett wrote: >> >> I just got PuppetStore to a point where I think it''s ready for >> testing. The way it currently works is this: >> >> 1. A client requests it''s config from puppetmaster >> 2. Puppetmaster checks if puppetstore is configured, if so it spawns >> a new thread and returns >> 3. The new thread calls storeconfig (lib/puppet/network/handler/ >> master.rb) >> 4. Storeconfig posts a YAML dump of data to puppetstore and includes >> the client hostname/ip in the URL params. >> 5. Puppet store chugs through the YAML and puts everything in the DB >> with pretty associations and junk. >> >> New configuration options: >> puppetstore = <true|false> >> puppetstorehost = <hostname> >> puppetstoreport = <port> > > Sounds good. > >> I was thinking of adding a puppetstoreurl option too for those that >> wanted to customize it... >> >> #5 is currently taking around ~17 seconds to store a host the first >> time and ~10 seconds for updates roundtrip, but it doesn''t block the >> client at all. I''m sure We can speed this up significantly to by >> doing things in a more DB friendly way (ala collection_merge). > > How does that compare to doing it inline as it works right now? That > is, is this process now significantly slower overall but doesn''t > block the client, or is it about the same speed and now doesn''t block > the client?Well, I''m not going to be over-enthusiastic until others run it, but from what I can see, it''s faster and doesn''t block the client. I haven''t gone to the extent that collection_merge does yet though.> > Is there any queuing or anything? We''ll still need to handle the > case where 100 clients (or 1000) check in quickly, and if the > webservices server is swamped we might start losing connections > unless the configs are queued, right?Because ActiveRecord has complete control of the DB connections, as far as I know, our max is whatever the default for the database engine is. (MySQL on Debian is 100 I believe). I''ll setup a stress tester next week to see what concurrency does to it.> >> I''m not sure if it''s best to commit the whole app? Keep it separate? > > Are all of the tests passing, including the export/collect ones? > > I''m planning on releasing a 0.22.4 this week with very simple > bugfixes, so I''d at least wait until after that.I haven''t had a chance to add the collection code yet, so it''s nowhere near ready. I wanted to talk to you about where/when the best place to do that is. -Blake
Luke Kanies
2007-Apr-17 04:34 UTC
Re: Wtih version 0.22.1 using SQLite storage, only resource -> host foreign keys are present
On Apr 16, 2007, at 10:46 PM, Blake Barnett wrote:>> >> How does that compare to doing it inline as it works right now? That >> is, is this process now significantly slower overall but doesn''t >> block the client, or is it about the same speed and now doesn''t block >> the client? > > Well, I''m not going to be over-enthusiastic until others run it, but > from what I can see, it''s faster and doesn''t block the client. I > haven''t gone to the extent that collection_merge does yet though.Ok. I just remember you saying your storage times were consistently around 2 seconds (this was a while ago), so seeing you now see 10-17 seconds made me think this might be a lot slower.>> Is there any queuing or anything? We''ll still need to handle the >> case where 100 clients (or 1000) check in quickly, and if the >> webservices server is swamped we might start losing connections >> unless the configs are queued, right? > > Because ActiveRecord has complete control of the DB connections, as > far as I know, our max is whatever the default for the database > engine is. (MySQL on Debian is 100 I believe). I''ll setup a stress > tester next week to see what concurrency does to it.Okay, great.> I haven''t had a chance to add the collection code yet, so it''s > nowhere near ready. I wanted to talk to you about where/when the > best place to do that is.Seems like a branch is a good idea. -- I have an answering machine in my car. It says, "I''m home now. But leave a message and I''ll call when I''m out. -- Stephen Wright --------------------------------------------------------------------- Luke Kanies | reductivelabs.com | madstop.com
Blake Barnett
2007-Apr-17 17:27 UTC
Re: Wtih version 0.22.1 using SQLite storage, only resource -> host foreign keys are present
On Apr 16, 2007, at 9:34 PM, Luke Kanies wrote:> On Apr 16, 2007, at 10:46 PM, Blake Barnett wrote: >>> >>> How does that compare to doing it inline as it works right now? >>> That >>> is, is this process now significantly slower overall but doesn''t >>> block the client, or is it about the same speed and now doesn''t >>> block >>> the client? >> >> Well, I''m not going to be over-enthusiastic until others run it, but >> from what I can see, it''s faster and doesn''t block the client. I >> haven''t gone to the extent that collection_merge does yet though. > > Ok. I just remember you saying your storage times were consistently > around 2 seconds (this was a while ago), so seeing you now see 10-17 > seconds made me think this might be a lot slower.That was to store a host in the test tree. Actual host storage in production was taking from 60 to 125 seconds. I had to disable it during 0.22.1 though, so I haven''t been testing the latest changes to the internal rails code in production (shouldn''t have to begin with ;)). -Blake