Puppet users - May 2007 - Anyone using storeconfigs with a DB other than sqlite?

Just wondering if anyone is using a backend database other than sqlite,
and if so, how easy/hard it was to configure.

I''m constantly receiving "SQLite3::BusyException: database is
locked..."
which I presume is due to sqlite''s relatively coarse locking method.

Cheers,

James

--
Senior Linux Platform Engineer
Midrange Services
AXA Technology Services - Asia Pacific

*********************************************************************************
Important Note
This email (including any attachments) contains information which is 
confidential and may be subject to legal privilege.  If you are not 
the intended recipient you must not use, distribute or copy this 
email.  If you have received this email in error please notify the 
sender immediately and delete this email. Any views expressed in this 
email are not necessarily the views of AXA-Tech Australia.   Thank you.
**********************************************************************************

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Monday 28 May 2007, HARRIS Jimmy (AXA-Tech-AU) wrote:
> Just wondering if anyone is using a backend database other than sqlite,
> and if so, how easy/hard it was to configure.
>
> I''m constantly receiving "SQLite3::BusyException: database is
locked..."
> which I presume is due to sqlite''s relatively coarse locking
method.
I did try it with postgres back around 0.22.0 -- which was a breeze to 
setup -- but it didn''t help with the locking stuff. If you
don''t need
export/collect features, you can just disable the "storeconfig"
setting.



Regards, David
- -- 
- - hallo... wie gehts heute?
- - *hust* gut *rotz* *keuch*
- - gott sei dank kommunizieren wir über ein septisches medium ;)
 -- Matthias Leeb, Uni f. angewandte Kunst, 2005-02-15
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFGWrJ7/Pp1N6Uzh0URApwsAJ0TijcS++4mTe3eUUi0l2LTG5Zb1ACfUYBB
+OBu4JDXi/86HxpUfdbHZsM=EiCT
-----END PGP SIGNATURE-----

On May 27, 2007, at 8:28 PM, HARRIS Jimmy ((AXA-Tech-AU)) wrote:
> Just wondering if anyone is using a backend database other than  
> sqlite,
> and if so, how easy/hard it was to configure.
It''s very easy to configure -- set up the dbadapter, dbserver,  
dbuser, and dbpassword, then start the master once with --migrate.
> I''m constantly receiving "SQLite3::BusyException: database is
> locked..."
> which I presume is due to sqlite''s relatively coarse locking
method.
I would have assumed the same thing, and David''s saying the problem  
persisted with postgres is worrying.  I can''t imagine that postgres  
or mysql would have locking problems.

  --
  Somebody has to do something, and it''s just incredibly pathetic  
that it
  has to be us.       --Jerry Garcia
  ---------------------------------------------------------------------
  Luke Kanies | http://reductivelabs.com | http://madstop.com

> > Just wondering if anyone is using a backend database other than 
> > sqlite, and if so, how easy/hard it was to configure.
> 
> It''s very easy to configure -- set up the dbadapter, 
> dbserver, dbuser, and dbpassword, then start the master once 
> with --migrate.
Where do I configure these settings?  Is it within Puppet or does it use
a more ROR-oriented location?
> > I''m constantly receiving "SQLite3::BusyException:
database is
> > locked..."
> > which I presume is due to sqlite''s relatively coarse locking
method.
> 
> I would have assumed the same thing, and David''s saying the 
> problem persisted with postgres is worrying.  I can''t imagine 
> that postgres or mysql would have locking problems.
I wouldn''t have thought they''d have locking problems either. 
I''ll give
MySQL a go and report back.  At the moment, I''m using Puppet to manage
only about fifteen servers so it''s not like there are constand
check-ins.

*********************************************************************************
Important Note
This email (including any attachments) contains information which is 
confidential and may be subject to legal privilege.  If you are not 
the intended recipient you must not use, distribute or copy this 
email.  If you have received this email in error please notify the 
sender immediately and delete this email. Any views expressed in this 
email are not necessarily the views of AXA-Tech Australia.   Thank you.
**********************************************************************************

On May 28, 2007, at 6:48 PM, HARRIS Jimmy ((AXA-Tech-AU))
wrote:
>
> Where do I configure these settings?  Is it within Puppet or does  
> it use
> a more ROR-oriented location?
The same way you''d configure any other Puppet parameters -- -- 
dbadapter, or in puppetmasterd.conf.
> I wouldn''t have thought they''d have locking problems
either.  I''ll
> give
> MySQL a go and report back.  At the moment, I''m using Puppet to
manage
> only about fifteen servers so it''s not like there are constand
> check-ins.
Yeah, please let us know.  I''m interested in hearing about storage  
times, too, and Allen Ballman is working on rearchitecting the  
database schemas to hopefully drastically reduce the times here, so  
it''d be great if you were willing to test when that arrives.

  --
  If two men agree on everything, you may be sure that one of them is
  doing the thinking.    -- Lyndon B. Johnson
  ---------------------------------------------------------------------
  Luke Kanies | http://reductivelabs.com | http://madstop.com

Hy there,
here are my experiences with storeconfigs:
On 29.05.2007, at 05:54, Luke Kanies wrote:
> On May 28, 2007, at 6:48 PM, HARRIS Jimmy ((AXA-Tech-AU)) wrote:
>> I wouldn''t have thought they''d have locking problems
either.  I''ll
>> give
>> MySQL a go and report back.  At the moment, I''m using Puppet
to
>> manage
>> only about fifteen servers so it''s not like there are constand
>> check-ins.
>
> Yeah, please let us know.  I''m interested in hearing about storage
> times, too, and Allen Ballman is working on rearchitecting the
> database schemas to hopefully drastically reduce the times here, so
> it''d be great if you were willing to test when that arrives.
After enabling storeconfigs, our puppetmaster became really  
unresponsive. OK, it is not the most recent machine, (dual PII @ 450  
Mhz with 512Mbs of Ram and IDE (udma33) Software RAID5). I was  
wondering, why it took a hell of a long time to compile the  
configuration of a node. Suddenly it was about 300 seconds and more.  
After looking into the puppetmaster''s logs, I saw that a lot  of  
messages about the DB being locked were there.
Anyway, after switching to mysql (which runs on the same machine),  
The whole matter became a lot better.
Now, the second processor is used by mysql, this enables the main  
puppemaster process to react faster. Everything is fine again.
It seems to me that sqlite is really slow. Mysql works great as it  
seems.

Have a nice day,
udo.
-- 
---[ Institute of Cognitive Science @ University of Osnabrueck
---[ Albrechtstrasse 28, D-49076 Osnabrueck, 969-3362
---[ Eyes: http://www.zoide.net/
---[ Ears: http://www.auriculabovinari.de/

I decided to give mysql a try.

Luke Kanies wrote:
> On May 27, 2007, at 8:28 PM, HARRIS Jimmy ((AXA-Tech-AU)) wrote:
> 
>> Just wondering if anyone is using a backend database other than  
>> sqlite,
>> and if so, how easy/hard it was to configure.
> 
> It''s very easy to configure -- set up the dbadapter, dbserver,  
> dbuser, and dbpassword, then start the master once with --migrate.
Unfortunately, --migrate is not a valid option:

/usr/sbin/puppetmasterd: unrecognized option `--migrate''

this is with 0.22.4, i can probably live without the migration, but thought I
should point it out.

micah

On Jun 1, 2007, at 10:48 PM, Micah Anderson wrote:
>
> Unfortunately, --migrate is not a valid option:
>
> /usr/sbin/puppetmasterd: unrecognized option `--migrate''
>
> this is with 0.22.4, i can probably live without the migration, but  
> thought I
> should point it out.
Sorry, --dbmigrate.

  --
  The death rate on Earth is: .... (computing) .... One per person.
  ---------------------------------------------------------------------
  Luke Kanies | http://reductivelabs.com | http://madstop.com

Yesterday I got puppet 0.22.4 up and running with store configs on mysql 5.0 and
every thing is working beautifully.

The only issue I ran into was the error "value is a required option for
Puppet::Parser::Resource::Param".  I was able to resolve the error by
truncating all of the tables in the puppet database. I''m assuming that
my first pass resulted in the creation of some invalid virtual resources.

Is there an easy way to track down and remove the specific entry that would be
causing this kind of error in order to remove it without erasing everything?

On 6/2/07, Luke Kanies <luke@madstop.com> wrote:
> On Jun 1, 2007, at 10:48 PM, Micah Anderson wrote:
> >
> > Unfortunately, --migrate is not a valid option:
> >
> > /usr/sbin/puppetmasterd: unrecognized option `--migrate''
> >
> > this is with 0.22.4, i can probably live without the migration, but
> > thought I
> > should point it out.
> 
> Sorry, --dbmigrate.
> 
>   --
>   The death rate on Earth is: .... (computing) .... One per person.
>   ---------------------------------------------------------------------
>   Luke Kanies | http://reductivelabs.com | http://madstop.com
> 
> 
> _______________________________________________
> Puppet-users mailing list
> Puppet-users@madstop.com
> https://mail.madstop.com/mailman/listinfo/puppet-users
>

On Jun 7, 2007, at 1:26 PM, Brian Finney wrote:
> Yesterday I got puppet 0.22.4 up and running with store configs on  
> mysql 5.0 and every thing is working beautifully.
>
> The only issue I ran into was the error "value is a required option  
> for Puppet::Parser::Resource::Param".  I was able to resolve the  
> error by truncating all of the tables in the puppet database. I''m
> assuming that my first pass resulted in the creation of some  
> invalid virtual resources.
>
> Is there an easy way to track down and remove the specific entry  
> that would be causing this kind of error in order to remove it  
> without erasing everything?
Hmm, it shouldn''t normally be possible to create invalid resources,  
so I haven''t spent time making it easy to track them down.

If this happens consistently, I''ll add information to the errors to  
make sure that it''s clear what''s failing.

  --
  The great tragedy of Science - the slaying of a beautiful  
hypothesis by
  an ugly fact.   --Thomas H. Huxley
  ---------------------------------------------------------------------
  Luke Kanies | http://reductivelabs.com | http://madstop.com

Same thing here,
from time to time (I can not tell when or why), the databse gets  
corrupted, and then it must be truncated.
My guess is, that some values an be entered as ''NULL'', but
puppet
wants them not to be null when they are read.
I saw some days ago that some timestamps of the entries were NULL.  
Maybe those are the source of error?
udo.


On 07.06.2007, at 20:26, Brian Finney wrote:
> Yesterday I got puppet 0.22.4 up and running with store configs on  
> mysql 5.0 and every thing is working beautifully.
>
> The only issue I ran into was the error "value is a required option  
> for Puppet::Parser::Resource::Param".  I was able to resolve the  
> error by truncating all of the tables in the puppet database. I''m
> assuming that my first pass resulted in the creation of some  
> invalid virtual resources.
>
> Is there an easy way to track down and remove the specific entry  
> that would be causing this kind of error in order to remove it  
> without erasing everything?
>
> On 6/2/07, Luke Kanies <luke@madstop.com> wrote:
>> On Jun 1, 2007, at 10:48 PM, Micah Anderson wrote:
>>>
>>> Unfortunately, --migrate is not a valid option:
>>>
>>> /usr/sbin/puppetmasterd: unrecognized option `--migrate''
>>>
>>> this is with 0.22.4, i can probably live without the migration, but
>>> thought I
>>> should point it out.
>>
>> Sorry, --dbmigrate.
>>
>>   --
>>   The death rate on Earth is: .... (computing) .... One per person.
>>    
>> ---------------------------------------------------------------------
>>   Luke Kanies | http://reductivelabs.com | http://madstop.com
>>
>>
>> _______________________________________________
>> Puppet-users mailing list
>> Puppet-users@madstop.com
>> https://mail.madstop.com/mailman/listinfo/puppet-users
>>
> _______________________________________________
> Puppet-users mailing list
> Puppet-users@madstop.com
> https://mail.madstop.com/mailman/listinfo/puppet-users
>
-- 
---[ Institute of Cognitive Science @ University of Osnabrueck
---[ Albrechtstrasse 28, D-49076 Osnabrueck, 969-3362
---[ Eyes: http://www.zoide.net/
---[ Ears: http://www.auriculabovinari.de/

On Jun 13, 2007, at 3:06 AM, udo waechter wrote:
> Same thing here,
> from time to time (I can not tell when or why), the databse gets
> corrupted, and then it must be truncated.
> My guess is, that some values an be entered as ''NULL'',
but puppet
> wants them not to be null when they are read.
> I saw some days ago that some timestamps of the entries were NULL.
> Maybe those are the source of error?
Most of the direct database interaction has now been rewritten by  
Allen Ballman, and we''re just finishing the last bit of query  
optimization and it should make it into the impending release.

Since I don''t know the source of this problem, I can''t promise
that
the new code fixes it, but I certainly hope so.

If it crops up again with the new code, please bring it up again and  
we''ll track it down then.

  --
  There is nothing worse than aggressive stupidity.
      -- Johann Wolfgang von Goethe
  ---------------------------------------------------------------------
  Luke Kanies | http://reductivelabs.com | http://madstop.com

Thats awesome, sounds like this will likely solve the slow queries I
was going to investigate today.

I''ll be more than happy to bang it around when it''s ready.

Thanks
Brian

On 6/13/07, Luke Kanies <luke@madstop.com> wrote:
> On Jun 13, 2007, at 3:06 AM, udo waechter wrote:
>
> > Same thing here,
> > from time to time (I can not tell when or why), the databse gets
> > corrupted, and then it must be truncated.
> > My guess is, that some values an be entered as
''NULL'', but puppet
> > wants them not to be null when they are read.
> > I saw some days ago that some timestamps of the entries were NULL.
> > Maybe those are the source of error?
>
> Most of the direct database interaction has now been rewritten by
> Allen Ballman, and we''re just finishing the last bit of query
> optimization and it should make it into the impending release.
>
> Since I don''t know the source of this problem, I can''t
promise that
> the new code fixes it, but I certainly hope so.
>
> If it crops up again with the new code, please bring it up again and
> we''ll track it down then.
>
>   --
>   There is nothing worse than aggressive stupidity.
>       -- Johann Wolfgang von Goethe
>   ---------------------------------------------------------------------
>   Luke Kanies | http://reductivelabs.com | http://madstop.com
>
>
> _______________________________________________
> Puppet-users mailing list
> Puppet-users@madstop.com
> https://mail.madstop.com/mailman/listinfo/puppet-users
>