Puppet users - Jun 2010 - puppetqd and remote mysql server

Has anyone been using this in production to store facts remotely in a
distributed environment?  I have been testing it in my lab where latency and
bandwith to a remote mysql server is very good so it works well.  But if I
attempt to use it across a wan with ~150ms puppetqd seems to die.  If I
don''t have it running the stompserver/activemq looks to be in the path
but
trying to debug why puppetqd would fail.

Also where is it logging?  I don''t see anything for it in
/var/log/messages
or with apache..



-Chris

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Think I found the issue from an strace, the rubygems-sqlit3 rpm was
missing.. still curious on where the logs end up for it though.  The fact
that its using sqllite also doesn''t help as I think I am running into
locking issues if I try to run a cross environment update.

How to deal with sqllite locking as well as I see this quite a bit if I am
using puppetqd/mysql combination or a local sqlite setup (without the
queing).

Got 1 failure(s) while initializing: change from 664 to 660 failed: failed
to chmod /var/lib/puppet/state/clientconfigs.sqlite3: Operation not
permitted - /var/lib/puppet/state/clientconfigs.sqlite3


On Mon, Jun 7, 2010 at 3:22 PM, Christopher Johnston
<chjohnst@gmail.com>wrote:
> Has anyone been using this in production to store facts remotely in a
> distributed environment?  I have been testing it in my lab where latency
and
> bandwith to a remote mysql server is very good so it works well.  But if I
> attempt to use it across a wan with ~150ms puppetqd seems to die.  If I
> don''t have it running the stompserver/activemq looks to be in the
path but
> trying to debug why puppetqd would fail.
>
> Also where is it logging?  I don''t see anything for it in
/var/log/messages
> or with apache..
>
>
>
> -Chris
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Jun 7, 1:00 pm, Christopher Johnston <chjoh...@gmail.com>
wrote:
> Think I found the issue from an strace, the rubygems-sqlit3 rpm was
> missing.. still curious on where the logs end up for it though.  The fact
> that its using sqllite also doesn''t help as I think I am running
into
> locking issues if I try to run a cross environment update.

Chris, were you using sqlite or mysql as the backend store when you
had issues?

Evaluating remote sites with central storeconfigs is on the todo list
for me. Currently have it working just peachy with mysql, straight
storeconfig no puppetqd.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

I have about a dozen sites with puppet masters in each.  They are
geographically located with some slow network links between them.  The setup
I was evaluating is to have a centralized stored config server running mysql
with all of my remote masters sending their locations data back to the
central server.  Now on my fast links remote connectivity is OK if I just
point a master right to the database (eliminate puppetqd, etc), albeit still
slower then if I were to have it local but I was hoping I could use a
stomp+puppetqd to handle caching and asyncronously update my central mysql
database.  In my tests this seems to work however it looks like puppetdq is
using sqllite to keep a cached copy.  If I have 100 servers and I am doing a
mass update  during maintenance window I am seeing timeouts once we hit 5-6
servers which is likely sqllite.

Perhaps this solution is not the most viable right now?  I am OK with
running a local mysql instance instead of sqllite if it is supported but I
would like to get all of my stored config data in one location (using facter
data, reporting is of great value to us).  This would be useful as we can
plugin foreman on top of that database for a central dashboard instead of
having dozens of dashboards for each location.

-Chris

On Tue, Jun 8, 2010 at 2:40 PM, donavan <donavan@desinc.net> wrote:
>
> On Jun 7, 1:00 pm, Christopher Johnston <chjoh...@gmail.com> wrote:
> > Think I found the issue from an strace, the rubygems-sqlit3 rpm was
> > missing.. still curious on where the logs end up for it though.  The
fact
> > that its using sqllite also doesn''t help as I think I am
running into
> > locking issues if I try to run a cross environment update.
>
>
> Chris, were you using sqlite or mysql as the backend store when you
> had issues?
>
> Evaluating remote sites with central storeconfigs is on the todo list
> for me. Currently have it working just peachy with mysql, straight
> storeconfig no puppetqd.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
>
puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com>
> .
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

I evaluated this scenario a couple of years ago (before puppetq), and it was
never a feasible solution due to a. latency (e.g. everything is much
slower), b. if wan goes down, your master wont work.

One of the reasons that I''ve developed Foreman, is to overcome this
problem,
the way I do it is:
- I keep one central database, but don''t use it directly with puppet,
rather
import the data i need (e.g. for facts see the import_facts script in
foreman extras dir)
- I query the database using puppet functions (e.g. using foreman query
interface) - providing similar functionality to export / collection with
storeconfig.

in order to avoid the single point of failure, my function caches the output
locally, so if the db (foreman) is not reachable, it uses the last known
cache.
in any case, sqlite will never really work with many processes trying to
lock the file (it is a file after all), so you should really evaluate it
with some other db engine.

hope this helps,
Ohad

On Wed, Jun 9, 2010 at 11:50 PM, Christopher Johnston
<chjohnst@gmail.com>wrote:
> I have about a dozen sites with puppet masters in each.  They are
> geographically located with some slow network links between them.  The
setup
> I was evaluating is to have a centralized stored config server running
mysql
> with all of my remote masters sending their locations data back to the
> central server.  Now on my fast links remote connectivity is OK if I just
> point a master right to the database (eliminate puppetqd, etc), albeit
still
> slower then if I were to have it local but I was hoping I could use a
> stomp+puppetqd to handle caching and asyncronously update my central mysql
> database.  In my tests this seems to work however it looks like puppetdq is
> using sqllite to keep a cached copy.  If I have 100 servers and I am doing
a
> mass update  during maintenance window I am seeing timeouts once we hit 5-6
> servers which is likely sqllite.
>
> Perhaps this solution is not the most viable right now?  I am OK with
> running a local mysql instance instead of sqllite if it is supported but I
> would like to get all of my stored config data in one location (using
facter
> data, reporting is of great value to us).  This would be useful as we can
> plugin foreman on top of that database for a central dashboard instead of
> having dozens of dashboards for each location.
>
> -Chris
>
>
> On Tue, Jun 8, 2010 at 2:40 PM, donavan <donavan@desinc.net> wrote:
>
>>
>> On Jun 7, 1:00 pm, Christopher Johnston <chjoh...@gmail.com>
wrote:
>> > Think I found the issue from an strace, the rubygems-sqlit3 rpm
was
>> > missing.. still curious on where the logs end up for it though. 
The
>> fact
>> > that its using sqllite also doesn''t help as I think I am
running into
>> > locking issues if I try to run a cross environment update.
>>
>>
>> Chris, were you using sqlite or mysql as the backend store when you
>> had issues?
>>
>> Evaluating remote sites with central storeconfigs is on the todo list
>> for me. Currently have it working just peachy with mysql, straight
>> storeconfig no puppetqd.
>>
>> --
>> You received this message because you are subscribed to the Google
Groups
>> "Puppet Users" group.
>> To post to this group, send email to puppet-users@googlegroups.com.
>> To unsubscribe from this group, send email to
>>
puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com>
>> .
>> For more options, visit this group at
>> http://groups.google.com/group/puppet-users?hl=en.
>>
>>
>  --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
>
puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com>
> .
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

I don''t want to use sqllite... puppetqd doesn''t leave me much
of a choice
since it will default to it even if you tell it to use mysql as your
storedconfig DB since thats needed for the centraldb.  Ultimately I think
puppetqd needs support for keeping a cached copy of various database flavors
(mysql, sqllite, etc).  If mysql was supported for puppetqd then my remote
masters would always have a local cached copy of their stored configs for
the servers they manage and we would be able to eliminate the locking
issue.  They can then asyncronously replicate the data back to the central
DB.

# enables storedconfig support, supposed to speed things up
storeconfigs = true
dbadapter = mysql
dbuser = puppet
dbpassword = puppet
dbserver = centraldb
dbsocket = /var/run/mysqld/mysqld.sock

# use queuing
queue_type = stomp
queue_source = stomp://localhost:61613
async_storeconfigs = true


On Wed, Jun 9, 2010 at 11:57 AM, Ohad Levy <ohadlevy@gmail.com> wrote:
> I evaluated this scenario a couple of years ago (before puppetq), and it
> was never a feasible solution due to a. latency (e.g. everything is much
> slower), b. if wan goes down, your master wont work.
>
> One of the reasons that I''ve developed Foreman, is to overcome
this
> problem, the way I do it is:
> - I keep one central database, but don''t use it directly with
puppet,
> rather import the data i need (e.g. for facts see the import_facts script
in
> foreman extras dir)
> - I query the database using puppet functions (e.g. using foreman query
> interface) - providing similar functionality to export / collection with
> storeconfig.
>
> in order to avoid the single point of failure, my function caches the
> output locally, so if the db (foreman) is not reachable, it uses the last
> known cache.
> in any case, sqlite will never really work with many processes trying to
> lock the file (it is a file after all), so you should really evaluate it
> with some other db engine.
>
> hope this helps,
> Ohad
>
> On Wed, Jun 9, 2010 at 11:50 PM, Christopher Johnston
<chjohnst@gmail.com>wrote:
>
>> I have about a dozen sites with puppet masters in each.  They are
>> geographically located with some slow network links between them.  The
setup
>> I was evaluating is to have a centralized stored config server running
mysql
>> with all of my remote masters sending their locations data back to the
>> central server.  Now on my fast links remote connectivity is OK if I
just
>> point a master right to the database (eliminate puppetqd, etc), albeit
still
>> slower then if I were to have it local but I was hoping I could use a
>> stomp+puppetqd to handle caching and asyncronously update my central
mysql
>> database.  In my tests this seems to work however it looks like
puppetdq is
>> using sqllite to keep a cached copy.  If I have 100 servers and I am
doing a
>> mass update  during maintenance window I am seeing timeouts once we hit
5-6
>> servers which is likely sqllite.
>>
>> Perhaps this solution is not the most viable right now?  I am OK with
>> running a local mysql instance instead of sqllite if it is supported
but I
>> would like to get all of my stored config data in one location (using
facter
>> data, reporting is of great value to us).  This would be useful as we
can
>> plugin foreman on top of that database for a central dashboard instead
of
>> having dozens of dashboards for each location.
>>
>> -Chris
>>
>>
>> On Tue, Jun 8, 2010 at 2:40 PM, donavan <donavan@desinc.net>
wrote:
>>
>>>
>>> On Jun 7, 1:00 pm, Christopher Johnston <chjoh...@gmail.com>
wrote:
>>> > Think I found the issue from an strace, the rubygems-sqlit3
rpm was
>>> > missing.. still curious on where the logs end up for it
though.  The
>>> fact
>>> > that its using sqllite also doesn''t help as I think I
am running into
>>> > locking issues if I try to run a cross environment update.
>>>
>>>
>>> Chris, were you using sqlite or mysql as the backend store when you
>>> had issues?
>>>
>>> Evaluating remote sites with central storeconfigs is on the todo
list
>>> for me. Currently have it working just peachy with mysql, straight
>>> storeconfig no puppetqd.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
Groups
>>> "Puppet Users" group.
>>> To post to this group, send email to puppet-users@googlegroups.com.
>>> To unsubscribe from this group, send email to
>>>
puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com>
>>> .
>>> For more options, visit this group at
>>> http://groups.google.com/group/puppet-users?hl=en.
>>>
>>>
>>  --
>> You received this message because you are subscribed to the Google
Groups
>> "Puppet Users" group.
>> To post to this group, send email to puppet-users@googlegroups.com.
>> To unsubscribe from this group, send email to
>>
puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com>
>> .
>> For more options, visit this group at
>> http://groups.google.com/group/puppet-users?hl=en.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
>
puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com>
> .
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.