Puppet users - Mar 2009 - Connection timeout calling fileserver.describe: socket read timeout

Hi all,

I''ve puppet + mongrel running with 110 hosts. Seems that puppet
file-server cannot provide files to client when some are connected at
same time (many of them do it together cause they were installed at same
time, I use puppetd).

Playing a bit with puppet+mongrel I finally configured it with 20
puppetmaster :

PUPPETMASTER_PORTS=( 18140 18141 18142 18143 18144 18145 18146 18147
18148 18149 18150 18151 18152 18153 18154 18155 18156 18157 18158 18159)

So, when all hosts connect, I get 100 connections per daemon, more or
less. (When I started I did it with 3 puppetmasters and I got too many
connections, and I got " Connection timeout calling
puppetmaster.getconfig" many times, now it happens but not so often).

But now, nodes get the catalog, but they cannot get files... 
I read this:
http://reductivelabs.com/trac/puppet/wiki/FileServingConfiguration .

As file-serving is part of puppetmaterd, do I have to run more
puppetmater daemons? Is there any limit in puppetmaster daemons? Other
people who is managing same amount of nodes could be kind enough to
share their experience / conf ?

*I''m thinking on moving from puppetd to cron and run a couple of nodes
every minute, but it will give me some work...

TIA,
Arnau

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Not sure what''s going on with your specific setup but that many  
puppetmasterd is way overkill.

Also for tips on how to run puppetd as cron, visit this page:

http://reductivelabs.com/trac/puppet/wiki/Recipes/cron

-L

--
Larry Ludwig



--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

>
> I''ve puppet + mongrel running with 110 hosts. Seems that puppet
>
> PUPPETMASTER_PORTS=( 18140 18141 18142 18143 18144 18145 18146 18147
> 18148 18149 18150 18151 18152 18153 18154 18155 18156 18157 18158  
> 18159)
>
> So, when all hosts connect, I get 100 connections per daemon, more or
> less. (When I started I did it with 3 puppetmasters and I got too many
> connections, and I got " Connection timeout calling
> puppetmaster.getconfig" many times, now it happens but not so often).
>
That''s really odd I''m not sure how you are configured.  3-4  
puppetmasterd should be more than enough for your node count.

How much file handling are you doing?

-L

--
Larry Ludwig


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Mon, 2 Mar 2009 09:29:01 -0500
Larry Ludwig wrote:

Hi Larry,
> That''s really odd I''m not sure how you are configured. 
3-4
> puppetmasterd should be more than enough for your node count.
Yes, that''s what I heard here, but my experience with 3-4
masters was terrible.
Are you managing ~100 hosts? with mongrel?
 
> How much file handling are you doing?
[root@gridinstall puppet]# find files|grep -v .svn|wc -l
123

[root@gridinstall puppet]# du -sh files
3.1M	files

123 files and 3 MB. Only that.
> -L
Arnau

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Mar 2, 2009, at 9:48 AM, Arnau Bria wrote:
>
> On Mon, 2 Mar 2009 09:29:01 -0500
> Larry Ludwig wrote:
>
> Hi Larry,
>
>> That''s really odd I''m not sure how you are
configured.  3-4
>> puppetmasterd should be more than enough for your node count.
> Yes, that''s what I heard here, but my experience with 3-4
> masters was terrible.
> Are you managing ~100 hosts? with mongrel?
150, and yes. The obvious question is I assume you are not thrashing  
to swap and other issues related to the server config itself.  Based  
upon how big puppetmasterd can grow with 20+ puppetmasterd you would  
need a lot of memory.
>
> [root@gridinstall puppet]# find files|grep -v .svn|wc -l
> 123
>
> [root@gridinstall puppet]# du -sh files
> 3.1M	files
>
> 123 files and 3 MB. Only that.
>
We are doing much more than that.

What version of puppet?

-L

--
Larry Ludwig



--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Mon, 2 Mar 2009 09:51:34 -0500
Larry Ludwig wrote:

> 150, and yes. The obvious question is I assume you are not thrashing  
> to swap and other issues related to the server config itself. Based  
> upon how big puppetmasterd can grow with 20+ puppetmasterd you would  
> need a lot of memory.
My puppet server has 4GB of memory, and I don''t see it swapping... but
every 30 minutes, when clients connect, the load is about 10-15 and cpu
gets 100%, but no swap used.
> We are doing much more than that.
> 
> What version of puppet?
# rpm -qa|grep puppet
puppet-0.24.7-4.el5
puppet-server-0.24.7-4.el5
> -L
Arnau

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

using storedconfig?

-L

On Mar 2, 2009, at 10:02 AM, Arnau Bria wrote:
>
> On Mon, 2 Mar 2009 09:51:34 -0500
> Larry Ludwig wrote:
>
>
>> 150, and yes. The obvious question is I assume you are not thrashing
>> to swap and other issues related to the server config itself. Based
>> upon how big puppetmasterd can grow with 20+ puppetmasterd you would
>> need a lot of memory.
>
> My puppet server has 4GB of memory, and I don''t see it swapping...
but
> every 30 minutes, when clients connect, the load is about 10-15 and  
> cpu
> gets 100%, but no swap used.
>
>> We are doing much more than that.
>>
>> What version of puppet?
> # rpm -qa|grep puppet
> puppet-0.24.7-4.el5
> puppet-server-0.24.7-4.el5
>
>> -L
> Arnau
>
> >

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Mon, 2 Mar 2009 10:03:39 -0500
Larry Ludwig wrote:

 
> using storedconfig?
nop
> -L
Arnau

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Arnau Bria wrote:
> On Mon, 2 Mar 2009 09:51:34 -0500
> Larry Ludwig wrote:
> 
> 
>> 150, and yes. The obvious question is I assume you are not thrashing  
>> to swap and other issues related to the server config itself. Based  
>> upon how big puppetmasterd can grow with 20+ puppetmasterd you would  
>> need a lot of memory.
> 
> My puppet server has 4GB of memory, and I don''t see it swapping...
but
> every 30 minutes, when clients connect, the load is about 10-15 and cpu
> gets 100%, but no swap used.
> 
>> We are doing much more than that.
>>
>> What version of puppet?
> # rpm -qa|grep puppet
> puppet-0.24.7-4.el5
> puppet-server-0.24.7-4.el5
Do you have SELinux enabled?  If so, you may be running into this issue:

http://projects.reductivelabs.com/issues/1963

Note that this isn''t caused by SELinux blocking anything, so this bug
can bite
you even if you have SELinux enabled in permissive mode.

-- 
Frank Sweetser fs at wpi.edu  |  For every problem, there is a solution that
WPI Senior Network Engineer   |  is simple, elegant, and wrong. - HL Mencken
     GPG fingerprint = 6174 1257 129E 0D21 D8D4  E8A3 8E39 29E3 E2E8 8CEC

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Mon, 02 Mar 2009 10:20:30 -0500
Frank Sweetser wrote:

Hi Frank,
> Do you have SELinux enabled?  If so, you may be running into this
> issue:
> 
> http://projects.reductivelabs.com/issues/1963
> 
> Note that this isn''t caused by SELinux blocking anything, so this
bug
> can bite you even if you have SELinux enabled in permissive mode.
I don''t run selinux, but thanks for the suggestion.

Cheers,
Arnau

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

2009/3/2 Arnau Bria <arnaubria@pic.es>
>
> On Mon, 02 Mar 2009 10:20:30 -0500
> Frank Sweetser wrote:
>
> Hi Frank,
>
> > Do you have SELinux enabled?  If so, you may be running into this
> > issue:
> >
> > http://projects.reductivelabs.com/issues/1963
> >
> > Note that this isn''t caused by SELinux blocking anything, so
this bug
> > can bite you even if you have SELinux enabled in permissive mode.
> I don''t run selinux, but thanks for the suggestion.
>
> Cheers,
> Arnau
>

Maybe Im a bit blunt here, but what are you expecting to gain from that many
puppetmasterds? What I have observed is that when when a client request a
compilation of configuration the prossess can run into 100%. That means it
runs into full capability of that cpu core. I can not see how it gains you
running more puppetmasterds than you have cpu cores?

If you have a decent server with eg a quadcore cpu and 4gb ram as you say,
you shouldn''t have any problems. Could the problem be that you are
doing
heavy file serving that might be best taken care of by some other service?
(configured by puppet of course)

Is there any reason to have them connect all at once? You could throw a
script in cron.daily on the client that restarts puppet at a random interval
every day. Then it will even out.

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Mar 2, 2009, at 6:18 AM, Arnau Bria wrote:
> So, when all hosts connect, I get 100 connections per daemon, more or
> less. (When I started I did it with 3 puppetmasters and I got too many
> connections, and I got " Connection timeout calling
> puppetmaster.getconfig" many times, now it happens but not so often).
Having all your nodes connect at the same time is a Bad Idea. Try  
running puppetd with --splay=true. See the config reference for  
details, but it should allow you to easily handle 150 clients with  
four instances of puppetmasterd.

-Josh

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Mon, 02 Mar 2009 10:50:46 -0800
Joshua Anderson wrote:

Hi Joshua,
> Having all your nodes connect at the same time is a Bad Idea. Try  
> running puppetd with --splay=true. See the config reference for  
> details, but it should allow you to easily handle 150 clients with  
> four instances of puppetmasterd.
thanks for the tip, I''ll try it tomorrow and will come back ASAP with
results.
 
> -Josh
Arnau

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Mon, 2 Mar 2009 19:32:55 +0100
Bjørn Dyre Dyresen wrote:

Hi,
> Maybe Im a bit blunt here, but what are you expecting to gain from
> that many puppetmasterds? What I have observed is that when when a
> client request a compilation of configuration the prossess can run
> into 100%. That means it runs into full capability of that cpu core.
> I can not see how it gains you running more puppetmasterds than you
> have cpu cores?
Well, if one master gets hundresd''s coneections and it gives me
timeouts, and, if I add more puppetmasters they get less and less
connections, and the host is not swapping and it responds very well, I
try tun more masters as I get less connections timeouts.
 
> If you have a decent server with eg a quadcore cpu and 4gb ram as you
> say, you shouldn''t have any problems. Could the problem be that
you
> are doing heavy file serving that might be best taken care of by some
> other service? (configured by puppet of course)
Well, 3.1 MB and 100 files (IIRC) is a heavy file serving? most of the
files are under same dir, so I copy then recursevely... is it really
heaavy?
 
> Is there any reason to have them connect all at once? You could throw
> a script in cron.daily on the client that restarts puppet at a random
> interval every day. Then it will even out.
Sure, I could, but, as someone said here, puppet can handle 100 hosts,
and noone talked about having to run puppet via cron, I supposed that
puppet daemon was a good idea.

Thanks for your reply,
Arnau

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Mar 3, 2009, at 12:05 AM, Arnau Bria wrote:
>
> On Mon, 2 Mar 2009 19:32:55 +0100
> Bjørn Dyre Dyresen wrote:
>
> Hi,
>
>> Maybe Im a bit blunt here, but what are you expecting to gain from
>> that many puppetmasterds? What I have observed is that when when a
>> client request a compilation of configuration the prossess can run
>> into 100%. That means it runs into full capability of that cpu core.
>> I can not see how it gains you running more puppetmasterds than you
>> have cpu cores?
> Well, if one master gets hundresd''s coneections and it gives me
> timeouts, and, if I add more puppetmasters they get less and less
> connections, and the host is not swapping and it responds very well, I
> try tun more masters as I get less connections timeouts.
>
>> If you have a decent server with eg a quadcore cpu and 4gb ram as you
>> say, you shouldn''t have any problems. Could the problem be
that you
>> are doing heavy file serving that might be best taken care of by some
>> other service? (configured by puppet of course)
> Well, 3.1 MB and 100 files (IIRC) is a heavy file serving? most of the
> files are under same dir, so I copy then recursevely... is it really
> heaavy?
>
No, not heavy at all.

>> Is there any reason to have them connect all at once? You could throw
>> a script in cron.daily on the client that restarts puppet at a random
>> interval every day. Then it will even out.
> Sure, I could, but, as someone said here, puppet can handle 100 hosts,
> and noone talked about having to run puppet via cron, I supposed that
> puppet daemon was a good idea.
>
Sorry, didn''t mean to run puppet from cron. Just to trigger a random  
restart, which won''t be needed if you follow Joshua Andersons  
suggestion.  100 nodes is not much at all.
> Thanks for your reply,
> Arnau
>
> >

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Mon, Mar 2, 2009 at 7:50 PM, Joshua Anderson
<joshua_anderson@mac.com>wrote:
Hi Josh,

> Having all your nodes connect at the same time is a Bad Idea. Try running
> puppetd with *--splay=true*. See the config
reference<http://reductivelabs.com/trac/puppet/wiki/ConfigurationReference#splay>
for
> details, but it should allow you to easily handle 150 clients with four
> instances of puppetmasterd.
>
I did what you recommended me.

Now I have 4 puppetmaterd and splay=true in all client.
*I don''t see the option if I do a ps|grep puppet:
]# ps -ef|grep puppet
root     22239     1  3 10:30 ?        00:01:50 /usr/bin/ruby
/usr/sbin/puppetd --logdest=/var/log/puppet/puppet.log

 but if I do puppetd --genconfig|grep splay I see its value and it''s
true.
  splaylimit = 1800
    splay = true


Few seconds ago I found 94 conections from WN to http server in
puppetserver, and, if we take in count that I run puppetmaters in 1814[0-3]
]# netstat -puta|grep -c 1814
4590

a lot of connections, so seems that clients still try to connect all at
once.
Maybe i should start evaluating cron option...


> -Josh
>
Thanks for your reply,
Arnau

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Arnau Bria <arnaubria@pic.es> writes:
> On Mon, Mar 2, 2009 at 7:50 PM, Joshua Anderson
<joshua_anderson@mac.com> wrote:
[...]
> a lot of connections, so seems that clients still try to connect all
> at once.  Maybe i should start evaluating cron option...
We ended up using cron and fqdn_rand() to spread execution time, rather
than splay.  It worked fine for us, and meant we didn''t have cron
processes hanging about for several minutes unnecessarily.

Regards,
        Daniel

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---