Neil Prockter
2009-Dec-06 23:33 UTC
[Puppet Users] "other end went away" leads to multiple report mails
Evening All, I''ve been getting a random number of copies of report mails, the same report sent multiple times at the same second, and finally got tired enough of it to look into it. They are sent from puppet rather than a mail server issue. I get one extra copy every time the client gets a "warning: Other end went away; restarting connection and retrying". Updating the server from 0.24.8 to 0.25.1 did not improve matters. Running with client 0.24.5 debug ends with debug: Calling puppetreports.report warning: Other end went away; restarting connection and retrying warning: Other end went away; restarting connection and retrying info: Sent transaction report in 3.29 seconds notice: Finished catalog run in 11.40 seconds upgrading the client to 0.24.8 client debug ends with debug: Calling puppetreports.report info: Other end went away; restarting connection and retrying info: Other end went away; restarting connection and retrying info: Sent transaction report in 3.66 seconds notice: Finished catalog run in 12.61 seconds so the level has been reduced to info but I still get the multiple report mails updating to 0.25.1 seems to deal with the repeated mails though I see err: Reporting failed: end of file reached Updating clients to 0.25.1 across the board is probably not an option for me so I''d like to fix the reporting from a server side if possible. My tagmail.conf is just all: an-address@that.goes.to.me Neil Please access the attached hyperlink for an important electronic communications disclaimer: http://www.lse.ac.uk/collections/secretariat/legal/disclaimer.htm -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Greg
2009-Dec-07 01:15 UTC
[Puppet Users] Re: "other end went away" leads to multiple report mails
Neil,
I had a similar problem back when I was running 0.24.7 and 0.24.8 on
Solaris. My solution was to remove tagmail completely and put in a
seperate system that parsed the YAML reports and send a single email
combining all changes during a given period of time (in my case -
daily). Required a little coding, but its not too difficult. Then just
schedule it to run from cron on each of your master servers that
receive reports from clients.
To be honest, tagmail isn''t that useful outside of running it on a few
hosts... I personally don''t want to receive 100+ emails every time I
changed an entry that was common to every node... This way I receive
one email in a summarised format - if I need more detail there is
always the messages file on the node itself, or you can optionally
keep the reports for a few days... Just remember to set up a tidy on
the reports directory if you do that... (Oh, and if you set up a tidy
on the reports directory, you may want to bear in mind Bug 2701:
http://projects.reductivelabs.com/issues/2701)
The base of the code is something like:
require "puppet"
require "yaml"
require "find"
Puppet[:config] = "/etc/puppet/puppet.conf"
Puppet.parse_config
Puppet[:name] = "puppetmasterd"
Puppet::Node::Facts.terminus_class = :yaml
File.find("/var/puppet/reports") do | report |
next if FileTest.directory?(report)
thisreport = open(report) { |fh| YAML::load(fh) }
print "Host: " + thisreport.host
reporttime = File.stat(report).mtime.to_i # seconds since epoch...
thisreport.logs.each do |log|
# Whatever you want to find out about each message...
end
File.unlink(report) # If you don''t need them anymore...
end
On Dec 7, 10:33 am, Neil Prockter <n.prock...@lse.ac.uk>
wrote:> Evening All,
>
> I''ve been getting a random number of copies of report mails, the
same
> report sent multiple times at the same second, and finally got tired
> enough of it to look into it.
>
> They are sent from puppet rather than a mail server issue.
>
> I get one extra copy every time the client gets a "warning: Other end
> went away; restarting connection and retrying".
>
> Updating the server from 0.24.8 to 0.25.1 did not improve matters.
>
> Running with client 0.24.5 debug ends with
>
> debug: Calling puppetreports.report
> warning: Other end went away; restarting connection and retrying
> warning: Other end went away; restarting connection and retrying
> info: Sent transaction report in 3.29 seconds
> notice: Finished catalog run in 11.40 seconds
>
> upgrading the client to 0.24.8 client debug ends with
>
> debug: Calling puppetreports.report
> info: Other end went away; restarting connection and retrying
> info: Other end went away; restarting connection and retrying
> info: Sent transaction report in 3.66 seconds
> notice: Finished catalog run in 12.61 seconds
>
> so the level has been reduced to info but I still get the multiple
> report mails
>
> updating to 0.25.1 seems to deal with the repeated mails though I see
>
> err: Reporting failed: end of file reached
>
> Updating clients to 0.25.1 across the board is probably not an option
> for me so I''d like to fix the reporting from a server side if
possible.
>
> My tagmail.conf is just
> all: an-addr...@that.goes.to.me
>
> Neil
>
> Please access the attached hyperlink for an important electronic
communications
disclaimer:http://www.lse.ac.uk/collections/secretariat/legal/disclaimer.htm
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
Ohad Levy
2009-Dec-07 01:53 UTC
Re: [Puppet Users] "other end went away" leads to multiple report mails
Hi, when developing Foreman[1], I''ve also seen it similar problem (Reports are being sent more than once). I''ve worked around it in Foreman, maybe that would be a better alternative to tagmail altogether. Cheers, Ohad *1 http://theforeman.org On Mon, Dec 7, 2009 at 7:33 AM, Neil Prockter <n.prockter@lse.ac.uk> wrote:> Evening All, > > I''ve been getting a random number of copies of report mails, the same > report sent multiple times at the same second, and finally got tired > enough of it to look into it. > > They are sent from puppet rather than a mail server issue. > > I get one extra copy every time the client gets a "warning: Other end > went away; restarting connection and retrying". > > Updating the server from 0.24.8 to 0.25.1 did not improve matters. > > Running with client 0.24.5 debug ends with > > debug: Calling puppetreports.report > warning: Other end went away; restarting connection and retrying > warning: Other end went away; restarting connection and retrying > info: Sent transaction report in 3.29 seconds > notice: Finished catalog run in 11.40 seconds > > upgrading the client to 0.24.8 client debug ends with > > debug: Calling puppetreports.report > info: Other end went away; restarting connection and retrying > info: Other end went away; restarting connection and retrying > info: Sent transaction report in 3.66 seconds > notice: Finished catalog run in 12.61 seconds > > so the level has been reduced to info but I still get the multiple > report mails > > updating to 0.25.1 seems to deal with the repeated mails though I see > > err: Reporting failed: end of file reached > > Updating clients to 0.25.1 across the board is probably not an option > for me so I''d like to fix the reporting from a server side if possible. > > My tagmail.conf is just > all: an-address@that.goes.to.me > > Neil > > Please access the attached hyperlink for an important electronic > communications disclaimer: > http://www.lse.ac.uk/collections/secretariat/legal/disclaimer.htm > > -- > > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Peter Meier
2009-Dec-07 07:27 UTC
Re: [Puppet Users] Re: "other end went away" leads to multiple report mails
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1> The base of the code is something like: > > require "puppet" > require "yaml" > require "find" > Puppet[:config] = "/etc/puppet/puppet.conf" > Puppet.parse_config > Puppet[:name] = "puppetmasterd" > Puppet::Node::Facts.terminus_class = :yaml > File.find("/var/puppet/reports") do | report | > next if FileTest.directory?(report) > thisreport = open(report) { |fh| YAML::load(fh) } > print "Host: " + thisreport.host > reporttime = File.stat(report).mtime.to_i # seconds since epoch... > thisreport.logs.each do |log| > # Whatever you want to find out about each message... > end > File.unlink(report) # If you don''t need them anymore... > endwould be nice to have such a working example in ext/ cheers pete -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkscrloACgkQbwltcAfKi3/LiACfQXDDbclJkGlofQxYUkvw09me 9A0AoI/IHHoBJhEiCZKkLlecvdwd1cgb =MuV0 -----END PGP SIGNATURE----- -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Neil Prockter
2009-Dec-07 23:23 UTC
Re: [Puppet Users] Re: "other end went away" leads to multiple report mails
Thanks for this. I''ve stopping using the tagmail report and I''ll modify that script to give me latest applied changes and the servers they were applied to and maybe checking times/frequency for all servers over some period. Neil Greg wrote:> Neil, > > I had a similar problem back when I was running 0.24.7 and 0.24.8 on > Solaris. My solution was to remove tagmail completely and put in a > seperate system that parsed the YAML reports and send a single email > combining all changes during a given period of time (in my case - > daily). Required a little coding, but its not too difficult. Then just > schedule it to run from cron on each of your master servers that > receive reports from clients. > > To be honest, tagmail isn''t that useful outside of running it on a few > hosts... I personally don''t want to receive 100+ emails every time I > changed an entry that was common to every node... This way I receive > one email in a summarised format - if I need more detail there is > always the messages file on the node itself, or you can optionally > keep the reports for a few days... Just remember to set up a tidy on > the reports directory if you do that... (Oh, and if you set up a tidy > on the reports directory, you may want to bear in mind Bug 2701: > http://projects.reductivelabs.com/issues/2701) > > The base of the code is something like: > > require "puppet" > require "yaml" > require "find" > Puppet[:config] = "/etc/puppet/puppet.conf" > Puppet.parse_config > Puppet[:name] = "puppetmasterd" > Puppet::Node::Facts.terminus_class = :yaml > File.find("/var/puppet/reports") do | report | > next if FileTest.directory?(report) > thisreport = open(report) { |fh| YAML::load(fh) } > print "Host: " + thisreport.host > reporttime = File.stat(report).mtime.to_i # seconds since epoch... > thisreport.logs.each do |log| > # Whatever you want to find out about each message... > end > File.unlink(report) # If you don''t need them anymore... > end > > On Dec 7, 10:33 am, Neil Prockter <n.prock...@lse.ac.uk> wrote: >> Evening All, >> >> I''ve been getting a random number of copies of report mails, the same >> report sent multiple times at the same second, and finally got tired >> enough of it to look into it. >> >> They are sent from puppet rather than a mail server issue. >> >> I get one extra copy every time the client gets a "warning: Other end >> went away; restarting connection and retrying". >> >> Updating the server from 0.24.8 to 0.25.1 did not improve matters. >> >> Running with client 0.24.5 debug ends with >> >> debug: Calling puppetreports.report >> warning: Other end went away; restarting connection and retrying >> warning: Other end went away; restarting connection and retrying >> info: Sent transaction report in 3.29 seconds >> notice: Finished catalog run in 11.40 seconds >> >> upgrading the client to 0.24.8 client debug ends with >> >> debug: Calling puppetreports.report >> info: Other end went away; restarting connection and retrying >> info: Other end went away; restarting connection and retrying >> info: Sent transaction report in 3.66 seconds >> notice: Finished catalog run in 12.61 seconds >> >> so the level has been reduced to info but I still get the multiple >> report mails >> >> updating to 0.25.1 seems to deal with the repeated mails though I see >> >> err: Reporting failed: end of file reached >> >> Updating clients to 0.25.1 across the board is probably not an option >> for me so I''d like to fix the reporting from a server side if possible. >> >> My tagmail.conf is just >> all: an-addr...@that.goes.to.me >> >> Neil >> >> Please access the attached hyperlink for an important electronic communications disclaimer:http://www.lse.ac.uk/collections/secretariat/legal/disclaimer.htm > > -- > > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. > >Please access the attached hyperlink for an important electronic communications disclaimer: http://www.lse.ac.uk/collections/secretariat/legal/disclaimer.htm -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Luke Schierer
2009-Dec-08 19:20 UTC
Re: [Puppet Users] Re: "other end went away" leads to multiple report mails
On Mon, Dec 7, 2009 at 2:27 AM, Peter Meier <peter.meier@immerda.ch> wrote:> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > > The base of the code is something like: > > > > require "puppet" > > require "yaml" > > require "find" > > Puppet[:config] = "/etc/puppet/puppet.conf" > > Puppet.parse_config > > Puppet[:name] = "puppetmasterd" > > Puppet::Node::Facts.terminus_class = :yaml > > File.find("/var/puppet/reports") do | report | > > next if FileTest.directory?(report) > > thisreport = open(report) { |fh| YAML::load(fh) } > > print "Host: " + thisreport.host > > reporttime = File.stat(report).mtime.to_i # seconds since epoch... > > thisreport.logs.each do |log| > > # Whatever you want to find out about each message... > > end > > File.unlink(report) # If you don''t need them anymore... > > end > > > would be nice to have such a working example in ext/ > > I agree, a working example would be awesome.Luke -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.