Hi puppetiers, recently I switched to using Mongrel webserver in effort to get rid of those annoying "end of file reached" errors and increase performance. It did the trick beautifully, but now a lot of my reports are plagued by another error message: --- err: Could not call fileserver.describe: #<Errno::EPERM: Operation not permitted - connect(2)> //ns0/dns/File[/etc/namedb/master/db.example.com]/source (err): Could not describe /files/apps/named/ns0/master/db.example.com: Operation not permitted - connect(2) err: Could not call fileserver.list: #<Errno::EPERM: Operation not permitted - connect(2)> //ns0/dns/File[/etc/namedb/master/db.example.com] (err): Failed to generate additional resources during transaction: Operation not permitted - connect(2) --- These particular examples are from a simple dns class that deploys around 40 master zone files (source a directory recursively) on FreeBSD 6.2-RELEASE and puppet 0.23.2 from source on both client and server. Enabling the dns class on other machines results in the same behaviour. Other classes also yeld this error, although much less frequently, but they are also serving less files. On ns0, the error appears around once a puppetd run (which is every 5 minutes). There are 12 servers managed in total with 2 puppetmasterd''s. Adding more puppetmasters seemed to do make no difference. I inspected apache / proxy logs and tcpdump output, but could not find any indication of failed connection attempts / connection resets. I read the list discussion on fileserver scalability, but those guys are serving way more files and there is no mention of the above problem anywhere in sight. I''d appreciate any pointers as to what could be causing this, anything else to try etc, as right now Im quite baffled by this. Thanks, Jimbow -- the neptune project http://t-n-p.org
On Oct 23, 2007, at 5:04 AM, Jimbow wrote:> recently I switched to using Mongrel webserver in effort to get rid of > those annoying "end of file reached" errors and increase > performance. It > did the trick beautifully, but now a lot of my reports are plagued by > another error message: > > --- > err: Could not call fileserver.describe: #<Errno::EPERM: > Operation not permitted - connect(2)>Do you have a namespaceauth.conf file? Is it allowing the connections? If not, what do your apache logs say? Are they blocking the connections somehow? -- What''s the good of having mastery over cosmic balance and knowing the secrets of fate if you can''t blow something up? -- Terry Pratchett, "Reaper Man" --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
Thanks for the reply, Luke Kanies wrote:> On Oct 23, 2007, at 5:04 AM, Jimbow wrote: > > > err: Could not call fileserver.describe: #<Errno::EPERM: > > Operation not permitted - connect(2)> > > Do you have a namespaceauth.conf file? Is it allowing the connections?No, I dont have a namespaceauth.conf. Adding one with permissions of ''allow *'' for fileserver, puppetmaster and puppetreports made no difference. The above error is generated only on one or sometimes two files from the entire puppet client run - all other files are retrieved w/o problems. The files requested are always the same and the error does not appear on the same file each time.> If not, what do your apache logs say? Are they blocking the > connections somehow?Apache is allowing everything through, there are no errors in the logs anywhere. access.log shows successful xmlrpc entries only (http code 200). There are no tcp resets sent between puppet client - apache and apache - puppetmaster either. Any more ideas? -- the neptune project http://t-n-p.org
On Oct 24, 2007, at 5:49 PM, Jimbow wrote:> > No, I dont have a namespaceauth.conf. Adding one with permissions of > ''allow *'' for fileserver, puppetmaster and puppetreports made no > difference. > > The above error is generated only on one or sometimes two files > from the > entire puppet client run - all other files are retrieved w/o problems. > The files requested are always the same and the error does not > appear on > the same file each time.So it''s randomly saying "operation not allowed" on some file copies? This almost sounds like an operating system thing, maybe something failing on the server. I''ve never seen it before.> Apache is allowing everything through, there are no errors in the logs > anywhere. access.log shows successful xmlrpc entries only (http code > 200). There are no tcp resets sent between puppet client - apache and > apache - puppetmaster either. > > Any more ideas?Erm, not really? You can''t usefully snoop the traffic, but maybe you could up the logging on apache or mongrel? -- Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it. -- (attributed to) Brian W. Kernighan (unconfirmed) --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com