Puppet users - Sep 2012 - Puppet/Passenger :: Could not retrieve catalog from remote server:Error 403 on server

Greetings,

I have a tested, working setup of Puppet and Webrick. I can add nodes, 
classes, etc.
Then I switched to Puppet/Passenger and get the error  below.
Puppet, Apache and Passenger are all up.

I have installed using *YUM *repos and *GEMs*. So, I have the most updated 
packages they have.

Puppet version: 2.7.19
Ruby version: 1.8.7 (2011-06-30 patchlevel 352 i386)
Apache: 2.2.15

The error is below.
I have found little references on the web. Has anyone come across such 
problem recently?

[root@puppetm01 ~]# puppet agent --test
err: Could not retrieve catalog from remote server: Error 403 on SERVER:
*Forbidden
request*: puppetm01.example.com(xxx.xxx.xxx.xxx) access to 
/catalog/puppetm01.example.com [find] at line 53
warning: Not using cache on failed catalog
err: Could not retrieve catalog; skipping run
err: Could not send report: Error 403 on SERVER: *Forbidden request*: 
puppetm01.example.com(xxx.xxx.xxx.xxx) access to 
/report/puppetm01.example.com [save] at line 53

Below is the path to the catalog file to which I believe the error points.

[root@puppetm01 ]# find /var/lib/puppet | grep catalog
./client_yaml/catalog
./client_yaml/catalog/puppetm01.example.com.yaml

Thanks in advance for any pointers.
----

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/xms_wXhyV2EJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Check the owner of config.ru. The owner of this file is who passenger will run
the puppetmaster daemon as. I''m guessing that it''s not owned
by puppet.

On Sep 28, 2012, at 9:36 AM, Lunixer wrote:
> Greetings,
> 
> I have a tested, working setup of Puppet and Webrick. I can add nodes,
classes, etc.
> Then I switched to Puppet/Passenger and get the error  below.
> Puppet, Apache and Passenger are all up.
> 
> I have installed using YUM repos and GEMs. So, I have the most updated
packages they have.
> 
> Puppet version: 2.7.19
> Ruby version: 1.8.7 (2011-06-30 patchlevel 352 i386)
> Apache: 2.2.15
> 
> The error is below.
> I have found little references on the web. Has anyone come across such
problem recently?
> 
> [root@puppetm01 ~]# puppet agent --test
> err: Could not retrieve catalog from remote server: Error 403 on SERVER:
Forbidden request: puppetm01.example.com(xxx.xxx.xxx.xxx) access to
/catalog/puppetm01.example.com [find] at line 53
> warning: Not using cache on failed catalog
> err: Could not retrieve catalog; skipping run
> err: Could not send report: Error 403 on SERVER: Forbidden request:
puppetm01.example.com(xxx.xxx.xxx.xxx) access to /report/puppetm01.example.com
[save] at line 53
> 
> Below is the path to the catalog file to which I believe the error points.
> 
> [root@puppetm01 ]# find /var/lib/puppet | grep catalog
> ./client_yaml/catalog
> ./client_yaml/catalog/puppetm01.example.com.yaml
> 
> Thanks in advance for any pointers.
> ----
> 
> -- 
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/xms_wXhyV2EJ.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
-- 
Jo Rhett
Net Consonance : net philanthropy to improve open source and internet projects.



-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Thanks for the reply.

I have checked permissions per the master puppet.conf excerpt below
.
My understanding is that Passenger does not really install anything or 
copies files around. 
You only create a directory and copy the config.ru into it and change 
permissions to puppet.
The only thing that passenger does is to install a Apache module, then you 
configure your vhost with that info.

I don''t know whether I could blame the problem on any of the other
packages
(I.e. ruby), because things work perfectly fine with WEBrick.

Below I added more information. Please let me know If anyone spots 
something out of place.

*[root@puppetm01 puppet]# cat puppet.conf*
[main]
    user = puppet
    group = puppet


*[root@puppetm01 ]# ls -l /var/lib/puppetmaster/*
-rw-r--r-- 1 puppet puppet  431 Sep 27 21:51 config.ru
drwxr-xr-x 2 puppet puppet 4096 Sep 27 21:31 public
drwxr-xr-x 2 puppet puppet 4096 Sep 27 21:31 tmp

*[root@puppetm01 ~]# ps -ef | grep puppet*
avahi     1989     1  0 09:34 ?   00:00:00 avahi-daemon: running 
[puppetm01.local]
root      2666     1  0 09:34 ?   00:00:01 /usr/bin/ruby /usr/sbin/puppetd
puppet    9734  9541  2 12:35 ?   00:00:00 
master

puppet    9769     1  0 12:35 ?   00:00:00 Rack: /var/lib/puppetmaster  



*
[root@puppetm01 ]# grep puppet /etc/passwd*
puppet:x:52:52:Puppet:/var/lib/puppet:/sbin/nologin
puppetdb:x:494:488:PuppetDB daemon:/usr/share/puppetdb:/sbin/nologin
puppet-dashboard:x:492:489:Puppet 
Dashboard:/usr/share/puppet-dashboard:/sbin/nologin

*[root@puppetm01 ]# id -a puppet*
uid=52(puppet) gid=52(puppet) groups=52(puppet)
*
[root@puppetm01 ~]# passenger-memory-stats *

-------- Apache processes ---------
PID   PPID  VMSize   Private  Name
-----------------------------------
9534  1     26.8 MB  0.3 MB   /usr/sbin/httpd
9551  9534  26.7 MB  0.2 MB   /usr/sbin/httpd
9552  9534  26.8 MB  0.2 MB   /usr/sbin/httpd
9553  9534  27.0 MB  0.5 MB   /usr/sbin/httpd
9554  9534  27.0 MB  0.5 MB   /usr/sbin/httpd
9555  9534  26.8 MB  0.3 MB   /usr/sbin/httpd
9556  9534  26.8 MB  0.2 MB   /usr/sbin/httpd
9557  9534  26.9 MB  0.3 MB   /usr/sbin/httpd
9558  9534  26.8 MB  0.2 MB   /usr/sbin/httpd
9559  9534  26.8 MB  0.2 MB   /usr/sbin/httpd
### Processes: 10
### Total private dirty RSS: 3.00 MB


-------- Nginx processes --------

### Processes: 0
### Total private dirty RSS: 0.00 MB


---- Passenger processes ----
PID   VMSize   Private  Name
-----------------------------
9536  6.7 MB   0.2 MB   PassengerWatchdog
9539  17.8 MB  0.4 MB   PassengerHelperAgent
9541  18.7 MB  4.9 MB   Passenger spawn server
9544  13.2 MB  0.4 MB   PassengerLoggingAgent
9769  51.8 MB  26.0 MB  Rack: /var/lib/puppetmaster
9802  60.6 MB  36.6 MB  Passenger ApplicationSpawner: 
/usr/share/puppet-dashboard
9808  61.1 MB  37.2 MB  Rails: /usr/share/puppet-dashboard
### Processes: 7
### Total private dirty RSS: 105.69 MB

*
[root@puppetm01 ~]# passenger-status --verbose*
----------- General information -----------
max      = 12
count    = 2
active   = 0
inactive = 2
Waiting on global queue: 0

----------- Application groups -----------
/usr/share/puppet-dashboard:
  App root: /usr/share/puppet-dashboard
  * PID: 9808    Sessions: 0    Processed: 2       Uptime: 58s
      URL     : http://127.0.0.1:50447
      Password: xxxxxxxxxxxxxx

/var/lib/puppetmaster:
  App root: /var/lib/puppetmaster
  * PID: 9769    Sessions: 0    Processed: 2       Uptime: 1m 56s
      URL     : http://127.0.0.1:55087
      Password: xxxxxxxxxxxxxx

*[root@puppetm01 ~]# tail -f /var/log/httpd/access_log
xxx.xxx.xxx.xxx - - [28/Sep/2012:12:39:20 -0700] "POST 
/production/catalog/puppetm01.example.com HTTP/1.1" 403 138 "-"
"-"
xxx.xxx.xxx.xxx - - [28/Sep/2012:12:39:20 -0700] "PUT 
/production/report/puppetm01.example.com HTTP/1.1" 500 635 "-"
"-"
xxx.xxx.xxx.xxx - - [28/Sep/2012:12:39:30 -0700] "POST 
/production/catalog/puppetm01.example.com HTTP/1.1" 403 138 "-"
"-"
xxx.xxx.xxx.xxx - - [28/Sep/2012:12:39:33 -0700] "PUT 
/production/report/puppetm01.example.com HTTP/1.1" 403 137 "-"
"-"


[root@puppetm01 ~]# find /var/lib/puppet | grep catalog | xargs ls -l
-rw-r-----. 1 root root 13150 Sep 27 21:00 
/var/lib/puppet/client_yaml/catalog/puppetm01.example.com.yaml

/var/lib/puppet/client_yaml/catalog:
total 16
-rw-r-----. 1 root root 13150 Sep 27 21:00 puppetm01.example.com.yaml*



Thanks,
LL
-----


On Friday, September 28, 2012 10:53:35 AM UTC-7, Jo
wrote:
>
> Check the owner of config.ru. The owner of this file is who passenger 
> will run the puppetmaster daemon as. I''m guessing that
it''s not owned by
> puppet.
>
> On Sep 28, 2012, at 9:36 AM, Lunixer wrote:
>
> Greetings,
>
> I have a tested, working setup of Puppet and Webrick. I can add nodes, 
> classes, etc.
> Then I switched to Puppet/Passenger and get the error  below.
> Puppet, Apache and Passenger are all up.
>
> I have installed using *YUM *repos and *GEMs*. So, I have the most 
> updated packages they have.
>
> Puppet version: 2.7.19
> Ruby version: 1.8.7 (2011-06-30 patchlevel 352 i386)
> Apache: 2.2.15
>
> The error is below.
> I have found little references on the web. Has anyone come across such 
> problem recently?
>
> [root@puppetm01 ~]# puppet agent --test
> err: Could not retrieve catalog from remote server: Error 403 on SERVER:
*Forbidden
> request*: puppetm01.example.com(xxx.xxx.xxx.xxx) access to /catalog/
> puppetm01.example.com [find] at line 53
> warning: Not using cache on failed catalog
> err: Could not retrieve catalog; skipping run
> err: Could not send report: Error 403 on SERVER: *Forbidden request*: 
> puppetm01.example.com(xxx.xxx.xxx.xxx) access to /report/
> puppetm01.example.com [save] at line 53
>
> Below is the path to the catalog file to which I believe the error points.
>
> [root@puppetm01 ]# find /var/lib/puppet | grep catalog
> ./client_yaml/catalog
> ./client_yaml/catalog/puppetm01.example.com.yaml
>
> Thanks in advance for any pointers.
> ----
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Puppet Users" group.
> To view this discussion on the web visit 
> https://groups.google.com/d/msg/puppet-users/-/xms_wXhyV2EJ.
> To post to this group, send email to
puppet...@googlegroups.com<javascript:>
> .
> To unsubscribe from this group, send email to 
> puppet-users...@googlegroups.com <javascript:>.
> For more options, visit this group at 
> http://groups.google.com/group/puppet-users?hl=en.
>
>
> -- 
> Jo Rhett
> Net Consonance : net philanthropy to improve open source and internet 
> projects.
>
>
>  
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/sO4Ugfd1vh0J.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Adding more troubleshooting info at the link below.

http://pastebin.com/AvCJSQgk

I recreated the certificates and rebooted the system, but still same result.
I really hope to get to the bottom of this. I cannot find a meaningful 
reference anywhere.

----




-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/lYCWnVNWC8sJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Does anyone have a hint to address this problem?

Or,

Is this destined to stump many a puppet enthusiast?
If this is a bug, where does one notify puppet labs of it?

LL
----

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/gmqnS25CCdYJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

This is a trivial problem to solve, but only you can do it. tcpdump is your
friend.

On Oct 1, 2012, at 11:32 AM, Lunixer wrote:
> Does anyone have a hint to address this problem?
> 
> Or,
> 
> Is this destined to stump many a puppet enthusiast?
> If this is a bug, where does one notify puppet labs of it?
> 
> LL
> ----
> 
> -- 
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/gmqnS25CCdYJ.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
-- 
Jo Rhett
Net Consonance : net philanthropy to improve open source and internet projects.



-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

I don''t think this is trivial. If it were, I would have already found
the
problem by looking at the obvious things.
What I have seen from several posts is that there''s other error similar
to
the one I''ve seen. I even came across a bug report filed a while back
with
the same error I see, but I lost the link and cannot find it.

The problem is not even from a client to the master. The testing I''ve
done
is all in the master.

I''ll try strace instead of tcpdump, being that this is not a TCP 
communication problem over the wire but rather a file or directory access 
problem.


LL
----

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/8D5D3RJ5dw0J.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Oct 1, 2012, at 5:00 PM, Lunixer wrote:
> I''ll try strace instead of tcpdump, being that this is not a TCP
communication problem over the wire but rather a file or directory access
problem.

Um, no. Puppet client talks to the server over the network, even on the same
host. You really should listen to advice we provide.

-- 
Jo Rhett
Net Consonance : net philanthropy to improve open source and internet projects.



-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Wednesday, 3 October 2012 22:45:11 UTC-4, Jo wrote:
>
> On Oct 1, 2012, at 5:00 PM, Lunixer wrote:
>
> I''ll try strace instead of tcpdump, being that this is not a TCP 
> communication problem over the wire but rather a file or directory access 
> problem.
>
>
> Um, no. Puppet client talks to the server over the network, even on the 
> same host. You really should listen to advice we provide. 
>
>
So if the server responds with a 403 error over the network, what exactly 
do you think a tcpdump will show?  The exact same error message.

This is why you would use strace, to see what is happening inside the 
actual process.

Try not being so condescending, particularly when you''re wrong. 

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/DP9BCccRLqEJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.