I am currently trying to get mcollective 2.2.4 up and running on RHEL 6 activeqm appears to be running and bound to the correct port, I see the server and client connections to the activemq box. However when I run mco ping all "seems to go well" except I get "No responses recieved" and above that I get the following log messages, debug 2013/05/29 17:02:12: activemq.rb:233:in `receive'' Waiting for a message from ActiveMQ debug 2013/05/29 17:02:13: pluginmanager.rb:83:in `[]'' Returning cached plugin security_plugin with class MCollective::Security::Ssl debug 2013/05/29 17:02:13: ssl.rb:222:in `deserialize'' De-Serializing using marshal debug 2013/05/29 17:02:13: ssl.rb:175:in `validrequest?'' Validating request from debug 2013/05/29 17:02:13: runnerstats.rb:43:in `unvalidated'' Incrementing unvalidated stat warn 2013/05/29 17:02:13: client.rb:99:in `receive'' Ignoring a message that did not pass security validations debug 2013/05/29 17:02:13: activemq.rb:233:in `receive'' Waiting for a message from ActiveMQ debug 2013/05/29 17:02:17: client.rb:73:in `unsubscribe'' Unsubscribing reply target for discovery Does anyone have any suggestions as to where to look next? -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Does /etc/mcollective/server.cfg look correct (port, servername, etc.)? Is activemq running (and on correct port)? iptables blocking anything? The MCollective module can make things a little easier, btw: https://forge.puppetlabs.com/puppetlabs/mcollective On Wed, May 29, 2013 at 2:12 PM, Mike Bowden <michael.r.bowden@gmail.com>wrote:> I am currently trying to get mcollective 2.2.4 up and running on RHEL > 6 activeqm appears to be running and bound to the correct port, I see > the server and client connections to the activemq box. However when I > run mco ping all "seems to go well" except I get "No responses > recieved" and above that I get the following log messages, > > debug 2013/05/29 17:02:12: activemq.rb:233:in `receive'' Waiting for a > message from ActiveMQ > debug 2013/05/29 17:02:13: pluginmanager.rb:83:in `[]'' Returning > cached plugin security_plugin with class MCollective::Security::Ssl > debug 2013/05/29 17:02:13: ssl.rb:222:in `deserialize'' De-Serializing > using marshal > debug 2013/05/29 17:02:13: ssl.rb:175:in `validrequest?'' Validating > request from > debug 2013/05/29 17:02:13: runnerstats.rb:43:in `unvalidated'' > Incrementing unvalidated stat > warn 2013/05/29 17:02:13: client.rb:99:in `receive'' Ignoring a message > that did not pass security validations > debug 2013/05/29 17:02:13: activemq.rb:233:in `receive'' Waiting for a > message from ActiveMQ > debug 2013/05/29 17:02:17: client.rb:73:in `unsubscribe'' Unsubscribing > reply target for discovery > > > Does anyone have any suggestions as to where to look next? > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > > >-- GVoice: 707.410.0371 LinkedIn: http://www.linkedin.com/in/brendanobra -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
michael.r.bowden@gmail.com
2013-May-30 11:40 UTC
Re: [Puppet Users] mcollective setup question
Thank you for your reply; let me share Sentry is the server, lockbox is the client and scribe is activemq iptables shut off on all, all sit in same vlan Port up and listening with connection from server [root@scribe activemq]# netstat -na | grep 61614 tcp 0 0 :::61614 :::* LISTEN tcp 0 0 ::ffff:139.140.228.18:61614 ::ffff:139.140.228.15:58976 ESTABLISHED /etc/mcollective/server.cfg below # /etc/mcollective/server.cfg # Connector settings (required): # ----------------------------- connector = activemq direct_addressing = 1 # ActiveMQ connector settings: plugin.activemq.pool.size = 1 plugin.activemq.pool.1.host = scribe.bowdoin.edu plugin.activemq.pool.1.port = 61614 plugin.activemq.pool.1.user = mcollective plugin.activemq.pool.1.password = secret plugin.activemq.pool.1.ssl = 1 plugin.activemq.pool.1.ssl.ca = /var/lib/puppet/ssl/certs/ca.pem plugin.activemq.pool.1.ssl.cert = /var/lib/puppet/ssl/certs/sentry.bowdoin.edu.pem plugin.activemq.pool.1.ssl.key = /var/lib/puppet/ssl/private_keys/sentry.bowdoin.edu.pem plugin.activemq.pool.1.ssl.fallback = 0 # RabbitMQ connector settings: plugin.rabbitmq.vhost = /mcollective plugin.rabbitmq.pool.size = 1 plugin.rabbitmq.pool.1.host = scribe.bowdoin.edu # ... etc., similar to activemq settings # Security plugin settings (required): # ----------------------------------- securityprovider = ssl # SSL plugin settings: plugin.ssl_client_cert_dir = /etc/mcollective/clients plugin.ssl_server_private = /etc/mcollective/sentry.bowdoin.edu-priv.pem plugin.ssl_server_public = /etc/mcollective/sentry.bowdoin.edu-pub.pem # PSK plugin settings: plugin.psk = j9q8kx7fnuied9e # Facts, identity, and classes (recommended): # ------------------------------------------ factsource = yaml plugin.yaml = /etc/mcollective/facts.yaml fact_cache_time = 300 identity = sentry.bowdoin.edu classesfile = /var/lib/puppet/state/classes.txt # Registration (recommended): # ----------------------- registerinterval = 600 registration = agentlist registration_collective = mcollective # Subcollectives (optional): # ------------------------- #collectives = mcollective,uk_collective #main_collective = mcollective # Auditing (optional): # ------------------- rpcaudit = 1 rpcauditprovider = logfile plugin.rpcaudit.logfile = /var/log/mcollective-audit.log # Authorization (optional): # ------------------------ rpcauthorization = 1 rpcauthprovider = action_policy # Logging: # ------- logger_type = file loglevel = debug logfile = /var/log/mcollective.log keeplogs = 5 max_log_size = 2097152 logfacility = user # Platform defaults: # ----------------- daemonize = 1 libdir = /usr/libexec/mcollective ssl_cipher = aes-256-cbc Client Config located ~/.mcollective # ~/.mcollective # or # /etc/mcollective/client.cfg # Connector settings (required): # ----------------------------- connector = activemq direct_addressing = 1 # ActiveMQ connector settings: plugin.activemq.pool.size = 1 plugin.activemq.pool.1.host = scribe.bowdoin.edu plugin.activemq.pool.1.port = 61614 plugin.activemq.pool.1.user = mcollective plugin.activemq.pool.1.password = secret plugin.activemq.pool.1.ssl = 1 plugin.activemq.pool.1.ssl.ca = /home/testuser/.mcollective.d/credentials/certs/ca.pem plugin.activemq.pool.1.ssl.cert = /home/testuser/.mcollective.d/credentials/certs/lockbox.bowdoin.edu.pem plugin.activemq.pool.1.ssl.key = /home/testuser/.mcollective.d/credentials/private_keys/lockbox.bowdoin.edu.pem plugin.activemq.pool.1.ssl.fallback = 0 # RabbitMQ connector settings: plugin.rabbitmq.vhost = /mcollective plugin.rabbitmq.pool.size = 1 plugin.rabbitmq.pool.1.host = scribe.bowdoin.edu plugin.rabbitmq.pool.1.port = 61613 # ... etc., similar to activemq settings # Security plugin settings (required): # ----------------------------------- securityprovider = ssl # SSL plugin settings: plugin.ssl_server_public = /home/testuser/.mcollective.d/credentials/certs/mcollective-servers.pem #plugin.ssl_server_public = /home/testuser/.mcollective.d/credentials/certs/lockbox.bowdoin.edu.pem plugin.ssl_client_private = /home/testuser/.mcollective.d/credentials/private_keys/testuser.pem plugin.ssl_client_public = /home/testuser/.mcollective.d/credentials/certs/testuser.pem # PSK plugin settings: plugin.psk = j9q8kx7fnuied9e # Interface settings (optional): # ------------------------------ # Discovery settings: default_discovery_method = mc # default_discovery_options = /etc/mcollective/nodes.txt # Performance settings: direct_addressing_threshold = 10 ttl = 60 # Miscellaneous settings: color = 1 rpclimitmethod = first # Subcollectives (optional): # ----------------------------------- collectives = mcollective,uk_collective main_collective = mcollective # Advanced settings and platform defaults: # ----------------------------------- logger_type = console loglevel = debug logfile = /var/log/mcollective.log keeplogs = 5 max_log_size = 2097152 logfacility = user libdir = /usr/libexec/mcollective rpchelptemplate = /etc/mcollective/rpc-help.erb helptemplatedir = /etc/mcollective ssl_cipher = aes-256-cbc On Wednesday, May 29, 2013 5:18:20 PM UTC-4, Brendan O''Bra wrote:> Does /etc/mcollective/server.cfg look correct (port, servername, etc.)? > Is activemq running (and on correct port)? > iptables blocking anything? > > The MCollective module can make things a little easier, btw: > https://forge.puppetlabs.com/puppetlabs/mcollective > > > On Wed, May 29, 2013 at 2:12 PM, Mike Bowden <michael....@gmail.com<javascript:> > > wrote: > >> I am currently trying to get mcollective 2.2.4 up and running on RHEL >> 6 activeqm appears to be running and bound to the correct port, I see >> the server and client connections to the activemq box. However when I >> run mco ping all "seems to go well" except I get "No responses >> recieved" and above that I get the following log messages, >> >> debug 2013/05/29 17:02:12: activemq.rb:233:in `receive'' Waiting for a >> message from ActiveMQ >> debug 2013/05/29 17:02:13: pluginmanager.rb:83:in `[]'' Returning >> cached plugin security_plugin with class MCollective::Security::Ssl >> debug 2013/05/29 17:02:13: ssl.rb:222:in `deserialize'' De-Serializing >> using marshal >> debug 2013/05/29 17:02:13: ssl.rb:175:in `validrequest?'' Validating >> request from >> debug 2013/05/29 17:02:13: runnerstats.rb:43:in `unvalidated'' >> Incrementing unvalidated stat >> warn 2013/05/29 17:02:13: client.rb:99:in `receive'' Ignoring a message >> that did not pass security validations >> debug 2013/05/29 17:02:13: activemq.rb:233:in `receive'' Waiting for a >> message from ActiveMQ >> debug 2013/05/29 17:02:17: client.rb:73:in `unsubscribe'' Unsubscribing >> reply target for discovery >> >> >> Does anyone have any suggestions as to where to look next? >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to puppet-users...@googlegroups.com <javascript:>. >> To post to this group, send email to puppet...@googlegroups.com<javascript:> >> . >> Visit this group at http://groups.google.com/group/puppet-users?hl=en. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> > > > -- > GVoice: 707.410.0371 <#> > LinkedIn: http://www.linkedin.com/in/brendanobra > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
I''ve been having the same issue with my new MCollective setup and a
quick
look in the client.rb file reveals this function on the line 99 area:
raise(MsgDoesNotMatchRequestID, "Message reqid #{requestid} does not match
our reqid #{reply.requestid}") unless reply.requestid == requestid
rescue SecurityValidationFailed => e
Log.warn("Ignoring a message that did not pass security validations")
and looking in the logs you can find the request ID for the mco command:
base.rb:178:in `create_request'' Encoding a request for agent
''discovery'' in
collective mcollective with request id 2f1214f66d1f5741bb79daa26ed6316e
client.rb:51:in `sendreq'' Sending request
2f1214f66d1f5741bb79daa26ed6316e
to the discovery agent with ttl 60 in collective mcollective
So it would seem that the client is expecting the reponse from the server
to have the same requestid as the one it sent, but for some reason they are
different. If the server is using SSL to generate/verify these request
id''s
it may point to a cert or pki mismatch issue but so far I haven''t been
able
to sport anything wrong with either. Ruvy isn''t my strong suit so it
will
probably need a ruby dev or one of the puppet team to explain how these ids
are generated and verified.
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
Anybody else had any luck solving this? I''ve tried on 3 separate boxes now all seem to suffer the same problem, and the logs are somewhat unhelpful is tracking down why this is occurring. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Was there ever a solution to this issue?
I appear to be having a similar (if not the same) issue:
By adding some debug code to client.rb, it appears that the replies I''m
getting have no requestid at all.
< Log.warn("Ignoring a message that did not pass security
validations")
---> Log.warn("Ignoring a message that did not pass security
validations: requestid #{requestid} - #{reply.requestid}")
Which produces the following message when I run mco ping (or any other
command):
warn 2013/10/14 10:26:13: client.rb:99:in `rescue in receive'' Ignoring
a
message that did not pass security validations: requestid
8d0b3938113259a3ac7d878b365ad522 -
Looking at mcollective.log on the server (set to ''debug''
level), I see the
request being processed:
D, [2013-10-14T10:26:13.237675 #12265] DEBUG -- : runnerstats.rb:49:in
`received'' Incrementing total stat
D, [2013-10-14T10:26:13.237794 #12265] DEBUG -- : pluginmanager.rb:83:in
`[]'' Returning cached plugin security_plugin with class
MCollective::Security::Ssl
D, [2013-10-14T10:26:13.237896 #12265] DEBUG -- : ssl.rb:222:in
`deserialize'' De-Serializing using marshal
D, [2013-10-14T10:26:13.238027 #12265] DEBUG -- : ssl.rb:175:in
`validrequest?'' Validating request from cert=jeg-mc
D, [2013-10-14T10:26:13.238868 #12265] DEBUG -- : runnerstats.rb:38:in
`validated'' Incrementing validated stat
D, [2013-10-14T10:26:13.238958 #12265] DEBUG -- : ssl.rb:222:in
`deserialize'' De-Serializing using marshal
D, [2013-10-14T10:26:13.239067 #12265] DEBUG -- : pluginmanager.rb:83:in
`[]'' Returning cached plugin security_plugin with class
MCollective::Security::Ssl
D, [2013-10-14T10:26:13.239186 #12265] DEBUG -- : pluginmanager.rb:83:in
`[]'' Returning cached plugin security_plugin with class
MCollective::Security::Ssl
D, [2013-10-14T10:26:13.239316 #12265] DEBUG -- : base.rb:153:in
`validate_filter?'' Message passed the filter checks
D, [2013-10-14T10:26:13.239487 #12265] DEBUG -- : runnerstats.rb:26:in
`passed'' Incrementing passed stat
D, [2013-10-14T10:26:13.239557 #12265] DEBUG -- : runner.rb:80:in
`agentmsg'' Handling message for agent ''discovery'' on
collective
''mcollective''
D, [2013-10-14T10:26:13.239624 #12265] DEBUG -- : agents.rb:119:in
`dispatch'' Dispatching a message to agent discovery
D, [2013-10-14T10:26:13.239713 #12265] DEBUG -- : pluginmanager.rb:83:in
`[]'' Returning cached plugin discovery_agent with class
MCollective::Agent::Discovery
D, [2013-10-14T10:26:13.239782 #12265] DEBUG -- : activemq.rb:233:in
`receive'' Waiting for a message from ActiveMQ
D, [2013-10-14T10:26:13.240053 #12265] DEBUG -- : pluginmanager.rb:83:in
`[]'' Returning cached plugin security_plugin with class
MCollective::Security::Ssl
D, [2013-10-14T10:26:13.240360 #12265] DEBUG -- : pluginmanager.rb:83:in
`[]'' Returning cached plugin security_plugin with class
MCollective::Security::Ssl
D, [2013-10-14T10:26:13.240440 #12265] DEBUG -- : ssl.rb:208:in
`serialize''
Serializing using marshal
D, [2013-10-14T10:26:13.240525 #12265] DEBUG -- : ssl.rb:307:in
`makehash''
Creating message hash using /etc/mcollective/server_private.pem
D, [2013-10-14T10:26:13.243567 #12265] DEBUG -- : base.rb:168:in
`create_reply'' Encoded a message for request
8d0b3938113259a3ac7d878b365ad522
D, [2013-10-14T10:26:13.243670 #12265] DEBUG -- : ssl.rb:208:in
`serialize''
Serializing using marshal
D, [2013-10-14T10:26:13.243956 #12265] DEBUG -- : pluginmanager.rb:83:in
`[]'' Returning cached plugin connector_plugin with class
MCollective::Connector::Activemq
D, [2013-10-14T10:26:13.244059 #12265] DEBUG -- : activemq.rb:266:in
`publish'' Sending a broadcast message to ActiveMQ target
''/queue/mcollective.reply.wolf.shands.ufl.edu_24887'' with
headers ''{}''
D, [2013-10-14T10:26:13.245763 #12265] DEBUG -- : runnerstats.rb:56:in
`sent'' Incrementing replies stat
It looks suspiciously like the reply created by base.rb is empty-- But
I''m
not sure how to proceed from here with troubleshooting since I don''t
know
what a valid reply looks like.
On Friday, June 7, 2013 4:25:19 AM UTC-4, Richard Knight
wrote:>
> Anybody else had any luck solving this? I''ve tried on 3 separate
boxes now
> all seem to suffer the same problem, and the logs are somewhat unhelpful is
> tracking down why this is occurring.
>
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.