thr3ads.net - Puppet users - [Puppet Users] Puppet interacting with CloudStack API (Ruby API client) [Oct 2012]

If this information is useful, please help other people find it:
Share via:

Ray

2012-Oct-23 14:20 UTC

[Puppet Users] Puppet interacting with CloudStack API (Ruby API client)

Hi,

I want Puppet to talk to the CloudStack API. CloudStack has many 
asynchronous API calls and Puppet''s catalog run tends to expire even
during
simple tasks such as adding port forwarding rules:

root@puppet30client:~# puppet agent --test
Info: Retrieving plugin
Info: Loading facts in /var/lib/puppet/lib/facter/centos_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/network-bonds.rb
Info: Loading facts in /var/lib/puppet/lib/facter/powerstates.rb
Info: Loading facts in /var/lib/puppet/lib/facter/software_raid.rb
Info: Loading facts in /var/lib/puppet/lib/facter/centos-version.rb

Error: Could not retrieve catalog from remote server: execution expired
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
root@puppet30client:~# 


When I grant the puppetrun more time than the default 2 minutes, it comes 
back successfully:

root@puppet30client:~# puppet agent --test --configtimeout 5m
Info: Retrieving plugin
Info: Loading facts in /var/lib/puppet/lib/facter/centos_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/network-bonds.rb
Info: Loading facts in /var/lib/puppet/lib/facter/powerstates.rb
Info: Loading facts in /var/lib/puppet/lib/facter/software_raid.rb
Info: Loading facts in /var/lib/puppet/lib/facter/centos-version.rb

Info: Caching catalog for puppet30client.swisstxt.ch
Info: Applying configuration version ''1351000748''
Finished catalog run in 1.88 seconds
root@puppet30client:~# 

(Don''t believe the 1.88 seconds - this run took about three minutes(!))


My question is: What is the best practise to let Puppet interact with the 
CloudStack API?


What I did:

1) Created a csapiclient module and dumped a custom function: 
/etc/puppet/modules/csapiclient/lib/puppet/parser/functions/add_pf_rule.rb

require File.join(File.dirname(__FILE__), ''cloudstack_client'')

module Puppet::Parser::Functions
  newfunction(:add_pf_rule) do |args|
    apiurl = args[0]
    apikey = args[1]
    secretkey = args[2]
    proto = args[3]
    pubPort = args[4]
    privPort = args[5]
    ipId = args[6]
    vmId = args[7]

    cs = CloudstackClient::Connection.new(apiurl,apikey,secretkey)

    cs.create_port_forwarding_rule( ipId, privPort, proto, pubPort, vmId)
  end
end

2) Wrote a defined resource that calls the Ruby function:

    define addPFRule( $ipId, $vmId, $proto, $pubPort, $privPort ) {
        # Pass arguments to Ruby function…
        add_pf_rule( $apiUrl, $apiKey , $secretKey, $proto, $pubPort, 
$privPort, $ipId, $vmId )
    }

3) Call the define from a recipe, feeding it with parameters.

Is this a viable way to go or are there better/simpler/more beautiful 
solutions to talk to CloudStack?

Cheers,
Raiy

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/8iKDPrIYmFoJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Jeff McCune

2012-Oct-23 15:22 UTC

head link

Re: [Puppet Users] Puppet interacting with CloudStack API (Ruby API client)

On Tue, Oct 23, 2012 at 7:20 AM, Ray <ray@renegade.zapto.org> wrote:
> Hi,
>
> I want Puppet to talk to the CloudStack API. CloudStack has many
> asynchronous API calls and Puppet''s catalog run tends to expire
even during
> simple tasks such as adding port forwarding rules:
>
Is it the catalog run, or the catalog retrieval that''s expiring?  In
the
example you provided the configtimeout setting affects how long the agent
waits for the catalog to be compiled on the master and retrieved by the
agent.  Catalog runs, which is the process of applying a catalog to a
system, shouldn''t have a timeout by default.

Hope this helps,
-Jeff

> root@puppet30client:~# puppet agent --test
> Info: Retrieving plugin
> Info: Loading facts in /var/lib/puppet/lib/facter/centos_version.rb
> Info: Loading facts in /var/lib/puppet/lib/facter/network-bonds.rb
> Info: Loading facts in /var/lib/puppet/lib/facter/powerstates.rb
> Info: Loading facts in /var/lib/puppet/lib/facter/software_raid.rb
> Info: Loading facts in /var/lib/puppet/lib/facter/centos-version.rb
>
> Error: Could not retrieve catalog from remote server: execution expired
> Warning: Not using cache on failed catalog
> Error: Could not retrieve catalog; skipping run
> root@puppet30client:~#
>
>
> When I grant the puppetrun more time than the default 2 minutes, it comes
> back successfully:
>
> root@puppet30client:~# puppet agent --test --configtimeout 5m
> Info: Retrieving plugin
> Info: Loading facts in /var/lib/puppet/lib/facter/centos_version.rb
> Info: Loading facts in /var/lib/puppet/lib/facter/network-bonds.rb
> Info: Loading facts in /var/lib/puppet/lib/facter/powerstates.rb
> Info: Loading facts in /var/lib/puppet/lib/facter/software_raid.rb
> Info: Loading facts in /var/lib/puppet/lib/facter/centos-version.rb
>
> Info: Caching catalog for puppet30client.swisstxt.ch
> Info: Applying configuration version ''1351000748''
> Finished catalog run in 1.88 seconds
> root@puppet30client:~#
>
> (Don''t believe the 1.88 seconds - this run took about three
minutes(!))
>
There''s something wrong if Puppet is lying to you that much.  Could you
run
with --evaltrace, --trace, and --debug and paste the output somewhere.  A
copy of the YAML report might also help diagnose the issue.

Hope this helps,
-Jeff

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Ray

2012-Oct-23 16:02 UTC

head link

[Puppet Users] Re: Puppet interacting with CloudStack API (Ruby API client)

root@puppet30client:~# puppet agent --evaltrace --trace --test 
--configtimeout 5m
Info: Retrieving plugin
Info: Loading facts in /var/lib/puppet/lib/facter/centos_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/network-bonds.rb
Info: Loading facts in /var/lib/puppet/lib/facter/powerstates.rb
Info: Loading facts in /var/lib/puppet/lib/facter/software_raid.rb
Info: Loading facts in /var/lib/puppet/lib/facter/centos-version.rb
Info: Caching catalog for puppet30client
Info: Applying configuration version ''1351007512''
Info: /Schedule[daily]: Starting to evaluate the resource
Info: /Schedule[daily]: Evaluated in 0.00 seconds
Info: /Schedule[monthly]: Starting to evaluate the resource
Info: /Schedule[monthly]: Evaluated in 0.00 seconds
Info: /Schedule[hourly]: Starting to evaluate the resource
Info: /Schedule[hourly]: Evaluated in 0.00 seconds
Info: Stage[main]: Starting to evaluate the resource
Info: Stage[main]: Evaluated in 0.00 seconds
Info: Class[Main]: Starting to evaluate the resource
Info: Class[Main]: Evaluated in 0.00 seconds
Info: Class[Yum]: Starting to evaluate the resource
Info: Class[Yum]: Evaluated in 0.00 seconds
Info: /Stage[main]/Yum/Package[yum-plugin-priorities]: Starting to evaluate 
the resource
Info: /Stage[main]/Yum/Package[yum-plugin-priorities]: Evaluated in 0.00 
seconds
Info: Set_yum_repo[epel]: Starting to evaluate the resource
Info: Set_yum_repo[epel]: Evaluated in 0.04 seconds
Info: Class[Csapiclient]: Starting to evaluate the resource
Info: Class[Csapiclient]: Evaluated in 0.00 seconds
Info: Node[puppet30client]: Starting to evaluate the resource
Info: Node[puppet30client]: Evaluated in 0.00 seconds
Info: Class[Centos]: Starting to evaluate the resource
Info: Class[Centos]: Evaluated in 0.00 seconds
Info: /File[/etc/profile.d/raimunds_favorite_aliases.sh]: Starting to 
evaluate the resource
Info: /File[/etc/profile.d/raimunds_favorite_aliases.sh]: Evaluated in 0.00 
seconds
Info: Replace_line[Set_default_global_HISTSIZE]: Starting to evaluate the 
resource
Info: Replace_line[Set_default_global_HISTSIZE]: Evaluated in 0.00 seconds
Info: Class[Settings]: Starting to evaluate the resource
Info: Class[Settings]: Evaluated in 0.00 seconds
Info: Class[Settings]: Starting to evaluate the resource
Info: Class[Settings]: Evaluated in 0.00 seconds
Info: /Stage[main]/Yum/Package[yum-plugin-fastestmirror]: Starting to 
evaluate the resource
Info: /Stage[main]/Yum/Package[yum-plugin-fastestmirror]: Evaluated in 0.00 
seconds
Info: Class[Puppet::Client]: Starting to evaluate the resource
Info: Class[Puppet::Client]: Evaluated in 0.00 seconds
Info: /File[yum_epel.repo]: Starting to evaluate the resource
Info: /File[yum_epel.repo]: Evaluated in 0.00 seconds
Info: Set_yum_repo[epel]: Starting to evaluate the resource
Info: Set_yum_repo[epel]: Evaluated in 0.00 seconds
Info: /Stage[main]/Centos/Package[bind-utils]: Starting to evaluate the 
resource
Info: /Stage[main]/Centos/Package[bind-utils]: Evaluated in 0.00 seconds
Info: /Stage[main]/Centos/Package[xorg-x11-fonts-Type1]: Starting to 
evaluate the resource
Info: /Stage[main]/Centos/Package[xorg-x11-fonts-Type1]: Evaluated in 0.00 
seconds
Info: /Stage[main]/Centos/Package[make]: Starting to evaluate the resource
Info: /Stage[main]/Centos/Package[make]: Evaluated in 0.00 seconds
Info: /Stage[main]/Centos/Package[finger]: Starting to evaluate the resource
Info: /Stage[main]/Centos/Package[finger]: Evaluated in 0.00 seconds
Info: /Stage[main]/Centos/Package[openssh-clients]: Starting to evaluate 
the resource
Info: /Stage[main]/Centos/Package[openssh-clients]: Evaluated in 0.00 
seconds
Info: /Stage[main]/Centos/Package[traceroute]: Starting to evaluate the 
resource
Info: /Stage[main]/Centos/Package[traceroute]: Evaluated in 0.00 seconds
Info: /Stage[main]/Centos/Package[htop]: Starting to evaluate the resource
Info: /Stage[main]/Centos/Package[htop]: Evaluated in 0.00 seconds
Info: /Stage[main]/Centos/Package[tcpdump]: Starting to evaluate the 
resource
Info: /Stage[main]/Centos/Package[tcpdump]: Evaluated in 0.00 seconds
Info: /Stage[main]/Centos/Package[wget]: Starting to evaluate the resource
Info: /Stage[main]/Centos/Package[wget]: Evaluated in 0.00 seconds
Info: /Stage[main]/Centos/Package[screen]: Starting to evaluate the resource
Info: /Stage[main]/Centos/Package[screen]: Evaluated in 0.00 seconds
Info: /Stage[main]/Yum/Exec[yum-clean-all]: Starting to evaluate the 
resource
Info: /Stage[main]/Yum/Exec[yum-clean-all]: Evaluated in 0.00 seconds
Info: /Stage[main]/Centos/Package[vim-common]: Starting to evaluate the 
resource
Info: /Stage[main]/Centos/Package[vim-common]: Evaluated in 0.00 seconds
Info: /Stage[main]/Centos/Package[sysstat]: Starting to evaluate the 
resource
Info: /Stage[main]/Centos/Package[sysstat]: Evaluated in 0.00 seconds
Info: /Stage[main]/Centos/Package[man]: Starting to evaluate the resource
Info: /Stage[main]/Centos/Package[man]: Evaluated in 0.00 seconds
Info: /Stage[main]/Centos/Package[curl]: Starting to evaluate the resource
Info: /Stage[main]/Centos/Package[curl]: Evaluated in 0.00 seconds
Info: /Stage[main]/Yum/Package[yum-metadata-parser]: Starting to evaluate 
the resource
Info: /Stage[main]/Yum/Package[yum-metadata-parser]: Evaluated in 0.00 
seconds
Info: /Stage[main]/Centos/Package[vim-enhanced]: Starting to evaluate the 
resource
Info: /Stage[main]/Centos/Package[vim-enhanced]: Evaluated in 0.00 seconds
Info: /Stage[main]/Yum/Package[yum]: Starting to evaluate the resource
Info: /Stage[main]/Yum/Package[yum]: Evaluated in 0.00 seconds
Info: Class[Yum]: Starting to evaluate the resource
Info: Class[Yum]: Evaluated in 0.00 seconds
Info: Class[Common]: Starting to evaluate the resource
Info: Class[Common]: Evaluated in 0.00 seconds
Info: Class[Common]: Starting to evaluate the resource
Info: Class[Common]: Evaluated in 0.00 seconds
Info: /Stage[main]/Centos/Package[telnet]: Starting to evaluate the resource
Info: /Stage[main]/Centos/Package[telnet]: Evaluated in 0.00 seconds
Info: /File[/root/.ssh]: Starting to evaluate the resource
Info: /File[/root/.ssh]: Evaluated in 0.00 seconds
Info: /Stage[main]/Centos/Package[pbzip2]: Starting to evaluate the resource
Info: /Stage[main]/Centos/Package[pbzip2]: Evaluated in 0.00 seconds
Info: /Schedule[never]: Starting to evaluate the resource
Info: /Schedule[never]: Evaluated in 0.00 seconds
Info: /Stage[main]/Centos/Package[lsof]: Starting to evaluate the resource
Info: /Stage[main]/Centos/Package[lsof]: Evaluated in 0.00 seconds
Info: /Stage[main]/Centos/Package[sudo]: Starting to evaluate the resource
Info: /Stage[main]/Centos/Package[sudo]: Evaluated in 0.00 seconds
Info: /Stage[main]/Centos/Package[bc]: Starting to evaluate the resource
Info: /Stage[main]/Centos/Package[bc]: Evaluated in 0.00 seconds
Info: /File[/root/.vimrc]: Starting to evaluate the resource
Info: /File[/root/.vimrc]: Evaluated in 0.00 seconds
Info: /Filebucket[puppet]: Starting to evaluate the resource
Info: /Filebucket[puppet]: Evaluated in 0.00 seconds
Info: /Stage[main]/Centos/Package[nc]: Starting to evaluate the resource
Info: /Stage[main]/Centos/Package[nc]: Evaluated in 0.00 seconds
Info: /Stage[main]/Centos/Package[pigz]: Starting to evaluate the resource
Info: /Stage[main]/Centos/Package[pigz]: Evaluated in 0.00 seconds
Info: /Schedule[weekly]: Starting to evaluate the resource
Info: /Schedule[weekly]: Evaluated in 0.00 seconds
Info: Node[basenode]: Starting to evaluate the resource
Info: Node[basenode]: Evaluated in 0.00 seconds
Info: Node[basenode]: Starting to evaluate the resource
Info: Node[basenode]: Evaluated in 0.00 seconds
Info: /Schedule[puppet]: Starting to evaluate the resource
Info: /Schedule[puppet]: Evaluated in 0.00 seconds
Info: Class[Csapiclient]: Starting to evaluate the resource
Info: Class[Csapiclient]: Evaluated in 0.00 seconds
Info: Csapiclient::Addpfrule[forwardTCP324]: Starting to evaluate the 
resource
Info: Csapiclient::Addpfrule[forwardTCP324]: Evaluated in 0.00 seconds
Info: Csapiclient::Addpfrule[forwardTCP324]: Starting to evaluate the 
resource
Info: Csapiclient::Addpfrule[forwardTCP324]: Evaluated in 0.00 seconds
Info: Node[puppet30client]: Starting to evaluate the resource
Info: Node[puppet30client]: Evaluated in 0.00 seconds
Info: Class[Main]: Starting to evaluate the resource
Info: Class[Main]: Evaluated in 0.00 seconds
Info: /Stage[main]/Centos/Package[augeas]: Starting to evaluate the resource
Info: /Stage[main]/Centos/Package[augeas]: Evaluated in 0.00 seconds
Info: /File[/etc/profile.d/raimunds_favorite_bash_prompt.sh]: Starting to 
evaluate the resource
Info: /File[/etc/profile.d/raimunds_favorite_bash_prompt.sh]: Evaluated in 
0.09 seconds
Info: 
/Stage[main]/Centos/Replace_line[Set_default_global_HISTSIZE]/Exec[/bin/sed 
-i -r -e ''s/HISTSIZE=1000/HISTSIZE=1000000/'' /etc/profile]:
Starting to
evaluate the resource
Info: 
/Stage[main]/Centos/Replace_line[Set_default_global_HISTSIZE]/Exec[/bin/sed 
-i -r -e ''s/HISTSIZE=1000/HISTSIZE=1000000/'' /etc/profile]:
Evaluated in
0.06 seconds
Info: Replace_line[Set_default_global_HISTSIZE]: Starting to evaluate the 
resource
Info: Replace_line[Set_default_global_HISTSIZE]: Evaluated in 0.00 seconds
Info: Class[Centos]: Starting to evaluate the resource
Info: Class[Centos]: Evaluated in 0.00 seconds
Info: /Stage[main]/Puppet::Client/Package[puppet]: Starting to evaluate the 
resource
Info: /Stage[main]/Puppet::Client/Package[puppet]: Evaluated in 0.00 seconds
Info: /Stage[main]/Puppet::Client/Service[puppet]: Starting to evaluate the 
resource
Info: /Stage[main]/Puppet::Client/Service[puppet]: Evaluated in 0.05 seconds
Info: Class[Puppet::Client]: Starting to evaluate the resource
Info: Class[Puppet::Client]: Evaluated in 0.00 seconds
Info: Stage[main]: Starting to evaluate the resource
Info: Stage[main]: Evaluated in 0.00 seconds
Finished catalog run in 1.87 seconds

This again took like three minutes. When I drop the CSApiClient entry it 
runs quick, then the timing appears right.

I assume the Cloudstack API takes agaes to respond and that''s where
things
hang. As this is evaluated on the server, I assume the cataloge compilation 
(and therefore the retrieval also) is taking ages. The stuff that happens 
on the client appears to be quick.

Cheers,
Ray


Am Dienstag, 23. Oktober 2012 16:20:25 UTC+2 schrieb
Ray:>
> Hi,
>
> I want Puppet to talk to the CloudStack API. CloudStack has many 
> asynchronous API calls and Puppet''s catalog run tends to expire
even during
> simple tasks such as adding port forwarding rules:
>
> root@puppet30client:~# puppet agent --test
> Info: Retrieving plugin
> Info: Loading facts in /var/lib/puppet/lib/facter/centos_version.rb
> Info: Loading facts in /var/lib/puppet/lib/facter/network-bonds.rb
> Info: Loading facts in /var/lib/puppet/lib/facter/powerstates.rb
> Info: Loading facts in /var/lib/puppet/lib/facter/software_raid.rb
> Info: Loading facts in /var/lib/puppet/lib/facter/centos-version.rb
>
> Error: Could not retrieve catalog from remote server: execution expired
> Warning: Not using cache on failed catalog
> Error: Could not retrieve catalog; skipping run
> root@puppet30client:~# 
>
>
> When I grant the puppetrun more time than the default 2 minutes, it comes 
> back successfully:
>
> root@puppet30client:~# puppet agent --test --configtimeout 5m
> Info: Retrieving plugin
> Info: Loading facts in /var/lib/puppet/lib/facter/centos_version.rb
> Info: Loading facts in /var/lib/puppet/lib/facter/network-bonds.rb
> Info: Loading facts in /var/lib/puppet/lib/facter/powerstates.rb
> Info: Loading facts in /var/lib/puppet/lib/facter/software_raid.rb
> Info: Loading facts in /var/lib/puppet/lib/facter/centos-version.rb
>
> Info: Caching catalog for puppet30client.swisstxt.ch
> Info: Applying configuration version ''1351000748''
> Finished catalog run in 1.88 seconds
> root@puppet30client:~# 
>
> (Don''t believe the 1.88 seconds - this run took about three
minutes(!))
>
>
> My question is: What is the best practise to let Puppet interact with the 
> CloudStack API?
>
>
> What I did:
>
> 1) Created a csapiclient module and dumped a custom function: 
> /etc/puppet/modules/csapiclient/lib/puppet/parser/functions/add_pf_rule.rb
>
> require File.join(File.dirname(__FILE__),
''cloudstack_client'')
>
> module Puppet::Parser::Functions
>   newfunction(:add_pf_rule) do |args|
>     apiurl = args[0]
>     apikey = args[1]
>     secretkey = args[2]
>     proto = args[3]
>     pubPort = args[4]
>     privPort = args[5]
>     ipId = args[6]
>     vmId = args[7]
>
>     cs = CloudstackClient::Connection.new(apiurl,apikey,secretkey)
>
>     cs.create_port_forwarding_rule( ipId, privPort, proto, pubPort, vmId)
>   end
> end
>
> 2) Wrote a defined resource that calls the Ruby function:
>
>     define addPFRule( $ipId, $vmId, $proto, $pubPort, $privPort ) {
>         # Pass arguments to Ruby function…
>         add_pf_rule( $apiUrl, $apiKey , $secretKey, $proto, $pubPort, 
> $privPort, $ipId, $vmId )
>     }
>
> 3) Call the define from a recipe, feeding it with parameters.
>
> Is this a viable way to go or are there better/simpler/more beautiful 
> solutions to talk to CloudStack?
>
> Cheers,
> Raiy
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/nBxpu-11DlgJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Jeff McCune

2012-Oct-23 23:50 UTC

head link

Re: [Puppet Users] Re: Puppet interacting with CloudStack API (Ruby API client)

Ray,

Are you the author of Class[Csapiclient] ?  If so, the only suggestion I
have is to try and find API calls that return the data you need in a
quicker manner.  The way the catalog and the resource abstraction layer
operate is built on top of an assumption of synchronous and relatively fast
system calls.  Asynchronous and relatively slow network API calls
aren''t
well suited for types and providers.

Hope this helps,
-Jeff

On Tue, Oct 23, 2012 at 9:02 AM, Ray <ray@renegade.zapto.org> wrote:
> root@puppet30client:~# puppet agent --evaltrace --trace --test
> --configtimeout 5m
>
> Info: Retrieving plugin
> Info: Loading facts in /var/lib/puppet/lib/facter/centos_version.rb
> Info: Loading facts in /var/lib/puppet/lib/facter/network-bonds.rb
> Info: Loading facts in /var/lib/puppet/lib/facter/powerstates.rb
> Info: Loading facts in /var/lib/puppet/lib/facter/software_raid.rb
> Info: Loading facts in /var/lib/puppet/lib/facter/centos-version.rb
> Info: Caching catalog for puppet30client
> Info: Applying configuration version ''1351007512''
> Info: /Schedule[daily]: Starting to evaluate the resource
> Info: /Schedule[daily]: Evaluated in 0.00 seconds
> Info: /Schedule[monthly]: Starting to evaluate the resource
> Info: /Schedule[monthly]: Evaluated in 0.00 seconds
> Info: /Schedule[hourly]: Starting to evaluate the resource
> Info: /Schedule[hourly]: Evaluated in 0.00 seconds
> Info: Stage[main]: Starting to evaluate the resource
> Info: Stage[main]: Evaluated in 0.00 seconds
> Info: Class[Main]: Starting to evaluate the resource
> Info: Class[Main]: Evaluated in 0.00 seconds
> Info: Class[Yum]: Starting to evaluate the resource
> Info: Class[Yum]: Evaluated in 0.00 seconds
> Info: /Stage[main]/Yum/Package[yum-plugin-priorities]: Starting to
> evaluate the resource
> Info: /Stage[main]/Yum/Package[yum-plugin-priorities]: Evaluated in 0.00
> seconds
> Info: Set_yum_repo[epel]: Starting to evaluate the resource
> Info: Set_yum_repo[epel]: Evaluated in 0.04 seconds
> Info: Class[Csapiclient]: Starting to evaluate the resource
> Info: Class[Csapiclient]: Evaluated in 0.00 seconds
> Info: Node[puppet30client]: Starting to evaluate the resource
> Info: Node[puppet30client]: Evaluated in 0.00 seconds
> Info: Class[Centos]: Starting to evaluate the resource
> Info: Class[Centos]: Evaluated in 0.00 seconds
> Info: /File[/etc/profile.d/raimunds_favorite_aliases.sh]: Starting to
> evaluate the resource
> Info: /File[/etc/profile.d/raimunds_favorite_aliases.sh]: Evaluated in
> 0.00 seconds
> Info: Replace_line[Set_default_global_HISTSIZE]: Starting to evaluate the
> resource
> Info: Replace_line[Set_default_global_HISTSIZE]: Evaluated in 0.00 seconds
> Info: Class[Settings]: Starting to evaluate the resource
> Info: Class[Settings]: Evaluated in 0.00 seconds
> Info: Class[Settings]: Starting to evaluate the resource
> Info: Class[Settings]: Evaluated in 0.00 seconds
> Info: /Stage[main]/Yum/Package[yum-plugin-fastestmirror]: Starting to
> evaluate the resource
> Info: /Stage[main]/Yum/Package[yum-plugin-fastestmirror]: Evaluated in
> 0.00 seconds
> Info: Class[Puppet::Client]: Starting to evaluate the resource
> Info: Class[Puppet::Client]: Evaluated in 0.00 seconds
> Info: /File[yum_epel.repo]: Starting to evaluate the resource
> Info: /File[yum_epel.repo]: Evaluated in 0.00 seconds
> Info: Set_yum_repo[epel]: Starting to evaluate the resource
> Info: Set_yum_repo[epel]: Evaluated in 0.00 seconds
> Info: /Stage[main]/Centos/Package[bind-utils]: Starting to evaluate the
> resource
> Info: /Stage[main]/Centos/Package[bind-utils]: Evaluated in 0.00 seconds
> Info: /Stage[main]/Centos/Package[xorg-x11-fonts-Type1]: Starting to
> evaluate the resource
> Info: /Stage[main]/Centos/Package[xorg-x11-fonts-Type1]: Evaluated in 0.00
> seconds
> Info: /Stage[main]/Centos/Package[make]: Starting to evaluate the resource
> Info: /Stage[main]/Centos/Package[make]: Evaluated in 0.00 seconds
> Info: /Stage[main]/Centos/Package[finger]: Starting to evaluate the
> resource
> Info: /Stage[main]/Centos/Package[finger]: Evaluated in 0.00 seconds
> Info: /Stage[main]/Centos/Package[openssh-clients]: Starting to evaluate
> the resource
> Info: /Stage[main]/Centos/Package[openssh-clients]: Evaluated in 0.00
> seconds
> Info: /Stage[main]/Centos/Package[traceroute]: Starting to evaluate the
> resource
> Info: /Stage[main]/Centos/Package[traceroute]: Evaluated in 0.00 seconds
> Info: /Stage[main]/Centos/Package[htop]: Starting to evaluate the resource
> Info: /Stage[main]/Centos/Package[htop]: Evaluated in 0.00 seconds
> Info: /Stage[main]/Centos/Package[tcpdump]: Starting to evaluate the
> resource
> Info: /Stage[main]/Centos/Package[tcpdump]: Evaluated in 0.00 seconds
> Info: /Stage[main]/Centos/Package[wget]: Starting to evaluate the resource
> Info: /Stage[main]/Centos/Package[wget]: Evaluated in 0.00 seconds
> Info: /Stage[main]/Centos/Package[screen]: Starting to evaluate the
> resource
> Info: /Stage[main]/Centos/Package[screen]: Evaluated in 0.00 seconds
> Info: /Stage[main]/Yum/Exec[yum-clean-all]: Starting to evaluate the
> resource
> Info: /Stage[main]/Yum/Exec[yum-clean-all]: Evaluated in 0.00 seconds
> Info: /Stage[main]/Centos/Package[vim-common]: Starting to evaluate the
> resource
> Info: /Stage[main]/Centos/Package[vim-common]: Evaluated in 0.00 seconds
> Info: /Stage[main]/Centos/Package[sysstat]: Starting to evaluate the
> resource
> Info: /Stage[main]/Centos/Package[sysstat]: Evaluated in 0.00 seconds
> Info: /Stage[main]/Centos/Package[man]: Starting to evaluate the resource
> Info: /Stage[main]/Centos/Package[man]: Evaluated in 0.00 seconds
> Info: /Stage[main]/Centos/Package[curl]: Starting to evaluate the resource
> Info: /Stage[main]/Centos/Package[curl]: Evaluated in 0.00 seconds
> Info: /Stage[main]/Yum/Package[yum-metadata-parser]: Starting to evaluate
> the resource
> Info: /Stage[main]/Yum/Package[yum-metadata-parser]: Evaluated in 0.00
> seconds
> Info: /Stage[main]/Centos/Package[vim-enhanced]: Starting to evaluate the
> resource
> Info: /Stage[main]/Centos/Package[vim-enhanced]: Evaluated in 0.00 seconds
> Info: /Stage[main]/Yum/Package[yum]: Starting to evaluate the resource
> Info: /Stage[main]/Yum/Package[yum]: Evaluated in 0.00 seconds
> Info: Class[Yum]: Starting to evaluate the resource
> Info: Class[Yum]: Evaluated in 0.00 seconds
> Info: Class[Common]: Starting to evaluate the resource
> Info: Class[Common]: Evaluated in 0.00 seconds
> Info: Class[Common]: Starting to evaluate the resource
> Info: Class[Common]: Evaluated in 0.00 seconds
> Info: /Stage[main]/Centos/Package[telnet]: Starting to evaluate the
> resource
> Info: /Stage[main]/Centos/Package[telnet]: Evaluated in 0.00 seconds
> Info: /File[/root/.ssh]: Starting to evaluate the resource
> Info: /File[/root/.ssh]: Evaluated in 0.00 seconds
> Info: /Stage[main]/Centos/Package[pbzip2]: Starting to evaluate the
> resource
> Info: /Stage[main]/Centos/Package[pbzip2]: Evaluated in 0.00 seconds
> Info: /Schedule[never]: Starting to evaluate the resource
> Info: /Schedule[never]: Evaluated in 0.00 seconds
> Info: /Stage[main]/Centos/Package[lsof]: Starting to evaluate the resource
> Info: /Stage[main]/Centos/Package[lsof]: Evaluated in 0.00 seconds
> Info: /Stage[main]/Centos/Package[sudo]: Starting to evaluate the resource
> Info: /Stage[main]/Centos/Package[sudo]: Evaluated in 0.00 seconds
> Info: /Stage[main]/Centos/Package[bc]: Starting to evaluate the resource
> Info: /Stage[main]/Centos/Package[bc]: Evaluated in 0.00 seconds
> Info: /File[/root/.vimrc]: Starting to evaluate the resource
> Info: /File[/root/.vimrc]: Evaluated in 0.00 seconds
> Info: /Filebucket[puppet]: Starting to evaluate the resource
> Info: /Filebucket[puppet]: Evaluated in 0.00 seconds
> Info: /Stage[main]/Centos/Package[nc]: Starting to evaluate the resource
> Info: /Stage[main]/Centos/Package[nc]: Evaluated in 0.00 seconds
> Info: /Stage[main]/Centos/Package[pigz]: Starting to evaluate the resource
> Info: /Stage[main]/Centos/Package[pigz]: Evaluated in 0.00 seconds
> Info: /Schedule[weekly]: Starting to evaluate the resource
> Info: /Schedule[weekly]: Evaluated in 0.00 seconds
> Info: Node[basenode]: Starting to evaluate the resource
> Info: Node[basenode]: Evaluated in 0.00 seconds
> Info: Node[basenode]: Starting to evaluate the resource
> Info: Node[basenode]: Evaluated in 0.00 seconds
> Info: /Schedule[puppet]: Starting to evaluate the resource
> Info: /Schedule[puppet]: Evaluated in 0.00 seconds
> Info: Class[Csapiclient]: Starting to evaluate the resource
> Info: Class[Csapiclient]: Evaluated in 0.00 seconds
> Info: Csapiclient::Addpfrule[forwardTCP324]: Starting to evaluate the
> resource
> Info: Csapiclient::Addpfrule[forwardTCP324]: Evaluated in 0.00 seconds
> Info: Csapiclient::Addpfrule[forwardTCP324]: Starting to evaluate the
> resource
> Info: Csapiclient::Addpfrule[forwardTCP324]: Evaluated in 0.00 seconds
> Info: Node[puppet30client]: Starting to evaluate the resource
> Info: Node[puppet30client]: Evaluated in 0.00 seconds
> Info: Class[Main]: Starting to evaluate the resource
> Info: Class[Main]: Evaluated in 0.00 seconds
> Info: /Stage[main]/Centos/Package[augeas]: Starting to evaluate the
> resource
> Info: /Stage[main]/Centos/Package[augeas]: Evaluated in 0.00 seconds
> Info: /File[/etc/profile.d/raimunds_favorite_bash_prompt.sh]: Starting to
> evaluate the resource
> Info: /File[/etc/profile.d/raimunds_favorite_bash_prompt.sh]: Evaluated in
> 0.09 seconds
> Info:
> /Stage[main]/Centos/Replace_line[Set_default_global_HISTSIZE]/Exec[/bin/sed
> -i -r -e ''s/HISTSIZE=1000/HISTSIZE=1000000/''
/etc/profile]: Starting to
> evaluate the resource
> Info:
> /Stage[main]/Centos/Replace_line[Set_default_global_HISTSIZE]/Exec[/bin/sed
> -i -r -e ''s/HISTSIZE=1000/HISTSIZE=1000000/''
/etc/profile]: Evaluated in
> 0.06 seconds
> Info: Replace_line[Set_default_global_HISTSIZE]: Starting to evaluate the
> resource
> Info: Replace_line[Set_default_global_HISTSIZE]: Evaluated in 0.00 seconds
> Info: Class[Centos]: Starting to evaluate the resource
> Info: Class[Centos]: Evaluated in 0.00 seconds
> Info: /Stage[main]/Puppet::Client/Package[puppet]: Starting to evaluate
> the resource
> Info: /Stage[main]/Puppet::Client/Package[puppet]: Evaluated in 0.00
> seconds
> Info: /Stage[main]/Puppet::Client/Service[puppet]: Starting to evaluate
> the resource
> Info: /Stage[main]/Puppet::Client/Service[puppet]: Evaluated in 0.05
> seconds
> Info: Class[Puppet::Client]: Starting to evaluate the resource
> Info: Class[Puppet::Client]: Evaluated in 0.00 seconds
> Info: Stage[main]: Starting to evaluate the resource
> Info: Stage[main]: Evaluated in 0.00 seconds
> Finished catalog run in 1.87 seconds
>
> This again took like three minutes. When I drop the CSApiClient entry it
> runs quick, then the timing appears right.
>
> I assume the Cloudstack API takes agaes to respond and that''s
where things
> hang. As this is evaluated on the server, I assume the cataloge compilation
> (and therefore the retrieval also) is taking ages. The stuff that happens
> on the client appears to be quick.
>
> Cheers,
> Ray
>
>
> Am Dienstag, 23. Oktober 2012 16:20:25 UTC+2 schrieb Ray:
>
>> Hi,
>>
>> I want Puppet to talk to the CloudStack API. CloudStack has many
>> asynchronous API calls and Puppet''s catalog run tends to
expire even during
>> simple tasks such as adding port forwarding rules:
>>
>> root@puppet30client:~# puppet agent --test
>> Info: Retrieving plugin
>> Info: Loading facts in /var/lib/puppet/lib/facter/**centos_version.rb
>> Info: Loading facts in /var/lib/puppet/lib/facter/**network-bonds.rb
>> Info: Loading facts in /var/lib/puppet/lib/facter/**powerstates.rb
>> Info: Loading facts in /var/lib/puppet/lib/facter/**software_raid.rb
>> Info: Loading facts in /var/lib/puppet/lib/facter/**centos-version.rb
>>
>> Error: Could not retrieve catalog from remote server: execution expired
>> Warning: Not using cache on failed catalog
>> Error: Could not retrieve catalog; skipping run
>> root@puppet30client:~#
>>
>>
>> When I grant the puppetrun more time than the default 2 minutes, it
comes
>> back successfully:
>>
>> root@puppet30client:~# puppet agent --test --configtimeout 5m
>> Info: Retrieving plugin
>> Info: Loading facts in /var/lib/puppet/lib/facter/**centos_version.rb
>> Info: Loading facts in /var/lib/puppet/lib/facter/**network-bonds.rb
>> Info: Loading facts in /var/lib/puppet/lib/facter/**powerstates.rb
>> Info: Loading facts in /var/lib/puppet/lib/facter/**software_raid.rb
>> Info: Loading facts in /var/lib/puppet/lib/facter/**centos-version.rb
>>
>> Info: Caching catalog for puppet30client.swisstxt.ch
>> Info: Applying configuration version ''1351000748''
>> Finished catalog run in 1.88 seconds
>> root@puppet30client:~#
>>
>> (Don''t believe the 1.88 seconds - this run took about three
minutes(!))
>>
>>
>> My question is: What is the best practise to let Puppet interact with
the
>> CloudStack API?
>>
>>
>> What I did:
>>
>> 1) Created a csapiclient module and dumped a custom function:
>> /etc/puppet/modules/**csapiclient/lib/puppet/parser/**
>> functions/add_pf_rule.rb
>>
>> require File.join(File.dirname(__FILE_**_),
''cloudstack_client'')
>>
>> module Puppet::Parser::Functions
>>   newfunction(:add_pf_rule) do |args|
>>     apiurl = args[0]
>>     apikey = args[1]
>>     secretkey = args[2]
>>     proto = args[3]
>>     pubPort = args[4]
>>     privPort = args[5]
>>     ipId = args[6]
>>     vmId = args[7]
>>
>>     cs = CloudstackClient::Connection.**new(apiurl,apikey,secretkey)
>>
>>     cs.create_port_forwarding_**rule( ipId, privPort, proto, pubPort,
>> vmId)
>>   end
>> end
>>
>> 2) Wrote a defined resource that calls the Ruby function:
>>
>>     define addPFRule( $ipId, $vmId, $proto, $pubPort, $privPort ) {
>>         # Pass arguments to Ruby function…
>>         add_pf_rule( $apiUrl, $apiKey , $secretKey, $proto, $pubPort,
>> $privPort, $ipId, $vmId )
>>     }
>>
>> 3) Call the define from a recipe, feeding it with parameters.
>>
>> Is this a viable way to go or are there better/simpler/more beautiful
>> solutions to talk to CloudStack?
>>
>> Cheers,
>> Raiy
>>
>  --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/puppet-users/-/nBxpu-11DlgJ.
>
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

jcbollinger

2012-Oct-24 14:49 UTC

head link

[Puppet Users] Re: Puppet interacting with CloudStack API (Ruby API client)

On Tuesday, October 23, 2012 9:20:25 AM UTC-5, Ray
wrote:>
> Hi,
>
> I want Puppet to talk to the CloudStack API. CloudStack has many 
> asynchronous API calls and Puppet''s catalog run tends to expire
even during
> simple tasks such as adding port forwarding rules:
>
> root@puppet30client:~# puppet agent --test
> Info: Retrieving plugin
> Info: Loading facts in /var/lib/puppet/lib/facter/centos_version.rb
> Info: Loading facts in /var/lib/puppet/lib/facter/network-bonds.rb
> Info: Loading facts in /var/lib/puppet/lib/facter/powerstates.rb
> Info: Loading facts in /var/lib/puppet/lib/facter/software_raid.rb
> Info: Loading facts in /var/lib/puppet/lib/facter/centos-version.rb
>
> Error: Could not retrieve catalog from remote server: execution expired
> Warning: Not using cache on failed catalog
> Error: Could not retrieve catalog; skipping run
> root@puppet30client:~# 
>
>
> When I grant the puppetrun more time than the default 2 minutes, it comes 
> back successfully:
>
> root@puppet30client:~# puppet agent --test --configtimeout 5m
> Info: Retrieving plugin
> Info: Loading facts in /var/lib/puppet/lib/facter/centos_version.rb
> Info: Loading facts in /var/lib/puppet/lib/facter/network-bonds.rb
> Info: Loading facts in /var/lib/puppet/lib/facter/powerstates.rb
> Info: Loading facts in /var/lib/puppet/lib/facter/software_raid.rb
> Info: Loading facts in /var/lib/puppet/lib/facter/centos-version.rb
>
> Info: Caching catalog for puppet30client.swisstxt.ch
> Info: Applying configuration version ''1351000748''
> Finished catalog run in 1.88 seconds
> root@puppet30client:~# 
>
> (Don''t believe the 1.88 seconds - this run took about three
minutes(!))
>
>
> My question is: What is the best practise to let Puppet interact with the 
> CloudStack API?
>
>
> What I did:
>
> 1) Created a csapiclient module and dumped a custom function: 
> /etc/puppet/modules/csapiclient/lib/puppet/parser/functions/add_pf_rule.rb
>
> require File.join(File.dirname(__FILE__),
''cloudstack_client'')
>
> module Puppet::Parser::Functions
>   newfunction(:add_pf_rule) do |args|
>     apiurl = args[0]
>     apikey = args[1]
>     secretkey = args[2]
>     proto = args[3]
>     pubPort = args[4]
>     privPort = args[5]
>     ipId = args[6]
>     vmId = args[7]
>
>     cs = CloudstackClient::Connection.new(apiurl,apikey,secretkey)
>
>     cs.create_port_forwarding_rule( ipId, privPort, proto, pubPort, vmId)
>   end
> end
>
> 2) Wrote a defined resource that calls the Ruby function:
>
>     define addPFRule( $ipId, $vmId, $proto, $pubPort, $privPort ) {
>         # Pass arguments to Ruby function…
>         add_pf_rule( $apiUrl, $apiKey , $secretKey, $proto, $pubPort, 
> $privPort, $ipId, $vmId )
>     }
>
> 3) Call the define from a recipe, feeding it with parameters.
>
> Is this a viable way to go or are there better/simpler/more beautiful 
> solutions to talk to CloudStack?
>

Yes.  You want a custom (Ruby) type and provider instead of a custom 
function to wrap the Cloudstack API.  As a bonus, you then won''t have
any
use for a wrapper definition (supposing actually you have one now).

What you actually wrote is rather strange.  Puppet functions run on the 
master, so your code makes *the master* execute  the CloudStack 
configuration commands.  That can work, in principle, but it''s not the 
execution model that Puppet was designed for.  The master should be tasked 
only with the compiling a catalog; actual configuration tasks should run on 
clients.

Furthermore, because functions run on the master, it is rarely useful to 
write a definition that does nothing but wrap a function call.  The 
resulting defined-type resource instance is just an empty marker in the 
catalog (except in a handful of special cases such as when the function 
called is ''include'' or ''create_resources''). 
Usually, it is better and
clearer to just call a function directly than to declare an instance of a 
wrapper resource.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/FFW0uQ9UhmoJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Puppet users - Oct 2012 - Puppet interacting with CloudStack API (Ruby API client)

[Puppet Users] Puppet interacting with CloudStack API (Ruby API client)

Re: [Puppet Users] Puppet interacting with CloudStack API (Ruby API client)

[Puppet Users] Re: Puppet interacting with CloudStack API (Ruby API client)

Re: [Puppet Users] Re: Puppet interacting with CloudStack API (Ruby API client)

[Puppet Users] Re: Puppet interacting with CloudStack API (Ruby API client)