Hello, i wanted to know if we can use exported ressrouces to do things like this: - for all nodes tagged as xxx create on the node yyy a file which content is a template where we can have each hostname , or create one file per nodes matching exemple /etc/hosts.allow sshd: host1 sshd: host2 ... or /etc/backuppc/host1 /etc/backuppc/host2 ... from the docs it seems i can collect only ALL the object and then use them as a bloc but not use the results to build something else like having a list of nodes and create configuration files for each nodes on ra monitoring server or backup server ? Here the goal is , for all nodes tagged as having backup enabled , create on the backup server a configuration file and a directory so backup works :) -- Cordialement, Ghislain _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
On 2/8/07, ADNET Ghislain <gadnet@aqueos.com> wrote:> Hello, > > i wanted to know if we can use exported ressrouces to do things like this: > > - for all nodes tagged as xxx create on the node yyy a file which content is > a template where we can have each hostname > , or create one file per nodes matching > > exemple > > /etc/hosts.allow > sshd: host1 > sshd: host2 > ... > > or > /etc/backuppc/host1 > /etc/backuppc/host2 > ... > > > from the docs it seems i can collect only ALL the object and then use them > as a bloc but not use the results to build > something else like having a list of nodes and create configuration files > for each nodes on ra monitoring server or > backup server ? Here the goal is , for all nodes tagged as having backup > enabled , create on the backup server a > configuration file and a directory so backup works :) > > > -- > Cordialement, > Ghislain >
On Feb 8, 2007, at 5:18 PM, ADNET Ghislain wrote:> Hello, > > i wanted to know if we can use exported ressrouces to do things > like this: > > - for all nodes tagged as xxx create on the node yyy a file which > content is a template where we can have each hostname , or create > one file per nodes matching > > exemple > > /etc/hosts.allow > sshd: host1 > sshd: host2 > ... > > or > /etc/backuppc/host1 > /etc/backuppc/host2 > ... > > > from the docs it seems i can collect only ALL the object and then > use them as a bloc but not use the results to build something else > like having a list of nodes and create configuration files for each > nodes on ra monitoring server or backup server ? Here the goal is , > for all nodes tagged as having backup enabled , create on the > backup server a configuration file and a directory so backup works :)Currently, you can only query all instances of exported values, or by name, so if you do this you''ll get all exported files, not just the backup files. We''re working on that; in the meantime, I think this does what you want. class backupclient { @@file { "/etc/backup.d/$hostname": content => template("blah.erb") } } class backupserver { File <<| |>> } Plus you''ll have to add whatever code you need to turn those little files into one big one, but that should give you an idea. Is that what you want? -- It is dangerous to be right when the government is wrong. -- Voltaire --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
Luke Kanies a écrit :> On Feb 8, 2007, at 5:18 PM, ADNET Ghislain wrote: > >> Hello, >> >> i wanted to know if we can use exported ressrouces to do things >> like this: >> >> - for all nodes tagged as xxx create on the node yyy a file which > Currently, you can only query all instances of exported values, or by > name, so if you do this you''ll get all exported files, not just the > backup files. We''re working on that; in the meantime, I think this > does what you want. > > class backupclient { > @@file { "/etc/backup.d/$hostname": content => template("blah.erb") } > } > > class backupserver { > File <<| |>> > } > > Plus you''ll have to add whatever code you need to turn those little > files into one big one, but that should give you an idea. > > Is that what you want?yes and it works quite well ! i have an exec that is notified when config change and build the config file with a "cat /balh/* > config" that notify the service to restart it ! you know what ? this is AWESOME :) when we will be able to make queries on the files we want or do not want to include it will rocks even more ! right now i can only configure my backup server as if i configure my monitoring one it will use the virtual ressources defined by the backup for my monitoringserver classe and vice-versa as backuped servers are allmost allways also monitored so this should not work ? can i define a virtual ressources like : monitoringfiles(...){ file{ ..: } } and call @@monitoringfile to wrap the @@file into a @@monitoringfile that will then be different ? -- Cordialement, Ghislain _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
On Feb 13, 2007, at 12:04 PM, ADNET Ghislain wrote:> > right now i can only configure my backup server as if i configure > my monitoring one it will use the virtual ressources defined by > the backup for my monitoringserver classe and vice-versa as > backuped servers are allmost allways also monitored so this should > not work ? can i define a virtual ressources like : > > monitoringfiles(...){ > file{ ..: > } > } > > and call @@monitoringfile to wrap the @@file into a > @@monitoringfile that will then be different ?You can export defined resources like monitoringfiles, but you cannot collect them; you''ll still have to collect the contained resources, not the defined ones. -- The conception of two people living together for twenty-five years without having a cross word suggests a lack of spirit only to be admired in sheep. --Alan Patrick Herbert --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
ok thanks Luke, another thing. Since i put this virtual stuff in production i got puppet doing : 6682 puppet 25 0 154m 134m 2608 R 97.7 13.5 91:44.39 ruby it uses 97% of cpu in extended period of time and consume memory like never before. Beside this, I also got a warning: DEPRECATION WARNING: The :dependent => true option is deprecated and will be removed from Rails 2.0. Please use :dependent => :destroy instead. See http://www.rubyonrails.org/deprecation for details. See http://www.rubyonrails.org/deprecation for details. (called from has_many at /usr/lib/ruby/gems/1.8/gems/activerecord-1.15.2/lib/active_record/associations.rb:550) is this normal ? -- Cordialement, Ghislain _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
On Feb 14, 2007, at 1:34 AM, ADNET Ghislain wrote:> ok thanks Luke, > > another thing. Since i put this virtual stuff in production i got > puppet doing : > > 6682 puppet 25 0 154m 134m 2608 R 97.7 13.5 91:44.39 ruby > > it uses 97% of cpu in extended period of time and consume memory > like never before.This is the server, then, right? I just checked mine and it''s at 48mb, but I don''t have many clients. I would say that your number is definitely high, and it certainly shouldn''t be using 97% cpu all the time. How long are your client configurations taking to store in the database? Mine take quite a while sometimes (up to 15 secs), but everyone else who''s tested sees no more than about 2 secs; if you''re getting extended times here, then that could clog up the rest of the system.> Beside this, I also got a warning: > > > DEPRECATION WARNING: The :dependent => true option is deprecated > and will be removed from Rails 2.0. Please use :dependent > => :destroy instead. See http://www.rubyonrails.org/deprecation > for details. See http://www.rubyonrails.org/deprecation for > details. (called from has_many at /usr/lib/ruby/gems/1.8/gems/ > activerecord-1.15.2/lib/active_record/associations.rb:550)You''re using the latest version of Rails, 1.2, and we have not yet made Puppet compatible with that. -- Levy''s Law: The truth is always more interesting than your preconception of what it might be. --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
> This is the server, then, right? I just checked mine and it''s at > 48mb, but I don''t have many clients. I would say that your number is > definitely high, and it certainly shouldn''t be using 97% cpu all the > time. >yes this is the server :) i have 60 hosts to manage i think (is there a way to ask puppet how many nodes it manage) ?> How long are your client configurations taking to store in the > database? Mine take quite a while sometimes (up to 15 secs), but > everyone else who''s tested sees no more than about 2 secs; if you''re > getting extended times here, then that could clog up the rest of the > system.more than 100 seconds. and it take >90% cpu all the time it compute this. Feb 14 08:18:15 puppet puppetmasterd[6682]: Compiled configuration for backup02.aqueos.net in 100.35 seconds Feb 14 08:27:20 puppet puppetmasterd[6682]: Compiled configuration for backup02.aqueos.net in 105.84 seconds Feb 14 08:30:56 puppet puppetmasterd[6682]: Compiled configuration for backup02.aqueos.net in 132.65 seconds Feb 14 08:32:54 puppet puppetmasterd[6682]: Compiled configuration for backup02.aqueos.net in 104.39 seconds Feb 14 09:44:32 puppet puppetmasterd[6682]: Compiled configuration for backup02.aqueos.net in 107.12 seconds Feb 14 09:56:26 puppet puppetmasterd[6682]: Compiled configuration for backup02.aqueos.net in 107.66 seconds Feb 14 09:59:20 puppet puppetmasterd[6682]: Compiled configuration for backup02.aqueos.net in 119.57 seconds Feb 14 10:05:00 puppet puppetmasterd[6682]: Compiled configuration for backup02.aqueos.net in 107.28 seconds Feb 14 10:41:44 puppet puppetmasterd[6682]: Compiled configuration for backup02.aqueos.net in 108.68 seconds Feb 14 13:54:34 puppet puppetmasterd[24683]: Compiled configuration for backup02.aqueos.net in 101.49 seconds Feb 14 14:40:08 puppet puppetmasterd[24683]: Compiled configuration for backup02.aqueos.net in 101.04 seconds Feb 14 14:59:56 puppet puppetmasterd[24683]: Compiled configuration for backup02.aqueos.net in 101.00 seconds Feb 14 15:12:56 puppet puppetmasterd[24683]: Compiled configuration for backup02.aqueos.net in 110.20 seconds Feb 14 15:17:37 puppet puppetmasterd[24683]: Compiled configuration for backup02.aqueos.net in 105.58 seconds Feb 14 15:19:33 puppet puppetmasterd[24683]: Compiled configuration for backup02.aqueos.net in 102.08 seconds Feb 14 15:23:36 puppet puppetmasterd[24683]: Compiled configuration for backup02.aqueos.net in 102.17 seconds Feb 14 15:25:47 puppet puppetmasterd[24683]: Compiled configuration for backup02.aqueos.net in 103.22 seconds Feb 14 15:31:22 puppet puppetmasterd[24683]: Compiled configuration for backup02.aqueos.net in 102.45 seconds Feb 14 15:33:22 puppet puppetmasterd[24683]: Compiled configuration for backup02.aqueos.net in 102.61 seconds Feb 14 15:35:51 puppet puppetmasterd[28998]: Compiled configuration for backup02.aqueos.net in 101.48 seconds Feb 14 15:40:01 puppet puppetmasterd[29162]: Compiled configuration for backup02.aqueos.net in 106.38 seconds Feb 14 16:56:40 puppet puppetmasterd[29162]: Compiled configuration for backup02.aqueos.net in 101.02 seconds most of the other host takes 20 to 60 seconds to compile. (Intel(R) Pentium(R) 4 CPU 3.00GHz, 512Mo ram) i use sqllite3 for the nodes parhaps another provider could help ?> You''re using the latest version of Rails, 1.2, and we have not yet > made Puppet compatible with that.ok so perhaps some issues comme from this, i downgrade to rails 1.1.6 but the compile time is still 100+ seconds and i still got the warning on the client side :) -- Cordialement, Ghislain _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
On Feb 14, 2007, at 11:20 AM, ADNET Ghislain wrote:> >> This is the server, then, right? I just checked mine and it''s at >> 48mb, but I don''t have many clients. I would say that your number >> is definitely high, and it certainly shouldn''t be using 97% cpu >> all the time. > > yes this is the server :)That''s good, anyway; I couldn''t see any way this would affect the client.> i have 60 hosts to manage i think (is there a way to ask puppet how > many nodes it manage) ?At this point, your only real option is ot write a query of your own against the sql database. PuppetShow will provide this info some day, but I don''t know when that day will be.>> How long are your client configurations taking to store in the >> database? Mine take quite a while sometimes (up to 15 secs), but >> everyone else who''s tested sees no more than about 2 secs; if >> you''re getting extended times here, then that could clog up the >> rest of the system. > > more than 100 seconds. and it take >90% cpu all the time it > compute this. > > > Feb 14 08:18:15 puppet puppetmasterd[6682]: Compiled configuration > for backup02.aqueos.net in 100.35 seconds[...]> Feb 14 16:56:40 puppet puppetmasterd[29162]: Compiled configuration > for backup02.aqueos.net in 101.02 seconds > > > most of the other host takes 20 to 60 seconds to compile. (Intel(R) > Pentium(R) 4 CPU 3.00GHz, 512Mo ram) > i use sqllite3 for the nodes parhaps another provider could help ?Can you start the server in verbose mode and tell me what percentage of that time is storing to the database vs. compiling the configuration? I''m currently working supporting mod_ruby + apache, which should help compile times, but I don''t know what I can do about the write times. I''m trying to get some help from local Rails guys, and they''ve made me realize how stupid it is that we don''t have any indexes. I''ll test that as soon as I get home. My guess is that the difference is mostly writing to the database; at least, that''s what I hope. I''ve done what I can to optimize these writes, but I get wildly inconsistent results. This is part of what makes the Rails support still experimental.>> You''re using the latest version of Rails, 1.2, and we have not >> yet made Puppet compatible with that. > > ok so perhaps some issues comme from this, i downgrade to rails > 1.1.6 but the compile time is still 100+ seconds and i still got > the warning on the client side :)What warning do you get on the client? The rails warning? Hmm, that shouldn''t even be loaded on the client, but even if it is, you can ignore the warning until we get around to fixing it, since the Rails stuff isn''t used at all on the clients. -- I have learned to use the word ''impossible'' with the greatest caution. -- Wernher von Braun --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On Feb 14, 2007, at 11:06 AM, Luke Kanies wrote:> On Feb 14, 2007, at 11:20 AM, ADNET Ghislain wrote: > >> >>> This is the server, then, right? I just checked mine and it''s at >>> 48mb, but I don''t have many clients. I would say that your number >>> is definitely high, and it certainly shouldn''t be using 97% cpu >>> all the time. >> >> yes this is the server :) > > That''s good, anyway; I couldn''t see any way this would affect the > client. > >> i have 60 hosts to manage i think (is there a way to ask puppet how >> many nodes it manage) ? > > At this point, your only real option is ot write a query of your own > against the sql database. PuppetShow will provide this info some > day, but I don''t know when that day will be.If I can just finish these last few projects for work I can get back to this........ That was supposed to be last friday. :)> >>> How long are your client configurations taking to store in the >>> database? Mine take quite a while sometimes (up to 15 secs), but >>> everyone else who''s tested sees no more than about 2 secs; if >>> you''re getting extended times here, then that could clog up the >>> rest of the system. >> >> more than 100 seconds. and it take >90% cpu all the time it >> compute this. >> >> >> Feb 14 08:18:15 puppet puppetmasterd[6682]: Compiled configuration >> for backup02.aqueos.net in 100.35 seconds > [...] >> Feb 14 16:56:40 puppet puppetmasterd[29162]: Compiled configuration >> for backup02.aqueos.net in 101.02 seconds >> >> >> most of the other host takes 20 to 60 seconds to compile. (Intel(R) >> Pentium(R) 4 CPU 3.00GHz, 512Mo ram) >> i use sqllite3 for the nodes parhaps another provider could help ? > > Can you start the server in verbose mode and tell me what percentage > of that time is storing to the database vs. compiling the > configuration? I''m currently working supporting mod_ruby + apache, > which should help compile times, but I don''t know what I can do about > the write times. I''m trying to get some help from local Rails guys, > and they''ve made me realize how stupid it is that we don''t have any > indexes. I''ll test that as soon as I get home. > > My guess is that the difference is mostly writing to the database; at > least, that''s what I hope. > > I''ve done what I can to optimize these writes, but I get wildly > inconsistent results. This is part of what makes the Rails support > still experimental.Ack! I can''t believe I forgot to to do this. I''ve got a wildly different schema right now that I''m playing with, but I''ll stick indexing in there if nobody beats me to it.> >>> You''re using the latest version of Rails, 1.2, and we have not >>> yet made Puppet compatible with that. >> >> ok so perhaps some issues comme from this, i downgrade to rails >> 1.1.6 but the compile time is still 100+ seconds and i still got >> the warning on the client side :) > > What warning do you get on the client? The rails warning? Hmm, that > shouldn''t even be loaded on the client, but even if it is, you can > ignore the warning until we get around to fixing it, since the Rails > stuff isn''t used at all on the clients.There''s also the AR connection thing which is pretty much a show- stopper. I have to restart puppetmasterd constantly to work around it. -Blake
On Feb 14, 2007, at 3:15 PM, Blake Barnett wrote:>> I''ve done what I can to optimize these writes, but I get wildly >> inconsistent results. This is part of what makes the Rails support >> still experimental. > > Ack! I can''t believe I forgot to to do this. I''ve got a wildly > different schema right now that I''m playing with, but I''ll stick > indexing in there if nobody beats me to it.I''ve got a sample schema with indexes, but it didn''t seem to make much difference; you can see the diff at http://luke.madstop.com/ index_diff.txt . After talking with a friend, I think that the fact_name/fact_value and param_name/param_value tables should be merged. They provide exactly the same functionality, and I think it''s a bit redundant.> There''s also the AR connection thing which is pretty much a show- > stopper. I have to restart puppetmasterd constantly to work around > it.Yep. :( -- Dawkins''s Law of Adversarial Debate: When two incompatible beliefs are advocated with equal intensity, the truth does not lie half way between them. --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On Feb 14, 2007, at 1:22 PM, Luke Kanies wrote:> On Feb 14, 2007, at 3:15 PM, Blake Barnett wrote: >>> I''ve done what I can to optimize these writes, but I get wildly >>> inconsistent results. This is part of what makes the Rails support >>> still experimental. >> >> Ack! I can''t believe I forgot to to do this. I''ve got a wildly >> different schema right now that I''m playing with, but I''ll stick >> indexing in there if nobody beats me to it. > > I''ve got a sample schema with indexes, but it didn''t seem to make > much difference; you can see the diff at http://luke.madstop.com/ > index_diff.txt . > > After talking with a friend, I think that the fact_name/fact_value > and param_name/param_value tables should be merged. They provide > exactly the same functionality, and I think it''s a bit redundant.I have to admit, I forget why we needed this in the first place... -Blake
On Feb 14, 2007, at 3:49 PM, Blake Barnett wrote:>> >> After talking with a friend, I think that the fact_name/fact_value >> and param_name/param_value tables should be merged. They provide >> exactly the same functionality, and I think it''s a bit redundant. > > I have to admit, I forget why we needed this in the first place...I don''t think we "needed" it per se; I think it was just done that way. -- The whole secret of life is to be interested in one thing profoundly and in a thousand things well. -- Horace Walpole --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On Feb 14, 2007, at 2:52 PM, Luke Kanies wrote:> On Feb 14, 2007, at 3:49 PM, Blake Barnett wrote: >>> >>> After talking with a friend, I think that the fact_name/fact_value >>> and param_name/param_value tables should be merged. They provide >>> exactly the same functionality, and I think it''s a bit redundant. >> >> I have to admit, I forget why we needed this in the first place... > > I don''t think we "needed" it per se; I think it was just done that > way.Probably a misunderstanding on my part. I certainly wouldn''t have done it that way if I didn''t think it was required. It makes all the code much uglier. Anyway, with rails 1.2, we can do away with almost all the crap that requires saving objects before associating them, etc. I really hope I get a chance to work on it soon. -Blake
> Can you start the server in verbose mode and tell me what percentage > of that time is storing to the database vs. compiling the > configuration? I''m currently working supporting mod_ruby + apache, > which should help compile times, but I don''t know what I can do about > the write times. I''m trying to get some help from local Rails guys, > and they''ve made me realize how stupid it is that we don''t have any > indexes. I''ll test that as soon as I get home. >now i have a constant timeout: sudo /usr/bin/puppetd --config /usr/local/.aqadmin/etc/puppetd.conf --onetime --test --configtimeout=600 --filetimeout=600 notice: Ignoring --listen on onetime run info: Retrieving facts info: Loading fact local Pour en savoir davantage, faites: « uname --help ». err: Configuration retrieval timed out on the client side :( running the server with --debug do not give any timing value and just stop output for ages without outputing anything so i cannot tell a lot about this. It pause after : debug: File[/etc/BackupPC/pc/hotsofme.com.pl]: Adding default for backup info: Stored configuration for backup02.aqueos.net in 117.74 seconds notice: Compiled configuration for backup02.aqueos.net in 131.47 seconds and there the client timeout. regards, Ghislain. _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
On Feb 15, 2007, at 1:54 AM, ADNET Ghislain wrote:> > now i have a constant timeout: > > sudo /usr/bin/puppetd --config /usr/local/.aqadmin/etc/ > puppetd.conf --onetime --test --configtimeout=600 --filetimeout=600 > notice: Ignoring --listen on onetime run > info: Retrieving facts > info: Loading fact local > Pour en savoir davantage, faites: « uname --help ». > err: Configuration retrieval timed out > > > on the client side :( > > running the server with --debug do not give any timing value and > just stop output for ages without outputing anything so i cannot > tell a lot about this. It pause after : > > debug: File[/etc/BackupPC/pc/hotsofme.com.pl]: Adding default for > backup > > > info: Stored configuration for backup02.aqueos.net in 117.74 seconds > notice: Compiled configuration for backup02.aqueos.net in 131.47 > seconds > > > and there the client timeout.The fact that it''s taking 117 seconds to store the configuration is clearly intolerable. It shouldn''t take more than a couple of seconds. This still shouldn''t cause a timeout, becuse it''s less than the timeout you have specified, but if it''s taking that long, I recommend avoiding storeconfigs until we can get those speeds up. You might get some benefit from switching to mysql or postgres, but I haven''t seen any in my testing. -- Never esteem anything as of advantage to you that will make you break your word or lose your self-respect. -- Marcus Aurelius Antoninus --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
> > The fact that it''s taking 117 seconds to store the configuration is > clearly intolerable. It shouldn''t take more than a couple of > seconds. This still shouldn''t cause a timeout, becuse it''s less than > the timeout you have specified, but if it''s taking that long, I > recommend avoiding storeconfigs until we can get those speeds up. > > You might get some benefit from switching to mysql or postgres, but I > haven''t seen any in my testing.ok so i will switch to manual mode for my backup system until you can move on with the code :) thanks for taking the time to look at it and sorry to have such weird recipes ! :) i can''t wait to be able to configure my servers using virtual ressources ! -- Cordialement, Ghislain _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users