Stephan
2011-Nov-15 13:05 UTC
[Puppet Users] Virtual Define-based resources can only be realized once
Hi all, here is what I have: A module that is responsible to roll out and configure TSM (a backup tool). Basically I need a text file that mentions all directories that have to be backed up. I want to make it a virtual resource and let other modules configure which parts of it have to be backed up, e.g. configure in the apache module that the apache logs of each defined virtual server have to be backed up: The node.pp is configured like this: apache::config {"virtualserver1": blah } apache:config {"virtualserver2": blah } @tsm::config { "$hostname": tsm_backup => ["/custdir1","/custdir2"] } The @ means this is a virtual Define-based resource, so it is not implemented until it gets realized. /custdir1 and /custdir2 denote custom directories that have to be backed up, but aren''t managed by other puppet modules. One module is supposed to add some directories to tsm_backup, I use this configuration inside the apache module''s config.pp: Tsm::Config<| title == $hostname |> { tsm_backup +> ["$apache_logs,$docroot], } I can then use this array in the template for the conf file (dsm.sys, virtualmountpoint and Domain, for those who know TSM). Up until here it all works fine. The config file will contain /custdir1, /custdir2, / apache/virtualserver1/logs, /apache/virtualserver2/logs, apache/ virtualserver1/docroot, apache/virtualserver2/docroot But when I want to configure another module to also add its directories to tsm_backup, so basically have another module, say postfix config.pp which contains this: Tsm::Config<| title == $hostname |> { tsm_backup +> ["$postfix_logs], } It will either contain the apache backup dirs OR the postfix backup dirs, depending on which <| |> it finds first, but never both. This is unexpected, since if I use the same construct for non define- based virtual resources, like a user (see https://groups.google.com/group/puppet-users/browse_thread/thread/c13c1936dcd939fd/cff48a8c03734336 ) Does anybody have an idea? -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
jcbollinger
2011-Nov-16 15:37 UTC
[Puppet Users] Re: Virtual Define-based resources can only be realized once
On Nov 15, 7:05 am, Stephan <stephan.eckwei...@admin.ox.ac.uk> wrote:> Hi all, > > here is what I have: > > A module that is responsible to roll out and configure TSM (a backup > tool). Basically I need a text file that mentions all directories that > have to be backed up. I want to make it a virtual resource and let > other modules configure which parts of it have to be backed up, e.g. > configure in the apache module that the apache logs of each defined > virtual server have to be backed up: > > The node.pp is configured like this: > > apache::config {"virtualserver1": > blah} > > apache:config {"virtualserver2": > blah > > } > > @tsm::config { "$hostname": > tsm_backup => ["/custdir1","/custdir2"] > > } > > The @ means this is a virtual Define-based resource, so it is not > implemented until it gets realized. > > /custdir1 and /custdir2 denote custom directories that have to be > backed up, but aren''t managed by other puppet modules. > > One module is supposed to add some directories to tsm_backup, I use > this configuration inside the apache module''s config.pp: > > Tsm::Config<| title == $hostname |> { > tsm_backup +> ["$apache_logs,$docroot], > > } > > I can then use this array in the template for the conf file (dsm.sys, > virtualmountpoint and Domain, for those who know TSM). Up until here > it all works fine. The config file will contain /custdir1, /custdir2, / > apache/virtualserver1/logs, /apache/virtualserver2/logs, apache/ > virtualserver1/docroot, apache/virtualserver2/docroot > > But when I want to configure another module to also add its > directories to tsm_backup, so basically have another module, say > postfix config.pp which contains this: > > Tsm::Config<| title == $hostname |> { > tsm_backup +> ["$postfix_logs], > > } > > It will either contain the apache backup dirs OR the postfix backup > dirs, depending on which <| |> it finds first, but never both. > > This is unexpected, since if I use the same construct for non define- > based virtual resources, like a user (seehttps://groups.google.com/group/puppet-users/browse_thread/thread/c13... > ) > > Does anybody have an idea?Your realizations conflict with each other. One says "realize Tsm::config[$hostname] with the *declared* groups plus $apache_logs and $docroot", whereas the other says "realize Tsm::config[$hostname] with the *declared* groups plus $postfix_logs". There is no qualitative difference from overriding via the = operator, which I''m sure you will agree would not work either. I would be surprised if the approach worked for built-in resources. It was not obvious from the thread you referenced that the same situation applies; if it does, then I recommend that you check whether it is having the desired effect. Even if so, I would not personally be comfortable with it. Because it''s both undocumented and semantically questionable, I would assess a high risk of it breaking from one version of puppet to another. As for how to achieve your goal, I''d recommend one of 1) target the original resource declaration, applying whatever logic or external data is needed to declare all the needed backup directories in the first place, or 2) rework your design so that each module can declare its own, independent Tsm::Config instead of relying on resource overrides. That could be built around a fragment-based approach to building the needed config file, or it could leverage the configuration language''s mechanism for interpolating other files, if it has one. John -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Stephan
2011-Nov-17 16:57 UTC
[Puppet Users] Re: Virtual Define-based resources can only be realized once
Hi John, thanks for your reply, much appreciated.> I recommend that you check whether > it is having the desired effect.Yeah, I actually double-checked before writing my post, and it did work just fine ... but I agree with you that the term "realize" is semantically questionable. What I actually need is a collection of attributes across many modules to configure another module. That doesn''t go against puppet''s declarative nature, since such a collection would arrive at a certain set of attributes, no matter where in the policy one starts to collect them. I''m actually not so sure this "realize" I did for $tsm_backup doesn''t go against the declarative nature. If a realize is really supposed to act as you describe and how I observe it in my tsm module, then we have something like a race condition. Whichever spaceship is found first finalizes the content of $tsm_backup. That''s not very declarative. The more I think about it, the more I think I should file a bug for it. What do you reckon?> I would assess a high risk of it breaking > from one version of puppet to another.Good point.> As for how to achieve your goal, I''d recommend one of > 1) target the original resource declaration, applying whatever logic > or external data is needed to declare all the needed backup > directories in the first placeIf I understand you correctly you suggest that the tsm module rolls out a script which picks up all backup information from files rolled out by the modules that need backups, rebuilds it into a usable conf file for tsm and deals with making sure to restart the service and all that. Probably possible, but kind of a nightmare to develop and maintain. That''s actually all stuff I expect a managing software to do for me. Bit of a puppet-managed puppet replacement. But I agree it might be possible, I''ll think about it, thanks.> 2) rework your design so that each module can declare its own, > independent Tsm::Config instead of relying on resource overrides. > That could be built around a fragment-based approach to building the > needed config file, or it could leverage the configuration language''s > mechanism for interpolating other files, if it has one.Unfortunately tsm doesn''t offer something like a include of all files in a .d directory (I guess that''s what you mean with interpolating other files). But what exactly do you mean with a fragment-based approach to build the config file? Sounds like exactly what I want to do, just don''t know a way to realize it with puppet. Thanks again Stephan -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
jcbollinger
2011-Nov-17 23:04 UTC
[Puppet Users] Re: Virtual Define-based resources can only be realized once
On Nov 17, 10:57 am, Stephan <stephan.eckwei...@admin.ox.ac.uk> wrote:> Hi John, > > thanks for your reply, much appreciated. > > > I recommend that you check whether > > it is having the desired effect. > > Yeah, I actually double-checked before writing my post, and it did > work just fine ... but I agree with you that the term "realize" is > semantically questionable. What I actually need is a collection of > attributes across many modules to configure another module. That > doesn''t go against puppet''s declarative nature, since such a > collection would arrive at a certain set of attributes, no matter > where in the policy one starts to collect them. I''m actually not so > sure this "realize" I did for $tsm_backup doesn''t go against the > declarative nature. If a realize is really supposed to act as you > describe and how I observe it in my tsm module, then we have something > like a race condition. Whichever spaceship is found first finalizes > the content of $tsm_backup. That''s not very declarative. The more I > think about it, the more I think I should file a bug for it. What do > you reckon?A race condition is a reasonable analogy. I would file the ticket, provided there isn''t already one on this topic. At mininum, Puppet should issue a warning when it encounters potentially conflicting realizations of the same resource. Ideally, it would error, but only when there is a bona fide conflict. I suspect, however, that it is a lot easier to detect the possibility of a conflict (which is present whenever at least one of two or more realizations of the same resource employs an override) than it is to determine whether there is an conflict.> > I would assess a high risk of it breaking > > from one version of puppet to another. > > Good point. > > > As for how to achieve your goal, I''d recommend one of > > 1) target the original resource declaration, applying whatever logic > > or external data is needed to declare all the needed backup > > directories in the first place > > If I understand you correctly you suggest that the tsm module rolls > out a script which picks up all backup information from files rolled > out by the modules that need backups, rebuilds it into a usable conf > file for tsm and deals with making sure to restart the service and all > that. Probably possible, but kind of a nightmare to develop and > maintain. That''s actually all stuff I expect a managing software to do > for me. Bit of a puppet-managed puppet replacement. But I agree it > might be possible, I''ll think about it, thanks.I suppose what you describe is a possibility, but certainly not one I would pursue. I was intentionally vague on the precise mechanism for putting together the needed list of backup directories because I don''t have an adequate view of the problem and context. Maybe there''s no good solution available along this path.> > 2) rework your design so that each module can declare its own, > > independent Tsm::Config instead of relying on resource overrides. > > That could be built around a fragment-based approach to building the > > needed config file, or it could leverage the configuration language''s > > mechanism for interpolating other files, if it has one. > > Unfortunately tsm doesn''t offer something like a include of all files > in a .d directory (I guess that''s what you mean with interpolating > other files). But what exactly do you mean with a fragment-based > approach to build the config file? Sounds like exactly what I want to > do, just don''t know a way to realize it with puppet.Look up the Concat module. That seems to be the tool most often used for this sort of job, and several of the regulars around here can help you out with it. John -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Nan Liu
2011-Nov-18 00:50 UTC
Re: [Puppet Users] Virtual Define-based resources can only be realized once
On Tue, Nov 15, 2011 at 8:05 AM, Stephan <stephan.eckweiler@admin.ox.ac.uk> wrote:> Hi all, > > here is what I have: > > A module that is responsible to roll out and configure TSM (a backup > tool). Basically I need a text file that mentions all directories that > have to be backed up. I want to make it a virtual resource and let > other modules configure which parts of it have to be backed up, e.g. > configure in the apache module that the apache logs of each defined > virtual server have to be backed up: > > The node.pp is configured like this: > > apache::config {"virtualserver1": > blah > } > apache:config {"virtualserver2": > blah > } > > @tsm::config { "$hostname": > tsm_backup => ["/custdir1","/custdir2"] > } > > The @ means this is a virtual Define-based resource, so it is not > implemented until it gets realized. > > /custdir1 and /custdir2 denote custom directories that have to be > backed up, but aren''t managed by other puppet modules. > > One module is supposed to add some directories to tsm_backup, I use > this configuration inside the apache module''s config.pp: > > Tsm::Config<| title == $hostname |> { > tsm_backup +> ["$apache_logs,$docroot], > } > > I can then use this array in the template for the conf file (dsm.sys, > virtualmountpoint and Domain, for those who know TSM). Up until here > it all works fine. The config file will contain /custdir1, /custdir2, / > apache/virtualserver1/logs, /apache/virtualserver2/logs, apache/ > virtualserver1/docroot, apache/virtualserver2/docroot > > But when I want to configure another module to also add its > directories to tsm_backup, so basically have another module, say > postfix config.pp which contains this: > > Tsm::Config<| title == $hostname |> { > tsm_backup +> ["$postfix_logs], > } > > It will either contain the apache backup dirs OR the postfix backup > dirs, depending on which <| |> it finds first, but never both. > > This is unexpected, since if I use the same construct for non define- > based virtual resources, like a user (see > https://groups.google.com/group/puppet-users/browse_thread/thread/c13c1936dcd939fd/cff48a8c03734336 > ) > > Does anybody have an idea?I have a very rough proof of concept that might fit (i.e. not official and likely buggy, though feedback welcomed): https://github.com/nanliu/puppet-export_resources The github page have explanation some of the export resource limitation it''s trying to solve. The idea there is for exporting systems to have a default resource attributes: /data/${env}/tsm_backup/hostname.yaml Each app export it''s data to a subdirectory: /data/${env}/tsm_backup/hostname/tsm_backup/apache.yaml /data/${env}/tsm_backup/hostname/tsm_backup/postfix.yaml ... and finally on the target system import_resource will create a resource hash suitable for create_resources function (https://github.com/puppetlabs/puppetlabs-create_resources). Thanks, Nan -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Stephan
2011-Nov-18 17:13 UTC
[Puppet Users] Re: Virtual Define-based resources can only be realized once
Hi John,> Look up the Concat module. That seems to be the tool most often used > for this sort of job, and several of the regulars around here can help > you out with it.Thanks so much for this hint, this does the job perfectly! All set up now :-) Stephan -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.