I am interested in a custom type which allows for execs which are not idempotent. My plan was to create a custom type which drops a file to record successful execution. I hoped to reuse as much of exec as possible, but I am running into issues. I started with this code: Puppet::Type.newtype(:file_addition, :parent=>Puppet::Type::Exec) do newparam(:command, :overwrite=>true) do # Omit the isnamevar end newproperty(:returns, :overwrite=>true) do |property| # Blah, Blah, Blah warning("Hello") And it allows me to add this. The overwrite has no effect, so I am guessing that this was basically a no-op. Especially since I can not get the Hello to every be issued. So.. my main question is am I going about this the right way? I would prefer to not recreate all the code becuase all I need to do is to drop a file at the end of the returns. -- bk --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Bryan Kearney wrote:> I am interested in a custom type which allows for execs which are not > idempotent. My plan was to create a custom type which drops a file to > record successful execution. I hoped to reuse as much of exec as > possible, but I am running into issues. >Why? What''s wrong with the creates and onlyif attributes of the existing exec type? http://reductivelabs.com/trac/puppet/wiki/TypeReference#exec Regards James Turnbull - -- Author of: * Pulling Strings with Puppet (http://www.amazon.com/gp/product/1590599780/) * Pro Nagios 2.0 (http://www.amazon.com/gp/product/1590596099/) * Hardening Linux (http://www.amazon.com/gp/product/1590594444/) -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFId2yk9hTGvAxC30ARAtE8AJ90+c34+7GHj2HiYke9M3M6DwwYxACeI612 NRzdIz8RMREVEcbPLihhs+c=oQYI -----END PGP SIGNATURE----- --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
James Turnbull wrote:> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Bryan Kearney wrote: >> I am interested in a custom type which allows for execs which are not >> idempotent. My plan was to create a custom type which drops a file to >> record successful execution. I hoped to reuse as much of exec as >> possible, but I am running into issues. >> > > Why? What''s wrong with the creates and onlyif attributes of the > existing exec type? > > http://reductivelabs.com/trac/puppet/wiki/TypeReference#execCall it ease of use, or perhaps a better phrase would be "Excapsulated requires". I am trying to write some manifests which make it easy to install DB driven applications. Many of the applications come with plql scripts that need to be done in a certain order, and only once. My first throught was do to the following: define single_exec ($command, $user="root", $cwd="/") { include single_exec_class $filename = "/var/ace/single_execs/$name" $record_file_command = "/bin/touch $filename" # Run the command if the file has not been dropped exec {$command: cwd => $cwd, user => $user, creates => $filename, notify => Exec[$record_file_command] } # Drop the file to show that we ran it exec {$record_file_command: refreshonly => true } } And then create database defines which use this: define mysql::execute_sql_file($sql_file, $root_user="root", $root_user_password="admin", $database="") { $my_command = template("mysql/execute_sql_file_command.erb") single_exec { "mysql_execute_sql_file_$name": command => $my_command } } where the end result would be the following "high level" manifest: mysql::execute_sql_file{"step1": sql_file=>"file1"} mysql::execute_sql_file{"step2": sql_file=>"file2" requires Execute_sql_file[file1]} However, that requires will fail since it is a require on a define which uses a deinf. I hoped to create a type to avoid the initial single_exec define.. which would allow me do specify the requires at the "high level". I would like to avoind the "high level" author havinf to know about how the internals are done. All he cares about is that step2 needs to occur once and only once after step1. -- bk --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
what about something like this: exec { single_exec1: command => "/var/ace/single_execs/single_exec1 && touch /path/to/single_exec1.done", unless => "test -f /path/to/single_exec1.done", [...otherstuff...] } Bryan Kearney schrieb:> James Turnbull wrote: >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> Bryan Kearney wrote: >>> I am interested in a custom type which allows for execs which are not >>> idempotent. My plan was to create a custom type which drops a file to >>> record successful execution. I hoped to reuse as much of exec as >>> possible, but I am running into issues. >>> >> Why? What''s wrong with the creates and onlyif attributes of the >> existing exec type? >> >> http://reductivelabs.com/trac/puppet/wiki/TypeReference#exec > > Call it ease of use, or perhaps a better phrase would be "Excapsulated > requires". I am trying to write some manifests which make it easy to > install DB driven applications. Many of the applications come with plql > scripts that need to be done in a certain order, and only once. My first > throught was do to the following: > > define single_exec ($command, $user="root", $cwd="/") { > > include single_exec_class > > $filename = "/var/ace/single_execs/$name" > $record_file_command = "/bin/touch $filename" > > # Run the command if the file has not been dropped > exec {$command: > cwd => $cwd, > user => $user, > creates => $filename, > notify => Exec[$record_file_command] > } > > # Drop the file to show that we ran it > exec {$record_file_command: > refreshonly => true > } > } > > > And then create database defines which use this: > > define mysql::execute_sql_file($sql_file, $root_user="root", > $root_user_password="admin", $database="") { > > $my_command = template("mysql/execute_sql_file_command.erb") > > single_exec { "mysql_execute_sql_file_$name": > command => $my_command > } > } > > where the end result would be the following "high level" manifest: > > mysql::execute_sql_file{"step1": sql_file=>"file1"} > mysql::execute_sql_file{"step2": sql_file=>"file2" requires > Execute_sql_file[file1]} > > However, that requires will fail since it is a require on a define which > uses a deinf. I hoped to create a type to avoid the initial single_exec > define.. which would allow me do specify the requires at the "high > level". I would like to avoind the "high level" author havinf to know > about how the internals are done. All he cares about is that step2 needs > to occur once and only once after step1. > > -- bk > > > > --~--~---------~--~----~------------~-------~--~----~ > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com > To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en > -~----------~----~----~----~------~----~------~--~--- > >--
Phillip Scholz wrote:> what about something like this: > > exec { single_exec1: > command => "/var/ace/single_execs/single_exec1 && touch > /path/to/single_exec1.done", > unless => "test -f /path/to/single_exec1.done", > [...otherstuff...] > }I agree that this would work on most systems, I was hoping for a "cleaner" solution since I am seeing this pattern alot in my use cases. -- bk --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Bryan Kearney <bkearney@redhat.com> writes:> James Turnbull wrote: >> Bryan Kearney wrote: >>> I am interested in a custom type which allows for execs which are not >>> idempotent. My plan was to create a custom type which drops a file to >>> record successful execution. I hoped to reuse as much of exec as >>> possible, but I am running into issues. >> >> Why? What''s wrong with the creates and onlyif attributes of the >> existing exec type? >> >> http://reductivelabs.com/trac/puppet/wiki/TypeReference#exec > > Call it ease of use, or perhaps a better phrase would be "Excapsulated > requires". I am trying to write some manifests which make it easy to > install DB driven applications. Many of the applications come with > plql scripts that need to be done in a certain order, and only > once. My first throught was do to the following:If you forgive my comment, perhaps a better approach is something akin to this: --8<---------------cut here---------------start------------->8--- # I have not tested this, only proven it correct. define database_setup ($database, $table, $setupfile) { $setup_cmd = "psql $database -f $setupfile" $test_cmd = "psql $database -c ''\d $table''" # wouldn''t it be nice to have a sensible name for this, rather than # having to put the command string in as the name. exec { $setup_cmd: unless => $test_cmd, require => Database[$database] } } --8<---------------cut here---------------end--------------->8--- This is, obviously, not a very well developed example, but the basic logic is: Pass it a database and a table that must exist, as well as the name of the file to create the tables in the database. The ''$test_cmd'' checks if the table exists in the specified database, returning 0 if it does and 1 if it does not. Adapt for your database if the command line tools are not as helpful as PostgreSQL. If that returns false, that the database does not contain the tables, run the ''$setup_cmd'', which goes right ahead and runs your setup SQL. This turns your problem from "run this once" into "run this if the database is incorrect", which is an idempotent operation. Regards, Daniel --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Daniel Pittman wrote:> Bryan Kearney <bkearney@redhat.com> writes: >> James Turnbull wrote: >>> Bryan Kearney wrote: >>>> I am interested in a custom type which allows for execs which are not >>>> idempotent. My plan was to create a custom type which drops a file to >>>> record successful execution. I hoped to reuse as much of exec as >>>> possible, but I am running into issues. >>> Why? What''s wrong with the creates and onlyif attributes of the >>> existing exec type? >>> >>> http://reductivelabs.com/trac/puppet/wiki/TypeReference#exec >> Call it ease of use, or perhaps a better phrase would be "Excapsulated >> requires". I am trying to write some manifests which make it easy to >> install DB driven applications. Many of the applications come with >> plql scripts that need to be done in a certain order, and only >> once. My first throught was do to the following: > > If you forgive my comment, perhaps a better approach is something akin > to this: > > --8<---------------cut here---------------start------------->8--- > # I have not tested this, only proven it correct. > define database_setup ($database, $table, $setupfile) { > $setup_cmd = "psql $database -f $setupfile" > $test_cmd = "psql $database -c ''\d $table''" > # wouldn''t it be nice to have a sensible name for this, rather than > # having to put the command string in as the name. > exec { $setup_cmd: > unless => $test_cmd, > require => Database[$database] > } > } > --8<---------------cut here---------------end--------------->8--- > > This is, obviously, not a very well developed example, but the basic > logic is: > > Pass it a database and a table that must exist, as well as the name of > the file to create the tables in the database. > > The ''$test_cmd'' checks if the table exists in the specified database, > returning 0 if it does and 1 if it does not. Adapt for your database if > the command line tools are not as helpful as PostgreSQL. > > If that returns false, that the database does not contain the tables, > run the ''$setup_cmd'', which goes right ahead and runs your setup SQL. > > > This turns your problem from "run this once" into "run this if the > database is incorrect", which is an idempotent operation. > > Regards,> DanielI agree this is a much better solution for the "from scratch" use case. I am looking at the use case where install scripts already exist, and the user is creating a puppet recipe to automate the install. I dont know that I would want them to have to recreate their entire script as a series of puppet commands. FWIW, I am currently looking at custom types similar to the one attached. -- bk --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Bryan Kearney <bkearney@redhat.com> writes:> Daniel Pittman wrote: >> Bryan Kearney <bkearney@redhat.com> writes: >>> James Turnbull wrote: >>>> Bryan Kearney wrote:G''day Bryan.>>>>> I am interested in a custom type which allows for execs which are >>>>> not idempotent.[...]> I agree this is a much better solution for the "from scratch" use > case. I am looking at the use case where install scripts already > exist, and the user is creating a puppet recipe to automate the > install.Sure.> I dont know that I would want them to have to recreate their entire > script as a series of puppet commands.I don''t think they should, and wasn''t trying to advise that. I /was/ trying to advise that rather than "non-idempotent exec" you change your strategy: Detect if the install script needs to be run, and run it if it hasn''t already been run. In other words: don''t try to make a version of exec that is not idempotent, make your actions idempotent -- without rewriting the script. Regards, Daniel --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
> I don''t think they should, and wasn''t trying to advise that. I /was/ > trying to advise that rather than "non-idempotent exec" you change your > strategy: > > Detect if the install script needs to be run, and run it if it > hasn''t already been run. > > In other words: don''t try to make a version of exec that is not > idempotent, make your actions idempotent -- without rewriting the > script. >What Daniel said. You want idempotency - designing away from it not only goes against the grain of Puppet but at some point it''ll come back to bite you. Regards James Turnbull -- Author of: * Pulling Strings with Puppet (http://www.amazon.com/gp/product/1590599780/) * Pro Nagios 2.0 (http://www.amazon.com/gp/product/1590596099/) * Hardening Linux (http://www.amazon.com/gp/product/1590594444/)
On Mon, 2008-07-14 at 22:29 +1000, James Turnbull wrote:> What Daniel said. You want idempotency - designing away from it not > only goes against the grain of Puppet but at some point it''ll come back > to bite you.But that''s what Bryan is trying to do: turn something that''s not idempotent (the SQL install script) into something that is, with the added difficulty that he doesn''t control those SQL scripts, since they are generally written by others. How do others automate loading DB schemas ? David --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Tue, Jul 15, 2008 at 12:47 PM, David Lutterkort <dlutter@redhat.com> wrote:> > On Mon, 2008-07-14 at 22:29 +1000, James Turnbull wrote: >> What Daniel said. You want idempotency - designing away from it not >> only goes against the grain of Puppet but at some point it''ll come back >> to bite you. > > But that''s what Bryan is trying to do: turn something that''s not > idempotent (the SQL install script) into something that is, with the > added difficulty that he doesn''t control those SQL scripts, since they > are generally written by others. > > How do others automate loading DB schemas ? >I have a few things like this, and I take the approach that Daniel outlined. If the scripts create something then all you need to do is check for the existence of that file/directory/variable. If it does not exist then make it so. If it does exist, pat it on the head and send it on it''s way. I have also used the refreshonly and the notify types to trigger script execution only on the installation of a package. The third and final approaches is to use custom RPMS/deps/what not, that handle running the correct scripts when they are installed. I am seeing one massive problem with a non-idempotent exec, and that is where are you going to record the fact that it has run once? If it is supposed to run once, and only once ever, per install, then this has to be recorded somewhere that puppet can find it. Evan --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
David Lutterkort <dlutter@redhat.com> writes:> On Mon, 2008-07-14 at 22:29 +1000, James Turnbull wrote: > >> What Daniel said. You want idempotency - designing away from it not >> only goes against the grain of Puppet but at some point it''ll come back >> to bite you. > > But that''s what Bryan is trying to do: turn something that''s not > idempotent (the SQL install script) into something that is,I disagree: he seems to be trying to do something very similar, but quite different. In his case there is a type that executes only once, and keeps track of that fact internally. This runs the load script. In my case there is a test to determine if the database needs to be loaded, and the script is run if it does. These /look/ very similar, but are kind of different. One is based on running a script when it is needed, the other is based on running it only once.> with the added difficulty that he doesn''t control those SQL scripts, > since they are generally written by others. > > How do others automate loading DB schemas ?I use the technique I outlined: determine if the schema is loaded, then run the schema installation script (unmodified) if it isn''t. In cases where I do have control of the schema install script I ensure that it tags a version into the database, or equivalent, and use that to control running the script to upgrade the database as well. Regards, Daniel --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Daniel Pittman wrote:> In cases where I do have control of the schema install script I ensure > that it tags a version into the database, or equivalent, and use that to > control running the script to upgrade the database as well.This is the main different. I am looking at poring the current configuration bits of thincrust.net to use puppet. Basically, why write Yet Another Configuration System when a great one exists in puppet. A main use case I am trying to address is an ISV who has an exisitng package building an appliance. In many cases, that package includes some SQL scripts, and some steps to configure the product. For example: Copy X, Softlink Y, Execute this sql script. The frist 2 are easy in puppet. The last becomes difficult becuase I do not have control over the scripts, and I do not believe it will be well recieved to say "Create a custom appliance install process". So.. the next step is to provide an easy mechanism to Say "Do this once per box". Ideally, they would re-write their scripts.. but I dont think it is pragmatic in all cases. -- bk --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Evan Hisey wrote:> On Tue, Jul 15, 2008 at 12:47 PM, David Lutterkort <dlutter@redhat.com> wrote: >> On Mon, 2008-07-14 at 22:29 +1000, James Turnbull wrote: >>> What Daniel said. You want idempotency - designing away from it not >>> only goes against the grain of Puppet but at some point it''ll come back >>> to bite you. >> But that''s what Bryan is trying to do: turn something that''s not >> idempotent (the SQL install script) into something that is, with the >> added difficulty that he doesn''t control those SQL scripts, since they >> are generally written by others. >> >> How do others automate loading DB schemas ? >> > I have a few things like this, and I take the approach that Daniel > outlined. If the scripts create something then all you need to do is > check for the existence of that file/directory/variable. If it does > not exist then make it so. If it does exist, pat it on the head and > send it on it''s way. I have also used the refreshonly and the notify > types to trigger script execution only on the installation of a > package. The third and final approaches is to use custom > RPMS/deps/what not, that handle running the correct scripts when they > are installed. > > I am seeing one massive problem with a non-idempotent exec, and that > is where are you going to record the fact that it has run once? If it > is supposed to run once, and only once ever, per install, then this > has to be recorded somewhere that puppet can find it.Yes, there needs to be a magic file location. Similar to where manifests are found. -- bk --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Hi> So.. the next step is to provide an easy mechanism to Say "Do this once > per box". > > Ideally, they would re-write their scripts.. but I dont think it is > pragmatic in all cases.the idea of puppet is to ensure your boxes in a certain state. so what needs to be done is that you say: be in state, with script applied, if not in that state, apply this script. something like "Do this once per box" would in my opinion be rather opposite of the idea of puppet. and as far as I have understood the last philosphical discussions such a mechanism will never be part of puppet. as it is not the idea of puppet to do things like that. for sure in the end, it''s the same (have the boxes in the state, that scripts are applied vs. run this script once). however the way to get there and the way you are modeling such a problem are signifcant different. thinking the puppet way is different that you were maybe thinking before in your daily sysadmin life. greets pete --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Peter Meier wrote:> Hi > >> So.. the next step is to provide an easy mechanism to Say "Do this once >> per box". >> >> Ideally, they would re-write their scripts.. but I dont think it is >> pragmatic in all cases. > > > the idea of puppet is to ensure your boxes in a certain state. so what > needs to be done is that you say: be in state, with script applied, if > not in that state, apply this script. > > something like "Do this once per box" would in my opinion be rather > opposite of the idea of puppet. and as far as I have understood the last > philosphical discussions such a mechanism will never be part of puppet. > as it is not the idea of puppet to do things like that. > > for sure in the end, it''s the same (have the boxes in the state, that > scripts are applied vs. run this script once). however the way to get > there and the way you are modeling such a problem are signifcant different. > > thinking the puppet way is different that you were maybe thinking before > in your daily sysadmin life. > > greets peteThen please explain to me why I can use exec {"foo": creates => "dont_run_foo_if_present" } I believe the answer it pragmatic. Ideally, this would not exist.. but there are some scenarios where it is important. In any case, I am not asking for my scenario to be included in the base puppet. I think it is important, I will maintain the code for it. No issues. My initial question was one of extending a core type.. if I have a type which is Puppet::Type.newtype(:file_addition, :parent=>Puppet::Type::Exec) do newparam(:command, :overwrite=>true) do # Omit the isnamevar end newproperty(:returns, :overwrite=>true) do |property| # Blah, Blah, Blah warning("Hello") it seems like I can not replace the properties and parameters from above. Is this the expected behavior? I would have expected that passing the :overwrite=>true would have given me my own custom version of Exec with minimal forking. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Hi> Then please explain to me why I can use > > exec {"foo": > creates => "dont_run_foo_if_present" > } > > I believe the answer it pragmatic. Ideally, this would not exist.. but > there are some scenarios where it is important.yeah, and therefor it''s imho not explicitly breaking the philosophical idea. What i''d like to say is more, that if the current solutions for run_once and then i''m in the state, isn''t enough, then maybe your goal of how to achieve your state might be not the prefered one. so maybe a remodeling of your problem will lead you to a much prettier solution. greets pete --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Bryan Kearney <bkearney@redhat.com> writes:> Daniel Pittman wrote: > >> In cases where I do have control of the schema install script I >> ensure that it tags a version into the database, or equivalent, and >> use that to control running the script to upgrade the database as >> well. > > This is the main different. I am looking at poring the current > configuration bits of thincrust.net to use puppet. Basically, why > write Yet Another Configuration System when a great one exists in > puppet. > > A main use case I am trying to address is an ISV who has an exisitng > package building an appliance. In many cases, that package includes > some SQL scripts, and some steps to configure the product.You omitted the previous paragraph where I explained that, indeed, I do face *exactly* the same situation, and take the different approach to solving it.> For example:To extend your example:> Copy X,In puppet, determine if X exists and has the correct content, then copy it if not.> Softlink Y,Determine if Y exists and is a link to the correct place, then make it one if not.> Execute this sql script....and here, determine if the script has been run by looking for the *outcome* of running it, and run it if not. Not "run it once, then flag that", "find out if it needs to be run, then run it."> The frist 2 are easy in puppet. The last becomes difficult becuase I > do not have control over the scripts, and I do not believe it will be > well recieved to say "Create a custom appliance install process".I doubt it: the process creates a database, or depends on one being created, and populates it with tables. Make it test for the presence of the database or, better, the tables inside the database. Viola, something that checks for the outcome of the script in a better fashion than ...> So.. the next step is to provide an easy mechanism to Say "Do this > once per box".... has this run before, stored distinctly from the effect. However, if you are dedicated to building something that is, in essence, against the grain then you have identified the path: Replace your SQL script with something that is, essentially: #!/bin/sh mkdir -p /var/lib/bad-idea test -f /var/lib/bad-idea/my-flag || exit 0 mysql < /path/to/sql-file && touch /var/lib/bad-idea/my-flag You can wrap that up in a define that sets the name of the flag, probably from the command line, trivially enough. You could even use the exec ''onlyif'' thing to split the flag and the execution. None of which makes it a /good/ idea, but whatever. Regards, Daniel --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On 7/16/2008 9:18 AM, Daniel Pittman wrote:>> So.. the next step is to provide an easy mechanism to Say "Do this >> once per box". > > ... has this run before, stored distinctly from the effect.To be more literal, there are two different questions in play here: "Has this exec ever been run since I last reformatted this system?" "Are things set up they way they should be?" The first question examines evidence from the past, and does not examine the present state of the system. There is no way for the first question to determine if the database got deleted, moved to a different host, etc. The second question just looks for differences between the present state of the system and some idealized one. There''s also a bit of brittleness to the first question, in that there exists a chance that the flag file could be deleted and you''ll try to rerun your exec over the top of existing tables. Granted, the second method stands a chance of performing unnecessary or destructive tasks if your criteria for "are things set up right" is incorrect, but that''s a risk with any method. A non-database example I''ve had to use that''s similar to this is for my private package repository: file { "/root/caeftp_key.asc": source => "puppet:///apt/caeftp_key.asc", ensure => present, require => [ File["/root/.ssh/id_rsa.pub"], File["/root/.ssh/id_rsa"] ]; } exec { "install-cae-ftp-key": command => "/usr/bin/apt-key add /root/caeftp_key.asc", require => File["/root/caeftp_key.asc"], unless => "/usr/bin/apt-key list | /bin/grep -q ''Mike Renfro (CAE FTP Archive) <renfro@tntech.edu>''"; } aptrepo { "cae_private.list": uri => "ssh://someuser@somehost/somepath/", distribution => "etch", component => "main non-free", require => Exec["install-cae-ftp-key"]; } At some point during my system builds, I need to get access to my private repository. What are the steps to get access to it manually? 1. Get the repository''s GPG key. 2. Add it with apt-key add. 3. Since it''s an ssh repository, make sure I''ve got ssh identities set up on the client to get me passwordless access to the server. 4. Add a file in /etc/apt/sources.list.d pointing to the repository, and refresh "apt-get update". That''s not the only way to do it: order 1234 gives identical results to order 1324 and 3124. But this way, there is zero chance that this procedure will leave me with a broken source entry. There''s no way the key will be added until it''s been downloaded. There''s no way the key will be added twice. If I delete the key by accident, it''ll be added back. -- Mike Renfro / R&D Engineer, Center for Manufacturing Research, 931 372-3601 / Tennessee Technological University --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Wed, 2008-07-16 at 08:03 -0400, Bryan Kearney wrote:> > I am seeing one massive problem with a non-idempotent exec, and that > > is where are you going to record the fact that it has run once? If it > > is supposed to run once, and only once ever, per install, then this > > has to be recorded somewhere that puppet can find it. > > Yes, there needs to be a magic file location. Similar to where manifests > are found.Why not just something under /var/lib/puppet ? Like put empty files into a dir /var/lib/puppet/ran-once with the name of the file the same as the name of the exec. David --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Wed, 2008-07-16 at 11:20 +1000, Daniel Pittman wrote:> > with the added difficulty that he doesn''t control those SQL scripts, > > since they are generally written by others. > > > > How do others automate loading DB schemas ? > > I use the technique I outlined: determine if the schema is loaded, then > run the schema installation script (unmodified) if it isn''t.But that''s only a difference in degrees: how do you determine that the installation script did indeed run successfully ? A SQL install script generally creates lots of different objects (tables, views, sequences, data in tables) and each of them can fail, and that failure does not always lead to the SQL interpreter reporting failure. If you check for the existence of a table, and the table is not there, you still can''t repair the schema in every case by rerunning the install script - think of a case where the install script was run once, and then the one table you check for got dropped. Running SQL install scripts is inherently stateful, and therefore hard to turn into something that''s idempotent - that''s very different from making sure a service is running, since you can usually restart a service any number of times without any adverse effects. The main difference between your and Bryan''s approach is that you use a different substitute for ''required schema exists'' - you check for the existence of a table in the DB, he checks for a file somewhere.> In cases where I do have control of the schema install script I ensure > that it tags a version into the database, or equivalent, and use that to > control running the script to upgrade the database as well.Yes, ideally all SQL scripts would be written that way; unfortunately, not too many are. David --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
David Lutterkort wrote:> On Wed, 2008-07-16 at 08:03 -0400, Bryan Kearney wrote: >>> I am seeing one massive problem with a non-idempotent exec, and that >>> is where are you going to record the fact that it has run once? If it >>> is supposed to run once, and only once ever, per install, then this >>> has to be recorded somewhere that puppet can find it. >> Yes, there needs to be a magic file location. Similar to where manifests >> are found. > > Why not just something under /var/lib/puppet ? Like put empty files into > a dir /var/lib/puppet/ran-once with the name of the file the same as the > name of the exec. > > DavidYes. that would be the idea... put it into a known location. -- bk --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
David Lutterkort <dlutter@redhat.com> writes:> On Wed, 2008-07-16 at 11:20 +1000, Daniel Pittman wrote: > >> > with the added difficulty that he doesn''t control those SQL >> > scripts, since they are generally written by others. >> > >> > How do others automate loading DB schemas ? >> >> I use the technique I outlined: determine if the schema is loaded, then >> run the schema installation script (unmodified) if it isn''t. > > But that''s only a difference in degrees: how do you determine that the > installation script did indeed run successfully?Check the exit status from the command line tool.> A SQL install script generally creates lots of different objects > (tables, views, sequences, data in tables) and each of them can fail, > and that failure does not always lead to the SQL interpreter reporting > failure....which would be a problem. Perhaps better tools would be useful; on occasion I have had to write wrappers around MySQL tools to detect errors correctly where a real database would simply report them. Likewise, with a real database the script can simply be wrapped in a transaction and any changes will roll back completely if there is an error; it is a pity MySQL doesn''t support the same facility.> If you check for the existence of a table, and the table is not there, > you still can''t repair the schema in every case by rerunning the > install script - think of a case where the install script was run > once, and then the one table you check for got dropped.Sure, that is true. Usually I would handle errors, in that case, by either dropping the database or by simply not checking for it within puppet if there was nothing I could do to handle it. (For that, my usual test is: does the database exist? If not, create it, grant privileges, then run the load script. drop the database if that fails. Never check anything else.)> Running SQL install scripts is inherently stateful, and therefore hard > to turn into something that''s idempotentWell, no. Running SQL install scripts that are written in a stateful fashion may be, and many products ship that way. That doesn''t make it a law of nature. ;)> that''s very different from making sure a service is running, since you > can usually restart a service any number of times without any adverse > effects.Depends on your service. ;)> The main difference between your and Bryan''s approach is that you use > a different substitute for ''required schema exists'' - you check for > the existence of a table in the DB, he checks for a file somewhere.True: the reason I said the difference is subtle is that, yes, they are very similar. I suggest checking something that is a direct result of running the installation script. Byran is checking something that is a proxy for the result of the script. In my experience that second, using a proxy for the state rather than the state directly, is much more prone to failure, and is much more likely to lead to handling /other/ design decisions badly. Regards, Daniel --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Fri, Jul 11, 2008 at 6:38 AM, Bryan Kearney wrote:> So.. my main question is am I going about this the right way? I would > prefer to not recreate all the code becuase all I need to do is to drop > a file at the end of the returns.Probably not. As others have pointed out extensively in a discussion thread verging on vi vs. emacs territory, the best way to do this is to check something that occurs as a result of running the SQL. I realize you don''t have control over the SQL, but you do have *access*, so you will have to make it part of your change control process to check the SQL to ensure that the side-effect you are looking for will still be present. Another approach is to *get* some level of control over the SQL; push back against your software vendor for better SQL or a better installation process. I have been bitten several times by the ''put a file somewhere that says this was run'' approach. I cannot stress enough how much of a bad idea it is. - --Paul -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) Comment: http://getfiregpg.org iEYEARECAAYFAkiA1ggACgkQDUiRfwYfJfkxMACgpXiPRZnEoD9nQa2UmdrjDvBr nu0AoKWXJZCjB31JqQ6PXkIn/ha8/Peb =kAK/ -----END PGP SIGNATURE----- --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---