drmikecrowe
2009-Jun-14 21:05 UTC
[Puppet Users] POLL: Migrating web changes across load-balanced servers
Hi folks, I''m polling the group to see what some basic techniques are that people use to migrate web sites to load-balanced servers. In the past, IRC folks have mentioned rsysnc as the way. I''m specifically curious if people are: 1) Letting puppet MD5 a website via a file{} tag? Could that be done? 2) Using version control somehow? I''ve been toying with creating a mercurial plugin somehow. If you push some changes to a production machine, you can configure mercurial to auto-update. What''s some ideas? Any pointers? TIA Mike --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Tim Allingham
2009-Jun-14 23:06 UTC
[Puppet Users] Re: POLL: Migrating web changes across load-balanced servers
Hi Mike, I''m not sure if your after puppet-specific methods, or just general systems for this? When we set up our load balancing and DR, we preferred to keep it simple, the methods used are outlined below. 1) Site code - This is managed but subversion, we maintain separate repositories for testing and production environments - when a change is ready to go live it is merged into the stable codebase. - To then update the production servers, a number of methods are available 1) For minor updates, there is a nightly cronjob that updates all the live servers, and DR setup 2) For critical fixes. I usually use cssh to trigger the update immediately, alternatively we have an authenticating email daemon that can be used to trigger the update cron at any time. - For user contributed files (ie image/video uploads, pdf attachments etc) We use a centralised SAN store for the upload locations. All uploads are also written to a temporary location and then rsync''d to the DR location every 15 minutes. - For SQL - we use MySQL replication, in a Master-Master setup between live and DR locations, with each master then having slaves to balance the load Cheers Tim -----Original Message----- From: drmikecrowe <drmikecrowe@gmail.com> Reply-to: puppet-users@googlegroups.com To: Puppet Users <puppet-users@googlegroups.com> Subject: [Puppet Users] POLL: Migrating web changes across load-balanced servers Date: Sun, 14 Jun 2009 14:05:24 -0700 (PDT) Hi folks, I''m polling the group to see what some basic techniques are that people use to migrate web sites to load-balanced servers. In the past, IRC folks have mentioned rsysnc as the way. I''m specifically curious if people are: 1) Letting puppet MD5 a website via a file{} tag? Could that be done? 2) Using version control somehow? I''ve been toying with creating a mercurial plugin somehow. If you push some changes to a production machine, you can configure mercurial to auto-update. What''s some ideas? Any pointers? TIA Mike --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
drmikecrowe
2009-Jun-15 15:41 UTC
[Puppet Users] Re: POLL: Migrating web changes across load-balanced servers
Hi Tim, That''s very close to where I am now, except we use mercurial instead of svn. :) However, I''m more searching for non-same hosts (i.e. no SAN). I''m basically asking the question of puppet managed ( file { source=>"puppet://.... } ) vs. another scheme. My hunch is I do NOT want puppet managing 1000''s of files and propagating them. Maybe have puppet monitor a version file, and when that changes, execute an rsync to bring the directories into sync. Thanks for the response! Mike On Jun 14, 7:06 pm, Tim Allingham <deser...@gmail.com> wrote:> Hi Mike, > > I''m not sure if your after puppet-specific methods, or just general > systems for this? > > When we set up our load balancing and DR, we preferred to keep it > simple, the methods used are outlined below. > > 1) Site code - This is managed but subversion, we maintain separate > repositories for testing and production environments - when a change is > ready to go live it is merged into the stable codebase. > - To then update the production servers, a number of methods are > available > 1) For minor updates, there is a nightly cronjob that updates > all the live servers, and DR setup > 2) For critical fixes. I usually use cssh to trigger the update > immediately, alternatively we have an authenticating email daemon that > can be used to trigger the update cron at any time. > > - For user contributed files (ie image/video uploads, pdf > attachments etc) We use a centralised SAN store for the upload > locations. All uploads are also written to a temporary location and then > rsync''d to the DR location every 15 minutes. > > - For SQL - we use MySQL replication, in a Master-Master setup > between live and DR locations, with each master then having slaves to > balance the load > > Cheers > > Tim > > > > -----Original Message----- > From: drmikecrowe <drmikecr...@gmail.com> > > Reply-to: puppet-users@googlegroups.com > To: Puppet Users <puppet-users@googlegroups.com> > Subject: [Puppet Users] POLL: Migrating web changes across load-balanced > servers > Date: Sun, 14 Jun 2009 14:05:24 -0700 (PDT) > > Hi folks, > > I''m polling the group to see what some basic techniques are that > people use to migrate web sites to load-balanced servers. > > In the past, IRC folks have mentioned rsysnc as the way. > > I''m specifically curious if people are: > > 1) Letting puppet MD5 a website via a file{} tag? Could that be done? > 2) Using version control somehow? I''ve been toying with creating a > mercurial plugin somehow. If you push some changes to a production > machine, you can configure mercurial to auto-update. > > What''s some ideas? Any pointers? > > TIA > Mike--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Sam Bashton
2009-Jun-15 16:17 UTC
[Puppet Users] Re: POLL: Migrating web changes across load-balanced servers
On Mon, Jun 15, 2009 at 08:41:31AM -0700, drmikecrowe wrote:> > Hi Tim, > > That''s very close to where I am now, except we use mercurial instead > of svn. :) > > However, I''m more searching for non-same hosts (i.e. no SAN). I''m > basically asking the question of puppet managed ( file > { source=>"puppet://.... } ) vs. another scheme. > > My hunch is I do NOT want puppet managing 1000''s of files and > propagating them. Maybe have puppet monitor a version file, and when > that changes, execute an rsync to bring the directories into sync.Mike, I sync a shell script which has a version number in it which I increment. When the script gets deployed it gets executed, (refreshonly => true in the exec) which runs rsync to deploy the site. It also does other things, like using IPtables to block inbound connections whilst the rsync runs, so the load balancers don''t send any traffic to the node during the update. Regards, -- Sam Bashton - Bashton Ltd, Manchester, England Linux Consultancy / High Availability Systems www.bashton.com / 0161-424-9600 / DDI 0161-424-9611
Marc Fournier
2009-Jun-17 12:11 UTC
[Puppet Users] Re: POLL: Migrating web changes across load-balanced servers
On Mon, 15 Jun 2009 08:41:31 -0700 (PDT) drmikecrowe <drmikecrowe@gmail.com> wrote:> > Hi Tim, > > That''s very close to where I am now, except we use mercurial instead > of svn. :) > > However, I''m more searching for non-same hosts (i.e. no SAN). I''m > basically asking the question of puppet managed ( file > { source=>"puppet://.... } ) vs. another scheme. > > My hunch is I do NOT want puppet managing 1000''s of files and > propagating them. Maybe have puppet monitor a version file, and when > that changes, execute an rsync to bring the directories into sync.I have similar needs but have not found any satisfying solution yet. Maybe this feature might prove useful, once it is implemented: http://projects.reductivelabs.com/issues/184 I imagine this would allow us to do something like this: file { "/var/www/whatever/": ensure => "r123", source => "svn://myproject.com/svn/repository/" } Marc --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---