I have a Bash script, currently run a couple times an hour from cron, that pulls data from an old Windows DB by rsync, converts it to SQL, and injects it into a MySQL DB for display in a LAMP-based app. (Make and Perl are also involved to minimize the number of tables touched and to clean up the SQL generated by Pxlib.) I'd like to add the ability to refresh the data immediately from the web app, but I don't want it to trample on the periodic script and corrupt the data. I figure the ideal way to do this is to run the script in a loop in its own process, waiting on a semaphore that times out at the refresh period, and poke the semaphore from the web app to have it run before the next periodic cycle. Are there existing frameworks to wrap this kind of thing in? Something that handles starting the loop at server startup, shutting it down at server halt, and handles the IPC between the web server and the service script.
On Fri, 8 Jul 2011, Kenneth Porter wrote:> To: CentOS mailing list <centos at centos.org> > From: Kenneth Porter <shiva at sewingwitch.com> > Subject: [CentOS] Triggering script from cron or web client > > I have a Bash script, currently run a couple times an hour from cron, that > pulls data from an old Windows DB by rsync, converts it to SQL, and injects > it into a MySQL DB for display in a LAMP-based app. (Make and Perl are also > involved to minimize the number of tables touched and to clean up the SQL > generated by Pxlib.) > > I'd like to add the ability to refresh the data immediately from the web > app, but I don't want it to trample on the periodic script and corrupt the > data. > > I figure the ideal way to do this is to run the script in a loop in its own > process, waiting on a semaphore that times out at the refresh period, and > poke the semaphore from the web app to have it run before the next periodic > cycle.Is this something you could do with AJAX? Keith ----------------------------------------------------------------- Websites: http://www.karsites.net http://www.php-debuggers.net http://www.raised-from-the-dead.org.uk All email addresses are challenge-response protected with TMDA [http://tmda.net] -----------------------------------------------------------------
On 7/8/2011 4:58 PM, Kenneth Porter wrote:> I have a Bash script, currently run a couple times an hour from cron, that > pulls data from an old Windows DB by rsync, converts it to SQL, and injects > it into a MySQL DB for display in a LAMP-based app. (Make and Perl are also > involved to minimize the number of tables touched and to clean up the SQL > generated by Pxlib.) > > I'd like to add the ability to refresh the data immediately from the web > app, but I don't want it to trample on the periodic script and corrupt the > data. > > I figure the ideal way to do this is to run the script in a loop in its own > process, waiting on a semaphore that times out at the refresh period, and > poke the semaphore from the web app to have it run before the next periodic > cycle. > > Are there existing frameworks to wrap this kind of thing in? Something that > handles starting the loop at server startup, shutting it down at server > halt, and handles the IPC between the web server and the service script.You already have a DB connection in common - can't the update script itself lock something in the DB while it works? Or just make the web script wait for the cron job to complete if you are anywhere near the time for it to run. -- Les Mikesell lesmikesell at gmail.com
On Fri, Jul 8, 2011 at 5:58 PM, Kenneth Porter <shiva at sewingwitch.com> wrote:> I have a Bash script, currently run a couple times an hour from cron, that > pulls data from an old Windows DB by rsync, converts it to SQL, and injects > it into a MySQL DB for display in a LAMP-based app. (Make and Perl are also > involved to minimize the number of tables touched and to clean up the SQL > generated by Pxlib.) > > I'd like to add the ability to refresh the data immediately from the web > app, but I don't want it to trample on the periodic script and corrupt the > data. > > I figure the ideal way to do this is to run the script in a loop in its own > process, waiting on a semaphore that times out at the refresh period, and > poke the semaphore from the web app to have it run before the next periodic > cycle. > > Are there existing frameworks to wrap this kind of thing in? Something that > handles starting the loop at server startup, shutting it down at server > halt, and handles the IPC between the web server and the service script.Web page creates a temp file saying "update", then cron job runs once a minute looking for that file, kicks off update script, then deletes temp file. If you don't want to wait a full minute you could use 'incron' to monitor the temp directory and kick the update right away. Seems better than having something in a loop constantly checking. -? Brian Mathis ?-
Emmanuel Noobadmin
2011-Jul-10 05:22 UTC
[CentOS] Triggering script from cron or web client
On 7/9/11, Kenneth Porter <shiva at sewingwitch.com> wrote:> I have a Bash script, currently run a couple times an hour from cron, that > pulls data from an old Windows DB by rsync, converts it to SQL, and injects > it into a MySQL DB for display in a LAMP-based app. (Make and Perl are also > involved to minimize the number of tables touched and to clean up the SQL > generated by Pxlib.) > > I'd like to add the ability to refresh the data immediately from the web > app, but I don't want it to trample on the periodic script and corrupt the > data.Offhand, I think checking for a lock file before starting is probably the easiest way to do it. That's what I do for scripts that are cronjobs but can also be initiated manually. However, this might not work if your web app is inserting/updating the data in MySQL directly rather than simply starting a new import process as I'm assuming.
On Mon, Jul 11, 2011 at 10:16 PM, Kenneth Porter <shiva at sewingwitch.com> wrote:> --On Friday, July 08, 2011 8:23 PM -0400 Brian Mathis > <brian.mathis+centos at betteradmin.com> wrote: > >> If you don't want to wait a full minute you could use >> 'incron' to monitor the temp directory and kick the update right away. >> ?Seems better than having something in a loop constantly checking. > > incron sounds promising. I could use the regular cron to push a signal (ie. > touch a file) at the regular interval, whereupon incron launches the > existing script, and the webscript can touch the signal file when I need > the script run right away. Will incron not run the script again if it's > already running? Will it queue the file event until the handler exits?No, incron will not queue events or wait for the other one to finish. Your script should take care of this. One thing to watch out for is if the script is running and you use a lockfile to exit from a 2nd one running, the original script won't get the message about the 2nd file. Maybe that matters for this application, maybe not. -? Brian Mathis ?-