Looking back over the list archives I see there was quite a bit of discussion about slapping a web server "face" onto EM so Rails could do RESTful web service-style calls to it. Did anything become of this? On a related note, I saw a message from Kirk Haines from the beginning of the year where he said he was creating a syslogd server using EM so multiple processes could have EM handle writing their log files. I searched his site (enigo.com) but came up empty. Here''s the problem I am trying to solve. If anyone has alternative ideas as to how to solve my problem, I''m open to suggestions. Problem 1 I have application logs scattered over several servers. I need to consolidate those logs into one centralized log in real-time. I do not have the source code to these applications so I can''t force them to use a syslogd API for writing their logs. The servers are a mixture of Windows and Linux. BTW, I was going to use this project as my reason for writing a netstrings protocol handler for EM. It isn''t strictly necessary but would be fun for me. Problem 2 I need to parse this centralized log in real-time to generate a series of application performance metrics. These metrics may be stored in a database or written to another file in a specified format. Problem 3 I need to put up a web page that refreshes every X seconds and displays graphs of the metrics created in Problem 2. My original idea was to deploy EM to every server with some simple code to "tail" the log files and write each line back to a centralized server listening for this data. The centralized server would also be based on EM. It would write a consolidated log *and* have a built-in parser for generating the application statistics. Then I was going to write a simple Rails app which would request the metrics for each server and build a nice little graph using Gruff and paint it up on the screen. Using a little Web 2.0 magic the web browser would generate the XmlHttpRequest every X seconds to kick off a browser update. I like this design because it fits in with my reading of the UNIX philosophy... lots of small programs that do very specific tasks, plus I can glue them together using EM as a conceptual proxy for PIPES. Without a REST interface on EM I will need to do something like use BackgrounDRb to spawn an EM process to parse the centralized log and push the results into Ruby objects that Rails can get at. Anyone have any further suggestions?
On 10/17/07, Chuck Remes <cremes.devlist at mac.com> wrote:> Looking back over the list archives I see there was quite a bit of > discussion about slapping a web server "face" onto EM so Rails could > do RESTful web service-style calls to it. Did anything become of this? > > On a related note, I saw a message from Kirk Haines from the > beginning of the year where he said he was creating a syslogd server > using EM so multiple processes could have EM handle writing their log > files. I searched his site (enigo.com) but came up empty.I''m in the middle of something right now, so just a quick reply. analogger.swiftcore.org That''s the asynchronous logger. Kirk Haines
> > > My original idea was to deploy EM to every server with some simple > code to "tail" the log files and write each line back to a > centralized server listening for this data. The centralized server > would also be based on EM. It would write a consolidated log *and* > have a built-in parser for generating the application statistics. > Then I was going to write a simple Rails app which would request the > metrics for each server and build a nice little graph using Gruff and > paint it up on the screen. Using a little Web 2.0 magic the web > browser would generate the XmlHttpRequest every X seconds to kick off > a browser update. I like this design because it fits in with my > reading of the UNIX philosophy... lots of small programs that do very > specific tasks, plus I can glue them together using EM as a > conceptual proxy for PIPES.Sounds good--or use javascript to refresh every so often. Without a REST interface on EM I will need to do something like use> BackgrounDRb to spawn an EM process to parse the centralized log and > push the results into Ruby objects that Rails can get at.You could have EM create DB objects, then for every rails request have it recreate the pictures. Or have EM create DB objects, and create the files you want to load as well, every X seconds (just start an EM timer that runs ''create the graphs'' ever x seconds). Then in reality you might even not have to use rails at all. Just have the EM ''driver'' create the stuff constantly and put it statically in a directory. Create index.html with javascript to refresh them and you have static pages. Dunno :) -Roger -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20071017/5621b970/attachment.html
On 10/17/07, Chuck Remes <cremes.devlist at mac.com> wrote:> > Looking back over the list archives I see there was quite a bit of > discussion about slapping a web server "face" onto EM so Rails could > do RESTful web service-style calls to it. Did anything become of this? > > On a related note, I saw a message from Kirk Haines from the > beginning of the year where he said he was creating a syslogd server > using EM so multiple processes could have EM handle writing their log > files. I searched his site (enigo.com) but came up empty. > > Here''s the problem I am trying to solve. If anyone has alternative > ideas as to how to solve my problem, I''m open to suggestions.EM has a complete HTTP server "face" on it, but it''s in its own package. Look on the EM project page for eventmachine_httpserver. As far as REST is concerned, I actually have a RESTful framework that I''m using in production on several projects now. However, it''s tightly integrated into the EM HTTP server, and it was suggested to me (convincingly) that it would make much more sense to release a REST framework that could work with any Web server (webrick, apache, lighty, nginx). So I''ve been meaning to factor out the EM dependency, but haven''t gotten around to it yet. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20071017/77b15fc8/attachment.html
On Oct 17, 2007, at 6:30 PM, Roger Pack wrote:> Without a REST interface on EM I will need to do something like use > BackgrounDRb to spawn an EM process to parse the centralized log and > push the results into Ruby objects that Rails can get at. > > You could have EM create DB objects, then for every rails request > have it recreate the pictures. Or have EM create DB objects, and > create the files you want to load as well, every X seconds (just > start an EM timer that runs ''create the graphs'' ever x seconds). > Then in reality you might even not have to use rails at all. > Just have the EM ''driver'' create the stuff constantly and put it > statically in a directory. Create index.html with javascript to > refresh them and you have static pages.Roger, good idea! I like this. Much simpler than my original plan though a teeny bit less flexible. Right now the page is read-only (no interactivity) so I think I can live without the Rails/Merb/IOWA piece for a while. If EM is generating the HTML, I should probably look at using Markaby, Bluecloth or Redcloth and see how to pull them into my EM handler. Thanks for the idea! cr
On 10/18/07, Chuck Remes <cremes.devlist at mac.com> wrote:> > > On Oct 17, 2007, at 6:30 PM, Roger Pack wrote: > > > Without a REST interface on EM I will need to do something like use > > BackgrounDRb to spawn an EM process to parse the centralized log and > > push the results into Ruby objects that Rails can get at. > > > > You could have EM create DB objects, then for every rails request > > have it recreate the pictures. Or have EM create DB objects, and > > create the files you want to load as well, every X seconds (just > > start an EM timer that runs ''create the graphs'' ever x seconds). > > Then in reality you might even not have to use rails at all. > > Just have the EM ''driver'' create the stuff constantly and put it > > statically in a directory. Create index.html with javascript to > > refresh them and you have static pages.As long as you have EM refresh the static pages with the same refresh rate that your html pages get updated, it...should...work. I think it would, unless the browser caches it. You could test it.> > If EM is generating the HTML, I should probably look at using > Markaby, Bluecloth or Redcloth and see how to pull them into my EM > handler. Thanks for the idea! >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20071018/ac7d45c3/attachment.html