The first release is now complete. I''m just finishing up some tests and some documentation, and then I will release it. I ended up just writing the client side library using Ruby''s general socket support, since I couldn''t see how to employ EM appropriately for that while still allowing regular program execution and a persistent connection. The server is all EM, though, and it''s very, very swift. The first release will be barebones -- the server and the client library, and will be made available this weekend. Later releases should include modules that one can use to employ the logger as the default logging destination in various web frameworks and anything else that people request or provide code for. Kirk Haines
So if I got this right your asynchronous logging server can be used with any EM server? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070302/21bf1b40/attachment-0001.html
On 3/2/07, Daniel Aquino <mr.danielaquino at gmail.com> wrote:> So if I got this right your asynchronous logging server can be used with any > EM server?The server itself is intended to run standalone, though if I decouple a few things, it could be a more general class library that would provide asynchronous logging capabilities to any EM server, yes. Kirk Haines
On 3/2/07, Kirk Haines <wyhaines at gmail.com> wrote:> > The first release is now complete. I''m just finishing up some tests > and some documentation, and then I will release it. > > I ended up just writing the client side library using Ruby''s general > socket support, since I couldn''t see how to employ EM appropriately > for that while still allowing regular program execution and a > persistent connection. > > The server is all EM, though, and it''s very, very swift. > > The first release will be barebones -- the server and the client > library, and will be made available this weekend. > > Later releases should include modules that one can use to employ the > logger as the default logging destination in various web frameworks > and anything else that people request or provide code for.I''m really looking forward to this. I assume your logging server is scriptable, so I can add (for example) ferret support to it, and a get a full-text-searchable log? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070302/a34d8856/attachment.html
On 3/2/07, Francis Cianfrocca <garbagecat10 at gmail.com> wrote:> I''m really looking forward to this. I assume your logging server is > scriptable, so I can add (for example) ferret support to it, and a get a > full-text-searchable log?It needs some refactoring, which I am planning on doing anyway, since the 2nd release will allow one to define a db table as a logging destination. After that refactoring takes place, my intention is that the logger will be able to use pluggins to support whatever one wants it to. Kirk
On 3/2/07, Kirk Haines <wyhaines at gmail.com> wrote:> On 3/2/07, Francis Cianfrocca <garbagecat10 at gmail.com> wrote: > > > I''m really looking forward to this. I assume your logging server is > > scriptable, so I can add (for example) ferret support to it, and a get a > > full-text-searchable log? > > It needs some refactoring, which I am planning on doing anyway, since > the 2nd release will allow one to define a db table as a logging > destination. After that refactoring takes place, my intention is that > the logger will be able to use pluggins to support whatever one wants > it to.Speaking of databases, is everyone designing EM applications just assuming that all DB calls block (since the DBI and Mysql libraries weren''t written with async in mind) and living with it? My server has to fetch certain data out of a database and I haven''t bothered trying to make the calls nonblocking yet. --Michael
On 3/2/07, Michael S. Fischer <michael at dynamine.net> wrote:> Speaking of databases, is everyone designing EM applications just > assuming that all DB calls block (since the DBI and Mysql libraries > weren''t written with async in mind) and living with it? My server has > to fetch certain data out of a database and I haven''t bothered trying > to make the calls nonblocking yet.Does the MySQL API have nonblocking calls? And yes, I have just accepted that if I go out to a database, throughput is going to plummet. In the case of the logger, the best speed will probably be achieved by setting it up so that it''s aware of the db specific syntaxt that exists for some backends (such as MySQL) for fast bulk inserts. Pain in the behind. Kirk Haines
On 3/2/07, Kirk Haines <wyhaines at gmail.com> wrote:> And yes, I have just accepted that if I go out to a database, > throughput is going to plummet.For reads, I am trying to alleviate this problem somewhat by using Memcached. Even though memcached is insanely fast, the read call on the Ruby side will still block. --Michael
I''m doing this. I have multiple concurrent HTTP connections open with EM, and when one finishes I pass the data to an AR model to insert it into the database. It interrupts the event loop until the database operations finish. At one point I factored apart the database code into a separate process, but just ended up with it accumulating a giant backlog as the EM portion was fetching data faster than the database portion could insert it. So now I''m only fetching data as fast as it can be loaded into the database. - Tony On 3/2/07, Michael S. Fischer <michael at dynamine.net> wrote:> > On 3/2/07, Kirk Haines <wyhaines at gmail.com> wrote: > > On 3/2/07, Francis Cianfrocca <garbagecat10 at gmail.com> wrote: > > > > > I''m really looking forward to this. I assume your logging server is > > > scriptable, so I can add (for example) ferret support to it, and a get > a > > > full-text-searchable log? > > > > It needs some refactoring, which I am planning on doing anyway, since > > the 2nd release will allow one to define a db table as a logging > > destination. After that refactoring takes place, my intention is that > > the logger will be able to use pluggins to support whatever one wants > > it to. > > Speaking of databases, is everyone designing EM applications just > assuming that all DB calls block (since the DBI and Mysql libraries > weren''t written with async in mind) and living with it? My server has > to fetch certain data out of a database and I haven''t bothered trying > to make the calls nonblocking yet. > > --Michael > _______________________________________________ > Eventmachine-talk mailing list > Eventmachine-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/eventmachine-talk >-- Tony Arcieri ClickCaster, Inc. tony at clickcaster.com (970) 232-4208 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070302/cb8da0d7/attachment-0001.html
On 3/2/07, Kirk Haines <wyhaines at gmail.com> wrote:> > On 3/2/07, Michael S. Fischer <michael at dynamine.net> wrote: > > > Speaking of databases, is everyone designing EM applications just > > assuming that all DB calls block (since the DBI and Mysql libraries > > weren''t written with async in mind) and living with it? My server has > > to fetch certain data out of a database and I haven''t bothered trying > > to make the calls nonblocking yet. > > Does the MySQL API have nonblocking calls? > > And yes, I have just accepted that if I go out to a database, > throughput is going to plummet. In the case of the logger, the best > speed will probably be achieved by setting it up so that it''s aware of > the db specific syntaxt that exists for some backends (such as MySQL) > for fast bulk inserts. Pain in the behind.take a look at EventMachine#defer, which was intended for this situation. It manages an internal thread pool. I''ve actually given some thought to writing EM-capable implementations of the postgres and mysql protocols. I always got stuck on the thought that if you''re going to have more than a small handful of DB operations in flight at any given point in time, you''re going to have big problems on the backend anyway. And if your simultaneous max is small, then the overhead of threads (which EventMachine#defer uses) is probably small. I could be convinced otherwise if anyone disagrees. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070302/6725a496/attachment.html