Hi everyone Apologies if I''m trying to apply the evented model to the wrong subject here, but I was wondering about trying out parsing log files using an event loop... Skimming through the docs and a few google searches, the closest I can get would be using a keyboard handler and the LineText2 protocol to read from stdin. This doesn''t "feel right", and I was wondering if anyone has some advice for me on the topic. Thanks in advance! Best -- Kenneth Kalmer kenneth.kalmer at gmail.com http://opensourcery.co.za -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://rubyforge.org/pipermail/eventmachine-talk/attachments/20090330/c4f9f7a8/attachment.html>
On 30 Mar 2009, at 01:34, Kenneth Kalmer wrote:> Hi everyone > > Apologies if I''m trying to apply the evented model to the wrong > subject here, but I was wondering about trying out parsing log files > using an event loop...It''s possible, however, the present implementation is geared toward populating connections with the optimal amount of data, and has a synchronous API. There is some infrastructure now included in the distribution that provides a ruby interface to a memory mapped file. A quick example of raw memory map usage: require "rubygems" require "eventmachine" require ''fastfilereaderext'' ChunkSize = 16384 m = EM::FastFileReader::Mapper.new(''/var/log/system.log'') size = File.size?(''/var/log/system.log'') pos = 0 cs = [ChunkSize, size].min until pos >= size puts m.get_chunk(pos, cs) pos += cs end You''ll notice that there''s no EM.run here. The file read is not happening in the event loop. If you wanted to spread out operations over ticks, you could replace the until with a recursive next_tick: operation = lambda { puts m.get_chunk(pos, cs) pos += cs if pos >= size EM.stop else next_tick &operation end } EM.run do operation.call end This is probably not optimal for just IO read performance though.> Skimming through the docs and a few google searches, the closest I > can get would be using a keyboard handler and the LineText2 protocol > to read from stdin. This doesn''t "feel right", and I was wondering > if anyone has some advice for me on the topic.If you''re sending to connections in your file IO, then you''ll want to spread your sends out over time. There is more infrastructure for doing this, see EM::FileStreamer. This may also give you some ideas for your implementation.> > Thanks in advance! > > Best > > -- > Kenneth Kalmer > kenneth.kalmer at gmail.com > http://opensourcery.co.za > _______________________________________________ > Eventmachine-talk mailing list > Eventmachine-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/eventmachine-talk-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://rubyforge.org/pipermail/eventmachine-talk/attachments/20090330/ad9c8dc9/attachment.html>
2009/3/30 James Tucker <jftucker at gmail.com>> > On 30 Mar 2009, at 01:34, Kenneth Kalmer wrote: > > Hi everyone > > Apologies if I''m trying to apply the evented model to the wrong subject > here, but I was wondering about trying out parsing log files using an event > loop... > > > It''s possible, however, the present implementation is geared toward > populating connections with the optimal amount of data, and has a > synchronous API. There is some infrastructure now included in the > distribution that provides a ruby interface to a memory mapped file. >Thanks for the example and the advice, I was thinking about it again today and I realized that I shouldn''t be confusing code aesthetics (clean, readable callbacks) with the actual problem EM aims to solve (especially not at 3AM). The fastfilereaderext is making me curious, will do some benchmarks between that and standard roll-your-own-buffered-IO. Thanks again for the time and the great explanations, really appreciated. Best -- Kenneth Kalmer kenneth.kalmer at gmail.com http://opensourcery.co.za -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://rubyforge.org/pipermail/eventmachine-talk/attachments/20090330/54b1cdf2/attachment.html>
On 30 Mar 2009, at 14:22, Kenneth Kalmer wrote:> 2009/3/30 James Tucker <jftucker at gmail.com> > > On 30 Mar 2009, at 01:34, Kenneth Kalmer wrote: > >> Hi everyone >> >> Apologies if I''m trying to apply the evented model to the wrong >> subject here, but I was wondering about trying out parsing log >> files using an event loop... > > It''s possible, however, the present implementation is geared toward > populating connections with the optimal amount of data, and has a > synchronous API. There is some infrastructure now included in the > distribution that provides a ruby interface to a memory mapped file. > > Thanks for the example and the advice, I was thinking about it again > today and I realized that I shouldn''t be confusing code aesthetics > (clean, readable callbacks) with the actual problem EM aims to solve > (especially not at 3AM). > > The fastfilereaderext is making me curious, will do some benchmarks > between that and standard roll-your-own-buffered-IO.From what I hear, some gains can be made from memory mapping. That being said, you''ll get more details and ideas in this series: http://www.tbray.org/ongoing/When/200x/2007/09/20/Wide-Finder If you get the chance to write some benchmarks, I''d be interested to see them :)> > Thanks again for the time and the great explanations, really > appreciated. > > Best > > > -- > Kenneth Kalmer > kenneth.kalmer at gmail.com > http://opensourcery.co.za > _______________________________________________ > Eventmachine-talk mailing list > Eventmachine-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/eventmachine-talk-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://rubyforge.org/pipermail/eventmachine-talk/attachments/20090330/c85c944e/attachment-0001.html>