Francis, I read in the list archives back that a future EventMachine release will support epoll on Linux (i.e., it''s in the trunk). Better still, is there a possibility that EM will rely on libevent so that it will be architecture independent (i.e. epoll on Linux, kqueue on FreeBSD/Mac OS X, /dev/poll on Solaris)? This is how memcached is implemented, and it would be helpful to be able to assume maximum performance on any mainstream host OS (aside, perhaps, from Windows). Best regards, --Michael
IIRC, there''s a technical reason (having to do with a green threads conflict) that keeps libevent from playing well with Ruby; Zed Shaw, the Mongrel guy, tried with Ruby/Event and apparently failed. I''d love to know more of the details; I have many tens of thousands of lines of C code that relies on libevent. For my part, though, I''ve taken Zed Shaw''s word for it. On 2/26/07, Michael S. Fischer <michael at dynamine.net> wrote:> Francis, > > I read in the list archives back that a future EventMachine release > will support epoll on Linux (i.e., it''s in the trunk). > > Better still, is there a possibility that EM will rely on libevent so > that it will be architecture independent (i.e. epoll on Linux, kqueue > on FreeBSD/Mac OS X, /dev/poll on Solaris)? This is how memcached is > implemented, and it would be helpful to be able to assume maximum > performance on any mainstream host OS (aside, perhaps, from Windows). > > Best regards, > > --Michael > _______________________________________________ > Eventmachine-talk mailing list > Eventmachine-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/eventmachine-talk >
Getting libevent and Ruby to play nicely is monumentally difficult. Zed Shaw (who wrote Mongrel) tried for quite awhile and eventually gave up. You can check out the old Ruby/Event page here: http://www.zedshaw.com/projects/ruby_event/index.html Zed had this to say: "Hi folks. After struggling with problems getting libevent to play fair with Ruby''s threading, I decided that it was too much effort for the gain and am dropping Ruby/Event. Feel free to take the code here and use it. You can contact me if you feel like maintaining it." Any solution for adding stateful multiplexers to Ruby is going to have to get a lot closer to the metal than libevent allows. The actual monitoring call can only run for the scheduler quantum, and libevent provides no way of controlling that. Francis said he saw no noticeable performance improvement with an epoll-based implementation over Kernel#select (I''d REALLY like to see that code though). I''ve been talking with Ruby core about this problem and will probably try to write my own epoll-based patch to Ruby 1.9 which I''ll benchmark to see if it actually results in improved performance. Between the need to call epoll_wait in a loop every 10ms (alternating with select getting called every 10ms) and deficiencies in epoll''s implementation (there''s already epoll2 on the horizon, ugh) using epoll may add only a marginal performance improvement over Kernel#select. I''ll just have to benchmark it and see. - Tony On 2/26/07, Michael S. Fischer <michael at dynamine.net> wrote:> > Francis, > > I read in the list archives back that a future EventMachine release > will support epoll on Linux (i.e., it''s in the trunk). > > Better still, is there a possibility that EM will rely on libevent so > that it will be architecture independent (i.e. epoll on Linux, kqueue > on FreeBSD/Mac OS X, /dev/poll on Solaris)? This is how memcached is > implemented, and it would be helpful to be able to assume maximum > performance on any mainstream host OS (aside, perhaps, from Windows). > > Best regards, > > --Michael > _______________________________________________ > Eventmachine-talk mailing list > Eventmachine-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/eventmachine-talk >-- Tony Arcieri ClickCaster, Inc. tony at clickcaster.com (970) 232-4208 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070226/6dbec9a4/attachment.html
The problem is libevent runs its own event loop in a blocking call, and Ruby needs to run its own event loop as well. The only way to get something like it to work is to run it within the Ruby event loop, having it block only ~10ms per call. I''d like to get epoll and kqueue support into Ruby core for this very reason. My hope would be further refinements to thread.c in Ruby 1.9 might allow the call to block indefinitely (with the entire inner event loop running on epoll or kqueue), provided there are no extensions loaded which still depend on the green threads model. I''m planning on implementing an epoll-based version of this interface, integrating directly into Ruby''s io.c and thread.c (in Ruby 1.9), and benchmarking it to see if there''s a noticible performance improvement over Kernel#select: http://pastie.caboo.se/41536 - Tony On 2/26/07, Thomas Ptacek <tqbf at matasano.com> wrote:> > IIRC, there''s a technical reason (having to do with a green threads > conflict) that keeps libevent from playing well with Ruby; Zed Shaw, > the Mongrel guy, tried with Ruby/Event and apparently failed. > > I''d love to know more of the details; I have many tens of thousands of > lines of C code that relies on libevent. For my part, though, I''ve > taken Zed Shaw''s word for it. > > On 2/26/07, Michael S. Fischer <michael at dynamine.net> wrote: > > Francis, > > > > I read in the list archives back that a future EventMachine release > > will support epoll on Linux (i.e., it''s in the trunk). > > > > Better still, is there a possibility that EM will rely on libevent so > > that it will be architecture independent (i.e. epoll on Linux, kqueue > > on FreeBSD/Mac OS X, /dev/poll on Solaris)? This is how memcached is > > implemented, and it would be helpful to be able to assume maximum > > performance on any mainstream host OS (aside, perhaps, from Windows). > > > > Best regards, > > > > --Michael > > _______________________________________________ > > Eventmachine-talk mailing list > > Eventmachine-talk at rubyforge.org > > http://rubyforge.org/mailman/listinfo/eventmachine-talk > > > _______________________________________________ > Eventmachine-talk mailing list > Eventmachine-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/eventmachine-talk >-- Tony Arcieri ClickCaster, Inc. tony at clickcaster.com (970) 232-4208 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070226/22a8e360/attachment.html
On 2/26/07, Michael S. Fischer <michael at dynamine.net> wrote:> > Francis, > > I read in the list archives back that a future EventMachine release > will support epoll on Linux (i.e., it''s in the trunk). > > Better still, is there a possibility that EM will rely on libevent so > that it will be architecture independent (i.e. epoll on Linux, kqueue > on FreeBSD/Mac OS X, /dev/poll on Solaris)? This is how memcached is > implemented, and it would be helpful to be able to assume maximum > performance on any mainstream host OS (aside, perhaps, from Windows).Well, candidly, the select-based implementation we have now really does perform quite well on all supported platforms. I don''t see significant difficulties with integrating any of the other techniques, other than IOCP, which is not monumentally difficult but I''m not sure it''s worth doing. My feeling about libevent is basically this: the charter of EM is to get right down to the metal. The code base that it grew out of has been worked on and tuned for quite a few years to have high performance basically everywhere it runs. So I don''t see much value in interposing another layer between EM and the metal. Especially since your issue is with cross-platform interoperability. I''d rather do the work to get those extra bits of performance out of the existing code, which knows enough about Ruby to play nice enough in the sandbox with it. Having said that, I should stress that the nature of optimizing a general-use product like EM always needs to stress the 80% case. I''ve found that very few applications have the kind of connection density combined with a bursty traffic profile that would cause select to start being a problem. (Classic example: an always-connected aynchronous messaging system.) I''ve been promising Tony the EM epoll code for way too long now, I''ve got to make some time and send it along. At "typical" load levels, it doesn''t add anything noticeable over select. But you guys might come up with some apps where it might make a diff. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070226/6e3c97e2/attachment-0001.html
For small (hundreds, not thousands) numbers of file descriptors, why would you expect any performance benefit over select? On 2/26/07, Tony Arcieri <tony at clickcaster.com> wrote:> Francis said he saw no noticeable performance improvement with an > epoll-based implementation over Kernel#select (I''d REALLY like to see that > code though). I''ve been talking with Ruby core about this problem and will > probably try to write my own epoll-based patch to Ruby 1.9 which I''ll > benchmark to see if it actually results in improved performance.
On 2/26/07, Michael S. Fischer <michael at dynamine.net> wrote:> > Francis, > > I read in the list archives back that a future EventMachine release > will support epoll on Linux (i.e., it''s in the trunk). > > Better still, is there a possibility that EM will rely on libevent so > that it will be architecture independent (i.e. epoll on Linux, kqueue > on FreeBSD/Mac OS X, /dev/poll on Solaris)? This is how memcached is > implemented, and it would be helpful to be able to assume maximum > performance on any mainstream host OS (aside, perhaps, from Windows).I thought of another important point, Michael. While EM will always aim to be the fastest, most scalable I/O framework available for Ruby (that''s it''s fundamental charter), there''s another couple of areas where it''s still deficient. If we really want this product to give Ruby a complete answer to Python''s Twisted, then we''ll need to add support for events that are no I/O events. Like GUI events, keyboard events, page faults (maybe), and user-defined events. One thing Twisted has done a great deal of work with is enabling users to develop custom network protocols. EM does that as well, although there is a learning curve and sort of a "preferred" way of doing it, that is under-documented. I happen to believe that user applications should AVOID developing custom protocols and stick to standards like REST, XMPP, and maybe things like Bayeux. For that reason, I''ve de-emphasized really building out the custom-protocol development support in EM. It''s not bad or hard to do, understand. But it''s not as feature-rich as it could be if that were a core area of focus for the product. Eventually, I really want to see a full-featured, high-performance asynchronous messaging protocol built into EM. (I''ve done a fair bit of work developing on AMQP, but haven''t finished.) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070226/c47fe704/attachment.html
In the case of DistribuStream, we''re using a Ruby-based server to schedule data transfer across a network of peers. Every single peer needs a connection back to the server, so the size of the peer network is limited to the number of connections the server can sustain. So, what matters the most to us is having as high of a connection count as possible. It seems like Kernel#select is somehow scaling beyond FD_SETSIZE (or at least the typical default of 1024) and that mitigates part of the problem, but how well does it scale, to say, 10,000 concurrent connections? The problem with having the multiplexing in an extension rather than Ruby core is the sort of changes which need to be made to solve some of the fundamental I/O problems, namely the need to run more scalable system calls (epoll, kqueue) side-by-side with Ruby''s event loop. Real optimizations (post-YARV) could come from disentangling that entire process and having a single central event loop which uses the best call available on the given platform. If this could happen, it opens up the possibility of allowing these calls to run for longer periods of time, instead of the current approach which could end up running both select() and a high performance call multiple times every second. - Tony On 2/26/07, Francis Cianfrocca <garbagecat10 at gmail.com> wrote:> > On 2/26/07, Michael S. Fischer <michael at dynamine.net> wrote: > > > > Francis, > > > > I read in the list archives back that a future EventMachine release > > will support epoll on Linux (i.e., it''s in the trunk). > > > > Better still, is there a possibility that EM will rely on libevent so > > that it will be architecture independent (i.e. epoll on Linux, kqueue > > on FreeBSD/Mac OS X, /dev/poll on Solaris)? This is how memcached is > > implemented, and it would be helpful to be able to assume maximum > > performance on any mainstream host OS (aside, perhaps, from Windows). > > > > Well, candidly, the select-based implementation we have now really does > perform quite well on all supported platforms. I don''t see significant > difficulties with integrating any of the other techniques, other than IOCP, > which is not monumentally difficult but I''m not sure it''s worth doing. > > My feeling about libevent is basically this: the charter of EM is to get > right down to the metal. The code base that it grew out of has been worked > on and tuned for quite a few years to have high performance basically > everywhere it runs. So I don''t see much value in interposing another layer > between EM and the metal. Especially since your issue is with cross-platform > interoperability. I''d rather do the work to get those extra bits of > performance out of the existing code, which knows enough about Ruby to play > nice enough in the sandbox with it. > > Having said that, I should stress that the nature of optimizing a > general-use product like EM always needs to stress the 80% case. I''ve found > that very few applications have the kind of connection density combined with > a bursty traffic profile that would cause select to start being a problem. > (Classic example: an always-connected aynchronous messaging system.) I''ve > been promising Tony the EM epoll code for way too long now, I''ve got to make > some time and send it along. At "typical" load levels, it doesn''t add > anything noticeable over select. But you guys might come up with some apps > where it might make a diff. > > > _______________________________________________ > Eventmachine-talk mailing list > Eventmachine-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/eventmachine-talk >-- Tony Arcieri ClickCaster, Inc. tony at clickcaster.com (970) 232-4208 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070226/da39c2d6/attachment-0001.html
Well, that sure killed the conversation... :) There''s something I''m curious about though: how is EventMachine scaling beyond FD_SETSIZE? (I suppose I could Use The Source, Luke) Are you defining a larger FD_SETSIZE for the underlying select() call? Or is my testing code that''s showing me 1200+ concurrent connections buggy? - Tony On 2/26/07, Tony Arcieri <tony at clickcaster.com> wrote:> > In the case of DistribuStream, we''re using a Ruby-based server to schedule > data transfer across a network of peers. Every single peer needs a > connection back to the server, so the size of the peer network is limited to > the number of connections the server can sustain. > > So, what matters the most to us is having as high of a connection count as > possible. It seems like Kernel#select is somehow scaling beyond FD_SETSIZE > (or at least the typical default of 1024) and that mitigates part of the > problem, but how well does it scale, to say, 10,000 concurrent connections? > > The problem with having the multiplexing in an extension rather than Ruby > core is the sort of changes which need to be made to solve some of the > fundamental I/O problems, namely the need to run more scalable system calls > (epoll, kqueue) side-by-side with Ruby''s event loop. Real optimizations > (post-YARV) could come from disentangling that entire process and having a > single central event loop which uses the best call available on the given > platform. If this could happen, it opens up the possibility of allowing > these calls to run for longer periods of time, instead of the current > approach which could end up running both select() and a high performance > call multiple times every second. > > - Tony > > On 2/26/07, Francis Cianfrocca <garbagecat10 at gmail.com> wrote: > > > On 2/26/07, Michael S. Fischer <michael at dynamine.net> wrote: > > > > > > Francis, > > > > > > I read in the list archives back that a future EventMachine release > > > will support epoll on Linux (i.e., it''s in the trunk). > > > > > > Better still, is there a possibility that EM will rely on libevent so > > > that it will be architecture independent (i.e. epoll on Linux, kqueue > > > on FreeBSD/Mac OS X, /dev/poll on Solaris)? This is how memcached is > > > implemented, and it would be helpful to be able to assume maximum > > > performance on any mainstream host OS (aside, perhaps, from Windows). > > > > > > > > Well, candidly, the select-based implementation we have now really does > > perform quite well on all supported platforms. I don''t see significant > > difficulties with integrating any of the other techniques, other than IOCP, > > which is not monumentally difficult but I''m not sure it''s worth doing. > > > > My feeling about libevent is basically this: the charter of EM is to get > > right down to the metal. The code base that it grew out of has been worked > > on and tuned for quite a few years to have high performance basically > > everywhere it runs. So I don''t see much value in interposing another layer > > between EM and the metal. Especially since your issue is with cross-platform > > interoperability. I''d rather do the work to get those extra bits of > > performance out of the existing code, which knows enough about Ruby to play > > nice enough in the sandbox with it. > > > > Having said that, I should stress that the nature of optimizing a > > general-use product like EM always needs to stress the 80% case. I''ve found > > that very few applications have the kind of connection density combined with > > a bursty traffic profile that would cause select to start being a problem. > > (Classic example: an always-connected aynchronous messaging system.) I''ve > > been promising Tony the EM epoll code for way too long now, I''ve got to make > > some time and send it along. At "typical" load levels, it doesn''t add > > anything noticeable over select. But you guys might come up with some apps > > where it might make a diff. > > > > > > _______________________________________________ > > Eventmachine-talk mailing list > > Eventmachine-talk at rubyforge.org > > http://rubyforge.org/mailman/listinfo/eventmachine-talk > > > > > > -- > Tony Arcieri > ClickCaster, Inc. > tony at clickcaster.com > (970) 232-4208 >-- Tony Arcieri ClickCaster, Inc. tony at clickcaster.com (970) 232-4208 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070227/804f6ac9/attachment-0001.html
On 2/27/07, Tony Arcieri <tony at clickcaster.com> wrote:> > Well, that sure killed the conversation... :) > > There''s something I''m curious about though: how is EventMachine scaling > beyond FD_SETSIZE? (I suppose I could Use The Source, Luke) > > Are you defining a larger FD_SETSIZE for the underlying select() call? > > Or is my testing code that''s showing me 1200+ concurrent connections > buggy? > > - TonyNothing funky about SETSIZE, which gets inherited from the underlying Ruby build, afaik. I think you probably have bugs ;-) Scaling to 10,000: Until we have proper epoll (which would be responsive to ulimits and break down the FD_SETSIZE barrier), you can look at multiple processes. What OS do you run? I think Linux would have trouble with 10,000 connections in the kernel. Solaris can handle 50,000 at least, without breaking a sweat. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070227/a11fb16e/attachment.html