I wonder if it would be possible to integrate libevent''s own asynch DNS library in rev somehow. Heck if we''re going that deep then I guess we could even write a wrapper for their HTTP server :) Take care. -R http://www.monkey.org/~provos/libevent/doxygen-1.4.3/
On Tue, Aug 12, 2008 at 12:41 PM, Roger Pack < roger.pack at leadmediapartners.com> wrote:> I wonder if it would be possible to integrate libevent''s own asynch > DNS library in rev somehow. >That might be a bit difficult considering that Rev uses libev''s native API without libevent compat. Are you having problems with Rev''s pure Ruby async DNS? -- Tony Arcieri medioh.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://rubyforge.org/pipermail/rev-talk/attachments/20080812/e3642354/attachment.html>
Oh for some reason I thought it was using libevent not libev. I have no idea what the status of libev is for windows. I do hope it''s something better than using select, since I think we''re bound to 64 ports max on select, since we use the old msvcrt lib or something. Even resetting FD_SETSIZE didn''t help last time I tried it. -R On Tue, Aug 12, 2008 at 12:45 PM, Tony Arcieri <tony at medioh.com> wrote:> On Tue, Aug 12, 2008 at 12:41 PM, Roger Pack > <roger.pack at leadmediapartners.com> wrote: >> >> I wonder if it would be possible to integrate libevent''s own asynch >> DNS library in rev somehow. > > That might be a bit difficult considering that Rev uses libev''s native API > without libevent compat. Are you having problems with Rev''s pure Ruby async > DNS? > > -- > Tony Arcieri > medioh.com > > _______________________________________________ > Rev-talk mailing list > Rev-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/rev-talk > >
On Tue, Aug 12, 2008 at 1:43 PM, Roger Pack < roger.pack at leadmediapartners.com> wrote:> Oh for some reason I thought it was using libevent not libev. > I have no idea what the status of libev is for windows. I do hope > it''s something better than using select, since I think we''re bound to > 64 ports max on select, since we use the old msvcrt lib or something. > Even resetting FD_SETSIZE didn''t help last time I tried it. >Yes, there are relatively few solutions to multiplexing more than 64 file descriptors on Windows, and even fewer which allow you to write portable applications. This is due to a 64 event object limitation in WSAWaitForMultipleEvents() function which was imposed in early versions of Windows NT. For whatever reason, Microsoft never decided to increase this limit. Microsoft''s intended solution is for you to use I/O completion ports in conjunction with thread pools. This is a nicely scalable solution but not one which maps very well to paradigms on other platforms. Solaris''s event completion ports are perhaps the closest match but there''s still a bit of a mismatch. So all that said, the best solution is to use thread pools which multiplex up to 64 descriptors each. -- Tony Arcieri medioh.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://rubyforge.org/pipermail/rev-talk/attachments/20080812/483bc506/attachment.html>
I haven''t looked into this too much, but here are a few thoughts: I''ve heard some random rumors that if you increase the FD_SETSIZE that windows is able to use it. Except that I actually tried that on mingw and it didn''t work. So I guess your suggestion of multiple threads is about the only way to handle select. It''s apparently possible to emulate asynchronous calls using IOCP[3]. When I google for ''libev iocp'' I get something that looks like a feature request, with no news of it actually working. libevent does compile out of the box, but the test programs don''t seem to output anything, so I''m a little wary to say it works. one can download the libev doesn''t seem to compile. If I ever get around to it on windows I may try to see if I can hook up libevent instead of libev. Maybe it will work without threading. I can hope :) Also it appears possible to run the libevent http server using libev[1] so... it could happen :) Looks like Ry was bulding something similar[2]. Not that I plan on ever doing it, but it''s just thought fodder. Take care. -R [1] http://dist.schmorp.de/libev/README.how_to_choose_between_libevent_and_libev_tarballs [2] http://lists.schmorp.de/pipermail/libev/2008q3/000406.html [3] http://www.eggheadcafe.com/software/aspnet/32292639/integrate-windows-io-com.aspx On Tue, Aug 12, 2008 at 2:05 PM, Tony Arcieri <tony at medioh.com> wrote:> On Tue, Aug 12, 2008 at 1:43 PM, Roger Pack > <roger.pack at leadmediapartners.com> wrote: >> >> Oh for some reason I thought it was using libevent not libev. >> I have no idea what the status of libev is for windows. I do hope >> it''s something better than using select, since I think we''re bound to >> 64 ports max on select, since we use the old msvcrt lib or something. >> Even resetting FD_SETSIZE didn''t help last time I tried it. > > Yes, there are relatively few solutions to multiplexing more than 64 file > descriptors on Windows, and even fewer which allow you to write portable > applications. This is due to a 64 event object limitation in > WSAWaitForMultipleEvents() function which was imposed in early versions of > Windows NT. For whatever reason, Microsoft never decided to increase this > limit. > > Microsoft''s intended solution is for you to use I/O completion ports in > conjunction with thread pools. This is a nicely scalable solution but not > one which maps very well to paradigms on other platforms. Solaris''s event > completion ports are perhaps the closest match but there''s still a bit of a > mismatch. > > So all that said, the best solution is to use thread pools which multiplex > up to 64 descriptors each. > > -- > Tony Arcieri > medioh.com > > _______________________________________________ > Rev-talk mailing list > Rev-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/rev-talk > >
On Tue, Aug 12, 2008 at 2:53 PM, Roger Pack < roger.pack at leadmediapartners.com> wrote:> I''ve heard some random rumors that if you increase the FD_SETSIZE that > windows is able to use it.Nope, the 64 object limitation is an inherent limit of the Win32 API and cannot be circumvented.> It''s apparently possible to emulate asynchronous calls using IOCP[3]. >It''s possible to do something similar to POSIX async I/O, if you''d like high performance async file I/O. It wouldn''t really be applicable to network sockets if that''s your primary use case. I''ve talked to the libev author about this, who wrote a library called libeio for this purpose, and he thought the semantics between POSIX async I/O and IOCP were too different to really have a clean cross-platform abstraction.> If I ever get around to it on windows I may try to see if I can hook > up libevent instead of libev.Have you tried to build Rev on Windows? If you''re interested in getting it working I''d certainly help out in trying to get it working, although I don''t have a Windows machine to do any testing on myself. -- Tony Arcieri medioh.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://rubyforge.org/pipermail/rev-talk/attachments/20080812/676122f4/attachment.html>
> Nope, the 64 object limitation is an inherent limit of the Win32 API and > cannot be circumvented.Rumors exist [1] :)>> It''s apparently possible to emulate asynchronous calls using IOCP[3].I think what I meant is emulate "event-driven I/O." Files...no clue there. I know that IOCP are supposed to give you a way around the 64-fd limit, but I''m not sure if things'' like [2] do or not.>> If I ever get around to it on windows I may try to see if I can hook >> up libevent instead of libev. > > Have you tried to build Rev on Windows? If you''re interested in getting it > working I''d certainly help out in trying to get it working, although I don''t > have a Windows machine to do any testing on myself.I have. Here''s the output from trying to install rev [mingw with the devkit]: gem install rev gcc -I. -I. -Ic:/roger/dev/rubyinstaller_lavena/sandbox/ruby_mingw/lib/ruby/1.8/i386-mingw32 -I. -DRUBY_VERSION_CODE=186 -g -O2 -c rev_buffer.c rev_buffer.c:630:2: warning: no newline at end of file gcc -I. -I. -Ic:/roger/dev/rubyinstaller_lavena/sandbox/ruby_mingw/lib/ruby/1.8/i386-mingw32 -I. -DRUBY_VERSION_CODE=186 -g -O2 -c rev_ext.c In file included from rev_ext.c:10: ../libev/ev.c: In function `fd_intern'': ../libev/ev.c:981: warning: passing arg 3 of `rb_w32_ioctlsocket'' from incompatible pointer type In file included from ../libev/ev.c:1191, from rev_ext.c:10: ../libev/ev_select.c: In function `select_poll'': ../libev/ev_select.c:137: error: `NFDBITS'' undeclared (first use in this function) ../libev/ev_select.c:137: error: (Each undeclared identifier is reported only once ../libev/ev_select.c:137: error: for each function it appears in.) make: *** [rev_ext.o] Error 1 Here''s the output from trying to install libev: make all-am make[1]: Entering directory `/c/roger/dev/libev-3.43'' /bin/sh ./libtool --mode=compile gcc -DHAVE_CONFIG_H -I. -I. -I. -g -O2 -O3 -c -o event.lo `test -f ''event.c'' || echo ''./''`event.c gcc -DHAVE_CONFIG_H -I. -I. -I. -g -O2 -O3 -c event.c -DDLL_EXPORT -DPIC -o .l ibs/event.o In file included from event.c:51: event.h:115: warning: "struct timeval" declared inside parameter list event.h:115: warning: its scope is only this definition or declaration, which is probably not what you want event.h:126: warning: "struct timeval" declared inside parameter list event.h:128: warning: "struct timeval" declared inside parameter list event.h:132: warning: "struct timeval" declared inside parameter list event.h:139: warning: "struct timeval" declared inside parameter list event.h:141: warning: "struct timeval" declared inside parameter list event.c:71: warning: "struct timeval" declared inside parameter list event.c: In function `tv_set'': event.c:73: error: dereferencing pointer to incomplete type event.c:74: error: dereferencing pointer to incomplete type event.c:74: error: dereferencing pointer to incomplete type event.c: At top level: I don''t really care since I''m not developing on windows, but just doing it out of curiosity. Thanks! -=R [1] http://www.nntp.perl.org/group/perl.loop/2007/11/msg1082.html [2] http://article.gmane.org/gmane.comp.lib.libevent.user/158 other possibly useful: http://www.eggheadcafe.com/software/aspnet/32292639/integrate-windows-io-com.aspx http://code.jellycan.com/memcached/
On Tue, Aug 12, 2008 at 3:25 PM, Roger Pack < roger.pack at leadmediapartners.com> wrote:> Rumors exist [1] :) >Interesting... I hadn''t heard about that. If you''re really interested in this, I''d suggest posting about it on the libev mailing list, although they may be doing something to that effect already.> Here''s the output from trying to install rev [mingw with the devkit]: >I''ll try to figure out what I need to do in order to detect / build libev on Windows in the extconf.rb and ping you when I think I''ve found a fix.> I don''t really care since I''m not developing on windows, but just > doing it out of curiosity.Same here, however if there''s a quick fix to get Rev building on Windows I''m willing to try. The libev guy has certainly put in a decent amount of effort into supporting Windows, so it shouldn''t be too much work. -- Tony Arcieri medioh.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://rubyforge.org/pipermail/rev-talk/attachments/20080812/d1ccdf3e/attachment.html>
> Interesting... I hadn''t heard about that. If you''re really interested in > this, I''d suggest posting about it on the libev mailing list, although they > may be doing something to that effect already.I wonder if compiling all of Ruby, itself, with FD_SETSIZE set to a larger number would work. I also wonder if windows works with non-blocking file handles [sockets and handles]. [this guy [1] mentioned something] -- but I guess that''s an optimization for another day. I also wonder why libevent and libev don''t just work together to form one project that''s optimized. Seems like duplication there :) Thanks for your work. -=R [1] http://monkeymail.org/archives/libevent-users/2008-July/001343.html> >> >> Here''s the output from trying to install rev [mingw with the devkit]: > > I''ll try to figure out what I need to do in order to detect / build libev on > Windows in the extconf.rb and ping you when I think I''ve found a fix. > >> >> I don''t really care since I''m not developing on windows, but just >> doing it out of curiosity. > > Same here, however if there''s a quick fix to get Rev building on Windows I''m > willing to try. The libev guy has certainly put in a decent amount of > effort into supporting Windows, so it shouldn''t be too much work. > > -- > Tony Arcieri > medioh.com > > _______________________________________________ > Rev-talk mailing list > Rev-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/rev-talk > >-- Thanks! -=R