Mongrel development - Dec 2007 - Some silly benchs (was: 1.9)

Guys,

Just for fun, I tried to see (I know, a silly way to test it) how much
overhead we have calling the C functions of the extensions.

the benchmark script and the results:

http://pastie.caboo.se/128646

The naive C extension:

http://pastie.caboo.se/128647


I compared 1.8.6 (VC6 and mingw builds) against a fresh checkout of ruby trunk.

What I understand from that is 1.9 is slower than 1.8 regarding FFI
calling, but compared to pure-ruby, it''s almost equal, which differ
from 1.8 where pure-ruby is twice slow.

On another topic, it seems serverside is based on event-machine, like
swiftiply. Tried to checkout EM code but the repo is a mess without a
clear indication of what branch/directory should be used.

Anyway, it appears lot of libraries based on events perform better
than other alternatives.

But, just random thoughts.

-- 
Luis Lavena
Multimedia systems
-
A common mistake that people make when trying to design
something completely foolproof is to underestimate
the ingenuity of complete fools.
Douglas Adams

On Dec 14, 2007 12:43 PM, Luis Lavena <luislavena at gmail.com>
wrote:
> On another topic, it seems serverside is based on event-machine, like
> swiftiply. Tried to checkout EM code but the repo is a mess without a
> clear indication of what branch/directory should be used.
Just download the latest production release of serverside.  An
interesting benchmark would be to just compare the HTTP parsing piece
of it with Mongrel''s, both for performance and for correct parsing.


Kirk Haines

On Dec 14, 2007 5:08 PM, Kirk Haines <wyhaines at gmail.com>
wrote:
> On Dec 14, 2007 12:43 PM, Luis Lavena <luislavena at gmail.com>
wrote:
> > On another topic, it seems serverside is based on event-machine, like
> > swiftiply. Tried to checkout EM code but the repo is a mess without a
> > clear indication of what branch/directory should be used.
>
> Just download the latest production release of serverside.  An
> interesting benchmark would be to just compare the HTTP parsing piece
> of it with Mongrel''s, both for performance and for correct
parsing.
>
I''ll try to rip the ServerSide::HTTP::Parsing module and drop the same
tests mongrel does to see how it perform (and how valid it is).

Anyway, the pastie was to show the overhead of calling C functions in
1.9 compared to the same naive one in pure-ruby.

Hopefully I wouldn''t need to adapt lot of code from ServerSide to make
the tests pass on 1.9.

--
Luis Lavena
Multimedia systems
-
A common mistake that people make when trying to design
something completely foolproof is to underestimate
the ingenuity of complete fools.
Douglas Adams

On Fri, 14 Dec 2007 19:05:28 -0300
"Luis Lavena" <luislavena at gmail.com> wrote:
> I''ll try to rip the ServerSide::HTTP::Parsing module and drop the
same
> tests mongrel does to see how it perform (and how valid it is).
> 
> Anyway, the pastie was to show the overhead of calling C functions in
> 1.9 compared to the same naive one in pure-ruby.
> 
> Hopefully I wouldn''t need to adapt lot of code from ServerSide to
make
> the tests pass on 1.9.
I wouldn''t bother.  A cornerstone of Mongrel''s design is that
by using
a parser generated by a mathematically sound (hopefully) parser Mongrel
is able to reject about 80% of security hacks simply becuase they
usually violate the RFC grammar.

This is in essence creating a white list for HTTP requests and
rejecting anything else, with very exact reasons based on grammar
violations.

What Serverside (and every other web server) does is creates a black
list for HTTP requests and allowing nearly all of them through.  It
gets its speed from basically not being thorough, and it only needs to
do this in Ruby.  As the mongrel parser shows you can still use a
generator and get a fast as hell http parser.

So, running off to a web server that''s implemented half assed with the
goal of bringing said half-assedness to mongrel in order to improve
speed would remove nearly all the other advantages.  That''s simply bad
engineering all around.

It would be better to instead check a Ruby version of the .rl parser
against 1.9 and then see what can be done to speed it up without losing
it''s valuable exactness.

-- 
Zed A. Shaw
- Hate: http://savingtheinternetwithhate.com/
- Good: http://www.zedshaw.com/
- Evil: http://yearofevil.com/

Yeah I am not a fan of a regex parser.

What exactly is the deal with events on 1.9? How many connections
should we realistically expect? Would it be just as good to use
select()?

My understanding is that Fibers don''t cooperatively multitask the way
green threads do, so I don''t think Fibers will offer a way forward
without horrible hacks. On the other hand, I''m not totally clear how
events solve this situation but I haven''t looked at any evented lib

Is there a way we could re-use a small process pool, and queue extra
requests in a simple buffer? How many contexts do you need to get
decent CPU saturation in most situations? Do they usually wait on IO
20% of the time, or 60%, or... I realize I am asking for benchmarks
without any context.

My understand is that WSGI is excellent (Python). How do they handle
these issues? Python''s threads are just like YARV''s.

Evan

On Dec 14, 2007 7:41 PM, Zed A. Shaw <zedshaw at zedshaw.com>
wrote:
> On Fri, 14 Dec 2007 19:05:28 -0300
> "Luis Lavena" <luislavena at gmail.com> wrote:
>
> > I''ll try to rip the ServerSide::HTTP::Parsing module and drop
the same
> > tests mongrel does to see how it perform (and how valid it is).
> >
> > Anyway, the pastie was to show the overhead of calling C functions in
> > 1.9 compared to the same naive one in pure-ruby.
> >
> > Hopefully I wouldn''t need to adapt lot of code from
ServerSide to make
> > the tests pass on 1.9.
>
> I wouldn''t bother.  A cornerstone of Mongrel''s design is
that by using
> a parser generated by a mathematically sound (hopefully) parser Mongrel
> is able to reject about 80% of security hacks simply becuase they
> usually violate the RFC grammar.
>
> This is in essence creating a white list for HTTP requests and
> rejecting anything else, with very exact reasons based on grammar
> violations.
>
> What Serverside (and every other web server) does is creates a black
> list for HTTP requests and allowing nearly all of them through.  It
> gets its speed from basically not being thorough, and it only needs to
> do this in Ruby.  As the mongrel parser shows you can still use a
> generator and get a fast as hell http parser.
>
> So, running off to a web server that''s implemented half assed with
the
> goal of bringing said half-assedness to mongrel in order to improve
> speed would remove nearly all the other advantages.  That''s simply
bad
> engineering all around.
>
> It would be better to instead check a Ruby version of the .rl parser
> against 1.9 and then see what can be done to speed it up without losing
> it''s valuable exactness.
>
> --
> Zed A. Shaw
> - Hate: http://savingtheinternetwithhate.com/
> - Good: http://www.zedshaw.com/
> - Evil: http://yearofevil.com/
>
> _______________________________________________
> Mongrel-development mailing list
> Mongrel-development at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-development
>


-- 
Evan Weaver
Cloudburst, LLC

> events solve this situation but I haven''t looked at any evented
lib
Got cut off. I haven''t looked at any evented lib in detail.

On Dec 14, 2007 7:43 PM, Evan Weaver <evan at cloudbur.st>
wrote:
> Yeah I am not a fan of a regex parser.
>
> What exactly is the deal with events on 1.9? How many connections
> should we realistically expect? Would it be just as good to use
> select()?
>
> My understanding is that Fibers don''t cooperatively multitask the
way
> green threads do, so I don''t think Fibers will offer a way forward
> without horrible hacks. On the other hand, I''m not totally clear
how
> events solve this situation but I haven''t looked at any evented
lib
>
> Is there a way we could re-use a small process pool, and queue extra
> requests in a simple buffer? How many contexts do you need to get
> decent CPU saturation in most situations? Do they usually wait on IO
> 20% of the time, or 60%, or... I realize I am asking for benchmarks
> without any context.
>
> My understand is that WSGI is excellent (Python). How do they handle
> these issues? Python''s threads are just like YARV''s.
>
> Evan
>
>
> On Dec 14, 2007 7:41 PM, Zed A. Shaw <zedshaw at zedshaw.com> wrote:
> > On Fri, 14 Dec 2007 19:05:28 -0300
> > "Luis Lavena" <luislavena at gmail.com> wrote:
> >
> > > I''ll try to rip the ServerSide::HTTP::Parsing module and
drop the same
> > > tests mongrel does to see how it perform (and how valid it is).
> > >
> > > Anyway, the pastie was to show the overhead of calling C
functions in
> > > 1.9 compared to the same naive one in pure-ruby.
> > >
> > > Hopefully I wouldn''t need to adapt lot of code from
ServerSide to make
> > > the tests pass on 1.9.
> >
> > I wouldn''t bother.  A cornerstone of Mongrel''s
design is that by using
> > a parser generated by a mathematically sound (hopefully) parser
Mongrel
> > is able to reject about 80% of security hacks simply becuase they
> > usually violate the RFC grammar.
> >
> > This is in essence creating a white list for HTTP requests and
> > rejecting anything else, with very exact reasons based on grammar
> > violations.
> >
> > What Serverside (and every other web server) does is creates a black
> > list for HTTP requests and allowing nearly all of them through.  It
> > gets its speed from basically not being thorough, and it only needs to
> > do this in Ruby.  As the mongrel parser shows you can still use a
> > generator and get a fast as hell http parser.
> >
> > So, running off to a web server that''s implemented half assed
with the
> > goal of bringing said half-assedness to mongrel in order to improve
> > speed would remove nearly all the other advantages.  That''s
simply bad
> > engineering all around.
> >
> > It would be better to instead check a Ruby version of the .rl parser
> > against 1.9 and then see what can be done to speed it up without
losing
> > it''s valuable exactness.
> >
> > --
> > Zed A. Shaw
> > - Hate: http://savingtheinternetwithhate.com/
> > - Good: http://www.zedshaw.com/
> > - Evil: http://yearofevil.com/
> >
> > _______________________________________________
> > Mongrel-development mailing list
> > Mongrel-development at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/mongrel-development
> >
>
>
>
> --
> Evan Weaver
> Cloudburst, LLC
>


-- 
Evan Weaver
Cloudburst, LLC

On Fri, 14 Dec 2007 19:43:02 -0500
"Evan Weaver" <evan at cloudbur.st> wrote:
> Yeah I am not a fan of a regex parser.
> 
> What exactly is the deal with events on 1.9? How many connections
> should we realistically expect? Would it be just as good to use
> select()?
Part of this "events will save" us attitude is based on myth.
Originally green or real threads were just too heavyweight for most of
the early c10k problems so things like libevent came about.  The issue
with using libevent though (as I found in my Ruby/Event project) is
that Ruby likes being king of the events and doesn''t give them up.
When you combine the event loop in libevent with the one in ruby via
select you get deadlock.

Switching to eventmachine won''t work either because it''s just
a heavy
layer around select, so you''re still stuck at 1024 max open files.

When it comes down to it, the ideal situation has been always a fixed
set of threads/processes that use some form of event loop (select or
other) to handle massive amounts of IO.  This usually gives you the
most optimal spread of the IO events and give the best parallel
processing.

Ruby 1.9 might allow this, keeping select and using real threads, but
that''d take a serious re-engineer of mongrel.

But, I''m looking at maybe resurrecting my Ruby/Event library using
libev under 1.9 in order to solve this problem permanently (and
cleanly, unlike that garbage festival Ruby/EventMachine is).

-- 
Zed A. Shaw
- Hate: http://savingtheinternetwithhate.com/
- Good: http://www.zedshaw.com/
- Evil: http://yearofevil.com/

On Dec 14, 2007 9:41 PM, Zed A. Shaw <zedshaw at zedshaw.com> wrote:
[...]
>
> I wouldn''t bother.  A cornerstone of Mongrel''s design is
that by using
> a parser generated by a mathematically sound (hopefully) parser Mongrel
> is able to reject about 80% of security hacks simply becuase they
> usually violate the RFC grammar.
>
> This is in essence creating a white list for HTTP requests and
> rejecting anything else, with very exact reasons based on grammar
> violations.
>
> What Serverside (and every other web server) does is creates a black
> list for HTTP requests and allowing nearly all of them through.  It
> gets its speed from basically not being thorough, and it only needs to
> do this in Ruby.  As the mongrel parser shows you can still use a
> generator and get a fast as hell http parser.
>
> So, running off to a web server that''s implemented half assed with
the
> goal of bringing said half-assedness to mongrel in order to improve
> speed would remove nearly all the other advantages.  That''s simply
bad
> engineering all around.
>
That wasn''t my point, I was trying to comment that we could bomb
ServerSide HTTP parser with Mongrel tests and see how it behaves. I
wasn''t thinking on replace it.

The curious note is that a lot of server, the ruby based ones and
other based on twisted (python) take the evented approach, which is
"funny", just a foot note, not actually a desire to implement.
> It would be better to instead check a Ruby version of the .rl parser
> against 1.9 and then see what can be done to speed it up without losing
> it''s valuable exactness.
I still agree with you on that. I need to get a stable 1.9 version for
windows to abuse (er, I mean, to use).

-- 
Luis Lavena
Multimedia systems
-
A common mistake that people make when trying to design
something completely foolproof is to underestimate
the ingenuity of complete fools.
Douglas Adams

On Dec 14, 2007 10:37 PM, Zed A. Shaw <zedshaw at zedshaw.com>
wrote:
> On Fri, 14 Dec 2007 19:43:02 -0500
> "Evan Weaver" <evan at cloudbur.st> wrote:
>
> > Yeah I am not a fan of a regex parser.
> >
> > What exactly is the deal with events on 1.9? How many connections
> > should we realistically expect? Would it be just as good to use
> > select()?
>
> Part of this "events will save" us attitude is based on myth.
> Originally green or real threads were just too heavyweight for most of
> the early c10k problems so things like libevent came about.  The issue
> with using libevent though (as I found in my Ruby/Event project) is
> that Ruby likes being king of the events and doesn''t give them up.
> When you combine the event loop in libevent with the one in ruby via
> select you get deadlock.
>
As replied before, I don''t think event will save us, but is
interesting that lot of frameworks/server take that approach "as a
solution"... when ruby is the true limiting factor in the whole
equation.
> Switching to eventmachine won''t work either because it''s
just a heavy
> layer around select, so you''re still stuck at 1024 max open files.
>
We need to figure out how to elude this ruby limitation. Or fix ruby,
or drop it for good.
> When it comes down to it, the ideal situation has been always a fixed
> set of threads/processes that use some form of event loop (select or
> other) to handle massive amounts of IO.  This usually gives you the
> most optimal spread of the IO events and give the best parallel
> processing.
Worker pools? In my experience they are hard to debug and bugs often
are too deep and hidden to really fix them (not to mention some
deadlocks)...

But maybe that talks on my lack of experience with them.
> Ruby 1.9 might allow this, keeping select and using real threads, but
> that''d take a serious re-engineer of mongrel.
>
> But, I''m looking at maybe resurrecting my Ruby/Event library using
> libev under 1.9 in order to solve this problem permanently (and
> cleanly, unlike that garbage festival Ruby/EventMachine is).
Amen on that. I just took the eventmachine code to "see" what is in
there and couldn''t understand it well... (ala, a mess?)

Also tried to run the tests and ''see'' them pass on windows and
it crashed badly.

-- 
Luis Lavena
Multimedia systems
-
A common mistake that people make when trying to design
something completely foolproof is to underestimate
the ingenuity of complete fools.
Douglas Adams

I see now that WSGI is a interface spec, not a single server. What''s
the main WSGI server? Apache/mod_wsgi? The stdlib
wsgiref.simple_server looks like Webrick.

CherryPy''s server looks pretty good (
http://www.cherrypy.org/browser/trunk/cherrypy/wsgiserver/__init__.py
). They use a thread pool with a queue. I guess the GIL lets them
sidestep queue race issues?

Evan

On Dec 14, 2007 10:10 PM, Luis Lavena <luislavena at gmail.com>
wrote:
> On Dec 14, 2007 10:37 PM, Zed A. Shaw <zedshaw at zedshaw.com> wrote:
> > On Fri, 14 Dec 2007 19:43:02 -0500
> > "Evan Weaver" <evan at cloudbur.st> wrote:
> >
> > > Yeah I am not a fan of a regex parser.
> > >
> > > What exactly is the deal with events on 1.9? How many connections
> > > should we realistically expect? Would it be just as good to use
> > > select()?
> >
> > Part of this "events will save" us attitude is based on
myth.
> > Originally green or real threads were just too heavyweight for most of
> > the early c10k problems so things like libevent came about.  The issue
> > with using libevent though (as I found in my Ruby/Event project) is
> > that Ruby likes being king of the events and doesn''t give
them up.
> > When you combine the event loop in libevent with the one in ruby via
> > select you get deadlock.
> >
>
> As replied before, I don''t think event will save us, but is
> interesting that lot of frameworks/server take that approach "as a
> solution"... when ruby is the true limiting factor in the whole
> equation.
>
> > Switching to eventmachine won''t work either because
it''s just a heavy
> > layer around select, so you''re still stuck at 1024 max open
files.
> >
>
> We need to figure out how to elude this ruby limitation. Or fix ruby,
> or drop it for good.
>
> > When it comes down to it, the ideal situation has been always a fixed
> > set of threads/processes that use some form of event loop (select or
> > other) to handle massive amounts of IO.  This usually gives you the
> > most optimal spread of the IO events and give the best parallel
> > processing.
>
> Worker pools? In my experience they are hard to debug and bugs often
> are too deep and hidden to really fix them (not to mention some
> deadlocks)...
>
> But maybe that talks on my lack of experience with them.
>
> > Ruby 1.9 might allow this, keeping select and using real threads, but
> > that''d take a serious re-engineer of mongrel.
> >
> > But, I''m looking at maybe resurrecting my Ruby/Event library
using
> > libev under 1.9 in order to solve this problem permanently (and
> > cleanly, unlike that garbage festival Ruby/EventMachine is).
>
> Amen on that. I just took the eventmachine code to "see" what is
in
> there and couldn''t understand it well... (ala, a mess?)
>
> Also tried to run the tests and ''see'' them pass on
windows and it crashed badly.
>
> --
> Luis Lavena
> Multimedia systems
> -
> A common mistake that people make when trying to design
> something completely foolproof is to underestimate
> the ingenuity of complete fools.
> Douglas Adams
> _______________________________________________
>
> Mongrel-development mailing list
> Mongrel-development at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-development
>


-- 
Evan Weaver
Cloudburst, LLC

On Dec 15, 2007 5:01 AM, Evan Weaver <evan at cloudbur.st> wrote:
> I see now that WSGI is a interface spec, not a single server.
What''s
> the main WSGI server? Apache/mod_wsgi? The stdlib
> wsgiref.simple_server looks like Webrick.
>
> CherryPy''s server looks pretty good (
> http://www.cherrypy.org/browser/trunk/cherrypy/wsgiserver/__init__.py
> ). They use a thread pool with a queue. I guess the GIL lets them
> sidestep queue race issues?
>
> Evan
>
>
Is WSGI something worth pursuing? There is a  Nginx WSGI  extension now.

  ~Wayne
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-development/attachments/20071215/33b4f942/attachment-0001.html

Well, it''s a Python interface spec. I don''t think you can run
in any
WSGI container without Python.

Evan

On Dec 15, 2007 9:05 AM, Wayne E. Seguin <wayneeseguin at gmail.com>
wrote:
> On Dec 15, 2007 5:01 AM, Evan Weaver <evan at cloudbur.st> wrote:
>
> > I see now that WSGI is a interface spec, not a single server.
What''s
> > the main WSGI server? Apache/mod_wsgi? The stdlib
> > wsgiref.simple_server looks like Webrick.
> >
> > CherryPy''s server looks pretty good (
> > http://www.cherrypy.org/browser/trunk/cherrypy/wsgiserver/__init__.py
> > ). They use a thread pool with a queue. I guess the GIL lets them
> > sidestep queue race issues?
> >
> > Evan
> >
> >
> >
> >
> >
>
> Is WSGI something worth pursuing? There is a  Nginx WSGI  extension now.
>
>   ~Wayne
>
> _______________________________________________
> Mongrel-development mailing list
> Mongrel-development at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-development
>
>


-- 
Evan Weaver
Cloudburst, LLC

On Dec 14, 2007 6:37 PM, Zed A. Shaw <zedshaw at zedshaw.com>
wrote:
> On Fri, 14 Dec 2007 19:43:02 -0500
> Switching to eventmachine won''t work either because it''s
just a heavy
> layer around select, so you''re still stuck at 1024 max open files.
Not exactly.  The EM reactor currently supports select() and epoll().
Epoll is Linux only, but on Linux systems one can exceed that 1024
descriptor limit that select() has.  I have personally tested it up to
20k open descriptors with no problems.

kqueue() will probably be supported in the near future, as well, and
likely IOCP for Windows systems.  I''m also pushing for Solaris events.


Kirk

On Dec 15, 2007 7:05 AM, Wayne E. Seguin <wayneeseguin at gmail.com>
wrote:
> Is WSGI something worth pursuing? There is a  Nginx WSGI  extension now.
WSGI is just a pre-framework layer.  It standardizes an API for
request and response handling -- all those little bits that frameworks
and servers tend to invent/reinvent it left to do so.  Ruby has a
pretty well implemented WSGI-like layer, called Rack.


Kirk

On Dec 14, 2007 8:10 PM, Luis Lavena <luislavena at gmail.com> wrote:
> Amen on that. I just took the eventmachine code to "see" what is
in
> there and couldn''t understand it well... (ala, a mess?)
The API is well documented.  In the ext/* code, rubymain.cpp is the
extension code itself that glues Ruby to the reactor.

cmain.cpp is the main C++ code file.  It ties everything else together.

Most of the meat of the reactor is in em.cpp.

ed.cpp has the code for the eventable descriptors.  Most of the rest
of it are associated things like support for events on keyboard,
files, pipes; ssl support; epoll support, lightweight concurrency, and
misc other bits.
> Also tried to run the tests and ''see'' them pass on
windows and it crashed badly.
I''ve been looking at the tests.  On my Linux systems there are a
couple bugs in the 0.10.0 release''s tests.  The current gem release
for Windows is 0.8.1, and in addition to the bugs that show up on
Linux, there are some other testing issues to deal with.  I am
addressing them.  Hopefully we''ll have a 0.10.0 gem release for
Windows very soon.


Kirk

Don''t forget this classic paper about events:
http://www.kegel.com/c10k.html

Evan

On Dec 15, 2007 11:38 PM, Kirk Haines <wyhaines at gmail.com>
wrote:
> On Dec 14, 2007 8:10 PM, Luis Lavena <luislavena at gmail.com> wrote:
>
> > Amen on that. I just took the eventmachine code to "see"
what is in
> > there and couldn''t understand it well... (ala, a mess?)
>
> The API is well documented.  In the ext/* code, rubymain.cpp is the
> extension code itself that glues Ruby to the reactor.
>
> cmain.cpp is the main C++ code file.  It ties everything else together.
>
> Most of the meat of the reactor is in em.cpp.
>
> ed.cpp has the code for the eventable descriptors.  Most of the rest
> of it are associated things like support for events on keyboard,
> files, pipes; ssl support; epoll support, lightweight concurrency, and
> misc other bits.
>
> > Also tried to run the tests and ''see'' them pass on
windows and it crashed badly.
>
> I''ve been looking at the tests.  On my Linux systems there are a
> couple bugs in the 0.10.0 release''s tests.  The current gem
release
> for Windows is 0.8.1, and in addition to the bugs that show up on
> Linux, there are some other testing issues to deal with.  I am
> addressing them.  Hopefully we''ll have a 0.10.0 gem release for
> Windows very soon.
>
>
> Kirk
>
> _______________________________________________
> Mongrel-development mailing list
> Mongrel-development at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-development
>


-- 
Evan Weaver
Cloudburst, LLC

On Sat, 15 Dec 2007 20:15:04 -0700
"Kirk Haines" <wyhaines at gmail.com> wrote:
> On Dec 14, 2007 6:37 PM, Zed A. Shaw <zedshaw at zedshaw.com> wrote:
> > On Fri, 14 Dec 2007 19:43:02 -0500
> 
> > Switching to eventmachine won''t work either because
it''s just a heavy
> > layer around select, so you''re still stuck at 1024 max open
files.
> 
> Not exactly.  The EM reactor currently supports select() and epoll().
> Epoll is Linux only, but on Linux systems one can exceed that 1024
> descriptor limit that select() has.  I have personally tested it up to
> 20k open descriptors with no problems.
You know, for years EM has been full of lies so I''ll just take the time
(once again) to verify what''s being claimed before I trust it.
 
> kqueue() will probably be supported in the near future, as well, and
> likely IOCP for Windows systems.  I''m also pushing for Solaris
events.
Riiight, just after all that C++ is cleaned up right?  Right after it
goes "pure ruby"?

Face it, EM is a pile of dogshit and it pains me to see people use it.
Especially when the real solution is for Ruby to finally get its head
out of its ass and stop using select.

-- 
Zed A. Shaw
- Hate: http://savingtheinternetwithhate.com/
- Good: http://www.zedshaw.com/
- Evil: http://yearofevil.com/

On Dec 16, 2007 9:45 AM, Zed A. Shaw <zedshaw at zedshaw.com> wrote:
> You know, for years EM has been full of lies so I''ll just take the
time
> (once again) to verify what''s being claimed before I trust it.
You are so full of vitriol, Zed.  It boggles my mind.  I _use_ the
epoll support.  I''ve hammered it.  Lots of other people do, too.
> Riiight, just after all that C++ is cleaned up right?  Right after it
> goes "pure ruby"?
What''s your definition of "cleaned up"?   The code is pretty
damn
clean.  It''s easily as readable and understandable as the libev code,
and better documented.  So what if it''s C++ instead of C?  And yes,
there is a supported pure ruby version, as well as a pure Java
version.
> Face it, EM is a pile of dogshit and it pains me to see people use it.
> Especially when the real solution is for Ruby to finally get its head
> out of its ass and stop using select.
It performs.  Solidly.  It''s extremely simple to use.  It''s
simple to
work on, too.  Your anger, apart from being your shtick, is just
tired.

And even if, tomorrow, Ruby switched to a model where it could use
select for portability reasons, but would switch between epoll,
kqueue, solaris events, IOCP, etc... depending on platform, it
wouldn''t matter that much.  A very, very large piece of the EM
codebase is supporting code for the event reactor.  A stucture to
build protocols with; support for events other than network events;
lightweight concurrency features, deferables, futures, etc....  The
reactor would change, but everything else would still be applicable.


Kirk Haines

On Sun, 16 Dec 2007 12:39:17 -0700
"Kirk Haines" <wyhaines at gmail.com> wrote:
> On Dec 16, 2007 9:45 AM, Zed A. Shaw <zedshaw at zedshaw.com> wrote:
> 
> > You know, for years EM has been full of lies so I''ll just
take the time
> > (once again) to verify what''s being claimed before I trust
it.
> 
> You are so full of vitriol, Zed.  It boggles my mind.  I _use_ the
> epoll support.  I''ve hammered it.  Lots of other people do, too.
Fair enough, let''s become rational about this then.  First, my
statements aren''t vitriol at all, but instead caution over a project I
found dubious. You however seem to confuse hatred of bad code with
hatred of the people who wrote it.  No such hatred here at all.

And, honestly it''s been a little while since I wandered through the
code to see if it''s improved, so let''s take a new look.

Referring to this post:

http://rubyforge.org/pipermail/eventmachine-talk/2007-December/001296.html

In reply to Frankie Rai who was asking how to use large numbers of FDs
the reply was:

"EM''s reactor uses select(2), which is the most portable way to do
it.
It also supports epoll where available. It would not be difficult to
add kqueue support to the EM reactor. It''s quite similar to epoll and
it probably wouldn''t be a big job."

That says to me that, despite claims, there''s something up or people
are very misinformed.

Taking a look at the code in .cpp files it does have epoll support, I
see this in the epoll.cpp file:

#ifdef HAVE_EPOLL

#include "project.h"

#endif // HAVE_EPOLL

Nice, just include a project.h.  That totally works.  So now someone
has to dig through the .cpp to find out where epoll is being used.
After looking in rubymain.cpp, and then em.cpp I find that it seems to
be implemented like this:

1) epoll manages the sockets it needs to deal with, and then uses a
"LoopBreaker" to keep it''s event loop going.  This
LoopBreaker is an fd
that''s used to break some loop.  Let''s dig further.
2) Further down in _RunSelectOnce we see that it loads the LoopBreaker
and any "eventable descriptors" into the normal select fdread/fdwrite
list and then loops them.
  a) It seems to be calling something called EmSelect, geeeee I wonder
what THAT does.
  b) This EmSelect periodically runs via a Quantum of 90000 us_sec
(hardcoded too btw).
4) Inside this loop EM is basically "layering" the epoll events on top
of the normal ruby select loop and using a timebased polling system.
5) This makes me wonder, how''s it loading these, and is it putting in
fewer than select can handle.  Taking a look further up, there''s a loop
that goes through all Descriptors (wtf, is that a singleton? goodbye
thread safety) and adding any that are read/write ready.  What happens
if more than 1024 are ready?  Don''t see it limiting here.
6) In effect, my statement is correct.  EM layers an epoll event loop
on top of a regular select loop using a timed polling method.  It is a
complicated framework built on select.

Now I''m sure there''s all sorts of great reasons this was done,
and I''m
sure it works, but works and clean/well-written are two totally
different things.  Put all the comments you want, if people go around
saying that "it has epoll" and a quick look shows that even epoll uses
select, then that''s a lie.  Having epoll support and *only* using epoll
are very different.

I will say it has improved vastly in code quality since the 0.8.x
days.  I think if there was some more honest effort in making people
like me happy it''d be worth using.

Let''s take a look at security now.  If you go hit the page for probably
the most secure web server ever:

http://www.and.org/and-httpd/

The man has a $2000 reward if you find a hole in his stuff.  What''s his
special power?  One tool is grep:

egrep
''[^_.>a-zA-Z0-9](str(n?cpy|n?cat|xfrm|n?dup|str|pbrk|tok|_)|stpn?\
cpy|r?index[^.]|a?sn?printf|byte_)'' src/*.c

Yep, that''s right, a single grep can find all sorts of security holes.
I use it, and flag any place I *must* use those functions.  Let''s take
a look at the .cpp:

binder.cpp:     static int index = 0;
binder.cpp:     if ((index >= 1000000) || (seed.length() == 0)) {
binder.cpp:                     sprintf (u2 + (i * 2), "%02x", u1[i]);
binder.cpp:             index = 0;
binder.cpp:     ss << seed << (++index);
em.cpp:                 snprintf (buf, sizeof(buf)-1, "unable to create
epoll descriptor: %s", strerror(errno));
em.cpp:                                 snprintf (buf, sizeof(buf)-1,
"unable to delete epoll event: %s", strerror(errno));
em.cpp:                 snprintf (buf, sizeof(buf)-1, "unable to modify
epoll event: %s", strerror(errno)); em.cpp: strcpy (pun.sun_path,
server); em.cpp:                         snprintf (buf, sizeof(buf)-1,
"unable to add new descriptor: %s", strerror(errno)); em.cpp: strncpy
(s_sun.sun_path, filename, sizeof(s_sun.sun_path)-1);
rubymain.cpp:           snprintf (buf, sizeof(buf)-1, "no popen: %s",
(err?err:"???")); rubymain.cpp:           snprintf (buf,
sizeof(buf)-1,
": %s %s", StringValuePtr(filename),(err?err:"???"));
ssl.cpp:
strcpy (buf, "kittycat");

Nice, much better than when I first looked at the code, but still quite
a few potential attack vectors.  But wait a minute, what''s this
"kittycat" in ssl.cpp?

extern "C" int builtin_passwd_cb (char *buf, int bufsize, int rwflag,
void *userdata) {
	strcpy (buf, "kittycat");
	return 8;
}

WTF, let''s hope nobody''s using that.  Taking a closer look, it
seems
the ssl support has embedded SSL keys and other goodies.  Let''s hope
that''s been reviewed by someone competent.

Finally, let''s take a look at code size.  A simple wc -l on
the .cpp, .h, and .rb files shows the raw lines at around 9819 lines
including comments and the very limited amount of whitespace used in
the project (just hit the enter key once in a while ok?).  Not too bad
for a simple event loop project.

However, the Mongrel source is 3278 lines of .rb|.rl|.h (don''t count .c
since it''s generated).  So, an entire functional web server is almost
1/3rd the size of the whole EM project''s gear.  I know, I know,
"apples
and oranges" but not really.  When evaluating a project for possible
inclusion I first look at code size, then license, then go trolling to
potential warning signs and review the design.  After all that I review
the claims.

So, I''ll upgrade my assessment of EM''s code:

* It is still way too much .cpp for my tastes, and I bet a large
majority of this could be put in .rb.
* The security passwords and keys in ssl.cpp make me wonder.
* Use of the above str*() functions is always bad.  Just remove them
since there''s very good alternatives available.  In the rare cases you
*must* use it, comment the hell out of it.
* It *still* uses select.  I don''t care what anyone says, but if the
epoll event loop then calls select, it is still going to be limited by
select, just not as limited.  Don''t even make the claim it uses epoll
again unless it *only* uses epoll.
* It is still a huge project with dubious design decisions (Descriptors
for *all* descriptors?) that I''d rather avoid.


TWO GARBAGE CATS?

Now, my final question for anyone intersted is:

Who the fuck is  Francis Girardeau and why does Francis Cianfrocca keep
pretending to be him:

http://rubyforge.org/users/garbagecat/
http://rubyforge.org/users/blackhedd/
http://rubyforge.org/scm/?group_id=1555

Notice that blackhedd does most of the commits, but for some reason FC
likes being both people in his communications:

http://www.koders.com/?s=%22Francis+Cianfrocca%22&la=*&li=*&scope=SRPPH8V57VZFKPQF9K8KMLXB7C
http://www.koders.com/default.aspx?s=garbagecat10&btn=&la=*&li=*
http://www.koders.com/default.aspx?s=garbagecat20&btn=&la=*&li=*

Further research shows plenty of hits for an FC living in NYC,
but the only mention of FG is a hospital.

That one single act makes me not trust the project.  It''s been like
this for *years* and nobody really noticed, which is why I worry about
you guys sometimes and why I bring up that you shouldn''t include it in
Mongrel directly.  Everyone is free to use whatever they want, but I''m
also free to report my findings and opinions on the matter.

If Francis has a reasonable explanation as to why he''s pretending to be
two different people on the EventMachine project then I''d like to hear
it. Actually I wouldn''t, it would probably be all lies anyway, just
like the above.

-- 
Zed A. Shaw
- Hate: http://savingtheinternetwithhate.com/
- Good: http://www.zedshaw.com/
- Evil: http://yearofevil.com/

On Dec 16, 2007 2:08 PM, Zed A. Shaw <zedshaw at zedshaw.com>
wrote:
> 1) epoll manages the sockets it needs to deal with, and then uses a
> "LoopBreaker" to keep it''s event loop going.  This
LoopBreaker is an fd
> that''s used to break some loop.  Let''s dig further.
> 2) Further down in _RunSelectOnce we see that it loads the LoopBreaker
> and any "eventable descriptors" into the normal select
fdread/fdwrite
> list and then loops them.
>  a) It seems to be calling something called EmSelect, geeeee I wonder
> what THAT does.
>  b) This EmSelect periodically runs via a Quantum of 90000 us_sec
> (hardcoded too btw).
> 4) Inside this loop EM is basically "layering" the epoll events
on top
> of the normal ruby select loop and using a timebased polling system.
In order for EM to do what couldn''t be done with Ruby/Event, it has to
coexist with the Ruby threads.
EmSelect is ruby_thread_select.

The default Quantum is set to be just a little shorter than the
default scheduling interval for Ruby.  It can be set by the user,
though, to whatever works for one''s application:

EventMachine.set_timer_quantum(99)
> 6) In effect, my statement is correct.  EM layers an epoll event loop
> on top of a regular select loop using a timed polling method.  It is a
> complicated framework built on select.
It''s built so that it works with Ruby, allowing Ruby threads to run.
> Now I''m sure there''s all sorts of great reasons this was
done, and I''m
> sure it works, but works and clean/well-written are two totally
> different things.  Put all the comments you want, if people go around
> saying that "it has epoll" and a quick look shows that even epoll
uses
> select, then that''s a lie.  Having epoll support and *only* using
epoll
> are very different.
It coexists with rb_thread_select.  That''s the crucial missing part of
your analysis.

EM is using epoll, but EM does its work by interleaving with Ruby''s
select based thread management so that ruby threads outside of EM have
a chance to run, without deadlocks.

What EM will need to be doing in the face of the changes in 1.9 is a
discussion, much like the one we are having about Mongrel, that needs
to take place.
> I will say it has improved vastly in code quality since the 0.8.x
> days.  I think if there was some more honest effort in making people
> like me happy it''d be worth using.
And *I* am happy to put time towards addressing real issues.
> I use it, and flag any place I *must* use those functions.  Let''s
take
> a look at the .cpp:
>
> binder.cpp:     static int index = 0;
> binder.cpp:     if ((index >= 1000000) || (seed.length() == 0)) {
> binder.cpp:                     sprintf (u2 + (i * 2), "%02x",
u1[i]);
> binder.cpp:             index = 0;
> binder.cpp:     ss << seed << (++index);
> em.cpp:                 snprintf (buf, sizeof(buf)-1, "unable to
create
> epoll descriptor: %s", strerror(errno));
> em.cpp:                                 snprintf (buf, sizeof(buf)-1,
> "unable to delete epoll event: %s", strerror(errno));
> em.cpp:                 snprintf (buf, sizeof(buf)-1, "unable to
modify
> epoll event: %s", strerror(errno)); em.cpp: strcpy (pun.sun_path,
> server); em.cpp:                         snprintf (buf, sizeof(buf)-1,
> "unable to add new descriptor: %s", strerror(errno)); em.cpp:
strncpy
> (s_sun.sun_path, filename, sizeof(s_sun.sun_path)-1);
> rubymain.cpp:           snprintf (buf, sizeof(buf)-1, "no popen:
%s",
> (err?err:"???")); rubymain.cpp:           snprintf (buf,
sizeof(buf)-1,
> ": %s %s", StringValuePtr(filename),(err?err:"???"));
ssl.cpp:
> strcpy (buf, "kittycat");
Yep.  Some of those things probably deserve attention and maybe
changes.  I''m making a note of them.
> Nice, much better than when I first looked at the code, but still quite
> a few potential attack vectors.  But wait a minute, what''s this
> "kittycat" in ssl.cpp?
The first pass of the SSL support used hardcoded privkey and creds.
Those are still there, but anyone with any sense uses set_tls_params()
to provide their own privkey and cert.
> Finally, let''s take a look at code size.  A simple wc -l on
> the .cpp, .h, and .rb files shows the raw lines at around 9819 lines
> including comments and the very limited amount of whitespace used in
> the project (just hit the enter key once in a while ok?).  Not too bad
> for a simple event loop project.
???  2388 lines in the C++ and Ruby code files are blank lines.  And a
huge number of the remaining lines are comments.  Most of the code is
very comment heavy.
> TWO GARBAGE CATS?
>
> Now, my final question for anyone intersted is:
>
> Who the fuck is  Francis Girardeau and why does Francis Cianfrocca keep
> pretending to be him:
.
.
.
> That one single act makes me not trust the project.  It''s been
like
> this for *years* and nobody really noticed, which is why I worry about
> you guys sometimes and why I bring up that you shouldn''t include
it in
> Mongrel directly.  Everyone is free to use whatever they want, but
I''m
> also free to report my findings and opinions on the matter.
1) Do you honestly think that nobody has ever noticed this before?

2) Who cares?  If the code works; if the license terms are acceptable;
if my patches are accepted and the reasonable changes that I need in
the codebase happen, then I don''t really give a rat''s ass
about the
"Francis Girardeau" name that is on the rubyforge project.  It
doesn''t
appear anywhere in the source, nor the licensing, nor the copyright
notices.  It''s a curiosity, at most.


Kirk Haines

On Sun, 16 Dec 2007 17:00:38 -0700
"Kirk Haines" <wyhaines at gmail.com> wrote:
> In order for EM to do what couldn''t be done with Ruby/Event, it
has to
> coexist with the Ruby threads.
> EmSelect is ruby_thread_select.
And that means it''s *still* using select.  You haven''t
disproven this,
and it''s a shitty design.
 
> > 6) In effect, my statement is correct.  EM layers an epoll event loop
> > on top of a regular select loop using a timed polling method.  It is a
> > complicated framework built on select.
> 
> It''s built so that it works with Ruby, allowing Ruby threads to
run.
However, the statements you made, and other make about EM are that it
somehow sidesteps this.  That''s a lie, and one more reason not to use
the project.  If the statements were more honest like, "it layers an
epoll system on top of ruby''s select so not super fast as epol, but
getting better" then it might be excusable.  However, instead it''s
hyperbole about using epoll, and it''s been this way forever.

The project has claimed Ruby licensed for years and didn''t change the
license statements in source until I mentioned it.  You even missed
that, which you constantly seem to do which is why I don''t trust your
statements about even basic assertions.
> > Now I''m sure there''s all sorts of great reasons this
was done, and I''m
> > sure it works, but works and clean/well-written are two totally
> > different things.  Put all the comments you want, if people go around
> > saying that "it has epoll" and a quick look shows that even
epoll uses
> > select, then that''s a lie.  Having epoll support and *only*
using epoll
> > are very different.
> 
> It coexists with rb_thread_select.  That''s the crucial missing
part of
> your analysis.
> 
> EM is using epoll, but EM does its work by interleaving with
Ruby''s
> select based thread management so that ruby threads outside of EM have
> a chance to run, without deadlocks.
And again Kirk, that''s a lie.  It does not "interleave with
Ruby''s select" which implies that the epoll descriptors are
handled off
to the side and are not impacted by Ruby''s select or that
Ruby''s
select isn''t involved in the processing. If what you say is true, why
is
this code in em.cpp:

	// prepare the sockets for reading and writing
	size_t i;
	for (i = 0; i < Descriptors.size(); i++) {
		EventableDescriptor *ed = Descriptors[i];
		assert (ed);
		int sd = ed->GetSocket();
		assert (sd != INVALID_SOCKET);

		if (ed->SelectForRead())
			FD_SET (sd, &fdreads);
		if (ed->SelectForWrite())
			FD_SET (sd, &fdwrites);

		if (maxsocket < sd)
			maxsocket = sd;
	}

THAT is loading the file descriptors into a god damned read/write set
so they can be processed by Ruby.  THAT means that Ruby''s select
is still processing them.  THAT means you are still limited by
Ruby''s IO, GC, and thread limitations.  

Now Kirk, how is it that you missed this?  I know you''re not the kind
of guy to just bold-faced lie about something, so is it that you''re
having a hard time reading the C++ code?  Obviously all that code is a
problem then.
> > I will say it has improved vastly in code quality since the 0.8.x
> > days.  I think if there was some more honest effort in making people
> > like me happy it''d be worth using.
> 
> And *I* am happy to put time towards addressing real issues.
I''m guessing you mean nothing I''ve brought up with the 10-20
minutes of
effort nobody else seems to want to exert on the project before using
it.
> > I use it, and flag any place I *must* use those functions. 
Let''s take
> > a look at the .cpp:
> >
> > binder.cpp:     static int index = 0;
> > binder.cpp:     if ((index >= 1000000) || (seed.length() == 0)) {
> > binder.cpp:                     sprintf (u2 + (i * 2),
"%02x", u1[i]);
> > binder.cpp:             index = 0;
> > binder.cpp:     ss << seed << (++index);
> > em.cpp:                 snprintf (buf, sizeof(buf)-1, "unable to
create
> > epoll descriptor: %s", strerror(errno));
> > em.cpp:                                 snprintf (buf, sizeof(buf)-1,
> > "unable to delete epoll event: %s", strerror(errno));
> > em.cpp:                 snprintf (buf, sizeof(buf)-1, "unable to
modify
> > epoll event: %s", strerror(errno)); em.cpp: strcpy (pun.sun_path,
> > server); em.cpp:                         snprintf (buf, sizeof(buf)-1,
> > "unable to add new descriptor: %s", strerror(errno));
em.cpp: strncpy
> > (s_sun.sun_path, filename, sizeof(s_sun.sun_path)-1);
> > rubymain.cpp:           snprintf (buf, sizeof(buf)-1, "no popen:
%s",
> > (err?err:"???")); rubymain.cpp:           snprintf (buf,
sizeof(buf)-1,
> > ": %s %s",
StringValuePtr(filename),(err?err:"???")); ssl.cpp:
> > strcpy (buf, "kittycat");
> 
> Yep.  Some of those things probably deserve attention and maybe
> changes.  I''m making a note of them.
What the fuck, so you tell me that I''m vitriolic.  I show that a SINGLE
DAMN GREP finds massive security holes like loading a buffer in a loop
via sprintf (binder.cpp) or the use of a clear text password in the
source as the basis of my assertion. You don''t say, "Damn Zed, you
were
right, that does suck." You say, "Taking note, thanks man."

I''m guessing there''s no way you''d ever admit that EM
isn''t that good,
just what''s available, so I''ll just leave this email as my
last
statement on it until you start pimping it again.
> > Nice, much better than when I first looked at the code, but still
quite
> > a few potential attack vectors.  But wait a minute, what''s
this
> > "kittycat" in ssl.cpp?
> 
> The first pass of the SSL support used hardcoded privkey and creds.
> Those are still there, but anyone with any sense uses set_tls_params()
> to provide their own privkey and cert.
Riiiight, that''s never been an issue.  People *never* accidentally use
defaults for stuff.  Totally secure.
 
> > Finally, let''s take a look at code size.  A simple wc -l on
> > the .cpp, .h, and .rb files shows the raw lines at around 9819 lines
> > including comments and the very limited amount of whitespace used in
> > the project (just hit the enter key once in a while ok?).  Not too bad
> > for a simple event loop project.
> 
> ???  2388 lines in the C++ and Ruby code files are blank lines.  And a
> huge number of the remaining lines are comments.  Most of the code is
> very comment heavy.
It''s still an event loop system that''s larger than most of the
ones
written in pure C.  Hell, libev is damn sexy and has even more
features and is faster than anything cooked up in this mess, and it''s
only 5242 lines of *well commented* C code.
 
> > TWO GARBAGE CATS?
> 
> 1) Do you honestly think that nobody has ever noticed this before?
And this is what makes me cry: people don''t notice it.  They just go
about their business missing that point, which leads to...
 
> 2) Who cares?  If the code works; if the license terms are acceptable;
> if my patches are accepted and the reasonable changes that I need in
> the codebase happen, then I don''t really give a rat''s ass
about the
> "Francis Girardeau" name that is on the rubyforge project.  It
doesn''t
> appear anywhere in the source, nor the licensing, nor the copyright
> notices.  It''s a curiosity, at most.
Not in my book.  That''s a sign of stupidity at the least and
con-artistry at the most.  Either way its explained it points to
something weird about the individual that I''d rather avoid.  My
experiences has also showed being cautious when it comes to who''s
software you inject into your own is the best thing to do.  Anything
else is just bad business.

Now, you and anyone else can keep on using the damn thing--I care
less--but when you keep mentioning it for Mongrel it kind of pisses me
off. Do what you want outside Mongrel, but all of the above reasons
(and then some) make me not want to have it within 10 feet of the
Mongrel project.

So Please Kirk, quit mentioning it, quit mentioning that it is "faster
than Ruby''s select" which is a lie, and quit saying that my
objections
to it are based on anything other than the facts I find right in the
source or online.  It''s not vitriol or opinion, it''s based on
a review
of the source that you obviously aren''t doing.

-- 
Zed A. Shaw
- Hate: http://savingtheinternetwithhate.com/
- Good: http://www.zedshaw.com/
- Evil: http://yearofevil.com/

On Dec 18, 2007 3:46 AM, Zed A. Shaw <zedshaw at zedshaw.com>
wrote:
> On Sun, 16 Dec 2007 17:00:38 -0700
> "Kirk Haines" <wyhaines at gmail.com> wrote:
>
> > In order for EM to do what couldn''t be done with Ruby/Event,
it has to
> > coexist with the Ruby threads.
> > EmSelect is ruby_thread_select.
>
> And that means it''s *still* using select.  You haven''t
disproven this,
> and it''s a shitty design.
LOL.  No, that means that it is written so that it works with Ruby''s
design, to permit Ruby''s threads to run without deadlocking.
Specifically, EM''s epoll loop processes epoll events, and then it runs
rb_thread_select with no descriptors in order to give Ruby''s threads a
chance to run.  It''s a good design for the requirements, which involve
being about to work with Ruby''s threading mechanism.
> However, the statements you made, and other make about EM are that it
> somehow sidesteps this.  That''s a lie, and one more reason not to
use
> the project.  If the statements were more honest like, "it layers an
> epoll system on top of ruby''s select so not super fast as epol,
but
> getting better" then it might be excusable.  However, instead
it''s
> hyperbole about using epoll, and it''s been this way forever.
*I* have tested my apps with up to 20000 open, active simultaneous
connections on Linux platforms.
You''re calling me a liar.  Don''t do that.

As for your other assertion, EM supports Ruby thread interoperability
because it has to.  It''s as simple as that.  Your own experience with
Ruby/Event''s failure demonstrates why it has to.
> The project has claimed Ruby licensed for years and didn''t change
the
> license statements in source until I mentioned it.  You even missed
> that, which you constantly seem to do which is why I don''t trust
your
> statements about even basic assertions.
Again, you go on with the unsupported statements that I missed things.
I didn''t.

EventMachine has existed as a project since April of 2006 (i.e. less
than 2 years), and it has always been dual licensed.  In August of
2006 you pointed out some incorrect source file headings, and they
were fixed.
> > It coexists with rb_thread_select.  That''s the crucial
missing part of
> > your analysis.
> >
> > EM is using epoll, but EM does its work by interleaving with
Ruby''s
> > select based thread management so that ruby threads outside of EM have
> > a chance to run, without deadlocks.
>
> And again Kirk, that''s a lie.  It does not "interleave with
> Ruby''s select" which implies that the epoll descriptors are
handled off
> to the side and are not impacted by Ruby''s select or that
Ruby''s
> select isn''t involved in the processing. If what you say is true,
why is
> this code in em.cpp:
>
>        // prepare the sockets for reading and writing
>        size_t i;
>        for (i = 0; i < Descriptors.size(); i++) {
>                EventableDescriptor *ed = Descriptors[i];
>                assert (ed);
>                int sd = ed->GetSocket();
>                assert (sd != INVALID_SOCKET);
>
>                if (ed->SelectForRead())
>                        FD_SET (sd, &fdreads);
>                if (ed->SelectForWrite())
>                        FD_SET (sd, &fdwrites);
>
>                if (maxsocket < sd)
>                        maxsocket = sd;
>        }
>
> THAT is loading the file descriptors into a god damned read/write set
> so they can be processed by Ruby.  THAT means that Ruby''s select
> is still processing them.  THAT means you are still limited by
> Ruby''s IO, GC, and thread limitations.
>
> Now Kirk, how is it that you missed this?  I know you''re not the
kind
> of guy to just bold-faced lie about something, so is it that
you''re
> having a hard time reading the C++ code?  Obviously all that code is a
> problem then.
I didn''t miss anythiing, here, Zed, nor did I lie.  Don''t
misunderstand me, here.  Desist from that libelous assertion.  I don''t
appreciate your defamation.


/************************
EventMachine_t::_RunOnce
************************/

bool EventMachine_t::_RunOnce()
{
 if (bEpoll)
   return _RunEpollOnce();
 else
   return _RunSelectOnce();
}

This dispatches to the appropriate handler depending on whether epoll
is being used or not.

The code that you quoted is from _RunSelectOnce().  As you can see
above, that code is used only if one is using select.  If one is using
epoll, _RunEpollOnce() is called; select() isn''t used.  Your assertion
is incorrect.

And again, to be quite clear, quit calling me a liar.  I did not
misunderstand the code.  You very clearly did.  And *I* have not
intentionally misrepresented anything.
> > And *I* am happy to put time towards addressing real issues.
>
> I''m guessing you mean nothing I''ve brought up with the
10-20 minutes of
> effort nobody else seems to want to exert on the project before using
> it.
No.  It means that I am right here, listening to you, and am happy to
look at the issues and see if any of them are real problems.  That is,
I will look at them and analyze them instead of just taking your word
of authority on the matter.  If my analysis agrees with yours, then I
will fix them.
> What the fuck, so you tell me that I''m vitriolic.  I show that a
SINGLE
> DAMN GREP finds massive security holes like loading a buffer in a loop
> via sprintf (binder.cpp) or the use of a clear text password in the
> source as the basis of my assertion. You don''t say, "Damn
Zed, you were
> right, that does suck." You say, "Taking note, thanks man."
No, you showed that a single grep finds a number of things without context.

One of the things that grep looks for is the use of a variable named
''index''.  That''s hardly a security hole.  None of
those lines would
have showed up in the magic grep had a different variable name been
used in that place.

So what I am saying is that those lines deserve to be examined in
context, and the use of the dangerous str* functions needs to be
looked at closely, because, as you said, there usually are safer
functions to use.

I''m saying that there could well be room for improvement there.

What I am not doing is getting aggitated, or excited, or fearful, or
antagonistic.  In that place you pointed out some things that need to
be looked at more closely, and I will be doing so.

Speaking of looking closely at things, regarding the sprintf usage in
binder (which comprised a large number of the lines that your grep
found): That''s not a security violation. The value is being copied to
an automatic array with a carefully-computed size, and the data is not
coming from user input but is generated internally.
> I''m guessing there''s no way you''d ever admit
that EM isn''t that good,
> just what''s available, so I''ll just leave this email as
my last
> statement on it until you start pimping it again.
I haven''t pimped it for Mongrel.  I''ve _never_ said that it
should be
used in Mongrel core.  I''ve resisted that notion, in fact.  I just
resist your defamation of EM, and of me, for using it.  I would like
an easier, cleaner way for things to swap out the core request
handling pieces of Mongrel, but if Mongrel were mine, I''d work very
hard to make sure the _core_ of Mongrel is as simple, and as base-ruby
as possible.  For 1.9, that means one of two things, I think.  Whether
this is fibers, a thread pool that plucks things off the accept queue,
or some combination of the two, I don''t know.  On Linux and Windows,
system threads per request are going to be slow.  On Solaris, though,
it stands a real chance of performing all right because Solaris
threads are very lightweight.

And since, as you have often pointed out, Mongrel is a library, and a
LOT of people use it in a lot of different ways, were it mine, I''d be
embracing that fact and making sure that it continues to be a flexible
platform that people can adopt to their own needs.
> > > Nice, much better than when I first looked at the code, but still
quite
> > > a few potential attack vectors.  But wait a minute,
what''s this
> > > "kittycat" in ssl.cpp?
> >
> > The first pass of the SSL support used hardcoded privkey and creds.
> > Those are still there, but anyone with any sense uses set_tls_params()
> > to provide their own privkey and cert.
>
> Riiiight, that''s never been an issue.  People *never* accidentally
use
> defaults for stuff.  Totally secure.
"kittycat" is dead code.  It''s a placeholder for an
encryption key in
case users want to add one themselves.  It was originally there for
decrypting the default private key.  Since the private key right there
in open source code for all to see, there''s no point in encrypting it.
 Anyone who understands how X509 works would not use the default
credential for authentication, but only for encryption.  So in the end
it comes down to doing a little RTFM and knowing what you are doing.
The user can make the choices and is responsible for the outcome of
those choices.
> Now, you and anyone else can keep on using the damn thing--I care
> less--but when you keep mentioning it for Mongrel it kind of pisses me
> off. Do what you want outside Mongrel, but all of the above reasons
> (and then some) make me not want to have it within 10 feet of the
> Mongrel project.
As I mentioned above, I have never advocated using it for Mongrel
itself.  I have said as much to Evan, Wayne, Luis, and Fillipe during
IRC conversations. I have only advocated for EM itself in this forum
because you persist with the shouting, name calling, and inaccuracies
in your statements.
> So Please Kirk, quit mentioning it, quit mentioning that it is "faster
> than Ruby''s select" which is a lie, and quit saying that my
objections
> to it are based on anything other than the facts I find right in the
> source or online.  It''s not vitriol or opinion, it''s
based on a review
> of the source that you obviously aren''t doing.

/************************
EventMachine_t::_RunOnce
************************/

bool EventMachine_t::_RunOnce()
{
 if (bEpoll)
   return _RunEpollOnce();
 else
   return _RunSelectOnce();
}



Kirk Haines