Eventmachine talk - Jun 2007 - Some questions

Hi EM Devs!

I''ve just discovered EventMachine (and Ruby, come to that!) and
I''m
excited about getting started with a new project using them
(redux.rubyforge.org).

I have some questions... Being new to EM and Ruby, the questions may
be naive, so be kind!   =0)

I downloaded eventmachine-0.7.2 and found some small issues:

   There is a file with version 0.7.3 in it:
lib/eventmachine_version.rb; but the release notes are for 0.7.1! I
suppose it averages out!  =0)
   The tests seem to fail (for example, matching HTTP/1.1 instead of
1.0; line endings having an extra character, at least on Linux/Ruby
1.8.2)
   The POST content type in the http client test comes out as
application/octet-stream not text/plain as sent.

I tried running both manually and through rake and couldn''t get the
tests to pass. Like I said, I''m on Ruby 1.8.2 and Ubuntu.

More importantly, I was wondering if it would be possible to make some
small changes to the API:

- I would like to get a callback on writable, or at least when the
buffers are emptied, so I can decide whether to close the socket or
carry on reading (e.g. for HTTP 1.1 persistent connections and
pipeline implementation).  Also, I may not  want to fill up the write
buffers without being notified that the pipes are blocked and
nothing''s being sent.

- Most importantly of all, I''d like to be able to set non-blocking on
the main loop, so I can have single-threaded implementation of long
running processes, that do a chunk of work, then pop back to check if
any fds are active. In python, I go ''asyncore.loop(timeout=t,
count=1)'', for example, setting timeout to zero for non-blocking.
(I''m
porting from Python).

These changes would give my project a good foundation - you''d save me
a lot of work writing my own select loop, etc! Like you, I''ve been
doing this for years and years, just not in Ruby.

Anyway, EM looks like really good work and very promising for the kind
of applications people like me are looking at building these days,
with performance and the Ruby runtime being such an issue. I really
hope I can use it - and perhaps help make it even better!

Cheers!

Duncan Cragg

_________________________________
http://duncan-cragg.org/blog/

On 6/12/07, Duncan Cragg <ruby at cilux.org> wrote:
Thanks for the kind words, and welcome.

> I downloaded eventmachine-0.7.2 and found some small issues:
>
>    There is a file with version 0.7.3 in it:
> lib/eventmachine_version.rb; but the release notes are for 0.7.1! I
> suppose it averages out!  =0)

Yeah, sometimes we''re not disciplined enough about the version numbers.
EM
is production-stable so by rights it should have a version number that
starts with 1, but there you are. In a few days, you can expect to see the
0.8.0 release (with epoll) and soon afterward a 0.9.0 (with some Erlang-like
distributed computing features).


   The tests seem to fail (for example, matching HTTP/1.1 instead
of
> 1.0; line endings having an extra character, at least on Linux/Ruby
> 1.8.2)

The tests are definitely not all working! That''s something
I''ve been meaning
to clean out. The project Rakefile actually defines a batch of testing
subtasks that do work, but you''re right. The main testing task
shouldn''t
have stupid mistakes.

Ruby 1.8.2: I''m not sure you''ll have a lot of luck with EM on
such an old
Ruby version (but I could be wrong). For one thing, EM uses some nonblocking
I/O features that the Ruby core developers added about a year ago
specifically per our request.

   The POST content type in the http client test comes out
as
> application/octet-stream not text/plain as sent.

I''ll have a look at that.

> More importantly, I was wondering if it would be possible to make some
> small changes to the API:
>
> - I would like to get a callback on writable, or at least when the
> buffers are emptied, so I can decide whether to close the socket or
> carry on reading (e.g. for HTTP 1.1 persistent connections and
> pipeline implementation).  Also, I may not  want to fill up the write
> buffers without being notified that the pipes are blocked and
> nothing''s being sent.

Look at EventMachine::Connection#get_outbound_data_size. It tells you the
amount of data that the connection has scheduled for transmission to the
kernel buffers, so it''s not exactly what you asked for. But it may be
just
as useful. I recently used it in an LDAP server to break up the transmission
of search-results that might be hundreds of megabytes long, to interleave
processing of other requests. Very effective.

- Most importantly of all, I''d like to be able to set non-blocking
on
> the main loop, so I can have single-threaded implementation of long
> running processes, that do a chunk of work, then pop back to check if
> any fds are active. In python, I go ''asyncore.loop(timeout=t,
> count=1)'', for example, setting timeout to zero for non-blocking.
(I''m
> porting from Python).

Can you achieve the same thing with an EM timer?
EM.run {
    EM.add_periodic_timer(5) {
        # do some work
    }
}
This will execute the block passed to #add_periodic_timer every five
seconds. There is some new syntax that makes these timers cancellable, too.
Are you porting from straight Python or from Twisted? I''d love to hear
what
the relative performance is like.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070612/ceb445cd/attachment.html

Just thought of something:
>
>
> - Most importantly of all, I''d like to be able to set non-blocking
on
> the main loop, so I can have single-threaded implementation of long
> running processes, that do a chunk of work, then pop back to check if
> any fds are active. In python, I go ''asyncore.loop(timeout=t,
> count=1)'', for example, setting timeout to zero for non-blocking.
(I''m
> porting from Python).


EM.run {
    EM.add_timer(5) {
        EM.stop
    }
}
$>.puts "HERE"

will end the loop after 5 seconds, and you can restart it whenever you want.
Was that what you needed?
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070612/819cba62/attachment.html

Thanks for your quick and helpful response!
> > There is a file with version 0.7.3 in it:
> > lib/eventmachine_version.rb; but the release notes are for 0.7.1! I
Oops - actually I had a subversion repository knocking around, with
the 0.7.3 number in it! Sorry. The release notes /are/ frozen in time,
though!  =0)
> Ruby 1.8.2: I''m not sure you''ll have a lot of luck with
EM on such an old
> Ruby version
I''m on 1.9 now - thanks for the tip!
> > - Most importantly of all, I''d like to be able to set
non-blocking on
> > the main loop, so I can have single-threaded implementation of long
> > running processes, that do a chunk of work, then pop back to check if
> > any fds are active. In python, I go ''asyncore.loop(timeout=t,
> > count=1)'', for example, setting timeout to zero for
non-blocking. (I''m
> > porting from Python).
>
> Can you achieve the same thing with an EM timer?
Sorry - I should have come clean with what I was /really/ trying to do!

My potentially long-running processes are actually sending events
around on queues. I want an event loop that knows it has more queue
work to do, and runs the select() call non-blocking, so it can get
back to work on the queues if there''s no fd activity to handle. Once
the queues are empty, it can go back to quantum-timeout blocking mode.

It''s not urgent any more, as I''ve switched to pure-ruby for
now, and
patched the loop like this:

# -------------------------------------------------------------
# Override core loop in pr_eventmachine.rb to insert a callback
# Only works in Pure Ruby, then: don''t install rubyeventmachine.so
# i.e., /usr/local/lib/site_ruby/1.9/i486-linux/rubyeventmachine.so

module EventMachine

  def EventMachine::set_loop_callback &block
      @loop_callback = block
  end

  def EventMachine::run_loop_callback
      @loop_callback.call if @loop_callback
  end

  class Reactor
    def run
      @running = true
      open_loopbreaker
      loop {
        break if @stop_scheduled
        run_timers
        break if @stop_scheduled
        EventMachine::run_loop_callback
        crank_selectables
      }
      close_loopbreaker
      @selectables.each {|k, io| io.close}
      @selectables.clear
      @running = false
    end
  end
end

# -----------------------------------------------

I then do this in my user code:

# -----------------------------------------------
    NON_BLOCKING = 0
    BLOCKING = 100

    def run_loop
        EventMachine.run {
            init_modules
            EventMachine.set_loop_callback { run_queues_and_set_quantum }
        }
    end

    def run_queues_and_set_quantum
        more=run_queues
        t = if more
                NON_BLOCKING
            else
                BLOCKING
            end
        EventMachine.set_quantum(t)
    end
# -----------------------------------------------

So I''m happy for now, but will miss epoll()!!  =0)
> Are you porting from straight Python or from Twisted? I''d love to
hear what
> the relative performance is like.
So would I! I''ll let you know when I''m done! I got some nice
numbers
from httperf for my Python version.

I didn''t like Twisted. At all. I''m porting from simple Python
asyncore, which worked great.

My current issues are in the LineAndTextProtocol for my HTTP server -
indeed, the http client tests failed for the same issue: in Linux, you
get an extra line ending character (the CR from CRLF, I presume). I''m
dibbling around in BufferedTokenizer right now trying to find out
where the bug is. Not knowing Ruby is holding me back a bit (as is the
fact that I''m only working on it on the train to and from work!!).
I''d
appreciate any help you can offer! =0)

Did I see mention on this list of an already-written HTTP server using
EM? I could at least use some of it.

Oh - do I need to change the licence on Redux to Ruby in order to use
your library? Would you prefer I use the GPL?

So, thanks again for the reply!  And thanks again for EM!  =0)

Duncan

_________________________________
http://duncan-cragg.org/blog/

On 6/13/07, Duncan Cragg <ruby at cilux.org>
wrote:
>
>   Oops - actually I had a subversion repository knocking around, with
> the 0.7.3 number in it! Sorry. The release notes /are/ frozen in time,
> though!  =0)

I updated the release notes so they''re now reasonably close to reality.

>
> Sorry - I should have come clean with what I was /really/ trying to do!
>
> My potentially long-running processes are actually sending events
> around on queues. I want an event loop that knows it has more queue
> work to do, and runs the select() call non-blocking, so it can get
> back to work on the queues if there''s no fd activity to handle.
Once
> the queues are empty, it can go back to quantum-timeout blocking mode.

Ok, here''s my suggestion for this: look at EventMachine#next_tick. This
is a
method which takes either a Proc or a block, and it will execute the block
after the next pass through the reactor core, WITHOUT blocking on the timer
quantum. I think this is exactly what you want. You can schedule as many of
these as you want. If you have a long-running operation scheduled with
#next_tick, it can do part of its work, then schedule the rest with another
#next_tick call, and then return. That will allow the reactor to keep doing
its thing.

#next_tick was designed to support user-defined events, which will appear
when I finish the Erlang-like features I''ve been working on.



It''s not urgent any more, as I''ve switched to pure-ruby for
now, and
> patched the loop like this:

It''s good to know that the pure-ruby code works as expected, but you
probably shouldn''t get too used to it. It''s definitely behind
in terms of
features. Rather than continue to maintain the pure-ruby version, I''m
probably going to write a pure-Java EM that will work with JRuby. (The
current C++ EM will be fully supported indefinitely.)

Is there anyone else out there who uses the pure-Ruby EM? If so, I can
change my mind about whether to continue supporting it.

>
> So I''m happy for now, but will miss epoll()!!  =0)

Epoll seems to be working great. I''m pretty happy with it.
>
>
>
> My current issues are in the LineAndTextProtocol for my HTTP server -
> indeed, the http client tests failed for the same issue: in Linux, you
> get an extra line ending character (the CR from CRLF, I presume).
I''m
> dibbling around in BufferedTokenizer right now trying to find out
> where the bug is. Not knowing Ruby is holding me back a bit (as is the
> fact that I''m only working on it on the train to and from work!!).
I''d
> appreciate any help you can offer! =0)
>
> Did I see mention on this list of an already-written HTTP server using
> EM? I could at least use some of it.

Look on the EM Rubyforge page: there is another distro called
eventmachine_httpserver. This is a production-quality HTTP parser that calls
into user-written Ruby code to fulfill web requests.

There also is a complete Web-development framework (currently called
"unicycle") that builds on the published HTTP parser. This product
comes out
of a long chain of work that I started to address problems with Rails.
It''s
production-quality, but it''s not published yet because it needs
documentation.

Oh - do I need to change the licence on Redux to Ruby in order to
use
> your library? Would you prefer I use the GPL?


I assume Redux is something you''re writing. What license do you
currently
release it under? You can use either GPL or Ruby-license with EM, it''s
your
choice.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070613/c14b3b09/attachment.html

From: "Duncan Cragg" <ruby at cilux.org>
>
> My current issues are in the LineAndTextProtocol for my HTTP server -
> indeed, the http client tests failed for the same issue: in Linux, you
> get an extra line ending character (the CR from CRLF, I presume).
I ran into this as well, because I needed to accept lines both
ending in LF and CRLF.  So I went with LF as the delimeter.
This meant the lines received as CRLF were still passed to me
with a trailing CR.

What I did was just call String#chomp on the lines passed to
me in receive_line.

(Unlike String#chop, String#chomp is smart enough to conditionally
remove a trailing line-ending sequence only if it exists.)


Hope this helps,

Bill

If this undesired behavior you can change the default token used by
BufferedTokenizer to \r\n

I believe I was trying to duplicate the previous behavior of
LineAndTextProtocol, which tokenized on \n

--
Tony Arcieri
ClickCaster, Inc.
tony at clickcaster.com

On 6/13/07, Bill Kelly <billk at cts.com> wrote:
>
> From: "Duncan Cragg" <ruby at cilux.org>
> >
> > My current issues are in the LineAndTextProtocol for my HTTP server -
> > indeed, the http client tests failed for the same issue: in Linux, you
> > get an extra line ending character (the CR from CRLF, I presume).
>
> I ran into this as well, because I needed to accept lines both
> ending in LF and CRLF.  So I went with LF as the delimeter.
> This meant the lines received as CRLF were still passed to me
> with a trailing CR.
>
> What I did was just call String#chomp on the lines passed to
> me in receive_line.
>
> (Unlike String#chop, String#chomp is smart enough to conditionally
> remove a trailing line-ending sequence only if it exists.)
>
>
> Hope this helps,
>
> Bill
>
>
> _______________________________________________
> Eventmachine-talk mailing list
> Eventmachine-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/eventmachine-talk
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070613/f21ffe30/attachment.html

From: Tony Arcieri
>
> If this undesired behavior you can change the default token used
> by BufferedTokenizer to \r\n
I wasn''t sure about that, because I needed to handle lines sent
to me in both LF and CRLF format.

I would have thought I''d need to set the token to a regexp like
/\r?\n/

So I just went with LF and chomp... :)


Regards,

Bill

Perhaps the best thing to do would be to allow the delimiter token to be
specified when LineAndTextProtocol is initialized, with "\n" as the
default

- Tony

On 6/13/07, Bill Kelly <billk at cts.com> wrote:
>
>
> From: Tony Arcieri
> >
> > If this undesired behavior you can change the default token used
> > by BufferedTokenizer to \r\n
>
> I wasn''t sure about that, because I needed to handle lines sent
> to me in both LF and CRLF format.
>
> I would have thought I''d need to set the token to a regexp like
> /\r?\n/
>
> So I just went with LF and chomp... :)
>
>
> Regards,
>
> Bill
>
>
> _______________________________________________
> Eventmachine-talk mailing list
> Eventmachine-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/eventmachine-talk
>


--
Tony Arcieri
ClickCaster, Inc.
tony at clickcaster.com
720-227-0129 ext. 202
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070613/340d97f6/attachment.html

On 6/13/07, Tony Arcieri <tony at clickcaster.com>
wrote:
>
> If this undesired behavior you can change the default token used by
> BufferedTokenizer to \r\n
>
> I believe I was trying to duplicate the previous behavior of
> LineAndTextProtocol, which tokenized on \n


I can''t remember now but I''d be very surprised if the original
didn''t handle
either CRLF or LF. If it tokenized on LF, then it probably suppressed CR (if
present) somewhere else. If it didn''t, I''d consider that a
bug.

It would be great if you can make sure the current implementation can handle
either CRLF or LF, even intermixed in the same stream (which some buggy code
out there does).
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070613/69835cab/attachment.html

If that''s the desired behavior (to be handle buggy clients that
don''t always
send CRLF) then the above-suggested solution of a simple chomp will
suffice.  This will also eliminate all LF vs CRLF confusion

- Tony

On 6/13/07, Francis Cianfrocca <garbagecat10 at gmail.com>
wrote:
>
> On 6/13/07, Tony Arcieri <tony at clickcaster.com> wrote:
> >
> > If this undesired behavior you can change the default token used by
> > BufferedTokenizer to \r\n
> >
> > I believe I was trying to duplicate the previous behavior of
> > LineAndTextProtocol, which tokenized on \n
>
>
>
> I can''t remember now but I''d be very surprised if the
original didn''t
> handle either CRLF or LF. If it tokenized on LF, then it probably
suppressed
> CR (if present) somewhere else. If it didn''t, I''d
consider that a bug.
>
> It would be great if you can make sure the current implementation can
> handle either CRLF or LF, even intermixed in the same stream (which some
> buggy code out there does).
>
>
> _______________________________________________
> Eventmachine-talk mailing list
> Eventmachine-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/eventmachine-talk
>


--
Tony Arcieri
ClickCaster, Inc.
tony at clickcaster.com
720-227-0129 ext. 202
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070613/5b1d2d4c/attachment-0001.html

Is that something you can do in your buftok-based implementation?

On 6/13/07, Tony Arcieri <tony at clickcaster.com>
wrote:
>
> If that''s the desired behavior (to be handle buggy clients that
don''t
> always send CRLF) then the above-suggested solution of a simple chomp will
> suffice.  This will also eliminate all LF vs CRLF confusion
>
> - Tony
>
> On 6/13/07, Francis Cianfrocca <garbagecat10 at gmail.com> wrote:
>
> > On 6/13/07, Tony Arcieri <tony at clickcaster.com > wrote:
> > >
> > > If this undesired behavior you can change the default token used
by
> > > BufferedTokenizer to \r\n
> > >
> > > I believe I was trying to duplicate the previous behavior of
> > > LineAndTextProtocol, which tokenized on \n
> >
> >
> >
> > I can''t remember now but I''d be very surprised if
the original didn''t
> > handle either CRLF or LF. If it tokenized on LF, then it probably
suppressed
> > CR (if present) somewhere else. If it didn''t, I''d
consider that a bug.
> >
> > It would be great if you can make sure the current implementation can
> > handle either CRLF or LF, even intermixed in the same stream (which
some
> > buggy code out there does).
> >
> >
> > _______________________________________________
> > Eventmachine-talk mailing list
> > Eventmachine-talk at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/eventmachine-talk
> >
>
>
>
> --
> Tony Arcieri
> ClickCaster, Inc.
> tony at clickcaster.com
> 720-227-0129 ext. 202
>
> _______________________________________________
> Eventmachine-talk mailing list
> Eventmachine-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/eventmachine-talk
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070613/e6a48867/attachment.html

Yes, that can be accomplished here (line 42 of line_and_text.rb):

@lpb_buffer.extract(data).each { |line| receive_line(line) if
respond_to?(:receive_line) }

would become:

@lpb_buffer.extract(data).map(&:chomp).each do |line|
  receive_line(line) if respond_to?(:receive_line)
end

Or something to that effect (I''m guessing the procinator isn''t
available, so
perhaps .map { |s| s.chomp }.each)

I''m at revision 379... should I go ahead and commit that change?  I
don''t
forsee any unintended consequences...

- Tony

On 6/13/07, Francis Cianfrocca <garbagecat10 at gmail.com>
wrote:
>
> Is that something you can do in your buftok-based implementation?
>
> On 6/13/07, Tony Arcieri <tony at clickcaster.com > wrote:
> >
> > If that''s the desired behavior (to be handle buggy clients
that don''t
> > always send CRLF) then the above-suggested solution of a simple chomp
will
> > suffice.  This will also eliminate all LF vs CRLF confusion
> >
> > - Tony
> >
> > On 6/13/07, Francis Cianfrocca < garbagecat10 at gmail.com>
wrote:
> >
> > > On 6/13/07, Tony Arcieri <tony at clickcaster.com > wrote:
> > > >
> > > > If this undesired behavior you can change the default token
used by
> > > > BufferedTokenizer to \r\n
> > > >
> > > > I believe I was trying to duplicate the previous behavior of
> > > > LineAndTextProtocol, which tokenized on \n
> > >
> > >
> > >
> > > I can''t remember now but I''d be very surprised
if the original didn''t
> > > handle either CRLF or LF. If it tokenized on LF, then it probably
suppressed
> > > CR (if present) somewhere else. If it didn''t,
I''d consider that a bug.
> > >
> > > It would be great if you can make sure the current implementation
can
> > > handle either CRLF or LF, even intermixed in the same stream
(which some
> > > buggy code out there does).
> > >
> > >
> > > _______________________________________________
> > > Eventmachine-talk mailing list
> > > Eventmachine-talk at rubyforge.org
> > > http://rubyforge.org/mailman/listinfo/eventmachine-talk
> > >
> >
> >
> >
> > --
> > Tony Arcieri
> > ClickCaster, Inc.
> > tony at clickcaster.com
> > 720-227-0129 ext. 202
> >
> > _______________________________________________
> > Eventmachine-talk mailing list
> > Eventmachine-talk at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/eventmachine-talk
> >
>
>
> _______________________________________________
> Eventmachine-talk mailing list
> Eventmachine-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/eventmachine-talk
>


--
Tony Arcieri
ClickCaster, Inc.
tony at clickcaster.com
720-227-0129 ext. 202
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070613/75b1bc5c/attachment.html

Go for it.

On 6/13/07, Tony Arcieri <tony at clickcaster.com>
wrote:
>
> Yes, that can be accomplished here (line 42 of line_and_text.rb):
>
> @lpb_buffer.extract(data).each { |line| receive_line(line) if
> respond_to?(:receive_line) }
>
> would become:
>
> @lpb_buffer.extract(data).map(&:chomp).each do |line|
>   receive_line(line) if respond_to?(:receive_line)
> end
>
> Or something to that effect (I''m guessing the procinator
isn''t available,
> so perhaps .map { |s| s.chomp }.each)
>
> I''m at revision 379... should I go ahead and commit that change? 
I don''t
> forsee any unintended consequences...
>
> - Tony
>
> On 6/13/07, Francis Cianfrocca <garbagecat10 at gmail.com> wrote:
> >
> > Is that something you can do in your buftok-based implementation?
> >
> > On 6/13/07, Tony Arcieri < tony at clickcaster.com > wrote:
> > >
> > > If that''s the desired behavior (to be handle buggy
clients that don''t
> > > always send CRLF) then the above-suggested solution of a simple
chomp will
> > > suffice.  This will also eliminate all LF vs CRLF confusion
> > >
> > > - Tony
> > >
> > > On 6/13/07, Francis Cianfrocca < garbagecat10 at gmail.com>
wrote:
> > >
> > > > On 6/13/07, Tony Arcieri <tony at clickcaster.com >
wrote:
> > > > >
> > > > > If this undesired behavior you can change the default
token used
> > > > > by BufferedTokenizer to \r\n
> > > > >
> > > > > I believe I was trying to duplicate the previous
behavior of
> > > > > LineAndTextProtocol, which tokenized on \n
> > > >
> > > >
> > > >
> > > > I can''t remember now but I''d be very
surprised if the original
> > > > didn''t handle either CRLF or LF. If it tokenized on
LF, then it probably
> > > > suppressed CR (if present) somewhere else. If it
didn''t, I''d consider that a
> > > > bug.
> > > >
> > > > It would be great if you can make sure the current
implementation
> > > > can handle either CRLF or LF, even intermixed in the same
stream (which some
> > > > buggy code out there does).
> > > >
> > > >
> > > > _______________________________________________
> > > > Eventmachine-talk mailing list
> > > > Eventmachine-talk at rubyforge.org
> > > > http://rubyforge.org/mailman/listinfo/eventmachine-talk
> > > >
> > >
> > >
> > >
> > > --
> > > Tony Arcieri
> > > ClickCaster, Inc.
> > > tony at clickcaster.com
> > > 720-227-0129 ext. 202
> > >
> > > _______________________________________________
> > > Eventmachine-talk mailing list
> > > Eventmachine-talk at rubyforge.org
> > > http://rubyforge.org/mailman/listinfo/eventmachine-talk
> > >
> >
> >
> > _______________________________________________
> > Eventmachine-talk mailing list
> > Eventmachine-talk at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/eventmachine-talk
> >
>
>
>
> --
> Tony Arcieri
> ClickCaster, Inc.
> tony at clickcaster.com
> 720-227-0129 ext. 202
>
> _______________________________________________
> Eventmachine-talk mailing list
> Eventmachine-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/eventmachine-talk
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070613/100e7847/attachment.html

Cool... ended up going with:

@lpb_buffer.extract(data).each do |line|
  receive_line(line.chomp) if respond_to?(:receive_line)
end

On 6/13/07, Francis Cianfrocca <garbagecat10 at gmail.com>
wrote:
>
> Go for it.
>
> On 6/13/07, Tony Arcieri <tony at clickcaster.com> wrote:
> >
> > Yes, that can be accomplished here (line 42 of line_and_text.rb):
> >
> > @lpb_buffer.extract(data).each { |line| receive_line(line) if
> > respond_to?(:receive_line) }
> >
> > would become:
> >
> > @lpb_buffer.extract(data).map(&:chomp).each do |line|
> >   receive_line(line) if respond_to?(:receive_line)
> > end
> >
> > Or something to that effect (I''m guessing the procinator
isn''t
> > available, so perhaps .map { |s| s.chomp }.each)
> >
> > I''m at revision 379... should I go ahead and commit that
change?  I
> > don''t forsee any unintended consequences...
> >
> > - Tony
> >
> > On 6/13/07, Francis Cianfrocca < garbagecat10 at gmail.com>
wrote:
> > >
> > > Is that something you can do in your buftok-based implementation?
> > >
> > > On 6/13/07, Tony Arcieri < tony at clickcaster.com > wrote:
> > > >
> > > > If that''s the desired behavior (to be handle buggy
clients that
> > > > don''t always send CRLF) then the above-suggested
solution of a simple chomp
> > > > will suffice.  This will also eliminate all LF vs CRLF
confusion
> > > >
> > > > - Tony
> > > >
> > > > On 6/13/07, Francis Cianfrocca < garbagecat10 at
gmail.com> wrote:
> > > >
> > > > > On 6/13/07, Tony Arcieri <tony at clickcaster.com
> wrote:
> > > > > >
> > > > > > If this undesired behavior you can change the
default token used
> > > > > > by BufferedTokenizer to \r\n
> > > > > >
> > > > > > I believe I was trying to duplicate the previous
behavior of
> > > > > > LineAndTextProtocol, which tokenized on \n
> > > > >
> > > > >
> > > > >
> > > > > I can''t remember now but I''d be very
surprised if the original
> > > > > didn''t handle either CRLF or LF. If it
tokenized on LF, then it probably
> > > > > suppressed CR (if present) somewhere else. If it
didn''t, I''d consider that a
> > > > > bug.
> > > > >
> > > > > It would be great if you can make sure the current
implementation
> > > > > can handle either CRLF or LF, even intermixed in the
same stream (which some
> > > > > buggy code out there does).
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > Eventmachine-talk mailing list
> > > > > Eventmachine-talk at rubyforge.org
> > > > > http://rubyforge.org/mailman/listinfo/eventmachine-talk
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Tony Arcieri
> > > > ClickCaster, Inc.
> > > > tony at clickcaster.com
> > > > 720-227-0129 ext. 202
> > > >
> > > > _______________________________________________
> > > > Eventmachine-talk mailing list
> > > > Eventmachine-talk at rubyforge.org
> > > > http://rubyforge.org/mailman/listinfo/eventmachine-talk
> > > >
> > >
> > >
> > > _______________________________________________
> > > Eventmachine-talk mailing list
> > > Eventmachine-talk at rubyforge.org
> > > http://rubyforge.org/mailman/listinfo/eventmachine-talk
> > >
> >
> >
> >
> > --
> > Tony Arcieri
> > ClickCaster, Inc.
> > tony at clickcaster.com
> > 720-227-0129 ext. 202
> >
> > _______________________________________________
> > Eventmachine-talk mailing list
> > Eventmachine-talk at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/eventmachine-talk
> >
>
>
> _______________________________________________
> Eventmachine-talk mailing list
> Eventmachine-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/eventmachine-talk
>


--
Tony Arcieri
ClickCaster, Inc.
tony at clickcaster.com
720-227-0129 ext. 202
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070613/aa35a40a/attachment-0001.html

Bill and Duncan, can you verify that this change fixes your issues?
Thx Tony.

On 6/13/07, Tony Arcieri <tony at clickcaster.com>
wrote:
>
> Cool... ended up going with:
>
> @lpb_buffer.extract(data).each do |line|
>   receive_line(line.chomp) if respond_to?(:receive_line)
> end
>
> On 6/13/07, Francis Cianfrocca <garbagecat10 at gmail.com> wrote:
> >
> > Go for it.
> >
> > On 6/13/07, Tony Arcieri < tony at clickcaster.com> wrote:
> > >
> > > Yes, that can be accomplished here (line 42 of line_and_text.rb):
> > >
> > > @lpb_buffer.extract(data).each { |line| receive_line(line) if
> > > respond_to?(:receive_line) }
> > >
> > > would become:
> > >
> > > @lpb_buffer.extract(data).map(&:chomp).each do |line|
> > >   receive_line(line) if respond_to?(:receive_line)
> > > end
> > >
> > > Or something to that effect (I''m guessing the procinator
isn''t
> > > available, so perhaps .map { |s| s.chomp }.each)
> > >
> > > I''m at revision 379... should I go ahead and commit that
change?  I
> > > don''t forsee any unintended consequences...
> > >
> > > - Tony
> > >
> > > On 6/13/07, Francis Cianfrocca < garbagecat10 at gmail.com>
wrote:
> > > >
> > > > Is that something you can do in your buftok-based
implementation?
> > > >
> > > > On 6/13/07, Tony Arcieri < tony at clickcaster.com >
wrote:
> > > > >
> > > > > If that''s the desired behavior (to be handle
buggy clients that
> > > > > don''t always send CRLF) then the
above-suggested solution of a simple chomp
> > > > > will suffice.  This will also eliminate all LF vs CRLF
confusion
> > > > >
> > > > > - Tony
> > > > >
> > > > > On 6/13/07, Francis Cianfrocca < garbagecat10 at
gmail.com> wrote:
> > > > >
> > > > > > On 6/13/07, Tony Arcieri <tony at
clickcaster.com > wrote:
> > > > > > >
> > > > > > > If this undesired behavior you can change the
default token
> > > > > > > used by BufferedTokenizer to \r\n
> > > > > > >
> > > > > > > I believe I was trying to duplicate the
previous behavior of
> > > > > > > LineAndTextProtocol, which tokenized on \n
> > > > > >
> > > > > >
> > > > > >
> > > > > > I can''t remember now but I''d be
very surprised if the original
> > > > > > didn''t handle either CRLF or LF. If it
tokenized on LF, then it probably
> > > > > > suppressed CR (if present) somewhere else. If it
didn''t, I''d consider that a
> > > > > > bug.
> > > > > >
> > > > > > It would be great if you can make sure the current
> > > > > > implementation can handle either CRLF or LF, even
intermixed in the same
> > > > > > stream (which some buggy code out there does).
> > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > Eventmachine-talk mailing list
> > > > > > Eventmachine-talk at rubyforge.org
> > > > > >
http://rubyforge.org/mailman/listinfo/eventmachine-talk
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Tony Arcieri
> > > > > ClickCaster, Inc.
> > > > > tony at clickcaster.com
> > > > > 720-227-0129 ext. 202
> > > > >
> > > > > _______________________________________________
> > > > > Eventmachine-talk mailing list
> > > > > Eventmachine-talk at rubyforge.org
> > > > > http://rubyforge.org/mailman/listinfo/eventmachine-talk
> > > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > Eventmachine-talk mailing list
> > > > Eventmachine-talk at rubyforge.org
> > > > http://rubyforge.org/mailman/listinfo/eventmachine-talk
> > > >
> > >
> > >
> > >
> > > --
> > > Tony Arcieri
> > > ClickCaster, Inc.
> > > tony at clickcaster.com
> > > 720-227-0129 ext. 202
> > >
> > > _______________________________________________
> > > Eventmachine-talk mailing list
> > > Eventmachine-talk at rubyforge.org
> > > http://rubyforge.org/mailman/listinfo/eventmachine-talk
> > >
> >
> >
> > _______________________________________________
> > Eventmachine-talk mailing list
> > Eventmachine-talk at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/eventmachine-talk
> >
>
>
>
> --
> Tony Arcieri
> ClickCaster, Inc.
> tony at clickcaster.com
> 720-227-0129 ext. 202
>
> _______________________________________________
> Eventmachine-talk mailing list
> Eventmachine-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/eventmachine-talk
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070613/b5065c2f/attachment.html

> Bill and Duncan, can you verify that this change fixes your issues?
Not being so up on Ruby stuff, I solved it myself with a similar:

def receive_line line
      line.strip!

And this worked for me, so I assume the fix you suggested will also work.

I spent some time trying to fix the buftok stuff to allow both
''\n''
and ''\r\n'' as the delimiter, but gave up (the existing code
seemed a
bit, um Rococo.. ''specially to a Ruby nooby).

However, I''m now following Francis'' advice to go with the
HttpServer
code, so I don''t need the LineAndTextProtocol any more and
won''t be
able to verify your  solution for certain!!   =0)

This is going fine, but my latest issue is how do I use HttpResponse?
It calls send_data, but it''s a class not a module, so I can''t
include
it in my Connection.  The tests do all sorts of hacks (sorry if that
seems rude! =0) to check it''s working, but don''t really give a
good
pattern for actual use.

Apologies if I''ve missed something obvious. (It''s hard not to,
as I''m
doing this bumping around on the train home at night...  =0)

Any advice for me?

Oh - another thing I noticed is that HttpServer seems to assume
string/character data on a POST - I couldn''t make out what you do for
binary data, like file uploads or any other non-text Content-Type.

Cheers!  Although progress on Redux is slow, I''m having fun (and
learning a lot about Ruby).     =0)

Duncan

On 6/13/07, Francis Cianfrocca <garbagecat10 at gmail.com> wrote:
>
> On 6/13/07, Tony Arcieri <tony at clickcaster.com > wrote:
> > Cool... ended up going with:
> >
> > @lpb_buffer.extract(data).each do |line|
> >   receive_line(line.chomp) if respond_to?(:receive_line)
> > end

On 6/13/07, Duncan Cragg <ruby at cilux.org>
wrote:
>
>  This is going fine, but my latest issue is how do I use HttpResponse?
> It calls send_data, but it''s a class not a module, so I
can''t include
> it in my Connection.  The tests do all sorts of hacks (sorry if that
> seems rude! =0) to check it''s working, but don''t really
give a good
> pattern for actual use.

I''ll try to put together some use cases and post them here later on.
>
>
> Oh - another thing I noticed is that HttpServer seems to assume
> string/character data on a POST - I couldn''t make out what you do
for
> binary data, like file uploads or any other non-text Content-Type.

Binary data is fully supported. In Ruby, a String can be binary.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070613/60b79d3e/attachment.html

From: "Duncan Cragg" <ruby at cilux.org>
>
> I spent some time trying to fix the buftok stuff to allow both
''\n''
> and ''\r\n'' as the delimiter, but gave up (the existing
code seemed a
> bit, um Rococo.. ''specially to a Ruby nooby).
<grin> I had to look up "Rococo"... what a great adjective. :)

I had a pretty similar reaction.  Meaning no disrespect to the
author(s) of buftok, if I wanted to understand the code the
first thing I''d do is strip out all the comments.

. . . Indeed, after a: `grep -v ''^ *#''
lib/protocols/buftok.rb`
I can begin to understand how it works, although it seems to
go through more convolutions than I''d have expected.

Here is an alternate implementation (leaving out the size_limit
for clarity.)

class BufferedTokenizer
  def initialize(delimiter = "\n")
    @delimiter = delimiter
    @partial = ""
  end

  def extract(data)
    tokens = (@partial + data).split(@delimiter, -1)
    @partial = tokens.pop
    tokens
  end

  def flush
    tok = @partial
    @partial = ""
    tok
  end

  def empty?
    @partial.empty?
  end
end


Regards,

Bill

That looks a lot like the original implementation I replaced.

The point of BufferedTokenizer is to provide O(n) scalability that comes
from a loop which constantly rescanning the input buffer for the token.

(@partial + data).split(@delimiter, -1)

Here, split is running against @partial, which has already been scanned for
tokens or is an empty string.  This step is unnecessary, and occurs in a
loop.

- Tony

On 6/13/07, Bill Kelly <billk at cts.com> wrote:
>
> From: "Duncan Cragg" <ruby at cilux.org>
> >
> > I spent some time trying to fix the buftok stuff to allow both
''\n''
> > and ''\r\n'' as the delimiter, but gave up (the
existing code seemed a
> > bit, um Rococo.. ''specially to a Ruby nooby).
>
> <grin> I had to look up "Rococo"... what a great adjective.
:)
>
> I had a pretty similar reaction.  Meaning no disrespect to the
> author(s) of buftok, if I wanted to understand the code the
> first thing I''d do is strip out all the comments.
>
> . . . Indeed, after a: `grep -v ''^ *#''
lib/protocols/buftok.rb`
> I can begin to understand how it works, although it seems to
> go through more convolutions than I''d have expected.
>
> Here is an alternate implementation (leaving out the size_limit
> for clarity.)
>
> class BufferedTokenizer
>   def initialize(delimiter = "\n")
>     @delimiter = delimiter
>     @partial = ""
>   end
>
>   def extract(data)
>     tokens = (@partial + data).split(@delimiter, -1)
>     @partial = tokens.pop
>     tokens
>   end
>
>   def flush
>     tok = @partial
>     @partial = ""
>     tok
>   end
>
>   def empty?
>     @partial.empty?
>   end
> end
>
>
> Regards,
>
> Bill
>
>
> _______________________________________________
> Eventmachine-talk mailing list
> Eventmachine-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/eventmachine-talk
>


--
Tony Arcieri
ClickCaster, Inc.
tony at clickcaster.com
720-227-0129 ext. 202
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070613/5fd35d21/attachment.html

>  def extract(data)
>    tokens = (@partial + data).split(@delimiter, -1)
>    @partial = tokens.pop
>    tokens
>  end
Figures I''d find a bug right after hitting ''send''. 
:rolleyes:

  def extract(data)
    tokens = (@partial + data).split(@delimiter, -1)
    @partial = tokens.pop || ""
    tokens
  end

... added the (|| "") to handle data being an empty string.



Regards,

Bill

From: Tony Arcieri
>
> That looks a lot like the original implementation I replaced.
>
> The point of BufferedTokenizer is to provide O(n) scalability
> that comes from a loop which constantly rescanning the input
> buffer for the token.
>
> (@partial + data).split(@delimiter, -1)
>
> Here, split is running against @partial, which has already been
> scanned for tokens or is an empty string.  This step is unnecessary,
> and occurs in a loop.
Oh.  How about:

  def extract(data)
    return [] if data.empty?
    tokens = data.split(@delimiter, -1)
    if tokens.length > 1
      tokens[0] = @partial + tokens[0]
      @partial = ""
    end
    @partial << tokens.pop
    tokens
  end

Clearly I can''t make claims about performance without actually
having measured the two implementations... But the above
eliminates a split nested within a map!, and also a flatten.


Regards,

Bill

The convoluted bit with the nested within map! and flatten were added for
multibyte character support.  I''m not exactly sure what
they''re doing.

The original implementation looked like this:

http://pastie.caboo.se/70260

Yours may be a little faster.  Last time I benchmarked appending to a string
in a loop versus accumulating in an array and doing a join, appending to the
string was faster.  I''m guessing Ruby is doing some of the list/join
stuff
internally in C depending on how the string is used.

I think at this point only Benchmark.benchmark can say

- Tony

On 6/13/07, Bill Kelly <billk at cts.com> wrote:
>
>
> From: Tony Arcieri
> >
> > That looks a lot like the original implementation I replaced.
> >
> > The point of BufferedTokenizer is to provide O(n) scalability
> > that comes from a loop which constantly rescanning the input
> > buffer for the token.
> >
> > (@partial + data).split(@delimiter, -1)
> >
> > Here, split is running against @partial, which has already been
> > scanned for tokens or is an empty string.  This step is unnecessary,
> > and occurs in a loop.
>
> Oh.  How about:
>
>   def extract(data)
>     return [] if data.empty?
>     tokens = data.split(@delimiter, -1)
>     if tokens.length > 1
>       tokens[0] = @partial + tokens[0]
>       @partial = ""
>     end
>     @partial << tokens.pop
>     tokens
>   end
>
> Clearly I can''t make claims about performance without actually
> having measured the two implementations... But the above
> eliminates a split nested within a map!, and also a flatten.
>
>
> Regards,
>
> Bill
>
>
> _______________________________________________
> Eventmachine-talk mailing list
> Eventmachine-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/eventmachine-talk
>


--
Tony Arcieri
ClickCaster, Inc.
tony at clickcaster.com
720-227-0129 ext. 202
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070613/efe8646d/attachment.html

Hello again!
> Ok, here''s my suggestion for this: look at EventMachine#next_tick.
This is a
> method which takes either a Proc or a block, and it will execute the block
> after the next pass through the reactor core, WITHOUT blocking on the timer
> quantum. I think this is exactly what you want. You can schedule as many of
> these as you want. If you have a long-running operation scheduled with
> #next_tick, it can do part of its work, then schedule the rest with another
> #next_tick call, and then return. That will allow the reactor to keep doing
> its thing.
>
I''ve switched to next_tick. Thanks for that. I presume that as long as
there is both next_tick work to do and fd activity to do, /both/ will
be serviced, right? And there is no quantum delay as long as there''s
work to do and you keep re-calling next_tick?

This requires the SVN code, as the 0.7.2 release doesn''t have it. I
found an issue that is again probably just me not understanding
enough, but I had to comment out this:

+# linkso = Object.send :remove_const, "LINK_SO"
+# LINK_SO = linkso + "; strip $@"

To get it to compile.
> Is there anyone else out there who uses the pure-Ruby EM? If so, I can
> change my mind about whether to continue supporting it.
>
Not me any more!
>
> Epoll seems to be working great. I''m pretty happy with it.
>
Right, I can still only see 1022 connections from httperf (yes, I''m
getting nice 1000 rq/s performance as you promised!).  I''ve upped the
descriptor limit on the server process, and tried running two
httperfs, but still can''t get more than this number of connections.  I
checked it compiled with the EPOLL flag. Any ideas? How are you
testing it?

> Look on the EM Rubyforge page: there is another distro called
> eventmachine_httpserver. This is a production-quality HTTP parser that
calls
> into user-written Ruby code to fulfill web requests.
>
I''m using that now, thanks. I''m also borrowing from Webrick
for the
file fetching stuff.

I asked before about using HttpResponse. Here is my patch to enable me
to use it while waiting for official documentation!! Oh, and to add a
status message:

# -------------------------------------------------------------
# Override to allow access to response data and to add a status
# message

module EventMachine
    class HttpResponse
        attr_accessor :status_message
        attr_reader :output
        def send_headers
            @sent_headers = true
            fixup_headers
            ary = []
            ary << "HTTP/1.1 #{@status || 200} #{@status_message ||
''...''}\r\n"
            ary += generate_header_lines(@headers)
            ary << "\r\n"
            send_data ary.join
        end
        def send_data data
            @output ||= ""
            @output << data
        end
        def close_connection_after_writing
        end
    end
end

Anyway, things are still going slow but well with my port. At least it
looks like I''m going to beat my Python implementation with your
native/ext stuff driving the loop!  So thanks again for that!

Cheers!

Duncan

On 6/21/07, Duncan Cragg <ruby at cilux.org>
wrote:
>
> Hello again!
>  >
> I''ve switched to next_tick. Thanks for that. I presume that as
long as
> there is both next_tick work to do and fd activity to do, /both/ will
> be serviced, right? And there is no quantum delay as long as
there''s
> work to do and you keep re-calling next_tick?

Yes and yes.


This requires the SVN code, as the 0.7.2 release doesn''t have it.
I
> found an issue that is again probably just me not understanding
> enough, but I had to comment out this:
>
> +# linkso = Object.send :remove_const, "LINK_SO"
> +# LINK_SO = linkso + "; strip $@"
>
> To get it to compile.

Ouch. I added that very recently to allow stripped executables, which are
much smaller. And it was pretty tough to get it right. What platform are you
compiling on, which compiler version, and exactly what error message did you
get?

>
> >
> > Epoll seems to be working great. I''m pretty happy with it.
> >
> Right, I can still only see 1022 connections from httperf (yes,
I''m
> getting nice 1000 rq/s performance as you promised!).  I''ve upped
the
> descriptor limit on the server process, and tried running two
> httperfs, but still can''t get more than this number of
connections.  I
> checked it compiled with the EPOLL flag. Any ideas? How are you
> testing it?

You have to be superuser in order to up the per-process descriptor limit.
Read the rdoc for #set_descriptor_table_size, there is a lot of good info in
there, including how to drop superuser privs after you increase the
descriptor table size. I''ve tested epoll past 20,000 simultaneous
connections and it doesn''t break a sweat.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070621/13431e0b/attachment-0001.html

> .. I had to comment out this:
> >
> > +# linkso = Object.send :remove_const, "LINK_SO"
> > +# LINK_SO = linkso + "; strip $@"
> >
> > To get it to compile.
>
> ... What platform are you compiling on, which compiler version,
> and exactly what error message did you get?
I''m on Ubuntu / 2.6.17-11-386, gcc (GCC) 4.1.2 20060928 (prerelease)
(Ubuntu 4.1.1-13ubuntu5) and Ruby 1.9 - I''m not sure which Ubuntu
release, but it''s one behind the latest.

The problem is that ''ruby ext/extconf.rb'' gives:

checking for openssl/err.h... yes
*** ext/extconf.rb failed ***
Could not create Makefile due to some reason,...

Whereas ''ruby extconf.rb'' inside ext/ works fine.

So I can ''make'' in ext/, but not run ''ruby
setup.rb'' without
commenting out the lines above.


> You have to be superuser in order to up the per-process descriptor limit.
> Read the rdoc for #set_descriptor_table_size, there is a lot of good info
in
> there, including how to drop superuser privs after you increase the
> descriptor table size. I''ve tested epoll past 20,000 simultaneous
> connections and it doesn''t break a sweat.
I''d done all that, the problem was with my httperf being unable to go
above 1024!  So how are you testing the >1024 connections on the
client side?

Also, I notice you added some doc (EPOLL) to tell us how to use it. In
there I spotted a crucial nugget of information: call
''EventMachine.epoll'' before ''run''.  I spent
a long time finding that
out myself by tracing laboriously through the code, /before/ finding
this document on a recent ''svn up''!! =0(  Aaargh!  =0)  And
I''ve now
added the fd table size call as well, as per doc.

So - thanks again for EventMachine. Keep tapping away at the documentation!  =0)

Duncan

To test with >1024 connections, use ab, or else fork multiple processes of
your test client.

Not sure what to say about the makefile problem. I''m going to release
0.8.0shortly without changing anything and hope the problem is local
to your
environment. In any case, the two lines you commented out are only used to
create a stripped (smaller) shared object. Taking out those two lines
doesn''t change functionality at all.

On 6/27/07, Duncan Cragg <ruby at cilux.org>
wrote:
>
> > .. I had to comment out this:
> > >
> > > +# linkso = Object.send :remove_const, "LINK_SO"
> > > +# LINK_SO = linkso + "; strip $@"
> > >
> > > To get it to compile.
> >
> > ... What platform are you compiling on, which compiler version,
> > and exactly what error message did you get?
>
> I''m on Ubuntu / 2.6.17-11-386, gcc (GCC) 4.1.2 20060928
(prerelease)
> (Ubuntu 4.1.1-13ubuntu5) and Ruby 1.9 - I''m not sure which Ubuntu
> release, but it''s one behind the latest.
>
> The problem is that ''ruby ext/extconf.rb'' gives:
>
> checking for openssl/err.h... yes
> *** ext/extconf.rb failed ***
> Could not create Makefile due to some reason,...
>
> Whereas ''ruby extconf.rb'' inside ext/ works fine.
>
> So I can ''make'' in ext/, but not run ''ruby
setup.rb'' without
> commenting out the lines above.
>
>
>
> > You have to be superuser in order to up the per-process descriptor
> limit.
> > Read the rdoc for #set_descriptor_table_size, there is a lot of good
> info in
> > there, including how to drop superuser privs after you increase the
> > descriptor table size. I''ve tested epoll past 20,000
simultaneous
> > connections and it doesn''t break a sweat.
>
> I''d done all that, the problem was with my httperf being unable to
go
> above 1024!  So how are you testing the >1024 connections on the
> client side?
>
> Also, I notice you added some doc (EPOLL) to tell us how to use it. In
> there I spotted a crucial nugget of information: call
> ''EventMachine.epoll'' before ''run''.  I
spent a long time finding that
> out myself by tracing laboriously through the code, /before/ finding
> this document on a recent ''svn up''!! =0(  Aaargh!  =0) 
And I''ve now
> added the fd table size call as well, as per doc.
>
> So - thanks again for EventMachine. Keep tapping away at the
> documentation!  =0)
>
> Duncan
> _______________________________________________
> Eventmachine-talk mailing list
> Eventmachine-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/eventmachine-talk
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070627/dcadacd3/attachment.html