[Eventmachine-talk] DistribuStream released

Tony Arcieri
Fri, 19 Oct 2007 10:19:02 -0000

We observed it while running LineAndTextProtocol and HttpClient in the same
Reactor.  I can see if I can revert the codebase to when we were
experiencing the problem (months and months ago), but I think the better
approach is to start removing some of the threaded components in our current
codebase and if the problem comes up again we can have a look then.

We were also trying to use Kirk''s evented Mongrel at one point, however
it
looks like due to the way the Mongrel API works we can''t use it
properly for
what we want to do.  Mongrel has a method you call to synchronously consume
the request body (for example, in a POST or PUT request), but in the evented
Mongrel this is stubbed out and the whole request has been consumed
(including the body) before we even get access to the header information.
We''re using the header information to authenticate file transfers, so
it''s
somewhat pointless and wasteful if we can''t get access to that
information
before the entire transfer has occurred.

Francis, does your HTTP server let you check the header of a POST or PUT
request before the body is consumed?

On 10/19/07, Francis Cianfrocca <garbagecat10 at gmail.com>
wrote:
>
> On 10/18/07, Tony Arcieri <tony at clickcaster.com> wrote:
>
> > I''m glad to announce the first initial public release of
DistribuStream:
> >
> > http://distribustream.org/
> >
> > DistribuStream is a peercasting system, facilitating streaming media
> > over a managed peer-to-peer network.  You can think of it as being
similar
> > to BitTorrent, but employing progressive downloads so media can be
consumed
> > in on-demand streams (and it''s not written in Python! :).  It
was developed
> > by ClickCaster, Inc. <http://www.clickcaster.com/> through the
> > University of Colorado Computer Science Program.  The whole project is
> > written in Ruby and uses EventMachine as the basis of both the client
and
> > server.  It''s available as a gem (gem install distribustream)
and you can
> > find a basic writeup on how to use it here:
> >
> > http://www.clickcaster.com/items/distribustream-0-1-0-released
> >
> > This release targets primarily developers as it has some pretty major
> > issues that need to get resolved before it''s generally
usable.  The most
> > pressing is the client implementation, which currently stores the
entire
> > file in RAM before writing it out to an IO object.  This needs to be
changed
> > to a disk buffer, and / or combined with the protocol''s
ability to
> > "unprovide" certain file segments from the peer network, so
the client can
> > store a manageable moving window of data.
> >
> > In terms of implementation details that would nice to get resolved,
the
> > present implementation uses a hodgepodge of both EM and threaded
> > components.  While the underlying protocol (the Peer Distributed
Transfer
> > Protocol, see pdtp.org) is implemented using EventMachine, all of the
> > peer-to-peer file transfers utilize the regular threaded Mongrel (
i.e.
> > not Kirk''s Mongrel) and the regular (and abominable)
Net::HTTP client.
> >
> > I would very much like to eliminate all of the threaded components and
> > running everything inside the Reactor loop with the EM HttpClient and
either
> > Kirk''s Mongrel or Francis'' em_httpserver.  In
earlier implementations where
> > we tried this we ran into inexplicable stalling inside of the Reactor
> > itself, and because of that the team decided to go with the hybrid
Reactor /
> > threaded approach.  Now that the project is open source I''d
like to start
> > looking into this again, and would greatly appreciate whatever help
anyone
> > can offer.
> >
> > I''ll be announcing it to ruby-talk in a few days or so, but
thought I''d
> > announce it here first and get some feedback.
>
>
>
> Do you have any general information about the reactor stalling you
> observed? It would be great to figure out where that was coming from.
>
>
> _______________________________________________
> Eventmachine-talk mailing list
> Eventmachine-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/eventmachine-talk
>



--
Tony Arcieri
ClickCaster, Inc.
tony at clickcaster.com
720-227-0129 ext. 202
-------------- next part --------------
An HTML attachment was scrubbed...