[Eventmachine-talk] DistribuStream released

Francis Cianfrocca
Fri, 19 Oct 2007 01:32:06 -0000

On 10/18/07, Tony Arcieri <tony at clickcaster.com>
wrote:
>
> I''m glad to announce the first initial public release of
DistribuStream:
>
> http://distribustream.org/
>
> DistribuStream is a peercasting system, facilitating streaming media over
> a managed peer-to-peer network.  You can think of it as being similar to
> BitTorrent, but employing progressive downloads so media can be consumed in
> on-demand streams (and it''s not written in Python! :).  It was
developed by ClickCaster,
> Inc. <http://www.clickcaster.com/> through the University of Colorado
> Computer Science Program.  The whole project is written in Ruby and uses
> EventMachine as the basis of both the client and server.  It''s
available as
> a gem (gem install distribustream) and you can find a basic writeup on how
> to use it here:
>
> http://www.clickcaster.com/items/distribustream-0-1-0-released
>
> This release targets primarily developers as it has some pretty major
> issues that need to get resolved before it''s generally usable.
The most
> pressing is the client implementation, which currently stores the entire
> file in RAM before writing it out to an IO object.  This needs to be
changed
> to a disk buffer, and / or combined with the protocol''s ability to
> "unprovide" certain file segments from the peer network, so the
client can
> store a manageable moving window of data.
>
> In terms of implementation details that would nice to get resolved, the
> present implementation uses a hodgepodge of both EM and threaded
> components.  While the underlying protocol (the Peer Distributed Transfer
> Protocol, see pdtp.org) is implemented using EventMachine, all of the
> peer-to-peer file transfers utilize the regular threaded Mongrel (i.e. not
> Kirk''s Mongrel) and the regular (and abominable) Net::HTTP client.
>
> I would very much like to eliminate all of the threaded components and
> running everything inside the Reactor loop with the EM HttpClient and
either
> Kirk''s Mongrel or Francis'' em_httpserver.  In earlier
implementations where
> we tried this we ran into inexplicable stalling inside of the Reactor
> itself, and because of that the team decided to go with the hybrid Reactor
/
> threaded approach.  Now that the project is open source I''d like
to start
> looking into this again, and would greatly appreciate whatever help anyone
> can offer.
>
> I''ll be announcing it to ruby-talk in a few days or so, but
thought I''d
> announce it here first and get some feedback.



Do you have any general information about the reactor stalling you observed?
It would be great to figure out where that was coming from.
-------------- next part --------------
An HTML attachment was scrubbed...