Eventmachine talk - Dec 2007 - EM, reliable-messaging and Net-AMQP

Hi Francis C.,
I was killing some time searching on messaging protocols. One thing
lead to another lead to :
http://rubyforge.org/projects/net-amqp/

I saw Francis Cianfrocca is listed as the admin. and recalled an
earlier email about the reliable-messaging possibly being included in
EM.
When you have a few moments could you indicate where how Net::AMQP
fits in any reliable-messaging or EM plans?

Thanks
Mark

On Dec 28, 2007 5:14 AM, Mark Van De Vyver <mvyver at gmail.com> wrote:
> Hi Francis C.,
> I was killing some time searching on messaging protocols. One thing
> lead to another lead to :
> http://rubyforge.org/projects/net-amqp/
>
> I saw Francis Cianfrocca is listed as the admin. and recalled an
> earlier email about the reliable-messaging possibly being included in
> EM.
> When you have a few moments could you indicate where how Net::AMQP
> fits in any reliable-messaging or EM plans?
>

It''s long been my contention that Ruby needs a high-speed reliable
messaging
engine, to enable the construction of advanced application architectures.
It''s not entirely clear how many people agree with me.

At this point, it''s clear that there is a great deal of interest in
building
process-pool architectures, in which several processes (not threads) share
the load of applications like web servers or web services. So far the
emphasis seems to have been on simply getting a more robust and scalable
version of a thread pool and running it on either a single machine or a
cluster.

This is something that isn''t hard to roll yourself in EM, although
it''s
clear that the model needs to be supported "out of the box."

However, it''s far from a real loosely-coupled architecture that would
enable
people to write and deploy arbitrary units of functionality and have them
work together in an ad-hoc way. To do that, we need a real messaging engine.

That''s why I started the AMQP project. As you know, it''s a
state-of-the-art
wire protocol-cum-library API that doesn''t seem to have taken the world
by
storm.

If you want to integrate with enterprise messaging systems today, you can
use the Stomp protocol. The EM distro contains an evented implementation of
Stomp, and several people have worked on a Stomp engine using EM. If your
enterprise uses AMQ from the Jakarta project, you can integrate with it.

AMQP itself is not a protocol one wants to implement by hand. It''s
enormous
and complex. It is something you can probably implement by writing XSLT
scripts, however.

The near-term EM roadmap is all about working with Ruby 1.9. Significant
performance improvements will be available on 1.9, so watch this list for
announcements. After that, we need a major push to improve awareness of EM,
build a public site for it, write books, papers and other documents about
it, and generally build up the community.

EM has been around for a year and a half now, but it''s just in the last
few
weeks that Rubyists have started really discussing the benefits of the
event-driven model. And from the tenor of those conversations (which are all
about reinventing wheels that EM has had for a long time) it seems that
people just aren''t aware of what''s already been done here.

It might take a similar amount of time before the Ruby world is ready to
talk about loosely-coupled application architectures. In the meantime we
should keep thinking about it and laying groundwork.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/eventmachine-talk/attachments/20071228/70fd7b22/attachment-0001.html

Hi Francis -

On Dec 29, 2007 12:22 AM, Francis Cianfrocca <garbagecat10 at gmail.com>
wrote:
>
> On Dec 28, 2007 5:14 AM, Mark Van De Vyver <mvyver at gmail.com>
wrote:
>
>
> > Hi Francis C.,
> > I was killing some time searching on messaging protocols. One thing
> > lead to another lead to :
> > http://rubyforge.org/projects/net-amqp/
> >
> > I saw Francis Cianfrocca is listed as the admin. and recalled an
> > earlier email about the reliable-messaging possibly being included in
> > EM.
> > When you have a few moments could you indicate where how Net::AMQP
> > fits in any reliable-messaging or EM plans?
> >
>
> It''s long been my contention that Ruby needs a high-speed reliable
messaging
> engine, to enable the construction of advanced application architectures.
> It''s not entirely clear how many people agree with me.
Yes.  My experience is that you can quickly saturate a cpu/machine
and, while I came to understand messaging via MPI, I think it is good
to be able to scale to multiple machines without much change to your
code.
> At this point, it''s clear that there is a great deal of interest
in building
> process-pool architectures, in which several processes (not threads) share
> the load of applications like web servers or web services. So far the
> emphasis seems to have been on simply getting a more robust and scalable
> version of a thread pool and running it on either a single machine or a
> cluster.
>
> This is something that isn''t hard to roll yourself in EM, although
it''s
> clear that the model needs to be supported "out of the box."
>
> However, it''s far from a real loosely-coupled architecture that
would enable
> people to write and deploy arbitrary units of functionality and have them
> work together in an ad-hoc way. To do that, we need a real messaging
engine.
This appeals to my understanding of my use case.
> That''s why I started the AMQP project. As you know, it''s
a state-of-the-art
> wire protocol-cum-library API that doesn''t seem to have taken the
world by
> storm.
You''re right, not by storm, but there does seem to be some interest.
In case anyone else comes long this thread... after I wrote I came
across (all open source in some way):
openAMQ (broker):
http://www.openamq.org/
RabbitMQ (broker):
http://www.rabbitmq.com/
Apache Qpid (broker + Ruby client)
http://people.apache.org/dist/incubator/qpid/M2-incubating/Ruby/
Redhat Messaging (builds on Qpid):
http://labs.jboss.com/rhmessaging/rhm-overview.html

AFAICT Qpid is the Ruby client that everyone suggests to use (given
Ruby is what you want) - not having this registered/available on
rubyforge may stunt awareness?
It would seem that the Apache Qpid/RHM project is off to a reasonable
start - I''m largely ignorant of license or implementation issues that
may be lurking, but some may find these worth investigating.
> If you want to integrate with enterprise messaging systems today, you can
> use the Stomp protocol. The EM distro contains an evented implementation of
> Stomp, and several people have worked on a Stomp engine using EM. If your
> enterprise uses AMQ from the Jakarta project, you can integrate with it.
>
> AMQP itself is not a protocol one wants to implement by hand. It''s
enormous
> and complex. It is something you can probably implement by writing XSLT
> scripts, however.
Hopefully the Apache Qpid/RHM project will ''take-off''.
> The near-term EM roadmap is all about working with Ruby 1.9. Significant
> performance improvements will be available on 1.9, so watch this list for
> announcements. After that, we need a major push to improve awareness of EM,
> build a public site for it, write books, papers and other documents about
> it, and generally build up the community.
>
> EM has been around for a year and a half now, but it''s just in the
last few
> weeks that Rubyists have started really discussing the benefits of the
> event-driven model. And from the tenor of those conversations (which are
all
> about reinventing wheels that EM has had for a long time) it seems that
> people just aren''t aware of what''s already been done
here.
Yes EM pleasantly surprised me :)
> It might take a similar amount of time before the Ruby world is ready to
> talk about loosely-coupled application architectures. In the meantime we
> should keep thinking about it and laying groundwork.
OK.
Thanks again for all you efforts.

Cheers
>
>