Hi, == Context = I need a messaging system to have rails pushing tasks to processes in a distributed environment. I need this to be small and hacker friendly, so Stompserver and activemessaging looked good to me. There are some needs that I can''t address as is with Stompserver and ActiveMessaging so I''m currently hacking them for the following: - make Stompserver use a database for storing queues instead of relying on Madeleine. Madeleine is interesting but I''m not sure I understand the limitations of the journaling (I''ve only looked briefly at Madeleine''s code) + I need a way to inspect the queue backlogs to monitor the whole environment so a journal isn''t really what I''m after, a traditionnal database is far better for this. - as I want messages to stay in queue instead of being dispatched to subscribers if they are already busy, I''m delaying messages for subscribers with "ack: client" until they send an ack back (this helps with load distribution and can avoid stompserver being blocked for a while when a new subscriber comes). - this means that I have to modify ActiveMessaging to support acknowledgment (or support it more naturally). == Current state = Unreadable :-) I''m a little hard on myself, but even if for example the DB support in Stompserver works, I''ve lots of cleanups to do... I began with the Madeleine -> ActiveRecord port which put pieces of StompServer''s code all around on the floor :-) It runs now (though I''m seeing warnings when I serialize the frames to a TEXT column : Exception `TypeError'' at /usr/lib/ruby/gems/1.8/gems/activerecord-1.14.4/lib/active_record/base.rb:2068 - can''t clone Fixnum). I''ve implemented my modified ack/queuing behaviour in Stompserver and am currently switching my attention to ActiveMessaging. == Questions = Is anyone else interested in these modifications? The more fair load distribution is completely independent of the DB support so I could push a patch for Stompserver rather easily. Don''t know yet for ActiveMessaging. The DB support is more invasive that I though it would be, I began thinking I could somehow allow users to choose a storage backend between Madeleine and ActiveRecord at runtime but I let this behind for the moment. If there''s interest in having a one-size fits all, I can look into it. Has someone a tip for the TypeError I get (described above)? It seems harmless, but I don''t like having harmless warnings next to real error messages :-) == Thanks = Thanks for both StompServer and ActiveMessaging. It will probably save me quite some development time (this is why I''m willing to contribute back). Best regards, Lionel --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
On 7/8/07, Lionel Bouton <lionel-subscription-WTamNBQcZIx7tPAFqOLdPg@public.gmane.org> wrote:> Thanks for both StompServer and ActiveMessaging. It will probably save > me quite some development time (this is why I''m willing to contribute back).I am pretty busy right now (with other things), but as long as you a add tests and touch base with me before pushing a new gem welcome to the project (I just added your rubyforge id). I am all for switching to a database store on the persistent queues. I have encounters a couple cases where Madeline did not work as well as I would have hoped. pth --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
You might take a look in branches/snacktime. It has some other backing stores that work better, the file backing seemed to work the best and have the best performance. I tried an sql store and it was pathetic performance wise, can''t remember if I ever committed it. Chris --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
Patrick Hurley wrote:> On 7/8/07, Lionel Bouton <lionel-subscription-WTamNBQcZIx7tPAFqOLdPg@public.gmane.org> wrote: >> Thanks for both StompServer and ActiveMessaging. It will probably save >> me quite some development time (this is why I''m willing to contribute >> back). > > I am pretty busy right now (with other things), but as long as you a > add tests and touch base with me before pushing a new gem welcome to > the project (I just added your rubyforge id).Wow, ok. I''ll polish things until I''m not too ashamed of them. I''ll probably have some details to work out with you before touching the svn repository, I''ll contact you again when I''m ready.> I am all for switching > to a database store on the persistent queues. I have encounters a > couple cases where Madeline did not work as well as I would have > hoped. >Hum, I suspected it was not robust when I saw a Thread somewhere that looked like it was linked to the journaling (not that having a separate thread doing snapshots can''t be made to work correctly, just that it seemed quite difficult to do so when I though about it). Lionel. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
snacktime wrote:> You might take a look in branches/snacktime.Thanks for the tip.> It has some other > backing stores that work better, the file backing seemed to work the > best and have the best performance. I tried an sql store and it was > pathetic performance wise, can''t remember if I ever committed it. >Hum, I thought that the performance would suffer a lot (as I enforce saving to DB as soon as the message comes in). In my case performance isn''t a problem though (a message is triggering huge processings of minutes or hours so limiting the throughput at say 10 messages/s is completely ok for me) and a DB brings remote inspection of the queues for free (which I need to monitor the whole system). If we don''t enforce a realtime ''message queued'' <-> ''message dumped to DB'' rule and allow writing messages to the database only if they stay in queue/aren''t acknowledge for a configurable delay there is a huge performance potential (think about all the messages that get through and are processed in a matter of tens of milliseconds) to get at the expense of reliability. So although it never will be the fastest backend, hopefully it can meet some needs other than my own. Hum, I''d better subscribe to the project''s devel list or I will bore everyone to death here... Best regards, Lionel --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
Hi Lionel - glad to hear about such active adoption of stomp, activemessaging, and stompserver - very cool. I think I have some good news for you on ActiveMessaging, see below... On 7/8/07, Lionel Bouton <lionel-subscription-WTamNBQcZIx7tPAFqOLdPg@public.gmane.org> wrote:> > > Hi, > > == Context => > I need a messaging system to have rails pushing tasks to processes in a > distributed environment. I need this to be small and hacker friendly, so > Stompserver and activemessaging looked good to me. There are some needs > that I can''t address as is with Stompserver and ActiveMessaging so I''m > currently hacking them for the following:I also recently checked in support for ReliableMessaging which supports file and database persistence of messages. Are you aware of this gem? http://trac.labnotes.org/cgi-bin/trac.cgi/wiki/Ruby/ReliableMessaging It is now fully supported in ActiveMessaging as a simple, pure Ruby broker implementation, and pretty hackable as well. Sounds like you are deep into StompServer, making headway and adding good stuff (which I am happy to see - glad StompServer is getting such additional attention) so don''t let me deter you, just wanted you to know there is an alternative. - make Stompserver use a database for storing queues instead of relying> on Madeleine. Madeleine is interesting but I''m not sure I understand the > limitations of the journaling (I''ve only looked briefly at Madeleine''s > code) + I need a way to inspect the queue backlogs to monitor the whole > environment so a journal isn''t really what I''m after, a traditionnal > database is far better for this.I''ve looked at Madeleine, very interesting stuff, but I can understand your needs for better management/monitoring. ReliableMessaging uses mysql or file based message storage, so it is still an alternative on this front. One disad it has is that it is mysql specific, not using active record, so if you want to use a different db, you''re sol. - as I want messages to stay in queue instead of being dispatched to> subscribers if they are already busy, I''m delaying messages for > subscribers with "ack: client" until they send an ack back (this helps > with load distribution and can avoid stompserver being blocked for a > while when a new subscriber comes).Totally agree, which is why I fixed the ActiveMessaging stomp adapter to have ack mode support - checked it in last week :)> - this means that I have to modify ActiveMessaging to support > acknowledgment (or support it more naturally).Ummm not sure what you think is missing - please look at my recent changes; you should be happy. Also you can check out the active messaging list and you''ll see my mail about it. ActiveMessaging now supports passing the :ack=>''client'' mode in the subscription, and has built in sending of acknowledgements after sucessful dispatch using the ''ActiveMessaging::Adapters::Stomp.received'' method , which is standard now in ActiveMessgaing as the way to have any connection type acknowledge its messages (Amazon SQS uses this as well): def received message, headers={} #check to see if the ack mode for this subscription is auto or client # if the ack mode is client, send an ack if (headers[:ack] === ''client'') ack_headers = message.headers.has_key?(:transaction) ? message.headers[:transaction] : {} ack message.headers[''message-id''], ack_headers end end All you need to do is add the :ack=>''client'' header to the subscribes_to call, and you''re all set. Let me know if there is something else you think is missing. == Current state =>> Unreadable :-) I''m a little hard on myself, but even if for example the > DB support in Stompserver works, I''ve lots of cleanups to do... > > I began with the Madeleine -> ActiveRecord port which put pieces of > StompServer''s code all around on the floor :-) It runs now (though I''m > seeing warnings when I serialize the frames to a TEXT column : > Exception `TypeError'' at > /usr/lib/ruby/gems/1.8/gems/activerecord-1.14.4 > /lib/active_record/base.rb:2068 > - can''t clone Fixnum). > > I''ve implemented my modified ack/queuing behaviour in Stompserver and am > currently switching my attention to ActiveMessaging.Again, please look at my recent changes, I really would be suprised if there is anything else regarding ack modes that needs to be added. == Questions =>> Is anyone else interested in these modifications? > The more fair load distribution is completely independent of the DB > support so I could push a patch for Stompserver rather easily. Don''t > know yet for ActiveMessaging.Always happy to get patches :) The DB support is more invasive that I though it would be, I began> thinking I could somehow allow users to choose a storage backend between > Madeleine and ActiveRecord at runtime but I let this behind for the > moment. If there''s interest in having a one-size fits all, I can look > into it. > > Has someone a tip for the TypeError I get (described above)? It seems > harmless, but I don''t like having harmless warnings next to real error > messages :-) > > == Thanks => > Thanks for both StompServer and ActiveMessaging. It will probably save > me quite some development time (this is why I''m willing to contribute > back).Glad you''re getting some use out of all this. Out of curiousity, can you talk about what you are using ActiveMessaging for? In any case, let me know if the ack mode stuff, or anything else in ActiveMessaging, isn''t cutting it for ya. -Andrew Kuklewicz http://beginsinwonder.com Best regards,> > Lionel > > > >-- Andrew Kuklewicz --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---