Is anybody out there witch has already implemented a SMTP server protocol handler for event machine? Peter
I have one that I''ll publish shortly, unless someone else has one already. On 8/29/07, Peter Ehrenberg <pe at dipe.de> wrote:> > Is anybody out there witch has already implemented a SMTP > server protocol handler for event machine? > > Peter > _______________________________________________ > Eventmachine-talk mailing list > Eventmachine-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/eventmachine-talk >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070829/26ceb938/attachment.html
On 8/29/07, Peter Ehrenberg <pe at dipe.de> wrote:> > Is anybody out there witch has already implemented a SMTP > server protocol handler for event machine?Ok, so here''s an outline of what I have. I''ve been developing this in C++ as a Ruby extension, but shifted to pure Ruby just for the sake of getting it out there more quickly. (Eventually it will shift back to C++, since EventMachine is starting to be used in non-Ruby environments.) There is a new class called EventMachine::SmtpServer, which subclasses EventMachine::Connection. Your code would subclass EM::SmtpServer, and you would override specific methods that will be called automatically at certain points during the processing of incoming SMTP transactions. This allows you to customize the behavior of the server in a fine-grained way. For example, you will get individual callbacks containing the domain name in the EHLO statement, the mail sender (MAIL FROM:), the recipients (RCPT TO:) and the message headers and body (DATA). You can specify whether you want to get the data line by line (appropriate for huge messages you may want to buffer yourself) or just get one callback with the whole message. Sample code: #--------------------------------------------------------------------- require ''rubygems'' require ''eventmachine'' class MyClass < EM::SmtpServer def initialize *args super end def receive_mail_from( sender ) # do something and return true/false to accept/reject the sender end def receive_rcpt_to( recipients ) # recipients is a Ruby Array. return an array of true/falses. end def receive_message( headers, data ) # process the message and reset internal buffers for the next one. end end EM.run { EM.start_server "0.0.0.0", 25, MyClass } #-------------------------------------------------------------------- Was that more or less what you had in mind? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070830/54952d17/attachment.html
"Francis Cianfrocca" <garbagecat10 at gmail.com> writes:> On 8/29/07, Peter Ehrenberg <pe at dipe.de> wrote:>> Is anybody out there witch has already implemented a SMTP server >> protocol handler for event machine?> Ok, so here''s an outline of what I have. [...] Was that more or less > what you had in mind?(back on this project again -- sorry for the long delay) Yes this was exactly what I searched for, I think! I have to get the recipient and message, pull out the subject and write it into a database table. It seems as if this was really easy with your approach. Do you have any idea, when you would release your code? I''m willing to be your beta tester :-) /Peter
I''ll be checking the code into SVN in about an hour. On 9/10/07, Peter Ehrenberg <pe at dipe.de> wrote:> > "Francis Cianfrocca" <garbagecat10 at gmail.com> writes: > > > On 8/29/07, Peter Ehrenberg <pe at dipe.de> wrote: > > >> Is anybody out there witch has already implemented a SMTP server > >> protocol handler for event machine? > > > Ok, so here''s an outline of what I have. [...] Was that more or less > > what you had in mind? > > (back on this project again -- sorry for the long delay) > > > Yes this was exactly what I searched for, I think! > > > I have to get the recipient and message, pull out the subject and > write it into a database table. > > It seems as if this was really easy with your approach. > > > Do you have any idea, when you would release your code? I''m willing to > be your beta tester :-) > > > /Peter > _______________________________________________ > Eventmachine-talk mailing list > Eventmachine-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/eventmachine-talk >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070911/62d92cb6/attachment.html
"Francis Cianfrocca" <garbagecat10 at gmail.com> writes:> I''ll be checking the code into SVN in about an hour.Ok, I''ve seen it. Many Thanks! I begin to implement my server in the next days. Please stay tuned. /Peter
On 9/12/07, Peter Ehrenberg <pe at dipe.de> wrote:> > "Francis Cianfrocca" <garbagecat10 at gmail.com> writes: > > > I''ll be checking the code into SVN in about an hour. > > Ok, I''ve seen it. Many Thanks! > > I begin to implement my server in the next days. > Please stay tuned.I''m using this SMTP implementation (both client and server) in a production system now. Please send email to this list asap if there are problems or feature requests. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070912/806c5904/attachment.html
> Ok, so here''s an outline of what I have. I''ve been developing this in C++ as > a Ruby extension, but shifted to pure Ruby just for the sake of getting it > out there more quickly. (Eventually it will shift back to C++, since > EventMachine is starting to be used in non-Ruby environments.)What are the advantages of EventMachine over libevent (for C on Win32 and Unix) or ACE (for C++)? -- --- Thomas H. Ptacek // matasano security read us on the web: http://www.matasano.com/log
On 9/12/07, Thomas Ptacek <tqbf at matasano.com> wrote:> What are the advantages of EventMachine over libevent (for C on Win32 > and Unix) or ACE (for C++)?EM is a higher level construct that libevent. It takes care of many of the details that you would have to manage for yourself with libevent. I don''t know anything about ACE. Kirk Haines
That''s true, but the "heavy lifting" is the event and timer loop; the server and client code is pretty straightforward once you have the loop. I guess a way to restate the question is, for C/C++ code, why not just build EventMachine onto libevent? If you''re not familiar with ACE, think "EventMachine for C++", only much, much, much bigger. That''s a good thing and a (mostly) bad thing.> EM is a higher level construct that libevent. It takes care of many > of the details that you would have to manage for yourself with > libevent. I don''t know anything about ACE.-- --- Thomas H. Ptacek // matasano security read us on the web: http://www.matasano.com/log
On 9/12/07, Thomas Ptacek <tqbf at matasano.com> wrote:> That''s true, but the "heavy lifting" is the event and timer loop; the > server and client code is pretty straightforward once you have the > loop. > > I guess a way to restate the question is, for C/C++ code, why not just > build EventMachine onto libevent?I''d like to know this as well. epoll utilization is great if you''re on Linux; not so great if you''re using Mac OS X or FreeBSD, which have kqueue. Using libevent instead of directly calling epoll might have benefited more users by supporting more operating systems. Best regards, --Michael
The green thread scheduler ruled libevent out for Ruby.> I''d like to know this as well. epoll utilization is great if you''re > on Linux; not so great if you''re using Mac OS X or FreeBSD, which have > kqueue. Using libevent instead of directly calling epoll might have > benefited more users by supporting more operating systems.-- --- Thomas H. Ptacek // matasano security read us on the web: http://www.matasano.com/log
This can be fixed in YARV, by wrapping event_dispatch() with the BLOCKING_CALL macro. This releases the VM''s global lock and allows the blocking call to execute inside a native thread. On 9/12/07, Thomas Ptacek <tqbf at matasano.com> wrote:> > The green thread scheduler ruled libevent out for Ruby. > > > I''d like to know this as well. epoll utilization is great if you''re > > on Linux; not so great if you''re using Mac OS X or FreeBSD, which have > > kqueue. Using libevent instead of directly calling epoll might have > > benefited more users by supporting more operating systems. > > -- > --- > Thomas H. Ptacek // matasano security > read us on the web: http://www.matasano.com/log > _______________________________________________ > Eventmachine-talk mailing list > Eventmachine-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/eventmachine-talk >-- Tony Arcieri ClickCaster, Inc. tony at clickcaster.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070912/d24b8a1a/attachment.html
I guess. EventMachine works now. Does YARV? On 9/12/07, Tony Arcieri <tony at clickcaster.com> wrote:> This can be fixed in YARV, by wrapping event_dispatch() with the > BLOCKING_CALL macro. This releases the VM''s global lock and allows the > blocking call to execute inside a native thread.-- --- Thomas H. Ptacek // matasano security read us on the web: http://www.matasano.com/log
From: "Thomas Ptacek" <tqbf at matasano.com>> > I guess a way to restate the question is, for C/C++ code, why not just > build EventMachine onto libevent?Francis said a few words about this on ruby-talk awhile back: http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/253463 Regards, Bill
On 9/12/07, Thomas Ptacek <tqbf at matasano.com> wrote:> > > Ok, so here''s an outline of what I have. I''ve been developing this in > C++ as > > a Ruby extension, but shifted to pure Ruby just for the sake of getting > it > > out there more quickly. (Eventually it will shift back to C++, since > > EventMachine is starting to be used in non-Ruby environments.) > > What are the advantages of EventMachine over libevent (for C on Win32 > and Unix) or ACE (for C++)?As I''ve said in the past, libevent is a first-rate piece of work, but it has problems working with MRI. But the goal of EM is completely different. At bottom, EM is a high-performance I/O engine, very very stable and based on years on insights from high-performance network programming. It also (I will admit) enables programmers to avoid threads, which is a personal bias of mine. (In my defense, I''ll say that this bias is not a prejudice. I''ve been writing threaded programs on Windows since the earliest Win32 betas, 15 years ago, and on Unix since before Posix threads were a standard.) Above the base level, however, the goal is to provide as complete a set of tools for network programming and distributed computing as possible, congruent with the "Ruby way." I''ll be happy when Ruby programmers needing to write network-aware software can reach into the EM toolkit and just pull out what they need. A perfect example is the subject of this thread: a complete and correct implementation of SMTP (both server and client side) that any Ruby programmer can simply drop in. The SMTP implementation (like the already-existing HTTP and other implementations) throws off "events" that your programs can respond to. EXCEPT that they are events that make sense in an SMTP context! Things like "receive_sender," "receive_recipient" and "receive_message." EM''s layered approach to protocol handling makes this possible. That''s what makes EM different from low-level libraries like libevent. EM is also usable without Ruby. It''s not as easy, because not many things are as easy as Ruby, but still it wraps up all the nasty bits and the protocols. There is also an EM version for Java. Originally written for JRuby, it''s going to be a fully standalone framework, because as far as I''ve been able to tell, Java has nothing directly comparable to EM. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070912/335d7d68/attachment-0001.html
On 9/12/07, Michael S. Fischer <michael at dynamine.net> wrote:> > On 9/12/07, Thomas Ptacek <tqbf at matasano.com> wrote: > > That''s true, but the "heavy lifting" is the event and timer loop; the > > server and client code is pretty straightforward once you have the > > loop. > > > > I guess a way to restate the question is, for C/C++ code, why not just > > build EventMachine onto libevent? > > I''d like to know this as well. epoll utilization is great if you''re > on Linux; not so great if you''re using Mac OS X or FreeBSD, which have > kqueue. Using libevent instead of directly calling epoll might have > benefited more users by supporting more operating systems.EM works fine on Mac, BSD, Solaris, Windows, and other platforms that support the select(2) system call. I''d love to see someone step up and contribute an implementation of kqueue and IOCP. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070912/8c21dd40/attachment.html
On 9/12/07, Francis Cianfrocca <garbagecat10 at gmail.com> wrote:> EM works fine on Mac, BSD, Solaris, Windows, and other platforms that > support the select(2) system call. I''d love to see someone step up and > contribute an implementation of kqueue and IOCP.I''ll second that. I know for a fact that some of my stuff that I have written to use EM is being used on very high load production Solaris systems without problems. However, I''d especially love to see kqueue support. Kirk Haines
On 9/12/07, Kirk Haines <wyhaines at gmail.com> wrote:> > On 9/12/07, Francis Cianfrocca <garbagecat10 at gmail.com> wrote: > > > EM works fine on Mac, BSD, Solaris, Windows, and other platforms that > > support the select(2) system call. I''d love to see someone step up and > > contribute an implementation of kqueue and IOCP. > > I''ll second that. I know for a fact that some of my stuff that I have > written to use EM is being used on very high load production Solaris > systems without problems. > > However, I''d especially love to see kqueue support.Is kqueue supported on Solaris??? That''s news to me. Solaris has that Posix AIO support, which I played with once, and it SUCKS. (Unless they''ve improved it since then.) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070912/d7c0f417/attachment.html
On 9/12/07, Francis Cianfrocca <garbagecat10 at gmail.com> wrote:> On 9/12/07, Kirk Haines <wyhaines at gmail.com> wrote: > > On 9/12/07, Francis Cianfrocca <garbagecat10 at gmail.com> wrote: > > > > > EM works fine on Mac, BSD, Solaris, Windows, and other platforms that > > > support the select(2) system call. I''d love to see someone step up and > > > contribute an implementation of kqueue and IOCP. > > > > I''ll second that. I know for a fact that some of my stuff that I have > > written to use EM is being used on very high load production Solaris > > systems without problems. > > > > However, I''d especially love to see kqueue support. > > > Is kqueue supported on Solaris??? That''s news to me. Solaris has that Posix > AIO support, which I played with once, and it SUCKS. (Unless they''ve > improved it since then.)No. On Solaris 7+, select() wraps /dev/poll, which is pretty fast, though not quite as fast as kqueue. Solaris 10 introduces a new event completion framework, which many are calling IOCP (not to be confused with IOCP in Windows). See http://developers.sun.com/solaris/articles/event_completion.html Best regards, --Michael
POSIX AIO is worthless Solaris has /dev/poll, which is a fairly simple API built around writing pollfds to /dev/poll Solaris 10 added an elegant and robust event completion interface On 9/12/07, Francis Cianfrocca <garbagecat10 at gmail.com> wrote:> > On 9/12/07, Kirk Haines <wyhaines at gmail.com> wrote: > > > > On 9/12/07, Francis Cianfrocca <garbagecat10 at gmail.com> wrote: > > > > > EM works fine on Mac, BSD, Solaris, Windows, and other platforms that > > > support the select(2) system call. I''d love to see someone step up and > > > > > contribute an implementation of kqueue and IOCP. > > > > I''ll second that. I know for a fact that some of my stuff that I have > > written to use EM is being used on very high load production Solaris > > systems without problems. > > > > However, I''d especially love to see kqueue support. > > > > Is kqueue supported on Solaris??? That''s news to me. Solaris has that > Posix AIO support, which I played with once, and it SUCKS. (Unless they''ve > improved it since then.) > > > _______________________________________________ > Eventmachine-talk mailing list > Eventmachine-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/eventmachine-talk >-- Tony Arcieri ClickCaster, Inc. tony at clickcaster.com 720-227-0129 ext. 202 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070912/e9937549/attachment-0001.html
On 9/12/07, Francis Cianfrocca <garbagecat10 at gmail.com> wrote:> Is kqueue supported on Solaris??? That''s news to me. Solaris has that Posix > AIO support, which I played with once, and it SUCKS. (Unless they''ve > improved it since then.)Arg. Yeah. I was thinking Solaris shared that with the *BSDs. It''s /dev/poll that seems to be the thing with Solaris, though. Kirk
On 9/12/07, Michael S. Fischer <michael at dynamine.net> wrote:> > > No. On Solaris 7+, select() wraps /dev/poll, which is pretty fast, > though not quite as fast as kqueue. Solaris 10 introduces a new event > completion framework, which many are calling IOCP (not to be confused > with IOCP in Windows). See > http://developers.sun.com/solaris/articles/event_completion.html > > Best regards, > > --Michael > _Have you used it? Any good? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070912/ee9b97e2/attachment.html
/dev/poll is simple and provides a decent degree of scalability. It''s also portable to platforms like HP-UX and IRIX. The event completion ports interface is excellent and scales much better. On 9/12/07, Francis Cianfrocca <garbagecat10 at gmail.com> wrote:> > On 9/12/07, Michael S. Fischer <michael at dynamine.net> wrote: > > > > > > No. On Solaris 7+, select() wraps /dev/poll, which is pretty fast, > > though not quite as fast as kqueue. Solaris 10 introduces a new event > > completion framework, which many are calling IOCP (not to be confused > > with IOCP in Windows). See > > http://developers.sun.com/solaris/articles/event_completion.html > > > > Best regards, > > > > --Michael > > _ > > > > Have you used it? Any good? > > > _______________________________________________ > Eventmachine-talk mailing list > Eventmachine-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/eventmachine-talk >-- Tony Arcieri ClickCaster, Inc. tony at clickcaster.com 720-227-0129 ext. 202 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070912/08e5ba06/attachment.html
Doesn''t POSIX aio give you async filesystem access? That would save most of my designs at least one thread. On 9/12/07, Tony Arcieri <tony at clickcaster.com> wrote:> POSIX AIO is worthless > > Solaris has /dev/poll, which is a fairly simple API built around writing > pollfds to /dev/poll > > Solaris 10 added an elegant and robust event completion interface > > > On 9/12/07, Francis Cianfrocca <garbagecat10 at gmail.com> wrote: > > > > On 9/12/07, Kirk Haines <wyhaines at gmail.com> wrote: > > > > > On 9/12/07, Francis Cianfrocca <garbagecat10 at gmail.com> wrote: > > > > > > > EM works fine on Mac, BSD, Solaris, Windows, and other platforms that > > > > support the select(2) system call. I''d love to see someone step up and > > > > contribute an implementation of kqueue and IOCP. > > > > > > I''ll second that. I know for a fact that some of my stuff that I have > > > written to use EM is being used on very high load production Solaris > > > systems without problems. > > > > > > However, I''d especially love to see kqueue support. > > > > > > > > Is kqueue supported on Solaris??? That''s news to me. Solaris has that > Posix AIO support, which I played with once, and it SUCKS. (Unless they''ve > improved it since then.) > > > > > > _______________________________________________ > > Eventmachine-talk mailing list > > Eventmachine-talk at rubyforge.org > > http://rubyforge.org/mailman/listinfo/eventmachine-talk > > > > > > -- > Tony Arcieri > ClickCaster, Inc. > tony at clickcaster.com > 720-227-0129 ext. 202 > _______________________________________________ > Eventmachine-talk mailing list > Eventmachine-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/eventmachine-talk >-- --- Thomas H. Ptacek // matasano security read us on the web: http://www.matasano.com/log
On 9/12/07, Thomas Ptacek <tqbf at matasano.com> wrote:> > Doesn''t POSIX aio give you async filesystem access? That would save > most of my designs at least one thread.Have you really found that file i/o is slow enough to be worth doing on a separate thread? If you look carefully in the EM code, you''ll find the traces of the time I started adding event-driven file i/o. I never finished it because it didn''t seem worth the trouble, but maybe I was wrong. There is a lot of things in the file i/o channel that change perceived performance, like the firmware on the disk controller, the behavior of your SAN card if you have one, whether you use hardware or software RAID, and of course the buffering that the kernel itself does. Is there anything unusual about your systems or your software that makes disk i/o particularly latent? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070913/e292c2b8/attachment-0001.html
"Francis Cianfrocca" <garbagecat10 at gmail.com> writes:> I''m using this SMTP implementation (both client and server) in a > production system now. Please send email to this list asap if there > are problems or feature requests.OK -- to get up an runnning your test_smtp (using ruby 1.8.6 (2007-03-13 patchlevel 0) [i686-darwin8.9.1]), I''ve to work out a little patch: Index: version_0/lib/protocols/smtpserver.rb ==================================================================--- version_0/lib/protocols/smtpserver.rb (Revision 530) +++ version_0/lib/protocols/smtpserver.rb (Arbeitskopie) @@ -109,7 +109,8 @@ # def post_init #send_data "220 #{get_server_greeting}\r\n" (ORIGINAL) - (EM.spawn {send_data "220 #{get_server_greeting}\r\n"}).notify + other = self + (EM.spawn {other.send_data "220 #{other.get_server_greeting}\r\n"}).notify end /Peter
On 9/13/07, Peter Ehrenberg <pe at dipe.de> wrote:> > "Francis Cianfrocca" <garbagecat10 at gmail.com> writes: > > > I''m using this SMTP implementation (both client and server) in a > > production system now. Please send email to this list asap if there > > are problems or feature requests. > > OK -- to get up an runnning your test_smtp (using ruby 1.8.6 (2007-03-13 > patchlevel 0) [i686-darwin8.9.1]), I''ve to work out a little patch: > > > Index: version_0/lib/protocols/smtpserver.rb > ==================================================================> --- version_0/lib/protocols/smtpserver.rb (Revision 530) > +++ version_0/lib/protocols/smtpserver.rb (Arbeitskopie) > @@ -109,7 +109,8 @@ > # > def post_init > #send_data "220 #{get_server_greeting}\r\n" (ORIGINAL) > - (EM.spawn {send_data "220 #{get_server_greeting}\r\n"}).notify > + other = self > + (EM.spawn {other.send_data "220 #{other.get_server_greeting > }\r\n"}).notify > end > >Ooooh, interesting. I think I know what happened. I actually tweaked EventMachine#spawn about a day ago. I may have introduced a regression. Thanks for the patch. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070913/bccfb6fa/attachment.html
My SMTP-Server is now ready and in the testing stage. Until now without any issues. I planned to go into production at end of next week. Thanks for your work, Francis! Peter -- Dipl.-Ing. Peter Ehrenberg http://dipe.de/
On 9/26/07, Peter Ehrenberg <pe at dipe.de> wrote:> > My SMTP-Server is now ready and in the testing stage. Until now > without any issues. I planned to go into production at end of next > week.Ours has been in limited production at a couple of sites for a few days now. I added an event-driven DNS resolver, which made it possible to build a complete relaying SMTP server with perhaps 200 lines of extra Ruby, in addition to the library in EM. Please let us know here how your deployment goes. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070926/821dc999/attachment.html