thr3ads.net - Rails - Follow-Up: Receiving Inbound Email and Creating Data in Rails App [Apr 2007]

If this information is useful, please help other people find it:
Share via:

Nathan Leach

2007-Apr-06 19:18 UTC

Follow-Up: Receiving Inbound Email and Creating Data in Rails App

I just wanted to post back to the group with my potential solution for
receiving inbound email and creating data in my Rails application, including
attached images.  Hope this will be helpful for others.  Of course,
suggestions for improving my ideas is always welcome.

Thanks to everyone who took the time to post helpful responses to my
original (multiple) inquiries.

Nathan

Overall Requirement

I needed a feature in my Rails application that would receive inbound emails
(from a Palm Treo 680, in particular) including an attached image.  I wanted
to use this data to create an entry in a product database for an online
shopping web site.  This requirement supports a small business owner who
travels, finds products to buy and resell, and wanted immediate (easy)
posting of available items on the site using a mobile device.

Design:

I decided to use a plain text email, with a defined format...

   - Subject = product title
   - 1st line = product description
   - 2nd line = product price
   - 3rd line = product cost (for accounting application sync)
   - 4th line = product quantity
   - Attachment = product image

I created a "mailer" model inheriting from ActionMailer.  The mailer
model
had two important methods (among others).  The first was the "check"
method
which will use Net::IMAP to poll the INBOX folder for new mail.  The check
method would then call the "receive" method which will take an
individual
mail, create a product, and create an associated image in the database.
Finally, there is a "delete" method which removes the processed mail
from
the inbox.

PROBLEM: I could not get IMAP to allow deletion of the old messages.  I
actually had to use POP3 to do it.  Something wacky here that may just be my
ignorance on this topic.

One of the bigger questions for me was how to periodically process the
mail.  There were several possibilities here and may well be others that I
didn''t know about or consider.  Here are a couple of the more popular
ones...

   - Use a server script to periodically "push" the mail into the
Rails
   app using cron
   - Use BackgroundRb to create a Rails background process for reading
   the mail periodically

However, I chose something a little simpler and possibly weird.  I used the
application controller to call the check mail function once per user/browser
session.  So inside the application.rb I have a before filter that calls a
method.  The method checks for a session variable that indicates whether
mail has been checked yet or not.  If not, it calls the model method for
doing so.  I don''t know the complete implications or trade-offs here,
so
this is an area where I would be happy to get input from Ruby/Rails
experts.  Given the low usage of the site and small number of products being
entered, I chose to use this given that the only downside seemed to be a
small wait (1-2 seconds) on the initial site load for the mail to be checked
and the parts/images loaded.

Development/Code:

class Mailer < ActionMailer::Base
.
.
.
  def self.check_mail
    begin
      imap = Net::IMAP.new(''mail.***.com'')
      if RAILS_ENV == "production"
        imap.authenticate(''LOGIN'',
''prduser'', ''prdpasswd'')
      else
        imap.authenticate(''LOGIN'',
''devuser'', ''devpasswd'')
      end
      imap.examine(''INBOX'')
      imap.search([''FROM'',
''validuser@***.com'']).each do |message_id|
        msg =
imap.fetch(message_id,''RFC822'')[0].attr[''RFC822'']
        envelope = imap.fetch(message_id,
"ENVELOPE")[0].attr["ENVELOPE"]
        Mailer.receive(msg)
      end
      imap.logout
      return true
    rescue
      RAILS_DEFAULT_LOGGER.error("Mailer Import Error: " + $!)
      return false
    end
  end

  def receive(email) return unless email.has_attachments?
    emailbody = email.body.to_s.gsub(/\r/, '' '')
    part = Part.new
    part.description = email.subject
    part.notes = emailbody.split(/\n/)[0]
    part.sellprice = emailbody.split(/\n/)[1].to_f
    part.lastcost = emailbody.split(/\n/)[2].to_f
    part.onhand = emailbody.split(/\n/)[3].to_i
    part.partsgroup_id = 10280
    if part.save!
      email.attachments.each do |attachment|
        next unless attachment.original_filename[-4..-1] ==
''.jpg''
        picture = Picture.new
        picture.name = email.subject
        picture.part_id = part.id
        picture.imagedata = attachment.read
        picture.content_type = ''image/jpeg''
        picture.save!
      end
    end
  end

  def self.delete_mail
    if RAILS_ENV =="production"
      Net::POP3.delete_all(''mail.***.com'', 110,
''prduser'', ''prdpasswd'')
    else
      Net::POP3.delete_all(''mail.***.com'', 110,
''devuser'', ''devpasswd'')
    end
  rescue
    RAILS_DEFAULT_LOGGER.error("Mailer Import Error: " + $!)
  end
end

class ApplicationController < ActionController::Base
  before_filter :env

  def env
    if !session[:checked_mail]
      if Mailer.check_mail
        session[:checked_mail] = true
        Mailer.delete_mail
      end
    end
  end
.
.
.
end

Testing:

So far, this solution seems to work pretty well.  I have not tested large
numbers of emails or large numbers of concurrent users.  This is only a
potential solution and not one I would recommend for a larger site. 
Let''s
have it...what do you think?  How can this be improved?

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---

Jason Edgecombe

2007-Apr-06 22:47 UTC

head link

Re: Follow-Up: Receiving Inbound Email and Creating Data in Rails App

Nathan Leach wrote:> I just wanted to post back to the group with my potential solution for 
> receiving inbound email and creating data in my Rails application, 
> including attached images.  Hope this will be helpful for others.  Of 
> course, suggestions for improving my ideas is always welcome.
>
> Thanks to everyone who took the time to post helpful responses to my 
> original (multiple) inquiries.
>
> Nathan
>
> Overall Requirement
>
> I needed a feature in my Rails application that would receive inbound 
> emails (from a Palm Treo 680, in particular) including an attached 
> image.  I wanted to use this data to create an entry in a product 
> database for an online shopping web site.  This requirement supports a 
> small business owner who travels, finds products to buy and resell, 
> and wanted immediate (easy) posting of available items on the site 
> using a mobile device.
>
> Design:
>
> I decided to use a plain text email, with a defined format...
>
>     * Subject = product title
>     * 1st line = product description
>     * 2nd line = product price
>     * 3rd line = product cost (for accounting application sync)
>     * 4th line = product quantity
>     * Attachment = product image
>
> I created a "mailer" model inheriting from ActionMailer.  The
mailer
> model had two important methods (among others).  The first was the 
> "check" method which will use Net::IMAP to poll the INBOX folder
for
> new mail.  The check method would then call the "receive" method
which
> will take an individual mail, create a product, and create an 
> associated image in the database.  Finally, there is a "delete"
method
> which removes the processed mail from the inbox.
>
> PROBLEM: I could not get IMAP to allow deletion of the old messages.  
> I actually had to use POP3 to do it.  Something wacky here that may 
> just be my ignorance on this topic.
>
> One of the bigger questions for me was how to periodically process the 
> mail.  There were several possibilities here and may well be others 
> that I didn''t know about or consider.  Here are a couple of the
more
> popular ones...
>
>     * Use a server script to periodically "push" the mail into
the
>       Rails app using cron
>     * Use BackgroundRb to create a Rails background process for
>       reading the mail periodically
>
> However, I chose something a little simpler and possibly weird.  I 
> used the application controller to call the check mail function once 
> per user/browser session.  So inside the application.rb I have a 
> before filter that calls a method.  The method checks for a session 
> variable that indicates whether mail has been checked yet or not.  If 
> not, it calls the model method for doing so.  I don''t know the 
> complete implications or trade-offs here, so this is an area where I 
> would be happy to get input from Ruby/Rails experts.  Given the low 
> usage of the site and small number of products being entered, I chose 
> to use this given that the only downside seemed to be a small wait 
> (1-2 seconds) on the initial site load for the mail to be checked and 
> the parts/images loaded.
>
> Development/Code:
>
> class Mailer < ActionMailer::Base
> .
> .
> .
>   def self.check_mail
>     begin
>       imap = Net::IMAP.new(''mail.***.com'')
>       if RAILS_ENV == "production"
>         imap.authenticate (''LOGIN'',
''prduser'', ''prdpasswd'')
>       else
>         imap.authenticate(''LOGIN'',
''devuser'', ''devpasswd'')
>       end
>       imap.examine(''INBOX'')
>       imap.search([''FROM'',
''validuser@***.com'']).each do |message_id|
>         msg =
imap.fetch(message_id,''RFC822'')[0].attr[''RFC822'']
>         envelope = imap.fetch (message_id,
"ENVELOPE")[0].attr["ENVELOPE"]
>         Mailer.receive(msg)
>       end
>       imap.logout
>       return true
>     rescue
>       RAILS_DEFAULT_LOGGER.error("Mailer Import Error: " + $!)
>       return false
>     end
>   end 
>  
>   def receive(email) return unless email.has_attachments?
>     emailbody = email.body.to_s.gsub(/\r/, '' '')
>     part = Part.new
>     part.description = email.subject
>     part.notes = emailbody.split(/\n/)[0]
>     part.sellprice = emailbody.split(/\n/)[1].to_f
>     part.lastcost = emailbody.split(/\n/)[2].to_f
>     part.onhand = emailbody.split(/\n/)[3].to_i
>     part.partsgroup_id = 10280
>     if part.save!
>       email.attachments.each do |attachment|
>         next unless attachment.original_filename[-4..-1] ==
''.jpg''
>         picture = Picture.new
>         picture.name <http://picture.name> = email.subject
>         picture.part_id = part.id <http://part.id>
>         picture.imagedata = attachment.read
>         picture.content_type = ''image/jpeg''
>         picture.save!     
>       end
>     end
>   end
>
>   def self.delete_mail
>     if RAILS_ENV =="production"
>       Net::POP3.delete_all(''mail.***.com'', 110,
''prduser'', ''prdpasswd'')
>     else
>       Net::POP3.delete_all(''mail.***.com'', 110,
''devuser'', ''devpasswd'')
>     end
>   rescue
>     RAILS_DEFAULT_LOGGER.error("Mailer Import Error: " + $!)
>   end
> end
>
> class ApplicationController < ActionController::Base
>   before_filter :env
>  
>   def env
>     if !session[:checked_mail]
>       if Mailer.check_mail
>         session[:checked_mail] = true
>         Mailer.delete_mail
>       end
>     end
>   end
> .
> .
> .
> end
>
> Testing:
>
> So far, this solution seems to work pretty well.  I have not tested 
> large numbers of emails or large numbers of concurrent users.  This is 
> only a potential solution and not one I would recommend for a larger 
> site.  Let''s have it...what do you think?  How can this be
improved?hmmm,

The biggest thing is as you said, checking the email on each session 
will not scale. In fact, the first request for each user will have to 
wait on the mail to fetched and parsed. cron or backgroundrb would be 
much better for that.

I think there should be a more rubyesque way for the emailbody.split 
lines but nothing comes to mind.

You should put the dev and production email passwords into the 
environment  in the config folder along with the database password. Then 
just use the value set by the environment.


Sincerely,
Jason

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---

Nathan Leach

2007-Apr-07 22:25 UTC

head link

Re: Follow-Up: Receiving Inbound Email and Creating Data in Rails App

On Fri, 2007-04-06 at 18:47 -0400, Jason Edgecombe wrote:
> 
> The biggest thing is as you said, checking the email on each session 
> will not scale. In fact, the first request for each user will have to 
> wait on the mail to fetched and parsed. cron or backgroundrb would be 
> much better for that.
> 
> I think there should be a more rubyesque way for the emailbody.split 
> lines but nothing comes to mind.
> 
> You should put the dev and production email passwords into the 
> environment  in the config folder along with the database password. Then 
> just use the value set by the environment.
> 
> 
> Sincerely,
> Jason
> 
Jason,

Great points.  I will move the settings to the config folder and try to
find another way to process the email body.

I am considering the other two background email processing options.  I
do have another idea to improve the current call to the Mailer model.
Not sure how feasible it is for production, but it seems to be working
in development ok.

I was wondering about using a process fork to call the email
check/receive.  Take a look at this code that would reside in the
application controller...

    if !session[:checked_mail]
      Process.detach fork {
        if Mailer.check_mail
          session[:checked_mail] = true
        end
      }
    end

Would it alleviate the performance and/or scaling issues?  Will this
cause problems?

Nathan

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---

Julien Delgoulet

2007-Apr-11 18:34 UTC

head link

Re: Follow-Up: Receiving Inbound Email and Creating Data in

Nathan Leach wrote:> On Fri, 2007-04-06 at 18:47 -0400, Jason Edgecombe wrote:
> 
>> environment  in the config folder along with the database password.
Then
>> just use the value set by the environment.
>> 
>> 
>> Sincerely,
>> Jason
>> 
> 
> Jason,
> 
> Great points.  I will move the settings to the config folder and try to
> find another way to process the email body.
> 
> I am considering the other two background email processing options.  I
> do have another idea to improve the current call to the Mailer model.
> Not sure how feasible it is for production, but it seems to be working
> in development ok.
> 
> I was wondering about using a process fork to call the email
> check/receive.  Take a look at this code that would reside in the
> application controller...
> 
>     if !session[:checked_mail]
>       Process.detach fork {
>         if Mailer.check_mail
>           session[:checked_mail] = true
>         end
>       }
>     end
> 
> Would it alleviate the performance and/or scaling issues?  Will this
> cause problems?
> 
> Nathan

Hi nathan,

May be you need to look to check backgroundrb and create a scheduled 
worker that will check for email every x minutes .. or hours ...

I will have a try soon as I want my users to be able to upload photos by 
email ;-)

Backgroundrb is here : http://backgroundrb.rubyforge.org/

Note that it can integrate with rails easily ;-) as long as you work on 
a unix machine.

Happy rails coding everyone !


-- 
Posted via http://www.ruby-forum.com/.

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---

Nils Franzen

2007-Apr-11 19:02 UTC

head link

Re: Follow-Up: Receiving Inbound Email and Creating Data in

> Backgroundrbis here :http://backgroundrb.rubyforge.org/
>
> Note that it can integrate with rails easily ;-) as long as you work on
> a unix machine.

The latest Backgroundrb (0.2.1) runs fine on cygwin (=unix-like env
for windows)

/Nils


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---

Rails - Apr 2007 - Follow-Up: Receiving Inbound Email and Creating Data in Rails App

Follow-Up: Receiving Inbound Email and Creating Data in Rails App

Re: Follow-Up: Receiving Inbound Email and Creating Data in Rails App

Re: Follow-Up: Receiving Inbound Email and Creating Data in Rails App

Re: Follow-Up: Receiving Inbound Email and Creating Data in

Re: Follow-Up: Receiving Inbound Email and Creating Data in