Hi there, I''m making a site that does a lot of image processing. In order to avoid timeouts while the clients image is being processed I have used backgroundrb to process the images. However, I have a few small issues: 1. Images tend to come in short, intense bursts. I want all the images to be processed at the same time, but it seems like drb is processing the images in sequence, with each image waiting for the next to be processed before it starts processing. Is there any way to make drb process them all concurrently? 2. Sometimes my worker just doesn''t start. I can''t find any errors anywhere. But looking the log indicated that the worker never got the message to start. I can''t see any correlation between image size, type or anything. It seems pretty random but happens about once every 10 images. If it was a problem with my code I would expect to see an error message somewhere, but I can''t find one. I am assuming that something goes wrong, or gets lost, and that the error is never logged. Does anyone have any suggestions? Thanks, Jonzo. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
Hi Jonzo, I''m also using backgroundrb for images processing. Mine are processing the images one at a time - not really an issue for my application, after all the application has returned control to the user. I''m not getting any problems with ''bursts'' of activity or missing workers btw I''m using the latest backgroundrb. Processing time seems very linear. Deployment OS is Centos, RMagick for processing and Mysql for DB ( images stored in filesystem). regards Kevin On Apr 10, 5:27 am, Jonzo <j.fant...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:> Hi there, > > I''m making a site that does a lot of image processing. In order to > avoid timeouts while the clients image is being processed I have used > backgroundrb to process the images. > > However, I have a few small issues: > > 1. Images tend to come in short, intense bursts. I want all the > images to be processed at the same time, but it seems like drb is > processing the images in sequence, with each image waiting for the > next to be processed before it starts processing. Is there any way to > make drb process them all concurrently? > > 2. Sometimes my worker just doesn''t start. I can''t find any errors > anywhere. But looking the log indicated that the worker never got the > message to start. I can''t see any correlation between image size, type > or anything. It seems pretty random but happens about once every 10 > images. If it was a problem with my code I would expect to see an > error message somewhere, but I can''t find one. I am assuming that > something goes wrong, or gets lost, and that the error is never > logged. > > Does anyone have any suggestions? > > Thanks, > Jonzo.--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
Thanks Kevin, it''s good to know that it does work for some people!
The bursts of activity isn''t actually a bug or problem, it''s
just the
way my application behaves. Typically people will upload 10 or so
images at once so I am starting that many workers. It just seems like
sometimes the workers don''t start, and when one of the workers
doesn''t
start there is nothing printed out in any log files. no errors, and
none of the log entries I have put in my code either.
Can you see any errors in my logic/code?
Here is my worker code:
===========================================
class FileProcessorWorker < BackgrounDRb::MetaWorker
set_worker_name :file_processor_worker
def create(args = nil)
# this method is called, when worker is loaded for the first time
end
def process( file_id )
begin
logger.info "processing file now"
file = UploadedFile.find( file_id )
logger.info "file: #{ file.inspect }"
if file && file.original
file.process
logger.info "file processing completed"
else
logger.info "file not processed"
end
rescue
logger.info "file processing error: #{ $! }"
end
end
end
===========================================
And I am starting my worker like this (after my model is created):
def start_worker
MiddleMan.worker(:file_processor_worker).process(id)
end
Thanks,
Jonzo.
On Apr 11, 2:09 am, kevin evans
<kwev...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
wrote:> Hi Jonzo,
>
> I''m also using backgroundrb for images processing. Mine are
processing
> the images one at a time - not really an issue for my application,
> after all the application has returned control to the user. I''m
not
> getting any problems with ''bursts'' of activity or missing
workers btw
> I''m using the latest backgroundrb. Processing time seems very
linear.
> Deployment OS is Centos, RMagick for processing and Mysql for DB
> ( images stored in filesystem).
>
> regards
> Kevin
>
> On Apr 10, 5:27 am, Jonzo
<j.fant...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
> > Hi there,
>
> > I''m making a site that does a lot of image processing. In
order to
> > avoid timeouts while the clients image is being processed I have used
> > backgroundrb to process the images.
>
> > However, I have a few small issues:
>
> > 1. Images tend to come in short, intense bursts. I want all the
> > images to be processed at the same time, but it seems like drb is
> > processing the images in sequence, with each image waiting for the
> > next to be processed before it starts processing. Is there any way to
> > make drb process them all concurrently?
>
> > 2. Sometimes my worker just doesn''t start. I can''t
find any errors
> > anywhere. But looking the log indicated that the worker never got the
> > message to start. I can''t see any correlation between image
size, type
> > or anything. It seems pretty random but happens about once every 10
> > images. If it was a problem with my code I would expect to see an
> > error message somewhere, but I can''t find one. I am assuming
that
> > something goes wrong, or gets lost, and that the error is never
> > logged.
>
> > Does anyone have any suggestions?
>
> > Thanks,
> > Jonzo.
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---
I have a worker in BackgroundRB that i am using to process large amounts of data returned from the facebook api. The problem is that as I''m walking through the results, it''s an array of hashes for some reason the response is lost or something, i get nil object errrors (http://pastie.caboo.se/185350). I can get through like 90 elements and then.. death. The response is not the problem, it all shows up when i have it displayed in an output in a view but in the worker it can''t get past like 90 elements. Is there some kinda of cache limit or something or a setting taht i have to change to get this to work properly and not lose my junk. Any help would be appreciated. , --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---