I''ve played around in the past with a number of custom http handlers, but never one dedicated to handling image uploads. We have a rails app that gets around 3 million page views a day and we are releasing a feature to let people upload images. So I''m looking for a good little server that can handle this task and something EM based came to mind. Even with threads I''m pretty sure I would have to run multiple instances of an em solution to avoid blocking. Not sure how many threads I could run concurrently before ruby''s green threads start to block other uploads. Running multiple instances isn''t a big deal, we already run a number of mongrel clusters which we manage directly with monit instead of relying on mongrel_cluster. So we are already setup for that type of deployment. I was thinking a good strategy with EM would be to buffer uploads in memory until they are complete, then write them to disk in a thread. Then run enough instances so each instance only gets 5 or so max concurrent requests. Anyone have experience with this kind of setup? The images would all be under 1mb, usually around 100k. Chris
The filesystem isn''t asynchronous, but other local IPC is; you can get files "buffered in memory" implicitly by writing them through another socket or pipe in a sidecar process. On Wed, Dec 10, 2008 at 1:27 AM, snacktime <snacktime at gmail.com> wrote:> I''ve played around in the past with a number of custom http handlers, > but never one dedicated to handling image uploads. We have a rails > app that gets around 3 million page views a day and we are releasing a > feature to let people upload images. So I''m looking for a good little > server that can handle this task and something EM based came to mind. > Even with threads I''m pretty sure I would have to run multiple > instances of an em solution to avoid blocking. Not sure how many > threads I could run concurrently before ruby''s green threads start to > block other uploads. Running multiple instances isn''t a big deal, we > already run a number of mongrel clusters which we manage directly with > monit instead of relying on mongrel_cluster. So we are already setup > for that type of deployment. > > I was thinking a good strategy with EM would be to buffer uploads in > memory until they are complete, then write them to disk in a thread. > Then run enough instances so each instance only gets 5 or so max > concurrent requests. Anyone have experience with this kind of setup? > The images would all be under 1mb, usually around 100k. > > Chris > _______________________________________________ > Eventmachine-talk mailing list > Eventmachine-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/eventmachine-talk >-- --- Thomas H. Ptacek // matasano security read us on the web: http://www.matasano.com/log -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://rubyforge.org/pipermail/eventmachine-talk/attachments/20081210/2847ed5f/attachment-0001.html>
> I was thinking a good strategy with EM would be to buffer uploads in > memory until they are complete, then write them to disk in a thread. > Then run enough instances so each instance only gets 5 or so max > concurrent requests. Anyone have experience with this kind of setup? > The images would all be under 1mb, usually around 100k.You can also use nginx for this purpose, with the nginx upload plugin (http://brainspl.at/articles/2008/07/20/nginx-upload-module). Nginx will buffer the upload to a file and pass the filename to your app once the upload is complete. If you need to resize the image without blocking, you can use EM.popen("convert -resize ...") Aman
On Dec 10, 2008, at 12:21 AM, Aman Gupta wrote:>> I was thinking a good strategy with EM would be to buffer uploads in >> memory until they are complete, then write them to disk in a thread. >> Then run enough instances so each instance only gets 5 or so max >> concurrent requests. Anyone have experience with this kind of setup? >> The images would all be under 1mb, usually around 100k. > > You can also use nginx for this purpose, with the nginx upload plugin > (http://brainspl.at/articles/2008/07/20/nginx-upload-module). Nginx > will buffer the upload to a file and pass the filename to your app > once the upload is complete. > > If you need to resize the image without blocking, you can use > EM.popen("convert -resize ...")Yeah you best bet is to use said nginx module. It''s what we have customers use once they need to handle tons of uploads and it is dramatically more scalable then doing this in ruby even with eventmachine. Cheers- Ezra Zygmuntowicz ez at engineyard.com
> > You can also use nginx for this purpose, with the nginx upload plugin > (http://brainspl.at/articles/2008/07/20/nginx-upload-module). Nginx > will buffer the upload to a file and pass the filename to your app > once the upload is complete.That looks very promising. Chris