Hello, I''ve been using NFS for sharing files across a cluster of servers but I''ve had a lot of problems with that. I''m planning to move to MogileFS which seems to be more stable and more adapted to this usage (lots of user uploaded files). I wonder if someone has already worked on an integration of MogileFS into Mongrel (and Rails app) for serving the files ? If not, I think of writing a handler that would act as a bridge between the MogileFS keys and the actual files. I could use Perlbal from Danga but it involves a new technology (Perl) and a new setup. What do you think about this ? Thanks for all, Philippe
Hi Philippe, On 2.11.2006, at 11.32, Philippe Jayet wrote:> Hello, > > I''ve been using NFS for sharing files across a cluster of servers but > I''ve had a lot of problems with that. I''m planning to move to MogileFS > which seems to be more stable and more adapted to this usage (lots of > user uploaded files). > > I wonder if someone has already worked on an integration of MogileFS > into Mongrel (and Rails app) for serving the files ? If not, I > think of > writing a handler that would act as a bridge between the MogileFS keys > and the actual files.Have a look at Robot Co-op''s MogileFS client (http:// blog.segment7.net/articles/2006/03/22/mogilefs-for-ruby). //jarkko -- Jarkko Laine http://jlaine.net http://odesign.fi -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2417 bytes Desc: not available Url : http://rubyforge.org/pipermail/mongrel-users/attachments/20061102/88facb12/attachment.bin
> > What do you think about this ?There is a mogile-fs client from robotcoop http://dev.robotcoop.com/Libraries/mogilefs/index.html that can be easily integrated with your Rails app. Jonathan -- Jonathan Weiss http://blog.innerewut.de
Hi,> There is a mogile-fs client from robotcoop > > http://dev.robotcoop.com/Libraries/mogilefs/index.html > > that can be easily integrated with your Rails app.I''ll definitely be using this library in my application. But I need some way to serve the MogileFS files (mainly images and audio files) and I don''t want to use Rails for this. All the overhead is unnecessary. To serve the MogileFS files, Robot Co-op are using their webrick-high-performance server with a servlet for accessing MogileFS (through their mogilefs-client library). I think it would be easier in my setup to use Mongrel for this task. By the way, my setup is : Apache 2.2 load balancing requests to Mongrel/Rails. Another option would be to use Apache to serve the MogileFS files. Thanks for your help, Philippe
mod_xsendfile is pretty nice for serving files: http://celebnamer.celebworld.ws/stuff/mod_xsendfile/ Especially if you don''t want to put all your files in the public directory. -Steven Philippe Jayet wrote:> Hi, > > >> There is a mogile-fs client from robotcoop >> >> http://dev.robotcoop.com/Libraries/mogilefs/index.html >> >> that can be easily integrated with your Rails app. >> > > I''ll definitely be using this library in my application. But I need some > way to serve the MogileFS files (mainly images and audio files) and I > don''t want to use Rails for this. All the overhead is unnecessary. > > To serve the MogileFS files, Robot Co-op are using their > webrick-high-performance server with a servlet for accessing MogileFS > (through their mogilefs-client library). I think it would be easier in > my setup to use Mongrel for this task. > > By the way, my setup is : Apache 2.2 load balancing requests to > Mongrel/Rails. Another option would be to use Apache to serve the > MogileFS files. > > Thanks for your help, > > Philippe > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >
On 11/2/06, Philippe Jayet <hip at lan-attack.ch> wrote:> Hi, > > > There is a mogile-fs client from robotcoop > > > > http://dev.robotcoop.com/Libraries/mogilefs/index.html > > > > that can be easily integrated with your Rails app. > > I''ll definitely be using this library in my application. But I need some > way to serve the MogileFS files (mainly images and audio files) and I > don''t want to use Rails for this. All the overhead is unnecessary. > > To serve the MogileFS files, Robot Co-op are using their > webrick-high-performance server with a servlet for accessing MogileFS > (through their mogilefs-client library). I think it would be easier in > my setup to use Mongrel for this task. > > By the way, my setup is : Apache 2.2 load balancing requests to > Mongrel/Rails. Another option would be to use Apache to serve the > MogileFS files. > > Thanks for your help, > > PhilippeYou should be able to write one very easily. I have a mongrel_send_file plugin (unrelated to the sendfile header) that might be a good reference: http://rubyforge.org/projects/msend-file/ -- Rick Olson http://weblog.techno-weenie.net http://mephistoblog.com
Hi,> mod_xsendfile is pretty nice for serving files: > > http://celebnamer.celebworld.ws/stuff/mod_xsendfile/Yes, Zed posted that link some time ago. I''ll have to give it a try, seems interesting and still maintained ...> Especially if you don''t want to put all your files in the public directory.In the case of MogileFS, it creates a directory structure of its own to scale to a large number of files. So serving the files from these directories is not the solution I think of. And I plan to use MogileFS in HTTP mode (not NFS), so the files may even be unaccessible from the filesystem. That is why I think of developing a dynamic component that will ask the MogileFS tracker for the URL of the file (which may be local or not), get this file and return it to the client. Philippe
Philippe Jayet wrote:>In the case of MogileFS, it creates a directory structure of its own to >scale to a large number of files. So serving the files from these >directories is not the solution I think of. And I plan to use MogileFS >in HTTP mode (not NFS), so the files may even be unaccessible from the >filesystem. That is why I think of developing a dynamic component that >will ask the MogileFS tracker for the URL of the file (which may be >local or not), get this file and return it to the client. > >I see two solutions for you here: 1. Write a Rails application that uses the mogilefs-client stuff to save/retrieve files as users create/request them (as MogileFS was intended to be used). 2. Use the FUSE mogilefs client to mount it as a filesystem for your entire rails project, and have mongrel serve up files as if they were static directly from public/. Knowing the state of the FUSE client the last time I looked at it, I would probably go with the former solution rather than the latter. There are other relational-master-metadata-distributed-replicated-datastore GRID style filesystems out there somewhat similar to MogileFS (like GFarm, CleverSafe, or even Hadoop), but MogileFS is probably one of your better bets when it comes to simplicity and a proven track record for Rails development/integration at the moment (with mogilefs-client). I suppose, in theory, you could modify mongrel to use mogilefs as a filesystem abstraction layer, but I really don''t see the need for that code bloat. Smaller. Simpler. Faster. - Ian C. Blenke <ian at blenke.com> http://ian.blenke.com/
On Thu, 02 Nov 2006 10:32:15 +0100 Philippe Jayet <hip at lan-attack.ch> wrote:> Hello, > > I''ve been using NFS for sharing files across a cluster of servers but > I''ve had a lot of problems with that. I''m planning to move to MogileFS > which seems to be more stable and more adapted to this usage (lots of > user uploaded files). > > I wonder if someone has already worked on an integration of MogileFS > into Mongrel (and Rails app) for serving the files ? If not, I think of > writing a handler that would act as a bridge between the MogileFS keys > and the actual files. > > I could use Perlbal from Danga but it involves a new technology (Perl) > and a new setup.Yep, this is totally possible depending on how you need to use it. If you just need to serve files out of mogilefs then look at the FUSE adapter and have your main web server do the work using either x-sendfile or plain old a-href. (many others mentioned this) If you''re looking to do the upload and download with a bit of smarts behind it, then you''re looking at doing a Mongrel handler or even set of Mongrel handlers. There''s plenty of great examples you can leverage with Rick''s mongrel_send_file and Ezra''s merb stuff as examples. And of course, I''ll pimp the book since it covers writing Mongrel handlers and how Mongrel is designed. :-) http://safari.oreilly.com/0321483502 Good luck, and when you do it, try to open source it since others will probably like something like this. -- Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu http://www.zedshaw.com/ http://safari.oreilly.com/0321483502 -- The Mongrel Book http://mongrel.rubyforge.org/ http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.