Hi. When I upload a 10Mb file (from my browser) to my Rails app, a Tempfile object gets created on the server. When the contents gets written to the database, it''s a simple INSERT, so basically the data is read to memory and then inserted into MySQL. How come the data is not streamed to the DB so we avoid the memory issue? Is this a short coming of ActiveRecord or the MySQL/Ruby driver or something else? I guess that it''s not possible to stream from the database either? (At least the API doesn''t appear to support this). Any insight much appreciated. Morten
Simple solution: Don''t store files in a db. Streaming files to the db is not part of the SQL standard so we would have to support adapter specific extensions to get this done. Storing files in the db is never a good idea however so there shouldn''t be a problem that this corner case isn''t well served by the adapter. Your file system is already a database for files. Its several lorders of magnitude more efficient then a database can ever hope to be. On 4/29/06, Morten <lists@kikobu.com> wrote:> > Hi. > > When I upload a 10Mb file (from my browser) to my Rails app, a Tempfile > object gets created on the server. When the contents gets written to the > database, it''s a simple INSERT, so basically the data is read to memory > and then inserted into MySQL. > > How come the data is not streamed to the DB so we avoid the memory > issue? Is this a short coming of ActiveRecord or the MySQL/Ruby driver > or something else? > > I guess that it''s not possible to stream from the database either? (At > least the API doesn''t appear to support this). > > Any insight much appreciated. > > Morten > > _______________________________________________ > Rails mailing list > Rails@lists.rubyonrails.org > http://lists.rubyonrails.org/mailman/listinfo/rails >-- Tobi http://shopify.com - modern e-commerce software http://typo.leetsoft.com - Open source weblog engine http://blog.leetsoft.com - Technical weblog
Thanks for the clarification. I disagree that adhering to the SQL standard should be a show stopper for supporting streaming, and suspect that it would be in place if it was relatively easy to support. I store files in the DB for backup simplicity and explicit referential integrity, but I do serve cached versions to the user off the FS. Br, Morten Tobias L?tke wrote:> Simple solution: Don''t store files in a db. Streaming files to the db > is not part of the SQL standard so we would have to support adapter > specific extensions to get this done. > Storing files in the db is never a good idea however so there > shouldn''t be a problem that this corner case isn''t well served by the > adapter. > > Your file system is already a database for files. Its several lorders > of magnitude more efficient then a database can ever hope to be. > > On 4/29/06, Morten <lists@kikobu.com> wrote: >> >> Hi. >> >> When I upload a 10Mb file (from my browser) to my Rails app, a Tempfile >> object gets created on the server. When the contents gets written to the >> database, it''s a simple INSERT, so basically the data is read to memory >> and then inserted into MySQL. >> >> How come the data is not streamed to the DB so we avoid the memory >> issue? Is this a short coming of ActiveRecord or the MySQL/Ruby driver >> or something else? >> >> I guess that it''s not possible to stream from the database either? (At >> least the API doesn''t appear to support this). >> >> Any insight much appreciated. >> >> Morten >> >> _______________________________________________ >> Rails mailing list >> Rails@lists.rubyonrails.org >> http://lists.rubyonrails.org/mailman/listinfo/rails >> > > > -- > Tobi > http://shopify.com - modern e-commerce software > http://typo.leetsoft.com - Open source weblog engine > http://blog.leetsoft.com - Technical weblog