Hello Folks, Has anyone run into anything like ticket 1189? Does the repro code seem sane? I''ve heard others haven''t seen this issue with blobs sized upwards of 100M. rails seems to be sql escaping the blob before the insert. is this standard procedure for blobs? i figured one might be able to pass in a file descriptor or the like to have the mysql driver stream the blob into the column, but it seems to read the entire thing into a string buffer, sql-escape it, and send it into the query processor. doesn''t seem very scalable to me, given that longblobs can go to 1gb. thoughts? -mcclain
It would most likely require the fields being left as a parameter (?) and then having the blob data passed up to the database as a parameter value. Unfortunately, ActiveRecord is setup currently to switch everything into a string and pass only a sql query up to the database. The oracle driver gets around this when inserting a sequence based primary key but that''s a case that is really specific and almost prepared for with ActiveRecord. I''m looking at ways of getting around this within ActiveRecord myself. I have a couple of thoughts but I''m not sure any of them would work. The best thought would be to have the quote function handle blobs differently (leaving them as a parameter) and then storing the actual data somewhere until the execute statement is called so it can be pushed up. The problem is that currently we don''t always pass up the column to the quote function and that would be required to understand whether or not you where quoting a blob field. I''ve got the column being now passed up in at least the majority of cases, I''m just trying to make sure I''m getting all the possible calls into quote handled. Then I''m onto trying to find an appropriate location to store the data while waiting for the insert statement and then making sure the data gets deleted when finished with. On 4/25/05, McClain Looney <m@loonsoft.com> wrote:> Hello Folks, > > Has anyone run into anything like ticket 1189? Does the repro code seem > sane? I''ve heard others haven''t seen this issue with blobs sized > upwards of 100M. > _______________________________________________ > Rails mailing list > Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org > http://lists.rubyonrails.org/mailman/listinfo/rails >-- John W Higgins wishdev-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
On Apr 25, 2005, at 11:54 AM, John Higgins wrote:> It would most likely require the fields being left as a parameter (?) > and then having the blob data passed up to the database as a parameter > value. Unfortunately, ActiveRecord is setup currently to switch > everything into a string and pass only a sql query up to the database. > The oracle driver gets around this when inserting a sequence based > primary key but that''s a case that is really specific and almost > prepared for with ActiveRecord.dang, i was hoping it was something i was doing. have you had any success with your approach? -mcclain
I''m moving down my list of issues with creating a Firebird Adapter - it''s on the list with what I described as the first approach I''m going to try. Should be moving on it fairly soon and if it works - it could be moved to other adapters fairly easily I believe. I''m really looking to get the ability to deal with parameters better however. I''ve got a date/time "issue" which would go totally away if I could only pass up the date string as a parameter instead of inline. -- John W Higgins wishdev-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org On 4/25/05, McClain Looney <m@loonsoft.com> wrote:> > On Apr 25, 2005, at 11:54 AM, John Higgins wrote: > > > It would most likely require the fields being left as a parameter (?) > > and then having the blob data passed up to the database as a parameter > > value. Unfortunately, ActiveRecord is setup currently to switch > > everything into a string and pass only a sql query up to the database. > > The oracle driver gets around this when inserting a sequence based > > primary key but that''s a case that is really specific and almost > > prepared for with ActiveRecord. > > dang, i was hoping it was something i was doing. have you had any > success with your approach? > > > -mcclain > > _______________________________________________ > Rails mailing list > Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org > http://lists.rubyonrails.org/mailman/listinfo/rails >
John Higgins wrote:> It would most likely require the fields being left as a parameter (?) > and then having the blob data passed up to the database as a parameter > value. Unfortunately, ActiveRecord is setup currently to switch > everything into a string and pass only a sql query up to the database.I''ve wished for Active Record to use prepared statements when the database supports them. It''d solve the blob issues and give a performance boost. It doesn''t make sense to scan a 100MB blob in Ruby to quote NULs! The sanitize and quote methods already have the "look" of prepared statements. With a bit of work, this could be pushed down into the adapters with a default implementation in the abstract adapter. It''d elicit a bit of healthy refactoring as well. jeremy
On Apr 25, 2005, at 11:35 AM, McClain Looney wrote:> > thoughts? > >i''m getting the impression that mysql+rails just isn''t ready for storing longblobs. Is this a fair statement? Can my problem be reproduced? is this an issue with the pgsql driver as well? oracle? -mcclain
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jeremy Kemper wrote:> John Higgins wrote: >>It would most likely require the fields being left as a parameter (?) >>and then having the blob data passed up to the database as a parameter >>value. Unfortunately, ActiveRecord is setup currently to switch >>everything into a string and pass only a sql query up to the database. > > I''ve wished for Active Record to use prepared statements when the > database supports them. It''d solve the blob issues and give a > performance boost. It doesn''t make sense to scan a 100MB blob in Ruby > to quote NULs! > > The sanitize and quote methods already have the "look" of prepared > statements. With a bit of work, this could be pushed down into the > adapters with a default implementation in the abstract adapter. It''d > elicit a bit of healthy refactoring as well.Wow, I hd assumed that this was done already since it already has the placeholder look. Yes indeed, it should be passed on to the database driver to deal with in the most optimal way possible. The database driver is also in the best position to know what needs to be escaped and how. - -- David Morton Maia Mailguard server side anti-spam/anti-virus solution: http://www.maiamailguard.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCbUYASIxC85HZHLMRApl5AJ9mSX8aM8r60sZGD8UcgIdJ+WEbGwCgiKET /cMzv48qmY/kadA/qG/Tmak=2VXZ -----END PGP SIGNATURE-----
On 4/26/05, David Morton <mortonda-0/IDydmJJnNeoWH0uzbU5w@public.gmane.org> wrote:> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Jeremy Kemper wrote: > > John Higgins wrote: > >>It would most likely require the fields being left as a parameter (?) > >>and then having the blob data passed up to the database as a parameter > >>value. Unfortunately, ActiveRecord is setup currently to switch > >>everything into a string and pass only a sql query up to the database. > > > > I''ve wished for Active Record to use prepared statements when the > > database supports them. It''d solve the blob issues and give a > > performance boost. It doesn''t make sense to scan a 100MB blob in Ruby > > to quote NULs! > > > > The sanitize and quote methods already have the "look" of prepared > > statements. With a bit of work, this could be pushed down into the > > adapters with a default implementation in the abstract adapter. It''d > > elicit a bit of healthy refactoring as well. > > Wow, I hd assumed that this was done already since it already has the > placeholder look. Yes indeed, it should be passed on to the database > driver to deal with in the most optimal way possible.This issue was raised when I initially delivered the bind-variables functionality. Basically the reason it hasn''t been changed is because so far noone has needed it. I haven''t stored files in blob columns since I realised that oracle was just creating seperate files on disk anyways. The odds of this being changed before 1.0 are fairly slim unless someone who has the motivation for it submits a patch.> The database > driver is also in the best position to know what needs to be escaped and > how.We already delegate to the connection adapter for this, most of them use the database''s libraries.> - -- > David Morton > Maia Mailguard server side anti-spam/anti-virus solution: > http://www.maiamailguard.com > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.2.5 (GNU/Linux) > Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org > > iD8DBQFCbUYASIxC85HZHLMRApl5AJ9mSX8aM8r60sZGD8UcgIdJ+WEbGwCgiKET > /cMzv48qmY/kadA/qG/Tmak> =2VXZ > -----END PGP SIGNATURE----- > _______________________________________________ > Rails mailing list > Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org > http://lists.rubyonrails.org/mailman/listinfo/rails >-- Cheers Koz
On Apr 25, 2005, at 11:35 AM, McClain Looney wrote:> Hello Folks, > > Has anyone run into anything like ticket 1189? Does the repro code > seem sane? I''ve heard others haven''t seen this issue with blobs sized > upwards of 100M. > >apparently this is an issue with mysql. i''ve swapped in sqlite3, and my ingest process works like a charm.