The article _Improving_ActiveRecord_Part_1_ is now online. In it, I explore the current _internal_ communication structure of AR. I want to try to explore this topic non-critically in order to solicit feed back from other Rails developers. The end goal is to provide Rails the solid database connectivity all developers demand. Part 1 explores the problem and illustrates one of the first and most obvious symptoms of the underlying issue: http://www.voxclandestina.com/ruby/rails/2005-05-24/improving-activerecord-part-1/ Jon --- "The Supreme Personality of Godhead said: Time I am, the great destroyer of the worlds, and I have come here to destroy all people. With the exception of you, all the soldiers here on both sides will be slain." (Bagavad Gita 11.32)
I always love these little "slap in the face" posts/blog entries that come up from time to time. First, this has nothing at all to do with improving ActiveRecord''s "solid connectivity" - it has to do with improving a very very small section of ActiveRecord - blob handling. Even ignoring the standard "thou shall/shall not put blobs in databases" arguments - this really has very little to do with ActiveRecord. The problem lies 99.9% from the fact that there is no standard when it comes to handling blobs. Each DB handles it different and on top of that each ruby driver handles it differently. That said there really probably isn''t much that is going to be done to fix the memory issue because the bottom line is that you have to pass something to the driver and in most cases the driver is not going to open a file and process the contents (of course there may be one of the database drivers that can do that). I''m personally working on building an adapter for Firebird and it has issues with blobs at the moment (on top of the fact that the Oracle and DB2 drivers just ignore blobs at the moment). The fix really isn''t that difficult to do and in fact is quite simple - we simply pass it up as a parameter and it can get dealt with by the driver - but it still means that it''s going to get passed up as a "string" so it''s going to get loaded into memory. ActiveRecord, while it does replace parameters with their actual values at one stage of the process - does not in any way prevent the adapters from passing in parameters - other then some small issues with field ordering because AR makes a hash of the fields after they are assigned and that can make a mess of the field ordering. AR is very nice in that it gathers all of the information you could possibly need about the fields and therefore the adapters can easily pick out blobs and primary key fields so they can be implemented as parameters (just look at the way Oracle handles PK''s for an example of parameterizing field values). I will say that I do love how you mention how nice FastCGI is in that it takes very little memory up handling the blob - how does it do that - IT KEEPS THE BLOB IN A FILE - HINT HINT HINT............. I think you would be better off submitting the patch to handle parameterized blobs instead of blogging about "solid database connectivity" -- John W Higgins wishdev-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
On 5/25/05, Jon-Carlos Rivera <imbcmdth-PkbjNfxxIARBDgjK7y7TUQ@public.gmane.org> wrote:> The article _Improving_ActiveRecord_Part_1_ is now online. In it, I explore > the current _internal_ communication structure of AR. I want to try to > explore this topic non-critically in order to solicit feed back from other > Rails developers. The end goal is to provide Rails the solid database > connectivity all developers demand. > > Part 1 explores the problem and illustrates one of the first and most > obvious symptoms of the underlying issue: > http://www.voxclandestina.com/ruby/rails/2005-05-24/improving-activerecord-part-1/I''m pleased you''re looking into this, there probably is some functionality in ActiveRecord::Base which we could move into the adapters. However the kind of major change you''re talking about is *highly* unlikely to be accepted for rails 1.0. Also, for any changes to be accepted they have to be completely backwards compatible with existing code. I''m of the ''blobs are stupid'' school of thought, but if someone else is willing to do the work, I won''t stand in their way. Just bear in mind a few things: 1) Your blob problem is a corner case. 99% of applications don''t do file handling, and those that do can probably use the file system. Backpack handles big files, ask david how. If you propose changes which will either make the easy case hard, or break existing code, it''ll probably get rejected 2) This comes up a *lot*. Everytime the people involved just complain and explain how we don''t *get it*. We do get it, we just don''t need this functionality. Send in patches and we''ll listen, just whinge and you''ll probably be completely ignored.> Jon > --- > "The Supreme Personality of Godhead said: Time I am, the great destroyer of > the worlds, and I have come here to destroy all people. With the exception > of you, all the soldiers here on both sides will be slain." > (Bagavad Gita 11.32) > > > _______________________________________________ > Rails mailing list > Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org > http://lists.rubyonrails.org/mailman/listinfo/rails >-- Cheers Koz
Heya :)> -----Original Message----- > From: rails-bounces-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org [mailto:rails- > bounces-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org] On Behalf Of Michael Koziarski > Sent: Tuesday, May 24, 2005 3:59 PM > To: rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org > Subject: Re: [Rails] Improving ActiveRecord Part 1 > > Everytime the people involved just complain and explain how we don''t > *get it*. We do get it, we just don''t need this functionality. Send > in patches and we''ll listen, just whinge and you''ll probably be > completely ignored.I read the article, and I don''t see it as whining but the opening of a discussion. Are things so sensitive that saying, on ones own blog, "here is the problem as I see it, does anyone have an comments" is considered whining? Soulhuntre ---------- http://www.girl2.com - my girls http://www.the-estate.com - my legacy http://wiki.thegreybook.com - my project http://weblog.soulhuntre.com - my thoughts
Heya :)> -----Original Message----- > From: rails-bounces-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org [mailto:rails- > bounces-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org] On Behalf Of John Higgins > Sent: Tuesday, May 24, 2005 3:31 PM > To: rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org > Subject: Re: [Rails] Improving ActiveRecord Part 1 > > I always love these little "slap in the face" posts/blog entries that > come up from time to time.I am a little hard pressed to see this entry, discussion an issue with memory usage when BLOBS are involved, as a "slap in the face". Is any discussion or criticism seen as an attack? Is it helpful to Rails as a whole for the community to be so touchy about this stuff? Soulhuntre ---------- http://www.girl2.com - my girls http://www.the-estate.com - my legacy http://wiki.thegreybook.com - my project http://weblog.soulhuntre.com - my thoughts
On 5/24/05, Soulhuntre <soulhuntre-xtZVmrYH4z1ZroRs9YW3xA@public.gmane.org> wrote:> Heya :) > > > -----Original Message----- > > From: rails-bounces-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org [mailto:rails- > > bounces-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org] On Behalf Of John Higgins > > Sent: Tuesday, May 24, 2005 3:31 PM > > To: rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org > > Subject: Re: [Rails] Improving ActiveRecord Part 1 > > > > I always love these little "slap in the face" posts/blog entries that > > come up from time to time. > > I am a little hard pressed to see this entry, discussion an issue with > memory usage when BLOBS are involved, as a "slap in the face". Is any > discussion or criticism seen as an attack? Is it helpful to Rails as a whole > for the community to be so touchy about this stuff? > > Soulhuntre > ----------The problem is in the way the issue is worded - it''s not an attempt to make things better - nor is it really a technical discussion of what the issues are. First, he pounds rails for "incurring a massive memory hit" when using blobs. Then there is the little example which instead of pointing out that at least 1 if not 2 of these 25MB objects are absolutely going to be there no matter what because the database drivers - not the Rails Adapter or anything else to do with Rails. The only option around that is to somehow train database drivers to either deal with file handles or with file names that they open up themselves and immediately pass up to the database (not very likely). On top of that is the little jab about how great FastCGI handles the blob because it only takes up a few KB of ram - wonder why?????? The bottom line is that a small adjustment is required in the code to handle blobs as parameters - and as I''ve said it''s technically very feasible. I''d much rather see someone pound the hell out of the current implementation in his patch ticket with the solution then a cheap shot which neither helps anything or correctly states the true facts. I don''t think I''m being touchy - I just would prefer true facts to distorted opinions. -- John W Higgins wishdev-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
On May 24, 2005, at 1:14 PM, Soulhuntre wrote:> I read the article, and I don''t see it as whining but the opening of a > discussion.Likewise.> Are things so sensitive that saying, on ones own blog, "here is > the problem as I see it, does anyone have an comments" is considered > whining?No. This is mostly you being sensitive about others being sensitive. This is your eigenpost as far as I can tell. There''s a difference between "feature X is missing, they sure missed the boat" and "feature X is missing, I guess I''ll scratch that itch." It''s up to us to find a constructive place on that continuum. Folks will always differ in their judgement of criticism and others will criticize them for it. I hope you aren''t offended by mine. On topic, another no-brainer for AR adapters is support for prepared statements. If anyone has coding/testing time to kill and is looking to improve Active Record, check out these enhancement requests: http://dev.rubyonrails.com/report/2 Best, jeremy
> On topic, another no-brainer for AR adapters is support for prepared > statements. If anyone has coding/testing time to kill and is looking > to improve Active Record, check out these enhancement requests: > http://dev.rubyonrails.com/report/2Aren''t prepared statements (er, telling the server to reprepare a statement, etc) typically a database driver-level issue, in that some support/really like it (Interbase/Firebase), some don''t (Oracle)?> Best, > jeremy > _______________________________________________ > Rails mailing list > Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org > http://lists.rubyonrails.org/mailman/listinfo/rails >
On May 24, 2005, at 3:43 PM, Corey Lawson wrote:>> On topic, another no-brainer for AR adapters is support for prepared >> statements. If anyone has coding/testing time to kill and is looking >> to improve Active Record, check out these enhancement requests: >> http://dev.rubyonrails.com/report/2 > > Aren''t prepared statements (er, telling the server to reprepare a > statement, etc) typically a database driver-level issue, in that some > support/really like it (Interbase/Firebase), some don''t (Oracle)?Yes. AR adapters could take advantage of them, as available, to save the database from extra query parsing. jeremy
On 5/24/05, Corey Lawson <corey.ssf.lawson-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:> > On topic, another no-brainer for AR adapters is support for prepared > > statements. If anyone has coding/testing time to kill and is looking > > to improve Active Record, check out these enhancement requests: > > http://dev.rubyonrails.com/report/2 > > Aren''t prepared statements (er, telling the server to reprepare a > statement, etc) typically a database driver-level issue, in that some > support/really like it (Interbase/Firebase), some don''t (Oracle)?Oracle most certainly does, to my knowledge.
On 5/25/05, Soulhuntre <soulhuntre-xtZVmrYH4z1ZroRs9YW3xA@public.gmane.org> wrote:> Heya :) > > > -----Original Message----- > > From: rails-bounces-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org [mailto:rails- > > bounces-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org] On Behalf Of Michael Koziarski > > Sent: Tuesday, May 24, 2005 3:59 PM > > To: rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org > > Subject: Re: [Rails] Improving ActiveRecord Part 1 > > > > Everytime the people involved just complain and explain how we don''t > > *get it*. We do get it, we just don''t need this functionality. Send > > in patches and we''ll listen, just whinge and you''ll probably be > > completely ignored. > > I read the article, and I don''t see it as whining but the opening of a > discussion. Are things so sensitive that saying, on ones own blog, "here is > the problem as I see it, does anyone have an comments" is considered > whining?Apologies, no I didn''t consider the post as whining, quite the opposite. It was the first time someone posted a lucid ''way forward'' and I''m really looking forward to someone addressing it. Having said that, I was trying to warn the original poster that if it gets taken as ''an attack'', not to take it personally. It seems I failed fairly completely in my attempt ;)> Soulhuntre > ---------- > > http://www.girl2.com - my girls > http://www.the-estate.com - my legacy > http://wiki.thegreybook.com - my project > http://weblog.soulhuntre.com - my thoughts > > >-- Cheers Koz
Heya :)> -----Original Message----- > From: rails-bounces-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org [mailto:rails- > bounces-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org] On Behalf Of Jeremy Kemper > Sent: Tuesday, May 24, 2005 4:50 PM > To: rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org > Subject: Re: [Rails] Improving ActiveRecord Part 1 > > No. This is mostly you being sensitive about others being > sensitive. This is your eigenpost as far as I can tell.You know, your comment got me thinking - and I looked back and your right. I seem to be posting more about the perception of touchiness in the Rails community than I am about anything technical. I can''t imagine that is going to change anything, or help anyone :) I''ll try and tone it down and not react to this stuff. Thanks for the gentle prod. Soulhuntre ---------- http://www.girl2.com - my girls http://www.the-estate.com - my legacy http://wiki.thegreybook.com - my project http://weblog.soulhuntre.com - my thoughts