Rails - Oct 2008 - detecting width overflow in serialized column with mysql

So I''ve got an ActiveRecord model pointing to a MySQL db, with an
auto-serialized column ("serialize :columnName").

Thing is, MySQL, depending on how it''s configured (like, by default),
has a bad habit of just truncating your data if it''s too wide for the
column, with no error raised. Yeah, I can probably reconfigure MySQL
and/or my AR connection to it. But I''m wanting to distribute this code
to all kinds of people who won''t think of that or won''t want
to do that,
I''d really like it to work with the default setup.

But obviously, if a yaml serialization gets truncated, it''s not going
to
unserialize very well.

At the point of insert/update, is there any good way for me to detect
that the serialization was too big for the column, even though MySQL
isn''t complaining?
-- 
Posted via http://www.ruby-forum.com/.

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---

Jonathan Rochkind wrote:
> So I''ve got an ActiveRecord model pointing to a MySQL db, with an
> auto-serialized column ("serialize :columnName").
> 
> Thing is, MySQL, depending on how it''s configured (like, by
default),
> has a bad habit of just truncating your data if it''s too wide for
the
> column, with no error raised. Yeah, I can probably reconfigure MySQL
> and/or my AR connection to it. But I''m wanting to distribute this
code
> to all kinds of people who won''t think of that or won''t
want to do that,
> I''d really like it to work with the default setup.
> 
> But obviously, if a yaml serialization gets truncated, it''s not
going to
> unserialize very well.
> 
> At the point of insert/update, is there any good way for me to detect
> that the serialization was too big for the column, even though MySQL
> isn''t complaining?
This may do it:

def validate
   if columnName.to_yaml > column_for_attribute(:columnName).limit
     errors.add(:columnName, ''columnName is too big to be
serialized'')
   end
end

It''d be good if something like this was added automatically by
serialize.

-- 
Rails Wheels - Find Plugins, List & Sell Plugins - http://railswheels.com

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---

Interesting, thanks, lots of tricks there I wouldn''thave figured out
for
myself. But am I being really inefficient now by calling to_yaml an 
extra time? I''ll call it once to validate, and then throw away the 
results. Then serialize will call it again when it actually wants to 
save. Calling it twice for no reason. But I guess to_yaml shouldn''t be 
_that_ expensive?

Now I wonder if I should abandon the automated serialization altogether, 
and just handle it myself manually through callbacks. Or maybe I should 
try monkey-patching the auto-serialization stuff to perform this check 
itself?

Jonathan

Mark Reginald James wrote:
> Jonathan Rochkind wrote:
>> But obviously, if a yaml serialization gets truncated, it''s
not going to
>> unserialize very well.
>> 
>> At the point of insert/update, is there any good way for me to detect
>> that the serialization was too big for the column, even though MySQL
>> isn''t complaining?
> 
> This may do it:
> 
> def validate
>    if columnName.to_yaml > column_for_attribute(:columnName).limit
>      errors.add(:columnName, ''columnName is too big to be
serialized'')
>    end
> end
> 
> It''d be good if something like this was added automatically by
> serialize.
> 
> --
> Rails Wheels - Find Plugins, List & Sell Plugins - 
> http://railswheels.com
-- 
Posted via http://www.ruby-forum.com/.

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---

Jonathan Rochkind wrote:
> Interesting, thanks, lots of tricks there I wouldn''thave figured
out for
> myself. But am I being really inefficient now by calling to_yaml an 
> extra time? I''ll call it once to validate, and then throw away the
> results. Then serialize will call it again when it actually wants to 
> save. Calling it twice for no reason. But I guess to_yaml
shouldn''t be
> _that_ expensive?
> 
> Now I wonder if I should abandon the automated serialization altogether, 
> and just handle it myself manually through callbacks. Or maybe I should 
> try monkey-patching the auto-serialization stuff to perform this check 
> itself?
Yeah, the double yaml is a waste, but don''t worry about it
unless this is being done sufficiently often to slow down
your app in a way that generates more work or cost.

See if you can come up for a patch for Rails that automatically
adds a such a validation for serialized attributes, where this
validation method and the attributes_with_quotes method that
generates the DB request both draw from a cache of YAMLized
attribute values.

-- 
Rails Wheels - Find Plugins, List & Sell Plugins - http://railswheels.com

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---

Here''s a different (easier :) ) tack to monkey patching AR to solve
this
problem efficiently. Turns out the actual serialization is done in 
quote_value (and interestingly, done whether or not the column is 
declared serialized; the serialized decleration only effects 
de-serialization). Does anyone think this following approach is a good 
idea?

Does anyone think it''s a good enough idea I should submit it as a patch
to AR?  I''ve never submitted a patch before, not sure how it''s
done. I''m
still using Rails 1.8.x, not sure if this is still a problem in Rails 2, 
so that might make it hard to submit a patch.

class ActiveRecord::Base
    alias :old_quote_value :quote_value
    def quote_value(value, column = nil)
        if column && serialized_attributes[column.name]
            serialization = value.to_yaml
            raise new ActiveRecord::StatementInvalid("Can not serialize 
column #{column.name}, length #{serialization.length} is greater than 
column limit of #{column.limit}") if if serialization.length > 
column.limit
           
"#{connection.quoted_string_prefix}''#{connection.quote_string(serialization)}''"
        else
            old_quote_value
        end
    end
end

Mark Reginald James wrote:
> Jonathan Rochkind wrote:
>> itself?
> Yeah, the double yaml is a waste, but don''t worry about it
> unless this is being done sufficiently often to slow down
> your app in a way that generates more work or cost.
> 
> See if you can come up for a patch for Rails that automatically
> adds a such a validation for serialized attributes, where this
> validation method and the attributes_with_quotes method that
> generates the DB request both draw from a cache of YAMLized
> attribute values.
> 
> --
> Rails Wheels - Find Plugins, List & Sell Plugins - 
> http://railswheels.com
-- 
Posted via http://www.ruby-forum.com/.

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---

Jonathan Rochkind wrote:
> Here''s a different (easier :) ) tack to monkey patching AR to
solve this
> problem efficiently. Turns out the actual serialization is done in 
> quote_value (and interestingly, done whether or not the column is 
> declared serialized; the serialized decleration only effects 
> de-serialization). Does anyone think this following approach is a good 
> idea?
> 
> Does anyone think it''s a good enough idea I should submit it as a
patch
> to AR?  I''ve never submitted a patch before, not sure how
it''s done. I''m
> still using Rails 1.8.x, not sure if this is still a problem in Rails 2, 
> so that might make it hard to submit a patch.
> 
> class ActiveRecord::Base
>     alias :old_quote_value :quote_value
>     def quote_value(value, column = nil)
>         if column && serialized_attributes[column.name]
>             serialization = value.to_yaml
>             raise new ActiveRecord::StatementInvalid("Can not
serialize
> column #{column.name}, length #{serialization.length} is greater than 
> column limit of #{column.limit}") if if serialization.length > 
> column.limit
>            
"#{connection.quoted_string_prefix}''#{connection.quote_string(serialization)}''"
>         else
>             old_quote_value
>         end
>     end
> end
Jonathan, that''s a fine solution if a database exception
is a better way to handle it than an automatic validation.
The validation route may however be more compatible with
the current save! idiom.

Also note that ActiveRecord trunk does the YAML conversion
in the attributes_with_quotes method instead.

-- 
Rails Wheels - Find Plugins, List & Sell Plugins - http://railswheels.com

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---

On 22 Oct 2008, at 10:23, Jonathan Rochkind wrote:
>
> Here''s a different (easier :) ) tack to monkey patching AR to
solve
> this
> problem efficiently. Turns out the actual serialization is done in
> quote_value (and interestingly, done whether or not the column is
> declared serialized; the serialized decleration only effects
> de-serialization). Does anyone think this following approach is a good
> idea?
>
> Does anyone think it''s a good enough idea I should submit it as a
> patch
> to AR?  I''ve never submitted a patch before, not sure how
it''s done.
> I''m
> still using Rails 1.8.x, not sure if this is still a problem in  
> Rails 2,
> so that might make it hard to submit a patch.
>
You''re almost certainly not using Rails 1.8.x because that version has
never existed (but you are probably running ruby 1.8.x (where x is  
probably 6))
Patches can be submitted at rails.lighthouseapp.com and/or discussed  
on the rubyonrails-core google group mailing list

Fred
> class ActiveRecord::Base
>    alias :old_quote_value :quote_value
>    def quote_value(value, column = nil)
>        if column && serialized_attributes[column.name]
>            serialization = value.to_yaml
>            raise new ActiveRecord::StatementInvalid("Can not serialize
> column #{column.name}, length #{serialization.length} is greater than
> column limit of #{column.limit}") if if serialization.length >
> column.limit
>             
> "#{connection 
>
.quoted_string_prefix}''#{connection.quote_string(serialization)}''"
>        else
>            old_quote_value
>        end
>    end
> end
>
> Mark Reginald James wrote:
>> Jonathan Rochkind wrote:
>>> itself?
>> Yeah, the double yaml is a waste, but don''t worry about it
>> unless this is being done sufficiently often to slow down
>> your app in a way that generates more work or cost.
>>
>> See if you can come up for a patch for Rails that automatically
>> adds a such a validation for serialized attributes, where this
>> validation method and the attributes_with_quotes method that
>> generates the DB request both draw from a cache of YAMLized
>> attribute values.
>>
>> --
>> Rails Wheels - Find Plugins, List & Sell Plugins -
>> http://railswheels.com
>
> -- 
> Posted via http://www.ruby-forum.com/.
>
> >

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---