Yossef Mendelssohn
2006-Oct-12 22:05 UTC
Validation enhancements -- reflection, overriding, and schema tie-ins
I''ve been looking at getting more and more information from the underlying database and automatically inserting that into my models. I''ve read many of the arguments, thoughts, and ideas about this. I understand the different points of view. This sort of thing is more for an integration database and not an application database, and I''m using an integration database. Also, I don''t entirely know the etiquette of the list and I apologize if pasting large blocks of code is gauche, but this is more of a proof-of-concept request-for-comments than a finalized plugin or patch. So this, below, is a stab at taking validation information from the DDL. I was part-way through this before I learned about the enforce_schema_rules and schema_validations plugins, but those weren''t exactly for me. I learned some from them, but there were some things I needed to do differently. ############################################################ module ActiveRecordExtensions module Validation module FromDDL def self.included(base) base.extend(ClassMethods) end module ClassMethods def validations_from_ddl validations = [] validations << column_type_validations validations << constraint_validations validations << index_validations validations.flatten! class_eval validations.join("\n") end def column_type_validations validations = [] self.content_columns.each do |col| case col.type when :string validations << "validates_length_of :#{col.name}, :maximum => #{col.limit}" when :integer validations << "validates_numericality_of :#{col.name}, :only_integer => true" when :float validations << "validates_numericality_of :#{col.name}" when :datetime if col.name.ends_with?(''_date'') validations << "validates_date :#{col.name}" elsif col.name.ends_with?(''_time'') validations << "validates_date_time :#{col.name}" end when :time validations << "validates_time :#{col.name}" end validations << "validates_presence_of :#{col.name}" unless col.null end validations end def constraint_validations validations = [] self.core_content_columns.each do |col| constraints_on(col.name).each do |constraint| case constraint[:type] when ''inclusion'', ''exclusion'' validations << "validates_#{constraint[:type]}_of :#{col.name}, :in => [#{constraint[:in]}]" end end end validations end def index_validations validations = [] self.connection.indexes(self.table_name).each do |index| next unless index.unique # I''m not sure about this, but there needs to be a way to ensure the uniqueness validation # is formatted correctly no matter the order the columns appear in the index specification scope_columns = index.columns unique_col = scope_columns.detect { |c| !c.ends_with?(''_id'') } next unless unique_col scope_columns.delete(unique_col) validation = "validates_uniqueness_of :#{unique_col}" validation += ", :scope => [#{ scope_columns.collect { |c| ":#{c}" }.join('', '') }]" unless scope_columns.blank? validations << validation end validations end # This is *slightly* Oracle-specific def constraints_on(column_name) (owner, table_name) self.connection.instance_variable_get(''@connection'').describe(self.table_name) column_constraints_sql = <<-END_SQL select uc.search_condition from user_constraints uc, user_cons_columns ucc where uc.owner = ''#{owner}'' and uc.table_name = ''#{table_name}'' and uc.constraint_type = ''C'' and uc.status = ''ENABLED'' and uc.constraint_name = ucc.constraint_name and ucc.owner = ''#{owner}'' and ucc.table_name = ''#{table_name}'' and ucc.column_name = ''#{column_name.upcase}'' END_SQL constraints = [] self.connection.select_all(column_constraints_sql).each do |row| condition = row[''search_condition''] || '''' next unless condition.starts_with?(column_name) if tmp = condition.match(/^#{column_name} (not )?in \((.+)\)$/) values = tmp[2] constraints << { :type => tmp[1] ? ''exclusion'' : ''inclusion'', :in => values } end end constraints end end end end end module ActiveRecord class Base include ActiveRecordExtensions::Validation::FromDDL end end ############################################################ That automatically produces a set of validations in any model that includes a call to ''validations_from_ddl''. It''s not everything, but it keeps you from having to repeat yourself when putting constraints in the database *and* the model. As for the argument about how you can''t put minimum length information in the database, I don''t think anyone has mentioned how you can put a check constraint on that as well, at least in Oracle. And before anyone talks about the differences between Oracle and Postgres and MySQL and where the constraints should go, there are *always* constraints in the database. Even "lowly MySQL" which "doesn''t use constraints" has varchar fields with limits. And I think we all know what good ol'' MySQL does when you hand it a value too long to fit in your varchar column. But there''s more! I thought about inheritance and subclassing and maybe wanting something *more restrictive* than the database. What if I have multiple classes all residing in one table? (STI, anyone?) In that case, the table has to support the longest possible value. Now, if you have a parent class that gives a length maximum of 30 and a subclass that gives a length maximum of 20, what happens? For values <= 20, no errors. For values between 21 and 30, one errors. For values >= 31, two errors. TWO ERRORS! This is unacceptable. Well, maybe not "unacceptable", but at least unexpected and undesirable. So I looked into how to give validation overrides. I learned about the good work done on reflections by Michael Schuerig. Once again, the plugin didn''t exactly suit my needs, so I grabbed it and made it my own. ############################################################ module ActiveRecord module Reflection # :nodoc: # Holds all the meta-data about a validation as it was specified in the Active Record class. class ValidationReflection < MacroReflection attr_accessor :block def klass @active_record end def attr_name @name end def validation @macro end def configuration @options end end end end module ActiveRecordExtensions module Validation module Reflection VALIDATION_METHODS = ActiveRecord::Base.methods.select { |m| m.starts_with?(''validates_'') && m != ''validates_each'' && !m.match(/_with(out)?_/) }.freeze def self.included(base) VALIDATION_METHODS.each do |validation| base.class_eval <<-eval_string class << self alias :#{validation}_without_reflection :#{validation} def #{validation}_with_reflection(*attr_names) attrs = attr_names.dup configuration = attrs.last.is_a?(Hash) ? attrs.pop : {} val_type = validation_method(configuration[:on] || :save) val_blocks = { :before => [], :after => [], :new => [] } val_blocks[:before] read_inheritable_attribute(val_type) || [] #{validation}_without_reflection(*attr_names) val_blocks[:after] read_inheritable_attribute(val_type) || [] val_blocks[:new] = val_blocks[:after] - val_blocks[:before] # I don''t think this will work when giving multiple attr_names. # We just won''t do that. Only one attribute per validation line! attrs.zip(val_blocks[:new]).each do |attr_name, block| val ActiveRecord::Reflection::ValidationReflection.new(:#{validation}, attr_name, configuration, self) val.block = block add_validation_for(attr_name, val) end end alias :#{validation} :#{validation}_with_reflection end eval_string end base.extend(ClassMethods) end module ClassMethods # Returns a hash of ValidationReflection objects for all validations in the class def validations read_inheritable_attribute(''validations'') || {} end # Returns a hash of ValidationReflection objects for all validations defined for the field +attr_name+ def validations_for(attr_name) validations[attr_name.to_s] end # Returns an array of ValidationReflection objects for all +validation+ type validations defined for the field +attr_name+ def validation_for(attr_name, validation) (validations[attr_name.to_s] || {})[validation.to_s] end def add_validation_for(attr_name, validation) vals = validations ((vals[attr_name.to_s] ||= {})[validation.validation.to_s] ||= []) << validation write_inheritable_attribute(''validations'', vals) end end end module Unique def self.included(base) base.extend(ClassMethods) end module ClassMethods def add_validation_for(attr_name, validation) vals = validations (vals[attr_name.to_s] ||= {})[validation.macro.to_s] validation write_inheritable_attribute(''validations'', vals) write_unique_validations(validation_method(validation.configuration[:on] || :save)) end private def write_unique_validations(key) new_methods = [] validations.each do |attr_name, vals| vals.each do |method, val| new_methods << val.block if validation_method(val.configuration[:on] || :save) == key end end write_inheritable_attribute(key, new_methods) end end end end end module ActiveRecord class Base include ActiveRecordExtensions::Validation::Reflection include ActiveRecordExtensions::Validation::Unique end end ############################################################ All the Procs I can''t get any information about, they didn''t make me happy. The way the validations are stored as simple arrays of Procs, not so good. I fully accept that this approach is hackery, but it works (as long as you pay attention to the ''one attribute per validation'' comment). I know this breaks some things. For the lovers of validates_length_of, that''s not a problem for me. I don''t use it in its full overloaded glory. I use it only for exact lengths and have validates_maximum_length_of and validates_minimum_length_of to use when appropriate. This also breaks some of the simplicity that can come from multiple calls to validates_format_of. For instance, it''s simpler to check for presence of both a letter and a digit by having two separate regexes than one. Like I said, proof of concept, request for comments, not final. The uniqueness can be made to check more than just the validation command name and attribute name. But this is a start. And it keeps you from getting multiple nonsensical errors if you happen to do something like validates_presence_of :name validates_presence_of :name validates_presence_of :name or validates_length_of :name, :is => 8 validates_length_of :name, :is => 9 validates_length_of :name, :is => 10 I''m not saying you *would* or even *should* do something like that, but it could come up. Maybe. Somehow. And the validation errors would not be pretty. Of course, the next step would be to tie this in to client-side JavaScript validations (also Michael Schuerig). If done well (and that''s a bit of an if), the DRY principle can be upheld quite nicely by using the database to store constraints and have them be pulled into the model (where they can be overriden and more can be defined) and from there into the view. It''s DRY and it''s multi-level. It''s the best of both worlds? (Once again, I apologize for the length. And I''m looking forward to discussion.) -- -yossef --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk -~----------~----~----~----~------~----~------~--~---
Ezra Zygmuntowicz
2006-Oct-13 01:35 UTC
Re: Validation enhancements -- reflection, overriding, and schema tie-ins
On Oct 12, 2006, at 3:05 PM, Yossef Mendelssohn wrote:> > I''ve been looking at getting more and more information from the > underlying database and automatically inserting that into my models. > I''ve read many of the arguments, thoughts, and ideas about this. I > understand the different points of view. This sort of thing is more > for an integration database and not an application database, and I''m > using an integration database. >> <snip good stuff> >> (Once again, I apologize for the length. And I''m looking forward to > discussion.) > > -- > -yossef >Hey Yossef- Looks like you have done some very nice work here. Can I make a suggestion that you make this into a plugin and put it in svn somewhere like rubyforge or another free svn host? Then I think you would get a lot more feedback on this if people can just drop it into their plugins dir and try it out. Let me know if you need a hand making it work as a plugin. Cheers- -Ezra --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk -~----------~----~----~----~------~----~------~--~---
Yossef Mendelssohn
2006-Oct-13 14:03 UTC
Re: Validation enhancements -- reflection, overriding, and schema tie-ins
Thanks, Ezra. I figured that would be the case but didn''t go the pre-packaged plugin route because of some of the problems mentioned (breaking multiple calls to validates_length_of and validates_format_of). But if it''ll get more discussion going as a plugin, a plugin (with caveats) it''ll be. I should have time for that sometime soon, maybe this weekend. -- -yossef --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk -~----------~----~----~----~------~----~------~--~---
Yossef Mendelssohn
2006-Oct-21 22:30 UTC
Re: Validation enhancements -- reflection, overriding, and schema tie-ins
Ezra Zygmuntowicz wrote:> Hey Yossef- > > Looks like you have done some very nice work here. Can I make a > suggestion that you make this into a plugin and put it in svn > somewhere like rubyforge or another free svn host? Then I think you > would get a lot more feedback on this if people can just drop it into > their plugins dir and try it out. Let me know if you need a hand > making it work as a plugin. > > > Cheers- > -EzraEzra, Thanks again for your comment. I was finally able to get some time to make this a plugin and put it on Rubyforge. It''s available at http://rubyforge.org/projects/validate-ddl/. -- -yossef --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk -~----------~----~----~----~------~----~------~--~---
Yossef Mendelssohn
2006-Nov-04 02:36 UTC
[Rails] Validation enhancements -- reflection, overriding, and schema tie-ins
I''ve been looking at getting more and more information from the underlying database and automatically inserting that into my models. I''ve read many of the arguments, thoughts, and ideas about this. I understand the different points of view. This sort of thing is more for an integration database and not an application database, and I''m using an integration database. Also, I don''t entirely know the etiquette of the list and I apologize if pasting large blocks of code is gauche, but this is more of a proof-of-concept request-for-comments than a finalized plugin or patch. So this, below, is a stab at taking validation information from the DDL. I was part-way through this before I learned about the enforce_schema_rules and schema_validations plugins, but those weren''t exactly for me. I learned some from them, but there were some things I needed to do differently. module ActiveRecordExtensions module Validation module FromDDL def self.included(base) base.extend(ClassMethods) end module ClassMethods def validations_from_ddl validations = [] validations << column_type_validations validations << constraint_validations validations << index_validations validations.flatten! class_eval validations.join("\n") end def column_type_validations validations = [] self.content_columns.each do |col| case col.type when :string validations << "validates_length_of :#{col.name}, :maximum => #{col.limit}" when :integer validations << "validates_numericality_of :#{col.name}, :only_integer => true" when :float validations << "validates_numericality_of :#{col.name}" when :datetime if col.name.ends_with?(''_date'') validations << "validates_date :#{col.name}" elsif col.name.ends_with?(''_time'') validations << "validates_date_time :#{col.name}" end when :time validations << "validates_time :#{col.name}" end validations << "validates_presence_of :#{col.name}" unless col.null end validations end def constraint_validations validations = [] self.core_content_columns.each do |col| constraints_on(col.name).each do |constraint| case constraint[:type] when ''inclusion'', ''exclusion'' validations << "validates_#{constraint[:type]}_of :#{col.name}, :in => [#{constraint[:in]}]" end end end validations end def index_validations validations = [] self.connection.indexes(self.table_name).each do |index| next unless index.unique # I''m not sure about this, but there needs to be a way to ensure the uniqueness validation # is formatted correctly no matter the order the columns appear in the index specification scope_columns = index.columns unique_col = scope_columns.detect { |c| !c.ends_with?(''_id'') } next unless unique_col scope_columns.delete(unique_col) validation = "validates_uniqueness_of :#{unique_col}" validation += ", :scope => [#{ scope_columns.collect { |c| ":#{c}" }.join('', '') }]" unless scope_columns.blank? validations << validation end validations end # This is *slightly* Oracle-specific def constraints_on(column_name) (owner, table_name) = self.connection.instance_variable_get(''@connection'').describe(self.table_name) column_constraints_sql = <<-END_SQL select uc.search_condition from user_constraints uc, user_cons_columns ucc where uc.owner = ''#{owner}'' and uc.table_name = ''#{table_name}'' and uc.constraint_type = ''C'' and uc.status = ''ENABLED'' and uc.constraint_name = ucc.constraint_name and ucc.owner = ''#{owner}'' and ucc.table_name = ''#{table_name}'' and ucc.column_name = ''#{column_name.upcase}'' END_SQL constraints = [] self.connection.select_all(column_constraints_sql).each do |row| condition = row[''search_condition''] || '''' next unless condition.starts_with?(column_name) if tmp = condition.match(/^#{column_name} (not )?in \((.+)\)$/) values = tmp[2] constraints << { :type => tmp[1] ? ''exclusion'' : ''inclusion'', :in => values } end end constraints end end end end end module ActiveRecord class Base include ActiveRecordExtensions::Validation::FromDDL end end That automatically produces a set of validations in any model that includes a call to ''validations_from_ddl''. It''s not everything, but it keeps you from having to repeat yourself when putting constraints in the database *and* the model. As for the argument about how you can''t put minimum length information in the database, I don''t think anyone has mentioned how you can put a check constraint on that as well, at least in Oracle. And before anyone talks about the differences between Oracle and Postgres and MySQL and where the constraints should go, there are *always* constraints in the database. Even "lowly MySQL" which "doesn''t use constraints" has varchar fields with limits. And I think we all know what good ol'' MySQL does when you hand it a value too long to fit in your varchar column. But there''s more! I thought about inheritance and subclassing and maybe wanting something *more restrictive* than the database. What if I have multiple classes all residing in one table? (STI, anyone?) In that case, the table has to support the longest possible value. Now, if you have a parent class that gives a length maximum of 30 and a subclass that gives a length maximum of 20, what happens? For values <= 20, no errors. For values between 21 and 30, one errors. For values >= 31, two errors. TWO ERRORS! This is unacceptable. Well, maybe not "unacceptable", but at least unexpected and undesirable. So I looked into how to give validation overrides. I learned about the good work done on reflections by Michael Schuerig. Once again, the plugin didn''t exactly suit my needs, so I grabbed it and made it my own. module ActiveRecord module Reflection # :nodoc: # Holds all the meta-data about a validation as it was specified in the Active Record class. class ValidationReflection < MacroReflection attr_accessor :block def klass @active_record end def attr_name @name end def validation @macro end def configuration @options end end end end module ActiveRecordExtensions module Validation module Reflection VALIDATION_METHODS = ActiveRecord::Base.methods.select { |m| m.starts_with?(''validates_'') && m != ''validates_each'' && !m.match(/_with(out)?_/) }.freeze def self.included(base) VALIDATION_METHODS.each do |validation| base.class_eval <<-eval_string class << self alias :#{validation}_without_reflection :#{validation} def #{validation}_with_reflection(*attr_names) attrs = attr_names.dup configuration = attrs.last.is_a?(Hash) ? attrs.pop : {} val_type = validation_method(configuration[:on] || :save) val_blocks = { :before => [], :after => [], :new => [] } val_blocks[:before] = read_inheritable_attribute(val_type) || [] #{validation}_without_reflection(*attr_names) val_blocks[:after] = read_inheritable_attribute(val_type) || [] val_blocks[:new] = val_blocks[:after] - val_blocks[:before] # I don''t think this will work when giving multiple attr_names. # We just won''t do that. Only one attribute per validation line! attrs.zip(val_blocks[:new]).each do |attr_name, block| val = ActiveRecord::Reflection::ValidationReflection.new(:#{validation}, attr_name, configuration, self) val.block = block add_validation_for(attr_name, val) end end alias :#{validation} :#{validation}_with_reflection end eval_string end base.extend(ClassMethods) end module ClassMethods # Returns a hash of ValidationReflection objects for all validations in the class def validations read_inheritable_attribute(''validations'') || {} end # Returns a hash of ValidationReflection objects for all validations defined for the field +attr_name+ def validations_for(attr_name) validations[attr_name.to_s] end # Returns an array of ValidationReflection objects for all +validation+ type validations defined for the field +attr_name+ def validation_for(attr_name, validation) (validations[attr_name.to_s] || {})[validation.to_s] end def add_validation_for(attr_name, validation) vals = validations ((vals[attr_name.to_s] ||= {})[validation.validation.to_s] ||= []) << validation write_inheritable_attribute(''validations'', vals) end end end module Unique def self.included(base) base.extend(ClassMethods) end module ClassMethods def add_validation_for(attr_name, validation) vals = validations (vals[attr_name.to_s] ||= {})[validation.macro.to_s] = validation write_inheritable_attribute(''validations'', vals) write_unique_validations(validation_method(validation.configuration[:on] || :save)) end private def write_unique_validations(key) new_methods = [] validations.each do |attr_name, vals| vals.each do |method, val| new_methods << val.block if validation_method(val.configuration[:on] || :save) == key end end write_inheritable_attribute(key, new_methods) end end end end end module ActiveRecord class Base include ActiveRecordExtensions::Validation::Reflection include ActiveRecordExtensions::Validation::Unique end end All the Procs I can''t get any information about, they didn''t make me happy. The way the validations are stored as simple arrays of Procs, not so good. I fully accept that this approach is hackery, but it works (as long as you pay attention to the ''one attribute per validation'' comment). I know this breaks some things. For the lovers of validates_length_of, that''s not a problem for me. I don''t use it in its full overloaded glory. I use it only for exact lengths and have validates_maximum_length_of and validates_minimum_length_of to use when appropriate. This also breaks some of the simplicity that can come from multiple calls to validates_format_of. For instance, it''s simpler to check for presence of both a letter and a digit by having two separate regexes than one. Like I said, proof of concept, request for comments, not final. The uniqueness can be made to check more than just the validation command name and attribute name. But this is a start. And it keeps you from getting multiple nonsensical errors if you happen to do something like validates_presence_of :name validates_presence_of :name validates_presence_of :name or validates_length_of :name, :is => 8 validates_length_of :name, :is => 9 validates_length_of :name, :is => 10 I''m not saying you *would* or even *should* do something like that, but it could come up. Maybe. Somehow. And the validation errors would not be pretty. Of course, the next step would be to tie this in to client-side JavaScript validations (also Michael Schuerig). If done well (and that''s a bit of an if), the DRY principle can be upheld quite nicely by using the database to store constraints and have them be pulled into the model (where they can be overriden and more can be defined) and from there into the view. It''s DRY and it''s multi-level. It''s the best of both worlds? (Once again, I apologize for the length. And I''m looking forward to discussion.) -- -yossef