Rails - Apr 2013 - Dynamic creation of models and/or tables in Rails

I''m working on a project at the moment that has a rather unusual 
requirement and I''m hoping to get some advice on the best way to handle
it
or even some pointers to info that can help me build a solution.

Ok, so this is what I need to do. The application stores and manages 
various types of media files but each deployment of the application has 
completely different metadata requirements for the media files.

This metadata can contain an arbitrary number of fields of different types 
(single line text, multi-line text, checkboxes, selected values, etc.) and 
also often requires validation particularly presence and uniqueness 
validations.

The application needs to be able to easily retrieve values and *most 
importantly* has to be able to handle full searching capabilities on these 
fields.

One option I considered was using a property list arrangement where the 
database table simply contained a property name and value for each metadata 
field of each media file. However, when prototyping this solution it 
quickly became apparent that it simply wasn''t going to be efficient
enough
for the searching and retrieval of records particularly when the database 
can be reasonably large e.g. a recent deployment had 3000 media files and 
there were over 20 metadata fields. Also, the queries to do a search and 
retrieve the relevant records quickly became very complex.

Another option that the system is currently using is that the metadata 
config is defined upfront and a migration is run during deployment to 
create a the table and model with a standard name so that the media model 
can be associated with it which the system then uses. This generally works 
pretty fine but it does cause some significant deployment and testing 
issues.

For example, writing unit tests becomes much more challenging when you 
don''t know the config until deployment. Although I could write a sample
config and test the code that way, it won''t allow me to test the
specific
requirements of a particular deployment.

Similarly, in development, it currently requires me to copy a migration 
from the config into the main folder, run it, do all of my testing and 
development and then I have to remember to rollback and remove that 
migration from the main folder so that the application is in a standard 
state. This particularly becomes challenging when I''m bug fixing and I
need
to have the application in a specific configuration for testing and 
debugging purposes. Trying to switch between the various configurations 
becomes a real nightmare.

Ideally, what I would like is to be able to dynamically create the table 
and model including validations, etc. from a config file when the server is 
started. Even better would be if I could maintain multiple metadata setups 
in the one database with each one having its own table so that all I need 
to do to switch between them is change which config file the application is 
currently using.

I''m sure this can be done with Rails but there is very little
information
that I''ve been able to find that can point me in the right direction of
how
to build it during my research over the past few days so any help or 
suggestions would be much appreciated!

-- 
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to
rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit
https://groups.google.com/d/msg/rubyonrails-talk/-/WE31jU_kb10J.
For more options, visit https://groups.google.com/groups/opt_out.

I guess I would probably use the one table, and use single table inheritance on
each of your "deploys" if I''ve understood what you mean
correctly.

Each subclass model could "rename" fields according to its purpose...
of course, it all depends what you want to do with the metadata in the end.
There''s no use optimising for speed if flexibility is your requirement.

Julian

On 15/04/2013, at 10:19 AM, reagleton-vQs5aMGR+7nQT0dZR+AlfA@public.gmane.org
wrote:
> I''m working on a project at the moment that has a rather unusual
requirement and I''m hoping to get some advice on the best way to handle
it or even some pointers to info that can help me build a solution.
> 
> Ok, so this is what I need to do. The application stores and manages
various types of media files but each deployment of the application has
completely different metadata requirements for the media files.
> 
> This metadata can contain an arbitrary number of fields of different types
(single line text, multi-line text, checkboxes, selected values, etc.) and also
often requires validation particularly presence and uniqueness validations.
> 
> The application needs to be able to easily retrieve values and most
importantly has to be able to handle full searching capabilities on these
fields.
> 
> One option I considered was using a property list arrangement where the
database table simply contained a property name and value for each metadata
field of each media file. However, when prototyping this solution it quickly
became apparent that it simply wasn''t going to be efficient enough for
the searching and retrieval of records particularly when the database can be
reasonably large e.g. a recent deployment had 3000 media files and there were
over 20 metadata fields. Also, the queries to do a search and retrieve the
relevant records quickly became very complex.
> 
> Another option that the system is currently using is that the metadata
config is defined upfront and a migration is run during deployment to create a
the table and model with a standard name so that the media model can be
associated with it which the system then uses. This generally works pretty fine
but it does cause some significant deployment and testing issues.
> 
> For example, writing unit tests becomes much more challenging when you
don''t know the config until deployment. Although I could write a sample
config and test the code that way, it won''t allow me to test the
specific requirements of a particular deployment.
> 
> Similarly, in development, it currently requires me to copy a migration
from the config into the main folder, run it, do all of my testing and
development and then I have to remember to rollback and remove that migration
from the main folder so that the application is in a standard state. This
particularly becomes challenging when I''m bug fixing and I need to have
the application in a specific configuration for testing and debugging purposes.
Trying to switch between the various configurations becomes a real nightmare.
> 
> Ideally, what I would like is to be able to dynamically create the table
and model including validations, etc. from a config file when the server is
started. Even better would be if I could maintain multiple metadata setups in
the one database with each one having its own table so that all I need to do to
switch between them is change which config file the application is currently
using.
> 
> I''m sure this can be done with Rails but there is very little
information that I''ve been able to find that can point me in the right
direction of how to build it during my research over the past few days so any
help or suggestions would be much appreciated!
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
> To unsubscribe from this group and stop receiving emails from it, send an
email to
rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit
https://groups.google.com/d/msg/rubyonrails-talk/-/WE31jU_kb10J.
> For more options, visit https://groups.google.com/groups/opt_out.
>  
>  
-- 
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to
rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
For more options, visit https://groups.google.com/groups/opt_out.

On Sun, Apr 14, 2013 at 7:19 PM, 
<reagleton-vQs5aMGR+7nQT0dZR+AlfA@public.gmane.org>
wrote:
> I''m working on a project at the moment that has a rather unusual
requirement
> and I''m hoping to get some advice on the best way to handle it or
even some
> pointers to info that can help me build a solution.
>
> Ok, so this is what I need to do. The application stores and manages
various
> types of media files but each deployment of the application has completely
> different metadata requirements for the media files.
>
> This metadata can contain an arbitrary number of fields of different types
> (single line text, multi-line text, checkboxes, selected values, etc.) and
> also often requires validation particularly presence and uniqueness
> validations.
>
> The application needs to be able to easily retrieve values and most
> importantly has to be able to handle full searching capabilities on these
> fields.
>
> One option I considered was using a property list arrangement where the
> database table simply contained a property name and value for each metadata
> field of each media file. However, when prototyping this solution it
quickly
> became apparent that it simply wasn''t going to be efficient enough
for the
> searching and retrieval of records particularly when the database can be
> reasonably large e.g. a recent deployment had 3000 media files and there
> were over 20 metadata fields. Also, the queries to do a search and retrieve
> the relevant records quickly became very complex.
>
> Another option that the system is currently using is that the metadata
> config is defined upfront and a migration is run during deployment to
create
> a the table and model with a standard name so that the media model can be
> associated with it which the system then uses. This generally works pretty
> fine but it does cause some significant deployment and testing issues.
>
> For example, writing unit tests becomes much more challenging when you
don''t
> know the config until deployment. Although I could write a sample config
and
> test the code that way, it won''t allow me to test the specific
requirements
> of a particular deployment.
>
> Similarly, in development, it currently requires me to copy a migration
from
> the config into the main folder, run it, do all of my testing and
> development and then I have to remember to rollback and remove that
> migration from the main folder so that the application is in a standard
> state. This particularly becomes challenging when I''m bug fixing
and I need
> to have the application in a specific configuration for testing and
> debugging purposes. Trying to switch between the various configurations
> becomes a real nightmare.
>
> Ideally, what I would like is to be able to dynamically create the table
and
> model including validations, etc. from a config file when the server is
> started. Even better would be if I could maintain multiple metadata setups
> in the one database with each one having its own table so that all I need
to
> do to switch between them is change which config file the application is
> currently using.
>
> I''m sure this can be done with Rails but there is very little
information
> that I''ve been able to find that can point me in the right
direction of how
> to build it during my research over the past few days so any help or
> suggestions would be much appreciated!
So, this is after about 5 minutes thought, so bear that in mind. :)

Working on the premise that metadata is basically a key-value sort of
thing, maybe you could split it out of the application proper, and use
a Rails engine serving it up via a MongoDB implementation. The Rails
app proper could do all the normal sorts of things you have, and the
engine can be used to handle just the metadata parts. The app can
remain metadata agnostic that way, and provide the needed elements
based upon the information it retrieves from the engine.

Going a step further, you could combine more front end intelligence
into the app, say via backbone.js and underscore.js, and use them to
interact with the metadata serving part of your app.

-- 
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to
rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
For more options, visit https://groups.google.com/groups/opt_out.