Rails core - Apr 2013 - Suggested strategies for testing a gem against Rails 3.x and Rails 4?

I''ve seen a few examples of dummy Rails apps (for testing, so they live
under test or spec dirs, typically) for use with the Appraisals gem that 
supposedly work with both Rails 3.x and Rails 4, but they seem hackish and 
not fully functional. It is somewhat expected, as it is a stripped down 
Frankenstein monster that is trying to be compatible with various versions 
of Rails 3 as well as Rails 4.

I''ve referred to projects that attempt to do this sort of testing (as
of
late March 2013) like less-rails and ember-rails, but this way to test with 
various version of Rails doesn''t seem very clean, and it is non-trivial
to
try to debug a non-standard Rails application, especially in a beta version 
of Rails.

It would be great to have a cleaner way to test that allows you to have a 
full Rails application for each version of Rails to test with that through 
some magic is not that difficult to setup or maintain and don''t require
non-standard path hacks in places, etc.

What are the available strategies for testing gems with various versions of 
Rails (including at least latest Rails 3.1.x, 3.2.x, and 4.0.0.beta1), and 
what are the pros and cons of each?
I''ve also asked this question at:
http://stackoverflow.com/questions/15752774/strategies-for-gem-tests-to-ensure-the-gem-works-with-rails-3-x-and-4-0

Thanks!

-- 
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to rubyonrails-core+unsubscribe@googlegroups.com.
To post to this group, send email to rubyonrails-core@googlegroups.com.
Visit this group at http://groups.google.com/group/rubyonrails-core?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Note: the reason I bring this up on the core list is that I think it would 
help people upgrade to Rails 4 more quickly if it were easier or clearer as 
to how to test gems against (maybe) 3.1.x, 3.2.x, and 4.x, at least until 
mostly everyone switches to 4.

On Tuesday, April 2, 2013 9:04:30 AM UTC-4, Gary Weaver
wrote:
>
> I''ve seen a few examples of dummy Rails apps (for testing, so they
live
> under test or spec dirs, typically) for use with the Appraisals gem that 
> supposedly work with both Rails 3.x and Rails 4, but they seem hackish and 
> not fully functional. It is somewhat expected, as it is a stripped down 
> Frankenstein monster that is trying to be compatible with various versions 
> of Rails 3 as well as Rails 4.
>
> I''ve referred to projects that attempt to do this sort of testing
(as of
> late March 2013) like less-rails and ember-rails, but this way to test with
> various version of Rails doesn''t seem very clean, and it is
non-trivial to
> try to debug a non-standard Rails application, especially in a beta version
> of Rails.
>
> It would be great to have a cleaner way to test that allows you to have a 
> full Rails application for each version of Rails to test with that through 
> some magic is not that difficult to setup or maintain and don''t
require
> non-standard path hacks in places, etc.
>
> What are the available strategies for testing gems with various versions 
> of Rails (including at least latest Rails 3.1.x, 3.2.x, and 4.0.0.beta1), 
> and what are the pros and cons of each?
> I''ve also asked this question at:
>
>
http://stackoverflow.com/questions/15752774/strategies-for-gem-tests-to-ensure-the-gem-works-with-rails-3-x-and-4-0
>
> Thanks!
>
-- 
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to rubyonrails-core+unsubscribe@googlegroups.com.
To post to this group, send email to rubyonrails-core@googlegroups.com.
Visit this group at http://groups.google.com/group/rubyonrails-core?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Why not test it in travis? Or some other kind of CI solution.

On Apr 2, 2013, at 2:04 PM, Gary Weaver <garysweaver@gmail.com> wrote:
> I''ve seen a few examples of dummy Rails apps (for testing, so they
live under test or spec dirs, typically) for use with the Appraisals gem that
supposedly work with both Rails 3.x and Rails 4, but they seem hackish and not
fully functional. It is somewhat expected, as it is a stripped down Frankenstein
monster that is trying to be compatible with various versions of Rails 3 as well
as Rails 4.
> 
> I''ve referred to projects that attempt to do this sort of testing
(as of late March 2013) like less-rails and ember-rails, but this way to test
with various version of Rails doesn''t seem very clean, and it is
non-trivial to try to debug a non-standard Rails application, especially in a
beta version of Rails.
> 
> It would be great to have a cleaner way to test that allows you to have a
full Rails application for each version of Rails to test with that through some
magic is not that difficult to setup or maintain and don''t require
non-standard path hacks in places, etc.
> 
> What are the available strategies for testing gems with various versions of
Rails (including at least latest Rails 3.1.x, 3.2.x, and 4.0.0.beta1), and what
are the pros and cons of each?
> 
> I''ve also asked this question at:
>
http://stackoverflow.com/questions/15752774/strategies-for-gem-tests-to-ensure-the-gem-works-with-rails-3-x-and-4-0
> 
> Thanks!
> 
> -- 
> You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
> To unsubscribe from this group and stop receiving emails from it, send an
email to rubyonrails-core+unsubscribe@googlegroups.com.
> To post to this group, send email to rubyonrails-core@googlegroups.com.
> Visit this group at http://groups.google.com/group/rubyonrails-core?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>  
>  
Cumprimentos,
Luís Ferreira



-- 
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to rubyonrails-core+unsubscribe@googlegroups.com.
To post to this group, send email to rubyonrails-core@googlegroups.com.
Visit this group at http://groups.google.com/group/rubyonrails-core?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

On Apr 2, 2013, at 9:16 AM, Gary Weaver <garysweaver@gmail.com> wrote:
> Note: the reason I bring this up on the core list is that I think it would
help people upgrade to Rails 4 more quickly if it were easier or clearer as to
how to test gems against (maybe) 3.1.x, 3.2.x, and 4.x, at least until mostly
everyone switches to 4.

I test minitest-spec-rails against 3.0, 3.1, 3,2 and 4.0 using a mix of
appraisal and dummy_app that minimally configures itself depending which rails
version it is testing against. Some links:

https://github.com/metaskills/minitest-spec-rails
https://github.com/metaskills/minitest-spec-rails/blob/master/test/dummy_app/init.rb


 - HTH,
   Ken



-- 
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to rubyonrails-core+unsubscribe@googlegroups.com.
To post to this group, send email to rubyonrails-core@googlegroups.com.
Visit this group at http://groups.google.com/group/rubyonrails-core?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Ken,

It definitely helps to have more examples, and what you did in 
minitest-spec-rails is certainly very clean-looking!

I looked at that and the stuff you did in less-rails which was similar:
https://github.com/metaskills/less-rails/blob/master/test/dummy_app/init.rb

as well as what had been done in ember-rails:
https://github.com/emberjs/ember-rails/tree/master/test/dummy

The difficulty is that my confidence goes through the floor when I''m
the
one who is diff''ing a default Rails 4.0.0.beta1 app (that I have to 
manually generate and recursive diff) vs. the hybrid dummy app to see what 
is different and I''m just not *sure* that it is really testing what
I''m
intending to. Also, I''ve had issues with the Appraisal of Rails 3.2.x,
so I
need to do the same for it. What about files and directories used in one 
version, and not in another? If I''m looking at your gem or another as
an
example, how many options may have been stripped down in a dummy app for a 
particular gem, because they weren''t needed for the testing of that
gem?

On the flipside, having to check in a full Rails app in for every version 
of Rails that I want to test with in my project would have a scary amount 
of code duplication. I wouldn''t want to add all of that into the git
repo.

What you did seems as clean as could be done without external tools 
managing the creation and configuration of Rails apps.

But would we be better off if we had something where we could test with 
fully Rails version-specific "dummy" applications, that for all
functional
purposes would be "real" Rails apps, with different versions of Rails
(any
version of Rails in rubygems)? The app would be generated as part of the 
test setup using the normal method to generate a new Rails app, then 
changes would be applied. Maybe changes would be a directory tree of only 
the files that changed, except with some way to identify files and 
directories that should be deleted from the default Rails app (which 
hopefully would be the exception rather than the rule for deletion). Once 
created, we could browse and treat the dummy app as if it were a normal 
version-specific Rails app when trying to debug that specific version (we 
could even *run* it like normal), and there would be no worry of it mucking 
up the git repo with a ton of excess files for testing.

When everything is working in my gem''s tests, I would certainly opt for
what you''ve come up with, because it looks clean and is probably more 
common sense than the solution I just described, and you wouldn''t have
to
wait the one time during test setup for the app to be generated and 
overlaid (even if it might just be an infrequent thing). And that one-time 
wait locally is probably not one-time for CI, since it would take much 
longer. But in a lot of ways it would be more flexible and you won''t
run
into as many gotchas.

Thanks,
Gary



On Tuesday, April 2, 2013 9:40:24 AM UTC-4, Ken Collins
wrote:
>
>
> On Apr 2, 2013, at 9:16 AM, Gary Weaver <garys...@gmail.com
<javascript:>>
> wrote:
>
> Note: the reason I bring this up on the core list is that I think it would 
> help people upgrade to Rails 4 more quickly if it were easier or clearer as
> to how to test gems against (maybe) 3.1.x, 3.2.x, and 4.x, at least until 
> mostly everyone switches to 4.
>
>
> I test minitest-spec-rails against 3.0, 3.1, 3,2 and 4.0 using a mix of 
> appraisal and dummy_app that minimally configures itself depending which 
> rails version it is testing against. Some links:
>
> https://github.com/metaskills/minitest-spec-rails
>
>
https://github.com/metaskills/minitest-spec-rails/blob/master/test/dummy_app/init.rb
>
>
>  - HTH,
>    Ken
>
>
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to rubyonrails-core+unsubscribe@googlegroups.com.
To post to this group, send email to rubyonrails-core@googlegroups.com.
Visit this group at http://groups.google.com/group/rubyonrails-core?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

To expand on the idea some, basically the strategy might be to:

1. Have a list of the versions of Rails to test against (like Appraisals).

2. For each version of Rails, generate a new Rails application called
"Dummy" and put it in its own directory, e.g.
test/appraising/rails-4.0.0.beta1/pristine.

3. You would specify an overlay directory for one or more versions of Rails
in a config file and put the files to overlay in the configured directory,
e.g. test/appraising/overlays/rails-4.x/

4. You would specify paths to delete as part of the overlay in that config
file also.

5. When you run some command, it could copy
test/appraising/rails-4.0.0.beta1/default_app into
test/appraising/rails-4.0.0.beta1/dummy and then make any specified
deletions and recursively copy the overlay into the dummy (e.g.
test/appraising/overlays/rails-4.x/**/*).

6. Would run bundle install, etc. in that dummy app and then proceed to
test it.

On Tue, Apr 2, 2013 at 10:35 AM, Gary Weaver <garysweaver@gmail.com>
wrote:
> Ken,
>
> It definitely helps to have more examples, and what you did in
> minitest-spec-rails is certainly very clean-looking!
>
> I looked at that and the stuff you did in less-rails which was similar:
> https://github.com/metaskills/less-rails/blob/master/test/dummy_app/init.rb
>
> as well as what had been done in ember-rails:
> https://github.com/emberjs/ember-rails/tree/master/test/dummy
>
> The difficulty is that my confidence goes through the floor when
I''m the
> one who is diff''ing a default Rails 4.0.0.beta1 app (that I have
to
> manually generate and recursive diff) vs. the hybrid dummy app to see what
> is different and I''m just not *sure* that it is really testing
what I''m
> intending to. Also, I''ve had issues with the Appraisal of Rails
3.2.x, so I
> need to do the same for it. What about files and directories used in one
> version, and not in another? If I''m looking at your gem or another
as an
> example, how many options may have been stripped down in a dummy app for a
> particular gem, because they weren''t needed for the testing of
that gem?
>
> On the flipside, having to check in a full Rails app in for every version
> of Rails that I want to test with in my project would have a scary amount
> of code duplication. I wouldn''t want to add all of that into the
git repo.
>
> What you did seems as clean as could be done without external tools
> managing the creation and configuration of Rails apps.
>
> But would we be better off if we had something where we could test with
> fully Rails version-specific "dummy" applications, that for all
functional
> purposes would be "real" Rails apps, with different versions of
Rails (any
> version of Rails in rubygems)? The app would be generated as part of the
> test setup using the normal method to generate a new Rails app, then
> changes would be applied. Maybe changes would be a directory tree of only
> the files that changed, except with some way to identify files and
> directories that should be deleted from the default Rails app (which
> hopefully would be the exception rather than the rule for deletion). Once
> created, we could browse and treat the dummy app as if it were a normal
> version-specific Rails app when trying to debug that specific version (we
> could even *run* it like normal), and there would be no worry of it mucking
> up the git repo with a ton of excess files for testing.
>
> When everything is working in my gem''s tests, I would certainly
opt for
> what you''ve come up with, because it looks clean and is probably
more
> common sense than the solution I just described, and you wouldn''t
have to
> wait the one time during test setup for the app to be generated and
> overlaid (even if it might just be an infrequent thing). And that one-time
> wait locally is probably not one-time for CI, since it would take much
> longer. But in a lot of ways it would be more flexible and you
won''t run
> into as many gotchas.
>
> Thanks,
> Gary
>
>
>
> On Tuesday, April 2, 2013 9:40:24 AM UTC-4, Ken Collins wrote:
>
>>
>> On Apr 2, 2013, at 9:16 AM, Gary Weaver <garys...@gmail.com>
wrote:
>>
>> Note: the reason I bring this up on the core list is that I think it
>> would help people upgrade to Rails 4 more quickly if it were easier or
>> clearer as to how to test gems against (maybe) 3.1.x, 3.2.x, and 4.x,
at
>> least until mostly everyone switches to 4.
>>
>>
>> I test minitest-spec-rails against 3.0, 3.1, 3,2 and 4.0 using a mix of
>> appraisal and dummy_app that minimally configures itself depending
which
>> rails version it is testing against. Some links:
>>
>>
https://github.com/metaskills/**minitest-spec-rails<https://github.com/metaskills/minitest-spec-rails>
>> https://github.com/metaskills/**minitest-spec-rails/blob/**
>>
master/test/dummy_app/init.rb<https://github.com/metaskills/minitest-spec-rails/blob/master/test/dummy_app/init.rb>
>>
>>
>>  - HTH,
>>    Ken
>>
>>
>>
>>  --
> You received this message because you are subscribed to a topic in the
> Google Groups "Ruby on Rails: Core" group.
> To unsubscribe from this topic, visit
>
https://groups.google.com/d/topic/rubyonrails-core/AS6uGQhf8y0/unsubscribe?hl=en
> .
> To unsubscribe from this group and all its topics, send an email to
> rubyonrails-core+unsubscribe@googlegroups.com.
> To post to this group, send email to rubyonrails-core@googlegroups.com.
> Visit this group at http://groups.google.com/group/rubyonrails-core?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to rubyonrails-core+unsubscribe@googlegroups.com.
To post to this group, send email to rubyonrails-core@googlegroups.com.
Visit this group at http://groups.google.com/group/rubyonrails-core?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

On Apr 2, 2013, at 10:35 AM, Gary Weaver <garysweaver@gmail.com> wrote:
> On the flipside, having to check in a full Rails app in for every version
of Rails that I want to test with in my project would have a scary amount of
code duplication.
And that may be fine. Thanks to changes for engines in 3.0 and upward, this is
only a few lines of code. If you wanted to share models, etc, each directory can
be symlinked. So you could have test/dummy_app_3 and test/dummy_app_4. Each with
their own init.rb, etc. It just all depends on what you need, so the strategy
will vary, and I think that is OK. From where I sit, Rails has done the work we
need.

 - Ken

-- 
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to rubyonrails-core+unsubscribe@googlegroups.com.
To post to this group, send email to rubyonrails-core@googlegroups.com.
Visit this group at http://groups.google.com/group/rubyonrails-core?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

I''ve not seen that many symlinks checked into Github repos. I also
don''t
think that using them is all that clear because you may assume you are only 
changing a file in version X when that is really a symlink to version Y, 
and unless it is a hard link, then if you delete version Y, you just lost 
part of version X.

Overlays (or something similar) should allow for the most flexibility and 
longevity when it comes to testing a gem with various versions of Rails. 
Rails may have what is needed for each currently supported stable version 
when viewed as a silo of that version only, and provides some degree of 
ability to use different configuration for different Rails versions, but I 
am not convinced that conditional-laden config files on their own are the 
answer.

That said, I think that the approach you''ve taken (without symlinks
afaik)
is the best I''ve seen so far. I just wish that I knew that what I was 
testing was really something that a user could easily reproduce (i.e. it is 
working with known changes from a default Rails application that could be 
generated).

I don''t know that Rails itself needs to do anything other than what it
does
already to make this happen, but I brought it up here because it is 
something that affects anyone that wants to maintain a gem for Rails, and 
some of those gems might be part of Rails or used directly by Rails.


On Tuesday, April 2, 2013 12:01:20 PM UTC-4, Ken Collins
wrote:
>
>
> On Apr 2, 2013, at 10:35 AM, Gary Weaver <garys...@gmail.com
<javascript:>>
> wrote:
>
> On the flipside, having to check in a full Rails app in for every version 
> of Rails that I want to test with in my project would have a scary amount 
> of code duplication.
>
>
> And that may be fine. Thanks to changes for engines in 3.0 and upward, 
> this is only a few lines of code. If you wanted to share models, etc, each 
> directory can be symlinked. So you could have test/dummy_app_3 and 
> test/dummy_app_4. Each with their own init.rb, etc. It just all depends on 
> what you need, so the strategy will vary, and I think that is OK. From 
> where I sit, Rails has done the work we need.
>
>  - Ken
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to rubyonrails-core+unsubscribe@googlegroups.com.
To post to this group, send email to rubyonrails-core@googlegroups.com.
Visit this group at http://groups.google.com/group/rubyonrails-core?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

I''ve been meaning to discuss this topic more, as I''ve been
doing it for a
bunch of my gems lately. I have two that do this:

Draper: https://github.com/drapergem/draper

LocaleSetter: https://github.com/jcasimir/locale_setter/

Basically, I embed an entire Rails application into the gem, and then run
it against multiple versions of Rails on travis via env vars.

-- 
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to rubyonrails-core+unsubscribe@googlegroups.com.
To post to this group, send email to rubyonrails-core@googlegroups.com.
Visit this group at http://groups.google.com/group/rubyonrails-core?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Steve,

Thanks for sharing those! That''s a cool way to not have to use
appraisal
and to be able to use a single Gemfile instead.

I guess travis-ci starts up a new vm each time, so not managing gemsets or 
different bundles is not a problem unless you were to set a different env 
var value for for RAILS_VERSION at command-line, and did a bundle install, 
in which case you might end up with a newer version of a gem than you 
intended to use? Maybe you were going to manage gems in different bundles 
(vs. using rbenv-gemset or rvm gemsets)? It is good with the appraisal gem 
that it manages these separately so you can somewhat run in different 
environments locally before having to commit to test with travis. Did I 
miss something? I just did a quick look at it. The ability to test against 
Rails master looks cool.

Thanks again,
Gary

On Tuesday, April 2, 2013 1:23:36 PM UTC-4, Steve Klabnik
wrote:
>
> I''ve been meaning to discuss this topic more, as I''ve
been doing it for a
> bunch of my gems lately. I have two that do this: 
>
> Draper: https://github.com/drapergem/draper
>
> LocaleSetter: https://github.com/jcasimir/locale_setter/
>
> Basically, I embed an entire Rails application into the gem, and then run 
> it against multiple versions of Rails on travis via env vars.
>  
-- 
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to rubyonrails-core+unsubscribe@googlegroups.com.
To post to this group, send email to rubyonrails-core@googlegroups.com.
Visit this group at http://groups.google.com/group/rubyonrails-core?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Yes, there''s a theoretical possibility that it could happen. That said,
I
don''t ever run these tests by hand, they''re really just so
that Travis can
detect this kind of thing. If it was a problem, you can just blow away the
lock and re-bundle.

-- 
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to rubyonrails-core+unsubscribe@googlegroups.com.
To post to this group, send email to rubyonrails-core@googlegroups.com.
Visit this group at http://groups.google.com/group/rubyonrails-core?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.