Rails - May 2006 - second assert_tag failling in rails integration test

Hi all,

I''m fairly new to rails and ruby, but I''ve come across an
interesting
glitch and I''m not sure whether I''ve just got something wrong
with my
assumptions or if it really is an error in the underlying framework...

I''ve created a bare-bones set of tests to show you where the problem
is.
What follows are the steps I went through to reproduce the issue.

1) run "rails test"
2) create a controller called "test_controller":

class TestController < ApplicationController
  def one
    render_text(''one'')
  end

  def two
    render_text(''two'')
  end
end


3) create an integration test:
require "#{File.dirname(__FILE__)}/../test_helper"

class MyTest < ActionController::IntegrationTest
  def test_one
    post(''/test/one'')
    assert_tag(:content => /one/)
    post(''/test/two'')
    assert_match(/two/, @response.body)
    assert_tag(:content => /two/)
  end
end


4) ran the integration test.


So, I expected the assertions to all pass, but instead I got the
following error:

Loaded suite test/integration/test_one
Started
F
Finished in 0.131389 seconds.

  1) Failure:
test_one(MyTest)
    [test/integration/test_one.rb:9:in `test_one''
    
/usr/local/lib/ruby/gems/1.8/gems/actionpack-1.12.1/lib/action_controller/integration.rb:427:in
`run'']:
expected tag, but no tag found matching {:content=>/two/} in:
"two".
<nil> is not true.

1 tests, 3 assertions, 1 failures, 0 errors


As you can see, it successfully passed the match against @response.body,
but failed the second assert_tag.

What''s interesting is that if you remove the initial assert_tag it
works
just fine.

Any ideas?

Taryn
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://wrath.rubyonrails.org/pipermail/rails/attachments/20060504/5ccd6730/attachment.html

On May 3, 2006, at 10:53 PM, Taryn East wrote:
> require "#{File.dirname(__FILE__)}/../test_helper"
>
> class MyTest < ActionController::IntegrationTest
>   def test_one
>     post(''/test/one'')
>     assert_tag(:content => /one/)
>     post(''/test/two'')
>     assert_match(/two/, @response.body)
>     assert_tag(:content => /two/)
>   end
> end
>
> Any ideas?
assert_match /two/, @response.body

-- 
Eric Hodel - drbrain@segment7.net - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant

http://trackmap.robotcoop.com

On Thu, 2006-05-04 at 11:01 -0700, Eric Hodel wrote:
> On May 3, 2006, at 10:53 PM, Taryn East wrote:
> 
> > require "#{File.dirname(__FILE__)}/../test_helper"
> >
> > class MyTest < ActionController::IntegrationTest
> >   def test_one
> >     post(''/test/one'')
> >     assert_tag(:content => /one/)
> >     post(''/test/two'')
> >     assert_match(/two/, @response.body)
> >     assert_tag(:content => /two/)
> >   end
> > end
> >
> > Any ideas?
> 
> assert_match /two/, @response.body
> 
Yes, well that is the path we are currently forced to take (as you see
in the example I gave above). But I was still kind of wondering why
assert_tag doesn''t work the way that I expect it to.

To give a couple of other examples:

These ones work:

def test_one
   post(''/test/one'')
   assert_tag(:content => /one/)
   assert_tag(:content => /n/)
   post(''/test/two'')
   assert_match(/two/, @response.body)
end

def test_two
   post(''/test/one'')
   post(''/test/two'')
   assert_match(/two/, @response.body)
   assert_tag(:content => /two/)
   assert_tag(:content => /w/)
end


These ones don''t

def test_three
   post(''/test/one'')
   assert_tag(:content => /one/)
   post(''/test/two'')
   assert_match(/two/, @response.body)
   assert_tag(:content => /two/)  # breaks here
end
def test_four
   post(''/test/one'')
   assert_tag(:content => /one/)
   assert_tag(:content => /n/)
   post(''/test/two'')
   assert_match(/two/, @response.body)
   assert_tag(:content => /two/) # breaks here
   assert_tag(:content => /w/)
end

ie - you need:
 *  multiple assert_tag calls and at least two post/get calls.
 * At least one assert_tag has to occur both before and after the second
get/post.
 * If there are multiple calls before the get/post - they all work.
 * It fails on the first assert_tag that occurs after the second
get/post.

What I can only assume (not knowing the underlying architecture) is that
the first assert_tag does something to internal variables (the book says
it "parses the response into a DOM")... then when the next get/post
occurs, the response changes... but the next assert_tag doesn''t check
that the response has changed - it is still using the old parsed DOM.

Cheers,
Taryn
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://wrath.rubyonrails.org/pipermail/rails/attachments/20060505/6ebcfe0f/attachment.html

You know I never did get a real response[1] to this question...

Does anyone else here actually use integration tests? Does they work for
you? Do you not do what we are doing? Are we using the wrong kind of
test here?
Has anybody got any ideas?


To recap on the issue I had:
the steps I went through to reproduce the issue.

1) run "rails test"
2) create a controller called "test_controller":

class TestController < ApplicationController
  def one
    render_text(''one'')
  end
  def two
    render_text(''two'')
  end
end

3) create an integration test:
require "#{File.dirname(__FILE__)}/../test_helper"

class MyTest < ActionController::IntegrationTest
  def test_one
    post(''/test/one'')
    assert_tag(:content => /one/)
    post(''/test/two'')
    assert_match(/two/, @response.body)
    assert_tag(:content => /two/)
  end
end

4) run the integration test.

So, I expected the assertions to all pass, but instead I got the
following error:

Loaded suite test/integration/test_one
Started
F
Finished in 0.131389 seconds.

  1) Failure:
test_one(MyTest)
    [test/integration/test_one.rb:9:in `test_one''
    
/usr/local/lib/ruby/gems/1.8/gems/actionpack-1.12.1/lib/action_controller/integration.rb:427:in
`run'']:
expected tag, but no tag found matching {:content=>/two/} in:
"two".
<nil> is not true.

1 tests, 3 assertions, 1 failures, 0 errors



To give a couple of other examples:

These ones work:

def test_one
   post(''/test/one'')
   assert_tag(:content => /one/)
   assert_tag(:content => /n/)
   post(''/test/two'')
   assert_match(/two/, @response.body)
end

def test_two
   post(''/test/one'')
   post(''/test/two'')
   assert_match(/two/, @response.body)
   assert_tag(:content => /two/)
   assert_tag(:content => /w/)
end


These ones don''t

def test_three
   post(''/test/one'')
   assert_tag(:content => /one/)
   post(''/test/two'')
   assert_match(/two/, @response.body)
   assert_tag(:content => /two/)  # breaks here
end
def test_four
   post(''/test/one'')
   assert_tag(:content => /one/)
   assert_tag(:content => /n/)
   post(''/test/two'')
   assert_match(/two/, @response.body)
   assert_tag(:content => /two/) # breaks here
   assert_tag(:content => /w/)
end

ie - you need:
*  multiple assert_tag calls and at least two post/get calls.
* At least one assert_tag has to occur both before and after the second
get/post.
* If there are multiple calls before the get/post - they all work.
* It fails on the first assert_tag that occurs after the second
get/post.

What I can only assume (not knowing the underlying architecture) is that
the first assert_tag does something to internal variables (the book says
it "parses the response into a DOM")... then when the next get/post
occurs, the response changes... but the next assert_tag doesn''t check
that the response has changed - it is still using the old parsed DOM.



Cheers,
Taryn



[1] Yes, I got Eric''s response of:
assert_match /two/, @response.body
which we are already using... but while it''s possible to write a regexp
to match on something like:
     assert_tag(:tag => ''td'', 
                :attributes => {:id => ''status-cell''},
                :child => {:tag => ''img'', 
                           :attributes => {:alt =>
%r''Complete''}}
                )
it''s really annoying... :P


-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://wrath.rubyonrails.org/pipermail/rails/attachments/20060522/99d73073/attachment.html

On May 21, 2006, at 6:46 PM, Taryn East wrote:
> You know I never did get a real response[1] to this question...
>
> Does anyone else here actually use integration tests?
I don''t, but I know people do.
> Does they work for you?
They seem to for the people that use them.
> Do you not do what we are doing?
They probably don''t, integration testing isn''t about testing
HTML.
> Are we using the wrong kind of test here?
You should be writing functional tests when using assert_tag.

-- 
Eric Hodel - drbrain@segment7.net - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant

http://trackmap.robotcoop.com

On Mon, 2006-05-22 at 11:47 -0700, Eric Hodel wrote:
> On May 21, 2006, at 6:46 PM, Taryn East wrote:
> > Does anyone else here actually use integration tests?
> 
> I don''t, but I know people do.
I haven''t heard anything from anybody else that does... and I guess
I''m
just beginning to wonder...

> > Does they work for you?
> 
> They seem to for the people that use them.
> 
> > Do you not do what we are doing?
> 
> They probably don''t, integration testing isn''t about
testing HTML.
We''re not actually testing the html itself, we''re testing the
flow of
screens... which I assumed is what integration tests are for. It''s just
that to test that we''re arriving at the right screen with the right
data
on it we use assert_tag.

We also use assert_response and assert_template (which seem to be the
main thing used in integration tests I''ve found) but you have to test
not only that *an* edit screen popped up, but that the right one popped
up - and the way to test that is to check that it was populated by the
data that you actually asked for. We don''t test that every bit of data
is there (that''s what I believe belongs in the realm of functional
testing - though maybe I have that wrong too) but we need to check that
something is right so for example we check for the existence of one key
field, or that one input has been pre-populated with previously-chosen
data. We check that, for example, this occurs whether or not the user is
still using the same session, or if they login with a different session
and come to the same page.

If we are doing this in the wrong way then I would be very appreciative
of any assistance in directing us towards the correct way to test this
sort of thing.

> 
> > Are we using the wrong kind of test here?
> 
> You should be writing functional tests when using assert_tag.


Ok, so then what are integration tests for? It''s quite possible that we
have misunderstood what level of testing we need... but the article I
read on it here:
http://jamis.jamisbuck.org/articles/2006/03/09/integration-testing-in-rails-1-1

seemed to indicate that integration tests were used for screen flow...
which is what (afaics) we are doing.




In any case even if we are using integration tests completely
incorrectly.... there is still a bug in the framework if assert_tag is
failing.


Cheers and thanks,
Taryn


-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://wrath.rubyonrails.org/pipermail/rails/attachments/20060522/6f0ca00f/attachment.html

On May 22, 2006, at 4:09 PM, Taryn East wrote:
> On Mon, 2006-05-22 at 11:47 -0700, Eric Hodel wrote:
>> On May 21, 2006, at 6:46 PM, Taryn East wrote:
>
>>> Does they work for you? They seem to for the people that use them.
>>>
>>> Do you not do what we are doing?
>>
>> They probably don''t, integration testing isn''t about
testing HTML.
I''m going to take the liberty of rearranging this paragraph.
> we''re testing the flow of screens... which I assumed is what  
> integration tests are for.
Correct.
> We''re not actually testing the html itself, [...] It''s
just that to
> test that we''re arriving at the right screen with the right data
on
> it we use assert_tag.
If you''re using assert_tag you''re testing the HTML.

assert_tag tests at the wrong level.  Test assigns instead.

Your functional tests should ensure that data on the controller side  
correctly shows up in the right place on the HTML.  If you''re using  
assert_tag you''re testing in two places.
> We also use assert_response and assert_template (which seem to be  
> the main thing used in integration tests I''ve found) but you have
> to test not only that *an* edit screen popped up, but that the  
> right one popped up and the way to test that is to check that it  
> was populated by the data that you actually asked for.
This is the responsibility of functional tests.  When your functional  
tests ensure that data is in the right spot then you can test based  
purely on assigns in your integration tests and cut out a ton of  
(duplicated) code.
> We don''t test that every bit of data is there (that''s
what I
> believe belongs in the realm of functional testing - though maybe I  
> have that wrong too) but we need to check that something is right  
> so for example we check for the existence of one key field, or that  
> one input has been pre-populated with previously-chosen data. We  
> check that, for example, this occurs whether or not the user is  
> still using the same session, or if they login with a different  
> session and come to the same page.
And that data gets there via assigns, so exploit that fact and make  
your tests simpler.  The bigger the test the more likely it is wrong.
> If we are doing this in the wrong way then I would be very  
> appreciative of any assistance in directing us towards the correct  
> way to test this sort of thing.
I actually think that even functional tests are too high level, so I  
wrote Test::Rails to break functional tests down into controller  
tests and view tests.  I found my functional tests repeating  
themselves too much so I needed to get closer to what the real  
responsibilities of controllers and views are.

Controller tests ensure that URL params map to the correct assigns  
and view tests ensure that assigns map to the correct HTML.  Once I  
know these tests all pass then integration tests can rely strictly on  
assigns and I avoid tons of code.
>>> Are we using the wrong kind of test here?
>>
>> You should be writing functional tests when using assert_tag.
>
> Ok, so then what are integration tests for? It''s quite possible  
> that we have misunderstood what level of testing we need... but the  
> article I read on it here:
> http://jamis.jamisbuck.org/articles/2006/03/09/integration-testing- 
> in-rails-1-1
>
> seemed to indicate that integration tests were used for screen  
> flow...  which is what (afaics) we are doing.
Testing is broken into multiple layers.  What you test on one layer  
shouldn''t be tested on another.  What is tested on one layer you can  
rely on in a higher layer.

Here''s an untested example:

class EntryControllerTest < Test::Rails::ControllerTestCase

   fixtures :entries

   def test_view
     get :view, :id => entries(:frogs).id
     assert_response :success
     assert_template ''entry/view''
     assert_assigned :entry, entries(:frogs)
   end

   def test_edit
     get :edit, :id => entries(:frogs).id
     assert_response :success
     assert_template ''entry/edit''
     assert_assigned :entry, entries(:frogs)
   end

   def test_update
     post :update, :id => entries(:frogs).id, :body => ''Frogs
rock!''
     assert_redirected_to "/entry/view/#{entries(:frogs).id}"
     entries(:frogs).reload
     assert_equal entries(:frogs).body, ''Frogs rock!''
   end

end

class EntryViewTest < Test::Rails::ViewTestCase

   fixtures :entries

   def test_view
     assigns[:entry] = entries(:frogs)
     render
     assert_tag :tag => ''p'', :content =>
entries(:frogs).body
   end

   def test_edit
     assigns[:entry] = entries(:frogs)
     render
     form_url = "/entries/update/#{entries(:frogs).id}"
     assert_post_form form_url
     assert_tag_in_form form_url, :tag => ''textarea'',
                                  :content => entries(:frogs).body
   end

end

Now I know that all the URLs and data will show up correctly I can do  
an integration test.

class EntryIntegrationTest < ActionController::IntegrationTest

   fixtures :entries

   def test_editing
     entry = entries(:frogs)
     get "/entry/edit/#{entry.id}"
     assert_response :success
     assert_template ''entry/edit''

     post "/entry/update/#{entry.id}" :body => ''Frogs
rock!''
     assert_response :redirect

     follow_redirect!
     assert_success
     assert_template ''entry/view''
   end

end

I already know all the URLs in the HTML will be correct for that  
template so I don''t need to re-test that.
> In any case even if we are using integration tests completely  
> incorrectly.... there is still a bug in the framework if assert_tag  
> is failing.
I know exactly what the problem is, and it isn''t a problem if you  
write your integration tests correctly.

-- 
Eric Hodel - drbrain@segment7.net - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant

http://trackmap.robotcoop.com