rspec users - Apr 2009 - [rspec] loading specs as an object model without running them

I''d like to load all of my examples into memory without actually
running them, so I can traverse the object tree and get descriptions
etc. ?I''m finding it tricky to figure out how to do that though.
There seems to be an inherent assumption in the framework that that
examples will actually run.

So basically, I want to do something like this:

examples = load_examples_without_running
examples.each {|example| puts example.description}

def load_examples_without_running
?# what goes here?
end


Does anyone have an idea about how I might do that?

-- Marty Andrews

On 29 Apr 2009, at 07:23, Marty Andrews wrote:
> I''d like to load all of my examples into memory without actually
> running them, so I can traverse the object tree and get descriptions
> etc.  I''m finding it tricky to figure out how to do that though.
> There seems to be an inherent assumption in the framework that that
> examples will actually run.
>
> So basically, I want to do something like this:
>
> examples = load_examples_without_running
> examples.each {|example| puts example.description}
>
> def load_examples_without_running
>  # what goes here?
> end
>
>
> Does anyone have an idea about how I might do that?
Use the dry-run option?

Matt Wynne
http://blog.mattwynne.net
http://www.songkick.com

On Wed, Apr 29, 2009 at 1:23 AM, Marty Andrews <marty at martyandrews.net>
wrote:
> I''d like to load all of my examples into memory without actually
> running them, so I can traverse the object tree and get descriptions
> etc. ?I''m finding it tricky to figure out how to do that though.
> There seems to be an inherent assumption in the framework that that
> examples will actually run.
>
> So basically, I want to do something like this:
>
> examples = load_examples_without_running
> examples.each {|example| puts example.description}
There is a --dry-run command line switch that will do this for you,
but I''ve been considering deprecating it because it can''t work
consistently with matcher-generated descriptions:

describe RSpec do
  it { should be_concise }
end

$ spec rspec_spec.rb
RSpec
  should be concise

In this case, the be_concise matcher generates the text "should be
concise" when it is run. If you don''t run it, there is no
description.
As this approach becomes more and more popular, the --dry-run not only
becomes less useful, it actually becomes somewhat of a frustration.

As for getting a handle on the objects, this is something that is not
really baked into RSpec, but will definitely be baked into rspec-2.0.
Work on this will start later this year (probably not until after the
rspec book is off to print). Goals for it include a very clean
separation of the DSL and the underlying object model and a very
lightweight runner.

So in the short run, your best option is to use --dry-run and
understand that you only get value out of it if you write docstrings
for all of your examples.

Cheers,
David
>
> def load_examples_without_running
> ?# what goes here?
> end
>
>
> Does anyone have an idea about how I might do that?
>
> -- Marty Andrews
> _______________________________________________
> rspec-users mailing list
> rspec-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/rspec-users
>

On 29/04/2009, at 8:54 PM, David Chelimsky wrote:
> On Wed, Apr 29, 2009 at 1:23 AM, Marty Andrews  
> <marty at martyandrews.net> wrote:
>> I''d like to load all of my examples into memory without
actually
>> running them, so I can traverse the object tree and get descriptions
>> etc.  I''m finding it tricky to figure out how to do that
though.
>> There seems to be an inherent assumption in the framework that that
>> examples will actually run.
>>
>> So basically, I want to do something like this:
>>
>> examples = load_examples_without_running
>> examples.each {|example| puts example.description}
[snip]
> As for getting a handle on the objects, this is something that is not
> really baked into RSpec, but will definitely be baked into rspec-2.0.
> Work on this will start later this year (probably not until after the
> rspec book is off to print). Goals for it include a very clean
> separation of the DSL and the underlying object model and a very
> lightweight runner.
>
> So in the short run, your best option is to use --dry-run and
> understand that you only get value out of it if you write docstrings
> for all of your examples.
Ok.  It''s actually a bit more complicate than that.  Let me give you  
an insight into where I''m headed.

I''m working into a big corporate in Australia, and they are demanding  
traceability of tests back to requirements.  I''m already using Mingle  
to manage story cards, and I''m hoping to let the testers build in the  
traceability via code.  So I''ve got a Mixin to the examples so that I  
can declare the id of the story that the test maps to.  It looks  
something like this:

describe FunctionalArea do
   it "automates some acceptance test declared on a story" do
     traces_to_story 35
     # test implementation
   end
end

So now, I want to be able to walk the object tree, getting the  
descriptions *and* the story id''s.  Mingle has a nice RESTful API that
I''ve already used to slurp up detail about stories.  Now I just want  
to wire in this last bit of information to tick the traceability box  
without any human needing to do the work.

I''ll have a look at the --dry-run option and see where it gets me.  If
you''ve got any more tips, let me know.  I''ll send in a code
snippet if
I manage to get it going :)

-- Marty

On Wed, Apr 29, 2009 at 6:09 AM, Marty Andrews <marty at martyandrews.net>
wrote:
>
> On 29/04/2009, at 8:54 PM, David Chelimsky wrote:
>
>> On Wed, Apr 29, 2009 at 1:23 AM, Marty Andrews <marty at
martyandrews.net>
>> wrote:
>>>
>>> I''d like to load all of my examples into memory without
actually
>>> running them, so I can traverse the object tree and get
descriptions
>>> etc. ?I''m finding it tricky to figure out how to do that
though.
>>> There seems to be an inherent assumption in the framework that that
>>> examples will actually run.
>>>
>>> So basically, I want to do something like this:
>>>
>>> examples = load_examples_without_running
>>> examples.each {|example| puts example.description}
>
> [snip]
>
>> As for getting a handle on the objects, this is something that is not
>> really baked into RSpec, but will definitely be baked into rspec-2.0.
>> Work on this will start later this year (probably not until after the
>> rspec book is off to print). Goals for it include a very clean
>> separation of the DSL and the underlying object model and a very
>> lightweight runner.
>>
>> So in the short run, your best option is to use --dry-run and
>> understand that you only get value out of it if you write docstrings
>> for all of your examples.
>
> Ok. ?It''s actually a bit more complicate than that. ?Let me give
you an
> insight into where I''m headed.
>
> I''m working into a big corporate in Australia, and they are
demanding
> traceability of tests back to requirements. ?I''m already using
Mingle to
> manage story cards, and I''m hoping to let the testers build in the
> traceability via code. ?So I''ve got a Mixin to the examples so
that I can
> declare the id of the story that the test maps to. ?It looks something like
> this:
>
> describe FunctionalArea do
> ?it "automates some acceptance test declared on a story" do
> ? ?traces_to_story 35
> ? ?# test implementation
> ?end
> end
>
> So now, I want to be able to walk the object tree, getting the descriptions
> *and* the story id''s. ?Mingle has a nice RESTful API that
I''ve already used
> to slurp up detail about stories. ?Now I just want to wire in this last bit
> of information to tick the traceability box without any human needing to do
> the work.
>
> I''ll have a look at the --dry-run option and see where it gets me.
?If
> you''ve got any more tips, let me know. ?I''ll send in a
code snippet if I
> manage to get it going :)
There *is* something you can *almost* use for this, but it''s
incomplete at the moment. RSpec let''s you write your own output
formatters, which receive messages for every example group and example
as they are run. Currently, the object passed to example_started (and
a few other methods) is an ExampleGroupProxy, which includes an
options hash passed to the example. So if you do this:

it "automates some acceptance test declared on a story",
    :traces_to_story => 25 do
# ...
end

Then you could do stuff like:

def example_started(example_proxy)
  traces_to_story(example_proxy.options[:traces_to_story]
end

This should solve your problem assuming that you have docstrings for
all of your examples. The big missing piece at this moment is that the
ExampleGroupProxy (passed to example_group_started) doesn''t include
the options hash, but I''ll happily add that tonight.

To see this in action, try this:

* in options_formatter.rb

require ''spec/runner/formatter/base_text_formatter''

class OptionsFormatter < Spec::Runner::Formatter::BaseTextFormatter
  def example_started(proxy)
    puts proxy.options.inspect unless proxy.options.empty?
  end
end

* in example_spec.rb

require ''spec''

describe "example" do
  it "supports options", :like => ''these'' do
    # no need for anything here
  end
end

Now, on the command line:

$ spec --require options_formatter.rb example_spec.rb --format OptionsFormatter
{:like => ''these''}

Formatter API is here:
http://rspec.rubyforge.org/rspec/1.2.4/classes/Spec/Runner/Formatter/BaseFormatter.html

Let me know if this is a good direction for you and I''ll add the
options hash to the example group proxy.

Cheers,
David

>
> -- Marty
>
> _______________________________________________
> rspec-users mailing list
> rspec-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/rspec-users
>

On 29/04/2009, at 9:41 PM, David Chelimsky wrote:
> On Wed, Apr 29, 2009 at 6:09 AM, Marty Andrews  
> <marty at martyandrews.net> wrote:
>>
>> On 29/04/2009, at 8:54 PM, David Chelimsky wrote:
>>
>>> On Wed, Apr 29, 2009 at 1:23 AM, Marty Andrews <marty at
martyandrews.net
>>> >
>>> wrote:
>>>>
>>>> I''d like to load all of my examples into memory
without actually
>>>> running them, so I can traverse the object tree and get  
>>>> descriptions
>>>> etc.  I''m finding it tricky to figure out how to do
that though.
>>>> There seems to be an inherent assumption in the framework that
that
>>>> examples will actually run.
>>>>
>>>> So basically, I want to do something like this:
>>>>
>>>> examples = load_examples_without_running
>>>> examples.each {|example| puts example.description}
>>
>> [snip]
>>
>>> As for getting a handle on the objects, this is something that is  
>>> not
>>> really baked into RSpec, but will definitely be baked into  
>>> rspec-2.0.
>>> Work on this will start later this year (probably not until after  
>>> the
>>> rspec book is off to print). Goals for it include a very clean
>>> separation of the DSL and the underlying object model and a very
>>> lightweight runner.
>>>
>>> So in the short run, your best option is to use --dry-run and
>>> understand that you only get value out of it if you write
docstrings
>>> for all of your examples.
>>
>> Ok.  It''s actually a bit more complicate than that.  Let me
give
>> you an
>> insight into where I''m headed.
>>
>> I''m working into a big corporate in Australia, and they are
demanding
>> traceability of tests back to requirements.  I''m already using
>> Mingle to
>> manage story cards, and I''m hoping to let the testers build in
the
>> traceability via code.  So I''ve got a Mixin to the examples so
that
>> I can
>> declare the id of the story that the test maps to.  It looks  
>> something like
>> this:
>>
>> describe FunctionalArea do
>>  it "automates some acceptance test declared on a story" do
>>    traces_to_story 35
>>    # test implementation
>>  end
>> end
>>
>> So now, I want to be able to walk the object tree, getting the  
>> descriptions
>> *and* the story id''s.  Mingle has a nice RESTful API that
I''ve
>> already used
>> to slurp up detail about stories.  Now I just want to wire in this  
>> last bit
>> of information to tick the traceability box without any human  
>> needing to do
>> the work.
>>
>> I''ll have a look at the --dry-run option and see where it gets
me.
>> If
>> you''ve got any more tips, let me know.  I''ll send in
a code snippet
>> if I
>> manage to get it going :)
>
> There *is* something you can *almost* use for this, but it''s
> incomplete at the moment. RSpec let''s you write your own output
> formatters, which receive messages for every example group and example
> as they are run. Currently, the object passed to example_started (and
> a few other methods) is an ExampleGroupProxy, which includes an
> options hash passed to the example. So if you do this:
>
> it "automates some acceptance test declared on a story",
>    :traces_to_story => 25 do
> # ...
> end
>
> Then you could do stuff like:
>
> def example_started(example_proxy)
>  traces_to_story(example_proxy.options[:traces_to_story]
> end
>
> This should solve your problem assuming that you have docstrings for
> all of your examples. The big missing piece at this moment is that the
> ExampleGroupProxy (passed to example_group_started) doesn''t
include
> the options hash, but I''ll happily add that tonight.
Yep - no problems there.  I reckon about 98% have docstrings.  I can  
easily add the remainders.


> To see this in action, try this:
>
> * in options_formatter.rb
>
> require ''spec/runner/formatter/base_text_formatter''
>
> class OptionsFormatter < Spec::Runner::Formatter::BaseTextFormatter
>  def example_started(proxy)
>    puts proxy.options.inspect unless proxy.options.empty?
>  end
> end
>
> * in example_spec.rb
>
> require ''spec''
>
> describe "example" do
>  it "supports options", :like => ''these'' do
>    # no need for anything here
>  end
> end
>
> Now, on the command line:
>
> $ spec --require options_formatter.rb example_spec.rb --format  
> OptionsFormatter
> {:like => ''these''}
>
> Formatter API is here:
>
http://rspec.rubyforge.org/rspec/1.2.4/classes/Spec/Runner/Formatter/BaseFormatter.html
>
> Let me know if this is a good direction for you and I''ll add the
> options hash to the example group proxy.
I reckon this would be great *if* it will still work with the --dry- 
run option.  Most of the examples are actually integration tests that  
call out to other tools as they run, so the build is S..L..O..W (over  
an hour).  I want to be able to generate the report without actually  
running the tests.

-- Marty

On Wed, Apr 29, 2009 at 6:50 AM, Marty Andrews <marty at martyandrews.net>
wrote:
>
> On 29/04/2009, at 9:41 PM, David Chelimsky wrote:
>
>> On Wed, Apr 29, 2009 at 6:09 AM, Marty Andrews <marty at
martyandrews.net>
>> wrote:
>>>
>>> On 29/04/2009, at 8:54 PM, David Chelimsky wrote:
>>>
>>>> On Wed, Apr 29, 2009 at 1:23 AM, Marty Andrews <marty at
martyandrews.net>
>>>> wrote:
>>>>>
>>>>> I''d like to load all of my examples into memory
without actually
>>>>> running them, so I can traverse the object tree and get
descriptions
>>>>> etc. ?I''m finding it tricky to figure out how to
do that though.
>>>>> There seems to be an inherent assumption in the framework
that that
>>>>> examples will actually run.
>>>>>
>>>>> So basically, I want to do something like this:
>>>>>
>>>>> examples = load_examples_without_running
>>>>> examples.each {|example| puts example.description}
>>>
>>> [snip]
>>>
>>>> As for getting a handle on the objects, this is something that
is not
>>>> really baked into RSpec, but will definitely be baked into
rspec-2.0.
>>>> Work on this will start later this year (probably not until
after the
>>>> rspec book is off to print). Goals for it include a very clean
>>>> separation of the DSL and the underlying object model and a
very
>>>> lightweight runner.
>>>>
>>>> So in the short run, your best option is to use --dry-run and
>>>> understand that you only get value out of it if you write
docstrings
>>>> for all of your examples.
>>>
>>> Ok. ?It''s actually a bit more complicate than that. ?Let
me give you an
>>> insight into where I''m headed.
>>>
>>> I''m working into a big corporate in Australia, and they
are demanding
>>> traceability of tests back to requirements. ?I''m already
using Mingle to
>>> manage story cards, and I''m hoping to let the testers
build in the
>>> traceability via code. ?So I''ve got a Mixin to the
examples so that I can
>>> declare the id of the story that the test maps to. ?It looks
something
>>> like
>>> this:
>>>
>>> describe FunctionalArea do
>>> ?it "automates some acceptance test declared on a story"
do
>>> ? traces_to_story 35
>>> ? # test implementation
>>> ?end
>>> end
>>>
>>> So now, I want to be able to walk the object tree, getting the
>>> descriptions
>>> *and* the story id''s. ?Mingle has a nice RESTful API that
I''ve already
>>> used
>>> to slurp up detail about stories. ?Now I just want to wire in this
last
>>> bit
>>> of information to tick the traceability box without any human
needing to
>>> do
>>> the work.
>>>
>>> I''ll have a look at the --dry-run option and see where it
gets me. ?If
>>> you''ve got any more tips, let me know. ?I''ll send
in a code snippet if I
>>> manage to get it going :)
>>
>> There *is* something you can *almost* use for this, but it''s
>> incomplete at the moment. RSpec let''s you write your own
output
>> formatters, which receive messages for every example group and example
>> as they are run. Currently, the object passed to example_started (and
>> a few other methods) is an ExampleGroupProxy, which includes an
>> options hash passed to the example. So if you do this:
>>
>> it "automates some acceptance test declared on a story",
>> ? :traces_to_story => 25 do
>> # ...
>> end
>>
>> Then you could do stuff like:
>>
>> def example_started(example_proxy)
>> ?traces_to_story(example_proxy.options[:traces_to_story]
>> end
>>
>> This should solve your problem assuming that you have docstrings for
>> all of your examples. The big missing piece at this moment is that the
>> ExampleGroupProxy (passed to example_group_started) doesn''t
include
>> the options hash, but I''ll happily add that tonight.
>
> Yep - no problems there. ?I reckon about 98% have docstrings. ?I can easily
> add the remainders.
>
>
>
>> To see this in action, try this:
>>
>> * in options_formatter.rb
>>
>> require ''spec/runner/formatter/base_text_formatter''
>>
>> class OptionsFormatter < Spec::Runner::Formatter::BaseTextFormatter
>> ?def example_started(proxy)
>> ? puts proxy.options.inspect unless proxy.options.empty?
>> ?end
>> end
>>
>> * in example_spec.rb
>>
>> require ''spec''
>>
>> describe "example" do
>> ?it "supports options", :like => ''these''
do
>> ? # no need for anything here
>> ?end
>> end
>>
>> Now, on the command line:
>>
>> $ spec --require options_formatter.rb example_spec.rb --format
>> OptionsFormatter
>> {:like => ''these''}
>>
>> Formatter API is here:
>>
>>
http://rspec.rubyforge.org/rspec/1.2.4/classes/Spec/Runner/Formatter/BaseFormatter.html
>>
>> Let me know if this is a good direction for you and I''ll add
the
>> options hash to the example group proxy.
>
> I reckon this would be great *if* it will still work with the --dry-run
> option. ?Most of the examples are actually integration tests that call out
> to other tools as they run, so the build is S..L..O..W (over an hour). ?I
> want to be able to generate the report without actually running the tests.
I''m really sorry about this, but I just couldn''t wait for
tonight to add this:

 
http://github.com/dchelimsky/rspec/commit/a4b3aac62e940c50077bdeb7d9ad5ee11e02116d

The release *will* have to wait for tonight, but you can easily build
the gem locally: http://wiki.github.com/dchelimsky/rspec/edge

Cheers,
David
>
> -- Marty
>
>
> _______________________________________________
> rspec-users mailing list
> rspec-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/rspec-users
>

[snip]
> I''m really sorry about this, but I just couldn''t wait for
tonight to
> add this:
>
> 
http://github.com/dchelimsky/rspec/commit/a4b3aac62e940c50077bdeb7d9ad5ee11e02116d
>
> The release *will* have to wait for tonight, but you can easily build
> the gem locally: http://wiki.github.com/dchelimsky/rspec/edge
Thanks David :)

I''m in Australia, and it''s already getting late here.  Your
tonight
will be during the day tomorrow for me.  I''ll give it a go then.

-- Marty

>>> So basically, I want to do something like this:
>>>
>>> examples = load_examples_without_running
>>> examples.each {|example| puts example.description}
[snip]
> I''m working into a big corporate in Australia, and they are
demanding
> traceability of tests back to requirements.  I''m already using
Mingle to
> manage story cards, and I''m hoping to let the testers build in the
> traceability via code.  So I''ve got a Mixin to the examples so
that I
> can declare the id of the story that the test maps to.  It looks  
> something like this:
>
> describe FunctionalArea do
>   it "automates some acceptance test declared on a story" do
>     traces_to_story 35
>     # test implementation
>   end
> end
I would have expected Cucumber to map stories to scenarios/examples
by having them in the same file (and a @tag if someone insists that
a number referrring an external doc is better for tracability).

I would have expected RSpec to cover a lower level that is of no
interest to the person wanting traceability, as that person can not
judge whether the code works anyway (YMMV). Besides, a spec can
apply to more than one requirement; or to no particular requirement at all.

I''m expecting this from regulations that deal with medical devices.
Validation has to happen on story level; risk mitigation may apply
to lower levels, though. Our department is working on several projects
that''ll result in medical devices and we are learning how regulations,
audits and things work for real.

So I''d like to hear why you are trying to do what you do.

There can be other requirements for other reasons. What traceability
are you looking for in these tests? Why do you look at the RSpec
level?

Bye,
Kero.
___
How can I change the world if I can''t even change myself?
  -- Faithless, Salva Mea

On 30/04/2009, at 7:59 AM, Kero van Gelder wrote:
>>>> So basically, I want to do something like this:
>>>>
>>>> examples = load_examples_without_running
>>>> examples.each {|example| puts example.description}
>
> [snip]
>
>> I''m working into a big corporate in Australia, and they are
demanding
>> traceability of tests back to requirements.  I''m already using
>> Mingle to
>> manage story cards, and I''m hoping to let the testers build in
the
>> traceability via code.  So I''ve got a Mixin to the examples so
that I
>> can declare the id of the story that the test maps to.  It looks
>> something like this:
>>
>> describe FunctionalArea do
>>  it "automates some acceptance test declared on a story" do
>>    traces_to_story 35
>>    # test implementation
>>  end
>> end
>
> I would have expected Cucumber to map stories to scenarios/examples
> by having them in the same file (and a @tag if someone insists that
> a number referrring an external doc is better for tracability).
>
> I would have expected RSpec to cover a lower level that is of no
> interest to the person wanting traceability, as that person can not
> judge whether the code works anyway (YMMV). Besides, a spec can
> apply to more than one requirement; or to no particular requirement  
> at all.
>
> I''m expecting this from regulations that deal with medical
devices.
> Validation has to happen on story level; risk mitigation may apply
> to lower levels, though. Our department is working on several projects
> that''ll result in medical devices and we are learning how
regulations,
> audits and things work for real.
>
> So I''d like to hear why you are trying to do what you do.
Sure.

> There can be other requirements for other reasons. What traceability
> are you looking for in these tests? Why do you look at the RSpec
> level?
I don''t necessarily disagree with what you are saying, but the project
was going long before cucumber existed, and we already had a heavy  
investment in tests using rspec.  It''s a data warehousing project, so  
there''s little or no UI to speak of.  We built a DSL on top of RSpec  
to allow us to manipulate data and verify results after ETL jobs run.   
Most of tests set up data, run the ETL job (calling out to an external  
tool) and then verify that the results look the way they''re supposed
to.

So to us, RSpec isn''t a ''lower level''.  Perhaps if
the project was
starting now, I''d consider cucumber.  Our investment in a DSL on RSpec
is reasonably heavy though, and is working really well for us.  I''d  
rather enhance that a bit further than switch frameworks for the sake  
of one feature.

-- Marty