rspec users - Nov 2008 - How to spec a (Trollop-based) binary’s internal state?

I?m trying to spec a ?binary?, and as previously discussed on this list,
I?m trying to do it ?from outside? ? i.e., by calling it with Kernel#`
and observing the (side-)effects.

Unfortunately, this doesn?t really let me spec expectations about its
internals. Let?s assume I have a -d flag and I parse it with Trollop to
set Conf.debug:

opts = Trollop::options do
  opt :debug, ''Print debug info'', :default => false
end
Conf.debug = opts[:debug]

To have a spec that actually expects
this, I?d write something like this:

it ''should set the debug option properly'' do
  `binary -d`
  Conf.should_receive(:debug=).with true
end

This obviously doesn?t work, as the Kernel#` call is executed in
a context that is not visible from the current Ruby interpreter.
I assume that eval()-ing the binary (much like Rick Bradley does?
for specing flog) would be the way to go, but then I don?t know how
to pass the -d flag to Trollop.

? http://rubyhoedown2008.confreaks.com/11-rick-bradley-flog-test-new.html

What would be the best practice in this case? Prepare ARGV for Trollop
so that it believes we are calling the binary with the -d flag and then
eval the binary, or run the binary with Kernel#`, pass it something
that should generate debug output and then check whether the output
was generated (rather than check whether the binary set Conf.debug)?

-- Shot
-- 
I''ve found that nurturing one''s Zen nature is vital to dealing
with
technology. Violence is pretty damn useful too.   -- Lionel Lauer, asr
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 191 bytes
Desc: not available
URL:
<http://rubyforge.org/pipermail/rspec-users/attachments/20081107/9240fcbc/attachment.bin>

"Shot (Piotr Szotkowski)" <shot at hot.pl> writes:
> I?m trying to spec a ?binary?, and as previously discussed on this list,
> I?m trying to do it ?from outside? ? i.e., by calling it with Kernel#`
> and observing the (side-)effects.
>
> Unfortunately, this doesn?t really let me spec expectations about its
> internals. Let?s assume I have a -d flag and I parse it with Trollop to
> set Conf.debug:
>
> opts = Trollop::options do
>   opt :debug, ''Print debug info'', :default => false
> end
> Conf.debug = opts[:debug]
>
> To have a spec that actually expects
> this, I?d write something like this:
>
> it ''should set the debug option properly'' do
>   `binary -d`
>   Conf.should_receive(:debug=).with true
> end
>
> This obviously doesn?t work, as the Kernel#` call is executed in
> a context that is not visible from the current Ruby interpreter.
> I assume that eval()-ing the binary (much like Rick Bradley does?
> for specing flog) would be the way to go, but then I don?t know how
> to pass the -d flag to Trollop.
>
> ? http://rubyhoedown2008.confreaks.com/11-rick-bradley-flog-test-new.html
>
> What would be the best practice in this case? Prepare ARGV for Trollop
> so that it believes we are calling the binary with the -d flag and then
> eval the binary, or run the binary with Kernel#`, pass it something
> that should generate debug output and then check whether the output
> was generated (rather than check whether the binary set Conf.debug)?
>
> -- Shot
When you use backticks to execute the binary, it runs in a separate
process, so you can''t use mock expectations like that.

I think you''re kind of missing the intention though...what I think you
want to do is write features with cucumber that use the built binary
from the outside.  But when you write code-level examples, you are not
going to run the binary.  You''re just writing examples at a lower
level,
directly for the objects.  Make sense?

Pat

Pat Maddox:
> "Shot (Piotr Szotkowski)" <shot at hot.pl> writes:
>> it ''should set the debug option properly'' do
>>   `binary -d`
>>   Conf.should_receive(:debug=).with true
>> end
> When you use backticks to execute the binary, it runs in
> a separate process, so you can''t use mock expectations like that.
Right, that?s why I suggested I could Kernel#eval the binary?s contents
in the current process instead. This would require tricking Trollop, but
I assume I could trick it by hand-crafting ARGV.
> I think you''re kind of missing the intention though...
Yeah, I thought so ? that?s why I?m asking. :) The previous question
ended up with the conclusion that RSpecing a ?binary? through Kernel#`
is a sane way to go:

? http://groups.google.com/group/rspec/browse_thread/thread/66dc48522014c5eb
> what I think you want to do is write features with
> cucumber that use the built binary from the outside.
Ah, Cucumber. :) I?ve yet to write my first story; I somehow assumed
these are more oriented toward non-programmers and/or for driving
implementation on a rather higher level, and have to be ?implemented?
on the RSpec level somehow anyway. I take that these assumptions are
flawed. :)
> But when you write code-level examples, you are not going to run the
> binary. You''re just writing examples at a lower level, directly
for
> the objects. Make sense?
Hm, I guess I simply need to read up on Cucumber and stories;
I still can?t see how this kind of specification would differ
(in the end) from Kernel#`-based RSpec ? i.e., why implementing
the code behind these stories would side-step the underlying issue.
(But the last time I read on stories was pre-Cucumber.)

Is there an example of story/Cucumber-based specification
of a binary?s behaviour available somewhere out there?

-- Shot
-- 
init.d is complete in the sense that brainfuck
is Turing-complete.     -- SanityInAnarchy, /.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 191 bytes
Desc: not available
URL:
<http://rubyforge.org/pipermail/rspec-users/attachments/20081107/191a707f/attachment.bin>

On Nov 07, 2008, at 5:24 pm, Shot (Piotr Szotkowski) wrote:
> Right, that?s why I suggested I could Kernel#eval the binary?s  
> contents
> in the current process instead. This would require tricking Trollop,  
> but
> I assume I could trick it by hand-crafting ARGV.
You might find Rick Bradley''s talk[1] on regression testing flog  
useful here.  He starts by speccing the binary at a Ruby level.  Note,  
however, that later in the talk he admits he made a mistake in not  
writing end-to-end tests soon enough.  I think he should have run the  
binary as a black box and just pushed binary code into lib sooner.
> Ah, Cucumber. :) I?ve yet to write my first story; I somehow assumed
> these are more oriented toward non-programmers and/or for driving
> implementation on a rather higher level, and have to be ?implemented?
> on the RSpec level somehow anyway. I take that these assumptions are
> flawed. :)
Yes and no.  I wouldn''t say they are oriented towards the end user -  
Cucumber files are code and must be formatted strictly.  But they are  
readable enough that they can be understood by the end user.  You  
don''t actually need to use RSpec in Cucumber steps - just raising an  
exception when you see something you don''t like is enough - but the  
matchers make the steps readable.

I recommending stopping the line until you have your current  
functionality described in Cucumber feature definitions, it will save  
you a lot of pain in the long run.

> Hm, I guess I simply need to read up on Cucumber and stories;
> I still can?t see how this kind of specification would differ
> (in the end) from Kernel#`-based RSpec ? i.e., why implementing
> the code behind these stories would side-step the underlying issue.
> (But the last time I read on stories was pre-Cucumber.)
Cucumber with Kernel#` and RSpec with Kernel#` would achieve the same  
thing, the difference being the structure of the files.  Cucumber is  
optimised to deal with higher-lever concepts, where natural language  
is a more important part of the spec.  RSpec is more suited to lower- 
level specs where the aim is to cover every edge case or object  
interaction.  This would be too cumbersome with Cucumber.

> Is there an example of story/Cucumber-based specification
> of a binary?s behaviour available somewhere out there?
There is now :)

Well, it''s a really crappy, trivial example I just made it about 10  
mins.  No use of STDERR, no file input/output, no network  
connections.  Flagrant abuse of FIT tables.  Oh, and wanton sharing of  
data between steps.  But apart from that it''s model code.

Don''t take any of this as best practice!  It''s just to show
the
concept (seeing how "sort" is presumably written in C):

# sort.feature

   Feature: sort
     So that I can search faster
     As a person that likes searching
     I want to sort data

     Scenario: data ordered already
       Given the following data:
         | STDIN  |
         | apple  |
         | banana |
         | cherry |
         | mango  |
         | pear   |
       When I sort the data
       Then the output should be:
         | STDOUT |
         | apple  |
         | banana |
         | cherry |
         | mango  |
         | pear   |

     Scenario: data out of order
       Given the following data:
         | STDIN  |
         | pear   |
         | mango  |
         | apple  |
         | cherry |
         | banana |
       When I sort the data
       Then the output should be:
         | STDOUT |
         | apple  |
         | banana |
         | cherry |
         | mango  |
         | pear   |


# steps/sort.rb

   require ''spec''

   Given %r/^the following data:$/ do |table|
     @stdin = table.hashes.map { |r| r["STDIN"] }.join("\n")
   end

   When %r/^I sort the data$/ do
     @result = `sort <<END\n#@stdin\nEND\n`.split("\n")
   end

   Then %r/^the output should be:$/ do |table|
     @expected = table.hashes.map { |r| r["STDOUT"] }
     @result.should == @expected
   end

Hope this helps.

Ashley

[1] http://rubyhoedown2008.confreaks.com/11-rick-bradley-flog-test-new.html


-- 
http://www.patchspace.co.uk/
http://aviewfromafar.net/

Ashley Moran:
> On Nov 07, 2008, at 5:24 pm, Shot (Piotr Szotkowski) wrote:
>> Right, that?s why I suggested I could Kernel#eval the binary?s contents
>> in the current process instead. This would require tricking Trollop,
but
>> I assume I could trick it by hand-crafting ARGV.
> You might find Rick Bradley''s talk[1] on regression testing flog
> useful here. He starts by speccing the binary at a Ruby level.
Heh ? I brought exactly this talk up twice when talking about specing
binaries here (it was what pointed me in the direction that it would
be actually good to spec/test them, actually). :)
> Note, however, that later in the talk he admits he made a mistake in
> not writing end-to-end tests soon enough. I think he should have run
> the binary as a black box and just pushed binary code into lib sooner.
Right. I think I agree that?s the best
(and most elegant) way to test binaries.

In my case, asfter some specing and some coding, it turns out I can?t
really do this the way I wanted; most of the cases I need to test for
are triggered with input that makes the binary run a long time (it?s
the code behind my PhD thesis, so tons of computations). As I?m testing
a black box with Kernel#`, I can?t mock/stub anything out.

Seeing as my binary grew to over 80 lines, I guess my best bet is to
pull out the gist of it into a separate Runner class, spec it (with
mocks/stubs/fixtures) so it runs in a sane time and then only create
a think binary wrapper around it (which will basically parse the
options, set the config and then run the runner).

Thanks a lot for the nice Cucumber example! It was most informative.

-- Shot
-- 
As the Slashdot distributed brain engine has observed, the ultimate test of
any piece of equipment is to build a Beowulf cluster of it.   -- Peter Seebach
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 191 bytes
Desc: not available
URL:
<http://rubyforge.org/pipermail/rspec-users/attachments/20081114/93ccc33a/attachment.bin>

On 14 Nov 2008, at 20:56, Shot (Piotr Szotkowski) wrote:
> Seeing as my binary grew to over 80 lines, I guess my best bet is to
> pull out the gist of it into a separate Runner class, spec it (with
> mocks/stubs/fixtures) so it runs in a sane time and then only create
> a think binary wrapper around it (which will basically parse the
> options, set the config and then run the runner).
Have a look at the source code for cucumber''s binary, or
rspec''s
''spec'' command. They''re both about four lines long
creating a
(eminently spec-able) class that does the rest.

cheers,
Matt

On Nov 14, 2008, at 8:56 pm, Shot (Piotr Szotkowski) wrote:
> Heh ? I brought exactly this talk up twice when talking about specing
> binaries here (it was what pointed me in the direction that it would
> be actually good to spec/test them, actually). :)

Ha, so you did :)

There was a delay after I read your email to when I finally replied,  
in that time it clearly left my head...

> In my case, asfter some specing and some coding, it turns out I can?t
> really do this the way I wanted; most of the cases I need to test for
> are triggered with input that makes the binary run a long time (it?s
> the code behind my PhD thesis, so tons of computations). As I?m  
> testing
> a black box with Kernel#`, I can?t mock/stub anything out.
Well, like Matt says, all the binary needs to do is get your code up  
and running.  For that, I imagine you can use selectively-chosen  
inputs that don''t produce long run times.  The cucumber files just  
need to prove that all the command line options work, and that the app  
is directing IO to and from the right places.  The details of the  
algorithms can be dealt with in your spec suite.

Ashley


-- 
http://www.patchspace.co.uk/
http://aviewfromafar.net/