Mechanize users - Mar 2008 - Mechanize#get vs Mechanize#fetch_page

So I found myself wanting to call Mechanize#get with a hash for  
arguments like this:

WWW::Mechanize.new(''http://api.flickr.com/services/rest/'',
{:method =>
''flickr.auth.getFrob''... })

Granted, it looks like this isn''t supported but it led me to what  
looks like a bug.  Namely that get calls fetch_page(abs_uri, request,  
cur_page, &block) [mechanize.rb:167].  But the method signature for  
fetch_page [mechanize.rb:441] is def fetch_page(uri, request,  
cur_page=current_page(), request_data=[]).

What was the intention for the &block on Mechanize#get?  I can''t
find
any examples of it getting used in any of the test cases.  It''d be  
cool if I could make use of that parameter as request_data since I  
think it would be exactly the function I''m looking for.

Thanks again,
Mat

On Mon, Mar 10, 2008 at 09:07:14PM -0400, Mat Schaffer
wrote:
> So I found myself wanting to call Mechanize#get with a hash for  
> arguments like this:
> 
>
WWW::Mechanize.new(''http://api.flickr.com/services/rest/'',
{:method =>
> ''flickr.auth.getFrob''... })
> 
> Granted, it looks like this isn''t supported but it led me to what
> looks like a bug.  Namely that get calls fetch_page(abs_uri, request,  
> cur_page, &block) [mechanize.rb:167].  But the method signature for  
> fetch_page [mechanize.rb:441] is def fetch_page(uri, request,  
> cur_page=current_page(), request_data=[]).
You''re right.  That does look like a bug.  Also, this is another syntax
I like.  Would you be able to put up with:

  agent.get(''http://blah.com/'', nil, { :one =>
''two'' ... })
> 
> What was the intention for the &block on Mechanize#get?  I
can''t find
> any examples of it getting used in any of the test cases.  It''d be
> cool if I could make use of that parameter as request_data since I  
> think it would be exactly the function I''m looking for.
The problem is that you need to stop reading my code!  ;-)  IIRC my
intent was to yield the page to the block.  However there must have been
a disconnect between my brain and the keyboard.  I''ll fix it.

-- 
Aaron Patterson
http://tenderlovemaking.com/

On Mar 11, 2008, at 10:03 AM, Aaron Patterson wrote:
> You''re right.  That does look like a bug.  Also, this is another  
> syntax
> I like.  Would you be able to put up with:
>
>  agent.get(''http://blah.com/'', nil, { :one =>
''two'' ... })
This would be okay, but I''d be curious about moving the referrer  
argument to the end so you don''t need the nil.  The nil kinda throws  
off the DSL quality of it all, I think.

Sadly, I tried to just patch around it like I thought would work and  
it didn''t.  (patch attached) Basically when I hit mechanize.rb:513, it
told me that I was giving http_obj.request too many arguments.  And I  
couldn''t figure out what http_obj was.  Maybe some sort of delegate?   
I didn''t wanna wade through your connection caching code to figure it  
out.

I''ve been making due with this in the mean time:

agent.get(url + "?" + WWW::Mechanize.build_query_string(args))

Which might help you build that get syntax that I''m aiming for.   
Anyway, I''m living with it, so don''t rush into a sub-optimal
solution
on my account.
>> What was the intention for the &block on Mechanize#get?  I
can''t find
>> any examples of it getting used in any of the test cases. 
It''d be
>> cool if I could make use of that parameter as request_data since I
>> think it would be exactly the function I''m looking for.
>
> The problem is that you need to stop reading my code!  ;-)  IIRC my
> intent was to yield the page to the block.  However there must have  
> been
> a disconnect between my brain and the keyboard.  I''ll fix it.
PEBKAC!!!! :)

With any luck, I''ll have a flickr api library built around mechanize  
when this is all done.  Which could be fun :)

Thanks,
Mat

-------------- next part --------------
A non-text attachment was scrubbed...
Name: get_params.patch
Type: application/octet-stream
Size: 739 bytes
Desc: not available
Url :
http://rubyforge.org/pipermail/mechanize-users/attachments/20080311/acad567e/attachment.obj
-------------- next part --------------

On Tue, Mar 11, 2008 at 12:42:59PM -0400, Mat Schaffer
wrote:
>
> On Mar 11, 2008, at 10:03 AM, Aaron Patterson wrote:
>> You''re right.  That does look like a bug.  Also, this is
another syntax
>> I like.  Would you be able to put up with:
>>
>>  agent.get(''http://blah.com/'', nil, { :one =>
''two'' ... })
>
> This would be okay, but I''d be curious about moving the referrer
argument
> to the end so you don''t need the nil.  The nil kinda throws off
the DSL
> quality of it all, I think.
Ask an ye shall receive.  This now works:

  page = @agent.get(''http://localhost/'', { :q =>
''hello'' })

So does this:
  
  @agent.get(''http://localhost/'') do |page|
    ...
  end

As well as this:

  page = @agent.post(''http://localhost/file_upload'', {
    :name       => ''Some picture'',
    :userfile1  => File.open(__FILE__)
  })

Any other requests before I make a release?

-- 
Aaron Patterson
http://tenderlovemaking.com/

On Mar 11, 2008, at 5:17 PM, Aaron Patterson wrote:
> Ask an ye shall receive.  This now works:
>
>  page = @agent.get(''http://localhost/'', { :q =>
''hello'' })
>
> So does this:
>
>  @agent.get(''http://localhost/'') do |page|
>    ...
>  end
>
> As well as this:
>
>  page = @agent.post(''http://localhost/file_upload'', {
>    :name       => ''Some picture'',
>    :userfile1  => File.open(__FILE__)
>  })
>
> Any other requests before I make a release?
You frickin rule!  Seriously dude, you''re awesome.

Actually, not a request exactly, but I did notice an odd behavior now  
that you mention http://localhost.  It looks like mechanize is set up  
to have some tricky behavior when referencing localhost.  Namely, it  
seems to be sidestepping the webserver.  Just as an example, see the  
attached script and output that gets pretty different results for  
localhost and 127.0.0.1.  It certainly bit me for about 5 minutes  
before I noticed what was going on.  I haven''t really debugged it so  
I''m not sure if it''s mechanize or maybe something in
Net::HTTP.  I''ll
look a bit and let you know if I find anything.

This is on mechanize-0.7.2.
-Mat


-------------- next part --------------
A non-text attachment was scrubbed...
Name: tester.rb
Type: text/x-ruby-script
Size: 245 bytes
Desc: not available
Url :
http://rubyforge.org/pipermail/mechanize-users/attachments/20080311/d70a509d/attachment.bin
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: output.txt
Url:
http://rubyforge.org/pipermail/mechanize-users/attachments/20080311/d70a509d/attachment.txt

On Mar 11, 2008, at 8:09 PM, Mat Schaffer wrote:
> You frickin rule!  Seriously dude, you''re awesome.
>
> Actually, not a request exactly, but I did notice an odd behavior  
> now that you mention http://localhost.  It looks like mechanize is  
> set up to have some tricky behavior when referencing localhost.   
> Namely, it seems to be sidestepping the webserver.  Just as an  
> example, see the attached script and output that gets pretty  
> different results for localhost and 127.0.0.1.  It certainly bit me  
> for about 5 minutes before I noticed what was going on.  I haven''t
> really debugged it so I''m not sure if it''s mechanize or
maybe
> something in Net::HTTP.  I''ll look a bit and let you know if I
find
> anything.
>
> This is on mechanize-0.7.2.
> -Mat
Yep, it''s Net::HTTP.  Man...  I have such a bittersweet relationship  
with that library.  But its interesting to note that jruby gets it  
right.  Probably because jruby''s implementation of Net::HTTP is just  
based on Java''s HttpURLConnection.

Anyway, unless feel like taking your babysitting of Net::HTTP to a  
whole new level.  Release away! :)
-Mat

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: net_http_output.txt
Url:
http://rubyforge.org/pipermail/mechanize-users/attachments/20080311/91e7e345/attachment-0002.txt
-------------- next part --------------
A non-text attachment was scrubbed...
Name: net_http_tester.rb
Type: text/x-ruby-script
Size: 219 bytes
Desc: not available
Url :
http://rubyforge.org/pipermail/mechanize-users/attachments/20080311/91e7e345/attachment-0001.bin
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: net_http_output.jruby.txt
Url:
http://rubyforge.org/pipermail/mechanize-users/attachments/20080311/91e7e345/attachment-0003.txt

On Tue, Mar 11, 2008 at 08:09:09PM -0400, Mat Schaffer
wrote:
> On Mar 11, 2008, at 5:17 PM, Aaron Patterson wrote:
>> Ask an ye shall receive.  This now works:
>>
>>  page = @agent.get(''http://localhost/'', { :q =>
''hello'' })
>>
>> So does this:
>>
>>  @agent.get(''http://localhost/'') do |page|
>>    ...
>>  end
>>
>> As well as this:
>>
>>  page = @agent.post(''http://localhost/file_upload'', {
>>    :name       => ''Some picture'',
>>    :userfile1  => File.open(__FILE__)
>>  })
>>
>> Any other requests before I make a release?
>
> You frickin rule!  Seriously dude, you''re awesome.
Thanks.  I''ve got to keep my users happy.  :-)

Its released now.

-- 
Aaron Patterson
http://tenderlovemaking.com/

On Tue, Mar 11, 2008 at 08:29:27PM -0400, Mat Schaffer
wrote:
> On Mar 11, 2008, at 8:09 PM, Mat Schaffer wrote:
>> You frickin rule!  Seriously dude, you''re awesome.
>>
>> Actually, not a request exactly, but I did notice an odd behavior now
that
>> you mention http://localhost.  It looks like mechanize is set up to
have
>> some tricky behavior when referencing localhost.  Namely, it seems to
be
>> sidestepping the webserver.  Just as an example, see the attached
script
>> and output that gets pretty different results for localhost and
127.0.0.1.
>>  It certainly bit me for about 5 minutes before I noticed what was
going
>> on.  I haven''t really debugged it so I''m not sure if
it''s mechanize or
>> maybe something in Net::HTTP.  I''ll look a bit and let you
know if I find
>> anything.
>>
>> This is on mechanize-0.7.2.
>> -Mat
>
> Yep, it''s Net::HTTP.  Man...  I have such a bittersweet
relationship with
> that library.  But its interesting to note that jruby gets it right.  
> Probably because jruby''s implementation of Net::HTTP is just based
on
> Java''s HttpURLConnection.
I wonder how EventMachine would work with this....  I was thinking about
switching mechanize from net/http to EventMachine, but EventMachine
doesn''t have the features I need.  I sent in a patch, but
didn''t really
get any responses:

 http://rubyforge.org/pipermail/eventmachine-talk/2008-March/001754.html

I think I''ll try again because EventMachine is insanely fast compared
to
net/http.

-- 
Aaron Patterson
http://tenderlovemaking.com/

On Mar 12, 2008, at 2:22 PM, Aaron Patterson wrote:
> I wonder how EventMachine would work with this....  I was thinking  
> about
> switching mechanize from net/http to EventMachine, but EventMachine
> doesn''t have the features I need.  I sent in a patch, but
didn''t
> really
> get any responses:
>
> http://rubyforge.org/pipermail/eventmachine-talk/2008-March/ 
> 001754.html
>
> I think I''ll try again because EventMachine is insanely fast  
> compared to
> net/http.
Huh... I wasn''t even aware that EventMachine was good for anything  
other than servers.  Might have to take a second look at that.

Thanks!
Mat