Mechanize users - Jan 2010 - EOFError content-length != page length

I know it seems like EOFError should be raised if content-length !page length
but it would be nice to turn this off just like we have
quirks mode for html soup.

I have an issue where I have to target a broken web server that I don''t
control.

I had to resort to using mechanize in another language that didn''t
have the check in place rather than editing ruby-mechanize and then
having users download my patched version.

It would be very nice if there was an option to removed the need for
the length received to be equal to the content length header
message...

On Tue, Jan 12, 2010 at 7:53 PM, Daniel Aquino <mr.danielaquino at
gmail.com>wrote:
> I know it seems like EOFError should be raised if content-length !> page
length but it would be nice to turn this off just like we have
> quirks mode for html soup.
>
> I have an issue where I have to target a broken web server that I
don''t
> control.
>
>From the FAQ:
Q: I keep getting an EOFError:
  protocol.rb:133:in `sysread'': end of file reached (EOFError)

A:  Some people have experienced an EOFError during normal mechanize usage.
    Most of the time this occurs because the remote website claims to
support
    keep alives, but does not implement them correctly.  Try turning off
    keep alives on your mechanize object:

      mech.keep_alive = false

Please let us know if this works for you.

>
> I had to resort to using mechanize in another language that didn''t
> have the check in place rather than editing ruby-mechanize and then
> having users download my patched version.
>
> It would be very nice if there was an option to removed the need for
> the length received to be equal to the content length header
> message...
> _______________________________________________
> Mechanize-users mailing list
> Mechanize-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mechanize-users
>


-- 
mike dalessio
mike at csa.net
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://rubyforge.org/pipermail/mechanize-users/attachments/20100112/7c3238f6/attachment.html>

it doesn''t I knew about that issue from searching and it''s on
the faq.

My problem is that the page appears truncated but in fact </body> is
reached

Perhaps attempting to parse the page first and then complaining if
there is an error  would be better?

On Tue, Jan 12, 2010 at 11:21 PM, Mike Dalessio <mike at csa.net>
wrote:
>
>
> On Tue, Jan 12, 2010 at 7:53 PM, Daniel Aquino <mr.danielaquino at
gmail.com>
> wrote:
>>
>> I know it seems like EOFError should be raised if content-length
!>> page length but it would be nice to turn this off just like we have
>> quirks mode for html soup.
>>
>> I have an issue where I have to target a broken web server that I
don''t
>> control.
>
> From the FAQ:
>
> Q: I keep getting an EOFError:
> ? protocol.rb:133:in `sysread'': end of file reached (EOFError)
>
> A:? Some people have experienced an EOFError during normal mechanize usage.
> ??? Most of the time this occurs because the remote website claims to
> support
> ??? keep alives, but does not implement them correctly.? Try turning off
> ??? keep alives on your mechanize object:
>
> ????? mech.keep_alive = false
>
> Please let us know if this works for you.
>
>>
>> I had to resort to using mechanize in another language that
didn''t
>> have the check in place rather than editing ruby-mechanize and then
>> having users download my patched version.
>>
>> It would be very nice if there was an option to removed the need for
>> the length received to be equal to the content length header
>> message...
>> _______________________________________________
>> Mechanize-users mailing list
>> Mechanize-users at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/mechanize-users
>
>
>
> --
> mike dalessio
> mike at csa.net
>
>
> _______________________________________________
> Mechanize-users mailing list
> Mechanize-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mechanize-users
>

Any plan to make this a feature?  I can wip togetther a patch for it

On 1/12/10, Daniel Aquino <mr.danielaquino at gmail.com>
wrote:
> it doesn''t I knew about that issue from searching and
it''s on the faq.
>
> My problem is that the page appears truncated but in fact </body> is
> reached
>
> Perhaps attempting to parse the page first and then complaining if
> there is an error  would be better?
>
> On Tue, Jan 12, 2010 at 11:21 PM, Mike Dalessio <mike at csa.net>
wrote:
>>
>>
>> On Tue, Jan 12, 2010 at 7:53 PM, Daniel Aquino
>> <mr.danielaquino at gmail.com>
>> wrote:
>>>
>>> I know it seems like EOFError should be raised if content-length
!>>> page length but it would be nice to turn this off just like we
have
>>> quirks mode for html soup.
>>>
>>> I have an issue where I have to target a broken web server that I
don''t
>>> control.
>>
>> From the FAQ:
>>
>> Q: I keep getting an EOFError:
>> ? protocol.rb:133:in `sysread'': end of file reached (EOFError)
>>
>> A:? Some people have experienced an EOFError during normal mechanize
>> usage.
>> ??? Most of the time this occurs because the remote website claims to
>> support
>> ??? keep alives, but does not implement them correctly.? Try turning
off
>> ??? keep alives on your mechanize object:
>>
>> ????? mech.keep_alive = false
>>
>> Please let us know if this works for you.
>>
>>>
>>> I had to resort to using mechanize in another language that
didn''t
>>> have the check in place rather than editing ruby-mechanize and then
>>> having users download my patched version.
>>>
>>> It would be very nice if there was an option to removed the need
for
>>> the length received to be equal to the content length header
>>> message...
>>> _______________________________________________
>>> Mechanize-users mailing list
>>> Mechanize-users at rubyforge.org
>>> http://rubyforge.org/mailman/listinfo/mechanize-users
>>
>>
>>
>> --
>> mike dalessio
>> mike at csa.net
>>
>>
>> _______________________________________________
>> Mechanize-users mailing list
>> Mechanize-users at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/mechanize-users
>>
>
-- 
Sent from my mobile device

On Mon, Jan 18, 2010 at 11:26 AM, Daniel Aquino
<mr.danielaquino at gmail.com> wrote:
> Any plan to make this a feature? ?I can wip togetther a patch for it
Go for it.  If you include tests, I will apply it.  :-D

-- 
Aaron Patterson
http://tenderlovemaking.com/

what does the comment mean here ?

Does this mean there is a way around this by specifying the content
length while making the request ?

          # Net::HTTP ignores EOFError if Content-length is given, so
we emulate it here.
          unless res_klass <= Net::HTTPRedirection
            raise EOFError if
(!params[:request].is_a?(Net::HTTP::Head)) &&
@response.content_length() && @response.content_length() != total
          end




On Mon, Jan 18, 2010 at 2:38 PM, Aaron Patterson
<aaron.patterson at gmail.com> wrote:
> On Mon, Jan 18, 2010 at 11:26 AM, Daniel Aquino
> <mr.danielaquino at gmail.com> wrote:
>> Any plan to make this a feature? ?I can wip togetther a patch for it
>
> Go for it. ?If you include tests, I will apply it. ?:-D
>
> --
> Aaron Patterson
> http://tenderlovemaking.com/
> _______________________________________________
> Mechanize-users mailing list
> Mechanize-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mechanize-users
>

my problem here is that @response.content_length() = 1142 but total = 1109....

again as i said this is a broken http server that I must deal with so
it would be nice to have a way around this type of misconfiguration so
the lib gracefully deals with the situation... I clearly see the
entire page being loaded including the final </body></html> so if
you
ask me mechanize should be good enough to handle this if it expects to
deal with the web !

On Tue, Jan 19, 2010 at 10:50 AM, Daniel Aquino
<mr.danielaquino at gmail.com> wrote:
> what does the comment mean here ?
>
> Does this mean there is a way around this by specifying the content
> length while making the request ?
>
> ? ? ? ? ?# Net::HTTP ignores EOFError if Content-length is given, so
> we emulate it here.
> ? ? ? ? ?unless res_klass <= Net::HTTPRedirection
> ? ? ? ? ? ?raise EOFError if
> (!params[:request].is_a?(Net::HTTP::Head)) &&
> @response.content_length() && @response.content_length() != total
> ? ? ? ? ?end
>
>
>
>
> On Mon, Jan 18, 2010 at 2:38 PM, Aaron Patterson
> <aaron.patterson at gmail.com> wrote:
>> On Mon, Jan 18, 2010 at 11:26 AM, Daniel Aquino
>> <mr.danielaquino at gmail.com> wrote:
>>> Any plan to make this a feature? ?I can wip togetther a patch for
it
>>
>> Go for it. ?If you include tests, I will apply it. ?:-D
>>
>> --
>> Aaron Patterson
>> http://tenderlovemaking.com/
>> _______________________________________________
>> Mechanize-users mailing list
>> Mechanize-users at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/mechanize-users
>>
>

On Tue, Jan 19, 2010 at 7:56 AM, Daniel Aquino
<mr.danielaquino at gmail.com> wrote:
> my problem here is that @response.content_length() = 1142 but total =
1109....
>
> again as i said this is a broken http server that I must deal with so
> it would be nice to have a way around this type of misconfiguration so
> the lib gracefully deals with the situation... I clearly see the
> entire page being loaded including the final </body></html> so
if you
> ask me mechanize should be good enough to handle this if it expects to
> deal with the web !
Cool story!

So do you have a patch or tests?

-- 
Aaron Patterson
http://tenderlovemaking.com/

no cause i was wondering about those comments in the code if there was
a way to override it already...

I''m not sure how you would want to add this to the interface...

perhaps there could be a quirks flag or ignore_content_length flag ?

On Tue, Jan 19, 2010 at 12:09 PM, Aaron Patterson
<aaron.patterson at gmail.com> wrote:
> On Tue, Jan 19, 2010 at 7:56 AM, Daniel Aquino
> <mr.danielaquino at gmail.com> wrote:
>> my problem here is that @response.content_length() = 1142 but total =
1109....
>>
>> again as i said this is a broken http server that I must deal with so
>> it would be nice to have a way around this type of misconfiguration so
>> the lib gracefully deals with the situation... I clearly see the
>> entire page being loaded including the final </body></html>
so if you
>> ask me mechanize should be good enough to handle this if it expects to
>> deal with the web !
>
> Cool story!
>
> So do you have a patch or tests?
>
> --
> Aaron Patterson
> http://tenderlovemaking.com/
> _______________________________________________
> Mechanize-users mailing list
> Mechanize-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mechanize-users
>

any input ?

On Tue, Jan 19, 2010 at 12:25 PM, Daniel Aquino
<mr.danielaquino at gmail.com> wrote:
> no cause i was wondering about those comments in the code if there was
> a way to override it already...
>
> I''m not sure how you would want to add this to the interface...
>
> perhaps there could be a quirks flag or ignore_content_length flag ?
>
> On Tue, Jan 19, 2010 at 12:09 PM, Aaron Patterson
> <aaron.patterson at gmail.com> wrote:
>> On Tue, Jan 19, 2010 at 7:56 AM, Daniel Aquino
>> <mr.danielaquino at gmail.com> wrote:
>>> my problem here is that @response.content_length() = 1142 but total
= 1109....
>>>
>>> again as i said this is a broken http server that I must deal with
so
>>> it would be nice to have a way around this type of misconfiguration
so
>>> the lib gracefully deals with the situation... I clearly see the
>>> entire page being loaded including the final
</body></html> so if you
>>> ask me mechanize should be good enough to handle this if it expects
to
>>> deal with the web !
>>
>> Cool story!
>>
>> So do you have a patch or tests?
>>
>> --
>> Aaron Patterson
>> http://tenderlovemaking.com/
>> _______________________________________________
>> Mechanize-users mailing list
>> Mechanize-users at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/mechanize-users
>>
>

does nobody else consider this to be an issue ?

On Thu, Jan 21, 2010 at 12:57 PM, Daniel Aquino
<mr.danielaquino at gmail.com> wrote:
> any input ?
>
> On Tue, Jan 19, 2010 at 12:25 PM, Daniel Aquino
> <mr.danielaquino at gmail.com> wrote:
>> no cause i was wondering about those comments in the code if there was
>> a way to override it already...
>>
>> I''m not sure how you would want to add this to the
interface...
>>
>> perhaps there could be a quirks flag or ignore_content_length flag ?
>>
>> On Tue, Jan 19, 2010 at 12:09 PM, Aaron Patterson
>> <aaron.patterson at gmail.com> wrote:
>>> On Tue, Jan 19, 2010 at 7:56 AM, Daniel Aquino
>>> <mr.danielaquino at gmail.com> wrote:
>>>> my problem here is that @response.content_length() = 1142 but
total = 1109....
>>>>
>>>> again as i said this is a broken http server that I must deal
with so
>>>> it would be nice to have a way around this type of
misconfiguration so
>>>> the lib gracefully deals with the situation... I clearly see
the
>>>> entire page being loaded including the final
</body></html> so if you
>>>> ask me mechanize should be good enough to handle this if it
expects to
>>>> deal with the web !
>>>
>>> Cool story!
>>>
>>> So do you have a patch or tests?
>>>
>>> --
>>> Aaron Patterson
>>> http://tenderlovemaking.com/
>>> _______________________________________________
>>> Mechanize-users mailing list
>>> Mechanize-users at rubyforge.org
>>> http://rubyforge.org/mailman/listinfo/mechanize-users
>>>
>>
>