R devel - May 2003 - internet.R test hangs if http proxy needed (PR#3108)

Full_Name: Barry Rowlingson
Version: 1.7.0
OS: RH8/Mandrake 9.1
Submission from: (NULL) (194.80.32.8)


internet.R in the tests directory hangs during 'make check' if an http
proxy is
needed for http access from the machine. This happens in the httpget function
defined therein.

It tries to make a direct socket connection to port 80 of a remote machine, and
if local network policy does not permit that, then the test can hang. 

A fix? Perhaps skip that test if Sys.getenv("http_proxy") is defined.
Or rewrite
httpget to use the proxy if defined.

Barry

This is intended to be a test of the socket functions.  It should fail
within the time-out period, not hang.

On Mon, 26 May 2003 b.rowlingson@lancaster.ac.uk wrote:
> Full_Name: Barry Rowlingson
> Version: 1.7.0
> OS: RH8/Mandrake 9.1
> Submission from: (NULL) (194.80.32.8)
> 
> 
> internet.R in the tests directory hangs during 'make check' if an
http proxy is
> needed for http access from the machine. This happens in the httpget
function
> defined therein.
> 
> It tries to make a direct socket connection to port 80 of a remote machine,
and
> if local network policy does not permit that, then the test can hang. 
Why does it hang?  Isn't that the bug that needs fixing?  Do the other
tests fail gracefully if no proxy is set?  (They all do the couple of
places I have tried them in very restricted setups.)
> A fix? Perhaps skip that test if Sys.getenv("http_proxy") is
defined. Or rewrite
> httpget to use the proxy if defined.
There seem to be two possibilities:

a) make.socket is failing and hanging or

b) the local setup is listening on the port and not sending replies, or 
replies that cannot be parsed.

We need help, as to reproduce this needs some particular local setup and 
there have been no reports from anyone during the beta-test period for 
1.7.0, nor from previous versions.  (That makes me suspect it is rare 
local setup.)  Can you please debug this further.

-- 
Brian D. Ripley,                  ripley@stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595

Prof Brian Ripley wrote:
> We need help, as to reproduce this needs some particular local setup and 
> there have been no reports from anyone during the beta-test period for 
> 1.7.0, nor from previous versions.  (That makes me suspect it is rare 
> local setup.)  Can you please debug this further.
> 
It appears to be partly a bug in my impatience:
> 
system.time(httpget("http://www.stats.ox.ac.uk/pub/datasets/csb/ch11b.dat"))
Error in make.socket(host, port = port) : Socket not established
Timing stopped at: 0 0 189 0 0

  The function times out after 189 seconds, whereas my impatience
timeout is 3 minutes. I will upgrade my impatience to the latest version
so that it is compatible with everyone elses.

  At first I thought it was taking 3 * options()$timeout to produce the
timeout in make.socket. A quick investigation shows that to be purely 
coincidental with the initial value of options()$timeout:
> options(timeout=2)
> 
system.time(httpget("http://www.stats.ox.ac.uk/pub/datasets/csb/ch11b.dat"))
making socket
Error in make.socket(host, port = port) : Socket not established
Timing stopped at: 0 0 189.04 0 0

  The 189 second timeout appears to be part of our particular local
setup (port 80 is blocked off-site).

options()$timeout seems to have no effect on connections that fail due
to  non-use of our proxies. Is this a bug? The doc claims:

timeout: integer.  The timeout for Internet operations, in seconds.
           Default 60 seconds.

which is a bit vague. If I do something like:
  read.table("http://www.stats.ox.ac.uk/pub/datasets/csb/ch11b.dat")
without setting my proxy I get 189 seconds, regardless of
options()$timeout yet it is undoubtedly an internet operation.

  Back to the original problem: the httpget function in tests/internet.R 
can be modified to obey an http_proxy setting with a fairly simple 
adjustment near the start:

   hp <- Sys.getenv("http_proxy")
   if(hp != ""){
## we need to extract host and port from the proxy:
## split on / or : thusly:
## [http]:[]/[]/[hostname]:[port]
##   1     2  3      4        5
     hpbits <- strsplit(hp,"[/:]")[[1]]
     host <- hpbits[4]
     port <- hpbits[5]
## ask the proxy for the full URL with all its http:// glory:
     rurl <- url
   }else{
## no proxy, connect direct and ask for the URL relative to root:
     host <- urlel[3]
     rurl <- paste(c("", urlel[-(1:3)]), collapse = "/")
   }


  I still have some qualms about tests that rely on things external to 
the test site, but without including a socket server program (which 
would need its own set of tests, I suppose) I cant see any way of 
testing socket functionality without dribbling on the internet at large.

Barry