Icecast dev - Oct 2018 - Should Icecast send the "Connection: close" header?

I'm accessing a server using Icecast 2.4.0. It is streamed using HTTP 1.1
using the Transfer-Encoding: chunked - so far so good. The issue is that the
HTTP library I'm using uses connection pooling by default and attempts to
re-use connections if the server does not send the "Connection: close"
header. As part of that re-use, when you end a request/response cycle, it tries
to read from the socket through the end of the response, which doesn't work
well when the stream is infinite. The idea is that it's setting things up so
that there isn't data left on the socket and things are ready for the next
transaction.

You can see the following conversation for more context:
https://github.com/JuliaWeb/HTTP.jl/issues/339

AFAICT the issue is that there's no way deterministically for the client to
know whether the stream its receiving is infinite, so it doesn't know
whether it's safe to try to read through the rest of the response to get the
stream state ready for the next transaction. I imagine that putting Icecast
behind a reverse proxy would have similar issues if they tried to reuse the
connection, so I'm not sure if there's a standard client-side approach
to handling this, or if it would be better for Icecast to send the
"Connection: close" header.

One client-side workaround I can think of is to disable connection re-use for
any chunked transfers, but that seems maybe overly-conservative.

Here's the transaction I'm seeing:

$ curl -v doppler.media.mit.edu/impoundment.ogg
*   Trying 18.85.59.166...
* TCP_NODELAY set
* Connected to doppler.media.mit.edu (18.85.59.166) port 80
(#0)
> GET /impoundment.ogg HTTP/1.1
> Host: doppler.media.mit.edu
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Mon, 29 Oct 2018 15:58:27 GMT
< Server: Icecast 2.4.0
< Content-Type: application/ogg
< Cache-Control: no-cache
< Expires: Sat, 26 Jul 1997 05:00:00 GMT
< Pragma: no-cache
< icy-name: no name
< icy-pub: 0
< Transfer-Encoding: chunked
< 
...

Good afternoon,

On Mon, 2018-10-29 at 13:48 -0400, Spencer Russell
wrote:
> I'm accessing a server using Icecast 2.4.0. It is streamed using HTTP
> 1.1 using the Transfer-Encoding: chunked - so far so good.
I feel like I need to clarify a few things here:
      * Icecast 2.4.0 does not implement HTTP/1.1 but HTTP/1.0, which it
        correctly tells in it's response status lines.
      * Icecast 2.4.0 does not support chunked tansfer encoding.
      * Icecast 2.5.x does, yet does not use it for stream delivery.
      * Icecast 2.4.0 is VERY OUTDATED and contains MULTIPLE SECURITY
        vulnerabilities. Please upgrade to the newest stable version.

> The issue is that the HTTP library I'm using uses connection pooling
> by default and attempts to re-use connections if the server does not
> send the "Connection: close" header. As part of that re-use, when
you
> end a request/response cycle, it tries to read from the socket through
> the end of the response, which doesn't work well when the stream is
> infinite. The idea is that it's setting things up so that there
isn't
> data left on the socket and things are ready for the next transaction.
This is correct for HTTP/1.1. Icecast 2.5.x implements this correctly
(as it does implement HTTP/1.1). For HTTP/1.0 the default type of
connections is 'close' (keep-alive must specifically requested).
> You can see the following conversation for more context:
> https://github.com/JuliaWeb/HTTP.jl/issues/339
> 
> AFAICT the issue is that there's no way deterministically for the
> client to know whether the stream its receiving is infinite, so it
> doesn't know whether it's safe to try to read through the rest of
the
> response to get the stream state ready for the next transaction. I
> imagine that putting Icecast behind a reverse proxy would have similar
> issues if they tried to reuse the connection, so I'm not sure if
> there's a standard client-side approach to handling this, or if it
> would be better for Icecast to send the "Connection: close"
header.
see above.
> One client-side workaround I can think of is to disable connection
> re-use for any chunked transfers, but that seems maybe
> overly-conservative.
> 
> Here's the transaction I'm seeing:
> 
> $ curl -v doppler.media.mit.edu/impoundment.ogg
> *   Trying 18.85.59.166...
> * TCP_NODELAY set
> * Connected to doppler.media.mit.edu (18.85.59.166) port 80 (#0)
> > GET /impoundment.ogg HTTP/1.1
> > Host: doppler.media.mit.edu
> > User-Agent: curl/7.58.0
> > Accept: */*
> > 
> < HTTP/1.1 200 OK
> < Date: Mon, 29 Oct 2018 15:58:27 GMT
> < Server: Icecast 2.4.0
> < Content-Type: application/ogg
> < Cache-Control: no-cache
> < Expires: Sat, 26 Jul 1997 05:00:00 GMT
> < Pragma: no-cache
> < icy-name: no name
> < icy-pub: 0
> < Transfer-Encoding: chunked
I'm not sure what is answering your request, however it is not Icecast
2.4.0. I suspect it's a faulty reverse proxy that is in front of the
server.

Maybe that you can have a look at that.
I hope my reply is of help to you!

With best regards,

-- 
Philipp Schafft (CEO/Geschäftsführer) 
Telephon: +49.3535 490 17 92

Löwenfelsen UG (haftungsbeschränkt)     Registration number:
Bickinger Straße 21                     HRB 12308 CB
04916 Herzberg (Elster)                 VATIN/USt-ID:
Germany                                 DE305133015
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: This is a digitally signed message part
URL:
<http://lists.xiph.org/pipermail/icecast-dev/attachments/20181029/af7fbe66/attachment.sig>

On Mon, Oct 29, 2018, at 4:42 PM, Philipp Schafft wrote:
> ...
> 
> I'm not sure what is answering your request, however it is not Icecast
> 2.4.0. I suspect it's a faulty reverse proxy that is in front of the
> server.
> 
> ...
Hi Philipp,

Thanks for the detailed response, it was indeed very helpful. I checked with the
guy who administrates that server and confirmed that it's running behind a
proxy that "upgrades" the connection to HTTP/1.1 but does not add the
"Connection: close" header.

Mystery solved.

-s