Icecast - May 2021 - falling back

I?m afraid this will be a little long, but hope some can put in the time to read
and help.

As I mentioned, I need to transcode and want Icecast to start and stop according
to listeners. So this is what I?m trying:-

	<relay>
		<server>127.0.0.1</server>
		<port>8127</port>
		<mount>/BBCR2.aac</mount>
		<local-mount>/BBCR2</local-mount>
	</relay>
	<mount type="normal">
		<mount-name>/BBCR2</mount-name>
		<public>0</public>
		<on-connect>/usr/local/bin/transtart</on-connect>
		<on-disconnect>/usr/local/bin/transtop</on-disconnect>
	</mount>
	
	<mount type="normal">
		<mount-name>/BBCR2.aac</mount-name>
		<public>0</public>
		<fallback-mount>/silence.aac</fallback-mount>
    	<fallback-override>1</fallback-override>
	</mount>

So the first 2 blocks are the relay mount that ?pulls? from the mount configured
in the 3rd block. Yes, this does work, Icecast CAN relay from itself. The reason
why I do this is to take advantage of the on-(dis)connect scripts and the fact
that an Icecast relay only ?pulls? when there are listeners for that mountpoint.

Testing with a player (ffplay) to connect to the BBCR2 (relay) mount, it plays
silence and this is what Icecast says:-

[2021-05-21  17:20:53] INFO slave/start_relay_stream Starting relayed source at
mountpoint "/BBCR2"
[2021-05-21  17:20:53] INFO slave/open_relay_connection connecting to
127.0.0.1:8127
[2021-05-21  17:20:53] INFO source/source_main listener count on /silence.aac
now 1
[2021-05-21  17:20:53] WARN format/format_get_type Unsupported or legacy stream
type: "audio/aac". Falling back to generic minimal handler for best
effort.
[2021-05-21  17:20:54] INFO source/source_main listener count on /BBCR2 now 1

Ignoring the legacy WARN, when the listener connects to BBCR2, it tries to relay
from BBCR2.aac which has no source and so plays the silence.aac (ADTS format)
file. All correct so far, except the on-connect script (that starts transcoding
and streaming to BBCR2.aac) is not run. More on that later.

However, about 35 seconds after this first starts (i.e. listener connects to
BBCR2), Icecast logs this:-

[2021-05-21  17:21:28] INFO source/send_to_listener Client 377 (xxx.xxx.xxx.xxx)
has fallen too far behind, removing
[2021-05-21  17:21:28] INFO source/source_main listener count on /BBCR2 now 0
[2021-05-21  17:21:28] INFO source/source_shutdown Source from 127.0.0.1 at
"/BBCR2" exiting
[2021-05-21  17:21:28] INFO source/source_main listener count on /silence.aac
now 0

So Icecast kills everything, but the listener (ffplay) doesn?t know this and
continues running, obviously still silent so I cannot tell what it actually
thinks it is playing. Why does Icecast say the Client (ffplay on my Mac at
xxx.xxx.xxx.xxx) is too far behind? Surely the fallback file should simply loop
indefinitely until the real source comes on line - which it doesn?t as the
script is not run.

Now I try the same thing, but manually running the script that will start the
transcoder and send the stream to Icecast?s BBCR2.aac mount:-

[2021-05-21  17:23:48] INFO slave/start_relay_stream Starting relayed source at
mountpoint "/BBCR2"
[2021-05-21  17:23:48] INFO slave/open_relay_connection connecting to
127.0.0.1:8127
[2021-05-21  17:23:48] INFO source/source_main listener count on /silence.aac
now 1
[2021-05-21  17:23:48] WARN format/format_get_type Unsupported or legacy stream
type: "audio/aac". Falling back to generic minimal handler for best
effort.
[2021-05-21  17:23:49] INFO source/source_main listener count on /BBCR2 now 1
[2021-05-21  17:24:04] INFO connection/_handle_source_request Source logging in
at mountpoint "/BBCR2.aac" from 127.0.0.1
[2021-05-21  17:24:04] WARN format/format_get_type Unsupported or legacy stream
type: "audio/aac". Falling back to generic minimal handler for best
effort.
[2021-05-21  17:24:04] INFO source/source_move_clients passing 1 listeners to
"/BBCR2.aac"
[2021-05-21  17:24:04] INFO source/source_main listener count on /BBCR2.aac now
1
[2021-05-21  17:24:04] INFO source/source_main listener count on /silence.aac
now 0

So initially the same as the first run, but then it sees the source for
BBCR2.aac and switches the listener from the file to that mount - as it should.
But I still don?t hear the actual radio stream because same as last time:-

[2021-05-21  17:24:23] INFO source/send_to_listener Client 380 (xxx.xxx.xxx.xxx)
has fallen too far behind, removing
[2021-05-21  17:24:23] INFO source/source_main listener count on /BBCR2 now 0
[2021-05-21  17:24:23] INFO source/source_shutdown Source from 127.0.0.1 at
"/BBCR2" exiting
[2021-05-21  17:24:23] INFO source/source_main listener count on /BBCR2.aac now
0

Leaving me to manually stop ffplay and run the transcode stop script.

So several questions:-

Why does Icecast complain about the client being ?too far behind? and how can I
prevent this? Surely the fallback should simply operate as long as necessary? Is
this the ?looping too fast? problem that is swamping the client? How is this
fallback file feature supposed to work?

Why when switching the listener back from the fallback file to the actual
streaming source do I still not hear that? Is this perhaps just because of the
previous issue?

Why are the scripts not run? It seems to me that when the relay receives
anything, whether from the relayed mount (BBCR2.aac in this case) or its
fallback file, that event is the source connecting and should trigger the
on-connect script. But it does not. The script writes a log and I can see it is
simply not being run at all (unless I run it manually).

As I said, sorry it?s a bit of a long one, but hope some more experienced in
Icecast than me can shed some light on this.



Ken  G i l l e t t

_/_/_/_/_/_/_/_/



-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.xiph.org/pipermail/icecast/attachments/20210521/d29c933e/attachment.htm>

V?Fri, May 21, 2021 at 06:28:42PM +0100,?Ken Gillett
napsal(a):
> As I mentioned, I need to transcode and want Icecast to start and stop
> according to listeners. So this is what I?m trying:-
> 
> 	<relay>
> 		<server>127.0.0.1</server>
> 		<port>8127</port>
> 		<mount>/BBCR2.aac</mount>
> 		<local-mount>/BBCR2</local-mount>
> 	</relay>
> 	<mount type="normal">
> 		<mount-name>/BBCR2</mount-name>
> 		<public>0</public>
> 		<on-connect>/usr/local/bin/transtart</on-connect>
> 		<on-disconnect>/usr/local/bin/transtop</on-disconnect>
> 	</mount>
> 	
> 	<mount type="normal">
> 		<mount-name>/BBCR2.aac</mount-name>
> 		<public>0</public>
> 		<fallback-mount>/silence.aac</fallback-mount>
>     	<fallback-override>1</fallback-override>
> 	</mount>
> 
> So the first 2 blocks are the relay mount that ?pulls? from the mount
> configured in the 3rd block. Yes, this does work, Icecast CAN relay from
> itself. The reason why I do this is to take advantage of the
on-(dis)connect
> scripts and the fact that an Icecast relay only ?pulls? when there are
> listeners for that mountpoint.
> 
In my opinion you cannot achieve what you want with Icecast because it goes
against its design.

Icecast has two types of mount points: source-mounts which content is pushed
into Icecast and relay-mounts which content is pulled by Icacast.
A transmission of the content which is pulled can only be initiated by Icecast.
On the other hand, the pushed content cannot be initiated by Icecast. The
pushed content is always initated by an external stream source (transcode in
your case).

Basically, Icecast is a streaming proxy of which task is to multiply an external
source to multiple clients at the same time. The external source is either
pushed or pulled content. Streaming local files is only a trivial tool for
providing a fallback when the external source is unavailable.

What you need is an transcoder which waits on a connection from an ICY client
and then it opens a tunner and starts transcoding. And similarly it will close
the tunner after the client disconnects. The transcoder does not need and
should not be able to handle multiple clients. It should be a trivial program
for one tunner and one client. After having this setup, you can place Icecast
between the transcoder and the (thousands of) clients.

I cannot recommend you any ready-made transcoder because I have no experience
in this area. But I believe there exist plenty of transcoders which can work
as a source-client for Icecast and thus it should be easy to enhance them to
implement a one-client ICY server (the pull model from Icecast point of view)
in addition.

(Technically it could be possible to introduce a third type of a mount type
to Icecast: It would execute an external command for the first client,
streamed its standard output and terminated the command with the last client.
But handling execution of possibly blocking commands adds more complexity and
from security point of view its not wise to give Icecast server a permission
to access various hardware (e.g. tunners) directly.)
> However, about 35 seconds after this first starts (i.e. listener connects
to
> BBCR2), Icecast logs this:-
> 
> [2021-05-21  17:21:28] INFO source/send_to_listener Client 377
(xxx.xxx.xxx.xxx) has fallen too far behind, removing
> [2021-05-21  17:21:28] INFO source/source_main listener count on /BBCR2 now
0
> [2021-05-21  17:21:28] INFO source/source_shutdown Source from 127.0.0.1 at
"/BBCR2" exiting
> [2021-05-21  17:21:28] INFO source/source_main listener count on
/silence.aac now 0
> 
> So Icecast kills everything, but the listener (ffplay) doesn?t know this
and
> continues running, obviously still silent so I cannot tell what it actually
> thinks it is playing.  Why does Icecast say the Client (ffplay on my Mac at
> xxx.xxx.xxx.xxx) is too far behind?
I think this message means that the client does not consume the stream. So an
output buffer on the server has been filling up and after 35 seconds reached
a limit and thus the server disconnected the client.

I recommend you either to replace the silent stream with something audiable,
or better use a network analyzer tool (e.g. tcpdump) to see whether the stream
flows from the server to the client or not.

I suspect that either the client misunderstands the stream, or the server
misunderstands the /silence.aac file.
> Why are the scripts not run? It seems to me that when the relay receives
> anything, whether from the relayed mount (BBCR2.aac in this case) or its
> fallback file, that event is the source connecting and should trigger the
> on-connect script. But it does not. The script writes a log and I can see
it
> is simply not being run at all (unless I run it manually).
> 
Does something change if you remove the second block (mount at /BBCR2)?

I think it does not call the scripts because you defined /BBCR2 twice. Maybe
Icecast parser does not expect having them twice, ignores the second
block and uses only the first one.

-- Petr

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL:
<http://lists.xiph.org/pipermail/icecast/attachments/20210521/0256a959/attachment.sig>

Thanks for the comments Petr and Marvin.

I know Icecast was not specifically designed for this, but I am sure that I am
not asking it to do anything (apart from triggering scripts) that it cannot do.
I understand about the different mounts, but each is independent of the other
and as I have found, Icecast does not care that it is relaying from itself. That
works. If I start the transcode process to BBCR2.aac first, I can then
connect/listen to the BBCR2 mount. This works perfectly, so Icecast has no
problem with this setup. The reason for multiple mounts like this is because a
<mount type="normal?> can only trigger the on-connect script when the
source connects, but the script IS the source, so a chicken and egg problem
there. However, I figured a relay type mount might trigger when it starts
pulling from the relayed server as that is as close to the source connecting as
a relay can get. More on that later.

As Petr points out, BBCR2.aac is only required to stream to a single client,
whereas Icecast is designed to stream to multiple clients. However how can this
be a problem. Icecast cares not how many listeners, as long as it is below the
maximum and in the case of a relay, only to determine when to connect to and
disconnect from the relayed server mount. Having just a single listener does not
mean Icecast will malfunction in any way. What it possibly means though is more
resources being used than is strictly necessary. However this is actually
irrelevant since Icecast is already running anyway, so an additional mount for a
single listener has to consume less resources than a completely independent
process just for the purpose of streaming to a single client. I?m certainly not
against that however, but trying to set up an on-demand transcode process that
streamed to Icecast turned out to be far harder (I?ve failed so far) than simply
interposing an additional mount in Icecast.

The BBCR2 mount is not specified twice as I understand it. A <relay> mount
block specifically configures the relay and nothing else. If you need any other
configuration (like authorisation etc.) you HAVE to set up a <mount
type="normal?> block of the same name to contain all its other required
parameters, as that stuff cannot be held within the <relay> block. So
unless anyone can explain how the docs are incorrect, I am sure that this is
correct.

The problem I have with this Icecast setup is as I said, triggering the
transcode script and the streaming fallback file.

The latter seems likely to be as suggested and my next test will involve a file
of some sound(s) that will not compress so much and hopefully not fill up the
buffers. However it is interesting that I have read many mentions of the use of
a file of silence specifically for this exact task, even though it clearly has
problems. Anyway, I can try an alternative file and see how that goes. Something
is required however as without the fallback file configured, any attempt to
listen to the relay is immediately rejected (with a 404 I seem to recall).

That leaves the on-(dis)connect issue. This is where my plan might completely
fail. I am hoping that a relay mount can trigger these, however I have seen no
direct evidence that it will. The standard explanation is that on-connect is
triggered when the source connects, but maybe that only works when a source is
pushed to a normal Icecast mountpoint and not when a relay sees the mount from
which it is supposed to relay and so streams from it. In my case, the stream
will actually be the fallback file, but it should still count as a source. The
problem that occurs to me is that the relay mount code may simply not recognise
being able to pull from the relayed mount (internal or external) as a source
connect event and hence will never trigger the script.

This last has always been in my mind, but no-one has yet been able to confirm
how Icecast actually functions in this respect. Hence why I?ve been testing, but
the absence of success so far does not confirm absolutely that it cannot work
and may only indicate I have something else wrong, although I am coming to the
realisation that a relay mount will not work as I require.

In which case, back to the idea of the transcode process as an http streamer.
This however has to be very low ?footprint? as multiple streams would mean
multiple http servers, one for each transcoded stream. Better to have a single
server that can handle multiple listeners and http streams. And we?re back to
Icecast that is already running?

If anyone can actually suggest a solution to this conundrum, please let me know
as I?m all ears.


Ken  G i l l e t t

_/_/_/_/_/_/_/_/


> On 21 May 2021, at 20:50, Petr Pisar <petr.pisar at atlas.cz> wrote:
> 
> V Fri, May 21, 2021 at 06:28:42PM +0100, Ken Gillett napsal(a):
>> As I mentioned, I need to transcode and want Icecast to start and stop
>> according to listeners. So this is what I?m trying:-
>> 
>> 	<relay>
>> 		<server>127.0.0.1</server>
>> 		<port>8127</port>
>> 		<mount>/BBCR2.aac</mount>
>> 		<local-mount>/BBCR2</local-mount>
>> 	</relay>
>> 	<mount type="normal">
>> 		<mount-name>/BBCR2</mount-name>
>> 		<public>0</public>
>> 		<on-connect>/usr/local/bin/transtart</on-connect>
>> 		<on-disconnect>/usr/local/bin/transtop</on-disconnect>
>> 	</mount>
>> 	
>> 	<mount type="normal">
>> 		<mount-name>/BBCR2.aac</mount-name>
>> 		<public>0</public>
>> 		<fallback-mount>/silence.aac</fallback-mount>
>>    	<fallback-override>1</fallback-override>
>> 	</mount>
>> 
>> So the first 2 blocks are the relay mount that ?pulls? from the mount
>> configured in the 3rd block. Yes, this does work, Icecast CAN relay
from
>> itself. The reason why I do this is to take advantage of the
on-(dis)connect
>> scripts and the fact that an Icecast relay only ?pulls? when there are
>> listeners for that mountpoint.
>> 
> In my opinion you cannot achieve what you want with Icecast because it goes
> against its design.
> 
> Icecast has two types of mount points: source-mounts which content is
pushed
> into Icecast and relay-mounts which content is pulled by Icacast.
> A transmission of the content which is pulled can only be initiated by
Icecast.
> On the other hand, the pushed content cannot be initiated by Icecast. The
> pushed content is always initated by an external stream source (transcode
in
> your case).
> 
> Basically, Icecast is a streaming proxy of which task is to multiply an
external
> source to multiple clients at the same time. The external source is either
> pushed or pulled content. Streaming local files is only a trivial tool for
> providing a fallback when the external source is unavailable.
> 
> What you need is an transcoder which waits on a connection from an ICY
client
> and then it opens a tunner and starts transcoding. And similarly it will
close
> the tunner after the client disconnects. The transcoder does not need and
> should not be able to handle multiple clients. It should be a trivial
program
> for one tunner and one client. After having this setup, you can place
Icecast
> between the transcoder and the (thousands of) clients.
> 
> I cannot recommend you any ready-made transcoder because I have no
experience
> in this area. But I believe there exist plenty of transcoders which can
work
> as a source-client for Icecast and thus it should be easy to enhance them
to
> implement a one-client ICY server (the pull model from Icecast point of
view)
> in addition.
> 
> (Technically it could be possible to introduce a third type of a mount type
> to Icecast: It would execute an external command for the first client,
> streamed its standard output and terminated the command with the last
client.
> But handling execution of possibly blocking commands adds more complexity
and
> from security point of view its not wise to give Icecast server a
permission
> to access various hardware (e.g. tunners) directly.)
> 
>> However, about 35 seconds after this first starts (i.e. listener
connects to
>> BBCR2), Icecast logs this:-
>> 
>> [2021-05-21  17:21:28] INFO source/send_to_listener Client 377
(xxx.xxx.xxx.xxx) has fallen too far behind, removing
>> [2021-05-21  17:21:28] INFO source/source_main listener count on /BBCR2
now 0
>> [2021-05-21  17:21:28] INFO source/source_shutdown Source from
127.0.0.1 at "/BBCR2" exiting
>> [2021-05-21  17:21:28] INFO source/source_main listener count on
/silence.aac now 0
>> 
>> So Icecast kills everything, but the listener (ffplay) doesn?t know
this and
>> continues running, obviously still silent so I cannot tell what it
actually
>> thinks it is playing.  Why does Icecast say the Client (ffplay on my
Mac at
>> xxx.xxx.xxx.xxx) is too far behind?
> 
> I think this message means that the client does not consume the stream. So
an
> output buffer on the server has been filling up and after 35 seconds
reached
> a limit and thus the server disconnected the client.
> 
> I recommend you either to replace the silent stream with something
audiable,
> or better use a network analyzer tool (e.g. tcpdump) to see whether the
stream
> flows from the server to the client or not.
> 
> I suspect that either the client misunderstands the stream, or the server
> misunderstands the /silence.aac file.
> 
>> Why are the scripts not run? It seems to me that when the relay
receives
>> anything, whether from the relayed mount (BBCR2.aac in this case) or
its
>> fallback file, that event is the source connecting and should trigger
the
>> on-connect script. But it does not. The script writes a log and I can
see it
>> is simply not being run at all (unless I run it manually).
>> 
> Does something change if you remove the second block (mount at /BBCR2)?
> 
> I think it does not call the scripts because you defined /BBCR2 twice.
Maybe
> Icecast parser does not expect having them twice, ignores the second
> block and uses only the first one.
> 
> -- Petr
> 
> _______________________________________________
> Icecast mailing list
> Icecast at xiph.org
> http://lists.xiph.org/mailman/listinfo/icecast

Good morning,

first of all, thank you Petr Pisar for you mail. It gives a good
overview but I feel the need to go into details a bit more.

On Fri, 2021-05-21 at 21:50 +0200, Petr Pisar wrote:
> Icecast has two types of mount points: source-mounts which content is
> pushed into Icecast and relay-mounts which content is pulled by
> Icacast.
This is actually a old myth. And I honestly don't know where it
originated from.

Icecast only knows one kind of mount points. Each mount can be in
mounted state (a source connected) or unmounted state (no source
connected).

Once a client is connects Icecast first checks if the mount is part of
the Admin API. If so it will handle the request via the Admin API, if
not it will continue:
Next it is checked if the mount point is mounted[0]. If it is, the
client is connected to that source.
If it is unmounted Icecast will try to auto connect a source. There are
currently the following source types Icecast can auto connect:

 * Relays
 * Admin API (This is everything under /admin; And only used for Admin
   API)
 * fserve (this is "plain files" and "XSLT rendered stats")

So what does that mean:
If you configure a relay Icecast will start a new source client. But
that is build into Icecast. That source client then forwards the data
it reads from another source[1].

If nothing specific has been configured, Icecast will next try using
static files with fserve. So Icecast will start a fserve source that
will then send the file if there is one.

If nothing could mount the mount point Icecast will send a error
("404") indicating that there is no source for the mount.

This also means:
You can apply all mount settings to all mount points. However those not
applicable will be ignored by Icecast. (Also the Admin API ignores most
settings as special rules apply for it. Mostly noticeable with auth/ACL
settings.)

> A transmission of the content which is pulled can only be initiated
> by Icecast. On the other hand, the pushed content cannot be initiated
> by Icecast. The pushed content is always initated by an external
> stream source (transcode in your case).
Exactly:
 * External sources for externally generated events (e.g.: "Someone
   pressed the \"on air\" button")
 * Internal source for internally generated events (e.g.: "Some client
   connected").

> Basically, Icecast is a streaming proxy of which task is to multipy
> an external source to multiple clients at the same time. The external
> source is either pushed or pulled content.
> Streaming local files is only a trivial tool for providing a fallback
> when the external source is unavailable.
Actually streaming of local files is only there to provide the web
interface. That you can use it for media is just because there is no
difference for Icecast. To Icecast it's all sending data from a source
to a sink.

However to provide what you describe there is one tiny bit of extra
code (it's really just a few lines!) that will loop local files when
used as fallback. But this is just meant for a really last resort.



Also based on what we learned above:
If you want a source to be started only when there are listeners: Use a
relay. The relay source will notify an external component of the demand
(by requesting content). That is the way to go.

To make this work with an external data source just write a tiny CGI
script or similar that sends the necessary headers and start a
encoder/transcoder that sends data to stdout (which is what is send
back to the client for CGI).

That can for example be done with just a few lines of shell. Something
like:
   #!/bin/sh
   printf "some: headers\r\n\r\n";
   exec myencoder blabla -o -


Hope this mail was helpful everyone and people learned a bit more about
Icecast. :)


[0] Two little exceptions: a) everything under /admin is protected
against being mounted by non-API; b) connections that try to mount a
mount point are handled a bit differently.
[1] Actually it splices the connection. Which gives way better
performance.

-- 
Philipp Schafft (CEO/Gesch?ftsf?hrer) 
Telephon: +49.3535 490 17 92

L?wenfelsen UG (haftungsbeschr?nkt)     Registration number:
Bickinger Stra?e 21                     HRB 12308 CB
04916 Herzberg (Elster)                 VATIN/USt-ID:
Germany                                 DE305133015
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 523 bytes
Desc: This is a digitally signed message part
URL:
<http://lists.xiph.org/pipermail/icecast/attachments/20210522/a5574203/attachment.sig>