Icecast - Jul 2017 - Source Drops, metadata, and source limits in setup

Good morning,


On Fri, 2017-07-14 at 03:38 +0000, ScanCaster wrote:
> An issue I've been trying to track down for quite a while is
> 
> Source Drops... 
> 
> especially say 4 sources from the same IP, dropping all at once. 
> 
> Or ALL sources on the server just dropping ALL AT ONCE. Sources in this 
> case are from differing locations and differing ISP's.
> 
> Yes, internet has its spordaic issues, not the cause....
> 
> Why do I say that?
> 
> Well after a hunch I took my icecast.xml and just upped the sources and 
> client counts to some HUGE RIDICULOUS NUMBER.
> 
> And POOF! That seems to have made the issue disappear!Although I am 
> wondering if this is going to cause a core dump or something if these are 
> hogging RAM etc...
You can easily check the RAM used by Icecast2 using e.g. top(1). Most
relevant is 'RES' which is the amount of in-core memory. However there
is no memory related problem known to me on any of the stable official
Icecast2 releases.

> What I am wondering is if the cause of this is metadata updates! Why? For 
> my case metadata updates can come in fast and furious every 5-10-20 
> seconds or so... versus say 3-10 minutes for something spitting out music 
> or something... I am not doing music..
This posting confirms me in that I should write a separate e-mail to the
list on how metadata updates work and how they should be implemented on
the source side.

> I am wondering if there is something where the metadata updates are 
> CLOGGING UP the source connections and thus once full, even though the 
> clients have been connected for days, they now drop.
No. Manual updates using the admin web interface is exactly that: admin
interface requests. It's just normal clients.

You can confirm this by having a look at the accumulative counters for
sources and clients. The client counter will increase for every client
while the source counter only for source connections.

> I use BASH scripts with wget to pass the metadata update which gets data 
> from my another data stream:
> 
> wget -q --delete-after --http-user=admin --http-password=Top_SECRET 
> http://audio.domain.invalid:8000/admin/metadata?mount=/MyMount
> \&mode=updinfo\&song=[This+is+who+is+talking]+$RANDOM >/dev/null
Why don't you use -O /dev/null? Also make sure you both quote the song
parameter first to HTTP standards and then to shell standards.

I think the redirect of stdout is mostly useless.
This also will allow each and every process on the same system to spot
the password.

> Is there something where these updates, don't release the connection
for
> a while, and a SETTING TO CHANGE THAT??? source-timeout??? 
No.

The source timeout is only to allow some jitter within the TCP flow
(e.g. to compensate lost packets). The only reason to increase it is a
bad network. And if you're on a bad network you should very likely
rethink your setup.

> Sporadic issues with routers, data traversal on the public internet I 
> get.. When FOUR SOURCES DROP ALL AT THE SAME TIME on an IP where things 
> are running fine for other things...and changing this setting, seems to 
> cure it. Colors me curious... 
Which *exact* version of Icecast2 do you run and where have you got it
from? Which operating system are you using?


Icecast2 will write a message to the error.log in case there is some
problem with a client. What is the relevant section of the error.log?
Also have a look at the access.log. It will list all the clients after
they have disconnected. This includes the source connections as well as
admin interface interaction.

Without input from the log files it's hard to tell. Here are some of my
guesses:
      * Your network has some kind of a problem. Maybe a bad cable right
        on the server? (Would explain why source fail at the same time
        and why altering timeouts may help.)
      * You're running a non-official version of Icecast2. (could
        explain everything or nothing.)
      * One of your scripts do random things. E.g. because of faulty
        escaping. (Could explain random drops.)
      * Your admin and/or source credentials are known to someone else.
        (Could explain all of those things.)
      * You're not running one of the stable and supported versions.
        There could be an attack based on a security problem of an old
        version going on. (Could explain all of those things.)
      * Your machine has some kind of hardware malfunction, e.g. bad
        RAM. (Could explain all of those things.)
      * A still unknown bug in Icecast2. (Could explain all of those
        things.)
      * A mixture of what is above.

The logs will easily tell if there is a timeout related problem, that
always means the network has some problem. It will also tell if there
are related admin interface requests. Both of this considering one of
the stable and supported versions of Icecast2.

With best regards,

-- 
Philipp.
 (Rah of PH2)