Martiño Perez
2019-Aug-01 08:45 UTC
[Icecast] mount points down in relay after a network problem
Hello.
I have a problem in a master-relay scenario.
Sometimes, when a stream is down in master server and after a while the
stream is fine again (e.g. after a network problem), relay/s server can't
get the stream again, so I have to restart icecast in relay server to make
it work:
1- An incoming stream is stopped in icecast master
2 - The stream is up again in icecast master
3 - Relay server has the mount point up but is not possible to play
(timeout error when playing)
4 - When source-timeout is reached in icecast relay, relay server
does source_shutdown in each failed mount points. Then when I request a
mount point it gives a 404 error and I have to restart the service to make
it work
Relay server has the following config:
<master-update-interval>15</master-update-interval>
So each 15 seconds the relay server goes to check the stream in the master
server
Master Server has the following config:
<source-timeout>180</source-timeout>
So it has to be 3 minutes without stream to delete the mount point.
Why the relay server does not respond after the network failure and before
the source-timeout?
Why the relay server can't find the mount points after the source-timeout?
Are there any way to force the relay server to check the mount points in
master server to avoid a restart?
This is the master config:
<icecast>
<limits>
<clients>5000</clients>
<sources>1000</sources>
<threadpool>4</threadpool>
<queue-size>524288</queue-size>
<client-timeout>60</client-timeout>
<header-timeout>30</header-timeout>
<source-timeout>20</source-timeout>
<burst-on-connect>0</burst-on-connect>
<burst-size>0</burst-size>
</limits>
<authentication>
<source-password>1234</source-password>
<relay-password>1234</relay-password>
<admin-user>1234</admin-user>
<admin-password>1234</admin-password>
</authentication>
<hostname>icecast-master</hostname>
<listen-socket>
<port>8000</port>
</listen-socket>
<http-headers>
<header name="Access-Control-Allow-Origin"
value="*" />
</http-headers>
<relay>
<server>127.0.0.1</server>
<port>8000</port>
<mount>/test1</mount>
<local-mount>/test1.mp3</local-mount>
</relay>
<relay>
<server>127.0.0.1</server>
<port>8000</port>
<mount>/test2</mount>
<local-mount>/test2.mp3</local-mount>
</relay>
<paths>
<basedir>/usr/share/icecast</basedir>
<logdir>/var/log/icecast</logdir>
<webroot>/usr/share/icecast/web</webroot>
<adminroot>/usr/share/icecast/admin</adminroot>
</paths>
<logging>
<accesslog>access.log</accesslog>
<errorlog>error.log</errorlog>
<loglevel>3</loglevel> <!-- 4 Debug, 3 Info, 2 Warn, 1
Error -->
<logsize>200000</logsize> <!-- Max size of a logfile
-->
</logging>
<security>
<chroot>0</chroot>
<changeowner>
<user>icecast</user>
<group>icecast</group>
</changeowner>
</security>
</icecast>
##################################
And this is relay config:
<icecast>
<limits>
<clients>2310</clients>
<sources>10000</sources>
<threadpool>16</threadpool>
<queue-size>524288</queue-size>
<client-timeout>30</client-timeout>
<header-timeout>15</header-timeout>
<source-timeout>600</source-timeout>
<burst-on-connect>1</burst-on-connect>
<burst-size>65536</burst-size>
</limits>
<authentication>
<source-password>1234</source-password>
<relay-password>1234</relay-password>
<admin-user>1234</admin-user>
<admin-password>1234</admin-password>
</authentication>
<hostname>192.168.1.5</hostname>
<listen-socket>
<port>80</port>
<bind-address>192.168.1.5</bind-address>
</listen-socket>
<listen-socket>
<port>443</port>
<bind-address>192.168.1.5</bind-address>
<ssl>1</ssl>
</listen-socket>
<http-headers>
<header name="Access-Control-Allow-Origin"
value="*" />
</http-headers>
<master-server>192.168.1.4</master-server>
<master-server-port>8000</master-server-port>
<master-update-interval>15</master-update-interval>
<master-password>1234</master-password>
<mount>
<mount-name>/test1.mp3</mount-name>
<burst-on-connect>0</burst-on-connect>
<burst-size>0</burst-size>
</mount>
<mount>
<mount-name>/test2.mp3</mount-name>
<burst-on-connect>0</burst-on-connect>
<burst-size>0</burst-size>
</mount>
<fileserve>1</fileserve>
<paths>
<basedir>/usr/share/icecast</basedir>
<logdir>/var/log/icecast</logdir>
<webroot>/usr/share/icecast/web</webroot>
<adminroot>/usr/share/icecast/admin</adminroot>
<ssl-certificate>/etc/pki/tls/certs/certificate_chained.crt</ssl-certificate>
</paths>
<logging>
<accesslog>access.log</accesslog>
<errorlog>error.log</errorlog>
<loglevel>3</loglevel> <!-- 4 Debug, 3 Info, 2 Warn, 1
Error -->
<logsize>200000</logsize> <!-- Max size of a logfile
-->
</logging>
<security>
<chroot>0</chroot>
<changeowner>
<user>icecast</user>
<group>icecast</group>
</changeowner>
</security>
</icecast>
Relay's error log file:
[2019-07-30 02:11:15] WARN slave/open_relay_connection Failed to connect
to 192.168.1.4:8000
[2019-07-30 02:11:15] WARN slave/open_relay_connection Failed to connect
to 192.168.1.4:8000
[2019-07-30 02:13:32] WARN slave/open_relay_connection Failed to connect
to 192.168.1.4:8000
[2019-07-30 02:13:56] WARN slave/open_relay_connection Failed to connect
to 192.168.1.4:8000
...
(After the network problem, the stream is up in master server but relay
server gives a timeout. Despite this, the number of listeners in log file
continue growing)
[2019-07-30 02:20:32] INFO source/source_main listener count on /test1.mp3
now 787
[2019-07-30 02:20:34] INFO source/source_main listener count on /test1.mp3
now 788
[2019-07-30 02:20:34] INFO source/source_main listener count on /test1.mp3
now 789
[2019-07-30 02:20:35] INFO source/source_main listener count on /test1.mp3
now 790
[2019-07-30 02:20:35] INFO source/source_main listener count on /test1.mp3
now 791
[2019-07-30 02:20:36] INFO source/source_main listener count on /test1.mp3
now 792
[2019-07-30 02:20:36] INFO source/source_main listener count on /test1.mp3
now 793
[2019-07-30 02:20:37] INFO source/source_main listener count on /test1.mp3
now 794
[2019-07-30 02:20:38] INFO source/source_main listener count on /test1.mp3
now 796
[2019-07-30 02:20:39] INFO source/source_main listener count on /test1.mp3
now 797
...
(the source-timeout is applied and down mount points are cleared. It starts
to give a 404 error)
[2019-07-30 02:20:39] INFO source/source_shutdown Source from 192.168.1.4
at "/test1.mp3" exiting
[2019-07-30 02:20:39] INFO source/source_clear_source 796 active listeners
on /test1.mp3 released
[2019-07-30 02:20:39] INFO fserve/fserve_client_create checking for file
/test1.mp3 (/usr/share/icecast/web/test1.mp3)
[2019-07-30 02:20:39] WARN fserve/fserve_client_create req for file
"/usr/share/icecast/web/test1.mp3" No such file or directory
[2019-07-30 02:20:39] INFO fserve/fserve_client_create checking for file
/test1.mp3 (/usr/share/icecast/web/test1.mp3)
[2019-07-30 02:20:39] WARN fserve/fserve_client_create req for file
"/usr/share/icecast/web/test1.mp3" No such file or directory
[2019-07-30 02:20:39] INFO fserve/fserve_client_create checking for file
/test1.mp3 (/usr/share/icecast/web/test1.mp3)
[2019-07-30 02:20:39] WARN fserve/fserve_client_create req for file
"/usr/share/icecast/web/test1.mp3" No such file or directory
[2019-07-30 02:20:39] INFO fserve/fserve_client_create checking for file
/test1.mp3 (/usr/share/icecast/web/test1.mp3)
[2019-07-30 02:20:39] WARN fserve/fserve_client_create req for file
"/usr/share/icecast/web/test1.mp3" No such file or directory
[2019-07-30 02:20:39] INFO fserve/fserve_client_create checking for file
/test1.mp3 (/usr/share/icecast/web/test1.mp3)
...
(The service is restarted)
[2019-07-30 02:45:26] INFO source/source_main listener count on /test1.mp3
now 0
[2019-07-30 02:45:39] INFO source/source_main listener count on /test1.mp3
now 1
[2019-07-30 02:45:54] INFO source/source_main listener count on /test1.mp3
now 2
[2019-07-30 02:46:18] INFO source/source_main listener count on /test1.mp3
now 3
[2019-07-30 02:46:19] INFO source/source_main listener count on /test1.mp3
now 4
...
Kind regards.
Martinho Pérez.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.xiph.org/pipermail/icecast/attachments/20190801/637d864d/attachment.html>
Possibly Parallel Threads
Wisdom of the Ancients
