Icecast dev - Apr 2015 - Segmentation fault issue with new install

Hi all,
see my reply inline.
Cheers, Jan-Pieter


2015-04-26 7:23 GMT+02:00 "Thomas B. R?cker" <thomas at
ruecker.fi>:
> On 04/25/2015 07:32 PM, Jan-Pieter wrote:
> > Hi all,
> >
> > I am a long-time user of Icecast, always without problems. However i
> > made a new setup using icecast 2.4.1 on a fresh and quick CentOS VPS.
>
> EPEL package or from somewhere else?
>
> it's just a standard CentOS 6.6 release, no EPEL. Distri from hosting
provider. [root at h2429770 tmp]# uname -a
Linux h2429770.stratoserver.net 2.6.32-042stab103.6 #1 SMP Wed Jan 21
13:07:39 MSK 2015 x86_64 x86_64 x86_64 GNU/Linux

>
> > The issue is that starting icecast gives a segmentation fault at
> > around 80% of all starts. No core file nor a entry in error.log is
> posted.
>
> Enable core dumps using "ulimit -c unlimited". The PWD from where
you
> start Icecast needs to be writeable.
>
Even with ulimit -c unlimited and running from a writable PWD no core is
dumped.

>
>
> > In the remaining 20% of the starts icecast start with no problem.
> >
> > Contents of the error.log with a succesfull start (20% of icecast
> starts):
> > [2015-04-25  21:08:40] INFO main/main Icecast 2.4.1 server started
> > [2015-04-25  21:08:40] INFO connection/get_ssl_certificate No SSL
> > capability
> > [2015-04-25  21:08:40] INFO stats/_stats_thread stats thread started
> > [2015-04-25  21:08:40] INFO yp/yp_update_thread YP update thread
started
> > [2015-04-25  21:08:40] INFO source/source_fallback_file mountpoint
> > /geen-uitzending.mp3 is reserved
> > [2015-04-25  21:08:40] WARN format/format_get_type Unsupported or
> > legacy stream type: "audio/mpeg". Falling back to generic
minimal
> > handler for best effort.
> > [2015-04-25  21:08:40] WARN source/source_fallback_file mountpoint
> > "/geen-uitzending.mp3" already reserved
> > [2015-04-25  21:08:40] INFO source/source_main listener count on
> > /geen-uitzending.mp3 now 0
> >
> >
> > Contents of error.log with a segmentation fault direct after start
> > (80% of icecast starts):
> > 2015-04-25  21:09:05] INFO main/main Icecast 2.4.1 server started
> > [2015-04-25  21:09:05] INFO connection/get_ssl_certificate No SSL
> > capability
> > [2015-04-25  21:09:05] INFO yp/yp_update_thread YP update thread
started
> > [2015-04-25  21:09:05] INFO source/source_fallback_file mountpoint
> > /geen-uitzending.mp3 is reserved
> > [2015-04-25  21:09:05] WARN format/format_get_type Unsupported or
> > legacy stream type: "audio/mpeg". Falling back to generic
minimal
> > handler for best effort.
> > [2015-04-25  21:09:05] INFO stats/_stats_thread stats thread started
>
> Can you attach your icecast.xml please (with passwords removed)?
> This might help reproduce the problem.
>
>
I attached icecast.xml. I also turned debug errorlog level on.
Results with debug on:

good start:
[2015-04-28  15:56:39] INFO main/main Icecast 2.4.1 server started
[2015-04-28  15:56:39] DBUG yp/yp_recheck_config Updating YP configuration
[2015-04-28  15:56:39] INFO connection/get_ssl_certificate No SSL capability
[2015-04-28  15:56:39] INFO stats/_stats_thread stats thread started
[2015-04-28  15:56:39] INFO yp/yp_update_thread YP update thread started
[2015-04-28  15:56:39] INFO source/source_fallback_file mountpoint
/geen-uitzending.mp3 is reserved
[2015-04-28  15:56:39] DBUG connection/connection_complete_source sources
count is 0
[2015-04-28  15:56:39] WARN format/format_get_type Unsupported or legacy
stream type: "audio/mpeg". Falling back to generic minimal handler for
best
effort.
[2015-04-28  15:56:39] DBUG connection/connection_complete_source source is
ready to start
[2015-04-28  15:56:39] DBUG source/source_init Source creation complete
[2015-04-28  15:56:39] WARN source/source_fallback_file mountpoint
"/geen-uitzending.mp3" already reserved
[2015-04-28  15:56:39] INFO source/source_main listener count on
/geen-uitzending.mp3 now 0
[2015-04-28  15:56:39] DBUG stats/modify_node_event update global sources
(1)
[2015-04-28  15:56:39] DBUG stats/process_source_event new source stat
/geen-uitzending.mp3
[2015-04-28  15:56:39] DBUG stats/modify_node_event update global
source_client_connections (1)
[2015-04-28  15:56:39] DBUG stats/process_source_event new node listeners
(0)
[2015-04-28  15:56:39] DBUG stats/process_source_event new node listenurl (
http://stream3.server7.nl:80/geen-uitzending.mp3)
[2015-04-28  15:56:39] DBUG stats/modify_node_event update global
source_total_connections (1)
[2015-04-28  15:56:39] DBUG stats/process_source_event new node
slow_listeners (0)
[2015-04-28  15:56:39] DBUG stats/modify_node_event update
"/geen-uitzending.mp3" listeners (0)
[2015-04-28  15:56:39] DBUG stats/process_source_event new node
listener_peak (0)
[2015-04-28  15:56:39] DBUG stats/process_source_event new node
stream_start (Tue, 28 Apr 2015 15:56:39 +0200)
[2015-04-28  15:56:39] DBUG stats/process_source_event new node
stream_start_iso8601 (2015-04-28T15:56:39+0200)
[2015-04-28  15:56:39] DBUG stats/process_source_event new node
total_bytes_read (0)
[2015-04-28  15:56:39] DBUG stats/process_source_event new node
total_bytes_sent (0)
[2015-04-28  15:56:39] DBUG stats/modify_node_event update
"/geen-uitzending.mp3" listeners (0)
[2015-04-28  15:56:40] DBUG slave/_slave_thread checking master stream list

bad start (segfault):
[2015-04-28  15:56:53] INFO main/main Icecast 2.4.1 server started
[2015-04-28  15:56:53] DBUG yp/yp_recheck_config Updating YP configuration
[2015-04-28  15:56:53] INFO connection/get_ssl_certificate No SSL capability
[2015-04-28  15:56:53] INFO yp/yp_update_thread YP update thread started
[2015-04-28  15:56:53] INFO source/source_fallback_file mountpoint
/geen-uitzending.mp3 is reserved
[2015-04-28  15:56:53] DBUG connection/connection_complete_source sources
count is 0
[2015-04-28  15:56:53] WARN format/format_get_type Unsupported or legacy
stream type: "audio/mpeg". Falling back to generic minimal handler for
best
effort.
[2015-04-28  15:56:53] DBUG connection/connection_complete_source source is
ready to start
[2015-04-28  15:56:53] DBUG source/source_init Source creation complete
[2015-04-28  15:56:53] INFO stats/_stats_thread stats thread started
[2015-04-28  15:56:53] DBUG stats/process_source_event new source stat
/geen-uitzending.mp3




>
> > Both the stream and the fallback file are .mp3 type. I tried removing
> > the fallback entry. Then icecast starts always normally. So it seems
> > related to the .mp3 fallback.
> >
> > Also I tried to launch icecast with strace, and this NEVER fails. No
> > segmentation faults!
> >
> >
> > Any idea what's going wrong? Something going wrong with timing on
a
> > fast server?
>
> It sounds like you've found a race condition. I'm not exactly sure
> what's happening though.
> By the log output something seems to be instantly accessing the stream.
> Notice the order of the stats_thread is different in both.
>
>
Yes this is the only interesting thing. The segfault only happens when
"INFO stats/_stats_thread stats thread started"
is missing in the error.log. So with every succesful start this line gets
logged, and if the line is missing the segfault occurs.


I also tried running using valgrind on advice of Philipp, but when running
with valgrind the segfault does not occur and icecast starts normally...
(same as with gdb)

>
> Cheers
>
> Thomas
>
>
>
> _______________________________________________
> Icecast-dev mailing list
> Icecast-dev at xiph.org
> http://lists.xiph.org/mailman/listinfo/icecast-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.xiph.org/pipermail/icecast-dev/attachments/20150428/93fa448e/attachment-0001.htm
-------------- next part --------------
A non-text attachment was scrubbed...
Name: icecast.xml
Type: text/xml
Size: 11829 bytes
Desc: not available
Url :
http://lists.xiph.org/pipermail/icecast-dev/attachments/20150428/93fa448e/attachment-0001.bin

On 04/28/2015 02:27 PM, Jan-Pieter wrote:
> Hi all,
> see my reply inline.
> Cheers, Jan-Pieter
>
>
> 2015-04-26 7:23 GMT+02:00 "Thomas B. R?cker" <thomas at
ruecker.fi
> <mailto:thomas at ruecker.fi>>:
>
>     On 04/25/2015 07:32 PM, Jan-Pieter wrote:
>     > Hi all,
>     >
>     > I am a long-time user of Icecast, always without problems. However
i
>     > made a new setup using icecast 2.4.1 on a fresh and quick CentOS
>     VPS.
>
>     EPEL package or from somewhere else?
>
> it's just a standard CentOS 6.6 release, no EPEL. Distri from hosting
> provider. [root at h2429770 tmp]# uname -a
> Linux h2429770.stratoserver.net <http://h2429770.stratoserver.net>
> 2.6.32-042stab103.6 #1 SMP Wed Jan 21 13:07:39 MSK 2015 x86_64 x86_64
> x86_64 GNU/Linux
>  
>
>
>     > The issue is that starting icecast gives a segmentation fault at
>     > around 80% of all starts. No core file nor a entry in error.log
>     is posted.
>
>     Enable core dumps using "ulimit -c unlimited". The PWD from
where you
>     start Icecast needs to be writeable.
>
>
> Even with ulimit -c unlimited and running from a writable PWD no core
> is dumped.
Seems it's a bit more involved on Centos6:
http://www.unixmen.com/how-to-enable-core-dumps-in-rhel6/
I've tried it successfully on a quick install of centos6 that I've
compiled 2.4.2 on and am going to try with your config.
(As you don't use url-auth for sources, 2.4.1 is safe for you)

>     > In the remaining 20% of the starts icecast start with no problem.
>     >
>     > Contents of the error.log with a succesfull start (20% of
>     icecast starts):
>     > [2015-04-25  21:08:40] INFO main/main Icecast 2.4.1 server started
>     > [2015-04-25  21:08:40] INFO connection/get_ssl_certificate No SSL
>     > capability
>     > [2015-04-25  21:08:40] INFO stats/_stats_thread stats thread
started
>     > [2015-04-25  21:08:40] INFO yp/yp_update_thread YP update thread
>     started
>     > [2015-04-25  21:08:40] INFO source/source_fallback_file mountpoint
>     > /geen-uitzending.mp3 is reserved
>     > [2015-04-25  21:08:40] WARN format/format_get_type Unsupported or
>     > legacy stream type: "audio/mpeg". Falling back to
generic minimal
>     > handler for best effort.
>     > [2015-04-25  21:08:40] WARN source/source_fallback_file mountpoint
>     > "/geen-uitzending.mp3" already reserved
>     > [2015-04-25  21:08:40] INFO source/source_main listener count on
>     > /geen-uitzending.mp3 now 0
>     >
>     >
>     > Contents of error.log with a segmentation fault direct after start
>     > (80% of icecast starts):
>     > 2015-04-25  21:09:05] INFO main/main Icecast 2.4.1 server started
>     > [2015-04-25  21:09:05] INFO connection/get_ssl_certificate No SSL
>     > capability
>     > [2015-04-25  21:09:05] INFO yp/yp_update_thread YP update thread
>     started
>     > [2015-04-25  21:09:05] INFO source/source_fallback_file mountpoint
>     > /geen-uitzending.mp3 is reserved
>     > [2015-04-25  21:09:05] WARN format/format_get_type Unsupported or
>     > legacy stream type: "audio/mpeg". Falling back to
generic minimal
>     > handler for best effort.
>     > [2015-04-25  21:09:05] INFO stats/_stats_thread stats thread
started
>
>     Can you attach your icecast.xml please (with passwords removed)?
>     This might help reproduce the problem.
>
>
> I attached icecast.xml. I also turned debug errorlog level on.
> Results with debug on:
>
> good start:
> [2015-04-28  15:56:39] INFO main/main Icecast 2.4.1 server started
> [2015-04-28  15:56:39] DBUG yp/yp_recheck_config Updating YP configuration
> [2015-04-28  15:56:39] INFO connection/get_ssl_certificate No SSL
> capability
> [2015-04-28  15:56:39] INFO stats/_stats_thread stats thread started
> [2015-04-28  15:56:39] INFO yp/yp_update_thread YP update thread started
> [2015-04-28  15:56:39] INFO source/source_fallback_file mountpoint
> /geen-uitzending.mp3 is reserved
> [2015-04-28  15:56:39] DBUG connection/connection_complete_source
> sources count is 0
> [2015-04-28  15:56:39] WARN format/format_get_type Unsupported or
> legacy stream type: "audio/mpeg". Falling back to generic minimal
> handler for best effort.
> [2015-04-28  15:56:39] DBUG connection/connection_complete_source
> source is ready to start
> [2015-04-28  15:56:39] DBUG source/source_init Source creation complete
> [2015-04-28  15:56:39] WARN source/source_fallback_file mountpoint
> "/geen-uitzending.mp3" already reserved
> [2015-04-28  15:56:39] INFO source/source_main listener count on
> /geen-uitzending.mp3 now 0
> [2015-04-28  15:56:39] DBUG stats/modify_node_event update global
> sources (1)
> [2015-04-28  15:56:39] DBUG stats/process_source_event new source stat
> /geen-uitzending.mp3
> [2015-04-28  15:56:39] DBUG stats/modify_node_event update global
> source_client_connections (1)
> [2015-04-28  15:56:39] DBUG stats/process_source_event new node
> listeners (0)
> [2015-04-28  15:56:39] DBUG stats/process_source_event new node
> listenurl (http://stream3.server7.nl:80/geen-uitzending.mp3)
> [2015-04-28  15:56:39] DBUG stats/modify_node_event update global
> source_total_connections (1)
> [2015-04-28  15:56:39] DBUG stats/process_source_event new node
> slow_listeners (0)
> [2015-04-28  15:56:39] DBUG stats/modify_node_event update
> "/geen-uitzending.mp3" listeners (0)
> [2015-04-28  15:56:39] DBUG stats/process_source_event new node
> listener_peak (0)
> [2015-04-28  15:56:39] DBUG stats/process_source_event new node
> stream_start (Tue, 28 Apr 2015 15:56:39 +0200)
> [2015-04-28  15:56:39] DBUG stats/process_source_event new node
> stream_start_iso8601 (2015-04-28T15:56:39+0200)
> [2015-04-28  15:56:39] DBUG stats/process_source_event new node
> total_bytes_read (0)
> [2015-04-28  15:56:39] DBUG stats/process_source_event new node
> total_bytes_sent (0)
> [2015-04-28  15:56:39] DBUG stats/modify_node_event update
> "/geen-uitzending.mp3" listeners (0)
> [2015-04-28  15:56:40] DBUG slave/_slave_thread checking master stream
> list
I currently suspect it's something interacting with the stats thread and
the two identical fallbacks.

>
> bad start (segfault):
> [2015-04-28  15:56:53] INFO main/main Icecast 2.4.1 server started
> [2015-04-28  15:56:53] DBUG yp/yp_recheck_config Updating YP configuration
> [2015-04-28  15:56:53] INFO connection/get_ssl_certificate No SSL
> capability
> [2015-04-28  15:56:53] INFO yp/yp_update_thread YP update thread started
> [2015-04-28  15:56:53] INFO source/source_fallback_file mountpoint
> /geen-uitzending.mp3 is reserved
> [2015-04-28  15:56:53] DBUG connection/connection_complete_source
> sources count is 0
> [2015-04-28  15:56:53] WARN format/format_get_type Unsupported or
> legacy stream type: "audio/mpeg". Falling back to generic minimal
> handler for best effort.
> [2015-04-28  15:56:53] DBUG connection/connection_complete_source
> source is ready to start
> [2015-04-28  15:56:53] DBUG source/source_init Source creation complete
> [2015-04-28  15:56:53] INFO stats/_stats_thread stats thread started
> [2015-04-28  15:56:53] DBUG stats/process_source_event new source stat
> /geen-uitzending.mp3
>
>
>
>  
>
>
>     > Both the stream and the fallback file are .mp3 type. I tried
>     removing
>     > the fallback entry. Then icecast starts always normally. So it
seems
>     > related to the .mp3 fallback.
>     >
>     > Also I tried to launch icecast with strace, and this NEVER fails.
No
>     > segmentation faults!
>     >
>     >
>     > Any idea what's going wrong? Something going wrong with timing
on a
>     > fast server?
>
>     It sounds like you've found a race condition. I'm not exactly
sure
>     what's happening though.
>     By the log output something seems to be instantly accessing the
>     stream.
>     Notice the order of the stats_thread is different in both.
>
>
> Yes this is the only interesting thing. The segfault only happens when 
> "INFO stats/_stats_thread stats thread started" 
> is missing in the error.log. So with every succesful start this line
> gets logged, and if the line is missing the segfault occurs.
I suspect it's a very close race that might only occur on that specific
configuration/virtualization.
In such case invasive things like strace and valgrind won't help us. A
coredump might reveal something, but isn't guaranteed.

What BTW confuses me is that you override the mountpoint configuration
to ogg/vorbis, but fall back to an mp3 file.

> I also tried running using valgrind on advice of Philipp, but when
> running with valgrind the segfault does not occur and icecast starts
> normally... (same as with gdb)
yeah, expected.

I tried your config on a centos 6 machine and couldn't reproduce it. How
many CPU cores / threads does it have?


Cheers

Thomas

2015-04-29 8:57 GMT+02:00 "Thomas B. R?cker" <thomas at
ruecker.fi>:
> On 04/28/2015 02:27 PM, Jan-Pieter wrote:
> > Hi all,
> > see my reply inline.
> > Cheers, Jan-Pieter
> >
> >
> > 2015-04-26 7:23 GMT+02:00 "Thomas B. R?cker" <thomas at
ruecker.fi
> > <mailto:thomas at ruecker.fi>>:
> >
> >     On 04/25/2015 07:32 PM, Jan-Pieter wrote:
> >     > Hi all,
> >     >
> >     > I am a long-time user of Icecast, always without problems.
However
> i
> >     > made a new setup using icecast 2.4.1 on a fresh and quick
CentOS
> >     VPS.
> >
> >     EPEL package or from somewhere else?
> >
> > it's just a standard CentOS 6.6 release, no EPEL. Distri from
hosting
> > provider. [root at h2429770 tmp]# uname -a
> > Linux h2429770.stratoserver.net
<http://h2429770.stratoserver.net>
> > 2.6.32-042stab103.6 #1 SMP Wed Jan 21 13:07:39 MSK 2015 x86_64 x86_64
> > x86_64 GNU/Linux
> >
> >
> >
> >     > The issue is that starting icecast gives a segmentation fault
at
> >     > around 80% of all starts. No core file nor a entry in
error.log
> >     is posted.
> >
> >     Enable core dumps using "ulimit -c unlimited". The PWD
from where you
> >     start Icecast needs to be writeable.
> >
> >
> > Even with ulimit -c unlimited and running from a writable PWD no core
> > is dumped.
>
> Seems it's a bit more involved on Centos6:
> http://www.unixmen.com/how-to-enable-core-dumps-in-rhel6/
> I've tried it successfully on a quick install of centos6 that I've
> compiled 2.4.2 on and am going to try with your config.
> (As you don't use url-auth for sources, 2.4.1 is safe for you)
>
> I followed this steps, but still no core file with this type of segfault.
Other programs which segfaults do produce a core dump however. so bit
strange...

>
> >     > In the remaining 20% of the starts icecast start with no
problem.
> >     >
> >     > Contents of the error.log with a succesfull start (20% of
> >     icecast starts):
> >     > [2015-04-25  21:08:40] INFO main/main Icecast 2.4.1 server
started
> >     > [2015-04-25  21:08:40] INFO connection/get_ssl_certificate No
SSL
> >     > capability
> >     > [2015-04-25  21:08:40] INFO stats/_stats_thread stats thread
> started
> >     > [2015-04-25  21:08:40] INFO yp/yp_update_thread YP update
thread
> >     started
> >     > [2015-04-25  21:08:40] INFO source/source_fallback_file
mountpoint
> >     > /geen-uitzending.mp3 is reserved
> >     > [2015-04-25  21:08:40] WARN format/format_get_type
Unsupported or
> >     > legacy stream type: "audio/mpeg". Falling back to
generic minimal
> >     > handler for best effort.
> >     > [2015-04-25  21:08:40] WARN source/source_fallback_file
mountpoint
> >     > "/geen-uitzending.mp3" already reserved
> >     > [2015-04-25  21:08:40] INFO source/source_main listener count
on
> >     > /geen-uitzending.mp3 now 0
> >     >
> >     >
> >     > Contents of error.log with a segmentation fault direct after
start
> >     > (80% of icecast starts):
> >     > 2015-04-25  21:09:05] INFO main/main Icecast 2.4.1 server
started
> >     > [2015-04-25  21:09:05] INFO connection/get_ssl_certificate No
SSL
> >     > capability
> >     > [2015-04-25  21:09:05] INFO yp/yp_update_thread YP update
thread
> >     started
> >     > [2015-04-25  21:09:05] INFO source/source_fallback_file
mountpoint
> >     > /geen-uitzending.mp3 is reserved
> >     > [2015-04-25  21:09:05] WARN format/format_get_type
Unsupported or
> >     > legacy stream type: "audio/mpeg". Falling back to
generic minimal
> >     > handler for best effort.
> >     > [2015-04-25  21:09:05] INFO stats/_stats_thread stats thread
> started
> >
> >     Can you attach your icecast.xml please (with passwords removed)?
> >     This might help reproduce the problem.
> >
> >
> > I attached icecast.xml. I also turned debug errorlog level on.
> > Results with debug on:
> >
> > good start:
> > [2015-04-28  15:56:39] INFO main/main Icecast 2.4.1 server started
> > [2015-04-28  15:56:39] DBUG yp/yp_recheck_config Updating YP
> configuration
> > [2015-04-28  15:56:39] INFO connection/get_ssl_certificate No SSL
> > capability
> > [2015-04-28  15:56:39] INFO stats/_stats_thread stats thread started
> > [2015-04-28  15:56:39] INFO yp/yp_update_thread YP update thread
started
> > [2015-04-28  15:56:39] INFO source/source_fallback_file mountpoint
> > /geen-uitzending.mp3 is reserved
> > [2015-04-28  15:56:39] DBUG connection/connection_complete_source
> > sources count is 0
> > [2015-04-28  15:56:39] WARN format/format_get_type Unsupported or
> > legacy stream type: "audio/mpeg". Falling back to generic
minimal
> > handler for best effort.
> > [2015-04-28  15:56:39] DBUG connection/connection_complete_source
> > source is ready to start
> > [2015-04-28  15:56:39] DBUG source/source_init Source creation
complete
> > [2015-04-28  15:56:39] WARN source/source_fallback_file mountpoint
> > "/geen-uitzending.mp3" already reserved
> > [2015-04-28  15:56:39] INFO source/source_main listener count on
> > /geen-uitzending.mp3 now 0
> > [2015-04-28  15:56:39] DBUG stats/modify_node_event update global
> > sources (1)
> > [2015-04-28  15:56:39] DBUG stats/process_source_event new source stat
> > /geen-uitzending.mp3
> > [2015-04-28  15:56:39] DBUG stats/modify_node_event update global
> > source_client_connections (1)
> > [2015-04-28  15:56:39] DBUG stats/process_source_event new node
> > listeners (0)
> > [2015-04-28  15:56:39] DBUG stats/process_source_event new node
> > listenurl (http://stream3.server7.nl:80/geen-uitzending.mp3)
> > [2015-04-28  15:56:39] DBUG stats/modify_node_event update global
> > source_total_connections (1)
> > [2015-04-28  15:56:39] DBUG stats/process_source_event new node
> > slow_listeners (0)
> > [2015-04-28  15:56:39] DBUG stats/modify_node_event update
> > "/geen-uitzending.mp3" listeners (0)
> > [2015-04-28  15:56:39] DBUG stats/process_source_event new node
> > listener_peak (0)
> > [2015-04-28  15:56:39] DBUG stats/process_source_event new node
> > stream_start (Tue, 28 Apr 2015 15:56:39 +0200)
> > [2015-04-28  15:56:39] DBUG stats/process_source_event new node
> > stream_start_iso8601 (2015-04-28T15:56:39+0200)
> > [2015-04-28  15:56:39] DBUG stats/process_source_event new node
> > total_bytes_read (0)
> > [2015-04-28  15:56:39] DBUG stats/process_source_event new node
> > total_bytes_sent (0)
> > [2015-04-28  15:56:39] DBUG stats/modify_node_event update
> > "/geen-uitzending.mp3" listeners (0)
> > [2015-04-28  15:56:40] DBUG slave/_slave_thread checking master stream
> > list
>
> I currently suspect it's something interacting with the stats thread
and
> the two identical fallbacks.
>
> Agree, seems to have something to do with stats thread. But I tested also
with dual fall back files. Same issue. See attached new configfile.
>
> >
> > bad start (segfault):
> > [2015-04-28  15:56:53] INFO main/main Icecast 2.4.1 server started
> > [2015-04-28  15:56:53] DBUG yp/yp_recheck_config Updating YP
> configuration
> > [2015-04-28  15:56:53] INFO connection/get_ssl_certificate No SSL
> > capability
> > [2015-04-28  15:56:53] INFO yp/yp_update_thread YP update thread
started
> > [2015-04-28  15:56:53] INFO source/source_fallback_file mountpoint
> > /geen-uitzending.mp3 is reserved
> > [2015-04-28  15:56:53] DBUG connection/connection_complete_source
> > sources count is 0
> > [2015-04-28  15:56:53] WARN format/format_get_type Unsupported or
> > legacy stream type: "audio/mpeg". Falling back to generic
minimal
> > handler for best effort.
> > [2015-04-28  15:56:53] DBUG connection/connection_complete_source
> > source is ready to start
> > [2015-04-28  15:56:53] DBUG source/source_init Source creation
complete
> > [2015-04-28  15:56:53] INFO stats/_stats_thread stats thread started
> > [2015-04-28  15:56:53] DBUG stats/process_source_event new source stat
> > /geen-uitzending.mp3
> >
> >
> >
> >
> >
> >
> >     > Both the stream and the fallback file are .mp3 type. I tried
> >     removing
> >     > the fallback entry. Then icecast starts always normally. So
it
> seems
> >     > related to the .mp3 fallback.
> >     >
> >     > Also I tried to launch icecast with strace, and this NEVER
fails.
> No
> >     > segmentation faults!
> >     >
> >     >
> >     > Any idea what's going wrong? Something going wrong with
timing on a
> >     > fast server?
> >
> >     It sounds like you've found a race condition. I'm not
exactly sure
> >     what's happening though.
> >     By the log output something seems to be instantly accessing the
> >     stream.
> >     Notice the order of the stats_thread is different in both.
> >
> >
> > Yes this is the only interesting thing. The segfault only happens when
> > "INFO stats/_stats_thread stats thread started"
> > is missing in the error.log. So with every succesful start this line
> > gets logged, and if the line is missing the segfault occurs.
>
> I suspect it's a very close race that might only occur on that specific
> configuration/virtualization.
> In such case invasive things like strace and valgrind won't help us. A
> coredump might reveal something, but isn't guaranteed.
>
> What BTW confuses me is that you override the mountpoint configuration
> to ogg/vorbis, but fall back to an mp3 file.
>
> Sorry my mistake. Tried changing things here, but did not change the
behaviour. In the attached config file I set it all to mp3 and problem
persists.
Agree that it has probably something to do with virtualization platform. My
VPS is running  in a Virtuozzo container. However with other programs no
issue at all. Looks to me still as a timing issue that somehow the stats
thread is not started causing the segfault when loading the fallback
stream. Maybe a small noop/sleep just before the statsthread will solve my
issue...

> > I also tried running using valgrind on advice of Philipp, but when
> > running with valgrind the segfault does not occur and icecast starts
> > normally... (same as with gdb)
>
> yeah, expected.
>
> I tried your config on a centos 6 machine and couldn't reproduce it.
How
> many CPU cores / threads does it have?
>
> 4 cores/threads.

>
> Cheers
>
> Thomas
>
> Thanks!
Jan-Pieter
> _______________________________________________
> Icecast-dev mailing list
> Icecast-dev at xiph.org
> http://lists.xiph.org/mailman/listinfo/icecast-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.xiph.org/pipermail/icecast-dev/attachments/20150430/04f23245/attachment.htm