asterisk users - Sep 2016 - Upgrading asterisk 13.7 to 13.11. Segfaults

Hello.

Several months server working on asterisk 13.7 and pjproject 2.5 
(installed separately). Once a day the server crashes or hangs and is 
familiar sores that written watchdogs.

Yesterday I decided to upgrade to 13.11 and bundled pjproject (2.5.5). 
Solved all the problems with compilation I started asterisk several 
times and each time after 5-7 seconds was seg fault.

So I didn't get to use the new version of asterisk. And I really wanted 
to be able to find and remove the cause. I would be grateful for any help.


first segfault:

Program terminated with signal 11, Segmentation fault.
#0  0x00007fe19337357e in pj_atomic_dec_and_get (atomic_var=0x1a9a) at 
../src/pj/os_core_unix.c:962
962        pj_mutex_lock( atomic_var->mutex );

backtrace: https://ruvoip.net/_other/voip/2016-09-05-1/backtrace-threads.txt

log (16 MB) : https://ruvoip.net/_other/voip/2016-09-05-1/full.txt

second segfault:

Program terminated with signal 11, Segmentation fault.
#0  0x00007f5777412788 in pjsip_auth_clt_reinit_req 
(sess=0x7f56f4088a30, rdata=0x7f56ec2b0c98, old_request=0x7f56f47775a8, 
new_request=0x7f56d5b38980) at ../src/pjsip/sip_auth_client.c:1144
1144        PJ_ASSERT_RETURN(old_request->msg->type == PJSIP_REQUEST_MSG,

backtrace: https://ruvoip.net/_other/voip/2016-09-05-2/backtrace-threads.txt

log (15 MB) : https://ruvoip.net/_other/voip/2016-09-05-2/full.txt

third segfault:

Program terminated with signal 11, Segmentation fault.
#0  0x00007f592bcad53d in pj_pool_alloc (pool=0x7f580000002c, size=80) 
at ../include/pj/pool_i.h:60
60        void *ptr = pj_pool_alloc_from_block(pool->block_list.next, size);

backtrace: https://ruvoip.net/_other/voip/2016-09-05-3/backtrace-threads.txt

log (11 MB) : https://ruvoip.net/_other/voip/2016-09-05-3/full.txt

On Tue, Sep 6, 2016 at 7:32 AM, Dmitriy Serov <serov.d.p at gmail.com>
wrote:
> Hello.
>
> Several months server working on asterisk 13.7 and pjproject 2.5
> (installed separately). Once a day the server crashes or hangs and is
> familiar sores that written watchdogs.
>
> Yesterday I decided to upgrade to 13.11 and bundled pjproject (2.5.5).
> Solved all the problems with compilation I started asterisk several times
> and each time after 5-7 seconds was seg fault.
>
> So I didn't get to use the new version of asterisk. And I really wanted
to
> be able to find and remove the cause. I would be grateful for any help.
>
>
All 3 of the backtraces are in different pjproject places which is weird.
Makes me think there's still a library mismatch somewhere.  Is the
separately compiled pjproject still installed on both the build machine and
the server?  They *should* be ignored if --with-pjproject-bundled is
specified but you might want to remove them and try again.


>
> first segfault:
>
> Program terminated with signal 11, Segmentation fault.
> #0  0x00007fe19337357e in pj_atomic_dec_and_get (atomic_var=0x1a9a) at
> ../src/pj/os_core_unix.c:962
> 962        pj_mutex_lock( atomic_var->mutex );
>
> backtrace: https://ruvoip.net/_other/voip/2016-09-05-1/backtrace-thread
> s.txt
>
> log (16 MB) : https://ruvoip.net/_other/voip/2016-09-05-1/full.txt
>
> second segfault:
>
> Program terminated with signal 11, Segmentation fault.
> #0  0x00007f5777412788 in pjsip_auth_clt_reinit_req (sess=0x7f56f4088a30,
> rdata=0x7f56ec2b0c98, old_request=0x7f56f47775a8,
> new_request=0x7f56d5b38980) at ../src/pjsip/sip_auth_client.c:1144
> 1144        PJ_ASSERT_RETURN(old_request->msg->type ==
PJSIP_REQUEST_MSG,
>
> backtrace: https://ruvoip.net/_other/voip/2016-09-05-2/backtrace-thread
> s.txt
>
> log (15 MB) : https://ruvoip.net/_other/voip/2016-09-05-2/full.txt
>
> third segfault:
>
> Program terminated with signal 11, Segmentation fault.
> #0  0x00007f592bcad53d in pj_pool_alloc (pool=0x7f580000002c, size=80) at
> ../include/pj/pool_i.h:60
> 60        void *ptr = pj_pool_alloc_from_block(pool->block_list.next,
> size);
>
> backtrace: https://ruvoip.net/_other/voip/2016-09-05-3/backtrace-thread
> s.txt
>
> log (11 MB) : https://ruvoip.net/_other/voip/2016-09-05-3/full.txt
>
>
>
> --
> _____________________________________________________________________
> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>
> Join the Asterisk Community at the 13th AstriCon, September 27-29, 2016
>      http://www.asterisk.org/community/astricon-user-conference
>
> New to Asterisk? Start here:
>      https://wiki.asterisk.org/wiki/display/AST/Getting+Started
>
> asterisk-users mailing list
> To UNSUBSCRIBE or update options visit:
>   http://lists.digium.com/mailman/listinfo/asterisk-users
>


-- 
George Joseph
Digium, Inc. | Software Developer
445 Jan Davis Drive NW - Huntsville, AL 35806 - US
Check us out at: www.digium.com & www.asterisk.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.digium.com/pipermail/asterisk-users/attachments/20160906/ed70383b/attachment.html>

All your libraries, kernel, headers and build tools up to date?

The other thing that might be worth noting is the warning along the
lines of "contains modules that were not installed by this version of
Asterisk".

Might be worth deleting anything that appears there, and then starting Asterisk.

On 6 September 2016 at 14:42, George Joseph <gjoseph at digium.com>
wrote:
>
>
> On Tue, Sep 6, 2016 at 7:32 AM, Dmitriy Serov <serov.d.p at
gmail.com> wrote:
>>
>> Hello.
>>
>> Several months server working on asterisk 13.7 and pjproject 2.5
>> (installed separately). Once a day the server crashes or hangs and is
>> familiar sores that written watchdogs.
>>
>> Yesterday I decided to upgrade to 13.11 and bundled pjproject (2.5.5).
>> Solved all the problems with compilation I started asterisk several
times
>> and each time after 5-7 seconds was seg fault.
>>
>> So I didn't get to use the new version of asterisk. And I really
wanted to
>> be able to find and remove the cause. I would be grateful for any help.
>>
>
> All 3 of the backtraces are in different pjproject places which is weird.
> Makes me think there's still a library mismatch somewhere.  Is the
> separately compiled pjproject still installed on both the build machine and
> the server?  They should be ignored if --with-pjproject-bundled is
specified
> but you might want to remove them and try again.
>
>
>>
>>
>> first segfault:
>>
>> Program terminated with signal 11, Segmentation fault.
>> #0  0x00007fe19337357e in pj_atomic_dec_and_get (atomic_var=0x1a9a) at
>> ../src/pj/os_core_unix.c:962
>> 962        pj_mutex_lock( atomic_var->mutex );
>>
>> backtrace:
>> https://ruvoip.net/_other/voip/2016-09-05-1/backtrace-threads.txt
>>
>> log (16 MB) : https://ruvoip.net/_other/voip/2016-09-05-1/full.txt
>>
>> second segfault:
>>
>> Program terminated with signal 11, Segmentation fault.
>> #0  0x00007f5777412788 in pjsip_auth_clt_reinit_req
(sess=0x7f56f4088a30,
>> rdata=0x7f56ec2b0c98, old_request=0x7f56f47775a8,
>> new_request=0x7f56d5b38980) at ../src/pjsip/sip_auth_client.c:1144
>> 1144        PJ_ASSERT_RETURN(old_request->msg->type ==
PJSIP_REQUEST_MSG,
>>
>> backtrace:
>> https://ruvoip.net/_other/voip/2016-09-05-2/backtrace-threads.txt
>>
>> log (15 MB) : https://ruvoip.net/_other/voip/2016-09-05-2/full.txt
>>
>> third segfault:
>>
>> Program terminated with signal 11, Segmentation fault.
>> #0  0x00007f592bcad53d in pj_pool_alloc (pool=0x7f580000002c, size=80)
at
>> ../include/pj/pool_i.h:60
>> 60        void *ptr =
pj_pool_alloc_from_block(pool->block_list.next,
>> size);
>>
>> backtrace:
>> https://ruvoip.net/_other/voip/2016-09-05-3/backtrace-threads.txt
>>
>> log (11 MB) : https://ruvoip.net/_other/voip/2016-09-05-3/full.txt
>>
>>
>>
>> --
>> _____________________________________________________________________
>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>
>> Join the Asterisk Community at the 13th AstriCon, September 27-29, 2016
>>      http://www.asterisk.org/community/astricon-user-conference
>>
>> New to Asterisk? Start here:
>>      https://wiki.asterisk.org/wiki/display/AST/Getting+Started
>>
>> asterisk-users mailing list
>> To UNSUBSCRIBE or update options visit:
>>   http://lists.digium.com/mailman/listinfo/asterisk-users
>
>
>
>
> --
> George Joseph
> Digium, Inc. | Software Developer
> 445 Jan Davis Drive NW - Huntsville, AL 35806 - US
> Check us out at: www.digium.com & www.asterisk.org
>
>
> --
> _____________________________________________________________________
> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>
> Join the Asterisk Community at the 13th AstriCon, September 27-29, 2016
>       http://www.asterisk.org/community/astricon-user-conference
>
> New to Asterisk? Start here:
>       https://wiki.asterisk.org/wiki/display/AST/Getting+Started
>
> asterisk-users mailing list
> To UNSUBSCRIBE or update options visit:
>    http://lists.digium.com/mailman/listinfo/asterisk-users

06.09.2016 16:42, George Joseph ?????:
>
>
> On Tue, Sep 6, 2016 at 7:32 AM, Dmitriy Serov <serov.d.p at gmail.com 
> <mailto:serov.d.p at gmail.com>> wrote:
>
>     Hello.
>
>     Several months server working on asterisk 13.7 and pjproject 2.5
>     (installed separately). Once a day the server crashes or hangs and
>     is familiar sores that written watchdogs.
>
>     Yesterday I decided to upgrade to 13.11 and bundled pjproject
>     (2.5.5). Solved all the problems with compilation I started
>     asterisk several times and each time after 5-7 seconds was seg fault.
>
>     So I didn't get to use the new version of asterisk. And I really
>     wanted to be able to find and remove the cause. I would be
>     grateful for any help.
>
>
> All 3 of the backtraces are in different pjproject places which is 
> weird.  Makes me think there's still a library mismatch somewhere.  Is 
> the separately compiled pjproject still installed on both the build 
> machine and the server?  They /should/ be ignored if 
> --with-pjproject-bundled is specified but you might want to remove 
> them and try again.
>
George, thank you that you responded. What I was hoping :)
I was also very surprised that SF can happen so quickly and so 
consistently. With all this in completely different places.

When installed separately pjproject source code with bundled not even 
compiled. Of course, I uninstalled and cleaned pjproject according to 
this instructions: 
https://wiki.asterisk.org/wiki/display/AST/Building+and+Installing+pjproject

At the moment I have restore separately pjproject and asterisk 13.7. But 
there is a directory with configured and compiled asterisk 13.11. Happy 
to provide the contents of the autoconfiguration files.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.digium.com/pipermail/asterisk-users/attachments/20160906/d3d7f5ff/attachment.html>