thr3ads.net - asterisk users - [asterisk-users] Upgrading 13.7 (external pjproject) to 13.9 (bundled pjproject) [Apr 2016]

If this information is useful, please help other people find it:
Share via:

Dmitriy Serov

2016-Apr-28 07:10 UTC

[asterisk-users] Upgrading 13.7 (external pjproject) to 13.9 (bundled pjproject)

Today was another attempt to upgrade to version 13.9 (git).

1. The result was https://issues.asterisk.org/jira/browse/ASTERISK-25970

Had to temporarily block this contact and look forward to advice of how 
to fix it.

2. Also, an unpleasant surprise was the increase in CPU usage from 
10-50% to 200-400% (4 cores). Stable CPU overusage with the same build 
options (DONT_OPTIMIZE, DEBUG_THREADS, BETTER_BACKTRACES, BUILD_NATIVE)

3. After 20 minutes of this work the server has ceased to respond 
promptly to registrations. "core show locks" is attached. Suggestions 
about solving the problem are very welcome.

Dmitriy.


-------------- next part --------------
Setting max files open to 300000

=========================================================================
13.9.0-rc1
=== Currently Held Locks
===========================================================================
<pending> <lock#> (<file>): <lock type> <line num>
<function> <lock name> <lock addr> (times locked)
===== Thread ID: 0x7fbaedb2c700 LWP:25598 (worker_start         started at [
1077] threadpool.c worker_thread_start())
=== ---> Lock #0 (sorcery.c): RDLOCK 1883 ast_sorcery_retrieve_by_fields
&(&object_type->wizards)->lock 0x2243420 (1)
	main/backtrace.c:59 __ast_bt_get_addresses() (0x466440+1D)
	main/lock.c:866 __ast_rwlock_rdlock() (0x53652a+BA)
	main/sorcery.c:1884 ast_sorcery_retrieve_by_fields() (0x5bf860+C2)
	res_pjsip/pjsip_options.c:290 find_an_endpoint()
	res_pjsip/pjsip_options.c:342 qualify_contact()
	res_pjsip/pjsip_options.c:443 qualify_contact_task()
	main/taskprocessor.c:852 ast_taskprocessor_execute() (0x5e02d3+10D)
	main/threadpool.c:1320 execute_tasks()
	main/taskprocessor.c:852 ast_taskprocessor_execute() (0x5e02d3+10D)
	main/threadpool.c:351 threadpool_execute()
	main/threadpool.c:1103 worker_active()
	main/threadpool.c:1024 worker_start()
	main/utils.c:1235 dummy_start()
	:0 start_thread()
	:0 __clone() (0x7fbb8bb68860+6D)
=== -------------------------------------------------------------------
===== Thread ID: 0x7fbaedab0700 LWP:25596 (worker_start         started at [
1077] threadpool.c worker_thread_start())
=== ---> Lock #0 (sorcery.c): RDLOCK 1883 ast_sorcery_retrieve_by_fields
&(&object_type->wizards)->lock 0x2243420 (1)
	main/backtrace.c:59 __ast_bt_get_addresses() (0x466440+1D)
	main/lock.c:866 __ast_rwlock_rdlock() (0x53652a+BA)
	main/sorcery.c:1884 ast_sorcery_retrieve_by_fields() (0x5bf860+C2)
	res_pjsip/pjsip_options.c:290 find_an_endpoint()
	res_pjsip/pjsip_options.c:342 qualify_contact()
	res_pjsip/pjsip_options.c:443 qualify_contact_task()
	main/taskprocessor.c:852 ast_taskprocessor_execute() (0x5e02d3+10D)
	main/threadpool.c:1320 execute_tasks()
	main/taskprocessor.c:852 ast_taskprocessor_execute() (0x5e02d3+10D)
	main/threadpool.c:351 threadpool_execute()
	main/threadpool.c:1103 worker_active()
	main/threadpool.c:1024 worker_start()
	main/utils.c:1235 dummy_start()
	:0 start_thread()
	:0 __clone() (0x7fbb8bb68860+6D)
=== -------------------------------------------------------------------
===== Thread ID: 0x7fbb25795700 LWP:25597 (worker_start         started at [
1077] threadpool.c worker_thread_start())
=== ---> Lock #0 (sorcery.c): RDLOCK 1883 ast_sorcery_retrieve_by_fields
&(&object_type->wizards)->lock 0x2243420 (1)
	main/backtrace.c:59 __ast_bt_get_addresses() (0x466440+1D)
	main/lock.c:866 __ast_rwlock_rdlock() (0x53652a+BA)
	main/sorcery.c:1884 ast_sorcery_retrieve_by_fields() (0x5bf860+C2)
	res_pjsip/pjsip_options.c:290 find_an_endpoint()
	res_pjsip/pjsip_options.c:342 qualify_contact()
	res_pjsip/pjsip_options.c:443 qualify_contact_task()
	main/taskprocessor.c:852 ast_taskprocessor_execute() (0x5e02d3+10D)
	main/threadpool.c:1320 execute_tasks()
	main/taskprocessor.c:852 ast_taskprocessor_execute() (0x5e02d3+10D)
	main/threadpool.c:351 threadpool_execute()
	main/taskprocessor.c:852 ast_taskprocessor_execute() (0x5e02d3+8CB8)
	main/threadpool.c:1024 worker_start()
	main/utils.c:1235 dummy_start()
	:0 start_thread()
	:0 __clone() (0x7fbb8bb68860+6D)
=== -------------------------------------------------------------------
===== Thread ID: 0x7fbb254ad700 LWP:25895 (worker_start         started at [
1077] threadpool.c worker_thread_start())
=== ---> Lock #0 (sorcery.c): RDLOCK 1883 ast_sorcery_retrieve_by_fields
&(&object_type->wizards)->lock 0x2243420 (1)
	main/backtrace.c:59 __ast_bt_get_addresses() (0x466440+1D)
	main/lock.c:866 __ast_rwlock_rdlock() (0x53652a+BA)
	main/sorcery.c:1884 ast_sorcery_retrieve_by_fields() (0x5bf860+C2)
	res_pjsip/pjsip_options.c:290 find_an_endpoint()
	res_pjsip/pjsip_options.c:342 qualify_contact()
	res_pjsip/pjsip_options.c:443 qualify_contact_task()
	main/taskprocessor.c:852 ast_taskprocessor_execute() (0x5e02d3+10D)
	main/threadpool.c:1320 execute_tasks()
	main/taskprocessor.c:852 ast_taskprocessor_execute() (0x5e02d3+10D)
	main/threadpool.c:351 threadpool_execute()
	main/threadpool.c:1103 worker_active()
	main/threadpool.c:1024 worker_start()
	main/utils.c:1235 dummy_start()
	:0 start_thread()
	:0 __clone() (0x7fbb8bb68860+6D)
=== -------------------------------------------------------------------
===== Thread ID: 0x7fbb8abf4700 LWP:26536 (worker_start         started at [
1077] threadpool.c worker_thread_start())
=== ---> Lock #0 (sorcery.c): RDLOCK 1883 ast_sorcery_retrieve_by_fields
&(&object_type->wizards)->lock 0x2243420 (1)
	main/backtrace.c:59 __ast_bt_get_addresses() (0x466440+1D)
	main/lock.c:866 __ast_rwlock_rdlock() (0x53652a+BA)
	main/sorcery.c:1884 ast_sorcery_retrieve_by_fields()
(0x5bf860+FFFFFFFFFFFFFE63)
	res/res_pjsip_endpoint_identifier_user.c:101 username_identify()
	res/res_pjsip.c:2247 ast_sip_identify_endpoint() (0x7fbb2634d440+52)
	res_pjsip/pjsip_distributor.c:384 endpoint_lookup()
	main/taskprocessor.c:852 ast_taskprocessor_execute() (0x5e02d3+10D)
	main/threadpool.c:1320 execute_tasks()
	main/taskprocessor.c:852 ast_taskprocessor_execute() (0x5e02d3+10D)
	main/threadpool.c:351 threadpool_execute()
	main/threadpool.c:1103 worker_active()
	main/threadpool.c:1024 worker_start()
	main/utils.c:1235 dummy_start()
	:0 start_thread()
	:0 __clone() (0x7fbb8bb68860+6D)
=== -------------------------------------------------------------------
===== Thread ID: 0x7fbaec3ab700 LWP:26537 (worker_start         started at [
1077] threadpool.c worker_thread_start())
=== ---> Lock #0 (sorcery.c): RDLOCK 1883 ast_sorcery_retrieve_by_fields
&(&object_type->wizards)->lock 0x2243420 (1)
	main/backtrace.c:59 __ast_bt_get_addresses() (0x466440+1D)
	main/lock.c:866 __ast_rwlock_rdlock() (0x53652a+BA)
	main/sorcery.c:1884 ast_sorcery_retrieve_by_fields() (0x5bf860+C2)
	res_pjsip/pjsip_options.c:290 find_an_endpoint()
	res/res_pjsip.c:2247 ast_sip_identify_endpoint() (0x7fbb2634d440+52)
	res_pjsip/pjsip_distributor.c:384 endpoint_lookup()
	main/taskprocessor.c:852 ast_taskprocessor_execute() (0x5e02d3+10D)
	main/threadpool.c:1320 execute_tasks()
	main/taskprocessor.c:852 ast_taskprocessor_execute() (0x5e02d3+10D)
	main/threadpool.c:351 threadpool_execute()
	main/threadpool.c:1103 worker_active()
	main/threadpool.c:1024 worker_start()
	main/utils.c:1235 dummy_start()
	:0 start_thread()
	:0 __clone() (0x7fbb8bb68860+6D)
=== -------------------------------------------------------------------
===== Thread ID: 0x7fbaec693700 LWP:26535 (worker_start         started at [
1077] threadpool.c worker_thread_start())
=== ---> Lock #0 (sorcery.c): RDLOCK 1883 ast_sorcery_retrieve_by_fields
&(&object_type->wizards)->lock 0x2243420 (1)
	main/backtrace.c:59 __ast_bt_get_addresses() (0x466440+1D)
	main/lock.c:866 __ast_rwlock_rdlock() (0x53652a+BA)
	main/sorcery.c:1884 ast_sorcery_retrieve_by_fields() (0x5bf860+C2)
	res_pjsip/pjsip_options.c:290 find_an_endpoint()
	res_pjsip/pjsip_options.c:342 qualify_contact()
	res_pjsip/pjsip_options.c:443 qualify_contact_task()
	main/taskprocessor.c:852 ast_taskprocessor_execute() (0x5e02d3+7FBB8E58394A)
	res_pjsip/pjsip_distributor.c:461 distribute()
	main/taskprocessor.c:852 ast_taskprocessor_execute() (0x5e02d3+10D)
	main/threadpool.c:1320 execute_tasks()
	main/threadpool.c:1103 worker_active()
	main/threadpool.c:1024 worker_start()
	main/utils.c:1235 dummy_start()
	:0 start_thread()
	:0 __clone() (0x7fbb8bb68860+6D)
=== -------------------------------------------------------------------
===== Thread ID: 0x7fbaeca73700 LWP:26534 (worker_start         started at [
1077] threadpool.c worker_thread_start())
=== ---> Lock #0 (sorcery.c): RDLOCK 1883 ast_sorcery_retrieve_by_fields
&(&object_type->wizards)->lock 0x2243420 (1)
	main/backtrace.c:59 __ast_bt_get_addresses() (0x466440+1D)
	main/lock.c:866 __ast_rwlock_rdlock() (0x53652a+BA)
	main/sorcery.c:1884 ast_sorcery_retrieve_by_fields() (0x5bf860+C2)
	res_pjsip/pjsip_options.c:290 find_an_endpoint()
	res_pjsip/pjsip_options.c:342 qualify_contact()
	res_pjsip/pjsip_options.c:443 qualify_contact_task()
	main/taskprocessor.c:852 ast_taskprocessor_execute() (0x5e02d3+10D)
	main/threadpool.c:1320 execute_tasks()
	main/taskprocessor.c:852 ast_taskprocessor_execute() (0x5e02d3+10D)
	main/threadpool.c:351 threadpool_execute()
	main/threadpool.c:1103 worker_active()
	main/threadpool.c:1024 worker_start()
	main/utils.c:1235 dummy_start()
	:0 start_thread()
	:0 __clone() (0x7fbb8bb68860+6D)
=== -------------------------------------------------------------------
=========================================================================

George Joseph

2016-Apr-28 13:11 UTC

head link

[asterisk-users] Upgrading 13.7 (external pjproject) to 13.9 (bundled pjproject)

On Thu, Apr 28, 2016 at 1:10 AM, Dmitriy Serov <serov.d.p at gmail.com>
wrote:
> Today was another attempt to upgrade to version 13.9 (git).
>
> 1. The result was https://issues.asterisk.org/jira/browse/ASTERISK-25970
>
> Had to temporarily block this contact and look forward to advice of how to
> fix it.
>
> 2. Also, an unpleasant surprise was the increase in CPU usage from 10-50%
> to 200-400% (4 cores). Stable CPU overusage with the same build options
> (DONT_OPTIMIZE, DEBUG_THREADS, BETTER_BACKTRACES, BUILD_NATIVE)
>

Does this also happen with an external pjproject?

>
> 3. After 20 minutes of this work the server has ceased to respond promptly
> to registrations. "core show locks" is attached. Suggestions
about solving
> the problem are very welcome.
>
> Dmitriy.
>
>
>
> --
> _____________________________________________________________________
> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
> New to Asterisk? Join us for a live introductory webinar every Thurs:
>                http://www.asterisk.org/hello
>
> asterisk-users mailing list
> To UNSUBSCRIBE or update options visit:
>    http://lists.digium.com/mailman/listinfo/asterisk-users
>


-- 
George Joseph
Digium, Inc. | Software Developer
445 Jan Davis Drive NW - Huntsville, AL 35806 - US
Check us out at: www.digium.com & www.asterisk.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.digium.com/pipermail/asterisk-users/attachments/20160428/6b224698/attachment.html>

asterisk users - Apr 2016 - Upgrading 13.7 (external pjproject) to 13.9 (bundled pjproject)

[asterisk-users] Upgrading 13.7 (external pjproject) to 13.9 (bundled pjproject)

[asterisk-users] Upgrading 13.7 (external pjproject) to 13.9 (bundled pjproject)