We ultimately found this to be a voicemail issue. The voicemail is held in MYSQL as well (via ODBC). And we found when attempting to playback a customers voicemail unavail greeting is when the deadlock would occur (Immediately, every time. Throwing the same "task processors" errors, And making pjsip completely unresponsive). We had imported a number of greetings from a legacy asterisk system and the vast majority of them worked. When we deleted the row containing the customers unavail greeting (making asterisk revert to read the mailbox number) all issues went away. If we re-record the customers unavail greeting it works fine and the problem doesn't reoccur. This was one out of ~250 voicemails imported. Since then we've done a few more migrations and they've all gone smooth with the exception of the most recent one. ~50% of the imported greetings have caused asterisk to deadlock. We've been checking them now at time of migration. What I can't figure out is what it doesn't like about the greeting. It was on a previous asterisk system working fine. The row looks identical to a working one. The only thing I can guess is something about the blob for the recording goes wrong. It would be nice if asterisk handled that more gracefully. I post this mostly just for internet history. To hopefully help the next guy out who has this same issue. *Nick Olsen* Network Engineer Office: 321-408-5000 x103 Mobile: 321-794-0763 On Mon, Mar 2, 2020 at 8:29 PM Joshua C. Colp <jcolp at sangoma.com> wrote:> On Mon, Mar 2, 2020 at 4:24 PM Nick Olsen < > nick at floridavirtualsolutions.com> wrote: > >> Thanks for the info, Joshua. >> >> Does PJSIP handle database access the same way Chan_sip did? We had a >> number of boxes running chan_sip referencing the same mysql server without >> issue. >> >> We're going to attempt to get a backtrace on the next occurance. We're >> also going to run a local copy of the database on the same physical >> asterisk instance and have the system reference it. Just to "throw >> everything at the wall". >> > > It uses the same underlying API and layer. It can do more frequent > database access though due to queries and because PJSIP is multithreaded. > > -- > Joshua C. Colp > Asterisk Technical Lead > Sangoma Technologies > Check us out at www.sangoma.com and www.asterisk.org > -- > _____________________________________________________________________ > -- Bandwidth and Colocation Provided by http://www.api-digital.com -- > > Check out the new Asterisk community forum at: > https://community.asterisk.org/ > > New to Asterisk? Start here: > https://wiki.asterisk.org/wiki/display/AST/Getting+Started > > asterisk-users mailing list > To UNSUBSCRIBE or update options visit: > http://lists.digium.com/mailman/listinfo/asterisk-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.digium.com/pipermail/asterisk-users/attachments/20200401/3ab95a6d/attachment.html>
Hi All
This sounds just like a problem I have had and still investigating having
moved to 16.9 using chan_sip. I am still trying to repeat the problem it
looks from debug that the issue is either voicemail of call transfer but I
cant consistently repeat it.
Voicemail is using ODBC and I just imported the data from the old system
into the new database.
Nick - if you have any more info I would be grateful
TIA
Paddy
_____
From: asterisk-users [mailto:asterisk-users-bounces at lists.digium.com] On
Behalf Of Nick Olsen
Sent: 01 April 2020 18:54
To: Asterisk Users Mailing List - Non-Commercial Discussion
Subject: Re: [asterisk-users] PJSIP Lockup
We ultimately found this to be a voicemail issue. The voicemail is held in
MYSQL as well (via ODBC). And we found when attempting to playback a
customers voicemail unavail greeting is when the deadlock would occur
(Immediately, every time. Throwing the same "task processors" errors,
And
making pjsip completely unresponsive). We had imported a number of greetings
from a legacy asterisk system and the vast majority of them worked. When we
deleted the row containing the customers unavail greeting (making asterisk
revert to read the mailbox number) all issues went away. If we re-record the
customers unavail greeting it works fine and the problem doesn't reoccur.
This was one out of ~250 voicemails imported.
Since then we've done a few more migrations and they've all gone smooth
with
the exception of the most recent one. ~50% of the imported greetings have
caused asterisk to deadlock. We've been checking them now at time of
migration.
What I can't figure out is what it doesn't like about the greeting. It
was
on a previous asterisk system working fine. The row looks identical to a
working one. The only thing I can guess is something about the blob for the
recording goes wrong. It would be nice if asterisk handled that more
gracefully.
I post this mostly just for internet history. To hopefully help the next guy
out who has this same issue.
Nick Olsen
Network Engineer
Office: 321-408-5000 x103
Mobile: 321-794-0763
<http://dl.floridavirtualsolutions.com/emaillogo/logo.png>
On Mon, Mar 2, 2020 at 8:29 PM Joshua C. Colp <jcolp at sangoma.com>
wrote:
On Mon, Mar 2, 2020 at 4:24 PM Nick Olsen <nick at
floridavirtualsolutions.com>
wrote:
Thanks for the info, Joshua.
Does PJSIP handle database access the same way Chan_sip did? We had a number
of boxes running chan_sip referencing the same mysql server without issue.
We're going to attempt to get a backtrace on the next occurance. We're
also
going to run a local copy of the database on the same physical asterisk
instance and have the system reference it. Just to "throw everything at the
wall".
It uses the same underlying API and layer. It can do more frequent database
access though due to queries and because PJSIP is multithreaded.
--
Joshua C. Colp
Asterisk Technical Lead
Sangoma Technologies
Check us out at www.sangoma.com and www.asterisk.org
--
_____________________________________________________________________
-- Bandwidth and Colocation Provided by http://www.api-digital.com --
Check out the new Asterisk community forum at:
https://community.asterisk.org/
New to Asterisk? Start here:
https://wiki.asterisk.org/wiki/display/AST/Getting+Started
asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
http://lists.digium.com/mailman/listinfo/asterisk-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.digium.com/pipermail/asterisk-users/attachments/20200401/62da6104/attachment.html>
Paddy, It's pretty easy to spot from the CLI. A voicemail gets called. And the screen basically stops scrolling from there. Eventually you'll get the "Task processors exceeded 500 queued tasks" or something like that. And maybe channels attempting to hangup due to lack of RTP (If you have no-rtp timers configured). Once you find the problem mailbox, You can call it via any method and it'll deadlock every time as soon as it tries to play the mailboxes unavail greeting. I've never had it occur when there is no unavail greeting. Each case deleting the problem recording from the database fixes the issue, And subsequent recordings for the same mailbox have no issue. *Nick Olsen* Network Engineer Office: 321-408-5000 x103 Mobile: 321-794-0763 On Wed, Apr 1, 2020 at 9:04 PM Paddy Grice <paddy at wizaner.com> wrote:> Hi All > > This sounds just like a problem I have had and still investigating having > moved to 16.9 using chan_sip. I am still trying to repeat the problem it > looks from debug that the issue is either voicemail of call transfer but I > cant consistently repeat it. > > Voicemail is using ODBC and I just imported the data from the old system > into the new database. > > Nick - if you have any more info I would be grateful > > TIA > > Paddy > > ------------------------------ > *From:* asterisk-users [mailto:asterisk-users-bounces at lists.digium.com] *On > Behalf Of *Nick Olsen > *Sent:* 01 April 2020 18:54 > *To:* Asterisk Users Mailing List - Non-Commercial Discussion > *Subject:* Re: [asterisk-users] PJSIP Lockup > > We ultimately found this to be a voicemail issue. The voicemail is held in > MYSQL as well (via ODBC). And we found when attempting to playback a > customers voicemail unavail greeting is when the deadlock would occur > (Immediately, every time. Throwing the same "task processors" errors, And > making pjsip completely unresponsive). We had imported a number of > greetings from a legacy asterisk system and the vast majority of them > worked. When we deleted the row containing the customers unavail greeting > (making asterisk revert to read the mailbox number) all issues went away. > If we re-record the customers unavail greeting it works fine and the > problem doesn't reoccur. This was one out of ~250 voicemails imported. > > Since then we've done a few more migrations and they've all gone smooth > with the exception of the most recent one. ~50% of the imported greetings > have caused asterisk to deadlock. We've been checking them now at time of > migration. > > What I can't figure out is what it doesn't like about the greeting. It was > on a previous asterisk system working fine. The row looks identical to a > working one. The only thing I can guess is something about the blob for the > recording goes wrong. It would be nice if asterisk handled that more > gracefully. > > I post this mostly just for internet history. To hopefully help the next > guy out who has this same issue. > > *Nick Olsen* > Network Engineer > Office: 321-408-5000 x103 > Mobile: 321-794-0763 > > > > On Mon, Mar 2, 2020 at 8:29 PM Joshua C. Colp <jcolp at sangoma.com> wrote: > >> On Mon, Mar 2, 2020 at 4:24 PM Nick Olsen < >> nick at floridavirtualsolutions.com> wrote: >> >>> Thanks for the info, Joshua. >>> >>> Does PJSIP handle database access the same way Chan_sip did? We had a >>> number of boxes running chan_sip referencing the same mysql server without >>> issue. >>> >>> We're going to attempt to get a backtrace on the next occurance. We're >>> also going to run a local copy of the database on the same physical >>> asterisk instance and have the system reference it. Just to "throw >>> everything at the wall". >>> >> >> It uses the same underlying API and layer. It can do more frequent >> database access though due to queries and because PJSIP is multithreaded. >> >> -- >> Joshua C. Colp >> Asterisk Technical Lead >> Sangoma Technologies >> Check us out at www.sangoma.com and www.asterisk.org >> -- >> _____________________________________________________________________ >> -- Bandwidth and Colocation Provided by http://www.api-digital.com -- >> >> Check out the new Asterisk community forum at: >> https://community.asterisk.org/ >> >> New to Asterisk? Start here: >> https://wiki.asterisk.org/wiki/display/AST/Getting+Started >> >> asterisk-users mailing list >> To UNSUBSCRIBE or update options visit: >> http://lists.digium.com/mailman/listinfo/asterisk-users > > -- > _____________________________________________________________________ > -- Bandwidth and Colocation Provided by http://www.api-digital.com -- > > Check out the new Asterisk community forum at: > https://community.asterisk.org/ > > New to Asterisk? Start here: > https://wiki.asterisk.org/wiki/display/AST/Getting+Started > > asterisk-users mailing list > To UNSUBSCRIBE or update options visit: > http://lists.digium.com/mailman/listinfo/asterisk-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.digium.com/pipermail/asterisk-users/attachments/20200402/69a1b011/attachment.html>