Dan Cropp
2021-Sep-14 14:07 UTC
[asterisk-users] Large system seeing single CPU core spiking
I am working with a very large customer running Asterisk with PJSIP. Systems total channels have been over 2500 (which includes hundreds of local channels and ConfBridges) when the issues occur. It's running on a Hyper-V VM with 12 CPU cores. Things work fine most of the time. They periodically see 10-30 minute periods where audio starts sounding like jitter buffer type issues. Can literally have someone spelling their name and a ConfBridge recording of it shows the audio is missing a letter or two. The odd part is another system (not running Asterisk) was handling these calls previously. The overall network has plenty of bandwidth (as evidenced by another system able to handle the call volume) One area that has perplexed us is when using htop, we see a single CPU core will spike to 100%. Which core does keep changing. Asterisk is definitely the process using the vast majority of the CPU cycles. We recently found a setting on Hyper-V networking SR-IOV which improved things. Prior to changing this setting, we were seeing SIP OPTIONS packets/responses would occasionally take more than 3 seconds causing devices to drop and come back online. We have configured a similar system running at Amazon handling far more traffic and can't get the single CPU core to spike. Only small static pops during the calls. The sheer scale of the system is making it hard to diagnose the problem. Any thoughts on how to diagnose what is causing the single CPU core to spike? Any thoughts on how to diagnose the problem? Any other thoughts/comments? Dan This email is intended only for the use of the party to which it is addressed and may contain information that is privileged, confidential, or protected by law. If you are not the intended recipient you are hereby notified that any dissemination, copying or distribution of this email or its contents is strictly prohibited. If you have received this message in error, please notify us immediately by replying to the message and deleting it from your computer. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.digium.com/pipermail/asterisk-users/attachments/20210914/57ed321d/attachment.html>
George Joseph
2021-Sep-14 14:39 UTC
[asterisk-users] Large system seeing single CPU core spiking
On Tue, Sep 14, 2021 at 8:07 AM Dan Cropp <dan at amtelco.com> wrote:> I am working with a very large customer running Asterisk with PJSIP. > Systems total channels have been over 2500 (which includes hundreds of > local channels and ConfBridges) when the issues occur. > > It’s running on a Hyper-V VM with 12 CPU cores. > > Things work fine most of the time. > > > > They periodically see 10-30 minute periods where audio starts sounding > like jitter buffer type issues. Can literally have someone spelling their > name and a ConfBridge recording of it shows the audio is missing a letter > or two. > > The odd part is another system (not running Asterisk) was handling these > calls previously. The overall network has plenty of bandwidth (as > evidenced by another system able to handle the call volume) > > > > One area that has perplexed us is when using htop, we see a single CPU > core will spike to 100%. Which core does keep changing. > > > > Asterisk is definitely the process using the vast majority of the CPU > cycles. > > > > We recently found a setting on Hyper-V networking SR-IOV which improved > things. Prior to changing this setting, we were seeing SIP OPTIONS > packets/responses would occasionally take more than 3 seconds causing > devices to drop and come back online. > > > > We have configured a similar system running at Amazon handling far more > traffic and can’t get the single CPU core to spike. Only small static pops > during the calls. > > > > The sheer scale of the system is making it hard to diagnose the problem. > > > > Any thoughts on how to diagnose what is causing the single CPU core to > spike? > > Any thoughts on how to diagnose the problem? > > Any other thoughts/comments? >The first thing I'd do is see where the CPU is spending time: userspace, system, nice, wait, etc. Is it actually the asterisk process consuming the CPU? Is Asterisk running with local file-based configs, local database, remote database, etc? If call quality is really bad already and your customer agrees, you could try the following the next time it happens... 1. Run "top -p `pidof asterisk` -n 1 -H -b" to get a list of all of Asterisk's threads and their CPU utilization. 2. Run ast_coredumper with the --RUNNING option. This will pause Asterisk while the dump is being generated! 3. See if you can correlate the high cpu thread IDs from the top output to the threads listed in the coredumper's -brief.txt file. That _may_ give you an idea of where to look.> > > Dan > > This email is intended only for the use of the party to which it is > addressed and may contain information that is privileged, confidential, or > protected by law. If you are not the intended recipient you are hereby > notified that any dissemination, copying or distribution of this email or > its contents is strictly prohibited. If you have received this message in > error, please notify us immediately by replying to the message and deleting > it from your computer. > -- > _____________________________________________________________________ > -- Bandwidth and Colocation Provided by http://www.api-digital.com -- > > Check out the new Asterisk community forum at: > https://community.asterisk.org/ > > New to Asterisk? Start here: > https://wiki.asterisk.org/wiki/display/AST/Getting+Started > > asterisk-users mailing list > To UNSUBSCRIBE or update options visit: > http://lists.digium.com/mailman/listinfo/asterisk-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.digium.com/pipermail/asterisk-users/attachments/20210914/44ddbc8a/attachment.html>