asterisk users - Sep 2010 - Asterisk 1.62.13 - CPU spikes every 10 minutes

Hi,

 

I've been getting regular CPU usage spikes(50%-80%), due to asterisk
(according to top).  I never noticed this on 1.4, and I have top running in
the background pretty much all the time. In between those spikes Asterisk
stays under 10% CPU usage (I have a transcoder card, which helps).

 

It's very regular, never any missed spike, or any spike in between the
regular spikes.  I don`t have cron job running every 10 minutes
(asterisk-related or not).

 

Because it's so regular, I don`t think it's anything in my dialplan.  If
it
was it would be more random. SIP peers are reregistering every 60 seconds,
so the spikes would be every minute if that was the problem.  There is no
"scheduled" attack from outside that I know of, network traffic
(calculated
from the switch port and the server) does NOT spike. Memory is very stable
at a low value.

 

Does anyone have an idea of where to start looking? Can Asterisk report
somehow what is causing this?

 

Mike

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.digium.com/pipermail/asterisk-users/attachments/20100924/0a638fc8/attachment.htm

_____  

From: asterisk-users-bounces at lists.digium.com
[mailto:asterisk-users-bounces at lists.digium.com] On Behalf Of Mike
Sent: Friday, September 24, 2010 2:41 PM
To: 'Asterisk Users Mailing List - Non-Commercial Discussion'
Subject: [asterisk-users] Asterisk 1.62.13 - CPU spikes every 10 minutes

 

Hi,

 

I've been getting regular CPU usage spikes(50%-80%), due to asterisk
(according to top).  I never noticed this on 1.4, and I have top running in
the background pretty much all the time. In between those spikes Asterisk
stays under 10% CPU usage (I have a transcoder card, which helps).

 

It's very regular, never any missed spike, or any spike in between the
regular spikes.  I don`t have cron job running every 10 minutes
(asterisk-related or not).

 

Because it's so regular, I don`t think it's anything in my dialplan.  If
it
was it would be more random. SIP peers are reregistering every 60 seconds,
so the spikes would be every minute if that was the problem.  There is no
"scheduled" attack from outside that I know of, network traffic
(calculated
from the switch port and the server) does NOT spike. Memory is very stable
at a low value.

 

Does anyone have an idea of where to start looking? Can Asterisk report
somehow what is causing this?

 

Mike

 

I would start with /var/log/asterisk/messages and /var/log/asterisk/full to
see what is (was) happening (if anything) on the console at these spike
times.

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.digium.com/pipermail/asterisk-users/attachments/20100924/d064fd2b/attachment.htm

sip / other registrations...

~
Andrew "lathama" Latham
lathama at gmail.com

* Learn more about OSS http://en.wikipedia.org/wiki/Open-source_software
* Learn more about Linux http://en.wikipedia.org/wiki/Linux
* Learn more about Tux http://en.wikipedia.org/wiki/Tux



On Fri, Sep 24, 2010 at 3:40 PM, Mike <list at net-wall.com>
wrote:
> Hi,
>
>
>
> I've been getting regular CPU usage spikes(50%-80%), due to asterisk
> (according to top).? I never noticed this on 1.4, and I have top running in
> the background pretty much all the time. In between those spikes Asterisk
> stays under 10% CPU usage (I have a transcoder card, which helps).
>
>
>
> It's very regular, never any missed spike, or any spike in between the
> regular spikes.? I don`t have cron job running every 10 minutes
> (asterisk-related or not).
>
>
>
> Because it's so regular, I don`t think it's anything in my
dialplan.? If it
> was it would be more random. SIP peers are reregistering every 60 seconds,
> so the spikes would be every minute if that was the problem.? There is no
> "scheduled" attack from outside that I know of, network traffic
(calculated
> from the switch port and the server) does NOT spike. Memory is very stable
> at a low value.
>
>
>
> Does anyone have an idea of where to start looking? Can Asterisk report
> somehow what is causing this?
>
>
>
> Mike
>
>
>
> --
> _____________________________________________________________________
> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
> New to Asterisk? Join us for a live introductory webinar every Thurs:
> ? ? ? ? ? ? ? http://www.asterisk.org/hello
>
> asterisk-users mailing list
> To UNSUBSCRIBE or update options visit:
> ? http://lists.digium.com/mailman/listinfo/asterisk-users
>

For what it's worth... I also observed an unusually high
level of CPU activity on my small (home-system) Asterisk
installation, shortly after I switched to 1.6.2.13.

The pattern in my case was (as best as I could tell) a
CPU load of 2% to 10%, pulsing upwards every few seconds.
I couldn't resolve it any more finely than that, at the
time I saw it.

This is a very small Asterisk install (just two or three
SIP extensions registered at the time, no calls in progress,
no significant activity noted on the net interfaces at
the time), running on an Atom N270 CPU system.  I am using
the "internal_timing = yes" option (pthreads timing rather
than dahdi).

I did a "core restart when convenient", Asterisk restarted,
and the CPU load dropped to negligible levels.  It has
remained low ever since (about 7 minutes of CPU used, over
the space of a couple of days).

I have a hunch that in 1.6.2.13, some resource isn't being
cleaned up properly after calls terminate or extensions
de-register, and that the system is spending a significant
amount of time "chasing its tail" navigating through these
obsolete resources.  Just a hunch, though... nothing to back
this up.

...which, after upgrading to Dahdi 2.4.0 seem to be gone.  Not sure if
2.3.0.1 was the culprit, or just the fact that sonething was stuck (stopping
dahdi took about 5 attempts because it was apparently in use, even if
Asterisk had been shutdown).

Hopefully this isn't just the calm before the storm and Dahdi will behave
under load.

Thanks for the help Shaun, I appreciate you helping point me in the right
direction. Hopefully this will help someone.

Mike
> -----Original Message-----
> From: Mike [mailto:list at net-wall.com]
> Sent: Tuesday, September 28, 2010 2:55
> To: 'Shaun Ruffell'
> Subject: RE: [asterisk-users] Asterisk 1.62.13 - CPU spikes every 10
> minutes
> 
> Actually, after further investigation I found that every 10 minutes I get
> massive irq usage spikes.  The CPU is busy servicing hardware interrupts.
> 
> I do have a 4 port PRI card and a TCE400B card,  but tha happens even when
> asterisk activity=0.
> 
> Mike
> 
> > -----Original Message-----
> > From: Shaun Ruffell [mailto:sruffell at digium.com]
> > Sent: Monday, September 27, 2010 15:58
> > To: Mike
> > Subject: Re: [asterisk-users] Asterisk 1.62.13 - CPU spikes every 10
> > minutes
> >
> > When the asterisk process spikes, do you know if there is a
> > corresponding spike in us, sy, ni, hi, or si time?
> >
> > On 09/27/2010 02:53 PM, Mike wrote:
> > > That`s exactly what I did, and exactly what didn't happen.
> > >
> > > Mike
> > >
> > >> -----Original Message-----
> > >> From: Shaun Ruffell [mailto:sruffell at digium.com]
> > >> Sent: Monday, September 27, 2010 15:48
> > >> To: Mike
> > >> Subject: Re: [asterisk-users] Asterisk 1.62.13 - CPU spikes
every 10
> > >> minutes
> > >>
> > >> Just something like "echo 1200 >
> > >> /proc/sys/net/ipv4/route/secret_interval" should have
been sufficient
> to
> > >> half the frequency of the CPU spikes.
> > >>
> > >>
> > >> On 09/27/2010 02:42 PM, Mike wrote:
> > >>> I just changed the file, did I need to reboot or do
anything for the
> > >> change
> > >>> to take effect?
> > >>>
> > >>> Mike
> > >>>
> > >>>> -----Original Message-----
> > >>>> From: Shaun Ruffell [mailto:sruffell at digium.com]
> > >>>> Sent: Monday, September 27, 2010 15:34
> > >>>> To: Mike
> > >>>> Subject: Re: [asterisk-users] Asterisk 1.62.13 - CPU
spikes every
10
> > >>>> minutes
> > >>>>
> > >>>> Hmm..ok, then I'm not sure I'll be able to
help.  It's just that I
> > know
> > >>>> I've seen issues similar to what you're
reporting based on the
route
> > >>>> table flushing pushing some si time into the asterisk
process (and
> it
> > >>>> runs at 10 minute intervals by default).
> > >>>>
> > >>>> So I'm not as familiar with the Asterisk code
base to know what
sort
> > of
> > >>>> periodic tasks may have changed between the versions
you reported.
> > >>>>
> > >>>> On 09/27/2010 02:23 PM, Mike wrote:
> > >>>>> That wasn't it.  I wonder what happens
precisely every 10 minutes
> in
> > >>>>> Asterisk (or that is accounted to Asterisk's
CPU usage).
> > >>>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> Shaun Ruffell
> > >>>> Digium, Inc. | Linux Kernel Developer
> > >>>> 445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
> > >>>> Check us out at: www.digium.com &
www.asterisk.org
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> Shaun Ruffell
> > >> Digium, Inc. | Linux Kernel Developer
> > >> 445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
> > >> Check us out at: www.digium.com & www.asterisk.org
> > >
> >
> >
> > --
> > Shaun Ruffell
> > Digium, Inc. | Linux Kernel Developer
> > 445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
> > Check us out at: www.digium.com & www.asterisk.org

(sorry for the series of email, I realize I seem to be having a discussion
with myself).

I was wrong, the "problem" isn't fixed.  Is having IRQ spikes
every 10
minutes (under no load at all) the norm with Dahdi hardware?

Mike


> -----Original Message-----
> From: Mike [mailto:list at net-wall.com]
> Sent: Tuesday, September 28, 2010 3:34
> To: 'Asterisk Users Mailing List - Non-Commercial Discussion'
> Subject: RE: [asterisk-users] Asterisk 1.62.13 - CPU spikes every 10
> minutes - FIXED...?
> 
> ...which, after upgrading to Dahdi 2.4.0 seem to be gone.  Not sure if
> 2.3.0.1 was the culprit, or just the fact that sonething was stuck
> (stopping dahdi took about 5 attempts because it was apparently in use,
> even if Asterisk had been shutdown).
> 
> Hopefully this isn't just the calm before the storm and Dahdi will
behave
> under load.
> 
> Thanks for the help Shaun, I appreciate you helping point me in the right
> direction. Hopefully this will help someone.
> 
> Mike
> 
> > -----Original Message-----
> > From: Mike [mailto:list at net-wall.com]
> > Sent: Tuesday, September 28, 2010 2:55
> > To: 'Shaun Ruffell'
> > Subject: RE: [asterisk-users] Asterisk 1.62.13 - CPU spikes every 10
> > minutes
> >
> > Actually, after further investigation I found that every 10 minutes I
get
> > massive irq usage spikes.  The CPU is busy servicing hardware
interrupts.
> >
> > I do have a 4 port PRI card and a TCE400B card,  but tha happens even
> when
> > asterisk activity=0.
> >
> > Mike
> >
> > > -----Original Message-----
> > > From: Shaun Ruffell [mailto:sruffell at digium.com]
> > > Sent: Monday, September 27, 2010 15:58
> > > To: Mike
> > > Subject: Re: [asterisk-users] Asterisk 1.62.13 - CPU spikes every
10
> > > minutes
> > >
> > > When the asterisk process spikes, do you know if there is a
> > > corresponding spike in us, sy, ni, hi, or si time?
> > >
> > > On 09/27/2010 02:53 PM, Mike wrote:
> > > > That`s exactly what I did, and exactly what didn't
happen.
> > > >
> > > > Mike
> > > >
> > > >> -----Original Message-----
> > > >> From: Shaun Ruffell [mailto:sruffell at digium.com]
> > > >> Sent: Monday, September 27, 2010 15:48
> > > >> To: Mike
> > > >> Subject: Re: [asterisk-users] Asterisk 1.62.13 - CPU
spikes every
10
> > > >> minutes
> > > >>
> > > >> Just something like "echo 1200 >
> > > >> /proc/sys/net/ipv4/route/secret_interval" should
have been
> sufficient
> > to
> > > >> half the frequency of the CPU spikes.
> > > >>
> > > >>
> > > >> On 09/27/2010 02:42 PM, Mike wrote:
> > > >>> I just changed the file, did I need to reboot or do
anything for
> the
> > > >> change
> > > >>> to take effect?
> > > >>>
> > > >>> Mike
> > > >>>
> > > >>>> -----Original Message-----
> > > >>>> From: Shaun Ruffell [mailto:sruffell at
digium.com]
> > > >>>> Sent: Monday, September 27, 2010 15:34
> > > >>>> To: Mike
> > > >>>> Subject: Re: [asterisk-users] Asterisk 1.62.13 -
CPU spikes every
> 10
> > > >>>> minutes
> > > >>>>
> > > >>>> Hmm..ok, then I'm not sure I'll be able
to help.  It's just that
I
> > > know
> > > >>>> I've seen issues similar to what you're
reporting based on the
> route
> > > >>>> table flushing pushing some si time into the
asterisk process
(and
> > it
> > > >>>> runs at 10 minute intervals by default).
> > > >>>>
> > > >>>> So I'm not as familiar with the Asterisk
code base to know what
> sort
> > > of
> > > >>>> periodic tasks may have changed between the
versions you
reported.
> > > >>>>
> > > >>>> On 09/27/2010 02:23 PM, Mike wrote:
> > > >>>>> That wasn't it.  I wonder what happens
precisely every 10
minutes
> > in
> > > >>>>> Asterisk (or that is accounted to
Asterisk's CPU usage).
> > > >>>>>
> > > >>>>
> > > >>>>
> > > >>>> --
> > > >>>> Shaun Ruffell
> > > >>>> Digium, Inc. | Linux Kernel Developer
> > > >>>> 445 Jan Davis Drive NW - Huntsville, AL 35806 -
USA
> > > >>>> Check us out at: www.digium.com &
www.asterisk.org
> > > >>>
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Shaun Ruffell
> > > >> Digium, Inc. | Linux Kernel Developer
> > > >> 445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
> > > >> Check us out at: www.digium.com & www.asterisk.org
> > > >
> > >
> > >
> > > --
> > > Shaun Ruffell
> > > Digium, Inc. | Linux Kernel Developer
> > > 445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
> > > Check us out at: www.digium.com & www.asterisk.org