asterisk users - Apr 2007 - Monitor application inestability and high load

Hi,

I'm having high load, choppy sound and slow responsives with an asterisk
server (version 1.2.12.1) that make a peak of 90 channels (around 60 phones
calling at max, isn't necessary to reach this peak to get the problem). All
the traffic is SIP, with recording for every call. The server has:

Intel(R) Xeon(TM) CPU 3.20GHz (with HyperThreading disabled for inestability)
4G RAM
2 DD SCSI 150GB in RAID I via hardware.

The problems are detected in the high count of asterisk processes and sh
wrappers to soxmix which could be as old as 1 hour in the server without a
reason to stay idle, but for some unknow reason this "sh" don't
die fast. This is when the dialplan calls Monitor obviously. I already tried to
switch to MixMonitor but yesterday users reported that in some calls the
recording isn't complete. Which is similar to a bug that is mentioned in
mantis but for versions prior to 1.2.7. The asterisk logs don't show any
particular message in verbose level 3. Apart from the recording, I have a high
use of Manager and the mysql is used for some bussines logic but I think that
nothing to high load, indeed mysql never is the most important part in
processor, memmory and disk access statistics.

In http://linuxuanl.org/eald/random/ps.txt there is an example (no very
espectacular but is more or less what happens) of the status of the computer
with problems. I see that there are many sh without their soxmix or rm; Usually
this is done faster indeed I changed soxmix to a script that only copy the files
in an attempt to low the load of the server.

Any knows a solution to this problem? or has an explanation for it?

Thanks,
Edgar Luna
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.digium.com/pipermail/asterisk-users/attachments/20070418/8a667c0d/attachment.htm

Edgar A. Luna Diaz wrote:
>
> I'm having high load, choppy sound and slow responsives with an 
> asterisk server (version 1.2.12.1) that make a peak of 90 channels 
> (around 60 phones calling at max, isn't necessary to reach this peak 
> to get the problem). All the traffic is SIP, with recording for every 
> call.
>
What codecs are you using?  Are you performing any transcoding?  What 
format are you using for the recordings?
>
> The problems are detected in the high count of asterisk processes and 
> sh wrappers to soxmix which could be as old as 1 hour in the server 
> without a reason to stay idle, but for some unknow reason this
"sh"
> don't die fast. This is when the dialplan calls Monitor obviously. I 
> already tried to switch to MixMonitor but yesterday users reported 
> that in some calls the recording isn't complete. Which is similar to a 
> bug that is mentioned in mantis but for versions prior to 1.2.7. The 
> asterisk logs don't show any particular message in verbose level 3. 
> Apart from the recording, I have a high use of Manager and the mysql 
> is used for some bussines logic but I think that nothing to high load, 
> indeed mysql never is the most important part in processor, memmory 
> and disk access statistics.
>
What do your disk access statistics look like?  In my experience, your 
call quality will begin to seriously deteriorate as you approach 60 
simultaneous recordings.  This is because the Monitor() code places a 
disk write in the code path that bridges channels.  If it's possible to 
disable recordings for a while, you can confirm whether or not this is 
the source of your problem.
>
> Any knows a solution to this problem? or has an explanation for it?
>
In general, you should try to offload as many processes from the 
Asterisk server as possible.  MySQL is a good candidate for that, but 
the big one is soxmix.  Transcoding audio files is CPU intensive and I 
wouldn't be surprised if it impacted your call quality.

I've overcome the Monitor() problem by writing to a RAM disk.  At the 
end of each call, the recordings are moved from the RAM disk to a remote 
machine via NFS where they are mixed and indexed for retrieval.  My 
documentation of this process is available here:

<http://thread.gmane.org/gmane.comp.telephony.pbx.asterisk.user/118497>

and here:

<http://lists.digium.com/pipermail/asterisk-users/2005-October/120930.html>

Note that we are overconfigured for RAM.  You could probably get away 
with a 2 GB RAM disk.  I've also made some refinements to the setup that 
increased reliability.  If you choose to go down this path, I'll be 
happy to discuss them with you.

Matthew Roth
InterMedia Marketing Solutions
Software Engineer and Systems Developer