asterisk users - May 2007 - Monitor application inestability and high load

Thanks for the answer Matthew.
> >
> > I'm having high load, choppy sound and slow responsives with an
> > asterisk server (version 1.2.12.1) that make a peak of 90 channels
> > (around 60 phones calling at max, isn't necessary to reach this
peak
> > to get the problem). All the traffic is SIP, with recording for
every
> > call.
> >
> What codecs are you using?  Are you performing any transcoding?  What
> format are you using for the recordings?
> >
>From sip show channels I use ulaw as codec for calls (both to sip phones
and to PBX).
I record with wav49 format in Monitor.
> > The problems are detected in the high count of asterisk processes
and
> > sh wrappers to soxmix which could be as old as 1 hour in the server
> > without a reason to stay idle, but for some unknow reason this
"sh"
> > don't die fast. This is when the dialplan calls Monitor obviously.
I
> > already tried to switch to MixMonitor but yesterday users reported
> > that in some calls the recording isn't complete. Which is similar
to
a
> > bug that is mentioned in mantis but for versions prior to 1.2.7. The
> > asterisk logs don't show any particular message in verbose level
3.
> > Apart from the recording, I have a high use of Manager and the mysql
> > is used for some bussines logic but I think that nothing to high
load,
> > indeed mysql never is the most important part in processor, memmory
> > and disk access statistics.
> >
> What do your disk access statistics look like?  In my experience, your
> call quality will begin to seriously deteriorate as you approach 60
> simultaneous recordings.  This is because the Monitor() code places a
> disk write in the code path that bridges channels.  If it's possible
to
> disable recordings for a while, you can confirm whether or not this is
> the source of your problem.
> >
The real problem was found. The configuration of this server had a
recording path as /var/spool/asterisk/monitor/ for every call, so the
size of monitor (the directory) keeps growing at 2000 files per day. Its
peek was around 36MB, just containing the addresses of the files, I just
changed the filename variable for Monitor to store in a path like
/var/spool/asterisk/monitor/2007/05/29/. With this change the maximum
size of a directory is 700k. To find this I used strace and saw an
incredible amount of a system call that get the next entry of a
directory (don't remember the name, sorry). Maybe is common sense don't
do what we did here but now I can say why is bad :).
That is the reason why sh (the rm command) was alive after soxmix had
ended.
> > Any knows a solution to this problem? or has an explanation for it?
> >
> In general, you should try to offload as many processes from the
> Asterisk server as possible.  MySQL is a good candidate for that, but
> the big one is soxmix.  Transcoding audio files is CPU intensive and I
> wouldn't be surprised if it impacted your call quality.
> 
> I've overcome the Monitor() problem by writing to a RAM disk.  At the
> end of each call, the recordings are moved from the RAM disk to a
remote
> machine via NFS where they are mixed and indexed for retrieval.  My
> documentation of this process is available here:
> 
>
<http://thread.gmane.org/gmane.comp.telephony.pbx.asterisk.user/118497>
> 
> and here:
> 
> <http://lists.digium.com/pipermail/asterisk-users/2005-
> October/120930.html>
> 
> Note that we are overconfigured for RAM.  You could probably get away
> with a 2 GB RAM disk.  I've also made some refinements to the setup
that
> increased reliability.  If you choose to go down this path, I'll be
> happy to discuss them with you.
> 
Yes for some previous problems I read your threads about it, indeed I
use it as reference for everything I try here. And for the RAM, I have a
600M of files at the end of the day, so I was thinking exactly in that
2GB to use as ramdisk.

I have your way to do it as an option, the difference here is that
people want more lines in one server, I actually have 2 servers doing
this job (exactly same configuration in both) so changing to a
storage/mixing backend in one and calls in others don't give us a real
advantage in that point (the connection to others sites makes impossible
to send more calls to that servers), but if the quality becomes an issue
I'll have this as an option.

Regards,
Edgar Luna

On Tue, 29 May 2007, Edgar A. Luna Diaz wrote:
> The real problem was found. The configuration of this server had a
> recording path as /var/spool/asterisk/monitor/ for every call, so the
> size of monitor (the directory) keeps growing at 2000 files per day. Its
> peek was around 36MB, just containing the addresses of the files, I just
> changed the filename variable for Monitor to store in a path like
> /var/spool/asterisk/monitor/2007/05/29/. With this change the maximum
> size of a directory is 700k.
That's a common problem with many filesystems - there is a limit to the 
number of files it's "sensible" to put in one directory - it's
really a
SysAdmin issue, but it's going to be easy for others to enter this trap.

It really does depend on the filesystem though - Linux ext3 is fair to 
middling at handling large numbers of files - with the dir_index option 
set, but without it, it's really bad. There are other filesystems which 
are better (and worse!) Ext3 has an option at create time - dir_index 
which you can set to make it use a more efficient way to store filenames. 
It's usually too late to change at production time unless you can take the 
server offline for a bit. (The defaults also depend on your Linux 
distribution too!)

Your system of a new directory per day is a solution similar to many 
others adopted by other systems though! (eg. email spools, home 
directorys, squid cache files, etc.)

Gordon