Olle E. Johansson
2005-Oct-01 06:32 UTC
[Asterisk-Users] Updated presentation of Asterisk 1.2
Friends, I have updated my Asterisk 1.2 presentation with the latest information. It is still available in the same place as before: http://www.astricon.net/asterisk1-2/ Please continue to test the beta of Asterisk 1.2, available at ftp.digium.com. We need all the feedback we can get. If you are a developer and have some time for community work, please check in with the bug tracker and help us resolve the unresolved bugs. See you at Astricon in Anaheim! /Olle
AbdelRahman Tarzi
2005-Oct-01 14:02 UTC
[Asterisk-Users] Updated presentation of Asterisk 1.2
Please search for the word "collission" and replace with collision.. Damn! This 1.2 sure takes things to a different level. Good luck. ----- Original Message ----- From: "Olle E. Johansson" <oej@edvina.net> To: "Users Asterisk" <asterisk-users@lists.digium.com> Sent: Saturday, October 01, 2005 4:32 PM Subject: [Asterisk-Users] Updated presentation of Asterisk 1.2> Friends, > I have updated my Asterisk 1.2 presentation with the latest information. > It is still available in the same place as before: > > http://www.astricon.net/asterisk1-2/ > > Please continue to test the beta of Asterisk 1.2, available at > ftp.digium.com. We need all the feedback we can get. If you are a > developer and have some time for community work, please check in with > the bug tracker and help us resolve the unresolved bugs. > > See you at Astricon in Anaheim! > > /Olle > _______________________________________________ > --Bandwidth and Colocation sponsored by Easynews.com -- > > Asterisk-Users mailing list > Asterisk-Users@lists.digium.com > http://lists.digium.com/mailman/listinfo/asterisk-users > To UNSUBSCRIBE or update options visit: > http://lists.digium.com/mailman/listinfo/asterisk-users
Jean-Denis Girard
2005-Oct-01 18:46 UTC
[Asterisk-Users] Updated presentation of Asterisk 1.2
Olle E. Johansson a ?crit :> Friends, > I have updated my Asterisk 1.2 presentation with the latest information. > It is still available in the same place as before: > > http://www.astricon.net/asterisk1-2/ >Just wanted to thank you for making this available. -- Jean-Denis Girard SysNux Syst?mes Linux en Polyn?sie fran?aise http://www.sysnux.pf/ T?l: +689 483 527 / GSM: +689 797 527
Matt Roth
2005-Oct-03 14:54 UTC
[Asterisk-Users] UPDATE - 512 Calls w/ Dig Rec: NFS Setup and Benchmarks
List members,
My previous post "SUCCESS - 512 Simultaneous Calls with Digital
Recording" documents using a RAM disk to eliminate the I/O
bottleneck associated with digitally recording calls via the Monitor
application. By recording directly to a RAM disk I was able to maintain good
call quality on 512 simultaneous calls.
This post documents moving the calls from the RAM disk to a hard disk on a
remote machine via NFS. The setup is not resource intensive on the Asterisk
server and should not impact call quality. As always, I welcome suggestions for
improvement and the identification of errors and omissions.
------------------------------------------------------------
Asterisk Configuration
=====================
The two leg files of a call will be moved immediately after the call is complete
via the MONITOR_EXEC and MONITOR_EXEC_ARGS variables. MONITOR_EXEC is generally
used to replace soxmix as the application for mixing the raw leg files, but
I'm using it as a hook to move them to an NFS mounted drive specified by
MONITOR_EXEC_ARGS as follows:
-From the dialplan (extensions.conf):
exten => _XXXX,1,SetVar(MONITOR_EXEC=mvdr)
exten => _XXXX,2,SetVar(MONITOR_EXEC_ARGS=/digrec-nfs/)
exten => _XXXX,3,Monitor(pcm||m)
mvdr is a shell script sitting in /usr/sbin/:
#!/bin/bash
/bin/nice -n 19 mv $1 $4 &
/bin/nice -n 19 mv $2 $4 &
Digitally recording via Monitor can also be initiated from agent channels and
queues. The MONITOR_EXEC and MONITOR_EXEC_ARGS variables are still set from the
dialplan, but you must tell Asterisk to mix the files for them to be used. This
is accomplished as follows:
-Agents: The channels must be configured in agents.conf for recording:
recordagentcalls=yes ; The leg files are always joined
-Queues: Each queue must be configured in queues.conf for recording and joining
the leg files:
monitor-format=pcm
monitor-join=yes
Using this hook to trigger the moves of the leg files has two distinct
advantages. First, the leg files are removed from the RAM disk as soon as
possible, minimizing the amount of RAM needed to buffer the calls. Secondly, the
RAM disk is volatile storage so moving the leg files to stable storage as soon
as possible minimizes the number of digital recordings that will be lost in the
event of an Asterisk server crash.
NFS Configuration
================
A fast NFS connection is needed for two reasons. First, the size of the RAM disk
is limited by the amount of physical memory so we have to move data off it as
quickly as possible to avoid filling it. Secondly, minimizing the amount of time
needed to transmit the leg files prevents a large number of moves from building
up on the system. Too many background processes leads to resource consumption
which inhibits Asterisk's ability to maintain call quality.
To attain the needed speed, I chose asynchronous NFS (version 3) using UDP and
8K block sizes transmitted via a crossover Gigabit connection configured for
jumbo frames. The Asterisk server is the NFS client in order to minimize
resource consumption, and the Digital Recording server runs the NFS daemons.
I decided to use an asynchronous NFS transfer because it allows the NFS server
to reply to NFS client requests as soon as it has processed the request, without
waiting for the data to be written to disk. This yields better performance at
the cost of possible data corruption in the event of an NFS server crash.
UDP was chosen because it is a stateless protocol and will not cause the NFS
client to hang if the NFS server crashes in the middle of a packet transmission.
The Asterisk server (NFS client) uses a soft, interruptable mount to prevent
hanging if the Digital Recording server (NFS server) crashes, as well.
Jumbo frames are used to minimize the number of CPU interrupts and general
proccessing overhead for a given data transfer size. The 8K block sizes (plus
packet headers) fit into the 9000 byte MTU allowing for efficient transfers
between the NFS client and server without packet fragmentation.
nfsd is started at boot on the Digital Recording server (NFS server) in
runlevels 3, 4, and 5.
Note that throughout the configuration I am sacrificing data integrity at the
expense of speed and NFS client reliability. I feel this is an acceptable trade,
given the realtime nature of Asterisk and the criticality of speed to this
application.
This configuration involves hardware and software, so I'll review both. If
you would like full copies of any configuration files, please contact me off
list.
Hardware Profiles
-----------------
-Asterisk Server (NFS Client)
- Machine: Dell PowerEdge 6850
- CPU: Four Intel Xeon MP CPUs at 3.16 GHz - Hyperthreaded
- RAM: 20 GB (4 GB System / 16 GB RAM Disk)
- NIC: Intel Pro/1000 MT Dual Port Server Adapter
- Exp. Slot: PCI-X, 64 bit, 133 MHz
-Digital Recording Server (NFS Server)
- Machine: Dell PowerEdge 1850
- CPU: One Intel Xeon CPU 2.80 GHz - Hyperthreaded
- RAM: 2 GB
- NIC: Intel Pro/1000 MT Gigabit Ethernet Adapter - Onboard
-Hard Disks: Two 73 Gigabyte SCSI Drives Configured in a RAID 1 (Mirrored)
-Controller: PERC 4e/Si RAID Controller - Onboard
-Physical Connection
-Cat5e Crossover Cable
Software Profiles
-----------------
-Asterisk Server (NFS Client)
- OS: Fedora Core 3 - 2.6.12-1.1376_FC3smp Kernel
-RAM Disk FS: ext2
- NIC Driver: e1000
-Digital Recording Server (NFS Server)
- OS: Fedora Core 3 - 2.6.12-1.1376_FC3smp Kernel
-Hard Disk FS: ext3
- NIC Driver: e1000
RAM Disk Setup
--------------
-Asterisk Server (NFS Client)
-Edit the bootloader configuration file (/boot/grub/grub.conf in my case) to
pass the ramdisk_size parameter to the kernel:
kernel /vmlinuz-2.6.12-1.1376_FC3smp ro root=LABEL=/ quiet
ramdisk_size=16777216
-Append lines to the /etc/rc.local init script to format and mount the RAM disk
at boot time:
# Formats and mounts the RAM disk used to buffer digital recordings.
/sbin/mke2fs -q -b 1024 -m 0 /dev/ram0
/bin/mount /dev/ram0 /digrec
Subnet Setup
------------
-Asterisk Server (NFS Client)
-Edit the /etc/sysconfig/network-script/ifcfg-eth# file for the dedicated NIC
so that it is on a different subnet than the other NICS in the system. No
default gateway is necessary. The following values work on my network:
IPADDR=192.168.55.2
NETMASK=255.255.255.0
-Digital Recording Server (NFS Server)
-Edit the /etc/sysconfig/network-script/ifcfg-eth# file for the dedicated NIC
so that it is on a different subnet than the other NICS in the system. No
default gateway is necessary. The following values work on my network:
IPADDR=192.168.55.1
NETMASK=255.255.255.0
After rebooting both machines you can confirm the setup using the route command:
[root@asterisk ~]# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.55.0 * 255.255.255.0 U 0 0 0 eth3
192.168.51.0 * 255.255.255.0 U 0 0 0 eth2
XXX.XXX.X.X * 255.255.0.0 U 0 0 0 eth2
default XXX.XXX.XX.XX 0.0.0.0 UG 0 0 0 eth2
Note that the eth3 interface is only used for the 192.168.55.0 subnet. If you
see similar behavior on both of your machines, your subnet setup was successful.
Jumbo Frame Setup
-----------------
-Both Servers (NFS Client and NFS Server)
-Edit the /etc/sysconfig/network-scripts/ifcfg-eth# file for the dedicated NIC
so that it has a 9000 byte MTU:
MTU=9000
After rebooting both machines you can confirm the setup using the tracepath
command:
[root@asterisk ~]# tracepath 192.168.55.1
1: 192.168.55.2 (192.168.55.2) 0.147ms pmtu 9000
1: 192.168.55.1 (192.168.55.1) 1.852ms reached
Resume: pmtu 9000 hops 1 back 1
Note that the pmtu (path MTU) is 9000. If you see similar behavior on both of
your machines, your jumbo frame setup was successful.
Gigabit Ethernet Confirmation
-----------------------------
As long as both of the NICs you configured are Gigabit NICs they should
auto-negotiate a speed of 1000 Mbps. You can confirm this by using the ethtool
command on both machines:
[root@asterisk ~]# ethtool eth3
Settings for eth3:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: umbg
Wake-on: d
Current message level: 0x00000007 (7)
Link detected: yes
NFS Server Optimizations
------------------------
-Digital Recording Server (NFS Server)
-Edit the NFS init script (/etc/rc.d/init.d/nfs in my case) so that it starts
20 nfsd threads. This number is an estimate. Increase the number if netstat -s
reports UDP socket overflows, decrease it if uptime reports a rising load
average:
# Number of servers to be started by default
[ -z "$RPCNFSDCOUNT" ] && RPCNFSDCOUNT=20
-Edit the start section of the NFS init script (/etc/rc.d/init.d/nfs in my
case) to increase the socket input queue for nfsd. This is important for NFS
servers with heavy write loads, because each instance of nfsd uses this queue to
store write requests while processing them:
start)
# Save the original size of the socket input queue - MJR
rmem_default=`cat /proc/sys/net/core/rmem_default`
rmem_max=`cat /proc/sys/net/core/rmem_max`
# Increase the size of the socket input queue (32K per nfsd) - MJR
rmem_nfsd=`echo "32768*$RPCNFSDCOUNT" | bc -l`
echo $rmem_nfsd > /proc/sys/net/core/rmem_default
echo $rmem_nfsd > /proc/sys/net/core/rmem_max
# Start daemons.
...
# Return the socket input queue to its original size - MJR
echo $rmem_default > /proc/sys/net/core/rmem_default
echo $rmem_max > /proc/sys/net/core/rmem_max
;;
Export Options
--------------
-Digital Recording Server (NFS Server)
-Edit /etc/exports so that the desired directory is exported asynchronously for
reading and writing, without squashing root:
/var/Recorded 192.168.55.2(rw,async,no_root_squash)
Mount Options
-------------
-Asterisk Server (NFS Client)
-Edit /etc/fstab so that the exported directory is mounted with the appropriate
options for a fast connection (see NFS Configuration):
192.168.55.1:/var/Recorded /digrec-nfs nfs
rw,soft,intr,bg,nolock,rsize=8192,wsize=8192,nfsvers=3,udp 0 0
NFS and RAM Disk Confirmation
-----------------------------
After rebooting both machines, you can confirm the setups using the df command
on the Asterisk server (NFS client):
[root@asterisk ~]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda6 5162796 535948 4364592 11% /
/dev/sda2 124443 20067 97950 18% /boot
none 10286816 0 10286816 0% /dev/shm
/dev/sda5 5162796 1645400 3255140 34% /home
/dev/sda7 10317828 4500676 5293036 46% /usr
/dev/sda8 47524452 1687956 43422332 4% /var
/dev/ram0 16510000 3855 16506145 1% /digrec
192.168.55.1:/var/Recorded
47564616 1257512 43890936 3% /digrec-nfs
Note the last two entries. The RAM disk is shown as /dev/ram0 mounted on /digrec
and the NFS mount is shown as 192.168.55.1:/var/Recorded mounted on /digrec-nfs.
If you see similar behavior, your NFS and RAM disk setups were successful.
Further Optimization Options
----------------------------
The disks of my NFS server are configured as a RAID 1 (mirrored). This offers
redundancy at the expense of slower writes, since each write is made to both
disks. Configuring the disks as a RAID 0 (striped) would yield better NFS
performance by decreasing write times if disk I/O on the NFS server proves to be
a bottleneck.
The e1000 drivers used by my Intel NICs can be configured via command line
parameters set in /etc/modprobe.conf. By increasing the number of RxDescriptors
and TxDescriptors the drivers could buffer more incoming packets and queue more
transmits, respectively. As per Intel's driver guide, the default settings
are generally the recommended settings.
NFS Benchmarking
===============
My first benchmarks were performed by creating files of zeroed bytes from
/dev/zero across the NFS mount. The commands to do this for each set of
benchmarks follows:
2 GB - time dd if=/dev/zero of=/digrec-nfs/testfile bs=16k count=131072
1 GB - time dd if=/dev/zero of=/digrec-nfs/testfile bs=16k count=65536
256 MB - time dd if=/dev/zero of=/digrec-nfs/testfile bs=16k count=16384
30 MB - time dd if=/dev/zero of=/digrec-nfs/testfile bs=16k count=1920
3 MB - time dd if=/dev/zero of=/digrec-nfs/testfile bs=16k count=192
For reference, a 3 MB file is roughly 6 minutes of PCM audio and a 30 MB file is
roughly 1 hour of PCM audio. These sizes represent one leg of an average call
and one leg of a long call for my application. The results of testing on these
files are of the most interest to me, and the larger files are used simply to
stress test the NFS configuration under heavier loads than should ever be seen.
I used sar on the Asterisk server (NFS client) to monitor CPU usage and iostat
on the Digital Recording server (NFS server) to monitor CPU and disk usage. The
NFS mount was unmounted and remounted after each test to clear any caches.
Without further ado, here are the numbers:
Client CPU Server CPU Server Max. Blocks
Filesize Runtime Mbps Min. Idle% Min. Idle% Written / Sec.
-------- --------- ---- ---------- ---------- ------------------
2 GB 0m54.756s 299 84.00 03.50 134768.00
2 GB 0m45.843s 357 84.65 06.44 121317.65
2 GB 0m48.347s 339 77.13 07.04 126800.00
1 GB 0m23.355s 351 87.30 07.46 119041.58
1 GB 0m22.110s 371 87.41 07.46 114035.64
1 GB 0m24.061s 340 83.41 08.42 138233.66
256 MB 0m03.146s 651 90.91 40.50 75896.00
256 MB 0m03.150s 650 92.54 41.00 7045.65 *
256 MB 0m03.145s 651 93.40 41.79 106488.00
30 MB 0m00.374s 642 97.38 80.50 6579.06 *
30 MB 0m00.373s 643 97.62 83.00 60966.34
30 MB 0m00.373s 643 98.13 83.50 5328.15 *
3 MB 0m00.041s 585 99.62 95.17 4902.11
3 MB 0m00.041s 585 99.75 95.48 4584.44
3 MB 0m00.041s 585 99.63 95.60 4456.03
* These numbers look a little off. I think I stopped iostat before the NFS
server's write buffers had been flushed to disk. Ooops!
My second benchmarks were performed by creating 120 files, each of which was 3
MB in size (360 MB total). They were named in pairs to simulate sixty pairs of
leg files, each containing 6 minutes of PCM audio. The goal of this test was to
measure the stress on both machines when sixty average calls end and are
transferred using the mvdr script over a given interval. The first set of tests
had a 1 second sleep between each call termination (typical call volume), the
second set had a 0.1 second sleep (high call volume), and the third set of tests
had no sleep (unrealistically high call volume/stress test). The script used to
conduct these benchmarks had the following format:
#!/bin/bash
mvdr 001a.txt 001b.txt dummy /digrec-nfs/ &
sleep 1 # 0.1 # 0.0
mvdr 002a.txt 002b.txt dummy /digrec-nfs/ &
sleep 1 # 0.1 # 0.0
mvdr 003a.txt 003b.txt dummy /digrec-nfs/ &
sleep 1 # 0.1 # 0.0
...
The approximate run times that I reported are just that. On both the NFS client
and the NFS server these times were approximated by reviewing the sar and iostat
logs and noting when the CPU began to be utilized and when it returned to idle.
The maximum number of simultaneous processes on the NFS client (Asterisk server)
is also an approximation. I simply ran ps repeatedly during the tests and
counted how many instances of the mvdr script were running.
It is important to note that the leg files were being transferred from the RAM
disk of the Asterisk server (NFS client) to the hard disk of the Digital
Recording server (NFS server) in order to simulate the actual scenario of the
production environment. Once again, the NFS mount was unmounted and remounted
after each test to clear any caches.
Client Client Client Server Server Server
Total Sleep Approx. CPU Min. Max. Sim. Approx. CPU Min. Max.
Blocks
Filesize Interval Run Time Idle% Processes Run Time Idle% Written /
Sec.
-------- -------- -------- -------- --------- -------- --------
--------------
360 MB 1.0s 60s 97.63 2 60s 55.28 61624.00
360 MB 1.0s 60s 92.39 0 60s 75.00 62865.31
360 MB 1.0s 60s 95.13 2 60s 67.33 62214.14
360 MB 0.1s 07s 90.75 2 16s 18.91 97615.69
360 MB 0.1s 07s 90.51 2 15s 24.88 110518.81
360 MB 0.1s 07s 90.75 8 14s 23.38 95264.00
360 MB 0.0s 05s 75.00 95 14s 20.10 93259.41
360 MB 0.0s 05s 80.75 103 14s 24.50 106352.00
360 MB 0.0s 05s 81.40 99 14s 21.00 114091.09
If you would like a zip file containing the raw test data or copies of my test
scripts and files, please contact me off list.
Conclusions
-----------
The speed of the NFS connection is more than adequate for this application and
at realistic call volume levels the added strain on the Asterisk server should
not effect call quality. The bottleneck looks to be CPU and possibly disk I/O on
the NFS server. This is not unexpected, since NFS uses code embedded in the
kernel which must be executed on the host CPU and large amounts of data are
being transferred across the connection in small periods of time.
I will monitor memory usage (to see how large the RAM disk actually needs to be)
and CPU (for stress) on the production system once it is up and report my
findings to the list.
Future Plans and Unresolved Issues
=================================
I wrote Windows software for another project that mixes leg files, indexes them
by call time, and archives them after a given period of time. I plan to port
that code to a set of shell scripts that will be run on the Digital Recording
server out of cron. If anyone knows of an existing project that has accomplished
this already, please let me know.
Before writing these scripts, I have two questions that need answered:
1) How can I tell when a file is complete on the NFS server?
2) What will happen on the NFS client if the NFS server crashes (I expect the
leg files to be written to the local mount point until the mount is
reesablished)?
If you can answer either of these questions, I would greatly appreciate your
contribution.
NFS Test Tools
=============
df
ethtool
ifconfig
lspci
netstat
nfsstat
showmount
tracepath
uptime
/proc/net/rpc/nfsd
Sources
======
Linux Administration Handbook
by Evi Nemeth, Garth Snyder, Trent R. Hein
Chapter 17 - The Network File System
http://www.amazon.com/exec/obidos/tg/detail/-/0130084662/103-5601970-3037440?v=glance
Linux NFS-HOWTO
by Tavis Barr, Nicolai Langfeldt, Seth Vidal, Tom McNeal
Chapter 5 - Optimizing NFS Performance
http://nfs.sourceforge.net/nfs-howto/
Gigabit Ethernet Jumbo Frames and Why You Should Care
by Phil Dykstra
http://64.233.161.104/search?q=cache:_XJNulIGbj0J:sd.wareonearth.com/~phil/jumbo.html+%22gigabit+ethernet+jumbo+frames%22+%22and+why+you+should+care%22&hl=en
Linux* Base Driver for the Intel? PRO/1000 Family of Adapters
by Intel Corporation
http://www.intel.com/support/network/sb/cs-009209.htm
Linux Ramdisk mini-HOWTO
by Van Emery
http://www.vanemery.com/Linux/Ramdisk/ramdisk.html
------------------------------------------------------------
Thank you for taking the time to read this post. I will try create a wiki page
from it (with all of your additions, of course) when time permits. If anyone
would like to perform this task feel free to do so, but please contact me first.
I want to avoid multiple pages with the same content being created.
Yours truly,
Matthew Roth
InterMedia Marketing Solutions
Software Engineer and Systems Developer
Dinesh Nair
2005-Oct-03 21:08 UTC
[Asterisk-Users] UPDATE - 512 Calls w/ Dig Rec: NFS Setup and Benchmarks
On 10/04/05 05:54 Matt Roth said the following:> This post documents moving the calls from the RAM disk to a hard disk on > a remote machine via NFS. The setup is not resource intensive on the > Asterisk server and should not impact call quality. As always, I welcome > suggestions for improvement and the identification of errors and omissions.this is very helpful, matt. thanx. -- Regards, /\_/\ "All dogs go to heaven." dinesh@alphaque.com (0 0) http://www.alphaque.com/ +==========================----oOO--(_)--OOo----==========================+ | for a in past present future; do | | for b in clients employers associates relatives neighbours pets; do | | echo "The opinions here in no way reflect the opinions of my $a $b." | | done; done | +=========================================================================+
Matt Florell
2005-Oct-04 07:49 UTC
[Asterisk-Users] UPDATE - 512 Calls w/ Dig Rec: NFS Setup and Benchmarks
On 10/3/05, Matt Roth <mroth@imminc.com> wrote:> Before writing these scripts, I have two questions that need answered: > > 1) How can I tell when a file is complete on the NFS server?We do something similar with a perl script, it grabs the file sizes of the 2 legs of the recording, then waits 5 seconds and checks the filesizes again, and if the recording is finished the files should not have grown in size over 5 seconds so it mixes the files together. This of course would not work if you had something like recording-pausing enabled on your server though. MATT--- -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.digium.com/pipermail/asterisk-users/attachments/20051004/3563bcd3/attachment.htm
Matt Roth
2005-Oct-04 17:19 UTC
[Asterisk-Users] UPDATE - 512 Calls w/ Dig Rec: NFS Setup and Benchmarks
Matt,> Well, almost all of the reports, database schema and other code is all > fully GPL'd and available on the project site for download: > http://astguiclient.sf.netI guess I should've realized it was open source. Thanks for the link, we'll download everything and take a look for ourselves.> We do not use Asterisk Queues or Agents because we found them limiting > in terms of functionality and scalability as well as not being as open > to call manipulation as the system we built is. Because of this we > haven't really used the Queue stat analysis tools out there any more > than to look at the kind of stats they generate.I figured as much, but with your experience I didn't think it would hurt to ask. Your statement concerning scalability being a problem with Asterisk Queues and Agents concerns me, because we are planning on using both in our large scale (250-500 concurrent calls) call center setup. Could you expound on it? Are they resource intensive at large numbers of calls, does the code have hard limits, etc? We've dealt with the scalability issue of the Monitor application and thought that would be the last such hurdle. The Asterisk source code looks to be very well written and I was hoping that the other applications that aren't bound to disk I/O do not introduce other bottlenecks. Are there other Asterisk applications that present serious scaling issues? We were hoping that our hardware (four 3.16 GHz Intel Xeon processors and 4 GB of RAM) would help us overcome most of them.> Our reporting is entirely based on database tables where we log > everything from asterisk calls to agent actions, that way we can > easily generate real-time stats based on whatever we want.I believe that will be the core of our architecture as well. Capture EVERYTHING in a database and derive all of the reports from it. I'm sure your database schema will be quite helpful in this endeavor.> The ACQS is still viable because it is more of a streamlined interface > that is easier on the Asterisk server than some of the other proxies > being developed, and it can be tightly integrated into how our other > software like VICIDIAL works. It is also fully backward compatible > with all versions of Asterisk from 1.2beta1 all the way back to > CVS_2003-10-04Anything that reduces the load on our Asterisk server is a Very Good Thing. We'll take a look at building ACQS into our architecture.> If you have the time and a spare machine I would suggest installing > astGUIclient/VICIDIAL on a machine. That would give you the best idea > of how everything works inside of it.I'll probably do that when I get a break in my schedule.> The guys from GnuDialer also have been going through a lot of the > figuring-out of how all of this works as well. They started earlier > this year building their own GPL'd asterisk dialer and they are > already on their third major code rebuilding because of all of the > things that they have learned.Working on Asterisk is definitely a learning experience.> Could you tell me a little more about what it is exactly that you are > trying to build?Sure. We are developing a three server system to handle inbound calling in a call center environment: 1) Asterisk Server 2) Digital Recording Server 3) Reporting Server The Asterisk Server itself performs no codec translations and no DSP. It will be connected to a Cisco AS5400HPX Universal Gateway that terminates the Ts and handles all TDM to VoIP (SIP) processing. As you know it's a pretty beefy box and we've tried to reserve as many resources on it as possible for running Asterisk and its applications. We won't be doing call conferencing on it, but we are planning on having multiple queues with music on hold (our music files are converted to raw files and do not require any decoding) and queue announcements. The majority of calls will be digitally recorded as well, but this involves no transcoding and no writes to the local disk. Some calls will also involve transfers to outsides lines, three way calling, and unattended monitoring via the app_chanspy.so module. We are looking to scale to 500 agents spread across multiple queues, but I believe our initial numbers will be closer to 200 agents. Currently, we are running a beta of Asterisk Business Edition with a license limit of 1000 simultaneous calls. We are using Asterisk's standard queues, but would like to move to realtime/dynamic queues in the future. The Digital Recording Server receives the leg files at the end of each call and is responsible for mixing, indexing, and eventually archiving them. The digital recordings will be available for downloading via FTP from this server as well. The Reporting Server is the biggest unknown at this point. Currently, it has a mySQL database that receives call detail records from Asterisk via the cdr_addon_mysql.so module. AstManProxy also runs on this box and may be used to add additional data to the mySQL database. Any real-time or next-day reporting will be created from this database, so that Asterisk itself is not affected by the reporting process. Please let me know if you would like any more specific details. I would be happy to supply them to you. Thanks, Matthew Roth InterMedia Marketing Solutions Software Engineer and Systems Developer
Matt Roth
2005-Oct-05 08:21 UTC
[Asterisk-Users] Agent/Queue Scalability (Formerly: UPDATE - 512 Calls...)
List users, Please review this exchange between myself and Matt Florell. I am looking for ANY data concerning Asterisk servers running standard Agents and Queues. Hardware configurations, software configurations, Asterisk configurations (especially the number of agents and queues), and descriptions of identified bottlenecks would be ideal. For perspective, we have scaled a singler Asterisk server (4 x 3.16 GHz Xeon processors, 4 GB RAM) to 256 simultaneous calls with digital recording at 70% idle and to 512 simultaneous calls with digital recording at 20% idle. I would like to know what to expect when we add agent channels to handle these calls directed to them via queues in a standard inbound call center environment. I would GREATLY appreciate any user experiences or knowledge you can share. =========================================================== Matt, Thank you so much for all of your help. I hope I can reciprocate in the future. ===========================================================> On 10/4/05, *Matt Roth* <mroth@imminc.com <mailto:mroth@imminc.com>> > wrote: > > > We do not use Asterisk Queues or Agents because we found them > limiting > > in terms of functionality and scalability as well as not being > as open > > to call manipulation as the system we built is. Because of this we > > haven't really used the Queue stat analysis tools out there any more > > than to look at the kind of stats they generate. > > I figured as much, but with your experience I didn't think it > would hurt > to ask. Your statement concerning scalability being a problem with > Asterisk Queues and Agents concerns me, because we are planning on > using > both in our large scale (250-500 concurrent calls) call center setup. > Could you expound on it? Are they resource intensive at large > numbers > of calls, does the code have hard limits, etc? We've dealt with the > scalability issue of the Monitor application and thought that would be > the last such hurdle. The Asterisk source code looks to be very well > written and I was hoping that the other applications that aren't bound > to disk I/O do not introduce other bottlenecks. > > > I don't know of any installation that has over 100 agents on a single > Asterisk server. You may want to contact the GnuDialer guys, because > they thought that they would be able to take a quad Xeon server and > put 150 agents using agent/queue on it but they never were able to > reliably get over 50 agents on that server. the agent/queue functions > on Asterisk really are much more CPU-intensive than just a standard > SIP stream. In our initial test 2 years ago the agent/queue load was > actually not much less than the meetme architecture that we ended up > using and with meetme you have a lot more control and functionality. > Your best bet may be to have several agent servers that get their > calls sent to them from your main server through IAX2 channels or > something like that.It may be wishful thinking, but 50 seemed to be the magic number associated with digital recording scalability. We have contacted the GNU Dialer team but if they do not respond, is anyone aware if they were trying to record the agent channels? If this is strictly an Agents/Queues issue, does anyone know where the CPU spikes occur? For example, will we see significant resource consumption by simply logging in a large number of agents to the Asterisk server or does the load occur as calls are routed through the queues? We'd like to avoid a server farm if possible.> Are there other Asterisk applications that present serious scaling > issues? We were hoping that our hardware (four 3.16 GHz Intel Xeon > processors and 4 GB of RAM) would help us overcome most of them. > > > agent/queue has a lot of functionality (you can see that in the code) > and it is not exactly designed for a low memory/CPU footprint. Other > than that, if you want to be as optimized as possible, don't use AGI > scripts at all and deactivate every Asterisk module you can.Done and done. Are there any metrics concerning Agents/Queues and the required memory/CPU for varying numbers of each? Do they scale linearly? It seems that a large problem with scaling Asterisk is that numbers such as these are hard to come by.> > Could you tell me a little more about what it is exactly that > you are > > trying to build? > > Sure. We are developing a three server system to handle inbound > calling > in a call center environment: > > 1) Asterisk Server > 2) Digital Recording Server > 3) Reporting Server > > The Asterisk Server itself performs no codec translations and no DSP. > It will be connected to a Cisco AS5400HPX Universal Gateway that > terminates the Ts and handles all TDM to VoIP (SIP) > processing. As you > know it's a pretty beefy box and we've tried to reserve as many > resources on it as possible for running Asterisk and its applications. > We won't be doing call conferencing on it, but we are planning on > having > multiple queues with music on hold (our music files are converted > to raw > files and do not require any decoding) and queue announcements. The > majority of calls will be digitally recorded as well, but this > involves > no transcoding and no writes to the local disk. Some calls will also > involve transfers to outsides lines, three way calling, and unattended > monitoring via the app_chanspy.so module. We are looking to scale to > 500 agents spread across multiple queues, but I believe our initial > numbers will be closer to 200 agents. > > Currently, we are running a beta of Asterisk Business Edition with a > license limit of 1000 simultaneous calls. We are using Asterisk's > standard queues, but would like to move to realtime/dynamic queues in > the future. > > The Digital Recording Server receives the leg files at the end of each > call and is responsible for mixing, indexing, and eventually > archiving > them. The digital recordings will be available for downloading > via FTP > from this server as well. > > The Reporting Server is the biggest unknown at this point. Currently, > it has a mySQL database that receives call detail records from > Asterisk > via the cdr_addon_mysql.so module. AstManProxy also runs on this box > and may be used to add additional data to the mySQL database. Any > real-time or next-day reporting will be created from this database, so > that Asterisk itself is not affected by the reporting process. > > Please let me know if you would like any more specific details. I > would > be happy to supply them to you. > > > > In VICIDIAL we really haven't optimized it for inbound centers of your > size, but we are planning on implementing a way of having a single > large inbound queue that would allow agents from any server to take > calls from it.Good luck with your project!> You certainly have your work cut out for you, lwt me know how it goes.Remember when people said computers would make our lives easier? = ) Sincerely, Matthew Roth InterMedia Marketing Solutions Software Engineer and Systems Developer