asterisk users - Sep 2005 - SUCCESS - 512 Simultaneous Calls with Digital Recording

List users,

Over the last few days we have been working with MCI's development lab 
to test our Asterisk setup.  We were using a piece of hardware called an 
Abacus 5000 that is capable of creating and terminating thousands of SIP 
calls.  Initially, we could not get past 64 simultaneous digitally 
recorded calls without having call quality issues including dropped 
calls.  We identified an I/O bottleneck and rectified it by digitally 
recording to a RAM disk.  Using this method, we were able to digitally 
record 512 simultaneous SIP-to-SIP calls with 100% call completion.

Our plan is to use the MONITOR_EXEC hook to call a custom program that 
will copy files to the hard disk at call completion.  This should be no 
problem when calling Monitor directly from the dialplan, but I need to 
know if there are any complications when digitally recording from 
app_queue or chan_agent.  If anyone has experience with using 
MONITOR_EXEC and MONITOR_EXEC_ARGS with these applications, your input 
would be greatly appreciated.  Digital recording seems to be the 
limiting factor when scaling an Asterisk system and we will share any 
advances we make with the community.

We also noticed some "Avoided initial deadlock" and "Avoided
deadlock"
warnings in the messages file that Asterisk generates.  Our 128 and 256 
simultaneous call tests each produced one of these warnings.  Our 512 
simultaneous call test produced 18 of them all within a 10 second span, 
each on a separate SIP channel.  Can anyone shed light on what these 
warnings mean, how they can be avoided, and if they are something to 
worry about.  They did not seem to affect the outcome of the testing in 
any way.

Details of our testing (hardware configuration, OS tweaks, system 
resource usage, reports from the Abacus 5000, etc.) will be made 
available over time.  Please bear with me, as I am working on a deadline.

Thank you,

Matthew Roth
InterMedia Marketing Solutions
Software Engineer and Systems Developer

On Tue, 2005-09-20 at 18:37 -0400, Matt Roth wrote:
> List users,
> 
> Over the last few days we have been working with MCI's development lab 
> to test our Asterisk setup.  We were using a piece of hardware called an 
> Abacus 5000 that is capable of creating and terminating thousands of SIP 
> calls.  Initially, we could not get past 64 simultaneous digitally 
> recorded calls without having call quality issues including dropped 
> calls.  We identified an I/O bottleneck and rectified it by digitally 
> recording to a RAM disk.  Using this method, we were able to digitally 
> record 512 simultaneous SIP-to-SIP calls with 100% call completion.
> 
> Our plan is to use the MONITOR_EXEC hook to call a custom program that 
> will copy files to the hard disk at call completion.
Matt,

Interesting stuff. Did you test copying of the recordings on the ramdisk
to a local harddisk also during that 512 call load? Just wondering if
that copy action wouldn't also create an I/O bottleneck and cause call
quality issues under load. Did you consider using remote storage e.g.
via nfs, a fibre channel or iSCSI link to a SAN?

Regards,
Patrick

Also when you do things over the network, disable your onboard network
card, and go for some more expensive network card.
In our tests with small packets, we could increase the throughput with a
factor 2. (related to cpu load).

Zoa.

----------
www.asteriskguru.com

trixter http://www.0xdecafbad.com wrote:
>On Wed, 2005-09-21 at 10:07 +0300, Zoa wrote:
>
>
>>The reason i recommended you to use a ramdisk is because i think the
>>problem with recording to disk is saving 20ms of stream 1, then 20 ms of
>>stream 2, then 20ms of stream 3 etc etc.... meaning you write everytime
>>very small things. (with a lot of seeking).
>>Our best test results were with:
>>
>>
>>
>filesystems are also a consideration with larger scale projects.
>Different filesystems add different amounts of overheads on different
>types of operations.  Some are faster at moving small files around
>others faster with large files.  This adds to the disk latency.
>Removing the disk latency itself is a good thing, since that is
>typically slower, but to crank out that last little bit of performance
>some research into the different filesystems under the specific kernel
>that you are using could also be a consideration.  The most obvious
>(and easiest to update a running system) is to remove things like atime,
>whih with most linux distros is on by default.  This causes a write
>operation for the read of a file to update the last time accessed.  A
>couple little things can add up to a few percent improvement and
>generally make the cost go down.
>
>
>
>
>>- buffering the recordings to a ramdisk, then
>>- on low load (at night) copy the files over the network (easy to shape
>>the pipe, so that you dont overload anything),
>>
>>
>Or have a seperate network set up (dual nic card for example) where the
>2nd network is used just for NFS traffic.  Although NFS generally is
>ugly network wise, it is standard and makes things easier.  Just gotta
>watch the IO on the system given that the network card itself will cause
>cpu cycles to be used, but lets face it cpu is cheap now.  Different
>drivers also work differently, and then with the 2.6 series kernels you
>can use device polling instead of interupts which can help a little.
>
>
>
>
>
>>If you want to go even freakier, run asterisk (or you complete distro)
>>from a ramdisk.
>>
>>
>>
>When you say ramdisk here I assume you mean using conventional ram, its
>cheap yes but its volatile, do you have any plans for failure of the
>system or ram?  Or is the data integrity itself not as critical?  The
>reason that people like hard drives is because most of the time if the
>system goes down for any reason the data is still intact.
>
>
>
>
>>I thought over your suggestion to use a sniffer to do the recordings,
>>you might pull it off, but will have to write your own to do so. (or go
>>to the expensive version of commercial sniffer applications).
>>
>>
>>
>isnt vomit free?  It was a voip sniffer that worked with some codecs
>many years ago (I wanna say mid-late 90s but I may be thinking of
>another back then). http://vomit.xtdnet.nl/ does G.711 only.
>
>The bigger prIoblem that I see is that sniffers dont always get all the
>traffic that is on a network particularly when the network has more
>traffic on it.  While this generally isnt a concern and I would like to
>think that even a poorly configured network could allow for 512 calls,
>it is a factor to implement this type of a solution.
>
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>--Bandwidth and Colocation sponsored by Easynews.com --
>
>Asterisk-Users mailing list
>Asterisk-Users@lists.digium.com
>http://lists.digium.com/mailman/listinfo/asterisk-users
>To UNSUBSCRIBE or update options visit:
>   http://lists.digium.com/mailman/listinfo/asterisk-users
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 254 bytes
Desc: OpenPGP digital signature
Url :
http://lists.digium.com/pipermail/asterisk-users/attachments/20050921/c554f68b/signature.pgp

On Wed, 2005-09-21 at 11:11 +0300, Zoa wrote:
> Also when you do things over the network, disable your onboard network
> card, and go for some more expensive network card.
> In our tests with small packets, we could increase the throughput with a
> factor 2. (related to cpu load).
I wonder how much of that is a poorly written driver and not the card
itself.  I have seen some fairly poor drivers performance wise.  :/ 


-- 
Trixter http://www.0xdecafbad.com     Bret McDanel
UK +44 870 340 4605   Germany +49 801 777 555 3402
US +1 360 207 0479 or +1 516 687 5200
FreeWorldDialup: 635378
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url :
http://lists.digium.com/pipermail/asterisk-users/attachments/20050921/c96ebf9c/attachment.pgp

True...  But i tried several brands of cards, and several drivers, the
dual nic gigabit intel card was a lot better than all the other
combinations i tried.

zoa

trixter http://www.0xdecafbad.com wrote:
>On Wed, 2005-09-21 at 11:11 +0300, Zoa wrote:
>
>
>>Also when you do things over the network, disable your onboard network
>>card, and go for some more expensive network card.
>>In our tests with small packets, we could increase the throughput with a
>>factor 2. (related to cpu load).
>>
>>
>
>I wonder how much of that is a poorly written driver and not the card
>itself.  I have seen some fairly poor drivers performance wise.  :/
>
>
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>--Bandwidth and Colocation sponsored by Easynews.com --
>
>Asterisk-Users mailing list
>Asterisk-Users@lists.digium.com
>http://lists.digium.com/mailman/listinfo/asterisk-users
>To UNSUBSCRIBE or update options visit:
>   http://lists.digium.com/mailman/listinfo/asterisk-users
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 254 bytes
Desc: OpenPGP digital signature
Url :
http://lists.digium.com/pipermail/asterisk-users/attachments/20050921/84d1842f/signature.pgp