asterisk users - Jul 2011 - Benchmarking AGI performance in C, PHP, and Perl

Many times, I've made the statement that you can execute hundreds of AGIs 
written in C in the time it takes to load an interpreter and parse a 
script written in PHP or Perl.

Recently, a Doubting Thomas asked me to substantiate my claim.

I suspect nobody has made the effort to implement an AGI of any reasonable 
size and function in multiple languages.

I'm guessing it may not really be all that important and the results would 
be too task specific to be relevant.

I suspect once an AGI is executing, the choice of source language is 
unimportant. I'll go out on a limb and say executing:

 	select prompt_path from foo where bar;
 	stream file prompt_path "1234*"
 	stream file you_entered ""
 	say digits selected ""

will execute in effectively the same time regardless of source language. 
Waiting for Asterisk to play a file or for your database to return a row 
is beyond the scope of your AGI.

It's what you do between your AGI and database calls that will determine 
how much your choice of source language will impact the total execution 
time.

Unless you're doing a lot of stuff in between these API calls, the only 
place you can make an impact is getting your code into memory and ready to 
execute.

I 'wrote' 2 different AGIs in C, PHP, and Perl.

The first AGI, 'null-agi' reads the AGI environment variables from STDIN
and exits. To me, this is the bare minimum a program can do and call 
itself an AGI. Each AGI was less than 10 lines.

The second AGI, 'neutered-agi' is an AGI of 'production length'
(around
1,600 lines) and supporting access to a MySQL database. The AGI is of 
'production length' but still exits after reading the AGI environment 
variables because we are measuring program startup time.

For both AGIs, the C implementation used an AGI library I developed way 
too many years ago. The PHP implementation used PHPAGI. The Perl 
implementation used Asterisk::AGI.

The C version of neutered-agi was based on a 'voicemail-like' AGI I
wrote
many years ago that stored the user credentials and messages in MySQL.

The PHP version of neutered-agi was based on dialparties.agi (nicked from 
PIAF). dialparties.agi is only about 800 lines long, so I 'doubled it'
by
copying and pasting it into the same source file. While dialparties.agi 
does not use MySQL, MySQL is available in PHP without including additional 
header or class files.

The Perl version of neutered-agi was based on agi-VDAD_ALL_outbound.agi 
(nicked from Vicidial).

I ran the tests in 3 different environments:

|-------------------+--------+--------+----------|
| CPU               | RAM    | CentOS | Asterisk |
|-------------------+--------+--------+----------|
| Geode 500MHz      | 256 MB |    4.9 |   1.2.37 |
| Atom D525 1.80GHz | 4 GB   |    5.6 |  1.8.4.1 |
| Xeon 3.40GHz      | 2 GB   |    4.8 |   1.2.40 |
|-------------------+--------+--------+----------|

No swapping occurred during the tests.

My dialplan executed each AGI 1,000 times to make the cumulative execution 
time more measurable. I wrote the dialplan in both 'inline dialplan' and
an AEL 'for' loop. The initial execution times were the same so the test
runs were made with the AEL version because it is more manageable. (5 
lines versus 1,000 lines.)

Here's the results for executing each AGI 1,000 times on each host in 
seconds:

Geode:

|----------+----------+--------------|
| language | null-agi | neutered-agi |
|----------+----------+--------------|
| C        |        6 |            6 |
| PHP      |      116 |          160 |
| Perl     |       99 |          639 |
|----------+----------+--------------|

Atom:

|----------+----------+--------------|
| language | null-agi | neutered-agi |
|----------+----------+--------------|
| C        |        6 |            6 |
| PHP      |       52 |           65 |
| Perl     |       38 |          197 |
|----------+----------+--------------|

Xeon:

|----------+----------+--------------|
| language | null-agi | neutered-agi |
|----------+----------+--------------|
| C        |        2 |            2 |
| PHP      |       40 |           47 |
| Perl     |       10 |          107 |
|----------+----------+--------------|

Summary:

Geode - Perl / C: 106
Atom - Perl / C: 33
Xeon - Perl / C: 54

The C null-agi AGI was statically linked. I didn't have all the libraries 
needed to statically link neutered-agi on these boxes, but the dynamically 
linked versions of null-agi and neutered-agi took the same time to execute 
(16 seconds on the Geode) so I'm assuming statically linked versions of 
null-agi and neutered-agi would also take the same time to execute. It 
also helps support my original statement :)

I guessing the Perl version of neutered-agi took a big hit from having to 
load the database code as well as the AGI framework (use DBI; use 
Asterisk::AGI;) while PHP only had to load the AGI framework (require_once 
"phpagi.php";).

I'll let you decide if the methodology is meaningful to your environment. 
I don't consider myself to be a PHP or Perl expert, so if I've made some
colossal blunder in my methodology, please let me know.

I'm guessing you'd have to resurrect a Soekris net4801 from the 
way-back-machine to substantiate my orignal claim of 'hundreds.'
I'll have
to remind myself to say 'dozens' from now on.

(I don't have a 'thing' against Soekris -- I just bought 2 off Ebay
to
play with.)

-- 
Thanks in advance,
-------------------------------------------------------------------------
Steve Edwards       sedwards at sedwards.com      Voice: +1-760-468-3867 PST
Newline                                              Fax: +1-760-731-3000

On 12/07/11 9:29 AM, Steve Edwards wrote:
> Many times, I've made the statement that you can execute hundreds of
> AGIs written in C in the time it takes to load an interpreter and parse
> a script written in PHP or Perl.
It would be interesting to see the same types of tests run against 
fast-agi - personally if I write an agi that will be called 1000 times 
I'm going to leave it running and have network requests against it 
rather than starting and stopping every time.

Interesting tests nonetheless - I would have been pretty concerned if 
straight C wasn't faster :-)

Mind if I post it to the Daily Asterisk News?

-- 
Cheers,

Matt Riddell
_______________________________________________

http://www.venturevoip.com/news.php (Daily Asterisk News)
http://www.venturevoip.com/exchange.php (Full ITSP Solution)
http://www.venturevoip.com/cc.php (Call Centre Solutions)

On Mon, Jul 11, 2011 at 5:29 PM, Steve Edwards
<asterisk.org at sedwards.com> wrote:
> Many times, I've made the statement that you can execute hundreds of
AGIs
> written in C in the time it takes to load an interpreter and parse a script
> written in PHP or Perl.
I've truly enjoyed this thread. And while startup time is certainly
part of the equation, I'm curious whether you also recorded memory
overhead while you were doing your benchmarks.

In my personal observation of 'Perl running in production', there's
a
significant memory complexity associated with running lots of
simultaneous Perl interpreters. I'm guessing the PHP overhead is
smaller but still a factor.

Because C lib is both statically compiled and essentially built into
the system, there's no such 'waste' of running the interpreters.

Along with the narrowing gap you saw by 'throwing hardware at the
problem' that shrinks the difference between C and interpreted
languages, I personally now work with a production asterisk
environment where systems have dozens of GB of ram. As such, it's not
entirely crazy to say 'so what' about the whole thing.

Nonetheless, do you have any numbers to back up my theory?

> On 11/07/11 23:42, Steve Edwards wrote:
>> 'Standalone' AGIs still have advantages in lower complexity and
less
>> impact on failure. If a bug takes out your fastagi daemon it can affect
>> all calls.
On Tue, 12 Jul 2011, Vincent Sweeney wrote:
> I'm pretty sure if you have a bug in your AGI code it's going to
affect
> all calls whether its fastagi or not.
I was thinking of a situation where not all calls follow the exact same 
path through the code and one of those paths caused the daemon to 
segfault.
> In a production environment any code will have gone through a testing 
> phase, so your argument about a compiled language detecting stupid 
> errors is pretty much irrelevant. Also most critical bugs will be in the 
> script logic not the syntax.
In a perfect world, we would all have regression testing in place to 
exercise every decision point and every value threshold in our code. I've 
never found a client willing to pay for that.

It has been my experience with many scripting languages that they do not 
syntax check my code, warn me if I have too many or too few arguments to a 
function, warn of undeclared variables or warn of unused variables. Gcc 
does all of these for me. Will 'strict' or 'taint' do this for
me? In AEL,
a single missing semicolon can cause large blocks of code to disappear. 
Maybe I'm just more clumsy at the keyboard, but I have had characters 
appear and disappear within an editing session.
> I'm also curious why you think the poor performance of a scripting 
> language actually matters for AGI code? As stated most of the time will 
> be spent by Asterisk streaming audio / waiting for a prompt. The few 
> extra milliseconds a php script takes to start up are not going to be 
> noticeable by any human listening to the call.
It matters once you know. It influences your decision making process.

In my home Asterisk box, it just doesn't matter. The dual cores (plus 
hyperthreading) of an Atom D525 with 4GB of RAM can handle the occasional 
2 or 3 simultaneous calls in its sleep.

Think of a system handling hundreds of simultaneous calls, tens of 
thousands of calls per day. All those milliseconds, all the memory 
footprints, all the disk accesses reading the 'uses' and
'requires' will
add up.

Paraphrasing Senator Everett Dirksen, "A billion (th of a second) here, a 
billion (th of a second) there, and pretty soon you're talking about real 
money.?

Look back to the stats from my first post. Starting (not executing to 
completion) neutered-agi.pl (AKA agi-VDAD_ALL_outbound.agi) on the Geode 
took 2/3rds of a second. Obviously a Geode is the wrong platform for 
Vicidial, but that kind of delay can add up fast.

My first Asterisk system is still in production and making money for the 
client. It was designed (by me) very simplistically. Each AGI did one 
thing and then moved on. So an incoming call would execute set-globalid, 
block-ani, lookup-dnis, update-channel-status, update-caller-status and 
write-cdr all before the caller heard the first prompt. All the AGIs were 
coded in C, all but 1 access the database and it all happens in the blink 
of an ear.

I think I was lucky. I think if I had implemented the same design in Perl 
it would have fallen flat.

If I was to re-implement the system today I would obviously merge all 
these steps into a single AGI, but it works well and the client won't pay 
to fix what isn't broken.

-- 
Thanks in advance,
-------------------------------------------------------------------------
Steve Edwards       sedwards at sedwards.com      Voice: +1-760-468-3867 PST
Newline                                              Fax: +1-760-731-3000

On Mon, Jul 11, 2011 at 02:29:25PM -0700, Steve Edwards wrote:
> The second AGI, 'neutered-agi' is an AGI of 'production
length' (around
> 1,600 lines) and supporting access to a MySQL database. The AGI is of  
> 'production length' but still exits after reading the AGI
environment
> variables because we are measuring program startup time.
Perl's startup time also depends on the modules you load:

The following is a simple loop that runs a perl program doing nothing
(executing an empty statement) but firt loading a different set of
modules for the task. The loop repeats it 1000 times:

# No extra modules:
$ time for i in `seq 1000`; do perl  -e ''; done
real    0m5.093s
user    0m1.676s
sys     0m2.524s

# Asterisk::AGI:
$ time for i in `seq 1000`; do perl -MAsterisk::AGI -e ''; done
real    0m9.400s
user    0m6.368s
sys     0m2.060s

# MySQL access:
$ time for i in `seq 1000`; do perl -MDBI -MDBD::mysql -e ''; done
real    0m40.964s
user    0m33.122s
sys     0m5.416s

# Asterisk::AGI and MySQL access:
$ time for i in `seq 1000`; do perl -MAsterisk::AGI -MDBI -MDBD::mysql -e
''; done
real    0m45.898s
user    0m36.798s
sys     0m6.048s

-- 
               Tzafrir Cohen
icq#16849755              jabber:tzafrir.cohen at xorcom.com
+972-50-7952406           mailto:tzafrir.cohen at xorcom.com
http://www.xorcom.com  iax:guest at local.xorcom.com/tzafrir