asterisk users - Aug 2013 - Need input on scalable system design...

Hey All,

Growing call center. Currently at about 200 call center staff, running 
about 1000 calls per hour. Gearing up to double that. Not too sure that 
a single server will support that growth. So, I'm trying to come up with 
ways to scale the system and still maintain a simplistic design. So I'd 
like to bounce some ideas around.

Currently I am running on a Dell 1950, dual quad core 2.33ghz xeons, 
with 16gb ram, and 2 tce400p cards. This server is managing the full 
load of the company. We are recording all calls, running ivr, queues, 
cdr, cel, and web for reporting. I currently have another 1950 of the 
exact same specifications as a cold spare.

Here's where you can see drawings of my current connectivity and an 
optional connectivity I'm contemplating...

http://www.paydaysupportcenter.com/current.pdf 
<http://www.linkedin.com/redirect?url=http%3A%2F%2Fwww%2Epaydaysupportcenter%2Ecom%2Fcurrent%2Epdf&urlhash=qLsB&_t=tracking_anet>http://www.paydaysupportcenter.com/option.pdf
<http://www.linkedin.com/redirect?url=http%3A%2F%2Fwww%2Epaydaysupportcenter%2Ecom%2Foption%2Epdf&urlhash=CJG1&_t=tracking_anet>

As you can see I currently have a separate sql server and a separate 
storage server for the call recordings. This is all working fine.

However, I'm thinking for scalability I should be looking to migrate to 
a configuration similar to the one in option.pdf. Where I have a VOIP 
gateway server that simply relays traffic and possibly can do some load 
balancing or intellegent routing. But nothing more then that, and 
possibly a second one of these online as a hot failover.

Then have separate sql, storage, (i forgot it in the pic) web, and 
asterisk servers behind that on separate dedicated network. Here's my 
dilemma though, how do I balance the load across multiple machines for 
scalability...

Since 95% of our calls come into queues, I need to be able to maintain 
queue stats and presence across all of the servers. Thus far, I've got 
everything except the extensions.conf file into the mysql database. I 
thought about setting up 2 servers, 1 for sales, and 1 for customer 
service, then possibly break out each call queue to it's own server as 
things grow. Just not sure if that's the right way to go.

Then regarding extensions.conf, I've read that it too can be placed in 
the sql database and accessed via switch. however it's resource intense, 
so now I'm thinking of maybe putting that file on the nfs server for all 
of the boxes to read from.

As for the design of that file, I was kind of thinking of a modular 
design within the file using various goto's and gosubs. Our business 
model is based on affiliates and corporate marketing, so we have a ton 
of did's that follow the same call flow with minor modifications in some 
variables, as well as variations in call flow, and hours of operation. 
Thus the modular design of the call flow. Then the primary inbound 
context would simply be a list of did's pointing to a goto with a list 
of the variations and variables for the did.

Ok, now that I've melted your brains.... thoughts?

Thanks all in advance for the discussion...
Greg

-- 
Greg

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.digium.com/pipermail/asterisk-users/attachments/20130827/be6c4c1c/attachment.htm>

Hi Greg,

I have a similar setup with multiple asterisk boxes. I have used opensips
as a load balancer (+some pre routing logic before load balancing).

It is based on this idea:
http://www.opensips.org/Documentation/Tutorials-LoadBalancing-1-9

- Laszlo




2013/8/27 Gregory Malsack <gmalsack at coastalacq.com>
>   Hey All,
>
> Growing call center. Currently at about 200 call center staff, running
> about 1000 calls per hour. Gearing up to double that. Not too sure that a
> single server will support that growth. So, I'm trying to come up with
ways
> to scale the system and still maintain a simplistic design. So I'd like
to
> bounce some ideas around.
>
> Currently I am running on a Dell 1950, dual quad core 2.33ghz xeons, with
> 16gb ram, and 2 tce400p cards. This server is managing the full load of the
> company. We are recording all calls, running ivr, queues, cdr, cel, and web
> for reporting. I currently have another 1950 of the exact same
> specifications as a cold spare.
>
> Here's where you can see drawings of my current connectivity and an
> optional connectivity I'm contemplating...
>
>
http://www.paydaysupportcenter.com/current.pdf<http://www.linkedin.com/redirect?url=http%3A%2F%2Fwww%2Epaydaysupportcenter%2Ecom%2Fcurrent%2Epdf&urlhash=qLsB&_t=tracking_anet>
>
http://www.paydaysupportcenter.com/option.pdf<http://www.linkedin.com/redirect?url=http%3A%2F%2Fwww%2Epaydaysupportcenter%2Ecom%2Foption%2Epdf&urlhash=CJG1&_t=tracking_anet>
>
> As you can see I currently have a separate sql server and a separate
> storage server for the call recordings. This is all working fine.
>
> However, I'm thinking for scalability I should be looking to migrate to
a
> configuration similar to the one in option.pdf. Where I have a VOIP gateway
> server that simply relays traffic and possibly can do some load balancing
> or intellegent routing. But nothing more then that, and possibly a second
> one of these online as a hot failover.
>
> Then have separate sql, storage, (i forgot it in the pic) web, and
> asterisk servers behind that on separate dedicated network. Here's my
> dilemma though, how do I balance the load across multiple machines for
> scalability...
>
> Since 95% of our calls come into queues, I need to be able to maintain
> queue stats and presence across all of the servers. Thus far, I've got
> everything except the extensions.conf file into the mysql database. I
> thought about setting up 2 servers, 1 for sales, and 1 for customer
> service, then possibly break out each call queue to it's own server as
> things grow. Just not sure if that's the right way to go.
>
> Then regarding extensions.conf, I've read that it too can be placed in
the
> sql database and accessed via switch. however it's resource intense, so
now
> I'm thinking of maybe putting that file on the nfs server for all of
the
> boxes to read from.
>
> As for the design of that file, I was kind of thinking of a modular design
> within the file using various goto's and gosubs. Our business model is
> based on affiliates and corporate marketing, so we have a ton of did's
that
> follow the same call flow with minor modifications in some variables, as
> well as variations in call flow, and hours of operation. Thus the modular
> design of the call flow. Then the primary inbound context would simply be a
> list of did's pointing to a goto with a list of the variations and
> variables for the did.
>
> Ok, now that I've melted your brains.... thoughts?
>
> Thanks all in advance for the discussion...
> Greg
>
> --
> Greg
>
>
> --
> _____________________________________________________________________
> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
> New to Asterisk? Join us for a live introductory webinar every Thurs:
>                http://www.asterisk.org/hello
>
> asterisk-users mailing list
> To UNSUBSCRIBE or update options visit:
>    http://lists.digium.com/mailman/listinfo/asterisk-users
>


-- 

--
Kind regards,
Laszlo Bekesi
http://voipfreak.net
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.digium.com/pipermail/asterisk-users/attachments/20130827/0c3c5be5/attachment.htm>

On 27/08/13 18:16, Gregory Malsack wrote:
>
> Hey All,
>
> Growing call center. Currently at about 200 call center staff, running 
> about 1000 calls per hour. Gearing up to double that. Not too sure 
> that a single server will support that growth. So, I'm trying to come 
> up with ways to scale the system and still maintain a simplistic 
> design. So I'd like to bounce some ideas around.
>
> Currently I am running on a Dell 1950, dual quad core 2.33ghz xeons, 
> with 16gb ram, and 2 tce400p cards. This server is managing the full 
> load of the company. We are recording all calls, running ivr, queues, 
> cdr, cel, and web for reporting. I currently have another 1950 of the 
> exact same specifications as a cold spare.
>
> Here's where you can see drawings of my current connectivity and an 
> optional connectivity I'm contemplating...
>
> *MailScanner has detected a possible fraud attempt from 
> "www.linkedin.com" claiming to be* 
> http://www.paydaysupportcenter.com/current.pdf 
>
<http://www.linkedin.com/redirect?url=http%3A%2F%2Fwww%2Epaydaysupportcenter%2Ecom%2Fcurrent%2Epdf&urlhash=qLsB&_t=tracking_anet>*MailScanner
> has detected a possible fraud attempt from "www.linkedin.com"
claiming
> to be* http://www.paydaysupportcenter.com/option.pdf 
>
<http://www.linkedin.com/redirect?url=http%3A%2F%2Fwww%2Epaydaysupportcenter%2Ecom%2Foption%2Epdf&urlhash=CJG1&_t=tracking_anet>
>
> As you can see I currently have a separate sql server and a separate 
> storage server for the call recordings. This is all working fine.
>
> However, I'm thinking for scalability I should be looking to migrate 
> to a configuration similar to the one in option.pdf. Where I have a 
> VOIP gateway server that simply relays traffic and possibly can do 
> some load balancing or intellegent routing. But nothing more then 
> that, and possibly a second one of these online as a hot failover.
>
> Then have separate sql, storage, (i forgot it in the pic) web, and 
> asterisk servers behind that on separate dedicated network. Here's my 
> dilemma though, how do I balance the load across multiple machines for 
> scalability...
>
> Since 95% of our calls come into queues, I need to be able to maintain 
> queue stats and presence across all of the servers. Thus far, I've got 
> everything except the extensions.conf file into the mysql database. I 
> thought about setting up 2 servers, 1 for sales, and 1 for customer 
> service, then possibly break out each call queue to it's own server as 
> things grow. Just not sure if that's the right way to go.
>
> Then regarding extensions.conf, I've read that it too can be placed in 
> the sql database and accessed via switch. however it's resource 
> intense, so now I'm thinking of maybe putting that file on the nfs 
> server for all of the boxes to read from.
>
> As for the design of that file, I was kind of thinking of a modular 
> design within the file using various goto's and gosubs. Our business 
> model is based on affiliates and corporate marketing, so we have a ton 
> of did's that follow the same call flow with minor modifications in 
> some variables, as well as variations in call flow, and hours of 
> operation. Thus the modular design of the call flow. Then the primary 
> inbound context would simply be a list of did's pointing to a goto 
> with a list of the variations and variables for the did.
>
> Ok, now that I've melted your brains.... thoughts?
>
> Thanks all in advance for the discussion...
> Greg
>
We have a similar server but a single quad core at 3Ghz. It easily 
handles 400 concurrent calls with a lot shorter average call duration 
than you have. It doesnt do as much call recording but does do a lot of 
AGI stuff.

With regard to nfs thats fine for non real time stuff. Personally we 
have a test machine and multiple live machines and use subversion to 
commit any approved changes and then check them out on the live boxes. 
We dont need to worry about shared file space and we get version control 
of the configuration as an additional benefit.
Its similar for call recordings. We have call recordings going to a ram 
disc and then when they are complete there is a background process to 
copy them to the nfs volume. If nfs is unavailable then they are moved 
to the internal disk temporarily until the nfs is back online. We have 
never used this functionality but it add a little redundancy.

I would put opensips at the front end which looks at the incoming 
destination number and routes the call to the appropiate front end 
asterisk box depending on the queue it should go to. The other asterisk 
box(s) will be a backup so if one asterisk box fails then one of the 
others takes over running that queue automatically.

For call recording are you using mixmonitor?
I would consider using the normal monitor command and pass these 
recordings off to the nfs and have another machine process them by 
mixing both legs and perhaps converting to mp3 aswell.

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.digium.com/pipermail/asterisk-users/attachments/20130828/3f0297b8/attachment.htm>

Hi Greg,
I am aware of a couple of solutions that come prepackaged and offer
distributed queues for Asterisk. One of them, that seems to work well and
reliably, is the one from Raynet. I am sure there are more. On the other
side, I have seen a number of in-house solutions where you basically have a
daemon polling queues statuses and redirecting calls based on the relative
wait times. Rough but effective, and can be deployed easily.

About recordings, my suggestion would be to use something to offload them
right from the servers, like Oreka. Have a number of large clients using it
and they are quite happy (plus, the guys supporting it are superb).

Just my two cents,
l.




2013/8/27 Gregory Malsack <gmalsack at coastalacq.com>
>   Hey All,
>
> Growing call center. Currently at about 200 call center staff, running
> about 1000 calls per hour. Gearing up to double that. Not too sure that a
> single server will support that growth. So, I'm trying to come up with
ways
> to scale the system and still maintain a simplistic design. So I'd like
to
> bounce some ideas around.
>
> Currently I am running on a Dell 1950, dual quad core 2.33ghz xeons, with
> 16gb ram, and 2 tce400p cards. This server is managing the full load of the
> company. We are recording all calls, running ivr, queues, cdr, cel, and web
> for reporting. I currently have another 1950 of the exact same
> specifications as a cold spare.
>
> Here's where you can see drawings of my current connectivity and an
> optional connectivity I'm contemplating...
>
>
http://www.paydaysupportcenter.com/current.pdf<http://www.linkedin.com/redirect?url=http%3A%2F%2Fwww%2Epaydaysupportcenter%2Ecom%2Fcurrent%2Epdf&urlhash=qLsB&_t=tracking_anet>
>
http://www.paydaysupportcenter.com/option.pdf<http://www.linkedin.com/redirect?url=http%3A%2F%2Fwww%2Epaydaysupportcenter%2Ecom%2Foption%2Epdf&urlhash=CJG1&_t=tracking_anet>
>
> As you can see I currently have a separate sql server and a separate
> storage server for the call recordings. This is all working fine.
>
> However, I'm thinking for scalability I should be looking to migrate to
a
> configuration similar to the one in option.pdf. Where I have a VOIP gateway
> server that simply relays traffic and possibly can do some load balancing
> or intellegent routing. But nothing more then that, and possibly a second
> one of these online as a hot failover.
>
> Then have separate sql, storage, (i forgot it in the pic) web, and
> asterisk servers behind that on separate dedicated network. Here's my
> dilemma though, how do I balance the load across multiple machines for
> scalability...
>
> Since 95% of our calls come into queues, I need to be able to maintain
> queue stats and presence across all of the servers. Thus far, I've got
> everything except the extensions.conf file into the mysql database. I
> thought about setting up 2 servers, 1 for sales, and 1 for customer
> service, then possibly break out each call queue to it's own server as
> things grow. Just not sure if that's the right way to go.
>
> Then regarding extensions.conf, I've read that it too can be placed in
the
> sql database and accessed via switch. however it's resource intense, so
now
> I'm thinking of maybe putting that file on the nfs server for all of
the
> boxes to read from.
>
> As for the design of that file, I was kind of thinking of a modular design
> within the file using various goto's and gosubs. Our business model is
> based on affiliates and corporate marketing, so we have a ton of did's
that
> follow the same call flow with minor modifications in some variables, as
> well as variations in call flow, and hours of operation. Thus the modular
> design of the call flow. Then the primary inbound context would simply be a
> list of did's pointing to a goto with a list of the variations and
> variables for the did.
>
> Ok, now that I've melted your brains.... thoughts?
>
> Thanks all in advance for the discussion...
> Greg
>
> --
> Greg
>
>
> --
> _____________________________________________________________________
> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
> New to Asterisk? Join us for a live introductory webinar every Thurs:
>                http://www.asterisk.org/hello
>
> asterisk-users mailing list
> To UNSUBSCRIBE or update options visit:
>    http://lists.digium.com/mailman/listinfo/asterisk-users
>


-- 
Loway - home of QueueMetrics - http://queuemetrics.com
Try the WombatDialer auto-dialer @ http://wombatdialer.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.digium.com/pipermail/asterisk-users/attachments/20130829/f84f71fc/attachment.htm>