Hi, During tests with a IAX2/PSTN gateway I've been getting strange results for processor idle time and load. I (re)search(ed) this issue for a while, but I didn't get any good explainations. Can somebody help me? I have several sites that rely on a central server for connection to the PSTN. Calls to the PSTN are routed over the Internet to this PSTN gateway using IAX2 in trunk mode. To minimize bandwith usage, the Speex codec is used. The central PSTN gateway is a P4 3.0GHz, 1GByte mem, has a TE110P card supporting ISDN30 and runs Asterisk version 1.0.3 on Debian Sarge. While sustaining 5 connections dialed in through the TE110P (terminated at remote sites through IAX) and running top on the PSTN gateway, I see 98% CPU idle time most of the time. I also see short (around 10sec) bursts of high CPU usage (40-50%) by one of the asterisk processes supporting the connection. The bursts happen in irregular intervals, ranging from 30 to 60 sec. Meanwhile, the reported average load jumps up and down between 0.1 to 0.7. What's happening here? Is the processor load really this erratic, or am I looking at an artefact in cpu usage measurement? Maybe there is an aliasing effect caused by the periodic cpu load (20ms, default trunk frequency) and the cpu usage measurement (also periodic?), but I don't know how to check this. If this top reading is an artefact, is there a way to check the actual (realtime) load? Regarding the actual processor usage for speex encoding: this report suggests my processor is indeed quite busy encoding a few speex channels: http://astertest.com/astricon_performance.ppt. Given the results in this report, I doubt the PSTN gateway will support more than 10 speex encodings. At the same time, the same processor encodes 756x756 PAL television to mpeg-4 on my mythtv box at home. Twice, leaving room for scheduled jobs. Has anyone some references to documentation to put these figures into perspective? Thanks in advance, Eric.
Hi, Im the astertest guy. If you are referring to the graphs on page 41 - 42, please note that those are done on a embedded via 800mhz cpu and not on a system similar to yours. So i'm pretty sure you can do more than 10 speex encodings at the same time. (also some things changes since we did those tests, some optimizations and configuration options were done to the speex codec implementation in asterisk. Now you can choose your own quality vs cpu usage balance on asterisk) I suggest you go to cvs-head and check out the changes to the speex codec. Without changes to your asterisk machine, i estimate you will be able to do around 30 to 40 channels on your machine if it was without zaptel. Zaptel does not take a lot cpu especially compared to speex encodings. (i know from experience) About the periodic load, please see if any calls are being setup or tear down, or specific applications are used in those cases. Saddly enough, i still didnt find the time to do any load measurements on pri cards. Although i have a test setup ready to go. Zoa. Eric Giesselbach wrote:>Hi, > >During tests with a IAX2/PSTN gateway I've been getting strange results for processor idle time and load. I (re)search(ed) this issue for a while, but I didn't get any good explainations. Can somebody help me? > >I have several sites that rely on a central server for connection to the PSTN. Calls to the PSTN are routed over the Internet to this PSTN gateway using IAX2 in trunk mode. To minimize bandwith usage, the Speex codec is used. The central PSTN gateway is a P4 3.0GHz, 1GByte mem, has a TE110P card supporting ISDN30 and runs Asterisk version 1.0.3 on Debian Sarge. > >While sustaining 5 connections dialed in through the TE110P (terminated at remote sites through IAX) and running top on the PSTN gateway, I see 98% CPU idle time most of the time. I also see short (around 10sec) bursts of high CPU usage (40-50%) by one of the asterisk processes supporting the connection. The bursts happen in irregular intervals, ranging from 30 to 60 sec. Meanwhile, the reported average load jumps up and down between 0.1 to 0.7. > >What's happening here? Is the processor load really this erratic, or am I looking at an artefact in cpu usage measurement? Maybe there is an aliasing effect caused by the periodic cpu load (20ms, default trunk frequency) and the cpu usage measurement (also periodic?), but I don't know how to check this. If this top reading is an artefact, is there a way to check the actual (realtime) load? > >Regarding the actual processor usage for speex encoding: this report suggests my processor is indeed quite busy encoding a few speex channels: http://astertest.com/astricon_performance.ppt. Given the results in this report, I doubt the PSTN gateway will support more than 10 speex encodings. At the same time, the same processor encodes 756x756 PAL television to mpeg-4 on my mythtv box at home. Twice, leaving room for scheduled jobs. Has anyone some references to documentation to put these figures into perspective? > >Thanks in advance, >Eric. > >_______________________________________________ >Asterisk-Users mailing list >Asterisk-Users@lists.digium.com >http://lists.digium.com/mailman/listinfo/asterisk-users >To UNSUBSCRIBE or update options visit: > http://lists.digium.com/mailman/listinfo/asterisk-users > >-------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 254 bytes Desc: OpenPGP digital signature Url : http://lists.digium.com/pipermail/asterisk-users/attachments/20050329/9f2b5800/signature.pgp
> -----Original Message----- > From: Zoa [mailto:zoachien@securax.org] > Sent: Tuesday, March 29, 2005 1:59 PM > To: Asterisk Users Mailing List - Non-Commercial Discussion > Subject: Re: [Asterisk-Users] Erratic CPU load > > > Hi, > Im the astertest guy. > If you are referring to the graphs on page 41 - 42, please note that > those are done on a embedded via 800mhz cpu and not on a > system similar > to yours. So i'm pretty sure you can do more than 10 speex > encodings at > the same time. (also some things changes since we did those > tests, some > optimizations and configuration options were done to the speex codec > implementation in asterisk. Now you can choose your own quality vs cpu > usage balance on asterisk) > > I suggest you go to cvs-head and check out the changes to the > speex codec.I did test a IAX/Speex connection between two test machines with cvs head installed. They also showed bursty cpu usage, but max cpu usage during the bursts might be lower, I'll check again. And I didn't change any speex config options, so there's hope :)> Without changes to your asterisk machine, i estimate you will > be able to > do around 30 to 40 channels on your machine if it was without zaptel. > Zaptel does not take a lot cpu especially compared to speex encodings. > (i know from experience) > > About the periodic load, please see if any calls are being > setup or tear down, or specific applications are used in those cases.30 channels would be great. Enough to service the 30 ISDN (E1) channels. The periodic load was not due to call setup or tear down, and as far as I could see (ps, top) there were no processes getting in Asterisk's way. Zaptel "on it's own" gave low (and steady) load during a test with calls from PSTN via TE110P to MOH on the gateway itself. But I guess only a test with iax, with codec translation and *without* zap can rule out zap as part of the cause.> > Saddly enough, i still didnt find the time to do any load measurements > on pri cards. Although i have a test setup ready to go.I can continue testing using your hints, thanks. Eric.> > Zoa. > > > Eric Giesselbach wrote: > > >Hi, > > > >During tests with a IAX2/PSTN gateway I've been getting > strange results for processor idle time and load. I > (re)search(ed) this issue for a while, but I didn't get any > good explainations. Can somebody help me? > > > >I have several sites that rely on a central server for > connection to the PSTN. Calls to the PSTN are routed over the > Internet to this PSTN gateway using IAX2 in trunk mode. To > minimize bandwith usage, the Speex codec is used. The central > PSTN gateway is a P4 3.0GHz, 1GByte mem, has a TE110P card > supporting ISDN30 and runs Asterisk version 1.0.3 on Debian Sarge. > > > >While sustaining 5 connections dialed in through the TE110P > (terminated at remote sites through IAX) and running top on > the PSTN gateway, I see 98% CPU idle time most of the time. I > also see short (around 10sec) bursts of high CPU usage > (40-50%) by one of the asterisk processes supporting the > connection. The bursts happen in irregular intervals, ranging > from 30 to 60 sec. Meanwhile, the reported average load jumps > up and down between 0.1 to 0.7. > > > >What's happening here? Is the processor load really this > erratic, or am I looking at an artefact in cpu usage > measurement? Maybe there is an aliasing effect caused by the > periodic cpu load (20ms, default trunk frequency) and the cpu > usage measurement (also periodic?), but I don't know how to > check this. If this top reading is an artefact, is there a > way to check the actual (realtime) load? > > > >Regarding the actual processor usage for speex encoding: > this report suggests my processor is indeed quite busy > encoding a few speex channels: > http://astertest.com/astricon_performance.ppt. Given the > results in this report, I doubt the PSTN gateway will support > more than 10 speex encodings. At the same time, the same > processor encodes 756x756 PAL television to mpeg-4 on my > mythtv box at home. Twice, leaving room for scheduled jobs. > Has anyone some references to documentation to put these > figures into perspective? > > > >Thanks in advance, > >Eric. > > > >_______________________________________________ > >Asterisk-Users mailing list > >Asterisk-Users@lists.digium.com > >http://lists.digium.com/mailman/listinfo/asterisk-users > >To UNSUBSCRIBE or update options visit: > > http://lists.digium.com/mailman/listinfo/asterisk-users > > > > > >
> -------- Snip -------- > During tests with a IAX2/PSTN gateway I've been getting strange results for processor idle time and load. I (re)search(ed) this issue for a while, but I didn't get any good explainations. Can somebody help me?Yes, Speex is pretty cpu intensive compared to other codecs. The best perfoming codecs are g.711u/a, gsm and g.729 in that order.> I have several sites that rely on a central server for connection to the PSTN. Calls to the PSTN are routed over the Internet to this PSTN gateway using IAX2 in trunk mode. To minimize bandwith usage, the Speex codec is used. The central PSTN gateway is a P4 3.0GHz, 1GByte mem, has a TE110P card supporting ISDN30 and runs Asterisk version 1.0.3 on Debian Sarge. > > While sustaining 5 connections dialed in through the TE110P (terminated at remote sites through IAX) and running top on the PSTN gateway, I see 98% CPU idle time most of the time. I also see short (around 10sec) bursts of high CPU usage (40-50%) by one of the asterisk processes supporting the connection. The bursts happen in irregular intervals, ranging from 30 to 60 sec. Meanwhile, the reported average load jumps up and down between 0.1 to 0.7. > > What's happening here? Is the processor load really this erratic, or am I looking at an artefact in cpu usage measurement? Maybe there is an aliasing effect caused by the periodic cpu load (20ms, default trunk frequency) and the cpu usage measurement (also periodic?), but I don't know how to check this. If this top reading is an artefact, is there a way to check the actual (realtime) load?With top you can increase the number of seconds beteween refreshes. this will give you a better useage over time. but if your cpu is spiking too 100% or even over 75% using the default update of 3 seconds your codec or your free memory may be the problem. Codec transcoding can take some serious hits on memory.> > Regarding the actual processor usage for speex encoding: this report suggests my processor is indeed quite busy encoding a few speex channels: http://astertest.com/astricon_performance.ppt. Given the results in this report, I doubt the PSTN gateway will support more than 10 speex encodings. At the same time, the same processor encodes 756x756 PAL television to mpeg-4 on my mythtv box at home. Twice, leaving room for scheduled jobs. Has anyone some references to documentation to put these figures into perspective?
David, Zoa helped me, but were not working together. What's more, I cannot focus on load tests too much: the setup I work on must be ready in may and starts small scale. This system must be functional and reliable first and should scale well later. The scaling part determines how long I am allowed to play with Asterisk - so performance issues are just personal :) A question about your "snake" load tests: have you seen any unexplainable spikes in processor load, or machine hangups every few hours? Eric.> -----Original Message----- > From: David [mailto:dm@sangoma.com] > Sent: Friday, April 01, 2005 4:59 PM > To: Eric Giesselbach > Subject: RE: RE: [Asterisk-Users] Erratic CPU load > > > Hi Eric, > > Thanks very much for your quick reply. Are you working with Zoa? > > I have seen Zoa's presentation on Asterisk performance > testing. I thought it > was really excellent, and I wish I could have attended. Do > you have any > updated data? > > Regards, > > David Mandelstam > Sangoma Technologies Corporation > email: dm@sangoma.com > web: www.sangoma.com > Tel: 905-474-1990 x 106 > 800-388-2475 x 106 > FAX: 905-474-9223 > > > > > At the moment I don't have much to add to your test concepts. > > I'm working with max 5 concurrent calls, because I'm mainly > > testing iax trunk timing issues (timestamp issues in asterisk > > v1.0.3 are repaired in cvs), routing (queueing) and effects > > of packet loss. The Speex load / cpu spike issue was an > > unexpected outcome I was worried about. > > > > The snake is something I use working with IAX and SIP (I > > patched Asterisk to be able to prevent native bridges). For a > > snake using E1 I have to wait for a second E1 delivered > > around may 1st. In the mean time I can work with our telco's > > conference service... > > > > Regards, > > Eric. > > > > > -----Original Message----- > > > From: David [mailto:dm@sangoma.com] > > > Sent: Friday, April 01, 2005 12:50 AM > > > To: Eric Giesselbach > > > Subject: RE: RE: [Asterisk-Users] Erratic CPU load > > > > > > > > > Hi Eric, > > > > > > We at Sangoma have been doing T1/E1 cards for over 10 years, and > > > lately we have been doing some Asterisk integration. We > > would love to > > > come up with some simple test setups that would allow us > to locally > > > load up large TDM systems for integrity testing. I was > wondering if > > > you had any ideas. > > > > > > Our current load tests are done on 2 machines > back-to-back with all > > > T1/E1 line connected. We then push calls in a "snake": > each machine > > > calling the other so that a single call goes through > maybe 94 links > > > before terminating in a channel bank or sip phone. > > > > > > It seems to load things up quite nicely, but I was > > wondering how the > > > astertest guru would simulate a heavy load. > > > > > > Regards, > > > > > > David Mandelstam > > > Sangoma Technologies Corporation > > > email: dm@sangoma.com > > > web: www.sangoma.com > > > Tel: 905-474-1990 x 106 > > > 800-388-2475 x 106 > > > FAX: 905-474-9223 > > > > > > > > > > > > > > > >