Hello all, I'm migrating a box from PJSIP with normal Flatfiles to ODBC/Realtime, Also 16.0.1 to 16.3.0. After adding a few peers to the new RT box, I noticed a delay in call processing. All I had done thus far is added a few endpoints for upstream carriers, And one downstream for the legacy PBX it was replacing. After a bit of troubleshooting I found the issue to be related to having the identify section matching a domain name (Instead of an IP). Each endpoint with match=domain.name.here (The specific name didn't really matter) added a bit of noticeable latency. Once you were up to 4-5 of them, It was a ~5 second delay from the time the invite came in until the time the call was routed and stood up with audio. Even when you were routing between unrelated (non-dns matched) endpoints. Looking at the debug logs, It seems like Asterisk looks these peers up on some kind of time interval. As well as during _every_ inbound call. This also explains why the delay increased with every single DNS based endpoint I added. Each extra peer means another DNS lookup and process. To further test my theory, I stood up a local caching DNS resolver and this significantly improved the delay. But it was still ~1 second with 5 DNS based endpoints. Even with a caching resolver, This doesn't scale. Doing a large number (and growing with scale) of DNS lookups for every call (Maybe even every packet) just doesn't fly. So my questions are: 1. Anyone have any idea why asterisk does this when using realtime but NOT when using flatfiles? 2. Is there not a way to cache this in Asterisk without having to do a DNS lookup every time? 3. Any other tweaks I can make to asterisk to speed this up (Not really looking to match based on other objects, like header or username). Nick Olsen Network Engineer Office: 321-408-5000 Mobile: 321-794-0763 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.digium.com/pipermail/asterisk-users/attachments/20190516/9c4eb1df/attachment.html>
On Thu, May 16, 2019, at 2:54 PM, Nick Olsen wrote:> Hello all, > > I'm migrating a box from PJSIP with normal Flatfiles to ODBC/Realtime, > Also 16.0.1 to 16.3.0. After adding a few peers to the new RT box, I > noticed a delay in call processing. All I had done thus far is added a > few endpoints for upstream carriers, And one downstream for the legacy > PBX it was replacing. After a bit of troubleshooting I found the issue > to be related to having the identify section matching a domain name > (Instead of an IP). Each endpoint with match=domain.name.here (The > specific name didn't really matter) added a bit of noticeable latency. > Once you were up to 4-5 of them, It was a ~5 second delay from the time > the invite came in until the time the call was routed and stood up with > audio. Even when you were routing between unrelated (non-dns matched) > endpoints. Looking at the debug logs, It seems like Asterisk looks > these peers up on some kind of time interval. As well as during _every_ > inbound call. This also explains why the delay increased with every > single DNS based endpoint I added. Each extra peer means another DNS > lookup and process. > > To further test my theory, I stood up a local caching DNS resolver and > this significantly improved the delay. But it was still ~1 second with > 5 DNS based endpoints. Even with a caching resolver, This doesn't > scale. Doing a large number (and growing with scale) of DNS lookups for > every call (Maybe even every packet) just doesn't fly. > > So my questions are: > 1. Anyone have any idea why asterisk does this when using realtime but > NOT when using flatfiles?Unless you configure caching for the objects[1] then every time the identify object is pulled from the database it has to be looked up and resolved.> 2. Is there not a way to cache this in Asterisk without having to do a > DNS lookup every time?There's nothing built in to cache this at a DNS level.> 3. Any other tweaks I can make to asterisk to speed this up (Not really > looking to match based on other objects, like header or username).[1] https://wiki.asterisk.org/wiki/display/AST/Sorcery+Caching -- Joshua C. Colp Digium - A Sangoma Company | Senior Software Developer 445 Jan Davis Drive NW - Huntsville, AL 35806 - US Check us out at: www.digium.com & www.asterisk.org
On Thu, May 16, 2019, at 2:57 PM, Joshua C. Colp wrote:> On Thu, May 16, 2019, at 2:54 PM, Nick Olsen wrote: > > Hello all, > > > > I'm migrating a box from PJSIP with normal Flatfiles to ODBC/Realtime, > > Also 16.0.1 to 16.3.0. After adding a few peers to the new RT box, I > > noticed a delay in call processing. All I had done thus far is added a > > few endpoints for upstream carriers, And one downstream for the legacy > > PBX it was replacing. After a bit of troubleshooting I found the issue > > to be related to having the identify section matching a domain name > > (Instead of an IP). Each endpoint with match=domain.name.here (The > > specific name didn't really matter) added a bit of noticeable latency. > > Once you were up to 4-5 of them, It was a ~5 second delay from the time > > the invite came in until the time the call was routed and stood up with > > audio. Even when you were routing between unrelated (non-dns matched) > > endpoints. Looking at the debug logs, It seems like Asterisk looks > > these peers up on some kind of time interval. As well as during _every_ > > inbound call. This also explains why the delay increased with every > > single DNS based endpoint I added. Each extra peer means another DNS > > lookup and process. > > > > To further test my theory, I stood up a local caching DNS resolver and > > this significantly improved the delay. But it was still ~1 second with > > 5 DNS based endpoints. Even with a caching resolver, This doesn't > > scale. Doing a large number (and growing with scale) of DNS lookups for > > every call (Maybe even every packet) just doesn't fly. > > > > So my questions are: > > 1. Anyone have any idea why asterisk does this when using realtime but > > NOT when using flatfiles? > > Unless you configure caching for the objects[1] then every time the > identify object is pulled from the database it has to be looked up and > resolved.Oh, and it would need to be full backend caching. Just per-object caching doesn't do the trick because in order to iterate over all the identifies, you need all of them including any uncached ones. The full backend caching on the other hand caches everything so if I recall it doesn't hit the backend in that case. -- Joshua C. Colp Digium - A Sangoma Company | Senior Software Developer 445 Jan Davis Drive NW - Huntsville, AL 35806 - US Check us out at: www.digium.com & www.asterisk.org