asterisk users - Apr 2004 - Questions about alarm reporting in Asterisk

I am currently helping a friend build an Asterisk PBX that spans
several cities using anything from T1s to DSL connections to
link remote SIP phones, IAX gateways, etc. to a central Asterisk
PBX server that serves up voicemail, features, etc.  The biggest problem
that I have had with this system appears to be the leading problem that
my day job company finds with their VOIP deployments:  Most common
problems are on the infrastructure network but are reported as "phone
system
problems" because that is the piece that the customer directly interacts 
with.
I'm interested in hearing success stories in tying things like Asterisk 
YELLOW
and RED alarms and network problems into a central alarm reporting solution.

The most common problems that I have found are:
1. Someone unplugs a X100P from the Dmarc and nobody knows until people
    complain that calls are not coming in.
2. A network span goes down and nobody knows until they can't send or 
receive
    calls on that span.

Here are some ideas that I have thought about so far:
1. Installing a basic SNMP agent on each Linux box and using a central SNMP
    manager to monitor each node.  This would give notice when a remote 
node became
    isolated from the monitoring network.
2. Rolling in Asterisk alarm logs into a syslog server or even as SNMP 
traps.

Any good ideas would be appreciated!

On Wed, 21 Apr 2004, Clif Jones wrote:
> I am currently helping a friend build an Asterisk PBX that spans
> several cities using anything from T1s to DSL connections to
> link remote SIP phones, IAX gateways, etc. to a central Asterisk
> PBX server that serves up voicemail, features, etc.  The biggest problem
> that I have had with this system appears to be the leading problem that
> my day job company finds with their VOIP deployments:  Most common
> problems are on the infrastructure network but are reported as "phone
system
> problems" because that is the piece that the customer directly
interacts
> with.
> I'm interested in hearing success stories in tying things like Asterisk
> YELLOW
> and RED alarms and network problems into a central alarm reporting
solution.
> 
> The most common problems that I have found are:
> 1. Someone unplugs a X100P from the Dmarc and nobody knows until people
>     complain that calls are not coming in.
> 2. A network span goes down and nobody knows until they can't send or 
> receive
>     calls on that span.
> 
> Here are some ideas that I have thought about so far:
> 1. Installing a basic SNMP agent on each Linux box and using a central SNMP
>     manager to monitor each node.  This would give notice when a remote 
> node became
>     isolated from the monitoring network.
> 2. Rolling in Asterisk alarm logs into a syslog server or even as SNMP 
> traps.
> 
The manager interface sends events when a channel/span goes into alarm.
A simple app collecting this data should be able to handle this for you

James

> Any good ideas would be appreciated!
We use a package called Nagios to monitor our servers, which works quite
well. It has the ability to track service and host dependencies so you don't
get flooded with a bunch of "service down" alerts when the real cause
is a
bad switch (or similar).

It would seem logical for someone (hah!) to write a res_snmp.c for asterisk
that would expose a lot of asterisk's internal data. This would seem a
logical step toward writing fully functional monitoring applications as
well. The module would allow clients to add themselves to the list and
receive traps, as well as check for the current status of various variables.

<brainstorming>
Okay, this may be over the top, but here goes. Write an asterisk application
that sends (and receives) status information to another box over the PSTN.
My idea is not only to use this as a way to verify that * is running, but as
a way to RELIABLY tell that a remote * box is actively accepting incoming
calls. It wouldn't have to be anything complicated, just a heartbeat and
some basic details to let the caller know that "yes, I'm alive and
accepting
calls over this line".

Simplified protocol:
1) Monitoring box calls up and says (in DTMF):
	#<my CallerID>#<extension I am trying to reach>#<I'm a
machine, so
reply in DTMF instead of voice>#<the secret code is>#
2) The remote box says
	#<your CallerID>#<Your DNIS>#<yes I will accept a call to that
number>#
3) Monitoring box acknowledges and disconnects
4) Remote box disconnects
5) Monitoring box decides whether it likes the answers it received and
performs actions accordingly.

</brainstorming>

--Ernest
> -----Original Message-----
> From: asterisk-users-admin@lists.digium.com 
> [mailto:asterisk-users-admin@lists.digium.com] On Behalf Of Clif Jones
> Sent: Wednesday, April 21, 2004 10:16 AM
> To: asterisk users
> Subject: [Asterisk-Users] Questions about alarm reporting in Asterisk
> 
> I am currently helping a friend build an Asterisk PBX that spans
> several cities using anything from T1s to DSL connections to
> link remote SIP phones, IAX gateways, etc. to a central Asterisk
> PBX server that serves up voicemail, features, etc.  The 
> biggest problem
> that I have had with this system appears to be the leading 
> problem that
> my day job company finds with their VOIP deployments:  Most common
> problems are on the infrastructure network but are reported 
> as "phone system
> problems" because that is the piece that the customer 
> directly interacts 
> with.
> I'm interested in hearing success stories in tying things 
> like Asterisk 
> YELLOW
> and RED alarms and network problems into a central alarm 
> reporting solution.
> 
> The most common problems that I have found are:
> 1. Someone unplugs a X100P from the Dmarc and nobody knows 
> until people
>     complain that calls are not coming in.
> 2. A network span goes down and nobody knows until they can't send or 
> receive
>     calls on that span.
> 
> Here are some ideas that I have thought about so far:
> 1. Installing a basic SNMP agent on each Linux box and using 
> a central SNMP
>     manager to monitor each node.  This would give notice 
> when a remote 
> node became
>     isolated from the monitoring network.
> 2. Rolling in Asterisk alarm logs into a syslog server or 
> even as SNMP 
> traps.
> 
> Any good ideas would be appreciated!
> 
> _______________________________________________
> Asterisk-Users mailing list
> Asterisk-Users@lists.digium.com
> http://lists.digium.com/mailman/listinfo/asterisk-users
> To UNSUBSCRIBE or update options visit:
>    http://lists.digium.com/mailman/listinfo/asterisk-users
>

> I'm interested in hearing success stories in tying things like Asterisk
> YELLOW
> and RED alarms and network problems into a central alarm reporting
solution.
> 
> The most common problems that I have found are:
> 1. Someone unplugs a X100P from the Dmarc and nobody knows until people
>     complain that calls are not coming in.
> 2. A network span goes down and nobody knows until they can't send or 
> receive
>     calls on that span.
> 
> Here are some ideas that I have thought about so far:
> 1. Installing a basic SNMP agent on each Linux box and using a central SNMP
>     manager to monitor each node.  This would give notice when a remote 
> node became
>     isolated from the monitoring network.
> 2. Rolling in Asterisk alarm logs into a syslog server or even as SNMP 
> traps.
> 
> Any good ideas would be appreciated!
There are lots of different ways to sense problems including those you've
mentioned. Others include:
- writing a small app that simply interrogates those interfaces that are
  important to the operation (iax2/udp, sip/udp, etc, send a crafted pkt
  and interpret the returned result. Port not open is obvious, no response
  is obvious, incorrect response is not so obvious)
- test call to an outside number once per five minutes, hourly, or whatever
  trips your trigger (outside number only needs to respond with something
  that is predictable, doesn't have to be a person or company)
- monitoring logs looking for keywords (may take some time to identify the
  appropriate keywords)

There are open source apps available that already address some of those, but
weren't written specifically for *.

Rich