asterisk users - Jul 2012 - res_odbc crashing asterisk after freetds dsn reconnects

I have an Asterisk Open Source 10 system set up that is using res_odbc to
connect to a MSSQL database so that our users can clock in/out on our timeclock
system from their phones.  I've been having a consistent issue with Asterisk
crashing (completely restarting and dropping active calls) when there is a
network disruption that severs the connection between Asterisk and the MSSQL
server while someone is trying to punch the timeclock.

The setup is as follows:
Asterisk 10.4 (also had same issues on 10.2) running on CentOS 6.2 (VM on a
CentOS 6.3 KVM host cluster) - connected to Voice VLAN

-          freetds installed from epel yum repository, 0.91-2.el6 (most current
version available on epel)

-          unixODBC & unixODBC-devel 2.2.14-11.el6 installed

-          Asterisk also has an ODBC connection to a local MySQL server
configured and in use for a separate purpose
MSSQL 2008 R2 running on Server 2008 R2 (VM on a CentOS 6.3 KVM host cluster) -
connected to Data VLAN

The res_odbc.conf file:
[my_freetds_dsn]
enabled => yes
dsn => my_freetds_dsn
username => my_freetds_user
password => my_freetds_password
pre-connect => yes
sanitysql => select 1

odbcinst.ini:

[FreeTDS]
Description         = ODBC for Microsoft SQL
Driver                    = /usr/lib64/libtdsodbc.so.0
UsageCount       = 1
Threading            = 2

odbc.ini:
[my_freetds_instance]
Description = my_freetds_instance
Driver = FreeTDS
Database = my_freetds_instanace
Server = my_mssql_server
Trace = no
TDS_Version = 7.2
Port = my_mssql_port
timeout = 10
connect_timeout = 5

The steps to replicate the crash are:

1)      Network disruption that prevents the Asterisk server from communicating
with the MSSQL server occurs.

2)      While the network disruption is ongoing, a user dials into the Asterisk
server's timeclock extension and inputs their employee ID, which causes
Asterisk to perform a lookup on the MSSQL server.

3)      Asterisk "hangs" for 3-5 minutes while it waits for the ODBC
connection to the MSSQL server.

4)      I get made aware of the problem and log in to Asterisk.

5)      I execute "module reload res_odbc.so" and Asterisk reconnects
successfully to the ODBC connection and can process new calls to the timeclock.

6)      The "hung" calls continue to show in "core show
channels" even after the user hangs up and tries again (for what it's
worth users, typically create 3-4 hung calls each before one or more of them let
me know.  I've seen anywhere from 5-20 hung calls at the times I've
logged in to try to reconnect the ODBC connection).

7)      Asterisk crashes during or shortly after the module reload.  Sometimes
I've sent one or more "channel request hangup" commands from the
Asterisk CLI for the hung calls.  Sometimes it crashes immediately on the module
reload, sometimes it runs for a few minutes after the reload.  I don't think
it's ever run more than 5 minutes after I reload the ODBC connections.

I do have a 91MB "core" file from yesterday's incident in my /tmp
directory (I assume that's a core dump from the crash?)

Thank you,

Noah Engelberth
System Administration
MetaLINK Technologies

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.digium.com/pipermail/asterisk-users/attachments/20120725/e455efdb/attachment.htm>

----- Original Message ----- 
> From: "Noah Engelberth" <Noah at directlinkcomputers.com>
> To: "Asterisk Users Mailing List - Non-Commercial Discussion"
> <asterisk-users at lists.digium.com>
> Sent: Wednesday, July 25, 2012 8:10:04 AM
> Subject: [asterisk-users] res_odbc crashing asterisk after freetds
> dsn reconnects
<snip>
> I do have a 91MB ?core? file from yesterday?s incident in my /tmp
> directory (I assume that?s a core dump from the crash?)
Please use the instructions linked below to generate a backtrace
from a core dump.

https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace

Note that you may need to re-compile Asterisk with the appropriate
build options to get a usable backtrace.

Once you have a backtrace, please file an issue on the issue tracker.

https://issues.asterisk.org/jira/

Thanks!

--
Matthew Jordan
Digium, Inc. | Software Developer
445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
Check us out at: http://digium.com & http://asterisk.org