asterisk users - Jan 2015 - Asterisk executable suddenly about 40KB larger - modules not working

Hi all

I have a strange issue with 1.8.11.0 on a production Asterisk machine at our
head office, and the same issue with a production machine at a branch
office.

Every now and then, on the head office machine, ODBC CEL and CDR logging
will stop working. On examination in the CLI, Asterisk behaves as if the
config files for ODBC in the /etc directory are just gone.

Repeated tests have then proved that the config files
(/etc/asterisk/res_odbc.conf, /etc/asterisk/res_pgsql.conf, etc.) ARE in
/etc/asterisk folder and are readable and have the correct contents, and are
NOT gone.

On the branch machine, where we do not use ODBC but FreeTDS to log CDRs to
an MSSQL DB, TDS stops working randomly as well, with the cdr_tds.so module
refusing to load with a message (I forget now the exact wording) that seems
to indicate that the Asterisk version is incompatible with the cdr_tds.so
ELF object file.

Checking further, I discovered that in both situations, the asterisk
executable in /usr/sbin grew by about 40KB compared to its size just after
being compiled...

The fix on both machines is to re-copy a backup of the asterisk executable
to /usr/sbin to overwrite the new "suddenly larger" asterisk
executable, and
then restarting asterisk on both machines.

Everything then works correctly again until the next time the
/usr/sbin/asterisk executable again "grows" by +- 40kb - at Head
Office
stopping ODBC from working, at the branch stopping TDS from working.

This doesn't happen with our other 14 branches all running 1.8.11.0 on
mostly identical hardware.

Anybody encountered this "growing executable" error before? 

Thanks

Stefan

>> Anybody encountered this "growing executable" error before? 
I would guess that those systems have been compromised.

You should review the logs.

http://serverfault.com/questions/2783/how-do-i-know-if-my-linux-server-has-been-hacked

You can also try making the Asterisk executable immutable with chattr

http://www.aboutlinux.info/2005/11/make-your-files-immutable-which-even.html

Doug

In article <001a01d02a75$cf314fc0$6d93ef40$@verishare.co.za>,
Stefan Viljoen <viljoens at verishare.co.za>
wrote:
> Hi all
> 
> I have a strange issue with 1.8.11.0 on a production Asterisk machine at
our
> head office, and the same issue with a production machine at a branch
> office.
> 
> Every now and then, on the head office machine, ODBC CEL and CDR logging
> will stop working. On examination in the CLI, Asterisk behaves as if the
> config files for ODBC in the /etc directory are just gone.
> 
> Repeated tests have then proved that the config files
> (/etc/asterisk/res_odbc.conf, /etc/asterisk/res_pgsql.conf, etc.) ARE in
> /etc/asterisk folder and are readable and have the correct contents, and
are
> NOT gone.
> 
> On the branch machine, where we do not use ODBC but FreeTDS to log CDRs to
> an MSSQL DB, TDS stops working randomly as well, with the cdr_tds.so module
> refusing to load with a message (I forget now the exact wording) that seems
> to indicate that the Asterisk version is incompatible with the cdr_tds.so
> ELF object file.
> 
> Checking further, I discovered that in both situations, the asterisk
> executable in /usr/sbin grew by about 40KB compared to its size just after
> being compiled...
> 
> The fix on both machines is to re-copy a backup of the asterisk executable
> to /usr/sbin to overwrite the new "suddenly larger" asterisk
executable, and
> then restarting asterisk on both machines.
> 
> Everything then works correctly again until the next time the
> /usr/sbin/asterisk executable again "grows" by +- 40kb - at Head
Office
> stopping ODBC from working, at the branch stopping TDS from working.
> 
> This doesn't happen with our other 14 branches all running 1.8.11.0 on
> mostly identical hardware.
> 
> Anybody encountered this "growing executable" error before? 
It could be something to do with pre-linking. See "man prelink". This
is usually run from /etc/cron.daily

You can disable pre-linking by following the instructions here:
http://www.builddesigncreate.com/index.cgi?mode=webpage_list&pageid=2011080413332724848

If that prevents the problem, the next step would be to determine why
pre-linking causes the problem, although I'm not sure how you do so.

Cheers
Tony
-- 
Tony Mountifield
Work: tony at softins.co.uk - http://www.softins.co.uk
Play: tony at mountifield.org - http://tony.mountifield.org

On Wednesday 07 Jan 2015, Stefan Viljoen wrote:
> Hi all
> 
> I have a strange issue with 1.8.11.0 on a production Asterisk machine at
> our head office, and the same issue with a production machine at a branch
> office.
> 
> Every now and then, on the head office machine, ODBC CEL and CDR logging
> will stop working. On examination in the CLI, Asterisk behaves as if the
> config files for ODBC in the /etc directory are just gone.
> 
> Repeated tests have then proved that the config files
> (/etc/asterisk/res_odbc.conf, /etc/asterisk/res_pgsql.conf, etc.) ARE in
> /etc/asterisk folder and are readable and have the correct contents, and
> are NOT gone.
> 
> Checking further, I discovered that in both situations, the asterisk
> executable in /usr/sbin grew by about 40KB compared to its size just after
> being compiled...
This sounds suspiciously as though you have some kind of rootkit-like 
infection.  Which probably is trying to make calls at your expense, and 
without even doing you the courtesy of recording the fact of them being made 
in the usual database.

You are going to need to get your hands dirty, tracing system operations .....  
You want to look for a write to /usr/sbin/asterisk .

-- 
AJS

Note:  Originating address only accepts e-mail from list!  If replying off-
list, change address to asterisk1list at earthshod dot co dot uk .