Hi all!
I'm stuck on something really bizarre that is happening to a product
I "own" at work. It's a C program, built on CentOS, runs on CentOs
or
RHEL, has been in circulation since the early 00's, is in use at
hundreds of sites.
recently, at multiple customer sites it has started just going away.
no core file (yes, ulimit is configured), nothing in any of its
(several) log files. it's just gone.
running it under strace until it dies reveals that every thread has
been given a SIGKILL.
How does one figure out who deliverd a SIGKILL? For other, non-fatal,
signals it is possible to glean the PID of the sending process in a
signal handler, but obviously you can't do that for SIGKILL because
the app doesn't survive the signal.
I'm grasping at straws here, and am open to almost any kind of
suggestion that can be followed-up (as compared to "beats me" which
is where I am now).
I'm even wondering if systemd has something to do with it.
Thanks in advance!
--
---- Fred Smith -- fredex at fcshome.stoneham.ma.us
-----------------------------
But God demonstrates his own love for us in this:
While we were still sinners,
Christ died for us.
------------------------------- Romans 5:8 (niv) ------------------------------
Try checking your /var/log/messages for OOM killer log lines. If your machine is
running low on memory the oom killer will start killing high memory usage
programs.
Grant
________________________________________
From: CentOS <centos-bounces at centos.org> on behalf of Fred Smith
<fredex at fcshome.stoneham.ma.us>
Sent: Tuesday, 6 August 2019 10:57 AM
To: centos at centos.org
Subject: [CentOS] another bizarre thing...
Hi all!
I'm stuck on something really bizarre that is happening to a product
I "own" at work. It's a C program, built on CentOS, runs on CentOs
or
RHEL, has been in circulation since the early 00's, is in use at
hundreds of sites.
recently, at multiple customer sites it has started just going away.
no core file (yes, ulimit is configured), nothing in any of its
(several) log files. it's just gone.
running it under strace until it dies reveals that every thread has
been given a SIGKILL.
How does one figure out who deliverd a SIGKILL? For other, non-fatal,
signals it is possible to glean the PID of the sending process in a
signal handler, but obviously you can't do that for SIGKILL because
the app doesn't survive the signal.
I'm grasping at straws here, and am open to almost any kind of
suggestion that can be followed-up (as compared to "beats me" which
is where I am now).
I'm even wondering if systemd has something to do with it.
Thanks in advance!
--
---- Fred Smith -- fredex at fcshome.stoneham.ma.us
-----------------------------
But God demonstrates his own love for us in this:
While we were still sinners,
Christ died for us.
------------------------------- Romans 5:8 (niv) ------------------------------
_______________________________________________
CentOS mailing list
CentOS at centos.org
https://clicktime.symantec.com/39tX9Zv3dbX6w8rkcpnA46w7Vc?u=https%3A%2F%2Flists.centos.org%2Fmailman%2Flistinfo%2Fcentos
--
Grant Street
Senior Systems Engineer
T: +61 2 9383 4800 (main)
D: +61 2 8310 3582 (direct)
E: Grant.Street at al.com.au
Building 54 / FSA #19, Fox Studios Australia, 38 Driver Avenue
Moore Park, NSW 2021
AUSTRALIA
[LinkedIn] <https://www.linkedin.com/company/animal-logic> [Facebook]
<https://www.facebook.com/Animal-Logic-129284263808191/> [Twitter]
<https://twitter.com/AnimalLogic> [Instagram]
<https://www.instagram.com/animallogicstudios/>
[Animal Logic]<http://www.animallogic.com>
www.animallogic.com<http://www.animallogic.com>
CONFIDENTIALITY AND PRIVILEGE NOTICE
This email is intended only to be read or used by the addressee. It is
confidential and may contain privileged information. If you are not the intended
recipient, any use, distribution, disclosure or copying of this email is
strictly prohibited. Confidentiality and legal privilege attached to this
communication are not waived or lost by reason of the mistaken delivery to you.
If you have received this email in error, please delete it and notify us
immediately by telephone or email.
On Tue, Aug 06, 2019 at 01:54:56AM +0000, Grant Street wrote:> Try checking your /var/log/messages for OOM killer log lines. If your machine is running low on memory the oom killer will start killing high memory usage programs. > > Grantwe have watched top while it runs and there's no evidence of a memory shortage.> ________________________________________ > From: CentOS <centos-bounces at centos.org> on behalf of Fred Smith <fredex at fcshome.stoneham.ma.us> > Sent: Tuesday, 6 August 2019 10:57 AM > To: centos at centos.org > Subject: [CentOS] another bizarre thing... > > Hi all! > > I'm stuck on something really bizarre that is happening to a product > I "own" at work. It's a C program, built on CentOS, runs on CentOs or > RHEL, has been in circulation since the early 00's, is in use at > hundreds of sites. > > recently, at multiple customer sites it has started just going away. > no core file (yes, ulimit is configured), nothing in any of its > (several) log files. it's just gone. > > running it under strace until it dies reveals that every thread has > been given a SIGKILL. > > How does one figure out who deliverd a SIGKILL? For other, non-fatal, > signals it is possible to glean the PID of the sending process in a > signal handler, but obviously you can't do that for SIGKILL because > the app doesn't survive the signal. > > I'm grasping at straws here, and am open to almost any kind of > suggestion that can be followed-up (as compared to "beats me" which > is where I am now). > > I'm even wondering if systemd has something to do with it. > > Thanks in advance! > -- > ---- Fred Smith -- fredex at fcshome.stoneham.ma.us ----------------------------- > But God demonstrates his own love for us in this: > While we were still sinners, > Christ died for us. > ------------------------------- Romans 5:8 (niv) ------------------------------ > _______________________________________________ > CentOS mailing list > CentOS at centos.org > https://clicktime.symantec.com/39tX9Zv3dbX6w8rkcpnA46w7Vc?u=https%3A%2F%2Flists.centos.org%2Fmailman%2Flistinfo%2Fcentos > -- > Grant Street > Senior Systems Engineer > > T: +61 2 9383 4800 (main) > D: +61 2 8310 3582 (direct) > E: Grant.Street at al.com.au > > Building 54 / FSA #19, Fox Studios Australia, 38 Driver Avenue > Moore Park, NSW 2021 > AUSTRALIA > > [LinkedIn] <https://www.linkedin.com/company/animal-logic> [Facebook] <https://www.facebook.com/Animal-Logic-129284263808191/> [Twitter] <https://twitter.com/AnimalLogic> [Instagram] <https://www.instagram.com/animallogicstudios/> > > [Animal Logic]<http://www.animallogic.com> > > www.animallogic.com<http://www.animallogic.com> > > CONFIDENTIALITY AND PRIVILEGE NOTICE > This email is intended only to be read or used by the addressee. It is confidential and may contain privileged information. If you are not the intended recipient, any use, distribution, disclosure or copying of this email is strictly prohibited. Confidentiality and legal privilege attached to this communication are not waived or lost by reason of the mistaken delivery to you. If you have received this email in error, please delete it and notify us immediately by telephone or email. > _______________________________________________ > CentOS mailing list > CentOS at centos.org > https://lists.centos.org/mailman/listinfo/centos-- ---- Fred Smith -- fredex at fcshome.stoneham.ma.us ----------------------------- God made him who had no sin to be sin for us, so that in him we might become the righteousness of God." --------------------------- Corinthians 5:21 ---------------------------------
On Aug 5, 2019, at 6:57 PM, Fred Smith <fredex at fcshome.stoneham.ma.us> wrote:> > no core file (yes, ulimit is configured)That?s nowhere near sufficient. To restore classic core file dumps on CentOS 7, you must: 1. Remove Red Hat?s ABRT system, which wants to catch all of this and handle it directly. Say something like ?sudo yum remove abrt*? 2. Override the default sysctl telling where core dumps land by writing this file, /etc/sysctl.d/10-core.conf: kernel.core_pattern = /tmp/core-%e-%p kernel.core_uses_pid = 1 fs.suid_dumpable = 2 Then apply those settings with ?sudo sysctl ?system?. I don?t remember what the default is, which this overrides, but I definitely didn?t want it. You can choose any pattern you like, just remember what permissions the service runs under, because that?s the permission needed by the process that actually dumps the core to make the file hit the disk. That?s why I chose /tmp in this example: anyone can write there. 3. Raise the limits by writing the following to /etc/security/limits.d/10-core.conf: * hard core unlimited * soft core unlimited If this is what you meant by ?ulimit,? then great, but I suspect you actually meant ?ulimit -c unlimited?, but I believe until you do the above, the ulimit CLI app can have no effect. You have to log out and back in to make this take effect. Once the above is done, ?ulimit -c unlimited? can take effect, but it?s of no value at all in conjunction with systemd services, for example, since those don?t run under a standard shell, so your .bash_profile and such aren?t even exec?d. 4. If your program is launched via systemd, then you must edit /etc/systemd/system.conf and set DefaultLimitCORE=infinity then say ?sudo systemctl daemon-reeexec? Case matters; ?Core? won?t work. Ask me how I know. :) 5. If you have a systemd unit file for your service, you have to set a related value in there as well: LimitCore=infinity You need both because #4 sets the system-wide cap, while this sets the per-service value, which can go no higher than the system cap. 6. Restart the service to apply the above two changes. Yes, it really is that difficult to enable classic core dumps on CentOS 7. You?re welcome. :)
On Tue, 2019-08-06 at 05:27 -0600, Warren Young wrote:> On Aug 5, 2019, at 6:57 PM, Fred Smith <fredex at fcshome.stoneham.ma.us> wrote: > > no core file (yes, ulimit is configured) > > That?s nowhere near sufficient. To restore classic core file dumps > on CentOS 7, you must: >I was under the impression that a SIGKILL doesn't trigger a core dump anyway. It just kills the process. P.
Wow, thanks for the detailed recipe! How did we deserve this when it was so easy in the past :-)> On Aug 5, 2019, at 6:57 PM, Fred Smith <fredex at fcshome.stoneham.ma.us> > wrote: >> >> no core file (yes, ulimit is configured) > > That?s nowhere near sufficient. To restore classic core file dumps on > CentOS 7, you must: > > 1. Remove Red Hat?s ABRT system, which wants to catch all of this and > handle it directly. Say something like ?sudo yum remove abrt*? > > > 2. Override the default sysctl telling where core dumps land by writing > this file, /etc/sysctl.d/10-core.conf: > > kernel.core_pattern = /tmp/core-%e-%p > kernel.core_uses_pid = 1 > fs.suid_dumpable = 2 > > Then apply those settings with ?sudo sysctl ?system?. > > I don?t remember what the default is, which this overrides, but I > definitely didn?t want it. > > You can choose any pattern you like, just remember what permissions the > service runs under, because that?s the permission needed by the process > that actually dumps the core to make the file hit the disk. That?s why I > chose /tmp in this example: anyone can write there. > > > 3. Raise the limits by writing the following to > /etc/security/limits.d/10-core.conf: > > * hard core unlimited > * soft core unlimited > > If this is what you meant by ?ulimit,? then great, but I suspect you > actually meant ?ulimit -c unlimited?, but I believe until you do the > above, the ulimit CLI app can have no effect. You have to log out and > back in to make this take effect. > > Once the above is done, ?ulimit -c unlimited? can take effect, but it?s of > no value at all in conjunction with systemd services, for example, since > those don?t run under a standard shell, so your .bash_profile and such > aren?t even exec?d. > > > 4. If your program is launched via systemd, then you must edit > /etc/systemd/system.conf and set > > DefaultLimitCORE=infinity > > then say ?sudo systemctl daemon-reeexec? > > Case matters; ?Core? won?t work. Ask me how I know. :) > > > 5. If you have a systemd unit file for your service, you have to set a > related value in there as well: > > LimitCore=infinity > > You need both because #4 sets the system-wide cap, while this sets the > per-service value, which can go no higher than the system cap. > > > 6. Restart the service to apply the above two changes. > > > Yes, it really is that difficult to enable classic core dumps on CentOS 7. > You?re welcome. :) > _______________________________________________ > CentOS mailing list > CentOS at centos.org > https://lists.centos.org/mailman/listinfo/centos >
On Tue, Aug 06, 2019 at 05:27:54AM -0600, Warren Young wrote:> On Aug 5, 2019, at 6:57 PM, Fred Smith <fredex at fcshome.stoneham.ma.us> wrote: > > > > no core file (yes, ulimit is configured)yeah, I meant "ulimit -c unlimited" is in effect. I had no idea systemd had made such a drastic change. or is it that someone at RH decided to make it (nearly) impossible to do? I fail to see how it is beneficial to anyone to make it so hard to get core dump files. but thanks for the details! Fred> > That?s nowhere near sufficient. To restore classic core file dumps on CentOS 7, you must: > > 1. Remove Red Hat?s ABRT system, which wants to catch all of this and handle it directly. Say something like ?sudo yum remove abrt*? > > > 2. Override the default sysctl telling where core dumps land by writing this file, /etc/sysctl.d/10-core.conf: > > kernel.core_pattern = /tmp/core-%e-%p > kernel.core_uses_pid = 1 > fs.suid_dumpable = 2 > > Then apply those settings with ?sudo sysctl ?system?. > > I don?t remember what the default is, which this overrides, but I definitely didn?t want it. > > You can choose any pattern you like, just remember what permissions the service runs under, because that?s the permission needed by the process that actually dumps the core to make the file hit the disk. That?s why I chose /tmp in this example: anyone can write there. > > > 3. Raise the limits by writing the following to /etc/security/limits.d/10-core.conf: > > * hard core unlimited > * soft core unlimited > > If this is what you meant by ?ulimit,? then great, but I suspect you actually meant ?ulimit -c unlimited?, but I believe until you do the above, the ulimit CLI app can have no effect. You have to log out and back in to make this take effect. > > Once the above is done, ?ulimit -c unlimited? can take effect, but it?s of no value at all in conjunction with systemd services, for example, since those don?t run under a standard shell, so your .bash_profile and such aren?t even exec?d. > > > 4. If your program is launched via systemd, then you must edit /etc/systemd/system.conf and set > > DefaultLimitCORE=infinity > > then say ?sudo systemctl daemon-reeexec? > > Case matters; ?Core? won?t work. Ask me how I know. :) > > > 5. If you have a systemd unit file for your service, you have to set a related value in there as well: > > LimitCore=infinity > > You need both because #4 sets the system-wide cap, while this sets the per-service value, which can go no higher than the system cap. > > > 6. Restart the service to apply the above two changes. > > > Yes, it really is that difficult to enable classic core dumps on CentOS 7. You?re welcome. :) > _______________________________________________ > CentOS mailing list > CentOS at centos.org > https://lists.centos.org/mailman/listinfo/centos-- ---- Fred Smith -- fredex at fcshome.stoneham.ma.us ----------------------------- "For him who is able to keep you from falling and to present you before his glorious presence without fault and with great joy--to the only God our Savior be glory, majesty, power and authority, through Jesus Christ our Lord, before all ages, now and forevermore! Amen." ----------------------------- Jude 1:24,25 (niv) -----------------------------
Fred Smith wrote:> > Hi all! > > I'm stuck on something really bizarre that is happening to a product > I "own" at work. It's a C program, built on CentOS, runs on CentOs or > RHEL, has been in circulation since the early 00's, is in use at > hundreds of sites. > > recently, at multiple customer sites it has started just going away. > no core file (yes, ulimit is configured), nothing in any of its > (several) log files. it's just gone. > > running it under strace until it dies reveals that every thread has > been given a SIGKILL. > > How does one figure out who deliverd a SIGKILL? For other, non-fatal, > signals it is possible to glean the PID of the sending process in a > signal handler, but obviously you can't do that for SIGKILL because > the app doesn't survive the signal. > > I'm grasping at straws here, and am open to almost any kind of > suggestion that can be followed-up (as compared to "beats me" which > is where I am now). > > I'm even wondering if systemd has something to do with it.I had an issue a few years ago where 'something' was killing processes - I found it by writing a simple LD_PRELOAD hack that intercepted kill(2) and logged what is was doing via syslog before doing the actual kill - and used /etc/ld.so.preload to get it loaded by every process ... James Pearson
On Tue, Aug 06, 2019 at 03:49:29PM +0000, James Pearson wrote:> Fred Smith wrote: > > > > Hi all! > > > > I'm stuck on something really bizarre that is happening to a product > > I "own" at work. It's a C program, built on CentOS, runs on CentOs or > > RHEL, has been in circulation since the early 00's, is in use at > > hundreds of sites. > > > > recently, at multiple customer sites it has started just going away. > > no core file (yes, ulimit is configured), nothing in any of its > > (several) log files. it's just gone. > > > > running it under strace until it dies reveals that every thread has > > been given a SIGKILL. > > > > How does one figure out who deliverd a SIGKILL? For other, non-fatal, > > signals it is possible to glean the PID of the sending process in a > > signal handler, but obviously you can't do that for SIGKILL because > > the app doesn't survive the signal. > > > > I'm grasping at straws here, and am open to almost any kind of > > suggestion that can be followed-up (as compared to "beats me" which > > is where I am now). > > > > I'm even wondering if systemd has something to do with it. > > I had an issue a few years ago where 'something' was killing processes - > I found it by writing a simple LD_PRELOAD hack that intercepted kill(2) > and logged what is was doing via syslog before doing the actual kill - > and used /etc/ld.so.preload to get it loaded by every process ... > > James Pearson > > _______________________________________________ > CentOS mailing list > CentOS at centos.org > https://lists.centos.org/mailman/listinfo/centosJames: After posting my original mail, I found this URL: https://www.ibm.com/developerworks/community/blogs/aimsupport/entry/Finding_the_source_of_signals_on_Linux_with_strace_auditd_or_Systemtap?lang=en which shows a very simple recipe for programming system tap to report sigkills, the UID that sends it, and the target process. We've asked the customer who is helping troubleshoot to implement that and get back to us with the result. I suspect systemd has something to do with it, but I have absolutely no evidence, just a nagging feeling that since it has its little fingers in all the pies, it could be doing anything and I'd have no way of knowing. :( I try not to be one of the systemd bashers, but I seem to be losing that battle. Fred -- ------------------------------------------------------------------------------- .---- Fred Smith / ( /__ ,__. __ __ / __ : / / / / /__) / / /__) .+' Home: fredex at fcshome.stoneham.ma.us / / (__ (___ (__(_ (___ / :__ 781-438-5471 -------------------------------- Jude 1:24,25 ---------------------------------
On Mon, Aug 05, 2019 at 08:57:45PM -0400, Fred Smith wrote:> Hi all! > > I'm stuck on something really bizarre that is happening to a product > I "own" at work. It's a C program, built on CentOS, runs on CentOs or > RHEL, has been in circulation since the early 00's, is in use at > hundreds of sites. > > recently, at multiple customer sites it has started just going away. > no core file (yes, ulimit is configured), nothing in any of its > (several) log files. it's just gone. > > running it under strace until it dies reveals that every thread has > been given a SIGKILL. > > How does one figure out who deliverd a SIGKILL? For other, non-fatal, > signals it is possible to glean the PID of the sending process in a > signal handler, but obviously you can't do that for SIGKILL because > the app doesn't survive the signal. > > I'm grasping at straws here, and am open to almost any kind of > suggestion that can be followed-up (as compared to "beats me" which > is where I am now).OK, more information. Found a recipe to cause systemtap to emit a line of text identifying the sender of the SIGKILL. probe signal.send { if (sig_name == "SIGKILL") printf("%s was sent to %s (pid:%d) by %s uid:%d\n", sig_name, pid_name, sig_pid, execname(), uid()) unfortunately, it says the program is killing itself: SIGKILL was sent to myprog (pid:12269) by myprog uid:1000 So,... now I'm wondering how one figures that out. nowhere in my source code does it explicitly raise any signal, much less SIGKILL. So there must be some underlying library or system call or something doing it. -- ---- Fred Smith -- fredex at fcshome.stoneham.ma.us ----------------------------- I can do all things through Christ who strengthens me. ------------------------------ Philippians 4:13 -------------------------------
On Wed, 7 Aug 2019 13:38:54 -0400 Fred Smith wrote:> So,... now I'm wondering how one figures that out.Since it's your program you have the source code. printf is your friend. Start adding printf statements (to console and/or to a file at your option) with status reports ("widget counting executing", "addition function executing", "huge explosion executing") and use that to find out where it quits. Add more printf's as needed to narrow it down. -- MELVILLE THEATRE ~ Real D 3D Digital Cinema ~ www.melvilletheatre.com
Is this on both EL6 and EL7? If only EL7, it could be control groups causing the issue. The idea of cgroups is to prevent zombie processes, but if you need your program to spawn another process then restart itself while the other process continues to run, you need to launch it in a different control group, or the shutdown of the parent process will also kill the child. In my case, we have an upgrade script which needs to get called, then shut down the calling process in order to upgrade it. For example: # Clear any errors in the upgrade control group. /bin/systemctl reset-failed upgrade-trigger # Launch the upgrader in its own control group. /bin/systemd-run --unit=upgrade-trigger --slice=upgrade-trigger /bin/bash /opt/myapp/Upgrade.sh "$1" "$2" If we don't do this, the upgrade fails as the upgrader get's terminated when the parent application is shut down. Gregory Young -----Original Message----- From: CentOS <centos-bounces at centos.org> On Behalf Of Fred Smith Sent: August 7, 2019 1:39 PM To: centos at centos.org Subject: Re: [CentOS] another bizarre thing... On Mon, Aug 05, 2019 at 08:57:45PM -0400, Fred Smith wrote:> Hi all! > > I'm stuck on something really bizarre that is happening to a product I > "own" at work. It's a C program, built on CentOS, runs on CentOs or > RHEL, has been in circulation since the early 00's, is in use at > hundreds of sites. > > recently, at multiple customer sites it has started just going away. > no core file (yes, ulimit is configured), nothing in any of its > (several) log files. it's just gone. > > running it under strace until it dies reveals that every thread has > been given a SIGKILL. > > How does one figure out who deliverd a SIGKILL? For other, non-fatal, > signals it is possible to glean the PID of the sending process in a > signal handler, but obviously you can't do that for SIGKILL because > the app doesn't survive the signal. > > I'm grasping at straws here, and am open to almost any kind of > suggestion that can be followed-up (as compared to "beats me" which is > where I am now).OK, more information. Found a recipe to cause systemtap to emit a line of text identifying the sender of the SIGKILL. probe signal.send { if (sig_name == "SIGKILL") printf("%s was sent to %s (pid:%d) by %s uid:%d\n", sig_name, pid_name, sig_pid, execname(), uid()) unfortunately, it says the program is killing itself: SIGKILL was sent to myprog (pid:12269) by myprog uid:1000 So,... now I'm wondering how one figures that out. nowhere in my source code does it explicitly raise any signal, much less SIGKILL. So there must be some underlying library or system call or something doing it. -- ---- Fred Smith -- fredex at fcshome.stoneham.ma.us ----------------------------- I can do all things through Christ who strengthens me. ------------------------------ Philippians 4:13 ------------------------------- _______________________________________________ CentOS mailing list CentOS at centos.org https://lists.centos.org/mailman/listinfo/centos
On Mon, Aug 05, 2019 at 08:57:45PM -0400, Fred Smith (fredex at fcshome.stoneham.ma.us) wrote:> Hi all! > > I'm stuck on something really bizarre that is happening to a product > I "own" at work. It's a C program, built on CentOS, runs on CentOs or > RHEL, has been in circulation since the early 00's, is in use at > hundreds of sites. > > recently, at multiple customer sites it has started just going away. > no core file (yes, ulimit is configured), nothing in any of its > (several) log files. it's just gone. >Late to the thread but since it has not been suggested: Have you tried to statically link all libs? Then use Frank Cox's suggestion to use printf's at location thoughout the source code. I know it will be big (depending on the number of libs) But this way you are sure that the compile is against a known (yours) set of libs! Also have you recompiled it and given the new binaries to the customers? Just an idea .. -- Jobst Schmalenbach Nice computers don't go down.
On Mon, Aug 12, 2019 at 10:16:35AM +1000, Jobst Schmalenbach wrote:> On Mon, Aug 05, 2019 at 08:57:45PM -0400, Fred Smith (fredex at fcshome.stoneham.ma.us) wrote: > > Hi all! > > > > I'm stuck on something really bizarre that is happening to a product > > I "own" at work. It's a C program, built on CentOS, runs on CentOs or > > RHEL, has been in circulation since the early 00's, is in use at > > hundreds of sites. > > > > recently, at multiple customer sites it has started just going away. > > no core file (yes, ulimit is configured), nothing in any of its > > (several) log files. it's just gone. > > > > Late to the thread but since it has not been suggested: Have you tried to statically link all libs?I doubt modern Linux systems will produce a fully-static binary, since many of the system libs come only as .so files.> > Then use Frank Cox's suggestion to use printf's at location thoughout the source code. > > I know it will be big (depending on the number of libs) > But this way you are sure that the compile is against a known (yours) set of libs! > > > Also have you recompiled it and given the new binaries to the customers?Yes, every time there's a new RHEL/CentOS version released it gets completely rebuilt on that new release. I don't depend on compatibility between releases. Not to mention as maintenance and feeping-creaturism* strikes. * for those not in the know: feeping-creaturism ==> creeping featurism -- ---- Fred Smith -- fredex at fcshome.stoneham.ma.us ----------------------------- Show me your ways, O LORD, teach me your paths; Guide me in your truth and teach me, for you are God my Savior, And my hope is in you all day long. -------------------------- Psalm 25:4-5 (NIV) --------------------------------