Christopher Thorjussen
2008-Jan-03 12:09 UTC
[CentOS] Random files in homedir gets deleted
On one of my systems I seem to loose a file or two from time to time. Last night, one of my files (/home/online/sh/NattjobbPrivat.sh) was deleted/removed/vanished. Another time it was /home/online/sh/daemon that was deleted. But I can't seem to find anything strange in the logs or in the history, nor would any of my scripts running in crontab mess with those files. Where can I look for clues? And how do I enable audit for file operations in my home folder? /Christopher Thorjussen
On Thu, 3 Jan 2008 13:09:11 +0100 "Christopher Thorjussen" <Christopher.Thorjussen at carrot.no> wrote:> On one of my systems I seem to loose a file or two from time to time. > > Where can I look for clues?Is your system visible to the internet? Maybe it's running some kind of Apache with homedirs loosely enabled and one unsecure php script + one little h4x0r could do the trick? Is your file system sane? Is your hard drive(s) SMART-wise OK? CM -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
On Thu, 2008-01-03 at 13:09 +0100, Christopher Thorjussen wrote:> On one of my systems I seem to loose a file or two from time to time. > Last night, one of my files (/home/online/sh/NattjobbPrivat.sh) was > deleted/removed/vanished. Another time it was /home/online/sh/daemon > that was deleted. > > But I can't seem to find anything strange in the logs or in the history, > nor would any of my scripts running in crontab mess with those files. > > Where can I look for clues? And how do I enable audit for file > operations in my home folder?Although less common than "In The Day", if file system corruption was repaired by *fsck during boot or other times, the "lost+found" directory of the affected FS may contain some files with weird names, having mostly numbers. These numbers are i-node numbers associated with the "chunks" that appear in the lost+found directory. If you see these, it indicates that corruption occurred and was repaired. If the files are plain text, there is a possibility to reconstruct the original files via visual inspection and concatenation. Often they are mixed or binary and more difficult to reconstruct. If it's not corruption, finding the cause may not be so easy. Long ago, before the we had the plethora of tools now available, I found the cause of such a mystery with a multi-pronged approach. 1. Fire up the system accounting package (sar). 2. Start a "daemon" (really just a "nohup <your shell script name> &" that checked for the existence of the file every so many seconds. This if really nothing more than a if [ ! -r <file name> ] ; then <some commands> ; fi The "-r" can be replaced/compounded with other flags as desired. See "man bash", "CONDITIONAL EXPRESSIONS. When the file went missing, a "log" entry was generated. 3. Generate the sar reports and examine them to see what was running around that time. 4. As I did, go viciously beat the offending user that made a flawed script about the ears with a rather rigid clue bat. 5. Problem solved.> > > /Christopher Thorjussen > <snip sig stuff>HTH -- Bill
You can enable auditing to determine if the files are disappearing due to human/machine intervention (audit file system deletes) or if it is due to file system corruption (files disappear and no delete audits recorded). It may just be an errant rsync script. -Ross -----Original Message----- From: centos-bounces at centos.org <centos-bounces at centos.org> To: centos at centos.org <centos at centos.org> Sent: Thu Jan 03 07:09:11 2008 Subject: [CentOS] Random files in homedir gets deleted On one of my systems I seem to loose a file or two from time to time. Last night, one of my files (/home/online/sh/NattjobbPrivat.sh) was deleted/removed/vanished. Another time it was /home/online/sh/daemon that was deleted. But I can't seem to find anything strange in the logs or in the history, nor would any of my scripts running in crontab mess with those files. Where can I look for clues? And how do I enable audit for file operations in my home folder? /Christopher Thorjussen _______________________________________________ CentOS mailing list CentOS at centos.org http://lists.centos.org/mailman/listinfo/centos ______________________________________________________________________ This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender and permanently delete the original and any copy or printout thereof. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.centos.org/pipermail/centos/attachments/20080103/af801350/attachment-0003.html>
> > On one of my systems I seem to loose a file or two from time > to time. Last night, one of my files > (/home/online/sh/NattjobbPrivat.sh) was > deleted/removed/vanished. Another time it was > /home/online/sh/daemon that was deleted. > > But I can't seem to find anything strange in the logs or in > the history, nor would any of my scripts running in crontab > mess with those files. > > Where can I look for clues? And how do I enable audit for > file operations in my home folder? >If your system is capable, use the SMART tools to check your drive out (as CM suggests), something like this: smartctl -a /dev/sda <replace /dev/sda with the drive in question> See how your 'error count log' is doing. If there are errors, then you might want to run that command a few times and see if the error count is still rising. Is it everything in the /home/online/sh/ directory that is getting deleted, or can you see any pattern at all? (it sounds like it is random from what you said...but hard to think of why files would be deleted randomly....as you know!)
On Thursday 03 January 2008 19:09:11 Christopher Thorjussen wrote:> On one of my systems I seem to loose a file or two from time to time. > Last night, one of my files (/home/online/sh/NattjobbPrivat.sh) was > deleted/removed/vanished. Another time it was /home/online/sh/daemon > that was deleted. > > But I can't seem to find anything strange in the logs or in the history, > nor would any of my scripts running in crontab mess with those files. > > Where can I look for clues? And how do I enable audit for file > operations in my home folder?Hi, this really sounds weird. In order to audit it, the following checklist might help: 1. If the system was administered by an admin other than you and he got fired/dismissed with hard feeling on him, he might put a crontab that would do nasty thing randomly. Audit all the files in: /var/spool/cron /var/spool/at Also all the script in /etc/cron.{d,daily,weekly,monthly}, /etc/crontab 2, Audit all RPM files installed using: rpm -Va, looks for a difference in md5sum for binary files such as /bin/ls,/bin/ps, etc. You might want to use cracker detection script such as rkhunter. 3. Looks for the word "error" in log files: grep -r error /var/log See for related error such as filesystem corruption, etc 4. It's a long shot, but could be a misconfigured rsync script? HTH, pls let us know the result. -- Fajar Priyanto | Reg'd Linux User #327841 | Linux tutorial http://linux2.arinet.org 21:09:01 up 1:02, 2.6.22-14-generic GNU/Linux Let's use OpenOffice. http://www.openoffice.org The real challenge of teaching is getting your students motivated to learn. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. URL: <http://lists.centos.org/pipermail/centos/attachments/20080103/a0e1bcf3/attachment-0003.sig>
Christopher Thorjussen
2008-Jan-04 09:23 UTC
[CentOS] Random files in homedir gets deleted
> On Thursday 03 January 2008 19:09:11 Christopher Thorjussen wrote: > > On one of my systems I seem to loose a file or two from time totime.> > Last night, one of my files (/home/online/sh/NattjobbPrivat.sh) was > > deleted/removed/vanished. Another time it was /home/online/sh/daemon > > that was deleted. > > > > But I can't seem to find anything strange in the logs or in thehistory,> > nor would any of my scripts running in crontab mess with thosefiles.> > > > Where can I look for clues? And how do I enable audit for file > > operations in my home folder? > > Hi, this really sounds weird. In order to audit it, the following > checklist > might help: > 1. If the system was administered by an admin other than you and hegot> fired/dismissed with hard feeling on him, he might put a crontab that > would > do nasty thing randomly. Audit all the files in: > /var/spool/cron > /var/spool/at > Also all the script in /etc/cron.{d,daily,weekly,monthly},/etc/crontab No admin or anyone else with access have quit or been fired. The files and folders looks fine.> 2, Audit all RPM files installed using: > rpm -Va, looks for a difference in md5sum for binary files such > as /bin/ls,/bin/ps, etc. You might want to use cracker detectionscript> such as rkhunter.The files look fine. Some files are marked as MD5 mismatch but it's mostly config files I've changed. The only files I'm not sure of is: SM5....T /usr/share/rhn/rhn_applet/rhn_applet.pyc SM5....T /usr/share/rhn/rhn_applet/rhn_applet_animation.pyc SM5....T /usr/share/rhn/rhn_applet/rhn_applet_apt.pyc SM5....T /usr/share/rhn/rhn_applet/rhn_applet_dialogs.pyc SM5....T /usr/share/rhn/rhn_applet/rhn_applet_model.pyc SM5....T /usr/share/rhn/rhn_applet/rhn_applet_protocols.pyc SM5....T /usr/share/rhn/rhn_applet/rhn_applet_rpc.pyc SM5....T /usr/share/rhn/rhn_applet/rhn_applet_rpm.pyc SM5....T /usr/share/rhn/rhn_applet/rhn_applet_version.pyc SM5....T /usr/share/rhn/rhn_applet/rhn_applet_yum.pyc SM5....T /usr/share/rhn/rhn_applet/rhn_sources.pyc SM5....T /usr/share/rhn/rhn_applet/rhn_utils.pyc But I'm not running X so the applet isn't running.> > 3. Looks for the word "error" in log files: > grep -r error /var/log > See for related error such as filesystem corruption, etc[root at ora01 tmp]# grep -r error /var/log /var/log/Xorg.0.log: (WW) warning, (EE) error, (NI) not implemented, (??) unknown. /var/log/anaconda.log:* getting rpm error class /var/log/prelink.log:/usr/lib64/libgpg-error.so.0.1.3 0000003c50e00000-0000003c50f02878 /var/log/rpmpkgs.4:libgpg-error-1.0-1.x86_64.rpm /var/log/rpmpkgs.1:libgpg-error-1.0-1.x86_64.rpm /var/log/messages.2:Dec 17 08:13:10 ora01 kernel: daemon[1562]: segfault at 0000007fc0000000 rip 0000002a957af4b2 rsp 0000007fbfffe730 error 6 /var/log/scrollkeeper.log:I/O error : Attempt to load network entity http://scrollkeeper.sourceforge.net/dtds/scrollkeeper-omf-1.0/scrollkeep er-omf.dtd /var/log/scrollkeeper.log:I/O error : Attempt to load network entity http://scrollkeeper.sourceforge.net/dtds/scrollkeeper-omf-1.0/scrollkeep er-omf.dtd /var/log/scrollkeeper.log:I/O error : Attempt to load network entity http://scrollkeeper.sourceforge.net/dtds/scrollkeeper-omf-1.0/scrollkeep er-omf.dtd /var/log/scrollkeeper.log:I/O error : Attempt to load network entity http://scrollkeeper.sourceforge.net/dtds/scrollkeeper-omf-1.0/scrollkeep er-omf.dtd /var/log/scrollkeeper.log:I/O error : Attempt to load network entity http://scrollkeeper.sourceforge.net/dtds/scrollkeeper-omf-1.0/scrollkeep er-omf.dtd /var/log/scrollkeeper.log:I/O error : Attempt to load network entity http://scrollkeeper.sourceforge.net/dtds/scrollkeeper-omf-1.0/scrollkeep er-omf.dtd /var/log/rpmpkgs.2:libgpg-error-1.0-1.x86_64.rpm /var/log/Xorg.0.log.old: (WW) warning, (EE) error, (NI) not implemented, (??) unknown. /var/log/rpmpkgs.3:libgpg-error-1.0-1.x86_64.rpm /var/log/rpmpkgs:libgpg-error-1.0-1.x86_64.rpm /var/log/anaconda.xlog: (WW) warning, (EE) error, (NI) not implemented, (??) unknown. /var/log/anaconda.xlog:error opening security policy file /etc/X11/xserver/SecurityPolicy> 4. It's a long shot, but could be a misconfigured rsync script?Rsync is not running/used, but some custom scripts are running cleaning up some folders. I'm trying to battle through them to see if somethings wrong in them, but so far I've found nothing.> HTH, pls let us know the result.Will do. /Christopher
Christopher Thorjussen
2008-Jan-04 09:25 UTC
[CentOS] Random files in homedir gets deleted
> You can enable auditing to determine if the files are disappearing dueto human/machine intervention (audit file system deletes) or if it is due to file system corruption (files disappear and no delete audits recorded).> > It may just be an errant rsync script. > > -RossHow do I enable auditing of the home dir? /Christopher
Do what I do when I need to setup a new Linux facility. Google "linux audit" I remember getting a good hit near the top with that. There are cli tools for adding files/folders/mounts to the audit system and you can tailor which type of activity to audit. It's no where as difficult to do as it sounds. -Ross -----Original Message----- From: centos-bounces at centos.org <centos-bounces at centos.org> To: CentOS mailing list <centos at centos.org> Sent: Fri Jan 04 04:25:17 2008 Subject: RE: [CentOS] Random files in homedir gets deleted> You can enable auditing to determine if the files are disappearing dueto human/machine intervention (audit file system deletes) or if it is due to file system corruption (files disappear and no delete audits recorded).> > It may just be an errant rsync script. > > -RossHow do I enable auditing of the home dir? /Christopher _______________________________________________ CentOS mailing list CentOS at centos.org http://lists.centos.org/mailman/listinfo/centos ______________________________________________________________________ This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender and permanently delete the original and any copy or printout thereof. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.centos.org/pipermail/centos/attachments/20080104/6ea7fc4b/attachment-0003.html>