JCARRIZOSA at Crutchfield.com
2009-Apr-09 22:51 UTC
[CentOS] What is writing to my filesystem
I have a CentOS 5.2 box that every few months runs out of drivespace on its root filesystem. Last time I manually searched and deleted some big files, but don't remember what they were or what wrote to them. The applications I'm aware of on the box don't write to /. Is there a way to find the files that get written to the most, or grow the most over time? Doing a df gives me a snapshot, but it seems clunky to keep track of the diff on that output over time. I can then see what processes write to them. Any other ideas on how to investigate this are welcome. Thanks much. Jorge
> -----Original Message----- > From: centos-bounces at centos.org > [mailto:centos-bounces at centos.org] On Behalf Of > JCARRIZOSA at Crutchfield.com > Sent: Thursday, April 09, 2009 18:52 > To: CentOS at centos.org > Subject: [CentOS] What is writing to my filesystem > > I have a CentOS 5.2 box that every few months runs out of > drivespace on its root filesystem. Last time I manually > searched and deleted some big files, but don't remember what > they were or what wrote to them. The applications I'm aware > of on the box don't write to /.Not much help, but every time we have "this" problem, it is in /var/log> > Is there a way to find the files that get written to the > most, or grow the most over time? Doing a df gives me a > snapshot, but it seems clunky to keep track of the diff on > that output over time. I can then see what processes write to > them. Any other ideas on how to investigate this are welcome.du -s /*> > Thanks much. > > Jorge > > _______________________________________________ > CentOS mailing list > CentOS at centos.org > http://lists.centos.org/mailman/listinfo/centos >-- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- - - - Jason Pyeron PD Inc. http://www.pdinc.us - - Principal Consultant 10 West 24th Street #100 - - +1 (443) 269-1555 x333 Baltimore, Maryland 21218 - - - -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- This message is copyright PD Inc, subject to license 20080407P00.
On Thu, 9 Apr 2009, JCARRIZOSA at Crutchfield.com wrote:> I have a CentOS 5.2 box that every few months runs out of drivespace on > its root filesystem. Last time I manually searched and deleted some big > files, but don't remember what they were or what wrote to them. The > applications I'm aware of on the box don't write to /. > > Is there a way to find the files that get written to the most, or grow > the most over time? Doing a df gives me a snapshot, but it seems clunky > to keep track of the diff on that output over time. I can then see what > processes write to them. Any other ideas on how to investigate this are > welcome.I like filelight for finding big consumers, unfortunately it requires Qt and X. What is possible as well is to go to your mountpoint that is problematic and then use: du -xh --max-depth=1 and then follow your gut-feeling in what you think is normal, and what is not :) Doing this on my own laptop just to show, I found out this: ---- [root at moria var]# du -xh --max-depth=1 554M ./lib 57M ./log 16K ./ftp 942M ./cache 16K ./games 24K ./yp 8.0K ./nis 508K ./run 8.0K ./tux 8.0K ./cvs 8.0K ./preserve 12K ./account 32K ./empty 1.1M ./spool 64M ./clamav 8.0K ./tmp 8.0K ./racoon 168K ./lock 8.6M ./www 16K ./mrepo 76K ./gdm 8.0K ./local 7.3G ./crash 140K ./named 8.0K ./opt 28K ./db 8.9G . ---- ./crash has size 7.3G ?? ---- [root at moria crash]# du -xh --max-depth=1 8.0K ./2009-02-27-11:59 3.7G ./2009-04-06-19:37 8.0K ./2009-01-23-16:56 3.7G ./2009-04-08-19:55 7.3G . [root at moria crash]# ls -l */* -r-------- 1 root root 4054784396 Apr 6 19:38 2009-04-06-19:37/vmcore -r-------- 1 root root 4054784396 Apr 8 19:57 2009-04-08-19:55/vmcore ---- So even when I didn't know anything was wrong, it helped me :) -- -- dag wieers, dag at centos.org, http://dag.wieers.com/ -- [Any errors in spelling, tact or fact are transmission errors]
JCARRIZOSA at Crutchfield.com wrote:> I have a CentOS 5.2 box that every few months runs out of drivespace on > its root filesystem. Last time I manually searched and deleted some big > files, but don't remember what they were or what wrote to them. The > applications I'm aware of on the box don't write to /. > > Is there a way to find the files that get written to the most, or grow > the most over time? Doing a df gives me a snapshot, but it seems clunky > to keep track of the diff on that output over time. I can then see what > processes write to them. Any other ideas on how to investigate this are > welcome.find / -mount -ctime -1 #files that changed today find / -mount -size +50M # files larger than 50 megs. -- Les Mikesell lesmikesell at gmail.com
At Thu, 9 Apr 2009 18:51:42 -0400 CentOS mailing list <centos at centos.org> wrote:> > I have a CentOS 5.2 box that every few months runs out of drivespace on > its root filesystem. Last time I manually searched and deleted some big > files, but don't remember what they were or what wrote to them. The > applications I'm aware of on the box don't write to /.You have /var and/or /tmp on separate filesystems? If not, that is where things are being written to. Specificly, look in /tmp, /var/log/, and under /var/spool. Is /var/tmp separate from /tmp? What about /usr/tmp? If you don't run the machine 24/7, logrotate might not be run, this would mean your log files (under /var/log) will grow and grow and grow ... If you don't have an email alias for root (I assume you don't login as root!), then /var/spool/mail/root will grow and grow and grow ... as well (all sorts of silly messages sent to root). It is also possible that stuff is accumulating under /var/spool/lpd as well, although this will be small files.> > Is there a way to find the files that get written to the most, or grow > the most over time? Doing a df gives me a snapshot, but it seems clunky > to keep track of the diff on that output over time. I can then see what > processes write to them. Any other ideas on how to investigate this are > welcome. > > Thanks much. > > Jorge > > _______________________________________________ > CentOS mailing list > CentOS at centos.org > http://lists.centos.org/mailman/listinfo/centos > >-- Robert Heller -- 978-544-6933 Deepwoods Software -- Download the Model Railroad System http://www.deepsoft.com/ -- Binaries for Linux and MS-Windows heller at deepsoft.com -- http://www.deepsoft.com/ModelRailroadSystem/
The only thing I know that will give you a instant snapshot of what's happening is "mtop". It shows you whayt apps are using your harddrive very similar to top with processes. David On 4/9/09, JCARRIZOSA at crutchfield.com <JCARRIZOSA at crutchfield.com> wrote:> I have a CentOS 5.2 box that every few months runs out of drivespace on > its root filesystem. Last time I manually searched and deleted some big > files, but don't remember what they were or what wrote to them. The > applications I'm aware of on the box don't write to /. > > Is there a way to find the files that get written to the most, or grow > the most over time? Doing a df gives me a snapshot, but it seems clunky > to keep track of the diff on that output over time. I can then see what > processes write to them. Any other ideas on how to investigate this are > welcome. > > Thanks much. > > Jorge > > _______________________________________________ > CentOS mailing list > CentOS at centos.org > http://lists.centos.org/mailman/listinfo/centos >
2009/4/10 <JCARRIZOSA at crutchfield.com>:> I have a CentOS 5.2 box that every few months runs out of drivespace on > its root filesystem. Last time I manually searched and deleted some big > files, but don't remember what they were or what wrote to them. The > applications I'm aware of on the box don't write to /. >I have a Centos 3 system, which has its /boot in the / filesystem, and after several kernel upgrades it gets full, so I have to deinstall old kernels.
From: "JCARRIZOSA at Crutchfield.com" <JCARRIZOSA at Crutchfield.com>> I have a CentOS 5.2 box that every few months runs out of drivespace on > its root filesystem. Last time I manually searched and deleted some big > files, but don't remember what they were or what wrote to them. The > applications I'm aware of on the box don't write to /. > Is there a way to find the files that get written to the most, or grow > the most over time? Doing a df gives me a snapshot, but it seems clunky > to keep track of the diff on that output over time. I can then see what > processes write to them. Any other ideas on how to investigate this are > welcome.Do you use logrotate? Do you use logrotate's compression? Maybe use some kind of snapshots, like: #!/bin/bash find /var -type f -printf "%k %p\n" > /tmp/usedspace.new if [ -f /tmp/usedspace.old ]; then cat /tmp/usedspace.old | while read LINE do set $LINE OLDSIZE=$1 OLDFILE=$2 NEWSIZE=`grep "$OLDFILE\$" /tmp/usedspace.new | cut -d " " -f1` if [ -n "$NEWSIZE" -a "$OLDSIZE" != "$NEWSIZE" ]; then echo "$OLDFILE: $OLDSIZE => $NEWSIZE" fi done fi mv -f /tmp/usedspace.new /tmp/usedspace.old JD
Apparently Analagous Threads
- Re: virt-resize: support to MBR logical partitions and some question
- Re: [RFC PATCH] resize: add support for MBR logical partitions some question
- [LLVMdev] Prevent unbounded memory consuption of long lived JIT processes
- [PATCH v2 00/13] virt-resize: add support for resizing MBR logical partitions
- [RFC][PATCH 3/3] Btrfs: improve truncation of btrfs