Using kernel 2.4.20 with the following patches
ext3-scheduling-storm.patch
ext3-use-after-free.patch
sync_fs-fix-2.patch
sync_fs-fix.patch
sync_fs.patch
Note: this problem started happening before applying the patches
I have a small partition for / since I don't have much there.
Filesystem 1k-blocks Used Available Use% Mounted on
/dev/sda6 303344 98934 188749 35% /
After an average of 14 days / is completely filled. But doing a du on /
returns:
98934 /
but Available is 0
and Used is equal to the total available space - % reserved for root
If I reboot, then Use% goes back to 35% and available space goes back to
normal
After rebooting, when i do a read only fsck, i get something like this:
--------------
azbok {/home/kobaz} root# e2fsck -n /dev/sda6 -f
e2fsck 1.27 (8-Mar-2002)
Warning! /dev/sda6 is mounted.
Warning: skipping journal recovery because doing a read-only filesystem
check.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences: -8777
Fix? no
Free blocks count wrong for group #0 (23991, counted=23990).
Fix? no
Free blocks count wrong (3744071, counted=3744070).
Fix? no
/dev/sda6: ********** WARNING: Filesystem still has errors **********
/dev/sda6: 8235/78312 files (1.7% non-contiguous), 108826/313236 blocks
---------------
I thought it might be a filesystem problem, so i resized one of my
larger partitions for use as a new /
I did a cpio of everything in / to the new partition to make an exact
duplicate.
I booted with the new /
I did a full fsck of the new / and lo and behold I get similar problems
--------------------
Pass 5: Checking group summary information
Block bitmap differences: -(920--927) -(950--1006) -(1015--1038)
-(1040--1071) -(1073--1085)
--------------------
The longer the system is running, the more block bitmap differences
If left running for a few days, the block bitmap differences are a few
pages long.
I am completely clueless of what to do next. Having to reboot every two
weeks is getting annoying.
On Tue, Apr 08, 2003 at 12:56:21AM -0400, Mark wrote:> Using kernel 2.4.20 with the following patches > ext3-scheduling-storm.patch > ext3-use-after-free.patch > sync_fs-fix-2.patch > sync_fs-fix.patch > sync_fs.patch > > Note: this problem started happening before applying the patches > > I have a small partition for / since I don't have much there. > > Filesystem 1k-blocks Used Available Use% Mounted on > /dev/sda6 303344 98934 188749 35% / > > After an average of 14 days / is completely filled. But doing a du on / > returns: > > 98934 / > > but Available is 0 > and Used is equal to the total available space - % reserved for root > > If I reboot, then Use% goes back to 35% and available space goes back to > normalThis is an FAQ. The standard cause for this is some user process keeping a deleted file open. When this happens, the space is not visible via 'du', since the file is no longer visible in the directory tree. However, the space is still used by the file until it is deallocated, and that can only happen once the last process which has the file open either closes its file descriptor to the file, or the process exits. You can use the lsof program to try to find which process is keeping an open file. Usually it's some log file, or some large data base file which gets rotated out, but some older process are still keeping the log file open. - Ted P.S. See this URL for an example of the same problem under Open BSD: http://dbforums.com/arch/181/2002/9/510770 Here's another summary of answers to the same questions dating back to 1995 and SunOS. http://www.netsys.com/sunmgr/1995-09/msg00073.html P.P.S. Anyone want to volunteer to start and maintain an EXT3 faq? I'm not sure why this question keeps coming up all the time, but I'm getting tired of answering it again... and again... and again...
Hi, On Tue, 2003-04-08 at 05:56, Mark wrote:> If I reboot, then Use% goes back to 35% and available space goes back to > normalThat itself is a faq, and has already been replied to. But:> After rebooting, when i do a read only fsck, i get something like this: > > -------------- > azbok {/home/kobaz} root# e2fsck -n /dev/sda6 -f > e2fsck 1.27 (8-Mar-2002) > Warning! /dev/sda6 is mounted. > Warning: skipping journal recovery because doing a read-only filesystem > check.Looks like the filesystem was mounted read/write at this point. If so, fsck can do nothing except give you a rough hint about fs health --- it basically cannot do anything useful, because the fs will be changing beneath its feet. I'm astonished if you only see one or two errors in that case! You can only reliably fsck the root filesystem if it is not mounted for write. That requires either booting from a rescue CD, or booting with a kernel option like "init=/bin/sh" to drop you straight into a shell at boot time, rather than going through fs setup. Cheers, Stephen