We're encountering some problems on FreeBSD 6.1 SMP with a number of end-user systems where a Postgresql database is growing to the point where the filesystem is nearly full or completely runs out of free space. After this point, df shows wildly incorrect values for available space, sometimes even showing more space available than there allocated to the partition (and our systems rely on df to determine when to send warnings to end-users that the free disk space is getting low). We've found the only way to correct the situation is to take the system down to single-user mode and force a non-background fsck on the partition - simply rebooting the system does not fix the situation. We're going to try setting debug.mpsafevfs to 0 in /boot/loader.conf to see if this works around the problem. I've gone through the CVS logs for src/sys/ufs/ffs/* but I haven't seen anything that would specifically apply to this problem. Has anyone else seen this problem, and by chance is it resolved for 6.2? Thanks, Guy -- Guy Helmer, Ph.D. Chief System Architect Palisade Systems, Inc.
Kris Kennaway wrote:> On Tue, Nov 28, 2006 at 10:45:59AM -0600, Guy Helmer wrote: > >> We're encountering some problems on FreeBSD 6.1 SMP with a number of >> end-user systems where a Postgresql database is growing to the point >> where the filesystem is nearly full or completely runs out of free >> space. After this point, df shows wildly incorrect values for available >> space, sometimes even showing more space available than there allocated >> to the partition (and our systems rely on df to determine when to send >> warnings to end-users that the free disk space is getting low). We've >> found the only way to correct the situation is to take the system down >> to single-user mode and force a non-background fsck on the partition - >> simply rebooting the system does not fix the situation. >> >> We're going to try setting debug.mpsafevfs to 0 in /boot/loader.conf to >> see if this works around the problem. I've gone through the CVS logs >> for src/sys/ufs/ffs/* but I haven't seen anything that would >> specifically apply to this problem. Has anyone else seen this problem, >> and by chance is it resolved for 6.2? >> > > First I've heard of it. May not be resolved unless a developer can > reproduce it. > > Kris >I had hoped one of kib's or tegge's recent changes to src/sys/ufs/ffs/ffs_softdep.c might be related. I'll see what I can do to get hard details - maybe a kernel with INVARIANTS would find something. Guy -- Guy Helmer, Ph.D. Chief System Architect Palisade Systems, Inc.
Kris Kennaway
2006-Nov-28 12:38 UTC
6.x loosing record of free space after filesystem fills?
On Tue, Nov 28, 2006 at 10:45:59AM -0600, Guy Helmer wrote:> We're encountering some problems on FreeBSD 6.1 SMP with a number of > end-user systems where a Postgresql database is growing to the point > where the filesystem is nearly full or completely runs out of free > space. After this point, df shows wildly incorrect values for available > space, sometimes even showing more space available than there allocated > to the partition (and our systems rely on df to determine when to send > warnings to end-users that the free disk space is getting low). We've > found the only way to correct the situation is to take the system down > to single-user mode and force a non-background fsck on the partition - > simply rebooting the system does not fix the situation. > > We're going to try setting debug.mpsafevfs to 0 in /boot/loader.conf to > see if this works around the problem. I've gone through the CVS logs > for src/sys/ufs/ffs/* but I haven't seen anything that would > specifically apply to this problem. Has anyone else seen this problem, > and by chance is it resolved for 6.2?First I've heard of it. May not be resolved unless a developer can reproduce it. Kris -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 187 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20061128/1d3ca8be/attachment.pgp
Kris Kennaway wrote:> On Tue, Nov 28, 2006 at 10:45:59AM -0600, Guy Helmer wrote: > >> We're encountering some problems on FreeBSD 6.1 SMP [...]. >> >> df shows wildly incorrect values for available >> space, sometimes even showing more space available than there is allocated >> to the partition[...] > First I've heard of it. May not be resolved unless a developer can > reproduce it. > > Kris >I think we've finally found the cause of the problem - it wasn't just occurring after heavy use, but was visible right after filesystem creation! We regularly built new filesystems with "newfs -U -O 1 -b 65536 -f 8192" (a couple of our older products are still based on FreeBSD 4, so we built UFS1 filesystems for all our products). It seems that newfs with the -O1 option creates inconsistent metadata: fsck on a large, newly-built UFS1 filesystem complains in phase 5 with "SUMMARY INFORMATION BAD". I've filed a PR on it. Guy