thr3ads.net - Ocfs2 users - [Ocfs2-users] No space left on device [Dec 2009]

If this information is useful, please help other people find it:
Share via:

Brad Plant

2009-Dec-16 11:00 UTC

[Ocfs2-users] No space left on device

On Sat, Feb 28, 2009 at 18:13:12 PST 2009, Joel Becker
wrote:> On Sat, Feb 28, 2009 at 12:09:37PM +0000, Nuno Fernandes wrote:
> > > 	That's rather odd.  What is the blocksize and cluster size
of
> > > this filesystem?  Can you send me the output of
'debugfs.ocfs2 -R "stat
> > > /.zbr" /dev/hda3'?
> > >
> > > Joel
> > 
> > Hi,
> > 
> > Well.. we've deleted over 275 000 files and the problem
disappeared... at
> > least for now. We'll just have to wait for that amount of files to
be added.
> 
> 	I wanted to see what .zbr looked like, because my hunch was that
> you were very fragmented and couldn't allocate enough contiguous space
> to create more extent blocks.  Deleting files would perhaps have freed
> some in-use extent blocks, thus making it work again.
> 	If you run into the problem again, we'll revisit.
I have this happen on different file systems running different mainline kernel
versions created with different versions of ocfs2-tools. I've seen 1 file
system "fill up" with only 40% odd in use according to df. What's
interesting is that while 1 node can't write anymore, another node (only 2
nodes mounted rw) will still be able to write to the FS, but neither will be
able to completely fill the FS.

I have created new ocfs2 file systems and the problem doesn't seem to occur
(or it gets very close be actually being full), but weeks/months later after
being in use, the problem develops. The FS I mentioned above that filled up at
~40% use was about 6 months old and you could say it did it's fair share of
"work".

I just dumped some 1GB files in a directory using dd to fill up a FS (full at
87%) to that I could collect some debugfs.ocfs2 information. Upon not being able
to create any further 1GB files (operations on the FS got really slow), I
successfully created some 10MB files. I then created 100+ 1MB files using dd.
What seemed strange is that dd kept returning "no space left on
device", but it would always write 995328 bytes of data to the file.

dd if=/dev/zero of=2156 bs=1k count=1k
dd: writing `2156': No space left on device
973+0 records in
972+0 records out
995328 bytes (995 kB) copied, 0.0551648 s, 18.0 MB/s

dd if=/dev/zero of=2157 bs=1k count=1k
dd: writing `2157': No space left on device
973+0 records in
972+0 records out
995328 bytes (995 kB) copied, 0.0545104 s, 18.3 MB/s



echo "stats" | debugfs.ocfs2 /dev/xvdd1 |cat
debugfs.ocfs2 1.4.3
debugfs: 	Revision: 0.90
	Mount Count: 0   Max Mount Count: 20
	State: 0   Errors: 0
	Check Interval: 0   Last Check: Thu Oct  8 20:23:54 2009
	Creator OS: 0
	Feature Compat: 0 
	Feature Incompat: 0 
	Tunefs Incomplete: 0 
	Feature RO compat: 0 
	Root Blknum: 5   System Dir Blknum: 6
	First Cluster Group Blknum: 3
	Block Size Bits: 12   Cluster Size Bits: 12
	Max Node Slots: 16
	Extended Attributes Inline Size: 0
	Label: 
	UUID: BD88053E263B4CEAABC941FE3413B4EA
	UUID_hash: 0 (0x0)
	Cluster stack: classic o2cb
	Inode: 2   Mode: 00   Generation: 964566170 (0x397e1c9a)
	FS Generation: 964566170 (0x397e1c9a)
	CRC32: 00000000   ECC: 0000
	Type: Unknown   Attr: 0x0   Flags: Valid System Superblock 
	Dynamic Features: (0x0) 
	User: 0 (root)   Group: 0 (root)   Size: 0
	Links: 0   Clusters: 13107199
	ctime: 0x48f80fe8 -- Fri Oct 17 15:09:12 2008
	atime: 0x0 -- Thu Jan  1 10:00:00 1970
	mtime: 0x48f80fe8 -- Fri Oct 17 15:09:12 2008
	dtime: 0x0 -- Thu Jan  1 10:00:00 1970
	ctime_nsec: 0x00000000 -- 0
	atime_nsec: 0x00000000 -- 0
	mtime_nsec: 0x00000000 -- 0
	Last Extblk: 0
	Sub Alloc Slot: Global   Sub Alloc Bit: 65535
debugfs: 



echo "stat /brad/aaaaa" | debugfs.ocfs2 /dev/xvdd1 |cat
debugfs.ocfs2 1.4.3
debugfs: 	Inode: 11539619   Mode: 0750   Generation: 3859452415 (0xe60a91ff)
	FS Generation: 964566170 (0x397e1c9a)
	CRC32: 00000000   ECC: 0000
	Type: Directory   Attr: 0x0   Flags: Valid 
	Dynamic Features: (0x0) 
	User: 1000 (brad)   Group: 2000 (unknown)   Size: 4096
	Links: 2   Clusters: 1
	ctime: 0x4b28b899 -- Wed Dec 16 21:38:17 2009
	atime: 0x4b28b03b -- Wed Dec 16 21:02:35 2009
	mtime: 0x4b28b899 -- Wed Dec 16 21:38:17 2009
	dtime: 0x0 -- Thu Jan  1 10:00:00 1970
	ctime_nsec: 0x29bd9e6e -- 700292718
	atime_nsec: 0x31a68876 -- 832997494
	mtime_nsec: 0x29bd9e6e -- 700292718
	Last Extblk: 0
	Sub Alloc Slot: 2   Sub Alloc Bit: 87
	Tree Depth: 0   Count: 243   Next Free Rec: 1
	## Offset        Clusters       Block#          Flags
	0  0             1              9343355         0x0
debugfs: 



Hopefully this sheds some light on what's going on. Please let me know if
you need any further info.

Cheers,

Brad

Sunil Mushran

2009-Dec-17 00:31 UTC

head link

[Ocfs2-users] No space left on device

Brad Plant wrote:> I have this happen on different file systems running different mainline
kernel versions created with different versions of ocfs2-tools. I've seen 1
file system "fill up" with only 40% odd in use according to df.
What's interesting is that while 1 node can't write anymore, another
node (only 2 nodes mounted rw) will still be able to write to the FS, but
neither will be able to completely fill the FS.
>
> I have created new ocfs2 file systems and the problem doesn't seem to
occur (or it gets very close be actually being full), but weeks/months later
after being in use, the problem develops. The FS I mentioned above that filled
up at ~40% use was about 6 months old and you could say it did it's fair
share of "work".
>
> I just dumped some 1GB files in a directory using dd to fill up a FS (full
at 87%) to that I could collect some debugfs.ocfs2 information. Upon not being
able to create any further 1GB files (operations on the FS got really slow), I
successfully created some 10MB files. I then created 100+ 1MB files using dd.
What seemed strange is that dd kept returning "no space left on
device", but it would always write 995328 bytes of data to the file.
>
> dd if=/dev/zero of=2156 bs=1k count=1k
> dd: writing `2156': No space left on device
> 973+0 records in
> 972+0 records out
> 995328 bytes (995 kB) copied, 0.0551648 s, 18.0 MB/s
>
> dd if=/dev/zero of=2157 bs=1k count=1k
> dd: writing `2157': No space left on device
> 973+0 records in
> 972+0 records out
> 995328 bytes (995 kB) copied, 0.0545104 s, 18.3 MB/s
This looks like bz#1189.
http://oss.oracle.com/bugzilla/show_bug.cgi?id=1189

We are working on the fix. The patch is in review.
http://oss.oracle.com/osswiki/OCFS2/SmallTasksList

If all goes well, the next release will have the fix.

Brad Plant

2009-Dec-17 03:23 UTC

head link

[Ocfs2-users] No space left on device

Hi Sunil,

On Wed, 16 Dec 2009 18:31:32 -0800
Sunil Mushran <sunil.mushran at oracle.com> wrote:
> Brad Plant wrote:
> > Thanks for the reply Sunil. I read through the 2 links you provided, 
> > but I have a question. The info mentioned modifications to mkfs to 
> > pre-allocate some area for extent stuff. Does this mean that we would 
> > have to create a new FS to fix the problem or will there also be a fix
> > in the kernel that allows files to be moved on-disk to create some 
> > contiguous free space?
> 
> The fix is in both components. Most users will see the benefit with just
> the kernel change. The mkfs is for added safety.
Ok. Do you intend on submitting this patch for inclusion in previous mainline
kernel versions? Or do you think we will be able to patch older kernel versions
(2.6.28+) ourself?
> If you attach the stat_sysdir.sh output of your vol to that bz, I will 
> be able
> to confirm that.
Thanks Sunil, I have uploaded the stat_sysdir.sh output for 2 different file
systems to the bugzilla item.

Cheers,

Brad

Sunil Mushran

2009-Dec-17 21:05 UTC

head link

[Ocfs2-users] No space left on device

Brad Plant wrote:> Ok. Do you intend on submitting this patch for inclusion in previous
mainline kernel versions? Or do you think we will be able to patch older kernel
versions (2.6.28+) ourself?
We always push patches to the current mainline version. Older mainline
kernels are handled on a case-by-case basis... for serious data corruption/
hang type issues. This will be pushed only in mainline and only be 
backported
to the ocfs2 1.4 supported tree.
>
>> If you attach the stat_sysdir.sh output of your vol to that bz, I will 
>> be able
>> to confirm that.
>
> Thanks Sunil, I have uploaded the stat_sysdir.sh output for 2 different
file systems to the bugzilla item.
>
Comments in:
http://oss.oracle.com/bugzilla/show_bug.cgi?id=1189#c11

Brad Plant

2010-Jan-06 02:21 UTC

head link

[Ocfs2-users] No space left on device

Hi Sunil,

On Wed, 16 Dec 2009 16:31:22 -0800
Sunil Mushran <sunil.mushran at oracle.com> wrote:
> This looks like bz#1189.
> http://oss.oracle.com/bugzilla/show_bug.cgi?id=1189
> 
> We are working on the fix. The patch is in review.
> http://oss.oracle.com/osswiki/OCFS2/SmallTasksList
> 
> If all goes well, the next release will have the fix.
By next release, do you mean mainline 2.6.33? If not, can you give any
indication of which kernel release this might be included in?

Cheers,

Brad

Reasonably Related Threads

Search for more possibly parallel threads

Ocfs2 users - Dec 2009 - No space left on device

[Ocfs2-users] No space left on device

[Ocfs2-users] No space left on device

[Ocfs2-users] No space left on device

[Ocfs2-users] No space left on device

[Ocfs2-users] No space left on device

Reasonably Related Threads