Ocfs2 users - Jun 2010 - 'No space left on device' error with plenty of space.

I've got a busy FTP/Web cluster running OCFS2 v1.4.4.

I've started getting "No space on device" errors when users
attempt to write
to the file system.  Disk utilization is about 76% with more than 100gb
free.  Inode utilization is also at 76%.

I thought this was a manifestation of bug # 1189, so I decreased the number
of nodes via tunefs.ocfs2 from 8 (the default) down to 3 (there are only 2
nodes in the cluster, with no growth anticipated).

That got me out of the woods on Monday, but this morning the problem
manifested again.

I've opened bug # 1263 about this issue. (link:
http://oss.oracle.com/bugzilla/show_bug.cgi?id=1263 )

Does anyone have other ideas?

I'm more than happy to supply other information.

What seems to happen is that small writes are allowed, but bigger writes
failed.  On Monday, I could write multiple 325kb files, and I could cat them
together to make one file of ~2 mb, but when I tried to make a 10ish mb
file, it failed.

--Jason
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://oss.oracle.com/pipermail/ocfs2-users/attachments/20100609/78509556/attachment.html

And now it's starting to fail again.

--Jason

On Wed, Jun 9, 2010 at 9:51 AM, Jason Price <japrice at gmail.com> wrote:
> I've got a busy FTP/Web cluster running OCFS2 v1.4.4.
>
> I've started getting "No space on device" errors when users
attempt to
> write to the file system.  Disk utilization is about 76% with more than
> 100gb free.  Inode utilization is also at 76%.
>
> I thought this was a manifestation of bug # 1189, so I decreased the number
> of nodes via tunefs.ocfs2 from 8 (the default) down to 3 (there are only 2
> nodes in the cluster, with no growth anticipated).
>
> That got me out of the woods on Monday, but this morning the problem
> manifested again.
>
> I've opened bug # 1263 about this issue. (link:
> http://oss.oracle.com/bugzilla/show_bug.cgi?id=1263 )
>
> Does anyone have other ideas?
>
> I'm more than happy to supply other information.
>
> What seems to happen is that small writes are allowed, but bigger writes
> failed.  On Monday, I could write multiple 325kb files, and I could cat
them
> together to make one file of ~2 mb, but when I tried to make a 10ish mb
> file, it failed.
>
> --Jason
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://oss.oracle.com/pipermail/ocfs2-users/attachments/20100609/6e333783/attachment.html