Ocfs2 users - Feb 2012 - Concurrent write performance issues with OCFS2

I have a two-node RHEL5 cluster that runs the following Linux kernel and
accompanying OCFS2 module packages:

  * kernel-2.6.18-274.17.1.el5
  * ocfs2-2.6.18-274.17.1.el5-1.4.7-1.el5

A 2.5TB LUN is presented to both nodes via DM-Multipath. I have carved
out a single partition (using the entire LUN), and formatted it with OCFS2:

  # mkfs.ocfs2 -N 2 -L 'foofs' -T datafiles /dev/mapper/bams01p1

Finally, the filesystem is mounted to both nodes with the following options:

  # mount | grep bams01
/dev/mapper/bams01p1 on /foofs type ocfs2
(rw,_netdev,noatime,data=writeback,heartbeat=local)

----------

When a single node is writing arbitrary data (i.e. dd(1) with /dev/zero
as input) to a large (say, 10 GB) file in /foofs, I see the expected
performance of ~850 MB/sec.

If both nodes are concurrently writing large files full of zeros to
/foofs, performance drops way down to ~45 MB/s. I experimented with each
node writing to /foofs/test01/ and /foofs/test02/ subdirectories,
respectively, and found that performance increased slightly to a - still
poor - 65 MB/s.

----------

I understand from searching past mailing list threads that the culprit
is likely related to the negotiation of file locks, and waiting for data
to be flushed to journal / disk.

My two questions are:

1. Does this dramatic write performance slowdown sound reasonable and
expected?

2. Are there any OCFS2-level steps I can take to improve this situation?


Thanks -

-- 
Erik Schwartz <schwartz.erik.c at gmail.com> | GPG key 14F1139B

In 1.4, the local allocator window is small. 8MB. Meaning the node
has to hit the global bitmap after every 8MB. In later releases, the
window is much larger.

Second, a single node is not a good baseline. A better baseline is
multiple nodes writing concurrently to the block device. Not fs.
Use dd. Set different write offsets. This should help figure out how
the shared device works with multiple nodes.

On 2/28/2012 9:24 AM, Erik Schwartz wrote:
> I have a two-node RHEL5 cluster that runs the following Linux kernel and
> accompanying OCFS2 module packages:
>
>    * kernel-2.6.18-274.17.1.el5
>    * ocfs2-2.6.18-274.17.1.el5-1.4.7-1.el5
>
> A 2.5TB LUN is presented to both nodes via DM-Multipath. I have carved
> out a single partition (using the entire LUN), and formatted it with OCFS2:
>
>    # mkfs.ocfs2 -N 2 -L 'foofs' -T datafiles /dev/mapper/bams01p1
>
> Finally, the filesystem is mounted to both nodes with the following
options:
>
>    # mount | grep bams01
> /dev/mapper/bams01p1 on /foofs type ocfs2
> (rw,_netdev,noatime,data=writeback,heartbeat=local)
>
> ----------
>
> When a single node is writing arbitrary data (i.e. dd(1) with /dev/zero
> as input) to a large (say, 10 GB) file in /foofs, I see the expected
> performance of ~850 MB/sec.
>
> If both nodes are concurrently writing large files full of zeros to
> /foofs, performance drops way down to ~45 MB/s. I experimented with each
> node writing to /foofs/test01/ and /foofs/test02/ subdirectories,
> respectively, and found that performance increased slightly to a - still
> poor - 65 MB/s.
>
> ----------
>
> I understand from searching past mailing list threads that the culprit
> is likely related to the negotiation of file locks, and waiting for data
> to be flushed to journal / disk.
>
> My two questions are:
>
> 1. Does this dramatic write performance slowdown sound reasonable and
> expected?
>
> 2. Are there any OCFS2-level steps I can take to improve this situation?
>
>
> Thanks -
>