Ocfs2 users - Aug 2011 - Slow OCFS2 file creation / how to make ocfs2 generally faster?

Hi Guys,

all in all ocfs2 is a nice piece of software and works very well. Last 
week i made some benchmarks and was thinking if there is a way to make 
it faster.

Here are some results perhaps someone can comment them:

XFS iSCSI target:
100MB/s block write
  50MB/s block rewrite
140MB/s block read

OCFS2 iSCSI target (one node connected / two nodes connected)
  75MB/s block write
  24MB/s block rewrite
  82MB/s block read


File Creation / Deletion:

XFS iSCSI target:
5800 files/s Seq. create
5700 files/s Seq. delete
5800 files/s Rand. create
1300 files/s Rand. delete

OCFS2 iSCSI target (only ONE node connected):
5000 files/s Seq. create
5400 files/s Seq. delete
5300 files/s Rand. create
  480 files/s Rand. delete

OCFS2 iSCSI target (two or more nodes connected):
  800 files/s Seq. create
5800 files/s Seq. delete
  800 files/s Rand. create
  480 files/s Rand. delete

I would be very happy if someone can tell me if this is normal or if 
there are ways to speed this up.

All tests were made using vanilla 2.6.38.8 kernel and these mount 
options noatime,data=writeback,commit=60.

Greets Stefan

On 08/21/2011 11:36 PM, Stefan Priebe - Profihost AG
wrote:
> Hi Guys,
>
> all in all ocfs2 is a nice piece of software and works very well. Last
> week i made some benchmarks and was thinking if there is a way to make
> it faster.
>
> Here are some results perhaps someone can comment them:
>
> XFS iSCSI target:
> 100MB/s block write
>    50MB/s block rewrite
> 140MB/s block read
>
> OCFS2 iSCSI target (one node connected / two nodes connected)
>    75MB/s block write
>    24MB/s block rewrite
>    82MB/s block read
>
>
> File Creation / Deletion:
>
> XFS iSCSI target:
> 5800 files/s Seq. create
> 5700 files/s Seq. delete
> 5800 files/s Rand. create
> 1300 files/s Rand. delete
>
> OCFS2 iSCSI target (only ONE node connected):
> 5000 files/s Seq. create
> 5400 files/s Seq. delete
> 5300 files/s Rand. create
>    480 files/s Rand. delete
>
> OCFS2 iSCSI target (two or more nodes connected):
>    800 files/s Seq. create
> 5800 files/s Seq. delete
>    800 files/s Rand. create
>    480 files/s Rand. delete
>
> I would be very happy if someone can tell me if this is normal or if
> there are ways to speed this up.
>
> All tests were made using vanilla 2.6.38.8 kernel and these mount
> options noatime,data=writeback,commit=60.
What features do you have enabled on disk?
debugfs.ocfs2 -R "stats" /dev/sdX

Network latency:
cat /sys/kernel/debug/o2net/stats

Number of locks, etc:
cat /sys/kernel/debug/o2dlm/DOMAIN/dlm_state

> What features do you have enabled on disk?
> debugfs.ocfs2 -R "stats" /dev/sdX
The disks were formated using max-features flag. The output looks like this:
       Revision: 0.90
         Mount Count: 0   Max Mount Count: 20
         State: 0   Errors: 0
         Check Interval: 0   Last Check: Wed Aug  3 12:20:20 2011
         Creator OS: 0
         Feature Compat: 3 backup-super strict-journal-super
         Feature Incompat: 16208 sparse extended-slotmap inline-data 
metaecc xattr indexed-dirs refcount discontig-bg
         Tunefs Incomplete: 0
         Feature RO compat: 7 unwritten usrquota grpquota
         Root Blknum: 5   System Dir Blknum: 6
         First Cluster Group Blknum: 3
         Block Size Bits: 12   Cluster Size Bits: 12
         Max Node Slots: 8
         Extended Attributes Inline Size: 256
         Label: ocfs2
         UUID: 2E60AF2024744D63848A13417DE66984
         Hash: 2985671674 (0xb1f5bbfa)
         DX Seed[0]: 0xd7cb3bf0
         DX Seed[1]: 0xb630f905
         DX Seed[2]: 0xc126a149
         Cluster stack: classic o2cb
         Inode: 2   Mode: 00   Generation: 2304813421 (0x8960a96d)
         FS Generation: 2304813421 (0x8960a96d)
         CRC32: 8a254ba7   ECC: 0896
         Type: Unknown   Attr: 0x0   Flags: Valid System Superblock
         Dynamic Features: (0x0)
         User: 0 (root)   Group: 0 (root)   Size: 0
         Links: 0   Clusters: 21482248
         ctime: 0x4e3920e4 -- Wed Aug  3 12:20:20 2011
         atime: 0x0 -- Thu Jan  1 01:00:00 1970
         mtime: 0x4e3920e4 -- Wed Aug  3 12:20:20 2011
         dtime: 0x0 -- Thu Jan  1 01:00:00 1970
         ctime_nsec: 0x00000000 -- 0
         atime_nsec: 0x00000000 -- 0
         mtime_nsec: 0x00000000 -- 0
         Refcount Block: 0
         Last Extblk: 0   Orphan Slot: 0
         Sub Alloc Slot: Global   Sub Alloc Bit: 65535
> Network latency:
> cat /sys/kernel/debug/o2net/stats
> Number of locks, etc:
> cat /sys/kernel/debug/o2dlm/DOMAIN/dlm_state
/sys/kernel/debug/ is empty - what do i miss?

Stefan

On 08/22/2011 09:01 AM, Stefan Priebe - Profihost AG
wrote:
>
>> What features do you have enabled on disk?
>> debugfs.ocfs2 -R "stats" /dev/sdX
>
> The disks were formated using max-features flag. The output looks like 
> this:
>       Revision: 0.90
>         Mount Count: 0   Max Mount Count: 20
>         State: 0   Errors: 0
>         Check Interval: 0   Last Check: Wed Aug  3 12:20:20 2011
>         Creator OS: 0
>         Feature Compat: 3 backup-super strict-journal-super
>         Feature Incompat: 16208 sparse extended-slotmap inline-data 
> metaecc xattr indexed-dirs refcount discontig-bg
>         Tunefs Incomplete: 0
>         Feature RO compat: 7 unwritten usrquota grpquota
>         Root Blknum: 5   System Dir Blknum: 6
>         First Cluster Group Blknum: 3
>         Block Size Bits: 12   Cluster Size Bits: 12
>         Max Node Slots: 8
>         Extended Attributes Inline Size: 256
>         Label: ocfs2
>         UUID: 2E60AF2024744D63848A13417DE66984
>         Hash: 2985671674 (0xb1f5bbfa)
>         DX Seed[0]: 0xd7cb3bf0
>         DX Seed[1]: 0xb630f905
>         DX Seed[2]: 0xc126a149
>         Cluster stack: classic o2cb
>         Inode: 2   Mode: 00   Generation: 2304813421 (0x8960a96d)
>         FS Generation: 2304813421 (0x8960a96d)
>         CRC32: 8a254ba7   ECC: 0896
>         Type: Unknown   Attr: 0x0   Flags: Valid System Superblock
>         Dynamic Features: (0x0)
>         User: 0 (root)   Group: 0 (root)   Size: 0
>         Links: 0   Clusters: 21482248
>         ctime: 0x4e3920e4 -- Wed Aug  3 12:20:20 2011
>         atime: 0x0 -- Thu Jan  1 01:00:00 1970
>         mtime: 0x4e3920e4 -- Wed Aug  3 12:20:20 2011
>         dtime: 0x0 -- Thu Jan  1 01:00:00 1970
>         ctime_nsec: 0x00000000 -- 0
>         atime_nsec: 0x00000000 -- 0
>         mtime_nsec: 0x00000000 -- 0
>         Refcount Block: 0
>         Last Extblk: 0   Orphan Slot: 0
>         Sub Alloc Slot: Global   Sub Alloc Bit: 65535
>
>> Network latency:
>> cat /sys/kernel/debug/o2net/stats
>> Number of locks, etc:
>> cat /sys/kernel/debug/o2dlm/DOMAIN/dlm_state
> /sys/kernel/debug/ is empty - what do i miss?
>

mount -t debugfs none /sys/kernel/debug

HI,

here are all values. Just a side note all machines had a fresh reboot. 
So these values are not right "after" the test.
>>> Network latency:
>>> cat /sys/kernel/debug/o2net/stats
1,3,992696,1284798776,6583557287,129962217930,15822,103963160
1,0,991090,1242030681,7180479582,150003421435,74,189353
1,2,1002340,1360444027,7088690084,131107733099,28796,112167428
>>> Number of locks, etc:
>>> cat /sys/kernel/debug/o2dlm/DOMAIN/dlm_state
>> /sys/kernel/debug/ is empty - what do i miss?
Domain: 2E60AF2024744D63848A13417DE66984  Key: 0xcdc07c07  Protocol: 1.1
Thread Pid: 3669  Node: 1  State: JOINED
Number of Joins: 1  Joining Node: 255
Domain Map: 0 1 2 3
Live Map: 0 1 2 3
Lock Resources: 1863 (497079)
MLEs: 0 (510131)
   Blocking: 0 (13211)
   Mastery: 0 (496920)
   Migration: 0 (0)
Lists: Dirty=Empty  Purge=Empty  PendingASTs=Empty  PendingBASTs=Empty
Purge Count: 0  Refs: 1
Dead Node: 255
Recovery Pid: 3670  Master: 255  State: INACTIVE
Recovery Map:
Recovery Node State:
> mount -t debugfs none /sys/kernel/debug
thanks!

On 08/22/2011 09:08 AM, Stefan Priebe - Profihost AG
wrote:
> HI,
>
> here are all values. Just a side note all machines had a fresh reboot. 
> So these values are not right "after" the test.
>
>>>> Network latency:
>>>> cat /sys/kernel/debug/o2net/stats
> 1,3,992696,1284798776,6583557287,129962217930,15822,103963160
> 1,0,991090,1242030681,7180479582,150003421435,74,189353
> 1,2,1002340,1360444027,7088690084,131107733099,28796,112167428
>
>>>> Number of locks, etc:
>>>> cat /sys/kernel/debug/o2dlm/DOMAIN/dlm_state
>>> /sys/kernel/debug/ is empty - what do i miss?
> Domain: 2E60AF2024744D63848A13417DE66984  Key: 0xcdc07c07  Protocol: 1.1
> Thread Pid: 3669  Node: 1  State: JOINED
> Number of Joins: 1  Joining Node: 255
> Domain Map: 0 1 2 3
> Live Map: 0 1 2 3
> Lock Resources: 1863 (497079)
> MLEs: 0 (510131)
>   Blocking: 0 (13211)
>   Mastery: 0 (496920)
>   Migration: 0 (0)
> Lists: Dirty=Empty  Purge=Empty  PendingASTs=Empty  PendingBASTs=Empty
> Purge Count: 0  Refs: 1
> Dead Node: 255
> Recovery Pid: 3670  Master: 255  State: INACTIVE
> Recovery Map:
> Recovery Node State:
>
>> mount -t debugfs none /sys/kernel/debug
Well the values during and after the test will give better info.

BTW, how much memory do the nodes have?

Lock Resources: 1863 (497079)

This means that that node has created 497K lock resources
but have under 2000 cached. That could be due to deletes
or could be due to lack of memory (lock churning).

The network looks ok. 150usecs ping time between the nodes.

On 08/22/2011 09:11 AM, Medapuram, Gopala wrote:
> To all OCFS gurus,
>
> We have discussion scheduled on pros and cons of OCFS2 over NFS.
>
> Can you please guide me to some good notes and documentation to prepare for
proper discussion?
>
> Appreciate the help.
>
> Thank you,
> Gopal
NFS and OCFS2 are different file systems. Infact one can NFS export
an OCFS2 volume.

Your qs probably relates to a particular use case.

> Well the values during and after the test will give better info.
I will create a dedicated test setup and provide new values and stats.
> BTW, how much memory do the nodes have?
> Lock Resources: 1863 (497079)
>
> This means that that node has created 497K lock resources
> but have under 2000 cached. That could be due to deletes
> or could be due to lack of memory (lock churning).
They all have 8GB of mem but only 1,5-2,5 are in use regarding free 
command. What does lock churning mean and is there another setting / 
value i can look at?

Stefan

On 08/22/2011 09:57 AM, Stefan Priebe - Profihost AG
wrote:
>
>> Well the values during and after the test will give better info.
> I will create a dedicated test setup and provide new values and stats.
>
>> BTW, how much memory do the nodes have?
>> Lock Resources: 1863 (497079)
>>
>> This means that that node has created 497K lock resources
>> but have under 2000 cached. That could be due to deletes
>> or could be due to lack of memory (lock churning).
> They all have 8GB of mem but only 1,5-2,5 are in use regarding free 
> command. What does lock churning mean and is there another setting / 
> value i can look at?
Its tied to the inode cache. As long as the inode is cached,
the lock resource will be cached.