I have a 12 node cluster using ocfs2. Not all the nodes mount the same
filesystems, so for this issue I am just focusing on the 8 nodes that
mount the filesystem that has performance issues.
The 8 nodes have 10 ocfs2 volumes mounted on them, and the volume that
seems to be causing the most problems is 2TB (data), which is currently
90% full.
At times, data performs very poorly. Processes using the volume are
slow, df will hang for a long time accessing it, and nfs mounts to it
are slow to return or even give an i/o error.
This is becoming a big problem and I would like some help to resolve the
performance issue. I have included the output of both stat and fs_state
below. The kernel version on all nodes 3.2.29, and the ocfs2-utilities
version in 1.6.4.
I tried 1.8.2, but that version seemed to have issues reading
cluster.conf properly. It would give errors about the local node not
being in the configuration file and it would only list one node in
ocfs2console. I looked to me as if it was only looking at the first
node in the configuration file, 1.6.4 reads the same configuration file
fine.
Thanks,
Andy
Here is the stat for the data volume
Revision: 0.90
Mount Count: 0 Max Mount Count: 20
State: 0 Errors: 0
Check Interval: 0 Last Check: Tue Apr 10 11:00:52 2012
Creator OS: 0
Feature Compat: 3 backup-super strict-journal-super
Feature Incompat: 592 sparse inline-data xattr
Tunefs Incomplete: 0
Feature RO compat: 1 unwritten
Root Blknum: 513 System Dir Blknum: 514
First Cluster Group Blknum: 256
Block Size Bits: 12 Cluster Size Bits: 20
Max Node Slots: 16
Extended Attributes Inline Size: 256
Label:
UUID: B4B05BE929E54DFEB5FDC035E3F6E931
Hash: 1144689215 (0x443a923f)
DX Seed[0]: 0x00000000
DX Seed[1]: 0x00000000
DX Seed[2]: 0x00000000
Cluster stack: classic o2cb
Inode: 2 Mode: 00 Generation: 2584189387 (0x9a0799cb)
FS Generation: 2584189387 (0x9a0799cb)
CRC32: 00000000 ECC: 0000
Type: Unknown Attr: 0x0 Flags: Valid System Superblock
Dynamic Features: (0x0)
User: 0 (root) Group: 0 (root) Size: 0
Links: 0 Clusters: 1280000
ctime: 0x4f845934 -- Tue Apr 10 11:00:52 2012
atime: 0x0 -- Wed Dec 31 18:00:00 1969
mtime: 0x4f845934 -- Tue Apr 10 11:00:52 2012
dtime: 0x0 -- Wed Dec 31 18:00:00 1969
ctime_nsec: 0x00000000 -- 0
atime_nsec: 0x00000000 -- 0
mtime_nsec: 0x00000000 -- 0
Refcount Block: 0
Last Extblk: 0 Orphan Slot: 0
Sub Alloc Slot: Global Sub Alloc Bit: 65535
fs_state:
Device => Id: 254,2 Uuid: 45B8B88228BD4DFBBE91BFC85C748FB6 Gen:
0x364EAC5A Label:
Volume => State: 2 Flags: 0x0
Sizes => Block: 4096 Cluster: 4096
Features => Compat: 0x3 Incompat: 0x250 ROcompat: 0x1
Mount => Opts: 0x105 AtimeQuanta: 60
Cluster => Stack: o2cb Name: 45B8B88228BD4DFBBE91BFC85C748FB6
Version: 1.0
DownCnvt => Pid: 9804 Count: 0 WakeSeq: 42573 WorkSeq: 42573
Recovery => Pid: -1 Nodes: None
Commit => Pid: 9809 Interval: 0 Needs: 1
Journal => State: 1 TxnId: 57 NumTxns: 0
Stats => GlobalAllocs: 1 LocalAllocs: 393 SubAllocs: 108
LAWinMoves: 1 SAExtends: 0
LocalAlloc => State: 1 Descriptor: 0 Size: 27136 bits Default: 27136 bits
Steal => InodeSlot: -1 StolenInodes: 0, MetaSlot: -1 StolenMeta: 0
OrphanScan => Local: 126 Global: 1036 Last Scan: 851 seconds ago
Slots => Num RecoGen
0 23
1 18
* 2 5
3 16
4 3
5 12
6 6
7 1
8 3
9 0
10 0
11 0
12 0
13 0
14 0
15 0