Ocfs2 users - Feb 2012 - extreme slowdown (20 Bytes/s) in 5 node cluster

Hello List,

since yesterday we're experiencing hang ups and low downs on one of our 
ocfs2 clusters. Our 5 nodes connected through iSCSI i only able to write 
/ read with around 20bytes/s (dd if=/dev/zero of=/mnt/ocfs2/test bs=256 
count=2). At the same time the same nodes can write perfectly to another 
ocfs2 filesystem (through the same network card and switch). I also 
changed the whole fs part. to another backup storage device - but the 
same thing happens.

So i'm quite sure it is a problem with the ocfs2 filesystem. A fsck 
didn't show any problems.

Right now it is running fine - most probably because of low visits right 
now.

FS details:
~# debugfs.ocfs2 -R "stats" /dev/sdb
         Revision: 0.90
         Mount Count: 0   Max Mount Count: 20
         State: 0   Errors: 0
         Check Interval: 0   Last Check: Fri Feb 17 21:59:21 2012
         Creator OS: 0
         Feature Compat: 3 backup-super strict-journal-super
         Feature Incompat: 16208 sparse extended-slotmap inline-data 
metaecc xattr indexed-dirs refcount discontig-bg
         Tunefs Incomplete: 0
         Feature RO compat: 1 unwritten
         Root Blknum: 5   System Dir Blknum: 6
         First Cluster Group Blknum: 3
         Block Size Bits: 12   Cluster Size Bits: 12
         Max Node Slots: 10
         Extended Attributes Inline Size: 256
         Label: ocfs2disk
         UUID: B029D592BBB04D1ABA25111216009DD2
         Hash: 476522312 (0x1c672748)
         DX Seed[0]: 0x00000000
         DX Seed[1]: 0x00000000
         DX Seed[2]: 0x00000000
         Cluster stack: classic o2cb
         Inode: 2   Mode: 00   Generation: 2428650348 (0x90c2436c)
         FS Generation: 2428650348 (0x90c2436c)
         CRC32: f68f7e80   ECC: 0078
         Type: Unknown   Attr: 0x0   Flags: Valid System Superblock
         Dynamic Features: (0x0)
         User: 0 (root)   Group: 0 (root)   Size: 0
         Links: 0   Clusters: 47456155
         ctime: 0x4be7dd41 -- Mon May 10 12:17:37 2010
         atime: 0x0 -- Thu Jan  1 01:00:00 1970
         mtime: 0x4be7dd41 -- Mon May 10 12:17:37 2010
         dtime: 0x0 -- Thu Jan  1 01:00:00 1970
         ctime_nsec: 0x00000000 -- 0
         atime_nsec: 0x00000000 -- 0
         mtime_nsec: 0x00000000 -- 0
         Refcount Block: 0
         Last Extblk: 0   Orphan Slot: 0
         Sub Alloc Slot: Global   Sub Alloc Bit: 65535


~# cat /sys/kernel/debug/o2net/stats
1,5,1639450,1887540786,10231206351,220661950679,24433045,144802608142
1,2,1888282,2664740522,13589664854,299324139712,1342649,5540708637
1,3,1585381,2021551046,10894428756,238189878694,829297,3453566877
1,4,1573033,2025456266,10594374194,239043267702,1061860,5666562010

Any ideas? Thanks!

Greets
Stefan

Am 18.02.2012 09:42, schrieb Stefan Priebe:
> Hello List,
>
> since yesterday we're experiencing hang ups and low downs on one of our
> ocfs2 clusters. Our 5 nodes connected through iSCSI i only able to write
> / read with around 20bytes/s (dd if=/dev/zero of=/mnt/ocfs2/test bs=256
> count=2). At the same time the same nodes can write perfectly to another
> ocfs2 filesystem (through the same network card and switch). I also
> changed the whole fs part. to another backup storage device - but the
> same thing happens.
When i start something like this on MORE than one node.

time (c=0; while [ $c -lt 100 ]; do c=$(expr $c + 1); echo $c; dd 
if=/dev/zero of=/home/clusterfs/$c.`hostname` bs=1M count=1; done;);

everything seems to hang completely.

Running this on the 2nd OCFS FS beeing on another storage but running on 
the same nodes. It works without problems.

Stefan