Ocfs2 users - Aug 2011 - fsck hangs in Pass 0a

Hello list,

I have a ~10TB ocfs2 filesystem in a 8-node cluster. This sits on a
logical volume (I know lv is not cluster aware, but I make sure no one
touches the lv, while the cluster is running). The LV consists of 5x2TB
multipath devices.

I recently had errors like this on some nodes:

OCFS2: ERROR (device dm-7): ocfs2_check_group_descriptor: Group Descriptor # 0
has bad signature
File system is now read-only due to the potential of on-disk corruption. Please
run fsck.ocfs2 once the file system is unmounted.
(kvm,12322,1):ocfs2_search_chain:1363 ERROR: status = -5
(kvm,12322,1):ocfs2_claim_suballoc_bits:1524 ERROR: status = -5
(kvm,12322,1):__ocfs2_claim_clusters:1806 ERROR: status = -5
(kvm,12322,1):ocfs2_local_alloc_new_window:1013 ERROR: status = -5
(kvm,12322,1):ocfs2_local_alloc_slide_window:1116 ERROR: status = -5
(kvm,12322,1):ocfs2_reserve_local_alloc_bits:537 ERROR: status = -5
(kvm,12322,1):__ocfs2_reserve_clusters:816 ERROR: status = -5
(kvm,12322,1):ocfs2_lock_allocators:677 ERROR: status = -5
(kvm,12322,1):ocfs2_write_begin_nolock:1750 ERROR: status = -5
(kvm,12322,1):ocfs2_write_begin:1860 ERROR: status = -5
(kvm,12322,1):ocfs2_file_buffered_write:2039 ERROR: status = -5
OCFS2: ERROR (device dm-7): ocfs2_check_group_descriptor: Group Descriptor # 0
has bad signature


So I ran fsck.ocfs2 -f. But it hangs forever (>12h) with this output:

fsck.ocfs2 1.4.4
Checking OCFS2 filesystem in /dev/mapper/lv0:
  Label:              <NONE>
  UUID:               F27D7B8F7127436981A2B5D1C93FB204
  Number of blocks:   2684349440
  Block size:         4096
  Number of clusters: 2684349440
  Cluster size:       4096
  Number of slots:    16

/dev/mapper/lv0 was run with -f, check forced.
Pass 0a: Checking cluster allocation chains


I attaced strace to it, to see what is going on.
Before it hangs I get:

write(1, "Pass 0a: Checking cluster alloca"..., 44Pass 0a: Checking
cluster allocation chains) = 44
mmap(NULL, 4198400, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x2ad6a5001000
munmap(0x2ad6a5001000, 4198400)         = 0
mmap(NULL, 4202496, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x2ad6a5001000
pread(3,
"INODE01\0;Q&\354\377\377\7\0\0\0\0\0\0\354\377\237\0\0\0\0\0\0\0\0"...,
4096, 45056) = 4096
mprotect(0x2ad640020000, 4096, PROT_READ|PROT_WRITE) = 0
mprotect(0x2ad640021000, 4096, PROT_READ|PROT_WRITE) = 0
...
[a couple of hundred similar lines]
...
mprotect(0x2ad640803000, 4096, PROT_READ|PROT_WRITE) = 0


Then it hangs with 100% idle on one core.

Regards,
Matthias

Hello Matthias,
> I have a ~10TB ocfs2 filesystem in a 8-node cluster. This sits on a
> logical volume (I know lv is not cluster aware, but I make sure no one
> touches the lv, while the cluster is running). The LV consists of 5x2TB
> multipath devices.
> So I ran fsck.ocfs2 -f. But it hangs forever (>12h) with this output:
> 
> fsck.ocfs2 1.4.4
> Checking OCFS2 filesystem in /dev/mapper/lv0:
>   Label:              <NONE>
>   UUID:               F27D7B8F7127436981A2B5D1C93FB204
>   Number of blocks:   2684349440
>   Block size:         4096
>   Number of clusters: 2684349440
>   Cluster size:       4096
>   Number of slots:    16
> 
> /dev/mapper/lv0 was run with -f, check forced.
> Pass 0a: Checking cluster allocation chains
I wrote to the list about what probably was the same problem in April. You can 
access the thread here:

http://oss.oracle.com/pipermail/ocfs2-users/2011-April/005093.html

Sunil wrote a few days after, explaining there was a bug in fsck that caused 
it to enter an infinite loop when the filesystem was bigger than some value, and
that it had been corrected in version 1.6.4 . This is an excerpt of the 
message:
> Fixed in ocfs2-tools 1.6.4. The src tarball is on oss.oracle.com.
> 
> =================================================================> 
> $ git name-rev --tags 2d741da9367b33f559802dfabe62d96f6adc7777
> 2d741da9367b33f559802dfabe62d96f6adc7777 tags/ocfs2-tools-1.6.3~33
> 
> =================================================================>
commit 2d741da9367b33f559802dfabe62d96f6adc7777
> Author: Goldwyn Rodrigues <rgoldwyn at gmail.com>
> Date:   Mon Jul 26 15:19:25 2010 -0500
> 
>     fsck.ocfs2: Change local variable datatype to avoid infinite loop
>     
>     fsck on large filesystems goes in an infinite loop.
>     The problem is in verify_bitmap_descs(). i, a local variable is
>     declared as uint16_t and is compared with
>     ocfs2_cluster_group_sizes.cgs_cluster_groups which is uint32_t.
>     When cgs_cluster_groups is greater than 65535, i overflows and wraps
>     
>     creating an infinite loop of the following:
>             for (i = 0, blkno = ost->ost_fs->fs_first_cg_blkno;
>             
>                  i < cgs.cgs_cluster_groups;
>                  i++, blkno = i * ocfs2_clusters_to_blocks(ost->ost_fs,
>                  
>                                                            cgs.cgs_cpg)) {
So I downloaded the 1.6.4 version, compiled it by hand, and ran fsck on the 16 
TB filesystem. It took a long time (maybe 4 hours), but it did finish and 
corrected all the errors. Hope this helps.

Regards,

Josep Guerrero
IEEC