qingwei wei
2016-Dec-05 02:58 UTC
[Gluster-users] Assertion failed: lru_inode_ctx->block_num > 0
Hi, I encountered this when i do the FIO random write on the fuse mount gluster volume. After this assertion happen, the client log is filled with pending frames messages and FIO just show zero IO in the progress status. As i leave this test to run overnight, the client log file fill up with those pending frame messages and hit 28GB for around 12 hours. The client log: [2016-12-04 15:48:35.274208] W [MSGID: 109072] [dht-linkfile.c:50:dht_linkfile_lookup_cbk] 0-testSF-dht: got non-linkfile testSF-replicate-0:/.shard/21da7b64-45e5-4c6a-9244-53d0284bf7ed.7038, gfid = 00000000-0000-0000-0000-000000000000 [2016-12-04 15:48:35.277208] W [MSGID: 109072] [dht-linkfile.c:50:dht_linkfile_lookup_cbk] 0-testSF-dht: got non-linkfile testSF-replicate-0:/.shard/21da7b64-45e5-4c6a-9244-53d0284bf7ed.8957, gfid = 00000000-0000-0000-0000-000000000000 [2016-12-04 15:48:35.277588] W [MSGID: 109072] [dht-linkfile.c:50:dht_linkfile_lookup_cbk] 0-testSF-dht: got non-linkfile testSF-replicate-0:/.shard/21da7b64-45e5-4c6a-9244-53d0284bf7ed.11912, gfid = 00000000-0000-0000-0000-000000000000 [2016-12-04 15:48:35.312751] E [shard.c:460:__shard_update_shards_inode_list] (-->/usr/lib64/glusterfs/3.7.17/xlator/features/shard.so(shard_common_lookup_shards_cbk+0x2d) [0x7f86cc42efdd] -->/usr/lib64/glusterfs/3.7.17/xlator/features/shard.so(shard_link_block_inode+0xdf) [0x7f86cc42ef6f] -->/usr/lib64/glusterfs/3.7.17/xlator/features/shard.so(__shard_update_shards_inode_list+0x22e) [0x7f86cc42a1ce] ) 0-: Assertion failed: lru_inode_ctx->block_num > 0 pending frames: frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) Gluster info (i am testing this on one server with each disk representing one brick, this gluster volume is then mounted locally via fuse) Volume Name: testSF Type: Distributed-Replicate Volume ID: 3f205363-5029-40d7-b1b5-216f9639b454 Status: Started Number of Bricks: 2 x 3 = 6 Transport-type: tcp Bricks: Brick1: 192.168.123.4:/mnt/sdb_mssd/testSF Brick2: 192.168.123.4:/mnt/sdc_mssd/testSF Brick3: 192.168.123.4:/mnt/sdd_mssd/testSF Brick4: 192.168.123.4:/mnt/sde_mssd/testSF Brick5: 192.168.123.4:/mnt/sdf_mssd/testSF Brick6: 192.168.123.4:/mnt/sdg_mssd/testSF Options Reconfigured: features.shard-block-size: 16MB features.shard: on performance.readdir-ahead: on Gluster version: 3.7.17 The actual disk usage (Is about 91% full): /dev/sdb1 235G 202G 22G 91% /mnt/sdb_mssd /dev/sdc1 235G 202G 22G 91% /mnt/sdc_mssd /dev/sdd1 235G 202G 22G 91% /mnt/sdd_mssd /dev/sde1 235G 200G 23G 90% /mnt/sde_mssd /dev/sdf1 235G 200G 23G 90% /mnt/sdf_mssd /dev/sdg1 235G 200G 23G 90% /mnt/sdg_mssd Anyone encounter this issue before? Cw