Gluster users - Jan 2018 - Segfaults after upgrade to GlusterFS 3.10.9

Hi,

after upgrading to 3.10.9 I'm seing ganesha.nfsd segfaulting all the time:

[12407.918249] ganesha.nfsd[38104]: segfault at 0 ip 00007f872425fb00 sp
00007f867cefe5d0 error 4 in libglusterfs.so.0.0.1[7f8724223000+f1000]
[12693.119259] ganesha.nfsd[3610]: segfault at 0 ip 00007f716d8f5b00 sp
00007f71367e15d0 error 4 in libglusterfs.so.0.0.1[7f716d8b9000+f1000]
[14531.582667] ganesha.nfsd[17025]: segfault at 0 ip 00007f7cb8fa8b00 sp
00007f7c5878d5d0 error 4 in libglusterfs.so.0.0.1[7f7cb8f6c000+f1000]

ganesha-fgapi.log shows the following errors:

[2018-01-18 17:24:00.146094] W [inode.c:1341:inode_parent]
(-->/lib64/libgfapi.so.0(glfs_resolve_at+0x278) [0x7f7cb927f0b8]
-->/lib64/libglusterfs.so.0(glusterfs_normalize_dentry+0x8e) [0x7f7cb8fa8aee]
-->/lib64/libglusterfs.so.0(inode_parent+0xda) [0x7f7cb8fa670a] ) 0-gfapi:
inode not found
[2018-01-18 17:24:00.146210] E [inode.c:2567:inode_parent_null_check]
(-->/lib64/libgfapi.so.0(glfs_resolve_at+0x278) [0x7f7cb927f0b8]
-->/lib64/libglusterfs.so.0(glusterfs_normalize_dentry+0xa0) [0x7f7cb8fa8b00]
-->/lib64/libglusterfs.so.0(+0x398c4) [0x7f7cb8fa58c4] ) 0-inode: invalid
argument: inode [Invalid argument]

This leads to serious availability issues.

Is this a known issue? Any workaround available?

FWIW, my GlusterFS volume looks like this:

Volume Name: gfsvol
Type: Distributed-Replicate
Volume ID: f7985bf3-67e1-49d6-90bf-16816536533b
Status: Started
Snapshot Count: 0
Number of Bricks: 4 x 3 = 12
Transport-type: tcp
Bricks:
Brick1: AAA:/bricks/gfsvol/vol1/volume
Brick2: BBB:/bricks/gfsvol/vol1/volume
Brick3: CCC:/bricks/gfsvol/vol1/volume
Brick4: AAA:/bricks/gfsvol/vol2/volume
Brick5: BBB:/bricks/gfsvol/vol2/volume
Brick6: CCC:/bricks/gfsvol/vol2/volume
Brick7: AAA:/bricks/gfsvol/vol3/volume
Brick8: BBB:/bricks/gfsvol/vol3/volume
Brick9: CCC:/bricks/gfsvol/vol3/volume
Brick10: AAA:/bricks/gfsvol/vol4/volume
Brick11: BBB:/bricks/gfsvol/vol4/volume
Brick12: CCC:/bricks/gfsvol/vol4/volume
Options Reconfigured:
nfs.disable: on
transport.address-family: inet
features.cache-invalidation: off
ganesha.enable: on
auth.allow: *
nfs.rpc-auth-allow: *
nfs-ganesha: enable
cluster.enable-shared-storage: enable


Thanks
- Frank

Hi Frank,

It will be very easy to debug if u have core file with u. It looks like 
crash is coming from gfapi stack.

If there is core file can u please share bt of the core file.

Regards,

Jiffin


On Thursday 18 January 2018 11:18 PM, Frank Wall wrote:
> Hi,
>
> after upgrading to 3.10.9 I'm seing ganesha.nfsd segfaulting all the
time:
>
> [12407.918249] ganesha.nfsd[38104]: segfault at 0 ip 00007f872425fb00 sp
00007f867cefe5d0 error 4 in libglusterfs.so.0.0.1[7f8724223000+f1000]
> [12693.119259] ganesha.nfsd[3610]: segfault at 0 ip 00007f716d8f5b00 sp
00007f71367e15d0 error 4 in libglusterfs.so.0.0.1[7f716d8b9000+f1000]
> [14531.582667] ganesha.nfsd[17025]: segfault at 0 ip 00007f7cb8fa8b00 sp
00007f7c5878d5d0 error 4 in libglusterfs.so.0.0.1[7f7cb8f6c000+f1000]
>
> ganesha-fgapi.log shows the following errors:
>
> [2018-01-18 17:24:00.146094] W [inode.c:1341:inode_parent]
(-->/lib64/libgfapi.so.0(glfs_resolve_at+0x278) [0x7f7cb927f0b8]
-->/lib64/libglusterfs.so.0(glusterfs_normalize_dentry+0x8e) [0x7f7cb8fa8aee]
-->/lib64/libglusterfs.so.0(inode_parent+0xda) [0x7f7cb8fa670a] ) 0-gfapi:
inode not found
> [2018-01-18 17:24:00.146210] E [inode.c:2567:inode_parent_null_check]
(-->/lib64/libgfapi.so.0(glfs_resolve_at+0x278) [0x7f7cb927f0b8]
-->/lib64/libglusterfs.so.0(glusterfs_normalize_dentry+0xa0) [0x7f7cb8fa8b00]
-->/lib64/libglusterfs.so.0(+0x398c4) [0x7f7cb8fa58c4] ) 0-inode: invalid
argument: inode [Invalid argument]
>
> This leads to serious availability issues.
>
> Is this a known issue? Any workaround available?
>
> FWIW, my GlusterFS volume looks like this:
>
> Volume Name: gfsvol
> Type: Distributed-Replicate
> Volume ID: f7985bf3-67e1-49d6-90bf-16816536533b
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 4 x 3 = 12
> Transport-type: tcp
> Bricks:
> Brick1: AAA:/bricks/gfsvol/vol1/volume
> Brick2: BBB:/bricks/gfsvol/vol1/volume
> Brick3: CCC:/bricks/gfsvol/vol1/volume
> Brick4: AAA:/bricks/gfsvol/vol2/volume
> Brick5: BBB:/bricks/gfsvol/vol2/volume
> Brick6: CCC:/bricks/gfsvol/vol2/volume
> Brick7: AAA:/bricks/gfsvol/vol3/volume
> Brick8: BBB:/bricks/gfsvol/vol3/volume
> Brick9: CCC:/bricks/gfsvol/vol3/volume
> Brick10: AAA:/bricks/gfsvol/vol4/volume
> Brick11: BBB:/bricks/gfsvol/vol4/volume
> Brick12: CCC:/bricks/gfsvol/vol4/volume
> Options Reconfigured:
> nfs.disable: on
> transport.address-family: inet
> features.cache-invalidation: off
> ganesha.enable: on
> auth.allow: *
> nfs.rpc-auth-allow: *
> nfs-ganesha: enable
> cluster.enable-shared-storage: enable
>
>
> Thanks
> - Frank
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users

Hi Jiffin,

On Fri, Jan 19, 2018 at 09:19:45AM +0530, Jiffin Tony Thottan
wrote:
> It will be very easy to debug if u have core file with u. It looks like 
> crash is coming from gfapi stack.
> 
> If there is core file can u please share bt of the core file.
I'm sorry, there are no core files. :(
Is there something I could do to trigger the creation of core files?
Or anything else that would help to track down the root cause?


Regards
- Frank