Ocfs2 devel - May 2020 - NFS clients crash OCFS2 nodes (general protection fault: 0000 [#1] SMP PTI)

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Hi,

I'm experiencing quite reproducable crashs on my OCFS2 cluster the last
weeks, and I think I found out part of the Problem. But before I make a
bug report (I'm not sure where), I though I could post my research here
in case someone understands it better than I.

When it happens, the kernel is giving me a 'general protection fault:
0000 [#1] SMP PTI' or ' BUG: unable to handle page fault for
address:'
message, local access to the OCFS2 volumes are all hanging forever, and
the load is getting so high that only a hard reset "solves" the
problem.
It happened the same way on both nodes (always on the active node
serving NFS). It happens when both nodes are up, or when only one is
active.

First, a few words about my setup :

- - storage is Fiberchannel LUNs served by a EMC VNX5200 (4x 5TB).

- - LUNs are encrypted with LUKS

- - LUKS volumes are formated with OCFS2

- - heartbeat=local

- - The OCFS2 cluster is communicating on a dedictated IP over InfinyBand
direct link (actually a fail-over bonding of the 2 ib interfaces).

- - nodes are 2 debian buster hardware servers (24 cores, 128GB RAM).

- - I use corosync for cluster management, but I don't think it's
related

- - The cluster serves 4 NFS exports (3 are NFSv3, and one is NFSv4 with
kerberos) and samba shares

- - The samba shares are served in parallel by the 2 nodes managed by ctdb

- - The NFS exports are served by one of the node having the dedicated IP
for this export.

- - When corosync is moving the IP of a NFS export to the other node, it
restarts nfsd to issue a grace time of 10s on the new active node.

- - The bug appeared with default debian buster kernel 4.19.0-9-amd64 and
with the backported kernel 5.5.0-0.bpo.2-amd64 as well


My cluster went live a few weeks ago, and it first worked with both
nodes active. During some maintenance work where the active NFS node had
to switch, I had a a bug, that I was able to reproduce against my will
at least 20 times in the last nights :

1) around 100 NFS clients are connected with the exports and happy
working with it

2) The IPs of the NFS shares are switching from nodeA to nodeB

3) nodeB's kernel says "general protection fault: 0000 [#1] SMP
PTI"

4) nodeB's load is getting high

5) access to the OCFS2 is hanging (clean reboot never end, system cannot
unmount OCFS volumes)

6) I power cycle nodeB

7) I start nodeA again

8) nodeA give the same symptoms, as soon as some NFS clients do some
special requests


Most of the time, when the IP jumps to the other node, I get some of
this messages :


[? 501.707566] (nfsd,2133,14):ocfs2_test_inode_bit:2841 ERROR: unable to
get alloc inode in slot 65535
[? 501.716660] (nfsd,2133,14):ocfs2_test_inode_bit:2867 ERROR: status = -22
[? 501.726585] (nfsd,2133,6):ocfs2_test_inode_bit:2841 ERROR: unable to
get alloc inode in slot 65535
[? 501.735579] (nfsd,2133,6):ocfs2_test_inode_bit:2867 ERROR: status = -22

But it also happens when the node is not crashing, so I think it's
another Problem.

Last night, I saved a few of the kernel output while the servers where
crashing one after the other like in a ping-pong game :


- -----8< on nodeB? 8<-----

[? 502.070431] nfsd: nfsv4 idmapping failing: has idmapd not been started?
[? 502.475747] general protection fault: 0000 [#1] SMP PTI
[? 502.481027] CPU: 6 PID: 2104 Comm: nfsd Not tainted
5.5.0-0.bpo.2-amd64 #1 Debian 5.5.17-1~bpo10+1
[? 502.490016] Hardware name: IBM System x3650 M3
- -[7945UHV]-/94Y7614???? , BIOS -[D6E150CUS-1.11]- 02/08/2011
[? 502.499821] RIP: 0010:_raw_spin_lock+0xc/0x20
[? 502.504206] Code: 66 90 66 66 90 31 c0 ba ff 00 00 00 f0 0f b1 17 75
05 48 89 d8 5b c3 e8 12 5f 91 ff eb f4 66 66 66 66 90 31 c0 ba 01 00 00
00 <f0> 0f b1 17 75 01 c3 89 c6 e8 76 46 91 ff 66 90 c3 0f 1f 00 66 66
[? 502.523018] RSP: 0018:ffffa5af4838bac0 EFLAGS: 00010246
[? 502.528277] RAX: 0000000000000000 RBX: 0fdbbeded053c1d6 RCX:
0000000000000000
[? 502.535433] RDX: 0000000000000001 RSI: 0000000000000009 RDI:
0fdbbeded053c25e
[? 502.542588] RBP: 0fdbbeded053c25e R08: ffff9672ffa73068 R09:
000000000002c340
[? 502.549743] R10: 00000184d03a2115 R11: 0000000000000000 R12:
ffff9672eff43fd0
[? 502.556899] R13: ffff9672fff66bc0 R14: 000000000000ffff R15:
ffff9672fff660c8
[? 502.564055] FS:? 0000000000000000(0000) GS:ffff968203a80000(0000)
knlGS:0000000000000000
[? 502.572166] CS:? 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[? 502.577928] CR2: 00007f0a4bb5df50 CR3: 0000001e4169a004 CR4:
00000000000206e0
[? 502.585082] Call Trace:
[? 502.587546]? igrab+0x19/0x50
[? 502.590484]? ocfs2_get_system_file_inode+0x65/0x2c0 [ocfs2]
[? 502.596094]? ? ocfs2_read_blocks_sync+0x159/0x330 [ocfs2]
[? 502.601534]? ocfs2_test_inode_bit+0xe8/0x900 [ocfs2]
[? 502.606536]? ? ocfs2_free_dir_lookup_result+0x24/0x50 [ocfs2]
[? 502.612321]? ocfs2_get_parent+0xa3/0x300 [ocfs2]
[? 502.616960]? reconnect_path+0xa1/0x2c0
[? 502.620742]? ? nfsd_proc_setattr+0x1b0/0x1b0 [nfsd]
[? 502.625636]? exportfs_decode_fh+0x111/0x2d0
[? 502.629844]? ? exp_find_key+0xcd/0x160 [nfsd]
[? 502.634218]? ? __kmalloc+0x180/0x270
[? 502.637810]? ? security_prepare_creds+0x6f/0xa0
[? 502.642356]? ? tomoyo_write_self+0x1b0/0x1b0
[? 502.646642]? ? security_prepare_creds+0x49/0xa0
[? 502.651995]? fh_verify+0x3e5/0x5f0 [nfsd]
[? 502.656737]? nfsd3_proc_getattr+0x6b/0x100 [nfsd]
[? 502.662152]? nfsd_dispatch+0x9e/0x210 [nfsd]
[? 502.667136]? svc_process_common+0x386/0x6f0 [sunrpc]
[? 502.672791]? ? svc_sock_secure_port+0x12/0x30 [sunrpc]
[? 502.678628]? ? svc_recv+0x2ff/0x9c0 [sunrpc]
[? 502.683597]? ? nfsd_svc+0x2c0/0x2c0 [nfsd]
[? 502.688395]? ? nfsd_destroy+0x50/0x50 [nfsd]
[? 502.693377]? svc_process+0xd1/0x110 [sunrpc]
[? 502.698361]? nfsd+0xe3/0x140 [nfsd]
[? 502.702561]? kthread+0x112/0x130
[? 502.706504]? ? kthread_park+0x80/0x80
[? 502.710886]? ret_from_fork+0x35/0x40
[? 502.715181] Modules linked in: rpcsec_gss_krb5 nfsd nfs_acl lockd
grace ocfs2 quota_tree sctp libcrc32c nft_counter ipt_REJECT
nf_reject_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink binfmt_misc
ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager
cpufreq_userspace ocfs2_stackglue cpufreq_powersave cpufreq_conservative
configfs bonding dm_round_robin dm_multipath scsi_dh_rdac scsi_dh_emc
scsi_dh_alua intel_powerclamp coretemp kvm_intel ipmi_ssif kvm cdc_ether
irqbypass usbnet mii tpm_tis intel_cstate tpm_tis_core intel_uncore
joydev tpm ipmi_si ioatdma sg iTCO_wdt ib_mthca iTCO_vendor_support
watchdog dca pcspkr ipmi_devintf rng_core ib_uverbs ipmi_msghandler
i5500_temp i7core_edac evdev ib_umad ib_ipoib ib_cm ib_core auth_rpcgss
sunrpc ip_tables x_tables autofs4 algif_skcipher af_alg ext4 crc16
mbcache jbd2 crc32c_generic dm_crypt dm_mod sd_mod hid_generic usbhid
hid sr_mod cdrom crct10dif_pclmul crc32_pclmul crc32c_intel
ghash_clmulni_intel mgag200 drm_vram_helper drm_ttm_helper
[? 502.715218]? i2c_algo_bit ttm qla2xxx drm_kms_helper drm ata_generic
ata_piix aesni_intel libata nvme_fc ehci_pci uhci_hcd crypto_simd
ehci_hcd nvme_fabrics cryptd glue_helper lpc_ich nvme_core megaraid_sas
mfd_core usbcore scsi_transport_fc e1000e scsi_mod i2c_i801 usb_common
ptp pps_core bnx2 button
[? 502.837917] ---[ end trace 6e1a8e507ac30d8d ]---
[? 502.843379] RIP: 0010:_raw_spin_lock+0xc/0x20
[? 502.848584] Code: 66 90 66 66 90 31 c0 ba ff 00 00 00 f0 0f b1 17 75
05 48 89 d8 5b c3 e8 12 5f 91 ff eb f4 66 66 66 66 90 31 c0 ba 01 00 00
00 <f0> 0f b1 17 75 01 c3 89 c6 e8 76 46 91 ff 66 90 c3 0f 1f 00 66 66
[? 502.869295] RSP: 0018:ffffa5af4838bac0 EFLAGS: 00010246
[? 502.875404] RAX: 0000000000000000 RBX: 0fdbbeded053c1d6 RCX:
0000000000000000
[? 502.883423] RDX: 0000000000000001 RSI: 0000000000000009 RDI:
0fdbbeded053c25e
[? 502.891436] RBP: 0fdbbeded053c25e R08: ffff9672ffa73068 R09:
000000000002c340
[? 502.899446] R10: 00000184d03a2115 R11: 0000000000000000 R12:
ffff9672eff43fd0
[? 502.907455] R13: ffff9672fff66bc0 R14: 000000000000ffff R15:
ffff9672fff660c8
[? 502.915473] FS:? 0000000000000000(0000) GS:ffff968203a80000(0000)
knlGS:0000000000000000
[? 502.924496] CS:? 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[? 502.931147] CR2: 00007f0a4bb5df50 CR3: 0000001e4169a004 CR4:
00000000000206e0

- -----8< cut here 8<-----


shortly after when trying to start nodeA while nodeB was down :

- -----8< nodeA 8<-----

[? 728.927750] general protection fault: 0000 [#1] SMP PTI
[? 728.934647] CPU: 8 PID: 12156 Comm: nfsd Not tainted
5.5.0-0.bpo.2-amd64 #1 Debian 5.5.17-1~bpo10+1
[? 728.944800] Hardware name: IBM System x3650 M3 -[7945H2G]-/69Y4438,
BIOS -[D6E158AUS-1.16]- 11/26/2012
[? 728.955176] RIP: 0010:_raw_spin_lock+0xc/0x20
[? 728.960595] Code: 66 90 66 66 90 31 c0 ba ff 00 00 00 f0 0f b1 17 75
05 48 89 d8 5b c3 e8 12 5f 91 ff eb f4 66 66 66 66 90 31 c0 ba 01 00 00
00 <f0> 0f b1 17 75 01 c3 89 c6 e8 76 46 91 ff 66 90 c3 0f 1f 00 66 66
[? 728.981559] RSP: 0018:ffffa24d09187ac0 EFLAGS: 00010246
[? 728.987885] RAX: 0000000000000000 RBX: 625f656369766564 RCX:
0000000000000000
[? 728.996135] RDX: 0000000000000001 RSI: 0000000000000009 RDI:
625f6563697665ec
[? 729.004384] RBP: 625f6563697665ec R08: ffff91f1272e3680 R09:
000000000002c340
[? 729.012628] R10: 000002114c00d22b R11: 0000000000000000 R12:
ffff91e13858bbd0
[? 729.020877] R13: ffff91f13c03bbc0 R14: 000000000000ffff R15:
ffff91f13c03b0c8
[? 729.029108] FS:? 0000000000000000(0000) GS:ffff91f13fa80000(0000)
knlGS:0000000000000000
[? 729.038308] CS:? 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[? 729.045161] CR2: 000055fe95d1d9a8 CR3: 0000001c8420a003 CR4:
00000000000206e0
[? 729.053415] Call Trace:
[? 729.056978]? igrab+0x19/0x50
[? 729.061023]? ocfs2_get_system_file_inode+0x65/0x2c0 [ocfs2]
[? 729.067736]? ? ocfs2_read_blocks_sync+0x159/0x330 [ocfs2]
[? 729.074284]? ocfs2_test_inode_bit+0xe8/0x900 [ocfs2]
[? 729.080386]? ? ocfs2_free_dir_lookup_result+0x24/0x50 [ocfs2]
[? 729.087268]? ocfs2_get_parent+0xa3/0x300 [ocfs2]
[? 729.092995]? reconnect_path+0xa1/0x2c0
[? 729.097863]? ? nfsd_proc_setattr+0x1b0/0x1b0 [nfsd]
[? 729.103840]? exportfs_decode_fh+0x111/0x2d0
[? 729.109133]? ? exp_find_key+0xcd/0x160 [nfsd]
[? 729.114574]? ? dbs_update_util_handler+0x16/0x90
[? 729.120273]? ? __kmalloc+0x180/0x270
[? 729.124920]? ? security_prepare_creds+0x6f/0xa0
[? 729.130521]? ? tomoyo_write_self+0x1b0/0x1b0
[? 729.135858]? ? security_prepare_creds+0x49/0xa0
[? 729.141464]? fh_verify+0x3e5/0x5f0 [nfsd]
[? 729.146536]? nfsd3_proc_getattr+0x6b/0x100 [nfsd]
[? 729.152297]? nfsd_dispatch+0x9e/0x210 [nfsd]
[? 729.157647]? svc_process_common+0x386/0x6f0 [sunrpc]
[? 729.163675]? ? svc_sock_secure_port+0x12/0x30 [sunrpc]
[? 729.169873]? ? svc_recv+0x2ff/0x9c0 [sunrpc]
[? 729.175190]? ? nfsd_svc+0x2c0/0x2c0 [nfsd]
[? 729.180323]? ? nfsd_destroy+0x50/0x50 [nfsd]
[? 729.185631]? svc_process+0xd1/0x110 [sunrpc]
[? 729.190925]? nfsd+0xe3/0x140 [nfsd]
[? 729.195421]? kthread+0x112/0x130
[? 729.199636]? ? kthread_park+0x80/0x80
[? 729.204279]? ret_from_fork+0x35/0x40
[? 729.208818] Modules linked in: ocfs2 quota_tree dm_crypt crypto_simd
cryptd glue_helper algif_skcipher af_alg nfsd nfs_acl lockd grace sctp
libcrc32c nft_counter xt_tcpudp nft_compat nf_tables nfnetlink
binfmt_misc ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager
ocfs2_stackglue configfs bonding dm_round_robin dm_multipath
scsi_dh_rdac scsi_dh_emc scsi_dh_alua intel_powerclamp ipmi_ssif
coretemp kvm_intel kvm irqbypass cdc_ether intel_cstate usbnet ib_mthca
mii joydev intel_uncore ib_uverbs iTCO_wdt sg ioatdma
iTCO_vendor_support i7core_edac watchdog i5500_temp pcspkr ipmi_si
tpm_tis tpm_tis_core tpm ipmi_devintf ipmi_msghandler rng_core evdev
acpi_cpufreq ib_umad auth_rpcgss ib_ipoib ib_cm sunrpc ib_core ip_tables
x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic dm_mod
hid_generic sd_mod usbhid hid sr_mod cdrom uas usb_storage mgag200
drm_vram_helper drm_ttm_helper qla2xxx ata_generic i2c_algo_bit ttm
nvme_fc ixgbe nvme_fabrics ata_piix xfrm_algo ehci_pci uhci_hcd
[? 729.208868]? drm_kms_helper nvme_core ehci_hcd dca libata
megaraid_sas scsi_transport_fc libphy ptp drm usbcore scsi_mod
crc32c_intel lpc_ich i2c_i801 pps_core mfd_core usb_common bnx2 mdio button
[? 729.323900] ---[ end trace 212cefe75207e17f ]---
[? 729.329644] RIP: 0010:_raw_spin_lock+0xc/0x20
[? 729.335121] Code: 66 90 66 66 90 31 c0 ba ff 00 00 00 f0 0f b1 17 75
05 48 89 d8 5b c3 e8 12 5f 91 ff eb f4 66 66 66 66 90 31 c0 ba 01 00 00
00 <f0> 0f b1 17 75 01 c3 89 c6 e8 76 46 91 ff 66 90 c3 0f 1f 00 66 66
[? 729.356192] RSP: 0018:ffffa24d09187ac0 EFLAGS: 00010246
[? 729.362584] RAX: 0000000000000000 RBX: 625f656369766564 RCX:
0000000000000000
[? 729.370889] RDX: 0000000000000001 RSI: 0000000000000009 RDI:
625f6563697665ec
[? 729.379184] RBP: 625f6563697665ec R08: ffff91f1272e3680 R09:
000000000002c340
[? 729.387475] R10: 000002114c00d22b R11: 0000000000000000 R12:
ffff91e13858bbd0
[? 729.395767] R13: ffff91f13c03bbc0 R14: 000000000000ffff R15:
ffff91f13c03b0c8
[? 729.404066] FS:? 0000000000000000(0000) GS:ffff91f13fa80000(0000)
knlGS:0000000000000000
[? 729.413336] CS:? 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[? 729.420262] CR2: 000055fe95d1d9a8 CR3: 0000001c8420a003 CR4:
00000000000206e0

- -----8< cut here 8<-----

Since the backtrace is speaking about nfsd, I tried to isolate the
Problem by droping all NFS packet entering to the active node before
starting the services :

iptables -I INPUT -p tcp --dport 2049 -j DROP

Everything fine, access over samba were doing fine.

I started to allow NFS for part of my network with iptables, until the
crash occurs.

On this one, the crash occured after allowing only a NFSv3 client.

- -----8< nodeB 8<-----

[ 2494.893697] BUG: unable to handle page fault for address:
00000000000fab7d
[ 2494.900626] #PF: supervisor write access in kernel mode
[ 2494.905873] #PF: error_code(0x0002) - not-present page
[ 2494.911042] PGD 0 P4D 0
[ 2494.913596] Oops: 0002 [#1] SMP PTI
[ 2494.917106] CPU: 11 PID: 21342 Comm: nfsd Not tainted
5.5.0-0.bpo.2-amd64 #1 Debian 5.5.17-1~bpo10+1
[ 2494.926285] Hardware name: IBM System x3650 M3
- -[7945UHV]-/94Y7614???? , BIOS -[D6E150CUS-1.11]- 02/08/2011
[ 2494.936077] RIP: 0010:_raw_spin_lock+0xc/0x20
[ 2494.940461] Code: 66 90 66 66 90 31 c0 ba ff 00 00 00 f0 0f b1 17 75
05 48 89 d8 5b c3 e8 12 5f 91 ff eb f4 66 66 66 66 90 31 c0 ba 01 00 00
00 <f0> 0f b1 17 75 01 c3 89 c6 e8 76 46 91 ff 66 90 c3 0f 1f 00 66 66
[ 2494.959269] RSP: 0018:ffffb949cca7fac0 EFLAGS: 00010246
[ 2494.964511] RAX: 0000000000000000 RBX: 00000000000faaf5 RCX:
0000000000000000
[ 2494.971664] RDX: 0000000000000001 RSI: 0000000000000009 RDI:
00000000000fab7d
[ 2494.978815] RBP: 00000000000fab7d R08: ffff99c8f12737b8 R09:
000000000002c340
[ 2494.985968] R10: 000005dc16e333d8 R11: 0000000000000000 R12:
ffff99c903066bd0
[ 2494.993119] R13: ffff99c9014c7bc0 R14: 000000000000ffff R15:
ffff99c9014c70c8
[ 2495.000271] FS:? 0000000000000000(0000) GS:ffff99ba03bc0000(0000)
knlGS:0000000000000000
[ 2495.008382] CS:? 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2495.014141] CR2: 00000000000fab7d CR3: 0000000f38df2002 CR4:
00000000000206e0
[ 2495.021291] Call Trace:
[ 2495.023753]? igrab+0x19/0x50
[ 2495.026693]? ocfs2_get_system_file_inode+0x65/0x2c0 [ocfs2]
[ 2495.032301]? ? ocfs2_read_blocks_sync+0x159/0x330 [ocfs2]
[ 2495.037741]? ocfs2_test_inode_bit+0xe8/0x900 [ocfs2]
[ 2495.042742]? ? ocfs2_free_dir_lookup_result+0x24/0x50 [ocfs2]
[ 2495.048525]? ocfs2_get_parent+0xa3/0x300 [ocfs2]
[ 2495.053163]? reconnect_path+0xa1/0x2c0
[ 2495.056943]? ? nfsd_proc_setattr+0x1b0/0x1b0 [nfsd]
[ 2495.061838]? exportfs_decode_fh+0x111/0x2d0
[ 2495.066042]? ? exp_find_key+0xcd/0x160 [nfsd]
[ 2495.070417]? ? __kmalloc+0x180/0x270
[ 2495.074007]? ? security_prepare_creds+0x6f/0xa0
[ 2495.078551]? ? tomoyo_write_self+0x1b0/0x1b0
[ 2495.082834]? ? security_prepare_creds+0x49/0xa0
[ 2495.087387]? fh_verify+0x3e5/0x5f0 [nfsd]
[ 2495.092125]? nfsd3_proc_getattr+0x6b/0x100 [nfsd]
[ 2495.097517]? nfsd_dispatch+0x9e/0x210 [nfsd]
[ 2495.102489]? svc_process_common+0x386/0x6f0 [sunrpc]
[ 2495.108134]? ? svc_sock_secure_port+0x12/0x30 [sunrpc]
[ 2495.113946]? ? svc_recv+0x2ff/0x9c0 [sunrpc]
[ 2495.118883]? ? nfsd_svc+0x2c0/0x2c0 [nfsd]
[ 2495.123651]? ? nfsd_destroy+0x50/0x50 [nfsd]
[ 2495.128604]? svc_process+0xd1/0x110 [sunrpc]
[ 2495.133563]? nfsd+0xe3/0x140 [nfsd]
[ 2495.137733]? kthread+0x112/0x130
[ 2495.141643]? ? kthread_park+0x80/0x80
[ 2495.145993]? ret_from_fork+0x35/0x40
[ 2495.150262] Modules linked in: ocfs2 quota_tree nfsd nfs_acl lockd
grace sctp libcrc32c nft_counter xt_tcpudp nft_compat nf_tables
nfnetlink binfmt_misc ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm
ocfs2_nodemanager ocfs2_stackglue cpufreq_userspace cpufreq_powersave
cpufreq_conservative configfs bonding dm_round_robin dm_multipath
scsi_dh_rdac scsi_dh_emc scsi_dh_alua intel_powerclamp coretemp
kvm_intel kvm ipmi_ssif irqbypass cdc_ether usbnet intel_cstate mii
ib_mthca intel_uncore evdev iTCO_wdt joydev ipmi_si tpm_tis sg
tpm_tis_core tpm ipmi_devintf iTCO_vendor_support ib_uverbs pcspkr
watchdog ioatdma rng_core i7core_edac dca i5500_temp ipmi_msghandler
ib_umad ib_ipoib auth_rpcgss ib_cm sunrpc ib_core ip_tables x_tables
autofs4 algif_skcipher af_alg ext4 crc16 mbcache jbd2 crc32c_generic
dm_crypt dm_mod sr_mod cdrom sd_mod hid_generic usbhid hid
crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel mgag200
drm_vram_helper drm_ttm_helper i2c_algo_bit ttm qla2xxx drm_kms_helper
[ 2495.150298]? ata_generic drm ata_piix aesni_intel libata nvme_fc
uhci_hcd crypto_simd nvme_fabrics ehci_pci ehci_hcd megaraid_sas cryptd
nvme_core glue_helper usbcore scsi_transport_fc lpc_ich mfd_core e1000e
scsi_mod i2c_i801 usb_common bnx2 ptp pps_core button
[ 2495.269073] CR2: 00000000000fab7d
[ 2495.273205] ---[ end trace 44f000505f987296 ]---
[ 2495.278647] RIP: 0010:_raw_spin_lock+0xc/0x20
[ 2495.283825] Code: 66 90 66 66 90 31 c0 ba ff 00 00 00 f0 0f b1 17 75
05 48 89 d8 5b c3 e8 12 5f 91 ff eb f4 66 66 66 66 90 31 c0 ba 01 00 00
00 <f0> 0f b1 17 75 01 c3 89 c6 e8 76 46 91 ff 66 90 c3 0f 1f 00 66 66
[ 2495.304287] RSP: 0018:ffffb949cca7fac0 EFLAGS: 00010246
[ 2495.310368] RAX: 0000000000000000 RBX: 00000000000faaf5 RCX:
0000000000000000
[ 2495.318358] RDX: 0000000000000001 RSI: 0000000000000009 RDI:
00000000000fab7d
[ 2495.326339] RBP: 00000000000fab7d R08: ffff99c8f12737b8 R09:
000000000002c340
[ 2495.334314] R10: 000005dc16e333d8 R11: 0000000000000000 R12:
ffff99c903066bd0
[ 2495.342297] R13: ffff99c9014c7bc0 R14: 000000000000ffff R15:
ffff99c9014c70c8
[ 2495.350279] FS:? 0000000000000000(0000) GS:ffff99ba03bc0000(0000)
knlGS:0000000000000000
[ 2495.359224] CS:? 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2495.365825] CR2: 00000000000fab7d CR3: 0000000f38df2002 CR4:
00000000000206e0

- -----8< cut here 8<-----


I also made some record of NFS traffic with tcpdump when the crashs are
occuring, but I didn't had time to analyse it. I can provide them if it
helps (but I might have to clean them up first to avoid some data leaks :)

After this happening a few times, I had one case where one of the few
NFSv4 client was involved, and I took as aworkaround to migrate the
NFSv4 share to XFS since I though it could be NFSv4 related. Now that I
looked at the kernel dumps, I think it's also NFSv3 related.

Is it a kernel bug or am I missing something ?

For now, I will switch back to XFS with a failover setup of my cluster,
cause I cannot afford this kind of crashes of a central file server.



Cheers


Pierre


PS : I'll try to update with what I found in the PCAP files in some days



-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEjXhlMuC3bFbPA3S/qC81yulnAywFAl7DuYUACgkQqC81yuln
AyyWHg/+NUMgiysN+QesGedzDjfKiUSTKqPvbg3pUklq3bXfidLdIcM6HGZt31Gy
/k33m83+675c3vQeez8uvSVAh7uNzQW2m7BGrzEv/ZkpwIZxqIwiV+gi/zc7lnGP
vqkjAqVjAapk254W3aTonZKpExKCbp6h1g1eVWKk43Mu7/UaLkQcjDPq9AyheSt+
PAPpiQeQ9ox1+OVvEnpawJmcV9QUluHPwsar/yYa87ET2LIlBswtjOHr+Vw27kJU
qwak16rTDWqVQo5PYIi5N1jdCAEAPOV6EalNNwIQ9/u+lCdbVHYRV7Ze5nE58DDz
zP7Ai9Vaj/oBr/znjbaoE2VR0iWRVqbztBxts0hNv1IxAqJivZnaaJx6bXp51NeV
kXwTz0tHqCoy8RVlUXEGP8FBuaWgSLZyfN5IOk1KoocsLUuZh/BVxh+snDj4T52k
u/SRBFV4lW4RVYWHBItik8oqfMwYaJuwp2hdmU1hHsm1UnHQTQjum1L9ZfLDI7XU
NxHMGedneJePXdWH0eTqvHAPJ1U59gEL/cMPnOJF57q+eYoCs0svnRxCwQzviBok
Zak2uzIIXr3jc9bY3qiQlZivik5b+l6LK7ccoS7eEIqxOkKivOjwijuXc0EHOUR2
c0QjFUDwJ1gR8rkBE/rOiKLXhMGnv2lcIZWiu7JlxMq+ms7f/1k=6VH2
-----END PGP SIGNATURE-----

Hi Pierre,

You indeed provide detailed information on this problem and the research 
looks very nice.

On 5/19/20 6:48 PM, Pierre Dinh-van wrote:
> 
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
> 
> Hi,
> 
> I'm experiencing quite reproducable crashs on my OCFS2 cluster the last
> weeks, and I think I found out part of the Problem. But before I make a
> bug report (I'm not sure where), I though I could post my research here
> in case someone understands it better than I.
> 
> When it happens, the kernel is giving me a 'general protection fault:
> 0000 [#1] SMP PTI' or ' BUG: unable to handle page fault for
address:'
> message, local access to the OCFS2 volumes are all hanging forever, and
> the load is getting so high that only a hard reset "solves" the
problem.
> It happened the same way on both nodes (always on the active node
> serving NFS). It happens when both nodes are up, or when only one is
> active.
This makes me a little puzzled, you mean you crash the kernel manually 
or ocfs2 crash the kernel itself? From the backtrace you posted, I 
assume that you crash the kernel manually to migrate NFS export IP to 
the other standby node, right?
> 
> First, a few words about my setup :
> 
> - - storage is Fiberchannel LUNs served by a EMC VNX5200 (4x 5TB).
> 
> - - LUNs are encrypted with LUKS
> 
> - - LUKS volumes are formated with OCFS2
> 
> - - heartbeat=local
> 
> - - The OCFS2 cluster is communicating on a dedictated IP over InfinyBand
> direct link (actually a fail-over bonding of the 2 ib interfaces).
> 
> - - nodes are 2 debian buster hardware servers (24 cores, 128GB RAM).
> 
> - - I use corosync for cluster management, but I don't think it's
related
> 
> - - The cluster serves 4 NFS exports (3 are NFSv3, and one is NFSv4 with
> kerberos) and samba shares
> 
> - - The samba shares are served in parallel by the 2 nodes managed by ctdb
> 
> - - The NFS exports are served by one of the node having the dedicated IP
> for this export.
In fact, I am not quite experienced in this kind of setup where ocfs2 
serves as NFS/Samba backend filesystem, but the use case seems no 
problem to me. But considering that you have hundreds of clients 
connected to a single NFS/v4 server, do you have a performance 
limitation. This has nothing with the problem, just my personal curiosity.
> 
> - - When corosync is moving the IP of a NFS export to the other node, it
> restarts nfsd to issue a grace time of 10s on the new active node.
Sure, this looks normal step.
> 
> - - The bug appeared with default debian buster kernel 4.19.0-9-amd64 and
> with the backported kernel 5.5.0-0.bpo.2-amd64 as well
> 
> 
> My cluster went live a few weeks ago, and it first worked with both
> nodes active. During some maintenance work where the active NFS node had
> to switch, I had a a bug, that I was able to reproduce against my will
> at least 20 times in the last nights :
> 
> 1) around 100 NFS clients are connected with the exports and happy
> working with it
> 
> 2) The IPs of the NFS shares are switching from nodeA to nodeB
> 
> 3) nodeB's kernel says "general protection fault: 0000 [#1] SMP
PTI"
> 
> 4) nodeB's load is getting high
I suppose the reason is that ocfs2 can't successfully acquire spin_lock 
of a system inode(ocfs2 internal mechanism to store its metadata, not 
visible to end users ). As you know, locking path won't stop trying 
until it get the lock, so it consumes a lot of CPU cycles, you probably 
observed a high system workload.
> 
> 5) access to the OCFS2 is hanging (clean reboot never end, system cannot
> unmount OCFS volumes)
> 
> 6) I power cycle nodeB
> 
> 7) I start nodeA again
> 
> 8) nodeA give the same symptoms, as soon as some NFS clients do some
> special requests
> 
> 
> Most of the time, when the IP jumps to the other node, I get some of
> this messages :
> 
> 
> [? 501.707566] (nfsd,2133,14):ocfs2_test_inode_bit:2841 ERROR: unable to
> get alloc inode in slot 65535
This is very strange, I think ocfs2 passes the invalid slot number to 
subsequent consumer function, which makes it use a dangling pointer to a 
fake/illegal system inode. In other word, this why ocfs2 can't acquire 
the spin_lock of system inode.


Can you try to fix the logic in `ocfs2_get_suballoc_slot_bit` judging if 
slot number is valid? And perform your case once more?

Feel free to ask my help if you need. :-)

> [? 501.716660] (nfsd,2133,14):ocfs2_test_inode_bit:2867 ERROR: status = -22
Why this happens and if it is another issue needs further investigation.
> [? 501.726585] (nfsd,2133,6):ocfs2_test_inode_bit:2841 ERROR: unable to
> get alloc inode in slot 65535
> [? 501.735579] (nfsd,2133,6):ocfs2_test_inode_bit:2867 ERROR: status = -22
> 
> But it also happens when the node is not crashing, so I think it's
> another Problem.
> 
> Last night, I saved a few of the kernel output while the servers where
> crashing one after the other like in a ping-pong game :
> 
> 
> - -----8< on nodeB? 8<-----
> 
> [? 502.070431] nfsd: nfsv4 idmapping failing: has idmapd not been started?
> [? 502.475747] general protection fault: 0000 [#1] SMP PTI
> [? 502.481027] CPU: 6 PID: 2104 Comm: nfsd Not tainted
> 5.5.0-0.bpo.2-amd64 #1 Debian 5.5.17-1~bpo10+1
> [? 502.490016] Hardware name: IBM System x3650 M3
> - -[7945UHV]-/94Y7614???? , BIOS -[D6E150CUS-1.11]- 02/08/2011
> [? 502.499821] RIP: 0010:_raw_spin_lock+0xc/0x20
> [? 502.504206] Code: 66 90 66 66 90 31 c0 ba ff 00 00 00 f0 0f b1 17 75
> 05 48 89 d8 5b c3 e8 12 5f 91 ff eb f4 66 66 66 66 90 31 c0 ba 01 00 00
> 00 <f0> 0f b1 17 75 01 c3 89 c6 e8 76 46 91 ff 66 90 c3 0f 1f 00 66
66
> [? 502.523018] RSP: 0018:ffffa5af4838bac0 EFLAGS: 00010246
> [? 502.528277] RAX: 0000000000000000 RBX: 0fdbbeded053c1d6 RCX:
> 0000000000000000
> [? 502.535433] RDX: 0000000000000001 RSI: 0000000000000009 RDI:
> 0fdbbeded053c25e
> [? 502.542588] RBP: 0fdbbeded053c25e R08: ffff9672ffa73068 R09:
> 000000000002c340
> [? 502.549743] R10: 00000184d03a2115 R11: 0000000000000000 R12:
> ffff9672eff43fd0
> [? 502.556899] R13: ffff9672fff66bc0 R14: 000000000000ffff R15:
> ffff9672fff660c8
> [? 502.564055] FS:? 0000000000000000(0000) GS:ffff968203a80000(0000)
> knlGS:0000000000000000
> [? 502.572166] CS:? 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [? 502.577928] CR2: 00007f0a4bb5df50 CR3: 0000001e4169a004 CR4:
> 00000000000206e0
> [? 502.585082] Call Trace:
> [? 502.587546]? igrab+0x19/0x50
> [? 502.590484]? ocfs2_get_system_file_inode+0x65/0x2c0 [ocfs2]
> [? 502.596094]? ? ocfs2_read_blocks_sync+0x159/0x330 [ocfs2]
> [? 502.601534]? ocfs2_test_inode_bit+0xe8/0x900 [ocfs2]
> [? 502.606536]? ? ocfs2_free_dir_lookup_result+0x24/0x50 [ocfs2]
> [? 502.612321]? ocfs2_get_parent+0xa3/0x300 [ocfs2]
> [? 502.616960]? reconnect_path+0xa1/0x2c0
> [? 502.620742]? ? nfsd_proc_setattr+0x1b0/0x1b0 [nfsd]
> [? 502.625636]? exportfs_decode_fh+0x111/0x2d0
> [? 502.629844]? ? exp_find_key+0xcd/0x160 [nfsd]
> [? 502.634218]? ? __kmalloc+0x180/0x270
> [? 502.637810]? ? security_prepare_creds+0x6f/0xa0
> [? 502.642356]? ? tomoyo_write_self+0x1b0/0x1b0
> [? 502.646642]? ? security_prepare_creds+0x49/0xa0
> [? 502.651995]? fh_verify+0x3e5/0x5f0 [nfsd]
> [? 502.656737]? nfsd3_proc_getattr+0x6b/0x100 [nfsd]
> [? 502.662152]? nfsd_dispatch+0x9e/0x210 [nfsd]
> [? 502.667136]? svc_process_common+0x386/0x6f0 [sunrpc]
> [? 502.672791]? ? svc_sock_secure_port+0x12/0x30 [sunrpc]
> [? 502.678628]? ? svc_recv+0x2ff/0x9c0 [sunrpc]
> [? 502.683597]? ? nfsd_svc+0x2c0/0x2c0 [nfsd]
> [? 502.688395]? ? nfsd_destroy+0x50/0x50 [nfsd]
> [? 502.693377]? svc_process+0xd1/0x110 [sunrpc]
> [? 502.698361]? nfsd+0xe3/0x140 [nfsd]
> [? 502.702561]? kthread+0x112/0x130
> [? 502.706504]? ? kthread_park+0x80/0x80
> [? 502.710886]? ret_from_fork+0x35/0x40
> [? 502.715181] Modules linked in: rpcsec_gss_krb5 nfsd nfs_acl lockd
> grace ocfs2 quota_tree sctp libcrc32c nft_counter ipt_REJECT
> nf_reject_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink binfmt_misc
> ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager
> cpufreq_userspace ocfs2_stackglue cpufreq_powersave cpufreq_conservative
> configfs bonding dm_round_robin dm_multipath scsi_dh_rdac scsi_dh_emc
> scsi_dh_alua intel_powerclamp coretemp kvm_intel ipmi_ssif kvm cdc_ether
> irqbypass usbnet mii tpm_tis intel_cstate tpm_tis_core intel_uncore
> joydev tpm ipmi_si ioatdma sg iTCO_wdt ib_mthca iTCO_vendor_support
> watchdog dca pcspkr ipmi_devintf rng_core ib_uverbs ipmi_msghandler
> i5500_temp i7core_edac evdev ib_umad ib_ipoib ib_cm ib_core auth_rpcgss
> sunrpc ip_tables x_tables autofs4 algif_skcipher af_alg ext4 crc16
> mbcache jbd2 crc32c_generic dm_crypt dm_mod sd_mod hid_generic usbhid
> hid sr_mod cdrom crct10dif_pclmul crc32_pclmul crc32c_intel
> ghash_clmulni_intel mgag200 drm_vram_helper drm_ttm_helper
> [? 502.715218]? i2c_algo_bit ttm qla2xxx drm_kms_helper drm ata_generic
> ata_piix aesni_intel libata nvme_fc ehci_pci uhci_hcd crypto_simd
> ehci_hcd nvme_fabrics cryptd glue_helper lpc_ich nvme_core megaraid_sas
> mfd_core usbcore scsi_transport_fc e1000e scsi_mod i2c_i801 usb_common
> ptp pps_core bnx2 button
> [? 502.837917] ---[ end trace 6e1a8e507ac30d8d ]---
> [? 502.843379] RIP: 0010:_raw_spin_lock+0xc/0x20
> [? 502.848584] Code: 66 90 66 66 90 31 c0 ba ff 00 00 00 f0 0f b1 17 75
> 05 48 89 d8 5b c3 e8 12 5f 91 ff eb f4 66 66 66 66 90 31 c0 ba 01 00 00
> 00 <f0> 0f b1 17 75 01 c3 89 c6 e8 76 46 91 ff 66 90 c3 0f 1f 00 66
66
> [? 502.869295] RSP: 0018:ffffa5af4838bac0 EFLAGS: 00010246
> [? 502.875404] RAX: 0000000000000000 RBX: 0fdbbeded053c1d6 RCX:
> 0000000000000000
> [? 502.883423] RDX: 0000000000000001 RSI: 0000000000000009 RDI:
> 0fdbbeded053c25e
> [? 502.891436] RBP: 0fdbbeded053c25e R08: ffff9672ffa73068 R09:
> 000000000002c340
> [? 502.899446] R10: 00000184d03a2115 R11: 0000000000000000 R12:
> ffff9672eff43fd0
> [? 502.907455] R13: ffff9672fff66bc0 R14: 000000000000ffff R15:
> ffff9672fff660c8
> [? 502.915473] FS:? 0000000000000000(0000) GS:ffff968203a80000(0000)
> knlGS:0000000000000000
> [? 502.924496] CS:? 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [? 502.931147] CR2: 00007f0a4bb5df50 CR3: 0000001e4169a004 CR4:
> 00000000000206e0
> 
> - -----8< cut here 8<-----
> 
> 
> shortly after when trying to start nodeA while nodeB was down :
> 
> - -----8< nodeA 8<-----
> 
> [? 728.927750] general protection fault: 0000 [#1] SMP PTI
> [? 728.934647] CPU: 8 PID: 12156 Comm: nfsd Not tainted
> 5.5.0-0.bpo.2-amd64 #1 Debian 5.5.17-1~bpo10+1
> [? 728.944800] Hardware name: IBM System x3650 M3 -[7945H2G]-/69Y4438,
> BIOS -[D6E158AUS-1.16]- 11/26/2012
> [? 728.955176] RIP: 0010:_raw_spin_lock+0xc/0x20
> [? 728.960595] Code: 66 90 66 66 90 31 c0 ba ff 00 00 00 f0 0f b1 17 75
> 05 48 89 d8 5b c3 e8 12 5f 91 ff eb f4 66 66 66 66 90 31 c0 ba 01 00 00
> 00 <f0> 0f b1 17 75 01 c3 89 c6 e8 76 46 91 ff 66 90 c3 0f 1f 00 66
66
> [? 728.981559] RSP: 0018:ffffa24d09187ac0 EFLAGS: 00010246
> [? 728.987885] RAX: 0000000000000000 RBX: 625f656369766564 RCX:
> 0000000000000000
> [? 728.996135] RDX: 0000000000000001 RSI: 0000000000000009 RDI:
> 625f6563697665ec
> [? 729.004384] RBP: 625f6563697665ec R08: ffff91f1272e3680 R09:
> 000000000002c340
> [? 729.012628] R10: 000002114c00d22b R11: 0000000000000000 R12:
> ffff91e13858bbd0
> [? 729.020877] R13: ffff91f13c03bbc0 R14: 000000000000ffff R15:
> ffff91f13c03b0c8
> [? 729.029108] FS:? 0000000000000000(0000) GS:ffff91f13fa80000(0000)
> knlGS:0000000000000000
> [? 729.038308] CS:? 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [? 729.045161] CR2: 000055fe95d1d9a8 CR3: 0000001c8420a003 CR4:
> 00000000000206e0
> [? 729.053415] Call Trace:
> [? 729.056978]? igrab+0x19/0x50
> [? 729.061023]? ocfs2_get_system_file_inode+0x65/0x2c0 [ocfs2]
> [? 729.067736]? ? ocfs2_read_blocks_sync+0x159/0x330 [ocfs2]
> [? 729.074284]? ocfs2_test_inode_bit+0xe8/0x900 [ocfs2]
> [? 729.080386]? ? ocfs2_free_dir_lookup_result+0x24/0x50 [ocfs2]
> [? 729.087268]? ocfs2_get_parent+0xa3/0x300 [ocfs2]
> [? 729.092995]? reconnect_path+0xa1/0x2c0
> [? 729.097863]? ? nfsd_proc_setattr+0x1b0/0x1b0 [nfsd]
> [? 729.103840]? exportfs_decode_fh+0x111/0x2d0
> [? 729.109133]? ? exp_find_key+0xcd/0x160 [nfsd]
> [? 729.114574]? ? dbs_update_util_handler+0x16/0x90
> [? 729.120273]? ? __kmalloc+0x180/0x270
> [? 729.124920]? ? security_prepare_creds+0x6f/0xa0
> [? 729.130521]? ? tomoyo_write_self+0x1b0/0x1b0
> [? 729.135858]? ? security_prepare_creds+0x49/0xa0
> [? 729.141464]? fh_verify+0x3e5/0x5f0 [nfsd]
> [? 729.146536]? nfsd3_proc_getattr+0x6b/0x100 [nfsd]
> [? 729.152297]? nfsd_dispatch+0x9e/0x210 [nfsd]
> [? 729.157647]? svc_process_common+0x386/0x6f0 [sunrpc]
> [? 729.163675]? ? svc_sock_secure_port+0x12/0x30 [sunrpc]
> [? 729.169873]? ? svc_recv+0x2ff/0x9c0 [sunrpc]
> [? 729.175190]? ? nfsd_svc+0x2c0/0x2c0 [nfsd]
> [? 729.180323]? ? nfsd_destroy+0x50/0x50 [nfsd]
> [? 729.185631]? svc_process+0xd1/0x110 [sunrpc]
> [? 729.190925]? nfsd+0xe3/0x140 [nfsd]
> [? 729.195421]? kthread+0x112/0x130
> [? 729.199636]? ? kthread_park+0x80/0x80
> [? 729.204279]? ret_from_fork+0x35/0x40
> [? 729.208818] Modules linked in: ocfs2 quota_tree dm_crypt crypto_simd
> cryptd glue_helper algif_skcipher af_alg nfsd nfs_acl lockd grace sctp
> libcrc32c nft_counter xt_tcpudp nft_compat nf_tables nfnetlink
> binfmt_misc ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager
> ocfs2_stackglue configfs bonding dm_round_robin dm_multipath
> scsi_dh_rdac scsi_dh_emc scsi_dh_alua intel_powerclamp ipmi_ssif
> coretemp kvm_intel kvm irqbypass cdc_ether intel_cstate usbnet ib_mthca
> mii joydev intel_uncore ib_uverbs iTCO_wdt sg ioatdma
> iTCO_vendor_support i7core_edac watchdog i5500_temp pcspkr ipmi_si
> tpm_tis tpm_tis_core tpm ipmi_devintf ipmi_msghandler rng_core evdev
> acpi_cpufreq ib_umad auth_rpcgss ib_ipoib ib_cm sunrpc ib_core ip_tables
> x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic dm_mod
> hid_generic sd_mod usbhid hid sr_mod cdrom uas usb_storage mgag200
> drm_vram_helper drm_ttm_helper qla2xxx ata_generic i2c_algo_bit ttm
> nvme_fc ixgbe nvme_fabrics ata_piix xfrm_algo ehci_pci uhci_hcd
> [? 729.208868]? drm_kms_helper nvme_core ehci_hcd dca libata
> megaraid_sas scsi_transport_fc libphy ptp drm usbcore scsi_mod
> crc32c_intel lpc_ich i2c_i801 pps_core mfd_core usb_common bnx2 mdio button
> [? 729.323900] ---[ end trace 212cefe75207e17f ]---
> [? 729.329644] RIP: 0010:_raw_spin_lock+0xc/0x20
> [? 729.335121] Code: 66 90 66 66 90 31 c0 ba ff 00 00 00 f0 0f b1 17 75
> 05 48 89 d8 5b c3 e8 12 5f 91 ff eb f4 66 66 66 66 90 31 c0 ba 01 00 00
> 00 <f0> 0f b1 17 75 01 c3 89 c6 e8 76 46 91 ff 66 90 c3 0f 1f 00 66
66
> [? 729.356192] RSP: 0018:ffffa24d09187ac0 EFLAGS: 00010246
> [? 729.362584] RAX: 0000000000000000 RBX: 625f656369766564 RCX:
> 0000000000000000
> [? 729.370889] RDX: 0000000000000001 RSI: 0000000000000009 RDI:
> 625f6563697665ec
> [? 729.379184] RBP: 625f6563697665ec R08: ffff91f1272e3680 R09:
> 000000000002c340
> [? 729.387475] R10: 000002114c00d22b R11: 0000000000000000 R12:
> ffff91e13858bbd0
> [? 729.395767] R13: ffff91f13c03bbc0 R14: 000000000000ffff R15:
> ffff91f13c03b0c8
> [? 729.404066] FS:? 0000000000000000(0000) GS:ffff91f13fa80000(0000)
> knlGS:0000000000000000
> [? 729.413336] CS:? 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [? 729.420262] CR2: 000055fe95d1d9a8 CR3: 0000001c8420a003 CR4:
> 00000000000206e0
> 
> - -----8< cut here 8<-----
> 
> Since the backtrace is speaking about nfsd, I tried to isolate the
> Problem by droping all NFS packet entering to the active node before
> starting the services :
> 
> iptables -I INPUT -p tcp --dport 2049 -j DROP
> 
> Everything fine, access over samba were doing fine.
> 
> I started to allow NFS for part of my network with iptables, until the
> crash occurs.
> 
> On this one, the crash occured after allowing only a NFSv3 client.
> 
> - -----8< nodeB 8<-----
> 
> [ 2494.893697] BUG: unable to handle page fault for address:
> 00000000000fab7d
> [ 2494.900626] #PF: supervisor write access in kernel mode
> [ 2494.905873] #PF: error_code(0x0002) - not-present page
> [ 2494.911042] PGD 0 P4D 0
> [ 2494.913596] Oops: 0002 [#1] SMP PTI
> [ 2494.917106] CPU: 11 PID: 21342 Comm: nfsd Not tainted
> 5.5.0-0.bpo.2-amd64 #1 Debian 5.5.17-1~bpo10+1
> [ 2494.926285] Hardware name: IBM System x3650 M3
> - -[7945UHV]-/94Y7614???? , BIOS -[D6E150CUS-1.11]- 02/08/2011
> [ 2494.936077] RIP: 0010:_raw_spin_lock+0xc/0x20
> [ 2494.940461] Code: 66 90 66 66 90 31 c0 ba ff 00 00 00 f0 0f b1 17 75
> 05 48 89 d8 5b c3 e8 12 5f 91 ff eb f4 66 66 66 66 90 31 c0 ba 01 00 00
> 00 <f0> 0f b1 17 75 01 c3 89 c6 e8 76 46 91 ff 66 90 c3 0f 1f 00 66
66
> [ 2494.959269] RSP: 0018:ffffb949cca7fac0 EFLAGS: 00010246
> [ 2494.964511] RAX: 0000000000000000 RBX: 00000000000faaf5 RCX:
> 0000000000000000
> [ 2494.971664] RDX: 0000000000000001 RSI: 0000000000000009 RDI:
> 00000000000fab7d
> [ 2494.978815] RBP: 00000000000fab7d R08: ffff99c8f12737b8 R09:
> 000000000002c340
> [ 2494.985968] R10: 000005dc16e333d8 R11: 0000000000000000 R12:
> ffff99c903066bd0
> [ 2494.993119] R13: ffff99c9014c7bc0 R14: 000000000000ffff R15:
> ffff99c9014c70c8
> [ 2495.000271] FS:? 0000000000000000(0000) GS:ffff99ba03bc0000(0000)
> knlGS:0000000000000000
> [ 2495.008382] CS:? 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 2495.014141] CR2: 00000000000fab7d CR3: 0000000f38df2002 CR4:
> 00000000000206e0
> [ 2495.021291] Call Trace:
> [ 2495.023753]? igrab+0x19/0x50
> [ 2495.026693]? ocfs2_get_system_file_inode+0x65/0x2c0 [ocfs2]
> [ 2495.032301]? ? ocfs2_read_blocks_sync+0x159/0x330 [ocfs2]
> [ 2495.037741]? ocfs2_test_inode_bit+0xe8/0x900 [ocfs2]
> [ 2495.042742]? ? ocfs2_free_dir_lookup_result+0x24/0x50 [ocfs2]
> [ 2495.048525]? ocfs2_get_parent+0xa3/0x300 [ocfs2]
> [ 2495.053163]? reconnect_path+0xa1/0x2c0
> [ 2495.056943]? ? nfsd_proc_setattr+0x1b0/0x1b0 [nfsd]
> [ 2495.061838]? exportfs_decode_fh+0x111/0x2d0
> [ 2495.066042]? ? exp_find_key+0xcd/0x160 [nfsd]
> [ 2495.070417]? ? __kmalloc+0x180/0x270
> [ 2495.074007]? ? security_prepare_creds+0x6f/0xa0
> [ 2495.078551]? ? tomoyo_write_self+0x1b0/0x1b0
> [ 2495.082834]? ? security_prepare_creds+0x49/0xa0
> [ 2495.087387]? fh_verify+0x3e5/0x5f0 [nfsd]
> [ 2495.092125]? nfsd3_proc_getattr+0x6b/0x100 [nfsd]
> [ 2495.097517]? nfsd_dispatch+0x9e/0x210 [nfsd]
> [ 2495.102489]? svc_process_common+0x386/0x6f0 [sunrpc]
> [ 2495.108134]? ? svc_sock_secure_port+0x12/0x30 [sunrpc]
> [ 2495.113946]? ? svc_recv+0x2ff/0x9c0 [sunrpc]
> [ 2495.118883]? ? nfsd_svc+0x2c0/0x2c0 [nfsd]
> [ 2495.123651]? ? nfsd_destroy+0x50/0x50 [nfsd]
> [ 2495.128604]? svc_process+0xd1/0x110 [sunrpc]
> [ 2495.133563]? nfsd+0xe3/0x140 [nfsd]
> [ 2495.137733]? kthread+0x112/0x130
> [ 2495.141643]? ? kthread_park+0x80/0x80
> [ 2495.145993]? ret_from_fork+0x35/0x40
> [ 2495.150262] Modules linked in: ocfs2 quota_tree nfsd nfs_acl lockd
> grace sctp libcrc32c nft_counter xt_tcpudp nft_compat nf_tables
> nfnetlink binfmt_misc ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm
> ocfs2_nodemanager ocfs2_stackglue cpufreq_userspace cpufreq_powersave
> cpufreq_conservative configfs bonding dm_round_robin dm_multipath
> scsi_dh_rdac scsi_dh_emc scsi_dh_alua intel_powerclamp coretemp
> kvm_intel kvm ipmi_ssif irqbypass cdc_ether usbnet intel_cstate mii
> ib_mthca intel_uncore evdev iTCO_wdt joydev ipmi_si tpm_tis sg
> tpm_tis_core tpm ipmi_devintf iTCO_vendor_support ib_uverbs pcspkr
> watchdog ioatdma rng_core i7core_edac dca i5500_temp ipmi_msghandler
> ib_umad ib_ipoib auth_rpcgss ib_cm sunrpc ib_core ip_tables x_tables
> autofs4 algif_skcipher af_alg ext4 crc16 mbcache jbd2 crc32c_generic
> dm_crypt dm_mod sr_mod cdrom sd_mod hid_generic usbhid hid
> crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel mgag200
> drm_vram_helper drm_ttm_helper i2c_algo_bit ttm qla2xxx drm_kms_helper
> [ 2495.150298]? ata_generic drm ata_piix aesni_intel libata nvme_fc
> uhci_hcd crypto_simd nvme_fabrics ehci_pci ehci_hcd megaraid_sas cryptd
> nvme_core glue_helper usbcore scsi_transport_fc lpc_ich mfd_core e1000e
> scsi_mod i2c_i801 usb_common bnx2 ptp pps_core button
> [ 2495.269073] CR2: 00000000000fab7d
> [ 2495.273205] ---[ end trace 44f000505f987296 ]---
> [ 2495.278647] RIP: 0010:_raw_spin_lock+0xc/0x20
> [ 2495.283825] Code: 66 90 66 66 90 31 c0 ba ff 00 00 00 f0 0f b1 17 75
> 05 48 89 d8 5b c3 e8 12 5f 91 ff eb f4 66 66 66 66 90 31 c0 ba 01 00 00
> 00 <f0> 0f b1 17 75 01 c3 89 c6 e8 76 46 91 ff 66 90 c3 0f 1f 00 66
66
> [ 2495.304287] RSP: 0018:ffffb949cca7fac0 EFLAGS: 00010246
> [ 2495.310368] RAX: 0000000000000000 RBX: 00000000000faaf5 RCX:
> 0000000000000000
> [ 2495.318358] RDX: 0000000000000001 RSI: 0000000000000009 RDI:
> 00000000000fab7d
> [ 2495.326339] RBP: 00000000000fab7d R08: ffff99c8f12737b8 R09:
> 000000000002c340
> [ 2495.334314] R10: 000005dc16e333d8 R11: 0000000000000000 R12:
> ffff99c903066bd0
> [ 2495.342297] R13: ffff99c9014c7bc0 R14: 000000000000ffff R15:
> ffff99c9014c70c8
> [ 2495.350279] FS:? 0000000000000000(0000) GS:ffff99ba03bc0000(0000)
> knlGS:0000000000000000
> [ 2495.359224] CS:? 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 2495.365825] CR2: 00000000000fab7d CR3: 0000000f38df2002 CR4:
> 00000000000206e0
> 
> - -----8< cut here 8<-----
> 
> 
> I also made some record of NFS traffic with tcpdump when the crashs are
> occuring, but I didn't had time to analyse it. I can provide them if it
> helps (but I might have to clean them up first to avoid some data leaks :)
> 
> After this happening a few times, I had one case where one of the few
> NFSv4 client was involved, and I took as aworkaround to migrate the
> NFSv4 share to XFS since I though it could be NFSv4 related. Now that I
> looked at the kernel dumps, I think it's also NFSv3 related.
> 
> Is it a kernel bug or am I missing something ?
Probably a ocfs2 internal error handling bug. :-(
> 
> For now, I will switch back to XFS with a failover setup of my cluster,
> cause I cannot afford this kind of crashes of a central file server.
> 
> 
Interesting, do you mean you can use XFS as a NFSv4 backend filesystem 
which is formatted on a shared SCSI LUN with more than one host attached 
without filesystem corruption?


Thanks,
Changwei

> 
> Cheers
> 
> 
> Pierre
> 
> 
> PS : I'll try to update with what I found in the PCAP files in some
days
> 
> 
> 
> -----BEGIN PGP SIGNATURE-----
> 
> iQIzBAEBCgAdFiEEjXhlMuC3bFbPA3S/qC81yulnAywFAl7DuYUACgkQqC81yuln
> AyyWHg/+NUMgiysN+QesGedzDjfKiUSTKqPvbg3pUklq3bXfidLdIcM6HGZt31Gy
> /k33m83+675c3vQeez8uvSVAh7uNzQW2m7BGrzEv/ZkpwIZxqIwiV+gi/zc7lnGP
> vqkjAqVjAapk254W3aTonZKpExKCbp6h1g1eVWKk43Mu7/UaLkQcjDPq9AyheSt+
> PAPpiQeQ9ox1+OVvEnpawJmcV9QUluHPwsar/yYa87ET2LIlBswtjOHr+Vw27kJU
> qwak16rTDWqVQo5PYIi5N1jdCAEAPOV6EalNNwIQ9/u+lCdbVHYRV7Ze5nE58DDz
> zP7Ai9Vaj/oBr/znjbaoE2VR0iWRVqbztBxts0hNv1IxAqJivZnaaJx6bXp51NeV
> kXwTz0tHqCoy8RVlUXEGP8FBuaWgSLZyfN5IOk1KoocsLUuZh/BVxh+snDj4T52k
> u/SRBFV4lW4RVYWHBItik8oqfMwYaJuwp2hdmU1hHsm1UnHQTQjum1L9ZfLDI7XU
> NxHMGedneJePXdWH0eTqvHAPJ1U59gEL/cMPnOJF57q+eYoCs0svnRxCwQzviBok
> Zak2uzIIXr3jc9bY3qiQlZivik5b+l6LK7ccoS7eEIqxOkKivOjwijuXc0EHOUR2
> c0QjFUDwJ1gR8rkBE/rOiKLXhMGnv2lcIZWiu7JlxMq+ms7f/1k> =6VH2
> -----END PGP SIGNATURE-----
> 
> 
> _______________________________________________
> Ocfs2-devel mailing list
> Ocfs2-devel at oss.oracle.com
> https://oss.oracle.com/mailman/listinfo/ocfs2-devel
>

Pierre,

A similar if not identical bug was reported against Oracle Linux. The 
fix is still being developed. I will update that bug to reference this 
thread, and ask the developer to submit the fix upstream and/or to the 
ocfs2-devel list once it's ready.

Thanks,
Herbert.

On 5/19/20 3:48 AM, Pierre Dinh-van wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>
> Hi,
>
> I'm experiencing quite reproducable crashs on my OCFS2 cluster the last
> weeks, and I think I found out part of the Problem. But before I make a
> bug report (I'm not sure where), I though I could post my research here
> in case someone understands it better than I.
>
> When it happens, the kernel is giving me a 'general protection fault:
> 0000 [#1] SMP PTI' or ' BUG: unable to handle page fault for
address:'
> message, local access to the OCFS2 volumes are all hanging forever, and
> the load is getting so high that only a hard reset "solves" the
problem.
> It happened the same way on both nodes (always on the active node
> serving NFS). It happens when both nodes are up, or when only one is
> active.
>
> First, a few words about my setup :
>
> - - storage is Fiberchannel LUNs served by a EMC VNX5200 (4x 5TB).
>
> - - LUNs are encrypted with LUKS
>
> - - LUKS volumes are formated with OCFS2
>
> - - heartbeat=local
>
> - - The OCFS2 cluster is communicating on a dedictated IP over InfinyBand
> direct link (actually a fail-over bonding of the 2 ib interfaces).
>
> - - nodes are 2 debian buster hardware servers (24 cores, 128GB RAM).
>
> - - I use corosync for cluster management, but I don't think it's
related
>
> - - The cluster serves 4 NFS exports (3 are NFSv3, and one is NFSv4 with
> kerberos) and samba shares
>
> - - The samba shares are served in parallel by the 2 nodes managed by ctdb
>
> - - The NFS exports are served by one of the node having the dedicated IP
> for this export.
>
> - - When corosync is moving the IP of a NFS export to the other node, it
> restarts nfsd to issue a grace time of 10s on the new active node.
>
> - - The bug appeared with default debian buster kernel 4.19.0-9-amd64 and
> with the backported kernel 5.5.0-0.bpo.2-amd64 as well
>
>
> My cluster went live a few weeks ago, and it first worked with both
> nodes active. During some maintenance work where the active NFS node had
> to switch, I had a a bug, that I was able to reproduce against my will
> at least 20 times in the last nights :
>
> 1) around 100 NFS clients are connected with the exports and happy
> working with it
>
> 2) The IPs of the NFS shares are switching from nodeA to nodeB
>
> 3) nodeB's kernel says "general protection fault: 0000 [#1] SMP
PTI"
>
> 4) nodeB's load is getting high
>
> 5) access to the OCFS2 is hanging (clean reboot never end, system cannot
> unmount OCFS volumes)
>
> 6) I power cycle nodeB
>
> 7) I start nodeA again
>
> 8) nodeA give the same symptoms, as soon as some NFS clients do some
> special requests
>
>
> Most of the time, when the IP jumps to the other node, I get some of
> this messages :
>
>
> [? 501.707566] (nfsd,2133,14):ocfs2_test_inode_bit:2841 ERROR: unable to
> get alloc inode in slot 65535
> [? 501.716660] (nfsd,2133,14):ocfs2_test_inode_bit:2867 ERROR: status = -22
> [? 501.726585] (nfsd,2133,6):ocfs2_test_inode_bit:2841 ERROR: unable to
> get alloc inode in slot 65535
> [? 501.735579] (nfsd,2133,6):ocfs2_test_inode_bit:2867 ERROR: status = -22
>
> But it also happens when the node is not crashing, so I think it's
> another Problem.
>
> Last night, I saved a few of the kernel output while the servers where
> crashing one after the other like in a ping-pong game :
>
>
> - -----8< on nodeB? 8<-----
>
> [? 502.070431] nfsd: nfsv4 idmapping failing: has idmapd not been started?
> [? 502.475747] general protection fault: 0000 [#1] SMP PTI
> [? 502.481027] CPU: 6 PID: 2104 Comm: nfsd Not tainted
> 5.5.0-0.bpo.2-amd64 #1 Debian 5.5.17-1~bpo10+1
> [? 502.490016] Hardware name: IBM System x3650 M3
> - -[7945UHV]-/94Y7614???? , BIOS -[D6E150CUS-1.11]- 02/08/2011
> [? 502.499821] RIP: 0010:_raw_spin_lock+0xc/0x20
> [? 502.504206] Code: 66 90 66 66 90 31 c0 ba ff 00 00 00 f0 0f b1 17 75
> 05 48 89 d8 5b c3 e8 12 5f 91 ff eb f4 66 66 66 66 90 31 c0 ba 01 00 00
> 00 <f0> 0f b1 17 75 01 c3 89 c6 e8 76 46 91 ff 66 90 c3 0f 1f 00 66
66
> [? 502.523018] RSP: 0018:ffffa5af4838bac0 EFLAGS: 00010246
> [? 502.528277] RAX: 0000000000000000 RBX: 0fdbbeded053c1d6 RCX:
> 0000000000000000
> [? 502.535433] RDX: 0000000000000001 RSI: 0000000000000009 RDI:
> 0fdbbeded053c25e
> [? 502.542588] RBP: 0fdbbeded053c25e R08: ffff9672ffa73068 R09:
> 000000000002c340
> [? 502.549743] R10: 00000184d03a2115 R11: 0000000000000000 R12:
> ffff9672eff43fd0
> [? 502.556899] R13: ffff9672fff66bc0 R14: 000000000000ffff R15:
> ffff9672fff660c8
> [? 502.564055] FS:? 0000000000000000(0000) GS:ffff968203a80000(0000)
> knlGS:0000000000000000
> [? 502.572166] CS:? 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [? 502.577928] CR2: 00007f0a4bb5df50 CR3: 0000001e4169a004 CR4:
> 00000000000206e0
> [? 502.585082] Call Trace:
> [? 502.587546]? igrab+0x19/0x50
> [? 502.590484]? ocfs2_get_system_file_inode+0x65/0x2c0 [ocfs2]
> [? 502.596094]? ? ocfs2_read_blocks_sync+0x159/0x330 [ocfs2]
> [? 502.601534]? ocfs2_test_inode_bit+0xe8/0x900 [ocfs2]
> [? 502.606536]? ? ocfs2_free_dir_lookup_result+0x24/0x50 [ocfs2]
> [? 502.612321]? ocfs2_get_parent+0xa3/0x300 [ocfs2]
> [? 502.616960]? reconnect_path+0xa1/0x2c0
> [? 502.620742]? ? nfsd_proc_setattr+0x1b0/0x1b0 [nfsd]
> [? 502.625636]? exportfs_decode_fh+0x111/0x2d0
> [? 502.629844]? ? exp_find_key+0xcd/0x160 [nfsd]
> [? 502.634218]? ? __kmalloc+0x180/0x270
> [? 502.637810]? ? security_prepare_creds+0x6f/0xa0
> [? 502.642356]? ? tomoyo_write_self+0x1b0/0x1b0
> [? 502.646642]? ? security_prepare_creds+0x49/0xa0
> [? 502.651995]? fh_verify+0x3e5/0x5f0 [nfsd]
> [? 502.656737]? nfsd3_proc_getattr+0x6b/0x100 [nfsd]
> [? 502.662152]? nfsd_dispatch+0x9e/0x210 [nfsd]
> [? 502.667136]? svc_process_common+0x386/0x6f0 [sunrpc]
> [? 502.672791]? ? svc_sock_secure_port+0x12/0x30 [sunrpc]
> [? 502.678628]? ? svc_recv+0x2ff/0x9c0 [sunrpc]
> [? 502.683597]? ? nfsd_svc+0x2c0/0x2c0 [nfsd]
> [? 502.688395]? ? nfsd_destroy+0x50/0x50 [nfsd]
> [? 502.693377]? svc_process+0xd1/0x110 [sunrpc]
> [? 502.698361]? nfsd+0xe3/0x140 [nfsd]
> [? 502.702561]? kthread+0x112/0x130
> [? 502.706504]? ? kthread_park+0x80/0x80
> [? 502.710886]? ret_from_fork+0x35/0x40
> [? 502.715181] Modules linked in: rpcsec_gss_krb5 nfsd nfs_acl lockd
> grace ocfs2 quota_tree sctp libcrc32c nft_counter ipt_REJECT
> nf_reject_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink binfmt_misc
> ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager
> cpufreq_userspace ocfs2_stackglue cpufreq_powersave cpufreq_conservative
> configfs bonding dm_round_robin dm_multipath scsi_dh_rdac scsi_dh_emc
> scsi_dh_alua intel_powerclamp coretemp kvm_intel ipmi_ssif kvm cdc_ether
> irqbypass usbnet mii tpm_tis intel_cstate tpm_tis_core intel_uncore
> joydev tpm ipmi_si ioatdma sg iTCO_wdt ib_mthca iTCO_vendor_support
> watchdog dca pcspkr ipmi_devintf rng_core ib_uverbs ipmi_msghandler
> i5500_temp i7core_edac evdev ib_umad ib_ipoib ib_cm ib_core auth_rpcgss
> sunrpc ip_tables x_tables autofs4 algif_skcipher af_alg ext4 crc16
> mbcache jbd2 crc32c_generic dm_crypt dm_mod sd_mod hid_generic usbhid
> hid sr_mod cdrom crct10dif_pclmul crc32_pclmul crc32c_intel
> ghash_clmulni_intel mgag200 drm_vram_helper drm_ttm_helper
> [? 502.715218]? i2c_algo_bit ttm qla2xxx drm_kms_helper drm ata_generic
> ata_piix aesni_intel libata nvme_fc ehci_pci uhci_hcd crypto_simd
> ehci_hcd nvme_fabrics cryptd glue_helper lpc_ich nvme_core megaraid_sas
> mfd_core usbcore scsi_transport_fc e1000e scsi_mod i2c_i801 usb_common
> ptp pps_core bnx2 button
> [? 502.837917] ---[ end trace 6e1a8e507ac30d8d ]---
> [? 502.843379] RIP: 0010:_raw_spin_lock+0xc/0x20
> [? 502.848584] Code: 66 90 66 66 90 31 c0 ba ff 00 00 00 f0 0f b1 17 75
> 05 48 89 d8 5b c3 e8 12 5f 91 ff eb f4 66 66 66 66 90 31 c0 ba 01 00 00
> 00 <f0> 0f b1 17 75 01 c3 89 c6 e8 76 46 91 ff 66 90 c3 0f 1f 00 66
66
> [? 502.869295] RSP: 0018:ffffa5af4838bac0 EFLAGS: 00010246
> [? 502.875404] RAX: 0000000000000000 RBX: 0fdbbeded053c1d6 RCX:
> 0000000000000000
> [? 502.883423] RDX: 0000000000000001 RSI: 0000000000000009 RDI:
> 0fdbbeded053c25e
> [? 502.891436] RBP: 0fdbbeded053c25e R08: ffff9672ffa73068 R09:
> 000000000002c340
> [? 502.899446] R10: 00000184d03a2115 R11: 0000000000000000 R12:
> ffff9672eff43fd0
> [? 502.907455] R13: ffff9672fff66bc0 R14: 000000000000ffff R15:
> ffff9672fff660c8
> [? 502.915473] FS:? 0000000000000000(0000) GS:ffff968203a80000(0000)
> knlGS:0000000000000000
> [? 502.924496] CS:? 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [? 502.931147] CR2: 00007f0a4bb5df50 CR3: 0000001e4169a004 CR4:
> 00000000000206e0
>
> - -----8< cut here 8<-----
>
>
> shortly after when trying to start nodeA while nodeB was down :
>
> - -----8< nodeA 8<-----
>
> [? 728.927750] general protection fault: 0000 [#1] SMP PTI
> [? 728.934647] CPU: 8 PID: 12156 Comm: nfsd Not tainted
> 5.5.0-0.bpo.2-amd64 #1 Debian 5.5.17-1~bpo10+1
> [? 728.944800] Hardware name: IBM System x3650 M3 -[7945H2G]-/69Y4438,
> BIOS -[D6E158AUS-1.16]- 11/26/2012
> [? 728.955176] RIP: 0010:_raw_spin_lock+0xc/0x20
> [? 728.960595] Code: 66 90 66 66 90 31 c0 ba ff 00 00 00 f0 0f b1 17 75
> 05 48 89 d8 5b c3 e8 12 5f 91 ff eb f4 66 66 66 66 90 31 c0 ba 01 00 00
> 00 <f0> 0f b1 17 75 01 c3 89 c6 e8 76 46 91 ff 66 90 c3 0f 1f 00 66
66
> [? 728.981559] RSP: 0018:ffffa24d09187ac0 EFLAGS: 00010246
> [? 728.987885] RAX: 0000000000000000 RBX: 625f656369766564 RCX:
> 0000000000000000
> [? 728.996135] RDX: 0000000000000001 RSI: 0000000000000009 RDI:
> 625f6563697665ec
> [? 729.004384] RBP: 625f6563697665ec R08: ffff91f1272e3680 R09:
> 000000000002c340
> [? 729.012628] R10: 000002114c00d22b R11: 0000000000000000 R12:
> ffff91e13858bbd0
> [? 729.020877] R13: ffff91f13c03bbc0 R14: 000000000000ffff R15:
> ffff91f13c03b0c8
> [? 729.029108] FS:? 0000000000000000(0000) GS:ffff91f13fa80000(0000)
> knlGS:0000000000000000
> [? 729.038308] CS:? 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [? 729.045161] CR2: 000055fe95d1d9a8 CR3: 0000001c8420a003 CR4:
> 00000000000206e0
> [? 729.053415] Call Trace:
> [? 729.056978]? igrab+0x19/0x50
> [? 729.061023]? ocfs2_get_system_file_inode+0x65/0x2c0 [ocfs2]
> [? 729.067736]? ? ocfs2_read_blocks_sync+0x159/0x330 [ocfs2]
> [? 729.074284]? ocfs2_test_inode_bit+0xe8/0x900 [ocfs2]
> [? 729.080386]? ? ocfs2_free_dir_lookup_result+0x24/0x50 [ocfs2]
> [? 729.087268]? ocfs2_get_parent+0xa3/0x300 [ocfs2]
> [? 729.092995]? reconnect_path+0xa1/0x2c0
> [? 729.097863]? ? nfsd_proc_setattr+0x1b0/0x1b0 [nfsd]
> [? 729.103840]? exportfs_decode_fh+0x111/0x2d0
> [? 729.109133]? ? exp_find_key+0xcd/0x160 [nfsd]
> [? 729.114574]? ? dbs_update_util_handler+0x16/0x90
> [? 729.120273]? ? __kmalloc+0x180/0x270
> [? 729.124920]? ? security_prepare_creds+0x6f/0xa0
> [? 729.130521]? ? tomoyo_write_self+0x1b0/0x1b0
> [? 729.135858]? ? security_prepare_creds+0x49/0xa0
> [? 729.141464]? fh_verify+0x3e5/0x5f0 [nfsd]
> [? 729.146536]? nfsd3_proc_getattr+0x6b/0x100 [nfsd]
> [? 729.152297]? nfsd_dispatch+0x9e/0x210 [nfsd]
> [? 729.157647]? svc_process_common+0x386/0x6f0 [sunrpc]
> [? 729.163675]? ? svc_sock_secure_port+0x12/0x30 [sunrpc]
> [? 729.169873]? ? svc_recv+0x2ff/0x9c0 [sunrpc]
> [? 729.175190]? ? nfsd_svc+0x2c0/0x2c0 [nfsd]
> [? 729.180323]? ? nfsd_destroy+0x50/0x50 [nfsd]
> [? 729.185631]? svc_process+0xd1/0x110 [sunrpc]
> [? 729.190925]? nfsd+0xe3/0x140 [nfsd]
> [? 729.195421]? kthread+0x112/0x130
> [? 729.199636]? ? kthread_park+0x80/0x80
> [? 729.204279]? ret_from_fork+0x35/0x40
> [? 729.208818] Modules linked in: ocfs2 quota_tree dm_crypt crypto_simd
> cryptd glue_helper algif_skcipher af_alg nfsd nfs_acl lockd grace sctp
> libcrc32c nft_counter xt_tcpudp nft_compat nf_tables nfnetlink
> binfmt_misc ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager
> ocfs2_stackglue configfs bonding dm_round_robin dm_multipath
> scsi_dh_rdac scsi_dh_emc scsi_dh_alua intel_powerclamp ipmi_ssif
> coretemp kvm_intel kvm irqbypass cdc_ether intel_cstate usbnet ib_mthca
> mii joydev intel_uncore ib_uverbs iTCO_wdt sg ioatdma
> iTCO_vendor_support i7core_edac watchdog i5500_temp pcspkr ipmi_si
> tpm_tis tpm_tis_core tpm ipmi_devintf ipmi_msghandler rng_core evdev
> acpi_cpufreq ib_umad auth_rpcgss ib_ipoib ib_cm sunrpc ib_core ip_tables
> x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic dm_mod
> hid_generic sd_mod usbhid hid sr_mod cdrom uas usb_storage mgag200
> drm_vram_helper drm_ttm_helper qla2xxx ata_generic i2c_algo_bit ttm
> nvme_fc ixgbe nvme_fabrics ata_piix xfrm_algo ehci_pci uhci_hcd
> [? 729.208868]? drm_kms_helper nvme_core ehci_hcd dca libata
> megaraid_sas scsi_transport_fc libphy ptp drm usbcore scsi_mod
> crc32c_intel lpc_ich i2c_i801 pps_core mfd_core usb_common bnx2 mdio button
> [? 729.323900] ---[ end trace 212cefe75207e17f ]---
> [? 729.329644] RIP: 0010:_raw_spin_lock+0xc/0x20
> [? 729.335121] Code: 66 90 66 66 90 31 c0 ba ff 00 00 00 f0 0f b1 17 75
> 05 48 89 d8 5b c3 e8 12 5f 91 ff eb f4 66 66 66 66 90 31 c0 ba 01 00 00
> 00 <f0> 0f b1 17 75 01 c3 89 c6 e8 76 46 91 ff 66 90 c3 0f 1f 00 66
66
> [? 729.356192] RSP: 0018:ffffa24d09187ac0 EFLAGS: 00010246
> [? 729.362584] RAX: 0000000000000000 RBX: 625f656369766564 RCX:
> 0000000000000000
> [? 729.370889] RDX: 0000000000000001 RSI: 0000000000000009 RDI:
> 625f6563697665ec
> [? 729.379184] RBP: 625f6563697665ec R08: ffff91f1272e3680 R09:
> 000000000002c340
> [? 729.387475] R10: 000002114c00d22b R11: 0000000000000000 R12:
> ffff91e13858bbd0
> [? 729.395767] R13: ffff91f13c03bbc0 R14: 000000000000ffff R15:
> ffff91f13c03b0c8
> [? 729.404066] FS:? 0000000000000000(0000) GS:ffff91f13fa80000(0000)
> knlGS:0000000000000000
> [? 729.413336] CS:? 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [? 729.420262] CR2: 000055fe95d1d9a8 CR3: 0000001c8420a003 CR4:
> 00000000000206e0
>
> - -----8< cut here 8<-----
>
> Since the backtrace is speaking about nfsd, I tried to isolate the
> Problem by droping all NFS packet entering to the active node before
> starting the services :
>
> iptables -I INPUT -p tcp --dport 2049 -j DROP
>
> Everything fine, access over samba were doing fine.
>
> I started to allow NFS for part of my network with iptables, until the
> crash occurs.
>
> On this one, the crash occured after allowing only a NFSv3 client.
>
> - -----8< nodeB 8<-----
>
> [ 2494.893697] BUG: unable to handle page fault for address:
> 00000000000fab7d
> [ 2494.900626] #PF: supervisor write access in kernel mode
> [ 2494.905873] #PF: error_code(0x0002) - not-present page
> [ 2494.911042] PGD 0 P4D 0
> [ 2494.913596] Oops: 0002 [#1] SMP PTI
> [ 2494.917106] CPU: 11 PID: 21342 Comm: nfsd Not tainted
> 5.5.0-0.bpo.2-amd64 #1 Debian 5.5.17-1~bpo10+1
> [ 2494.926285] Hardware name: IBM System x3650 M3
> - -[7945UHV]-/94Y7614???? , BIOS -[D6E150CUS-1.11]- 02/08/2011
> [ 2494.936077] RIP: 0010:_raw_spin_lock+0xc/0x20
> [ 2494.940461] Code: 66 90 66 66 90 31 c0 ba ff 00 00 00 f0 0f b1 17 75
> 05 48 89 d8 5b c3 e8 12 5f 91 ff eb f4 66 66 66 66 90 31 c0 ba 01 00 00
> 00 <f0> 0f b1 17 75 01 c3 89 c6 e8 76 46 91 ff 66 90 c3 0f 1f 00 66
66
> [ 2494.959269] RSP: 0018:ffffb949cca7fac0 EFLAGS: 00010246
> [ 2494.964511] RAX: 0000000000000000 RBX: 00000000000faaf5 RCX:
> 0000000000000000
> [ 2494.971664] RDX: 0000000000000001 RSI: 0000000000000009 RDI:
> 00000000000fab7d
> [ 2494.978815] RBP: 00000000000fab7d R08: ffff99c8f12737b8 R09:
> 000000000002c340
> [ 2494.985968] R10: 000005dc16e333d8 R11: 0000000000000000 R12:
> ffff99c903066bd0
> [ 2494.993119] R13: ffff99c9014c7bc0 R14: 000000000000ffff R15:
> ffff99c9014c70c8
> [ 2495.000271] FS:? 0000000000000000(0000) GS:ffff99ba03bc0000(0000)
> knlGS:0000000000000000
> [ 2495.008382] CS:? 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 2495.014141] CR2: 00000000000fab7d CR3: 0000000f38df2002 CR4:
> 00000000000206e0
> [ 2495.021291] Call Trace:
> [ 2495.023753]? igrab+0x19/0x50
> [ 2495.026693]? ocfs2_get_system_file_inode+0x65/0x2c0 [ocfs2]
> [ 2495.032301]? ? ocfs2_read_blocks_sync+0x159/0x330 [ocfs2]
> [ 2495.037741]? ocfs2_test_inode_bit+0xe8/0x900 [ocfs2]
> [ 2495.042742]? ? ocfs2_free_dir_lookup_result+0x24/0x50 [ocfs2]
> [ 2495.048525]? ocfs2_get_parent+0xa3/0x300 [ocfs2]
> [ 2495.053163]? reconnect_path+0xa1/0x2c0
> [ 2495.056943]? ? nfsd_proc_setattr+0x1b0/0x1b0 [nfsd]
> [ 2495.061838]? exportfs_decode_fh+0x111/0x2d0
> [ 2495.066042]? ? exp_find_key+0xcd/0x160 [nfsd]
> [ 2495.070417]? ? __kmalloc+0x180/0x270
> [ 2495.074007]? ? security_prepare_creds+0x6f/0xa0
> [ 2495.078551]? ? tomoyo_write_self+0x1b0/0x1b0
> [ 2495.082834]? ? security_prepare_creds+0x49/0xa0
> [ 2495.087387]? fh_verify+0x3e5/0x5f0 [nfsd]
> [ 2495.092125]? nfsd3_proc_getattr+0x6b/0x100 [nfsd]
> [ 2495.097517]? nfsd_dispatch+0x9e/0x210 [nfsd]
> [ 2495.102489]? svc_process_common+0x386/0x6f0 [sunrpc]
> [ 2495.108134]? ? svc_sock_secure_port+0x12/0x30 [sunrpc]
> [ 2495.113946]? ? svc_recv+0x2ff/0x9c0 [sunrpc]
> [ 2495.118883]? ? nfsd_svc+0x2c0/0x2c0 [nfsd]
> [ 2495.123651]? ? nfsd_destroy+0x50/0x50 [nfsd]
> [ 2495.128604]? svc_process+0xd1/0x110 [sunrpc]
> [ 2495.133563]? nfsd+0xe3/0x140 [nfsd]
> [ 2495.137733]? kthread+0x112/0x130
> [ 2495.141643]? ? kthread_park+0x80/0x80
> [ 2495.145993]? ret_from_fork+0x35/0x40
> [ 2495.150262] Modules linked in: ocfs2 quota_tree nfsd nfs_acl lockd
> grace sctp libcrc32c nft_counter xt_tcpudp nft_compat nf_tables
> nfnetlink binfmt_misc ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm
> ocfs2_nodemanager ocfs2_stackglue cpufreq_userspace cpufreq_powersave
> cpufreq_conservative configfs bonding dm_round_robin dm_multipath
> scsi_dh_rdac scsi_dh_emc scsi_dh_alua intel_powerclamp coretemp
> kvm_intel kvm ipmi_ssif irqbypass cdc_ether usbnet intel_cstate mii
> ib_mthca intel_uncore evdev iTCO_wdt joydev ipmi_si tpm_tis sg
> tpm_tis_core tpm ipmi_devintf iTCO_vendor_support ib_uverbs pcspkr
> watchdog ioatdma rng_core i7core_edac dca i5500_temp ipmi_msghandler
> ib_umad ib_ipoib auth_rpcgss ib_cm sunrpc ib_core ip_tables x_tables
> autofs4 algif_skcipher af_alg ext4 crc16 mbcache jbd2 crc32c_generic
> dm_crypt dm_mod sr_mod cdrom sd_mod hid_generic usbhid hid
> crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel mgag200
> drm_vram_helper drm_ttm_helper i2c_algo_bit ttm qla2xxx drm_kms_helper
> [ 2495.150298]? ata_generic drm ata_piix aesni_intel libata nvme_fc
> uhci_hcd crypto_simd nvme_fabrics ehci_pci ehci_hcd megaraid_sas cryptd
> nvme_core glue_helper usbcore scsi_transport_fc lpc_ich mfd_core e1000e
> scsi_mod i2c_i801 usb_common bnx2 ptp pps_core button
> [ 2495.269073] CR2: 00000000000fab7d
> [ 2495.273205] ---[ end trace 44f000505f987296 ]---
> [ 2495.278647] RIP: 0010:_raw_spin_lock+0xc/0x20
> [ 2495.283825] Code: 66 90 66 66 90 31 c0 ba ff 00 00 00 f0 0f b1 17 75
> 05 48 89 d8 5b c3 e8 12 5f 91 ff eb f4 66 66 66 66 90 31 c0 ba 01 00 00
> 00 <f0> 0f b1 17 75 01 c3 89 c6 e8 76 46 91 ff 66 90 c3 0f 1f 00 66
66
> [ 2495.304287] RSP: 0018:ffffb949cca7fac0 EFLAGS: 00010246
> [ 2495.310368] RAX: 0000000000000000 RBX: 00000000000faaf5 RCX:
> 0000000000000000
> [ 2495.318358] RDX: 0000000000000001 RSI: 0000000000000009 RDI:
> 00000000000fab7d
> [ 2495.326339] RBP: 00000000000fab7d R08: ffff99c8f12737b8 R09:
> 000000000002c340
> [ 2495.334314] R10: 000005dc16e333d8 R11: 0000000000000000 R12:
> ffff99c903066bd0
> [ 2495.342297] R13: ffff99c9014c7bc0 R14: 000000000000ffff R15:
> ffff99c9014c70c8
> [ 2495.350279] FS:? 0000000000000000(0000) GS:ffff99ba03bc0000(0000)
> knlGS:0000000000000000
> [ 2495.359224] CS:? 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 2495.365825] CR2: 00000000000fab7d CR3: 0000000f38df2002 CR4:
> 00000000000206e0
>
> - -----8< cut here 8<-----
>
>
> I also made some record of NFS traffic with tcpdump when the crashs are
> occuring, but I didn't had time to analyse it. I can provide them if it
> helps (but I might have to clean them up first to avoid some data leaks :)
>
> After this happening a few times, I had one case where one of the few
> NFSv4 client was involved, and I took as aworkaround to migrate the
> NFSv4 share to XFS since I though it could be NFSv4 related. Now that I
> looked at the kernel dumps, I think it's also NFSv3 related.
>
> Is it a kernel bug or am I missing something ?
>
> For now, I will switch back to XFS with a failover setup of my cluster,
> cause I cannot afford this kind of crashes of a central file server.
>
>
>
> Cheers
>
>
> Pierre
>
>
> PS : I'll try to update with what I found in the PCAP files in some
days
>
>
>
> -----BEGIN PGP SIGNATURE-----
>
> iQIzBAEBCgAdFiEEjXhlMuC3bFbPA3S/qC81yulnAywFAl7DuYUACgkQqC81yuln
> AyyWHg/+NUMgiysN+QesGedzDjfKiUSTKqPvbg3pUklq3bXfidLdIcM6HGZt31Gy
> /k33m83+675c3vQeez8uvSVAh7uNzQW2m7BGrzEv/ZkpwIZxqIwiV+gi/zc7lnGP
> vqkjAqVjAapk254W3aTonZKpExKCbp6h1g1eVWKk43Mu7/UaLkQcjDPq9AyheSt+
> PAPpiQeQ9ox1+OVvEnpawJmcV9QUluHPwsar/yYa87ET2LIlBswtjOHr+Vw27kJU
> qwak16rTDWqVQo5PYIi5N1jdCAEAPOV6EalNNwIQ9/u+lCdbVHYRV7Ze5nE58DDz
> zP7Ai9Vaj/oBr/znjbaoE2VR0iWRVqbztBxts0hNv1IxAqJivZnaaJx6bXp51NeV
> kXwTz0tHqCoy8RVlUXEGP8FBuaWgSLZyfN5IOk1KoocsLUuZh/BVxh+snDj4T52k
> u/SRBFV4lW4RVYWHBItik8oqfMwYaJuwp2hdmU1hHsm1UnHQTQjum1L9ZfLDI7XU
> NxHMGedneJePXdWH0eTqvHAPJ1U59gEL/cMPnOJF57q+eYoCs0svnRxCwQzviBok
> Zak2uzIIXr3jc9bY3qiQlZivik5b+l6LK7ccoS7eEIqxOkKivOjwijuXc0EHOUR2
> c0QjFUDwJ1gR8rkBE/rOiKLXhMGnv2lcIZWiu7JlxMq+ms7f/1k> =6VH2
> -----END PGP SIGNATURE-----
>
>
> _______________________________________________
> Ocfs2-devel mailing list
> Ocfs2-devel at oss.oracle.com
> https://oss.oracle.com/mailman/listinfo/ocfs2-devel

Hi Herbert,

Is there somewhere a Bugtracker for it ? Or are the bugs @oracle not
public anymore ?

Browsing/searching bugzilla don't show me any bug :
https://bugzilla.oracle.com/bugzilla/describecomponents.cgi

Cheers

Pierre

On 5/21/20 12:01 AM, herbert.van.den.bergh at oracle.com
wrote:
> Pierre,
>
> A similar if not identical bug was reported against Oracle Linux. The
> fix is still being developed. I will update that bug to reference this
> thread, and ask the developer to submit the fix upstream and/or to the
> ocfs2-devel list once it's ready.
>
> Thanks,
> Herbert.
>
> On 5/19/20 3:48 AM, Pierre Dinh-van wrote:
> Hi,
> 
> I'm experiencing quite reproducable crashs on my OCFS2 cluster the last
> weeks, and I think I found out part of the Problem. But before I make a
> bug report (I'm not sure where), I though I could post my research here
> in case someone understands it better than I.
> 
> When it happens, the kernel is giving me a 'general protection fault:
> 0000 [#1] SMP PTI' or ' BUG: unable to handle page fault for
address:'
> message, local access to the OCFS2 volumes are all hanging forever, and
> the load is getting so high that only a hard reset "solves" the
problem.
> It happened the same way on both nodes (always on the active node
> serving NFS). It happens when both nodes are up, or when only one is
> active.
> 
> First, a few words about my setup :
> 
> - storage is Fiberchannel LUNs served by a EMC VNX5200 (4x 5TB).
> 
> - LUNs are encrypted with LUKS
> 
> - LUKS volumes are formated with OCFS2
> 
> - heartbeat=local
> 
> - The OCFS2 cluster is communicating on a dedictated IP over InfinyBand
> direct link (actually a fail-over bonding of the 2 ib interfaces).
> 
> - nodes are 2 debian buster hardware servers (24 cores, 128GB RAM).
> 
> - I use corosync for cluster management, but I don't think it's
related
> 
> - The cluster serves 4 NFS exports (3 are NFSv3, and one is NFSv4 with
> kerberos) and samba shares
> 
> - The samba shares are served in parallel by the 2 nodes managed by
> ctdb
> 
> - The NFS exports are served by one of the node having the dedicated IP
> for this export.
> 
> - When corosync is moving the IP of a NFS export to the other node, it
> restarts nfsd to issue a grace time of 10s on the new active node.
> 
> - The bug appeared with default debian buster kernel 4.19.0-9-amd64 and
> with the backported kernel 5.5.0-0.bpo.2-amd64 as well
> 
> 
> My cluster went live a few weeks ago, and it first worked with both
> nodes active. During some maintenance work where the active NFS node had
> to switch, I had a a bug, that I was able to reproduce against my will
> at least 20 times in the last nights :
> 
> 1) around 100 NFS clients are connected with the exports and happy
> working with it
> 
> 2) The IPs of the NFS shares are switching from nodeA to nodeB
> 
> 3) nodeB's kernel says "general protection fault: 0000 [#1] SMP
PTI"
> 
> 4) nodeB's load is getting high
> 
> 5) access to the OCFS2 is hanging (clean reboot never end, system cannot
> unmount OCFS volumes)
> 
> 6) I power cycle nodeB
> 
> 7) I start nodeA again
> 
> 8) nodeA give the same symptoms, as soon as some NFS clients do some
> special requests
> 
> 
> Most of the time, when the IP jumps to the other node, I get some of
> this messages :
> 
> 
> [  501.707566] (nfsd,2133,14):ocfs2_test_inode_bit:2841 ERROR: unable to
> get alloc inode in slot 65535
> [  501.716660] (nfsd,2133,14):ocfs2_test_inode_bit:2867 ERROR: status
> = -22
> [  501.726585] (nfsd,2133,6):ocfs2_test_inode_bit:2841 ERROR: unable to
> get alloc inode in slot 65535
> [  501.735579] (nfsd,2133,6):ocfs2_test_inode_bit:2867 ERROR: status >
-22
> 
> But it also happens when the node is not crashing, so I think it's
> another Problem.
> 
> Last night, I saved a few of the kernel output while the servers where
> crashing one after the other like in a ping-pong game :
> 
> 
> -----8< on nodeB  8<-----
> 
> [  502.070431] nfsd: nfsv4 idmapping failing: has idmapd not been
> started?
> [  502.475747] general protection fault: 0000 [#1] SMP PTI
> [  502.481027] CPU: 6 PID: 2104 Comm: nfsd Not tainted
> 5.5.0-0.bpo.2-amd64 #1 Debian 5.5.17-1~bpo10+1
> [  502.490016] Hardware name: IBM System x3650 M3
> -[7945UHV]-/94Y7614     , BIOS -[D6E150CUS-1.11]- 02/08/2011
> [  502.499821] RIP: 0010:_raw_spin_lock+0xc/0x20
> [  502.504206] Code: 66 90 66 66 90 31 c0 ba ff 00 00 00 f0 0f b1 17 75
> 05 48 89 d8 5b c3 e8 12 5f 91 ff eb f4 66 66 66 66 90 31 c0 ba 01 00 00
> 00 <f0> 0f b1 17 75 01 c3 89 c6 e8 76 46 91 ff 66 90 c3 0f 1f 00 66
66
> [  502.523018] RSP: 0018:ffffa5af4838bac0 EFLAGS: 00010246
> [  502.528277] RAX: 0000000000000000 RBX: 0fdbbeded053c1d6 RCX:
> 0000000000000000
> [  502.535433] RDX: 0000000000000001 RSI: 0000000000000009 RDI:
> 0fdbbeded053c25e
> [  502.542588] RBP: 0fdbbeded053c25e R08: ffff9672ffa73068 R09:
> 000000000002c340
> [  502.549743] R10: 00000184d03a2115 R11: 0000000000000000 R12:
> ffff9672eff43fd0
> [  502.556899] R13: ffff9672fff66bc0 R14: 000000000000ffff R15:
> ffff9672fff660c8
> [  502.564055] FS:  0000000000000000(0000) GS:ffff968203a80000(0000)
> knlGS:0000000000000000
> [  502.572166] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [  502.577928] CR2: 00007f0a4bb5df50 CR3: 0000001e4169a004 CR4:
> 00000000000206e0
> [  502.585082] Call Trace:
> [  502.587546]  igrab+0x19/0x50
> [  502.590484]  ocfs2_get_system_file_inode+0x65/0x2c0 [ocfs2]
> [  502.596094]  ? ocfs2_read_blocks_sync+0x159/0x330 [ocfs2]
> [  502.601534]  ocfs2_test_inode_bit+0xe8/0x900 [ocfs2]
> [  502.606536]  ? ocfs2_free_dir_lookup_result+0x24/0x50 [ocfs2]
> [  502.612321]  ocfs2_get_parent+0xa3/0x300 [ocfs2]
> [  502.616960]  reconnect_path+0xa1/0x2c0
> [  502.620742]  ? nfsd_proc_setattr+0x1b0/0x1b0 [nfsd]
> [  502.625636]  exportfs_decode_fh+0x111/0x2d0
> [  502.629844]  ? exp_find_key+0xcd/0x160 [nfsd]
> [  502.634218]  ? __kmalloc+0x180/0x270
> [  502.637810]  ? security_prepare_creds+0x6f/0xa0
> [  502.642356]  ? tomoyo_write_self+0x1b0/0x1b0
> [  502.646642]  ? security_prepare_creds+0x49/0xa0
> [  502.651995]  fh_verify+0x3e5/0x5f0 [nfsd]
> [  502.656737]  nfsd3_proc_getattr+0x6b/0x100 [nfsd]
> [  502.662152]  nfsd_dispatch+0x9e/0x210 [nfsd]
> [  502.667136]  svc_process_common+0x386/0x6f0 [sunrpc]
> [  502.672791]  ? svc_sock_secure_port+0x12/0x30 [sunrpc]
> [  502.678628]  ? svc_recv+0x2ff/0x9c0 [sunrpc]
> [  502.683597]  ? nfsd_svc+0x2c0/0x2c0 [nfsd]
> [  502.688395]  ? nfsd_destroy+0x50/0x50 [nfsd]
> [  502.693377]  svc_process+0xd1/0x110 [sunrpc]
> [  502.698361]  nfsd+0xe3/0x140 [nfsd]
> [  502.702561]  kthread+0x112/0x130
> [  502.706504]  ? kthread_park+0x80/0x80
> [  502.710886]  ret_from_fork+0x35/0x40
> [  502.715181] Modules linked in: rpcsec_gss_krb5 nfsd nfs_acl lockd
> grace ocfs2 quota_tree sctp libcrc32c nft_counter ipt_REJECT
> nf_reject_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink binfmt_misc
> ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager
> cpufreq_userspace ocfs2_stackglue cpufreq_powersave cpufreq_conservative
> configfs bonding dm_round_robin dm_multipath scsi_dh_rdac scsi_dh_emc
> scsi_dh_alua intel_powerclamp coretemp kvm_intel ipmi_ssif kvm cdc_ether
> irqbypass usbnet mii tpm_tis intel_cstate tpm_tis_core intel_uncore
> joydev tpm ipmi_si ioatdma sg iTCO_wdt ib_mthca iTCO_vendor_support
> watchdog dca pcspkr ipmi_devintf rng_core ib_uverbs ipmi_msghandler
> i5500_temp i7core_edac evdev ib_umad ib_ipoib ib_cm ib_core auth_rpcgss
> sunrpc ip_tables x_tables autofs4 algif_skcipher af_alg ext4 crc16
> mbcache jbd2 crc32c_generic dm_crypt dm_mod sd_mod hid_generic usbhid
> hid sr_mod cdrom crct10dif_pclmul crc32_pclmul crc32c_intel
> ghash_clmulni_intel mgag200 drm_vram_helper drm_ttm_helper
> [  502.715218]  i2c_algo_bit ttm qla2xxx drm_kms_helper drm ata_generic
> ata_piix aesni_intel libata nvme_fc ehci_pci uhci_hcd crypto_simd
> ehci_hcd nvme_fabrics cryptd glue_helper lpc_ich nvme_core megaraid_sas
> mfd_core usbcore scsi_transport_fc e1000e scsi_mod i2c_i801 usb_common
> ptp pps_core bnx2 button
> [  502.837917] ---[ end trace 6e1a8e507ac30d8d ]---
> [  502.843379] RIP: 0010:_raw_spin_lock+0xc/0x20
> [  502.848584] Code: 66 90 66 66 90 31 c0 ba ff 00 00 00 f0 0f b1 17 75
> 05 48 89 d8 5b c3 e8 12 5f 91 ff eb f4 66 66 66 66 90 31 c0 ba 01 00 00
> 00 <f0> 0f b1 17 75 01 c3 89 c6 e8 76 46 91 ff 66 90 c3 0f 1f 00 66
66
> [  502.869295] RSP: 0018:ffffa5af4838bac0 EFLAGS: 00010246
> [  502.875404] RAX: 0000000000000000 RBX: 0fdbbeded053c1d6 RCX:
> 0000000000000000
> [  502.883423] RDX: 0000000000000001 RSI: 0000000000000009 RDI:
> 0fdbbeded053c25e
> [  502.891436] RBP: 0fdbbeded053c25e R08: ffff9672ffa73068 R09:
> 000000000002c340
> [  502.899446] R10: 00000184d03a2115 R11: 0000000000000000 R12:
> ffff9672eff43fd0
> [  502.907455] R13: ffff9672fff66bc0 R14: 000000000000ffff R15:
> ffff9672fff660c8
> [  502.915473] FS:  0000000000000000(0000) GS:ffff968203a80000(0000)
> knlGS:0000000000000000
> [  502.924496] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [  502.931147] CR2: 00007f0a4bb5df50 CR3: 0000001e4169a004 CR4:
> 00000000000206e0
> 
> -----8< cut here 8<-----
> 
> 
> shortly after when trying to start nodeA while nodeB was down :
> 
> -----8< nodeA 8<-----
> 
> [  728.927750] general protection fault: 0000 [#1] SMP PTI
> [  728.934647] CPU: 8 PID: 12156 Comm: nfsd Not tainted
> 5.5.0-0.bpo.2-amd64 #1 Debian 5.5.17-1~bpo10+1
> [  728.944800] Hardware name: IBM System x3650 M3 -[7945H2G]-/69Y4438,
> BIOS -[D6E158AUS-1.16]- 11/26/2012
> [  728.955176] RIP: 0010:_raw_spin_lock+0xc/0x20
> [  728.960595] Code: 66 90 66 66 90 31 c0 ba ff 00 00 00 f0 0f b1 17 75
> 05 48 89 d8 5b c3 e8 12 5f 91 ff eb f4 66 66 66 66 90 31 c0 ba 01 00 00
> 00 <f0> 0f b1 17 75 01 c3 89 c6 e8 76 46 91 ff 66 90 c3 0f 1f 00 66
66
> [  728.981559] RSP: 0018:ffffa24d09187ac0 EFLAGS: 00010246
> [  728.987885] RAX: 0000000000000000 RBX: 625f656369766564 RCX:
> 0000000000000000
> [  728.996135] RDX: 0000000000000001 RSI: 0000000000000009 RDI:
> 625f6563697665ec
> [  729.004384] RBP: 625f6563697665ec R08: ffff91f1272e3680 R09:
> 000000000002c340
> [  729.012628] R10: 000002114c00d22b R11: 0000000000000000 R12:
> ffff91e13858bbd0
> [  729.020877] R13: ffff91f13c03bbc0 R14: 000000000000ffff R15:
> ffff91f13c03b0c8
> [  729.029108] FS:  0000000000000000(0000) GS:ffff91f13fa80000(0000)
> knlGS:0000000000000000
> [  729.038308] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [  729.045161] CR2: 000055fe95d1d9a8 CR3: 0000001c8420a003 CR4:
> 00000000000206e0
> [  729.053415] Call Trace:
> [  729.056978]  igrab+0x19/0x50
> [  729.061023]  ocfs2_get_system_file_inode+0x65/0x2c0 [ocfs2]
> [  729.067736]  ? ocfs2_read_blocks_sync+0x159/0x330 [ocfs2]
> [  729.074284]  ocfs2_test_inode_bit+0xe8/0x900 [ocfs2]
> [  729.080386]  ? ocfs2_free_dir_lookup_result+0x24/0x50 [ocfs2]
> [  729.087268]  ocfs2_get_parent+0xa3/0x300 [ocfs2]
> [  729.092995]  reconnect_path+0xa1/0x2c0
> [  729.097863]  ? nfsd_proc_setattr+0x1b0/0x1b0 [nfsd]
> [  729.103840]  exportfs_decode_fh+0x111/0x2d0
> [  729.109133]  ? exp_find_key+0xcd/0x160 [nfsd]
> [  729.114574]  ? dbs_update_util_handler+0x16/0x90
> [  729.120273]  ? __kmalloc+0x180/0x270
> [  729.124920]  ? security_prepare_creds+0x6f/0xa0
> [  729.130521]  ? tomoyo_write_self+0x1b0/0x1b0
> [  729.135858]  ? security_prepare_creds+0x49/0xa0
> [  729.141464]  fh_verify+0x3e5/0x5f0 [nfsd]
> [  729.146536]  nfsd3_proc_getattr+0x6b/0x100 [nfsd]
> [  729.152297]  nfsd_dispatch+0x9e/0x210 [nfsd]
> [  729.157647]  svc_process_common+0x386/0x6f0 [sunrpc]
> [  729.163675]  ? svc_sock_secure_port+0x12/0x30 [sunrpc]
> [  729.169873]  ? svc_recv+0x2ff/0x9c0 [sunrpc]
> [  729.175190]  ? nfsd_svc+0x2c0/0x2c0 [nfsd]
> [  729.180323]  ? nfsd_destroy+0x50/0x50 [nfsd]
> [  729.185631]  svc_process+0xd1/0x110 [sunrpc]
> [  729.190925]  nfsd+0xe3/0x140 [nfsd]
> [  729.195421]  kthread+0x112/0x130
> [  729.199636]  ? kthread_park+0x80/0x80
> [  729.204279]  ret_from_fork+0x35/0x40
> [  729.208818] Modules linked in: ocfs2 quota_tree dm_crypt crypto_simd
> cryptd glue_helper algif_skcipher af_alg nfsd nfs_acl lockd grace sctp
> libcrc32c nft_counter xt_tcpudp nft_compat nf_tables nfnetlink
> binfmt_misc ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager
> ocfs2_stackglue configfs bonding dm_round_robin dm_multipath
> scsi_dh_rdac scsi_dh_emc scsi_dh_alua intel_powerclamp ipmi_ssif
> coretemp kvm_intel kvm irqbypass cdc_ether intel_cstate usbnet ib_mthca
> mii joydev intel_uncore ib_uverbs iTCO_wdt sg ioatdma
> iTCO_vendor_support i7core_edac watchdog i5500_temp pcspkr ipmi_si
> tpm_tis tpm_tis_core tpm ipmi_devintf ipmi_msghandler rng_core evdev
> acpi_cpufreq ib_umad auth_rpcgss ib_ipoib ib_cm sunrpc ib_core ip_tables
> x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic dm_mod
> hid_generic sd_mod usbhid hid sr_mod cdrom uas usb_storage mgag200
> drm_vram_helper drm_ttm_helper qla2xxx ata_generic i2c_algo_bit ttm
> nvme_fc ixgbe nvme_fabrics ata_piix xfrm_algo ehci_pci uhci_hcd
> [  729.208868]  drm_kms_helper nvme_core ehci_hcd dca libata
> megaraid_sas scsi_transport_fc libphy ptp drm usbcore scsi_mod
> crc32c_intel lpc_ich i2c_i801 pps_core mfd_core usb_common bnx2 mdio
> button
> [  729.323900] ---[ end trace 212cefe75207e17f ]---
> [  729.329644] RIP: 0010:_raw_spin_lock+0xc/0x20
> [  729.335121] Code: 66 90 66 66 90 31 c0 ba ff 00 00 00 f0 0f b1 17 75
> 05 48 89 d8 5b c3 e8 12 5f 91 ff eb f4 66 66 66 66 90 31 c0 ba 01 00 00
> 00 <f0> 0f b1 17 75 01 c3 89 c6 e8 76 46 91 ff 66 90 c3 0f 1f 00 66
66
> [  729.356192] RSP: 0018:ffffa24d09187ac0 EFLAGS: 00010246
> [  729.362584] RAX: 0000000000000000 RBX: 625f656369766564 RCX:
> 0000000000000000
> [  729.370889] RDX: 0000000000000001 RSI: 0000000000000009 RDI:
> 625f6563697665ec
> [  729.379184] RBP: 625f6563697665ec R08: ffff91f1272e3680 R09:
> 000000000002c340
> [  729.387475] R10: 000002114c00d22b R11: 0000000000000000 R12:
> ffff91e13858bbd0
> [  729.395767] R13: ffff91f13c03bbc0 R14: 000000000000ffff R15:
> ffff91f13c03b0c8
> [  729.404066] FS:  0000000000000000(0000) GS:ffff91f13fa80000(0000)
> knlGS:0000000000000000
> [  729.413336] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [  729.420262] CR2: 000055fe95d1d9a8 CR3: 0000001c8420a003 CR4:
> 00000000000206e0
> 
> -----8< cut here 8<-----
> 
> Since the backtrace is speaking about nfsd, I tried to isolate the
> Problem by droping all NFS packet entering to the active node before
> starting the services :
> 
> iptables -I INPUT -p tcp --dport 2049 -j DROP
> 
> Everything fine, access over samba were doing fine.
> 
> I started to allow NFS for part of my network with iptables, until the
> crash occurs.
> 
> On this one, the crash occured after allowing only a NFSv3 client.
> 
> -----8< nodeB 8<-----
> 
> [ 2494.893697] BUG: unable to handle page fault for address:
> 00000000000fab7d
> [ 2494.900626] #PF: supervisor write access in kernel mode
> [ 2494.905873] #PF: error_code(0x0002) - not-present page
> [ 2494.911042] PGD 0 P4D 0
> [ 2494.913596] Oops: 0002 [#1] SMP PTI
> [ 2494.917106] CPU: 11 PID: 21342 Comm: nfsd Not tainted
> 5.5.0-0.bpo.2-amd64 #1 Debian 5.5.17-1~bpo10+1
> [ 2494.926285] Hardware name: IBM System x3650 M3
> -[7945UHV]-/94Y7614     , BIOS -[D6E150CUS-1.11]- 02/08/2011
> [ 2494.936077] RIP: 0010:_raw_spin_lock+0xc/0x20
> [ 2494.940461] Code: 66 90 66 66 90 31 c0 ba ff 00 00 00 f0 0f b1 17 75
> 05 48 89 d8 5b c3 e8 12 5f 91 ff eb f4 66 66 66 66 90 31 c0 ba 01 00 00
> 00 <f0> 0f b1 17 75 01 c3 89 c6 e8 76 46 91 ff 66 90 c3 0f 1f 00 66
66
> [ 2494.959269] RSP: 0018:ffffb949cca7fac0 EFLAGS: 00010246
> [ 2494.964511] RAX: 0000000000000000 RBX: 00000000000faaf5 RCX:
> 0000000000000000
> [ 2494.971664] RDX: 0000000000000001 RSI: 0000000000000009 RDI:
> 00000000000fab7d
> [ 2494.978815] RBP: 00000000000fab7d R08: ffff99c8f12737b8 R09:
> 000000000002c340
> [ 2494.985968] R10: 000005dc16e333d8 R11: 0000000000000000 R12:
> ffff99c903066bd0
> [ 2494.993119] R13: ffff99c9014c7bc0 R14: 000000000000ffff R15:
> ffff99c9014c70c8
> [ 2495.000271] FS:  0000000000000000(0000) GS:ffff99ba03bc0000(0000)
> knlGS:0000000000000000
> [ 2495.008382] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 2495.014141] CR2: 00000000000fab7d CR3: 0000000f38df2002 CR4:
> 00000000000206e0
> [ 2495.021291] Call Trace:
> [ 2495.023753]  igrab+0x19/0x50
> [ 2495.026693]  ocfs2_get_system_file_inode+0x65/0x2c0 [ocfs2]
> [ 2495.032301]  ? ocfs2_read_blocks_sync+0x159/0x330 [ocfs2]
> [ 2495.037741]  ocfs2_test_inode_bit+0xe8/0x900 [ocfs2]
> [ 2495.042742]  ? ocfs2_free_dir_lookup_result+0x24/0x50 [ocfs2]
> [ 2495.048525]  ocfs2_get_parent+0xa3/0x300 [ocfs2]
> [ 2495.053163]  reconnect_path+0xa1/0x2c0
> [ 2495.056943]  ? nfsd_proc_setattr+0x1b0/0x1b0 [nfsd]
> [ 2495.061838]  exportfs_decode_fh+0x111/0x2d0
> [ 2495.066042]  ? exp_find_key+0xcd/0x160 [nfsd]
> [ 2495.070417]  ? __kmalloc+0x180/0x270
> [ 2495.074007]  ? security_prepare_creds+0x6f/0xa0
> [ 2495.078551]  ? tomoyo_write_self+0x1b0/0x1b0
> [ 2495.082834]  ? security_prepare_creds+0x49/0xa0
> [ 2495.087387]  fh_verify+0x3e5/0x5f0 [nfsd]
> [ 2495.092125]  nfsd3_proc_getattr+0x6b/0x100 [nfsd]
> [ 2495.097517]  nfsd_dispatch+0x9e/0x210 [nfsd]
> [ 2495.102489]  svc_process_common+0x386/0x6f0 [sunrpc]
> [ 2495.108134]  ? svc_sock_secure_port+0x12/0x30 [sunrpc]
> [ 2495.113946]  ? svc_recv+0x2ff/0x9c0 [sunrpc]
> [ 2495.118883]  ? nfsd_svc+0x2c0/0x2c0 [nfsd]
> [ 2495.123651]  ? nfsd_destroy+0x50/0x50 [nfsd]
> [ 2495.128604]  svc_process+0xd1/0x110 [sunrpc]
> [ 2495.133563]  nfsd+0xe3/0x140 [nfsd]
> [ 2495.137733]  kthread+0x112/0x130
> [ 2495.141643]  ? kthread_park+0x80/0x80
> [ 2495.145993]  ret_from_fork+0x35/0x40
> [ 2495.150262] Modules linked in: ocfs2 quota_tree nfsd nfs_acl lockd
> grace sctp libcrc32c nft_counter xt_tcpudp nft_compat nf_tables
> nfnetlink binfmt_misc ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm
> ocfs2_nodemanager ocfs2_stackglue cpufreq_userspace cpufreq_powersave
> cpufreq_conservative configfs bonding dm_round_robin dm_multipath
> scsi_dh_rdac scsi_dh_emc scsi_dh_alua intel_powerclamp coretemp
> kvm_intel kvm ipmi_ssif irqbypass cdc_ether usbnet intel_cstate mii
> ib_mthca intel_uncore evdev iTCO_wdt joydev ipmi_si tpm_tis sg
> tpm_tis_core tpm ipmi_devintf iTCO_vendor_support ib_uverbs pcspkr
> watchdog ioatdma rng_core i7core_edac dca i5500_temp ipmi_msghandler
> ib_umad ib_ipoib auth_rpcgss ib_cm sunrpc ib_core ip_tables x_tables
> autofs4 algif_skcipher af_alg ext4 crc16 mbcache jbd2 crc32c_generic
> dm_crypt dm_mod sr_mod cdrom sd_mod hid_generic usbhid hid
> crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel mgag200
> drm_vram_helper drm_ttm_helper i2c_algo_bit ttm qla2xxx drm_kms_helper
> [ 2495.150298]  ata_generic drm ata_piix aesni_intel libata nvme_fc
> uhci_hcd crypto_simd nvme_fabrics ehci_pci ehci_hcd megaraid_sas cryptd
> nvme_core glue_helper usbcore scsi_transport_fc lpc_ich mfd_core e1000e
> scsi_mod i2c_i801 usb_common bnx2 ptp pps_core button
> [ 2495.269073] CR2: 00000000000fab7d
> [ 2495.273205] ---[ end trace 44f000505f987296 ]---
> [ 2495.278647] RIP: 0010:_raw_spin_lock+0xc/0x20
> [ 2495.283825] Code: 66 90 66 66 90 31 c0 ba ff 00 00 00 f0 0f b1 17 75
> 05 48 89 d8 5b c3 e8 12 5f 91 ff eb f4 66 66 66 66 90 31 c0 ba 01 00 00
> 00 <f0> 0f b1 17 75 01 c3 89 c6 e8 76 46 91 ff 66 90 c3 0f 1f 00 66
66
> [ 2495.304287] RSP: 0018:ffffb949cca7fac0 EFLAGS: 00010246
> [ 2495.310368] RAX: 0000000000000000 RBX: 00000000000faaf5 RCX:
> 0000000000000000
> [ 2495.318358] RDX: 0000000000000001 RSI: 0000000000000009 RDI:
> 00000000000fab7d
> [ 2495.326339] RBP: 00000000000fab7d R08: ffff99c8f12737b8 R09:
> 000000000002c340
> [ 2495.334314] R10: 000005dc16e333d8 R11: 0000000000000000 R12:
> ffff99c903066bd0
> [ 2495.342297] R13: ffff99c9014c7bc0 R14: 000000000000ffff R15:
> ffff99c9014c70c8
> [ 2495.350279] FS:  0000000000000000(0000) GS:ffff99ba03bc0000(0000)
> knlGS:0000000000000000
> [ 2495.359224] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 2495.365825] CR2: 00000000000fab7d CR3: 0000000f38df2002 CR4:
> 00000000000206e0
> 
> -----8< cut here 8<-----
> 
> 
> I also made some record of NFS traffic with tcpdump when the crashs are
> occuring, but I didn't had time to analyse it. I can provide them if it
> helps (but I might have to clean them up first to avoid some data
> leaks :)
> 
> After this happening a few times, I had one case where one of the few
> NFSv4 client was involved, and I took as aworkaround to migrate the
> NFSv4 share to XFS since I though it could be NFSv4 related. Now that I
> looked at the kernel dumps, I think it's also NFSv3 related.
> 
> Is it a kernel bug or am I missing something ?
> 
> For now, I will switch back to XFS with a failover setup of my cluster,
> cause I cannot afford this kind of crashes of a central file server.
> 
> 
> 
> Cheers
> 
> 
> Pierre
> 
> 
> PS : I'll try to update with what I found in the PCAP files in some
days
> 
> 
> 
>>
>>
>> _______________________________________________
>> Ocfs2-devel mailing list
>> Ocfs2-devel at oss.oracle.com
>> https://oss.oracle.com/mailman/listinfo/ocfs2-devel