Gluster users - Nov 2017 - Changing performance.parallel-readdir to on causes CPU soft lockup and very high load all glusterd nodes

Hi,

We're using a 3-node setup where GlusterFS is running as both a client and
a server with a fuse mount-point.

We tried to change the performance.parallel-readdir setting to on for a
volume, but after that the load on all 3 nodes skyrocketed due to the
glusterd process and we saw CPU soft lockup errors in the console.
I had to completely bring down/reboot all 3 nodes and disable the setting
again.

There were tons of errors like mentioned below, does anyone know what could
cause this?

Nov 10 20:55:53 n01c01 kernel: [196591.960126] BUG: soft lockup - CPU#6
stuck for 22s! [touch:25995]
Nov 10 20:55:53 n01c01 kernel: [196591.960168] Modules linked in:
xt_multiport binfmt_misc hcpdriver(PO) nf_conntrack_ipv6 nf_defrag_ipv6
ip6table_filter ip6_tables nf_conntrack_ipv4 nf_defrag_ipv4 xt_tcpudp
xt_conntrack iptable_filter ip_tables x_tables xfs libcrc32c crc32c_generic
nls_utf8 nls_cp437 vfat fat x86_pkg_temp_thermal coretemp iTCO_wdt
iTCO_vendor_support kvm_intel kvm crc32_pclmul ast ttm aesni_intel
drm_kms_helper efi_pstore aes_x86_64 lrw gf128mul evdev glue_helper
ablk_helper joydev cryptd pcspkr efivars drm i2c_algo_bit mei_me lpc_ich
mei mfd_core shpchp tpm_tis wmi tpm ipmi_si ipmi_msghandler acpi_pad
processor button acpi_power_meter thermal_sys nf_conntrack_ftp nf_conntrack
fuse autofs4 ext4 crc16 mbcache jbd2 dm_mod hid_generic usbhid hid sg
sd_mod crc_t10dif crct10dif_generic ehci_pci xhci_hcd ehci_hcd ahci libahci
i2c_i801 crct10dif_pclmul crct10dif_common libata ixgbe i2c_core
crc32c_intel dca usbcore scsi_mod ptp usb_common pps_core nvme mdio
Nov 10 20:55:53 n01c01 kernel: [196591.960224] CPU: 6 PID: 25995 Comm:
touch Tainted: P           O  3.16.0-4-amd64 #1 Debian 3.16.43-2+deb8u5
Nov 10 20:55:53 n01c01 kernel: [196591.960226] Hardware name: Supermicro
SYS-1028U-TNR4T+/X10DRU-i+, BIOS 2.0c 04/21/2017
Nov 10 20:55:53 n01c01 kernel: [196591.960228] task: ffff88184bf872f0 ti:
ffff88182cbc0000 task.ti: ffff88182cbc0000
Nov 10 20:55:53 n01c01 kernel: [196591.960229] RIP:
0010:[<ffffffff81519fe5>]  [<ffffffff81519fe5>]
_raw_spin_lock+0x25/0x30
Nov 10 20:55:53 n01c01 kernel: [196591.960237] RSP: 0018:ffff88182cbc3b78
EFLAGS: 00000287
Nov 10 20:55:53 n01c01 kernel: [196591.960239] RAX: 0000000000005e5c RBX:
ffffffff811646a5 RCX: 0000000000005e69
Nov 10 20:55:53 n01c01 kernel: [196591.960240] RDX: 0000000000005e69 RSI:
ffffffff811bffa0 RDI: ffff88182e42bc70
Nov 10 20:55:53 n01c01 kernel: [196591.960241] RBP: ffff88182e42bc18 R08:
0000000000200008 R09: 0000000000000001
Nov 10 20:55:53 n01c01 kernel: [196591.960242] R10: 0000000000012f40 R11:
0000000000000010 R12: ffff88182cbc3af0
Nov 10 20:55:53 n01c01 kernel: [196591.960243] R13: 0000000000000286 R14:
0000000000000010 R15: ffffffff81519fce
Nov 10 20:55:53 n01c01 kernel: [196591.960244] FS:  00007fc005c67700(0000)
GS:ffff88187fc80000(0000) knlGS:0000000000000000
Nov 10 20:55:53 n01c01 kernel: [196591.960246] CS:  0010 DS: 0000 ES: 0000
CR0: 0000000080050033
Nov 10 20:55:53 n01c01 kernel: [196591.960247] CR2: 00007f498c1ab148 CR3:
0000000b0fcf4000 CR4: 00000000003407e0
Nov 10 20:55:53 n01c01 kernel: [196591.960248] DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
Nov 10 20:55:53 n01c01 kernel: [196591.960249] DR3: 0000000000000000 DR6:
00000000fffe0ff0 DR7: 0000000000000400
Nov 10 20:55:53 n01c01 kernel: [196591.960250] Stack:
Nov 10 20:55:53 n01c01 kernel: [196591.960251]  ffffffff81513196
ffff88182e42bc18 ffff880aa062e8d8 ffff880aa062e858
Nov 10 20:55:53 n01c01 kernel: [196591.960254]  ffffffff811c1120
ffff88182cbc3bd0 ffff88182e42bc18 0000000000032b68
Nov 10 20:55:53 n01c01 kernel: [196591.960256]  ffff88182943b010
ffffffff811c1988 ffff88182e42bc18 ffff8817de0db218
Nov 10 20:55:53 n01c01 kernel: [196591.960258] Call Trace:
Nov 10 20:55:53 n01c01 kernel: [196591.960263]  [<ffffffff81513196>] ?
lock_parent.part.17+0x1d/0x43
Nov 10 20:55:53 n01c01 kernel: [196591.960268]  [<ffffffff811c1120>] ?
shrink_dentry_list+0x1f0/0x240
Nov 10 20:55:53 n01c01 kernel: [196591.960270]  [<ffffffff811c1988>] ?
check_submounts_and_drop+0x68/0x90
Nov 10 20:55:53 n01c01 kernel: [196591.960278]  [<ffffffffa017f7f8>] ?
fuse_dentry_revalidate+0x1e8/0x300 [fuse]
Nov 10 20:55:53 n01c01 kernel: [196591.960281]  [<ffffffff811b4e5e>] ?
lookup_fast+0x25e/0x2b0
Nov 10 20:55:53 n01c01 kernel: [196591.960283]  [<ffffffff811b5ebb>] ?
link_path_walk+0x1ab/0x870
Nov 10 20:55:53 n01c01 kernel: [196591.960285]  [<ffffffff811ba2ec>] ?
path_openat+0x9c/0x680
Nov 10 20:55:53 n01c01 kernel: [196591.960289]  [<ffffffff8116c0fc>] ?
handle_mm_fault+0x63c/0x1150
Nov 10 20:55:53 n01c01 kernel: [196591.960292]  [<ffffffff8117253c>] ?
mmap_region+0x19c/0x650
Nov 10 20:55:53 n01c01 kernel: [196591.960294]  [<ffffffff811bb07a>] ?
do_filp_open+0x3a/0x90
Nov 10 20:55:53 n01c01 kernel: [196591.960297]  [<ffffffff811c736c>] ?
__alloc_fd+0x7c/0x120
Nov 10 20:55:53 n01c01 kernel: [196591.960302]  [<ffffffff811aa169>] ?
do_sys_open+0x129/0x220
Nov 10 20:55:53 n01c01 kernel: [196591.960305]  [<ffffffff8151a48d>] ?
system_call_fast_compare_end+0x10/0x15
Nov 10 20:55:53 n01c01 kernel: [196591.960306] Code: 84 00 00 00 00 00 0f
1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 89 d1 75 01
c3 0f b7 07 66 39 d0 74 f7 f3 90 <0f> b7 07 66 39 c8 75 f6 c3 66 90 0f 1f
44 00 00 65 81 04 25 60


We are running Debian 8 with Glusterfs 3.12.2

Here are our settings:
gluster volume info

Volume Name: ssl
Type: Replicate
Volume ID: 8aa7bab2-a8dc-4855-ab46-bbcb4a9ff174
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 10.0.0.1:/storage/gluster/ssl
Brick2: 10.0.0.2:/storage/gluster/ssl
Brick3: 10.0.0.3:/storage/gluster/ssl
Options Reconfigured:
performance.client-io-threads: off
nfs.disable: on
transport.address-family: inet
network.ping-timeout: 5

Volume Name: www
Type: Replicate
Volume ID: d8d87920-f92d-4669-8509-acd1936ba365
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 10.0.0.1:/storage/gluster/www
Brick2: 10.0.0.2:/storage/gluster/www
Brick3: 10.0.0.3:/storage/gluster/www
Options Reconfigured:
features.scrub: Active
features.bitrot: on
performance.flush-behind: on
network.ping-timeout: 3
features.cache-invalidation-timeout: 600
performance.write-behind-window-size: 4MB
performance.quick-read: off
performance.md-cache-timeout: 600
performance.io-thread-count: 64
cluster.readdir-optimize: on
performance.write-behind: on
performance.readdir-ahead: on
performance.cache-size: 1GB
performance.cache-invalidation: on
features.cache-invalidation: on
diagnostics.brick-log-level: WARNING
performance.cache-priority: *.php:3,*.temp:3,*:1
network.inode-lru-limit: 90000
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
performance.parallel-readdir: off

And here are our /etc/fstab entries for the fuse mountpoint:
localhost:/ssl /etc/ssl/shared glusterfs
backup-volfile-servers=10.0.0.2:10.0.0.3,log-level=WARNING
0 0
localhost:/www /var/www glusterfs
backup-volfile-servers=10.0.0.2:10.0.0.3,log-level=WARNING
0 0

Fuse version:
dpkg -l | grep fuse
ii  fuse                                 2.9.3-15+deb8u2
                               amd64        Filesystem in Userspace
ii  libfuse2:amd64                       2.9.3-15+deb8u2
                               amd64        Filesystem in Userspace
(library)


Thank you!
Niels Hendriks
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20171114/3cb051e7/attachment.html>

By chance... it's not 3 replica, 1 arbiter is it?

--
Sam McLeod
https://smcleod.net
https://twitter.com/s_mcleod

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20171115/82fd8c74/attachment.html>

By chance... it's not 3 replica, 1 arbiter is it?

See: https://bugzilla.redhat.com/show_bug.cgi?id=1512371
<https://bugzilla.redhat.com/show_bug.cgi?id=1512371>

--
Sam McLeod
https://smcleod.net
https://twitter.com/s_mcleod

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20171115/5b633b40/attachment.html>

Hi Sam,

Thanks for your response. No, we do not have any arbiters. Just 3x replica.

Best regards,
Niels Hendriks

On 14 November 2017 at 23:02, Sam McLeod <mailinglists at smcleod.net>
wrote:
> By chance... it's not 3 replica, 1 arbiter is it?
>
> See: https://bugzilla.redhat.com/show_bug.cgi?id=1512371
>
> --
> Sam McLeod
> https://smcleod.net
> https://twitter.com/s_mcleod
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20171115/cd756cfd/attachment.html>