Gluster users - Nov 2017 - Changing performance.parallel-readdir to on causes CPU soft lockup and very high load all glusterd nodes

Hi,

Just to update this thread. We updated from Gluster 3.12.2 to 3.12.3 which
resolved the issue it seems. I checked the changelog but don't see anything
that looks like this issue, but I'm glad it seems like it's OK now.

Niels Hendriks

On 14 November 2017 at 09:42, Niels Hendriks <niels at nuvini.com> wrote:
> Hi,
>
> We're using a 3-node setup where GlusterFS is running as both a client
and
> a server with a fuse mount-point.
>
> We tried to change the performance.parallel-readdir setting to on for a
> volume, but after that the load on all 3 nodes skyrocketed due to the
> glusterd process and we saw CPU soft lockup errors in the console.
> I had to completely bring down/reboot all 3 nodes and disable the setting
> again.
>
> There were tons of errors like mentioned below, does anyone know what
> could cause this?
>
> Nov 10 20:55:53 n01c01 kernel: [196591.960126] BUG: soft lockup - CPU#6
> stuck for 22s! [touch:25995]
> Nov 10 20:55:53 n01c01 kernel: [196591.960168] Modules linked in:
> xt_multiport binfmt_misc hcpdriver(PO) nf_conntrack_ipv6 nf_defrag_ipv6
> ip6table_filter ip6_tables nf_conntrack_ipv4 nf_defrag_ipv4 xt_tcpudp
> xt_conntrack iptable_filter ip_tables x_tables xfs libcrc32c crc32c_generic
> nls_utf8 nls_cp437 vfat fat x86_pkg_temp_thermal coretemp iTCO_wdt
> iTCO_vendor_support kvm_intel kvm crc32_pclmul ast ttm aesni_intel
> drm_kms_helper efi_pstore aes_x86_64 lrw gf128mul evdev glue_helper
> ablk_helper joydev cryptd pcspkr efivars drm i2c_algo_bit mei_me lpc_ich
> mei mfd_core shpchp tpm_tis wmi tpm ipmi_si ipmi_msghandler acpi_pad
> processor button acpi_power_meter thermal_sys nf_conntrack_ftp nf_conntrack
> fuse autofs4 ext4 crc16 mbcache jbd2 dm_mod hid_generic usbhid hid sg
> sd_mod crc_t10dif crct10dif_generic ehci_pci xhci_hcd ehci_hcd ahci libahci
> i2c_i801 crct10dif_pclmul crct10dif_common libata ixgbe i2c_core
> crc32c_intel dca usbcore scsi_mod ptp usb_common pps_core nvme mdio
> Nov 10 20:55:53 n01c01 kernel: [196591.960224] CPU: 6 PID: 25995 Comm:
> touch Tainted: P           O  3.16.0-4-amd64 #1 Debian 3.16.43-2+deb8u5
> Nov 10 20:55:53 n01c01 kernel: [196591.960226] Hardware name: Supermicro
> SYS-1028U-TNR4T+/X10DRU-i+, BIOS 2.0c 04/21/2017
> Nov 10 20:55:53 n01c01 kernel: [196591.960228] task: ffff88184bf872f0 ti:
> ffff88182cbc0000 task.ti: ffff88182cbc0000
> Nov 10 20:55:53 n01c01 kernel: [196591.960229] RIP:
> 0010:[<ffffffff81519fe5>]  [<ffffffff81519fe5>]
_raw_spin_lock+0x25/0x30
> Nov 10 20:55:53 n01c01 kernel: [196591.960237] RSP: 0018:ffff88182cbc3b78
> EFLAGS: 00000287
> Nov 10 20:55:53 n01c01 kernel: [196591.960239] RAX: 0000000000005e5c RBX:
> ffffffff811646a5 RCX: 0000000000005e69
> Nov 10 20:55:53 n01c01 kernel: [196591.960240] RDX: 0000000000005e69 RSI:
> ffffffff811bffa0 RDI: ffff88182e42bc70
> Nov 10 20:55:53 n01c01 kernel: [196591.960241] RBP: ffff88182e42bc18 R08:
> 0000000000200008 R09: 0000000000000001
> Nov 10 20:55:53 n01c01 kernel: [196591.960242] R10: 0000000000012f40 R11:
> 0000000000000010 R12: ffff88182cbc3af0
> Nov 10 20:55:53 n01c01 kernel: [196591.960243] R13: 0000000000000286 R14:
> 0000000000000010 R15: ffffffff81519fce
> Nov 10 20:55:53 n01c01 kernel: [196591.960244] FS:  00007fc005c67700(0000)
> GS:ffff88187fc80000(0000) knlGS:0000000000000000
> Nov 10 20:55:53 n01c01 kernel: [196591.960246] CS:  0010 DS: 0000 ES: 0000
> CR0: 0000000080050033
> Nov 10 20:55:53 n01c01 kernel: [196591.960247] CR2: 00007f498c1ab148 CR3:
> 0000000b0fcf4000 CR4: 00000000003407e0
> Nov 10 20:55:53 n01c01 kernel: [196591.960248] DR0: 0000000000000000 DR1:
> 0000000000000000 DR2: 0000000000000000
> Nov 10 20:55:53 n01c01 kernel: [196591.960249] DR3: 0000000000000000 DR6:
> 00000000fffe0ff0 DR7: 0000000000000400
> Nov 10 20:55:53 n01c01 kernel: [196591.960250] Stack:
> Nov 10 20:55:53 n01c01 kernel: [196591.960251]  ffffffff81513196
> ffff88182e42bc18 ffff880aa062e8d8 ffff880aa062e858
> Nov 10 20:55:53 n01c01 kernel: [196591.960254]  ffffffff811c1120
> ffff88182cbc3bd0 ffff88182e42bc18 0000000000032b68
> Nov 10 20:55:53 n01c01 kernel: [196591.960256]  ffff88182943b010
> ffffffff811c1988 ffff88182e42bc18 ffff8817de0db218
> Nov 10 20:55:53 n01c01 kernel: [196591.960258] Call Trace:
> Nov 10 20:55:53 n01c01 kernel: [196591.960263]  [<ffffffff81513196>]
?
> lock_parent.part.17+0x1d/0x43
> Nov 10 20:55:53 n01c01 kernel: [196591.960268]  [<ffffffff811c1120>]
?
> shrink_dentry_list+0x1f0/0x240
> Nov 10 20:55:53 n01c01 kernel: [196591.960270]  [<ffffffff811c1988>]
?
> check_submounts_and_drop+0x68/0x90
> Nov 10 20:55:53 n01c01 kernel: [196591.960278]  [<ffffffffa017f7f8>]
?
> fuse_dentry_revalidate+0x1e8/0x300 [fuse]
> Nov 10 20:55:53 n01c01 kernel: [196591.960281]  [<ffffffff811b4e5e>]
?
> lookup_fast+0x25e/0x2b0
> Nov 10 20:55:53 n01c01 kernel: [196591.960283]  [<ffffffff811b5ebb>]
?
> link_path_walk+0x1ab/0x870
> Nov 10 20:55:53 n01c01 kernel: [196591.960285]  [<ffffffff811ba2ec>]
?
> path_openat+0x9c/0x680
> Nov 10 20:55:53 n01c01 kernel: [196591.960289]  [<ffffffff8116c0fc>]
?
> handle_mm_fault+0x63c/0x1150
> Nov 10 20:55:53 n01c01 kernel: [196591.960292]  [<ffffffff8117253c>]
?
> mmap_region+0x19c/0x650
> Nov 10 20:55:53 n01c01 kernel: [196591.960294]  [<ffffffff811bb07a>]
?
> do_filp_open+0x3a/0x90
> Nov 10 20:55:53 n01c01 kernel: [196591.960297]  [<ffffffff811c736c>]
?
> __alloc_fd+0x7c/0x120
> Nov 10 20:55:53 n01c01 kernel: [196591.960302]  [<ffffffff811aa169>]
?
> do_sys_open+0x129/0x220
> Nov 10 20:55:53 n01c01 kernel: [196591.960305]  [<ffffffff8151a48d>]
?
> system_call_fast_compare_end+0x10/0x15
> Nov 10 20:55:53 n01c01 kernel: [196591.960306] Code: 84 00 00 00 00 00 0f
> 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 89 d1 75 01
> c3 0f b7 07 66 39 d0 74 f7 f3 90 <0f> b7 07 66 39 c8 75 f6 c3 66 90
0f 1f
> 44 00 00 65 81 04 25 60
>
>
> We are running Debian 8 with Glusterfs 3.12.2
>
> Here are our settings:
> gluster volume info
>
> Volume Name: ssl
> Type: Replicate
> Volume ID: 8aa7bab2-a8dc-4855-ab46-bbcb4a9ff174
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x 3 = 3
> Transport-type: tcp
> Bricks:
> Brick1: 10.0.0.1:/storage/gluster/ssl
> Brick2: 10.0.0.2:/storage/gluster/ssl
> Brick3: 10.0.0.3:/storage/gluster/ssl
> Options Reconfigured:
> performance.client-io-threads: off
> nfs.disable: on
> transport.address-family: inet
> network.ping-timeout: 5
>
> Volume Name: www
> Type: Replicate
> Volume ID: d8d87920-f92d-4669-8509-acd1936ba365
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x 3 = 3
> Transport-type: tcp
> Bricks:
> Brick1: 10.0.0.1:/storage/gluster/www
> Brick2: 10.0.0.2:/storage/gluster/www
> Brick3: 10.0.0.3:/storage/gluster/www
> Options Reconfigured:
> features.scrub: Active
> features.bitrot: on
> performance.flush-behind: on
> network.ping-timeout: 3
> features.cache-invalidation-timeout: 600
> performance.write-behind-window-size: 4MB
> performance.quick-read: off
> performance.md-cache-timeout: 600
> performance.io-thread-count: 64
> cluster.readdir-optimize: on
> performance.write-behind: on
> performance.readdir-ahead: on
> performance.cache-size: 1GB
> performance.cache-invalidation: on
> features.cache-invalidation: on
> diagnostics.brick-log-level: WARNING
> performance.cache-priority: *.php:3,*.temp:3,*:1
> network.inode-lru-limit: 90000
> transport.address-family: inet
> nfs.disable: on
> performance.client-io-threads: off
> performance.parallel-readdir: off
>
> And here are our /etc/fstab entries for the fuse mountpoint:
> localhost:/ssl /etc/ssl/shared glusterfs
backup-volfile-servers=10.0.0.2:10.0.0.3,log-level=WARNING
> 0 0
> localhost:/www /var/www glusterfs
backup-volfile-servers=10.0.0.2:10.0.0.3,log-level=WARNING
> 0 0
>
> Fuse version:
> dpkg -l | grep fuse
> ii  fuse                                 2.9.3-15+deb8u2
>                                  amd64        Filesystem in Userspace
> ii  libfuse2:amd64                       2.9.3-15+deb8u2
>                                  amd64        Filesystem in Userspace
> (library)
>
>
> Thank you!
> Niels Hendriks
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20171124/fd39e71b/attachment.html>