Hello, can anybody help with gluster performace ? I'm running gluster replica 3 arbiter 1 ( 2+1) with following version OS Version:RHEL - 7 - 3.1611.el7.centos OS Description:CentOS Linux 7 (Core) Kernel Version:3.10.0 - 514.6.1.el7.x86_64 KVM Version:2.6.0 - 28.el7_3.3.1 LIBVIRT Version:libvirt-2.0.0-10.el7_3.4 VDSM Version:vdsm-4.18.21-1.el7.centos SPICE Version:0.12.4 - 19.el7 GlusterFS Version:glusterfs-3.7.19-1.el7 CEPH Version:librbd1-0.94.5-1.el7 After huge gluster load ( runnig DB + 2x import over 10Gb Lan ) system generate messages to "messages" and dramaticaly goes down, even node rebooted . Jan 28 09:45:19 1kvm2 kernel: INFO: task worker:11384 blocked for more than 120 seconds. Jan 28 09:45:19 1kvm2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 28 09:45:19 1kvm2 kernel: worker D ffff880167566dd0 0 11384 1 0x00000080 Jan 28 09:45:19 1kvm2 kernel: ffff8807ab76fa60 0000000000000086 ffff882fba75de20 ffff8807ab76ffd8 Jan 28 09:45:19 1kvm2 kernel: ffff8807ab76ffd8 ffff8807ab76ffd8 ffff882fba75de20 ffff882fbf396c40 Jan 28 09:45:19 1kvm2 kernel: 0000000000000000 7fffffffffffffff ffff882fa8ee2500 ffff8807ab76fd60 Jan 28 09:45:19 1kvm2 kernel: Call Trace: Jan 28 09:45:19 1kvm2 kernel: [<ffffffff8168b899>] schedule+0x29/0x70 Jan 28 09:45:19 1kvm2 kernel: [<ffffffff816892e9>] schedule_timeout+0x239/0x2d0 Jan 28 09:45:19 1kvm2 kernel: [<ffffffffa0889ea2>] ? fuse_direct_io+0x492/0x820 [fuse] Jan 28 09:45:19 1kvm2 kernel: [<ffffffff8168ae3e>] io_schedule_timeout+0xae/0x130 Jan 28 09:45:19 1kvm2 kernel: [<ffffffff8168aed8>] io_schedule+0x18/0x20 Jan 28 09:45:19 1kvm2 kernel: [<ffffffff8124d945>] wait_on_sync_kiocb+0x35/0x80 Jan 28 09:45:19 1kvm2 kernel: [<ffffffffa088b091>] fuse_direct_IO+0x231/0x380 [fuse] Jan 28 09:45:19 1kvm2 kernel: [<ffffffffa088d1f8>] ? fuse_change_attributes_common+0x88/0x120 [fuse] Jan 28 09:45:19 1kvm2 kernel: [<ffffffffa088d32f>] ? fuse_change_attributes+0x9f/0x100 [fuse] Jan 28 09:45:19 1kvm2 kernel: [<ffffffff8118283c>] generic_file_aio_read+0x70c/0x790 Jan 28 09:45:19 1kvm2 kernel: [<ffffffffa0887d78>] fuse_file_aio_read+0x78/0xb0 [fuse] Jan 28 09:45:19 1kvm2 kernel: [<ffffffff811fdad9>] do_sync_readv_writev+0x79/0xd0 Jan 28 09:45:19 1kvm2 kernel: [<ffffffff811ff17e>] do_readv_writev+0xce/0x260 Jan 28 09:45:19 1kvm2 kernel: [<ffffffff810f7bae>] ? do_futex+0xfe/0x5b0 Jan 28 09:45:19 1kvm2 kernel: [<ffffffff811ff342>] vfs_readv+0x32/0x60 Jan 28 09:45:19 1kvm2 kernel: [<ffffffff811ff6b2>] SyS_preadv+0xc2/0xf0 Jan 28 09:45:19 1kvm2 kernel: [<ffffffff816967c9>] system_call_fastpath+0x16/0x1b Jan 28 09:50:01 1kvm2 systemd: Started Session 58 of user root. Jan 28 09:50:01 1kvm2 systemd: Starting Session 58 of user root. I tried to set kernel params via web workaround recommendations, but without any success ( current /etc/sysctl.conf settings ) vm.swappiness=1 vm.vfs_cache_pressure=100 vm.min_free_kbytes = 512000 vm.zone_reclaim_mode = 1 Any idea ?? regs. Paf1 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170128/c26a5f28/attachment.html>