Yuhao Zhang
2018-Aug-05 07:14 UTC
[Gluster-users] Gluster High CPU/Clients Hanging on Heavy Writes
Hi, I am running into a situation that heavy write causes Gluster server went into zombie with many high CPU processes and all clients hangs, it is almost 100% reproducible on my machine. Hope someone can help. I started to observe this issue when running rsync to copy files from another server and I thought it might be because Gluster doesn't like rsync's delta transfer with a lot of small writes. However, I was able to reproduce this with "rsync --whole-file --inplace", or even with cp or scp. It usually appears after starting the transfer for a few hours, but sometimes can happen within several minutes. Since this is a single node Gluster distributed volume, I tried to transfer files directly onto the server bypassing Gluster clients, but it still caused the same issue. It is running on top of a ZFS RAIDZ2 dataset. Options are attached. Also, I attached the statedump generated when my clients hung, and volume options. - Ubuntu 16.04 x86_64 / 4.4.0-116-generic - GlusterFS 3.12.8 Thank you, Yuhao -------------- next part -------------- A non-text attachment was scrubbed... Name: zfs_options Type: application/octet-stream Size: 4220 bytes Desc: not available URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180805/77c0c46e/attachment-0003.obj> -------------- next part -------------- A non-text attachment was scrubbed... Name: volume_options Type: application/octet-stream Size: 14895 bytes Desc: not available URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180805/77c0c46e/attachment-0004.obj> -------------- next part -------------- A non-text attachment was scrubbed... Name: zfs-my_vol_name.6856.dump.1533387154 Type: application/octet-stream Size: 3681718 bytes Desc: not available URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180805/77c0c46e/attachment-0005.obj>
Raghavendra Gowdappa
2018-Aug-05 07:38 UTC
[Gluster-users] Gluster High CPU/Clients Hanging on Heavy Writes
On Sun, Aug 5, 2018 at 12:44 PM, Yuhao Zhang <zzyzxd at gmail.com> wrote:> Hi, > > I am running into a situation that heavy write causes Gluster server went > into zombie with many high CPU processes and all clients hangs, it is > almost 100% reproducible on my machine. Hope someone can help. >Can you give us the output of monitioring these processes with High cpu usage captured in the duration when your tests are running? - MON_INTERVAL=10 # can be increased for very long runs - top -bd $MON_INTERVAL > /tmp/top_proc.${HOSTNAME}.txt # CPU utilization by process - top -bHd $MON_INTERVAL > /tmp/top_thr.${HOSTNAME}.txt # CPU utilization by thread> > I started to observe this issue when running rsync to copy files from > another server and I thought it might be because Gluster doesn't like > rsync's delta transfer with a lot of small writes. However, I was able to > reproduce this with "rsync --whole-file --inplace", or even with cp or scp. > It usually appears after starting the transfer for a few hours, but > sometimes can happen within several minutes. > > Since this is a single node Gluster distributed volume, I tried to > transfer files directly onto the server bypassing Gluster clients, but it > still caused the same issue. > > It is running on top of a ZFS RAIDZ2 dataset. Options are attached. Also, > I attached the statedump generated when my clients hung, and volume options. > > - Ubuntu 16.04 x86_64 / 4.4.0-116-generic > - GlusterFS 3.12.8 > > Thank you, > Yuhao > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180805/cb370904/attachment.html>