Hi guys, My glusterfs is causing 100% CPU WAIT according to `top`. This has been going on for hours and I have no idea what is causing it. How can I troubleshoot? Iotop reports this: Total DISK READ: 268.60 K/s | Total DISK WRITE: 0.00 B/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND 7899 be/4 root 268.60 K/s 0.00 B/s 0.00 % 96.70 % glusterfsd -s server01 --volfile-id myvol.server01.glusterfs-brick1 -p /var/lib/glusterd/vols/myvol/run/server01-glusterfs-brick1.pid -S /var/run/a7562806405853d2b9382d6fc59051cc.socket --brick-name /glusterfs/brick1 -l /var/log/glusterfs/bricks/glusterfs-brick1.log --xlator-option *-posix.glusterd-uuid=07acd5b2-85e6-46f1-8477-038028e8ef7f --brick-port 49152 --xlator-option myvol-server.listen-port=49152 1885 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.98 % glusterfsd -s server01 --volfile-id myvol.server01.glusterfs-brick1 -p /var/lib/glusterd/vols/myvol/run/server01-glusterfs-brick1.pid -S /var/run/a7562806405853d2b9382d6fc59051cc.socket --brick-name /glusterfs/brick1 -l /var/log/glusterfs/bricks/glusterfs-brick1.log --xlator-option *-posix.glusterd-uuid=07acd5b2-85e6-46f1-8477-038028e8ef7f --brick-port 49152 --xlator-option myvol-server.listen-port=49152 Kind regards, Tom van Leeuwen
Hi Tom, We experience this behavior when a self heal is running after a bad communication between 2 nodes, or after a node crashed. How we diagnose it is usually by looking into the mount log (tail -f /var/log/gluster/mnt-log), and you should see entries such as afr ... self-heal -----Original Message----- From: gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at gluster.org] On Behalf Of Tom van Leeuwen Sent: 3 octobre 2014 06:00 To: gluster-users at gluster.org Subject: [Gluster-users] 100% CPU WAIT Hi guys, My glusterfs is causing 100% CPU WAIT according to `top`. This has been going on for hours and I have no idea what is causing it. How can I troubleshoot? Iotop reports this: Total DISK READ: 268.60 K/s | Total DISK WRITE: 0.00 B/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND 7899 be/4 root 268.60 K/s 0.00 B/s 0.00 % 96.70 % glusterfsd -s server01 --volfile-id myvol.server01.glusterfs-brick1 -p /var/lib/glusterd/vols/myvol/run/server01-glusterfs-brick1.pid -S /var/run/a7562806405853d2b9382d6fc59051cc.socket --brick-name /glusterfs/brick1 -l /var/log/glusterfs/bricks/glusterfs-brick1.log --xlator-option *-posix.glusterd-uuid=07acd5b2-85e6-46f1-8477-038028e8ef7f --brick-port 49152 --xlator-option myvol-server.listen-port=49152 1885 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.98 % glusterfsd -s server01 --volfile-id myvol.server01.glusterfs-brick1 -p /var/lib/glusterd/vols/myvol/run/server01-glusterfs-brick1.pid -S /var/run/a7562806405853d2b9382d6fc59051cc.socket --brick-name /glusterfs/brick1 -l /var/log/glusterfs/bricks/glusterfs-brick1.log --xlator-option *-posix.glusterd-uuid=07acd5b2-85e6-46f1-8477-038028e8ef7f --brick-port 49152 --xlator-option myvol-server.listen-port=49152 Kind regards, Tom van Leeuwen _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
On 10/03/2014 03:30 PM, Tom van Leeuwen wrote:> Hi guys, > > My glusterfs is causing 100% CPU WAIT according to `top`. > This has been going on for hours and I have no idea what is causing > it. How can I troubleshoot? > > Iotop reports this: > Total DISK READ: 268.60 K/s | Total DISK WRITE: 0.00 B/s > TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND > 7899 be/4 root 268.60 K/s 0.00 B/s 0.00 % 96.70 % glusterfsd > -s server01 --volfile-id myvol.server01.glusterfs-brick1 -p > /var/lib/glusterd/vols/myvol/run/server01-glusterfs-brick1.pid -S > /var/run/a7562806405853d2b9382d6fc59051cc.socket --brick-name > /glusterfs/brick1 -l /var/log/glusterfs/bricks/glusterfs-brick1.log > --xlator-option > *-posix.glusterd-uuid=07acd5b2-85e6-46f1-8477-038028e8ef7f > --brick-port 49152 --xlator-option myvol-server.listen-port=49152 > 1885 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.98 % glusterfsd > -s server01 --volfile-id myvol.server01.glusterfs-brick1 -p > /var/lib/glusterd/vols/myvol/run/server01-glusterfs-brick1.pid -S > /var/run/a7562806405853d2b9382d6fc59051cc.socket --brick-name > /glusterfs/brick1 -l /var/log/glusterfs/bricks/glusterfs-brick1.log > --xlator-option > *-posix.glusterd-uuid=07acd5b2-85e6-46f1-8477-038028e8ef7f > --brick-port 49152 --xlator-option myvol-server.listen-port=49152Could you provide us with the logs please. Pranith> > Kind regards, > Tom van Leeuwen > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-users