Gluster users - Feb 2019 - glusterfsd Ubuntu 18.04 high iowait issues

We're running gluster on two hypervisors running Ubuntu. When we 
upgraded from Ubuntu 14.04 to 18.04, it upgraded gluster from 3.4.2 to 
3.13.2. As soon as we upgraded and since then, we've been seeing 
substantially higher iowait on the system, as measured by top and iotop, 
and iotop indicates that glusterfsd is the culprit. For some reason, 
glusterfsd is doing more disk reads and/or those reads are being held up 
up at a greater rate. The guest VMs are also seeing more iowait -- their 
images are hosted on the gluster volume. This is causing inconsistent 
responsiveness from the services hosted on the VMs.

I'm looking for any recommendations on how to troubleshoot and/or 
resolve this problem. We have other sites that are still running 14.04, 
so I can compare/contrast any configuration parameters and performance.

The block scheduler on 14.04 was set to deadline and 18.04 was set to 
cfq. But changing the 18.04 scheduler to deadline didn't make any 
difference.

I was wondering whether glusterfsd on 18.04 isn't caching as much as it 
should. We tried increasing performance.cache-size substantially but 
that didn't make any difference.

Another option we're considering but haven't tried yet is upgrading to 
gluster 5.3 by back-porting the package from Ubuntu 19.04 to 18.04. Does 
anyone think this might help?

Is there any particular debug logging we could set up or other commands 
we could run to troubleshoot this better? Any thoughts, suggestions, 
ideas would be greatly appreciated.

Thanks,

 ??? -Kartik

If you have both systems to get some idea, can you get the `gluster profile
info' output? That helps a bit to understand the issue.


On Thu, Feb 21, 2019 at 8:20 AM Kartik Subbarao <subbarao at computer.org>
wrote:
> We're running gluster on two hypervisors running Ubuntu. When we
> upgraded from Ubuntu 14.04 to 18.04, it upgraded gluster from 3.4.2 to
> 3.13.2. As soon as we upgraded and since then, we've been seeing
> substantially higher iowait on the system, as measured by top and iotop,
> and iotop indicates that glusterfsd is the culprit. For some reason,
> glusterfsd is doing more disk reads and/or those reads are being held up
> up at a greater rate. The guest VMs are also seeing more iowait -- their
> images are hosted on the gluster volume. This is causing inconsistent
> responsiveness from the services hosted on the VMs.
>
> I'm looking for any recommendations on how to troubleshoot and/or
> resolve this problem. We have other sites that are still running 14.04,
> so I can compare/contrast any configuration parameters and performance.
>
> The block scheduler on 14.04 was set to deadline and 18.04 was set to
> cfq. But changing the 18.04 scheduler to deadline didn't make any
> difference.
>
> I was wondering whether glusterfsd on 18.04 isn't caching as much as it
> should. We tried increasing performance.cache-size substantially but
> that didn't make any difference.
>
> Another option we're considering but haven't tried yet is upgrading
to
> gluster 5.3 by back-porting the package from Ubuntu 19.04 to 18.04. Does
> anyone think this might help?
>
> Is there any particular debug logging we could set up or other commands
> we could run to troubleshoot this better? Any thoughts, suggestions,
> ideas would be greatly appreciated.
>
> Thanks,
>
>      -Kartik
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users


-- 
Amar Tumballi (amarts)
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20190221/0fcfa607/attachment.html>