Gluster users - Mar 2019 - Lots of connections on clients - appropriate values for various thread parameters

what is the per thread CPU usage like on these clients? With highly
concurrent workloads we've seen single thread that reads requests from
/dev/fuse (fuse reader thread) becoming bottleneck. Would like to know what
is the cpu usage of this thread looks like (you can use top -H).

On Mon, Mar 4, 2019 at 3:39 PM Hu Bert <revirii at googlemail.com> wrote:
> Good morning,
>
> we use gluster v5.3 (replicate with 3 servers, 2 volumes, raid10 as
> brick) with at the moment 10 clients; 3 of them do heavy I/O
> operations (apache tomcats, read+write of (small) images). These 3
> clients have a quite high I/O wait (stats from yesterday) as can be
> seen here:
>
> client: https://abload.de/img/client1-cpu-dayulkza.png
> server: https://abload.de/img/server1-cpu-dayayjdq.png
>
> The iowait in the graphics differ a lot. I checked netstat for the
> different clients; the other clients have 8 open connections:
> https://pastebin.com/bSN5fXwc
>
> 4 for each server and each volume. The 3 clients with the heavy I/O
> have (at the moment) according to netstat 170, 139 and 153
> connections. An example for one client can be found here:
> https://pastebin.com/2zfWXASZ
>
> gluster volume info: https://pastebin.com/13LXPhmd
> gluster volume status: https://pastebin.com/cYFnWjUJ
>
> I just was wondering if the iowait is based on the clients and their
> workflow: requesting a lot of files (up to hundreds per second),
> opening a lot of connections and the servers aren't able to answer
> properly. Maybe something can be tuned here?
>
> Especially the server|client.event-threads (both set to 4) and
> performance.(high|normal|low|least)-prio-threads (all at default value
> 16) and performance.io-thread-count (32) options, maybe these aren't
> properly configured for up to 170 client connections.
>
> Both servers and clients have a Xeon CPU (6 cores, 12 threads), a 10
> GBit connection and 128G (servers) respectively 256G (clients) RAM.
> Enough power :-)
>
>
> Thx for reading && best regards,
>
> Hubert
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20190304/09463b3d/attachment.html>

Hi Raghavendra,

at the moment iowait and cpu consumption is quite low, the main
problems appear during the weekend (high traffic, especially on
sunday), so either we have to wait until next sunday or use a time
machine ;-)

I made a screenshot of top (https://abload.de/img/top-hvvjt2.jpg) and
a text output (https://pastebin.com/TkTWnqxt), maybe that helps. Seems
like processes like glfs_fuseproc (>204h) and glfs_epoll (64h for each
process) consume a lot of CPU (uptime 24 days). Is that already
helpful?


Hubert

Am Mo., 4. M?rz 2019 um 11:31 Uhr schrieb Raghavendra Gowdappa
<rgowdapp at redhat.com>:
>
> what is the per thread CPU usage like on these clients? With highly
concurrent workloads we've seen single thread that reads requests from
/dev/fuse (fuse reader thread) becoming bottleneck. Would like to know what is
the cpu usage of this thread looks like (you can use top -H).
>
> On Mon, Mar 4, 2019 at 3:39 PM Hu Bert <revirii at googlemail.com>
wrote:
>>
>> Good morning,
>>
>> we use gluster v5.3 (replicate with 3 servers, 2 volumes, raid10 as
>> brick) with at the moment 10 clients; 3 of them do heavy I/O
>> operations (apache tomcats, read+write of (small) images). These 3
>> clients have a quite high I/O wait (stats from yesterday) as can be
>> seen here:
>>
>> client: https://abload.de/img/client1-cpu-dayulkza.png
>> server: https://abload.de/img/server1-cpu-dayayjdq.png
>>
>> The iowait in the graphics differ a lot. I checked netstat for the
>> different clients; the other clients have 8 open connections:
>> https://pastebin.com/bSN5fXwc
>>
>> 4 for each server and each volume. The 3 clients with the heavy I/O
>> have (at the moment) according to netstat 170, 139 and 153
>> connections. An example for one client can be found here:
>> https://pastebin.com/2zfWXASZ
>>
>> gluster volume info: https://pastebin.com/13LXPhmd
>> gluster volume status: https://pastebin.com/cYFnWjUJ
>>
>> I just was wondering if the iowait is based on the clients and their
>> workflow: requesting a lot of files (up to hundreds per second),
>> opening a lot of connections and the servers aren't able to answer
>> properly. Maybe something can be tuned here?
>>
>> Especially the server|client.event-threads (both set to 4) and
>> performance.(high|normal|low|least)-prio-threads (all at default value
>> 16) and performance.io-thread-count (32) options, maybe these
aren't
>> properly configured for up to 170 client connections.
>>
>> Both servers and clients have a Xeon CPU (6 cores, 12 threads), a 10
>> GBit connection and 128G (servers) respectively 256G (clients) RAM.
>> Enough power :-)
>>
>>
>> Thx for reading && best regards,
>>
>> Hubert
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users