Gluster users - Feb 2014 - Possible to put some sort of qos/rate-limit on glusterfsd?

Hi all,

Sadly my google skills aren't finding me any results - is there an option
to limit the CPU usage and/or the disk IO intensity of glusterfsd.

Example scenario, oVirt + gluster on the same host when it comes to adding
an extra host + replicated brick the original host with the brick goes
crazy with 500% cpu as it copies just under 1TB of data across to the new
replicated brick. Going crazy I mean everything else will hang, simple
"ls"
command will take 30+ seconds.

Limiting the network bandwidth to 200Mbps seems to solve this issue, I'm
quite sure this is a CPU issue rather than IO so I was wondering if there's
any possibility to limit this down so the nic's itself don't have to get
rate limited.

Thanks,
Andrew
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140202/b9a01999/attachment.html>

On Sun, Feb 2, 2014 at 12:46 AM, Andrew Lau <andrew at andrewklau.com>
wrote:
> Hi all,
Hi,
>
> Sadly my google skills aren't finding me any results - is there an
option to
> limit the CPU usage and/or the disk IO intensity of glusterfsd.
This is what cgroups are for. Looking forward to you setting this up
and telling us how it goes and what the optimized values are. AFAIK,
gluster doesn't have anything built in to limit this. I actually don't
think that it makes sense for it to either.
>
> Example scenario, oVirt + gluster on the same host when it comes to adding
> an extra host + replicated brick the original host with the brick goes
crazy
> with 500% cpu as it copies just under 1TB of data across to the new
> replicated brick. Going crazy I mean everything else will hang, simple
"ls"
> command will take 30+ seconds.
Not sure what the correct solution is here. Maybe other gluster dev's
can comment. Although it's really not detailed and you comment is
unclear, so i'd elaborate before expecting a good response.
>
> Limiting the network bandwidth to 200Mbps seems to solve this issue,
I'm
> quite sure this is a CPU issue rather than IO so I was wondering if
there's
> any possibility to limit this down so the nic's itself don't have
to get
> rate limited.
I occasionally use shorewall+puppet-gluster to rate limit the network
interfaces. Puppet-Gluster could do per interface and per port/gluster
connection throttling.
>
> Thanks,
> Andrew
HTH,
James

Try experimenting with performance.io-thread-count to see if that has an impact.

-Dan
----------------
Dan Mons
Skunk Works
Cutting Edge
http://cuttingedge.com.au


On 2 February 2014 15:46, Andrew Lau <andrew at andrewklau.com>
wrote:
> Hi all,
>
> Sadly my google skills aren't finding me any results - is there an
option to
> limit the CPU usage and/or the disk IO intensity of glusterfsd.
>
> Example scenario, oVirt + gluster on the same host when it comes to adding
> an extra host + replicated brick the original host with the brick goes
crazy
> with 500% cpu as it copies just under 1TB of data across to the new
> replicated brick. Going crazy I mean everything else will hang, simple
"ls"
> command will take 30+ seconds.
>
> Limiting the network bandwidth to 200Mbps seems to solve this issue,
I'm
> quite sure this is a CPU issue rather than IO so I was wondering if
there's
> any possibility to limit this down so the nic's itself don't have
to get
> rate limited.
>
> Thanks,
> Andrew
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users