Gluster users - Jan 2014 - [Users] Creation of preallocated disk with Gluster replication

Adding gluster-users.

On 01/02/2014 08:50 PM, gregoire.leroy at retenodus.net
wrote:
> Hello,
>
> I have a Gluster volume in distributed/replicated mode. I have 2 hosts.
> When I try to create a VM with a preallocated disk, it uses 100% of the
> available CPU and bandwidth (I have 1 Gigabit network card).
> The result is I can't even create a preallocated disk because the
engine
> detects a network failure.
>
> I get that kind of messages in /var/log/messages :
> "
> Jan  2 14:13:54 localhost sanlock[3811]: 2014-01-02 14:13:54+0100 167737
> [3811]: s4 kill 21114 sig 15 count 1
> Jan  2 14:13:54 localhost wdmd[3800]: test failed rem 51 now 167737 ping
> 167718 close 167728 renewal 167657 expire 167737 client 3811
> sanlock_ef4978d6-5711-4e01-a0ec-7ffbd9     cdbe5d:1
> "
>
> And that in the Ovirt Gui :
> "
> 2014-janv.-02, 15:35 Operation Add-Disk failed to complete.
> 2014-janv.-02, 15:35 Storage Pool Manager runs on Host HOST2 (Address:
> X.X.X.X).
> 2014-janv.-02, 15:35 Invalid status on Data Center GlusterSewan. Setting
> Data Center status to Non Responsive (On host HOST2, Error: done).
> 2014-janv.-02, 15:35 State was set to Up for host HOST2.
> 2014-janv.-02, 15:33 Used Network resources of host HOST2 [98%] exceeded
> defined threshold [95%].
> 2014-janv.-02, 15:33 Add-Disk operation of test_Disk1 was initiated on
> VM test by admin at internal.
>
> I understand that the creation of a 10 Go disk image generates a lot of
> traffic, but is there a way to limit it so that it doesn't have an
> impact on the production ? Furthermore, Why does it use so much CPU
> ressources ? I can see on my monitoring graph a big peak of CPU usage
> when I launched the operation (probably until 100%).
Do you happen to notice what is consuming CPU? Since the same cluster 
does both virtualization and storage, a GigE network might get saturated 
very quickly. Is it possible to separate out the management and 
data/gluster traffic in this setup?

Regards,
Vijay

> Do you happen to notice what is consuming CPU?
When I check with top, glusterfs and glusterfsd are the only process who 
use a significant amount of CPU. Load average is between 5 and 6, and I 
don't have any started VM.
> Since the same cluster
> does both virtualization and storage, a GigE network might get
> saturated very quickly. Is it possible to separate out the management
> and data/gluster traffic in this setup?
Unfortunately, it's not possible. I only have two hosts, both for 
virtualization and storage. Why does glusterfs use so much CPU ?

About the network traffic, the creation of a pre-allocated disk is not 
something which absolutely needs to be fast : if it takes 5 minutes, 
well... it takes 5 minutes. I guess that if there's a way to limit the 
bandwidth available for glusterfs, it'll still have an impact on 
production because glusterfs and VM will be in competition for 
bandwidth, right ? Is there a way to limit the bandwidth only for the 
creation of a pre-allocated disk ?

Thank you,
Regards,
Gr?goire Leroy

Hello,

Do you need more informations about this issue ? Do you think this 
problem is likely to show up in other cases ? I mean, is that an 
expected behaviour with my environment, or is it unexpected ?

Is there a way to limit the bandwidth usage for creation of 
pre-allocated disk so that it doesn't impact production ?

Thank you,
Regards,
Gr?goire

Le 2014-01-02 17:42, Vijay Bellur a ?crit?:
> Adding gluster-users.
> 
> On 01/02/2014 08:50 PM, gregoire.leroy at retenodus.net wrote:
>> Hello,
>> 
>> I have a Gluster volume in distributed/replicated mode. I have 2 
>> hosts.
>> When I try to create a VM with a preallocated disk, it uses 100% of 
>> the
>> available CPU and bandwidth (I have 1 Gigabit network card).
>> The result is I can't even create a preallocated disk because the 
>> engine
>> detects a network failure.
>> 
>> I get that kind of messages in /var/log/messages :
>> "
>> Jan  2 14:13:54 localhost sanlock[3811]: 2014-01-02 14:13:54+0100 
>> 167737
>> [3811]: s4 kill 21114 sig 15 count 1
>> Jan  2 14:13:54 localhost wdmd[3800]: test failed rem 51 now 167737 
>> ping
>> 167718 close 167728 renewal 167657 expire 167737 client 3811
>> sanlock_ef4978d6-5711-4e01-a0ec-7ffbd9     cdbe5d:1
>> "
>> 
>> And that in the Ovirt Gui :
>> "
>> 2014-janv.-02, 15:35 Operation Add-Disk failed to complete.
>> 2014-janv.-02, 15:35 Storage Pool Manager runs on Host HOST2 (Address:
>> X.X.X.X).
>> 2014-janv.-02, 15:35 Invalid status on Data Center GlusterSewan. 
>> Setting
>> Data Center status to Non Responsive (On host HOST2, Error: done).
>> 2014-janv.-02, 15:35 State was set to Up for host HOST2.
>> 2014-janv.-02, 15:33 Used Network resources of host HOST2 [98%] 
>> exceeded
>> defined threshold [95%].
>> 2014-janv.-02, 15:33 Add-Disk operation of test_Disk1 was initiated on
>> VM test by admin at internal.
>> 
>> I understand that the creation of a 10 Go disk image generates a lot 
>> of
>> traffic, but is there a way to limit it so that it doesn't have an
>> impact on the production ? Furthermore, Why does it use so much CPU
>> ressources ? I can see on my monitoring graph a big peak of CPU usage
>> when I launched the operation (probably until 100%).
> 
> Do you happen to notice what is consuming CPU? Since the same cluster
> does both virtualization and storage, a GigE network might get
> saturated very quickly. Is it possible to separate out the management
> and data/gluster traffic in this setup?
> 
> Regards,
> Vijay