Gluster users - Jan 2018 - Problem with Gluster 3.12.4, VM and sharding

Hi to everyone.

I've got a strange problem with a gluster setup: 3 nodes with Centos 
7.4, Gluster 3.12.4 from Centos/Gluster repositories, QEMU-KVM version 
2.9.0 (compiled from RHEL sources).

I'm running volumes in replica 3 arbiter 1 mode (but I've got a volume 
in "pure" replica 3 mode too). I've applied the "virt"
group settings to
my volumes since they host VM images.

If I try to install something (eg: Ubuntu Server 16.04.3) on a VM (and 
so I generate a bit of I/O inside it) and configure KVM to access 
gluster volume directly (via libvirt), install fails after a while 
because the disk content is corrupted. If I inspect the block inside the 
disk (by accessing the image directly from outside) I can found many 
files filled with "^@".

If, instead, I configure KVM to access VM images via a FUSE mount, 
everything seems to work correctly.

Note that the problem with install is verified 100% time with QCOW2 
image, while it appears only after with RAW disk images.

Is there anyone who experienced the same problem ?

Thank you,


-- 
Ing. Luca Lazzeroni
Responsabile Ricerca e Sviluppo
Trend Servizi Srl
Tel: 0376/631761
Web: https://www.trendservizi.it

Please share the volume-info output and the logs under /var/log/glusterfs/
from all your nodes. for investigating the issue.

-Krutika

On Tue, Jan 16, 2018 at 1:30 PM, Ing. Luca Lazzeroni - Trend Servizi Srl <
luca at gvnet.it> wrote:
> Hi to everyone.
>
> I've got a strange problem with a gluster setup: 3 nodes with Centos
7.4,
> Gluster 3.12.4 from Centos/Gluster repositories, QEMU-KVM version 2.9.0
> (compiled from RHEL sources).
>
> I'm running volumes in replica 3 arbiter 1 mode (but I've got a
volume in
> "pure" replica 3 mode too). I've applied the "virt"
group settings to my
> volumes since they host VM images.
>
> If I try to install something (eg: Ubuntu Server 16.04.3) on a VM (and so
> I generate a bit of I/O inside it) and configure KVM to access gluster
> volume directly (via libvirt), install fails after a while because the disk
> content is corrupted. If I inspect the block inside the disk (by accessing
> the image directly from outside) I can found many files filled with
"^@".
>
> If, instead, I configure KVM to access VM images via a FUSE mount,
> everything seems to work correctly.
>
> Note that the problem with install is verified 100% time with QCOW2 image,
> while it appears only after with RAW disk images.
>
> Is there anyone who experienced the same problem ?
>
> Thank you,
>
>
> --
> Ing. Luca Lazzeroni
> Responsabile Ricerca e Sviluppo
> Trend Servizi Srl
> Tel: 0376/631761
> Web: https://www.trendservizi.it
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180116/3c50fc57/attachment.html>

Also to help isolate the component, could you answer these:

1. on a different volume with shard not enabled, do you see this issue?
2. on a plain 3-way replicated volume (no arbiter), do you see this issue?



On Tue, Jan 16, 2018 at 4:03 PM, Krutika Dhananjay <kdhananj at
redhat.com>
wrote:
> Please share the volume-info output and the logs under /var/log/glusterfs/
> from all your nodes. for investigating the issue.
>
> -Krutika
>
> On Tue, Jan 16, 2018 at 1:30 PM, Ing. Luca Lazzeroni - Trend Servizi Srl
<
> luca at gvnet.it> wrote:
>
>> Hi to everyone.
>>
>> I've got a strange problem with a gluster setup: 3 nodes with
Centos 7.4,
>> Gluster 3.12.4 from Centos/Gluster repositories, QEMU-KVM version 2.9.0
>> (compiled from RHEL sources).
>>
>> I'm running volumes in replica 3 arbiter 1 mode (but I've got a
volume in
>> "pure" replica 3 mode too). I've applied the
"virt" group settings to my
>> volumes since they host VM images.
>>
>> If I try to install something (eg: Ubuntu Server 16.04.3) on a VM (and
so
>> I generate a bit of I/O inside it) and configure KVM to access gluster
>> volume directly (via libvirt), install fails after a while because the
disk
>> content is corrupted. If I inspect the block inside the disk (by
accessing
>> the image directly from outside) I can found many files filled with
"^@".
>>
>
Also, what exactly do you mean by accessing the image directly from
outside? Was it from the brick directories directly? Was it from the mount
point of the volume? Could you elaborate? Which files exactly did you check?

-Krutika

>> If, instead, I configure KVM to access VM images via a FUSE mount,
>> everything seems to work correctly.
>>
>> Note that the problem with install is verified 100% time with QCOW2
>> image, while it appears only after with RAW disk images.
>>
>> Is there anyone who experienced the same problem ?
>>
>> Thank you,
>>
>>
>> --
>> Ing. Luca Lazzeroni
>> Responsabile Ricerca e Sviluppo
>> Trend Servizi Srl
>> Tel: 0376/631761
>> Web: https://www.trendservizi.it
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180116/c15f998d/attachment.html>