thr3ads.net - Gluster users - [Gluster-users] [Possibile SPAM] Re: Problem with Gluster 3.12.4, VM and sharding [Jan 2018]

If this information is useful, please help other people find it:
Share via:

Krutika Dhananjay

2018-Jan-17 10:00 UTC

[Gluster-users] [Possibile SPAM] Re: Problem with Gluster 3.12.4, VM and sharding

On Tue, Jan 16, 2018 at 10:47 PM, Ing. Luca Lazzeroni - Trend Servizi Srl <
luca at gvnet.it> wrote:
> I've made the test with raw image format (preallocated too) and the
> corruption problem is still there (but without errors in bricks' log
file).
>
> What does the "link" error in bricks log files means ?
>
> I've seen the source code looking for the lines where it happens and it
> seems a warning (it doesn't imply a failure).
>
Indeed, it only represents a transient state when the shards are created
for the first time and does not indicate a failure.
Could you also get the logs of the gluster fuse mount process? It should be
under /var/log/glusterfs of your client machine with the filename as a
hyphenated mount point path.

For example, if your volume was mounted at /mnt/glusterfs, then your log
file would be named mnt-glusterfs.log.

-Krutika

>
> Il 16/01/2018 17:39, Ing. Luca Lazzeroni - Trend Servizi Srl ha scritto:
>
> An update:
>
> I've tried, for my tests, to create the vm volume as
>
> qemu-img create -f qcow2 -o preallocation=full
> gluster://gluster1/Test/Test-vda.img 20G
>
> et voila !
>
> No errors at all, neither in bricks' log file (the "link
failed" message
> disappeared), neither in VM (no corruption and installed succesfully).
>
> I'll do another test with a fully preallocated raw image.
>
>
>
> Il 16/01/2018 16:31, Ing. Luca Lazzeroni - Trend Servizi Srl ha scritto:
>
> I've just done all the steps to reproduce the problem.
>
> Tha VM volume has been created via "qemu-img create -f qcow2
> Test-vda2.qcow2 20G" on the gluster volume mounted via FUSE. I've
tried
> also to create the volume with preallocated metadata, which moves the
> problem a bit far away (in time). The volume is a replice 3 arbiter 1
> volume hosted on XFS bricks.
>
> Here are the informations:
>
> [root at ovh-ov1 bricks]# gluster volume info gv2a2
>
> Volume Name: gv2a2
> Type: Replicate
> Volume ID: 83c84774-2068-4bfc-b0b9-3e6b93705b9f
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x (2 + 1) = 3
> Transport-type: tcp
> Bricks:
> Brick1: gluster1:/bricks/brick2/gv2a2
> Brick2: gluster3:/bricks/brick3/gv2a2
> Brick3: gluster2:/bricks/arbiter_brick_gv2a2/gv2a2 (arbiter)
> Options Reconfigured:
> storage.owner-gid: 107
> storage.owner-uid: 107
> user.cifs: off
> features.shard: on
> cluster.shd-wait-qlength: 10000
> cluster.shd-max-threads: 8
> cluster.locking-scheme: granular
> cluster.data-self-heal-algorithm: full
> cluster.server-quorum-type: server
> cluster.quorum-type: auto
> cluster.eager-lock: enable
> network.remote-dio: enable
> performance.low-prio-threads: 32
> performance.io-cache: off
> performance.read-ahead: off
> performance.quick-read: off
> transport.address-family: inet
> nfs.disable: off
> performance.client-io-threads: off
>
> /var/log/glusterfs/glusterd.log:
>
> [2018-01-15 14:17:50.196228] I [MSGID: 106488] [glusterd-handler.c:1548:__
> glusterd_handle_cli_get_volume] 0-management: Received get vol req
> [2018-01-15 14:25:09.555214] I [MSGID: 106488] [glusterd-handler.c:1548:__
> glusterd_handle_cli_get_volume] 0-management: Received get vol req
>
> (empty because today it's 2018-01-16)
>
> /var/log/glusterfs/glustershd.log:
>
> [2018-01-14 02:23:02.731245] I [glusterfsd-mgmt.c:1821:mgmt_getspec_cbk]
> 0-glusterfs: No change in volfile,continuing
>
> (empty too)
>
> /var/log/glusterfs/bricks/brick-brick2-gv2a2.log (the interested volume):
>
> [2018-01-16 15:14:37.809965] I [MSGID: 115029]
[server-handshake.c:793:server_setvolume]
> 0-gv2a2-server: accepted client from ovh-ov1-10302-2018/01/16-15:
> 14:37:790306-gv2a2-client-0-0-0 (version: 3.12.4)
> [2018-01-16 15:16:41.471751] E [MSGID: 113020] [posix.c:1485:posix_mknod]
> 0-gv2a2-posix: setting gfid on /bricks/brick2/gv2a2/.shard/
> 62335cb9-c7b5-4735-a879-59cff93fe622.4 failed
> [2018-01-16 15:16:41.471745] W [MSGID: 113096]
[posix-handle.c:770:posix_handle_hard]
> 0-gv2a2-posix: link
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.4
> -> 
/bricks/brick2/gv2a2/.glusterfs/a0/14/a0144df3-8d89-4aed-872e-5fef141e9e1efailed
> [File exists]
> [2018-01-16 15:16:42.593392] W [MSGID: 113096]
[posix-handle.c:770:posix_handle_hard]
> 0-gv2a2-posix: link
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.5
> ->
/bricks/brick2/gv2a2/.glusterfs/eb/04/eb044e6e-3a23-40a4-9ce1-f13af148eb67failed
> [File exists]
> [2018-01-16 15:16:42.593426] E [MSGID: 113020] [posix.c:1485:posix_mknod]
> 0-gv2a2-posix: setting gfid on /bricks/brick2/gv2a2/.shard/
> 62335cb9-c7b5-4735-a879-59cff93fe622.5 failed
> [2018-01-16 15:17:04.129593] W [MSGID: 113096]
[posix-handle.c:770:posix_handle_hard]
> 0-gv2a2-posix: link
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.8
> ->
/bricks/brick2/gv2a2/.glusterfs/dc/92/dc92bd0a-0d46-4826-a4c9-d073a924dd8dfailed
> [File exists]
> The message "W [MSGID: 113096] [posix-handle.c:770:posix_handle_hard]
> 0-gv2a2-posix: link
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.8
> ->
/bricks/brick2/gv2a2/.glusterfs/dc/92/dc92bd0a-0d46-4826-a4c9-d073a924dd8dfailed
> [File exists]" repeated 5 times between [2018-01-16 15:17:04.129593]
and
> [2018-01-16 15:17:04.129593]
> [2018-01-16 15:17:04.129661] E [MSGID: 113020] [posix.c:1485:posix_mknod]
> 0-gv2a2-posix: setting gfid on /bricks/brick2/gv2a2/.shard/
> 62335cb9-c7b5-4735-a879-59cff93fe622.8 failed
> [2018-01-16 15:17:08.279162] W [MSGID: 113096]
[posix-handle.c:770:posix_handle_hard]
> 0-gv2a2-posix: link
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.9
> ->
/bricks/brick2/gv2a2/.glusterfs/c9/b7/c9b71b00-a09f-4df1-b874-041820ca8241failed
> [File exists]
> [2018-01-16 15:17:08.279162] W [MSGID: 113096]
[posix-handle.c:770:posix_handle_hard]
> 0-gv2a2-posix: link
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.9
> ->
/bricks/brick2/gv2a2/.glusterfs/c9/b7/c9b71b00-a09f-4df1-b874-041820ca8241failed
> [File exists]
> The message "W [MSGID: 113096] [posix-handle.c:770:posix_handle_hard]
> 0-gv2a2-posix: link
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.9
> ->
/bricks/brick2/gv2a2/.glusterfs/c9/b7/c9b71b00-a09f-4df1-b874-041820ca8241failed
> [File exists]" repeated 2 times between [2018-01-16 15:17:08.279162]
and
> [2018-01-16 15:17:08.279162]
>
> [2018-01-16 15:17:08.279177] E [MSGID: 113020] [posix.c:1485:posix_mknod]
> 0-gv2a2-posix: setting gfid on /bricks/brick2/gv2a2/.shard/
> 62335cb9-c7b5-4735-a879-59cff93fe622.9 failed
> The message "W [MSGID: 113096] [posix-handle.c:770:posix_handle_hard]
> 0-gv2a2-posix: link
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.4
> ->
/bricks/brick2/gv2a2/.glusterfs/a0/14/a0144df3-8d89-4aed-872e-5fef141e9e1efailed
> [File exists]" repeated 6 times between [2018-01-16 15:16:41.471745]
and
> [2018-01-16 15:16:41.471807]
> The message "W [MSGID: 113096] [posix-handle.c:770:posix_handle_hard]
> 0-gv2a2-posix: link
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.5
> ->
/bricks/brick2/gv2a2/.glusterfs/eb/04/eb044e6e-3a23-40a4-9ce1-f13af148eb67failed
> [File exists]" repeated 2 times between [2018-01-16 15:16:42.593392]
and
> [2018-01-16 15:16:42.593430]
> [2018-01-16 15:17:32.229689] W [MSGID: 113096]
[posix-handle.c:770:posix_handle_hard]
> 0-gv2a2-posix: link
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.14
> ->
/bricks/brick2/gv2a2/.glusterfs/53/04/530449fa-d698-4928-a262-9a0234232323failed
> [File exists]
> [2018-01-16 15:17:32.229720] E [MSGID: 113020] [posix.c:1485:posix_mknod]
> 0-gv2a2-posix: setting gfid on /bricks/brick2/gv2a2/.shard/
> 62335cb9-c7b5-4735-a879-59cff93fe622.14 failed
> [2018-01-16 15:18:07.154330] W [MSGID: 113096]
[posix-handle.c:770:posix_handle_hard]
> 0-gv2a2-posix: link
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.17
> ->
/bricks/brick2/gv2a2/.glusterfs/81/96/8196dd19-84bc-4c3d-909f-8792e9b4929dfailed
> [File exists]
> [2018-01-16 15:18:07.154375] E [MSGID: 113020] [posix.c:1485:posix_mknod]
> 0-gv2a2-posix: setting gfid on /bricks/brick2/gv2a2/.shard/
> 62335cb9-c7b5-4735-a879-59cff93fe622.17 failed
> The message "W [MSGID: 113096] [posix-handle.c:770:posix_handle_hard]
> 0-gv2a2-posix: link
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.14
> ->
/bricks/brick2/gv2a2/.glusterfs/53/04/530449fa-d698-4928-a262-9a0234232323failed
> [File exists]" repeated 7 times between [2018-01-16 15:17:32.229689]
and
> [2018-01-16 15:17:32.229806]
> The message "W [MSGID: 113096] [posix-handle.c:770:posix_handle_hard]
> 0-gv2a2-posix: link
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.17
> ->
/bricks/brick2/gv2a2/.glusterfs/81/96/8196dd19-84bc-4c3d-909f-8792e9b4929dfailed
> [File exists]" repeated 3 times between [2018-01-16 15:18:07.154330]
and
> [2018-01-16 15:18:07.154357]
> [2018-01-16 15:19:23.618794] W [MSGID: 113096]
[posix-handle.c:770:posix_handle_hard]
> 0-gv2a2-posix: link
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.21
> ->
/bricks/brick2/gv2a2/.glusterfs/6d/02/6d02bd98-83de-43e8-a7af-b1d5f5160403failed
> [File exists]
> [2018-01-16 15:19:23.618827] E [MSGID: 113020] [posix.c:1485:posix_mknod]
> 0-gv2a2-posix: setting gfid on /bricks/brick2/gv2a2/.shard/
> 62335cb9-c7b5-4735-a879-59cff93fe622.21 failed
> The message "W [MSGID: 113096] [posix-handle.c:770:posix_handle_hard]
> 0-gv2a2-posix: link
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.21
> ->
/bricks/brick2/gv2a2/.glusterfs/6d/02/6d02bd98-83de-43e8-a7af-b1d5f5160403failed
> [File exists]" repeated 3 times between [2018-01-16 15:19:23.618794]
and
> [2018-01-16 15:19:23.618794]
>
> Thank you,
>
> Il 16/01/2018 11:40, Krutika Dhananjay ha scritto:
>
> Also to help isolate the component, could you answer these:
>
> 1. on a different volume with shard not enabled, do you see this issue?
> 2. on a plain 3-way replicated volume (no arbiter), do you see this issue?
>
>
>
> On Tue, Jan 16, 2018 at 4:03 PM, Krutika Dhananjay <kdhananj at
redhat.com>
> wrote:
>
>> Please share the volume-info output and the logs under
>> /var/log/glusterfs/ from all your nodes. for investigating the issue.
>>
>> -Krutika
>>
>> On Tue, Jan 16, 2018 at 1:30 PM, Ing. Luca Lazzeroni - Trend Servizi
Srl
>> <luca at gvnet.it> wrote:
>>
>>> Hi to everyone.
>>>
>>> I've got a strange problem with a gluster setup: 3 nodes with
Centos
>>> 7.4, Gluster 3.12.4 from Centos/Gluster repositories, QEMU-KVM
version
>>> 2.9.0 (compiled from RHEL sources).
>>>
>>> I'm running volumes in replica 3 arbiter 1 mode (but I've
got a volume
>>> in "pure" replica 3 mode too). I've applied the
"virt" group settings to my
>>> volumes since they host VM images.
>>>
>>> If I try to install something (eg: Ubuntu Server 16.04.3) on a VM
(and
>>> so I generate a bit of I/O inside it) and configure KVM to access
gluster
>>> volume directly (via libvirt), install fails after a while because
the disk
>>> content is corrupted. If I inspect the block inside the disk (by
accessing
>>> the image directly from outside) I can found many files filled with
"^@".
>>>
>>
> Also, what exactly do you mean by accessing the image directly from
> outside? Was it from the brick directories directly? Was it from the mount
> point of the volume? Could you elaborate? Which files exactly did you
check?
>
> -Krutika
>
>
>>> If, instead, I configure KVM to access VM images via a FUSE mount,
>>> everything seems to work correctly.
>>>
>>> Note that the problem with install is verified 100% time with QCOW2
>>> image, while it appears only after with RAW disk images.
>>>
>>> Is there anyone who experienced the same problem ?
>>>
>>> Thank you,
>>>
>>>
>>> --
>>> Ing. Luca Lazzeroni
>>> Responsabile Ricerca e Sviluppo
>>> Trend Servizi Srl
>>> Tel: 0376/631761
>>> Web: https://www.trendservizi.it
>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>>
>>
>
> --
> Ing. Luca Lazzeroni
> Responsabile Ricerca e Sviluppo
> Trend Servizi Srl
> Tel: 0376/631761
> Web: https://www.trendservizi.it
>
>
>
> _______________________________________________
> Gluster-users mailing listGluster-users at
gluster.orghttp://lists.gluster.org/mailman/listinfo/gluster-users
>
>
> --
> Ing. Luca Lazzeroni
> Responsabile Ricerca e Sviluppo
> Trend Servizi Srl
> Tel: 0376/631761
> Web: https://www.trendservizi.it
>
>
>
> _______________________________________________
> Gluster-users mailing listGluster-users at
gluster.orghttp://lists.gluster.org/mailman/listinfo/gluster-users
>
>
> --
> Ing. Luca Lazzeroni
> Responsabile Ricerca e Sviluppo
> Trend Servizi Srl
> Tel: 0376/631761
> Web: https://www.trendservizi.it
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180117/449723d9/attachment.html>

Ing. Luca Lazzeroni - Trend Servizi Srl

2018-Jan-17 10:29 UTC

head link

[Gluster-users] [Possibile SPAM] Re: Problem with Gluster 3.12.4, VM and sharding

Of course. Here's the full log. Please, note that in FUSE mode 
everything works apparently without problems. I've installed 4 vm and 
updated them without problems.



Il 17/01/2018 11:00, Krutika Dhananjay ha scritto:>
>
> On Tue, Jan 16, 2018 at 10:47 PM, Ing. Luca Lazzeroni - Trend Servizi 
> Srl <luca at gvnet.it <mailto:luca at gvnet.it>> wrote:
>
>     I've made the test with raw image format (preallocated too) and
>     the corruption problem is still there (but without errors in
>     bricks' log file).
>
>     What does the "link" error in bricks log files means ?
>
>     I've seen the source code looking for the lines where it happens
>     and it seems a warning (it doesn't imply a failure).
>
>
> Indeed, it only represents a transient state when the shards are 
> created for the first time and does not indicate a failure.
> Could you also get the logs of the gluster fuse mount process? It 
> should be under /var/log/glusterfs of your client machine with the 
> filename as a hyphenated mount point path.
>
> For example, if your volume was mounted at /mnt/glusterfs, then your 
> log file would be named mnt-glusterfs.log.
>
> -Krutika
>
>
>
>     Il 16/01/2018 17:39, Ing. Luca Lazzeroni - Trend Servizi Srl ha
>     scritto:
>>
>>     An update:
>>
>>     I've tried, for my tests, to create the vm volume as
>>
>>     qemu-img create -f qcow2 -o preallocation=full
>>     gluster://gluster1/Test/Test-vda.img 20G
>>
>>     et voila !
>>
>>     No errors at all, neither in bricks' log file (the "link
failed"
>>     message disappeared), neither in VM (no corruption and installed
>>     succesfully).
>>
>>     I'll do another test with a fully preallocated raw image.
>>
>>
>>
>>     Il 16/01/2018 16:31, Ing. Luca Lazzeroni - Trend Servizi Srl ha
>>     scritto:
>>>
>>>     I've just done all the steps to reproduce the problem.
>>>
>>>     Tha VM volume has been created via "qemu-img create -f
qcow2
>>>     Test-vda2.qcow2 20G" on the gluster volume mounted via
FUSE.
>>>     I've tried also to create the volume with preallocated
metadata,
>>>     which moves the problem a bit far away (in time). The volume is
>>>     a replice 3 arbiter 1 volume hosted on XFS bricks.
>>>
>>>     Here are the informations:
>>>
>>>     [root at ovh-ov1 bricks]# gluster volume info gv2a2
>>>
>>>     Volume Name: gv2a2
>>>     Type: Replicate
>>>     Volume ID: 83c84774-2068-4bfc-b0b9-3e6b93705b9f
>>>     Status: Started
>>>     Snapshot Count: 0
>>>     Number of Bricks: 1 x (2 + 1) = 3
>>>     Transport-type: tcp
>>>     Bricks:
>>>     Brick1: gluster1:/bricks/brick2/gv2a2
>>>     Brick2: gluster3:/bricks/brick3/gv2a2
>>>     Brick3: gluster2:/bricks/arbiter_brick_gv2a2/gv2a2 (arbiter)
>>>     Options Reconfigured:
>>>     storage.owner-gid: 107
>>>     storage.owner-uid: 107
>>>     user.cifs: off
>>>     features.shard: on
>>>     cluster.shd-wait-qlength: 10000
>>>     cluster.shd-max-threads: 8
>>>     cluster.locking-scheme: granular
>>>     cluster.data-self-heal-algorithm: full
>>>     cluster.server-quorum-type: server
>>>     cluster.quorum-type: auto
>>>     cluster.eager-lock: enable
>>>     network.remote-dio: enable
>>>     performance.low-prio-threads: 32
>>>     performance.io-cache: off
>>>     performance.read-ahead: off
>>>     performance.quick-read: off
>>>     transport.address-family: inet
>>>     nfs.disable: off
>>>     performance.client-io-threads: off
>>>
>>>     /var/log/glusterfs/glusterd.log:
>>>
>>>     [2018-01-15 14:17:50.196228] I [MSGID: 106488]
>>>     [glusterd-handler.c:1548:__glusterd_handle_cli_get_volume]
>>>     0-management: Received get vol req
>>>     [2018-01-15 14:25:09.555214] I [MSGID: 106488]
>>>     [glusterd-handler.c:1548:__glusterd_handle_cli_get_volume]
>>>     0-management: Received get vol req
>>>
>>>     (empty because today it's 2018-01-16)
>>>
>>>     /var/log/glusterfs/glustershd.log:
>>>
>>>     [2018-01-14 02:23:02.731245] I
>>>     [glusterfsd-mgmt.c:1821:mgmt_getspec_cbk] 0-glusterfs: No
change
>>>     in volfile,continuing
>>>
>>>     (empty too)
>>>
>>>     /var/log/glusterfs/bricks/brick-brick2-gv2a2.log (the
interested
>>>     volume):
>>>
>>>     [2018-01-16 15:14:37.809965] I [MSGID: 115029]
>>>     [server-handshake.c:793:server_setvolume] 0-gv2a2-server:
>>>     accepted client from
>>>     ovh-ov1-10302-2018/01/16-15:14:37:790306-gv2a2-client-0-0-0
>>>     (version: 3.12.4)
>>>     [2018-01-16 15:16:41.471751] E [MSGID: 113020]
>>>     [posix.c:1485:posix_mknod] 0-gv2a2-posix: setting gfid on
>>>    
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.4
>>>     failed
>>>     [2018-01-16 15:16:41.471745] W [MSGID: 113096]
>>>     [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link
>>>    
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.4
>>>     ->
>>>    
/bricks/brick2/gv2a2/.glusterfs/a0/14/a0144df3-8d89-4aed-872e-5fef141e9e1efailed
>>>     [File exists]
>>>     [2018-01-16 15:16:42.593392] W [MSGID: 113096]
>>>     [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link
>>>    
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.5
>>>     ->
>>>    
/bricks/brick2/gv2a2/.glusterfs/eb/04/eb044e6e-3a23-40a4-9ce1-f13af148eb67failed
>>>     [File exists]
>>>     [2018-01-16 15:16:42.593426] E [MSGID: 113020]
>>>     [posix.c:1485:posix_mknod] 0-gv2a2-posix: setting gfid on
>>>    
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.5
>>>     failed
>>>     [2018-01-16 15:17:04.129593] W [MSGID: 113096]
>>>     [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link
>>>    
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.8
>>>     ->
>>>    
/bricks/brick2/gv2a2/.glusterfs/dc/92/dc92bd0a-0d46-4826-a4c9-d073a924dd8dfailed
>>>     [File exists]
>>>     The message "W [MSGID: 113096]
>>>     [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link
>>>    
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.8
>>>     ->
>>>    
/bricks/brick2/gv2a2/.glusterfs/dc/92/dc92bd0a-0d46-4826-a4c9-d073a924dd8dfailed
>>>     [File exists]" repeated 5 times between [2018-01-16
>>>     15:17:04.129593] and [2018-01-16 15:17:04.129593]
>>>     [2018-01-16 15:17:04.129661] E [MSGID: 113020]
>>>     [posix.c:1485:posix_mknod] 0-gv2a2-posix: setting gfid on
>>>    
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.8
>>>     failed
>>>     [2018-01-16 15:17:08.279162] W [MSGID: 113096]
>>>     [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link
>>>    
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.9
>>>     ->
>>>    
/bricks/brick2/gv2a2/.glusterfs/c9/b7/c9b71b00-a09f-4df1-b874-041820ca8241failed
>>>     [File exists]
>>>     [2018-01-16 15:17:08.279162] W [MSGID: 113096]
>>>     [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link
>>>    
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.9
>>>     ->
>>>    
/bricks/brick2/gv2a2/.glusterfs/c9/b7/c9b71b00-a09f-4df1-b874-041820ca8241failed
>>>     [File exists]
>>>     The message "W [MSGID: 113096]
>>>     [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link
>>>    
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.9
>>>     ->
>>>    
/bricks/brick2/gv2a2/.glusterfs/c9/b7/c9b71b00-a09f-4df1-b874-041820ca8241failed
>>>     [File exists]" repeated 2 times between [2018-01-16
>>>     15:17:08.279162] and [2018-01-16 15:17:08.279162]
>>>
>>>     [2018-01-16 15:17:08.279177] E [MSGID: 113020]
>>>     [posix.c:1485:posix_mknod] 0-gv2a2-posix: setting gfid on
>>>    
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.9
>>>     failed
>>>     The message "W [MSGID: 113096]
>>>     [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link
>>>    
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.4
>>>     ->
>>>    
/bricks/brick2/gv2a2/.glusterfs/a0/14/a0144df3-8d89-4aed-872e-5fef141e9e1efailed
>>>     [File exists]" repeated 6 times between [2018-01-16
>>>     15:16:41.471745] and [2018-01-16 15:16:41.471807]
>>>     The message "W [MSGID: 113096]
>>>     [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link
>>>    
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.5
>>>     ->
>>>    
/bricks/brick2/gv2a2/.glusterfs/eb/04/eb044e6e-3a23-40a4-9ce1-f13af148eb67failed
>>>     [File exists]" repeated 2 times between [2018-01-16
>>>     15:16:42.593392] and [2018-01-16 15:16:42.593430]
>>>     [2018-01-16 15:17:32.229689] W [MSGID: 113096]
>>>     [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link
>>>    
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.14
>>>     ->
>>>    
/bricks/brick2/gv2a2/.glusterfs/53/04/530449fa-d698-4928-a262-9a0234232323failed
>>>     [File exists]
>>>     [2018-01-16 15:17:32.229720] E [MSGID: 113020]
>>>     [posix.c:1485:posix_mknod] 0-gv2a2-posix: setting gfid on
>>>    
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.14
>>>     failed
>>>     [2018-01-16 15:18:07.154330] W [MSGID: 113096]
>>>     [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link
>>>    
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.17
>>>     ->
>>>    
/bricks/brick2/gv2a2/.glusterfs/81/96/8196dd19-84bc-4c3d-909f-8792e9b4929dfailed
>>>     [File exists]
>>>     [2018-01-16 15:18:07.154375] E [MSGID: 113020]
>>>     [posix.c:1485:posix_mknod] 0-gv2a2-posix: setting gfid on
>>>    
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.17
>>>     failed
>>>     The message "W [MSGID: 113096]
>>>     [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link
>>>    
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.14
>>>     ->
>>>    
/bricks/brick2/gv2a2/.glusterfs/53/04/530449fa-d698-4928-a262-9a0234232323failed
>>>     [File exists]" repeated 7 times between [2018-01-16
>>>     15:17:32.229689] and [2018-01-16 15:17:32.229806]
>>>     The message "W [MSGID: 113096]
>>>     [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link
>>>    
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.17
>>>     ->
>>>    
/bricks/brick2/gv2a2/.glusterfs/81/96/8196dd19-84bc-4c3d-909f-8792e9b4929dfailed
>>>     [File exists]" repeated 3 times between [2018-01-16
>>>     15:18:07.154330] and [2018-01-16 15:18:07.154357]
>>>     [2018-01-16 15:19:23.618794] W [MSGID: 113096]
>>>     [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link
>>>    
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.21
>>>     ->
>>>    
/bricks/brick2/gv2a2/.glusterfs/6d/02/6d02bd98-83de-43e8-a7af-b1d5f5160403failed
>>>     [File exists]
>>>     [2018-01-16 15:19:23.618827] E [MSGID: 113020]
>>>     [posix.c:1485:posix_mknod] 0-gv2a2-posix: setting gfid on
>>>    
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.21
>>>     failed
>>>     The message "W [MSGID: 113096]
>>>     [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link
>>>    
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.21
>>>     ->
>>>    
/bricks/brick2/gv2a2/.glusterfs/6d/02/6d02bd98-83de-43e8-a7af-b1d5f5160403failed
>>>     [File exists]" repeated 3 times between [2018-01-16
>>>     15:19:23.618794] and [2018-01-16 15:19:23.618794]
>>>
>>>     Thank you,
>>>
>>>
>>>     Il 16/01/2018 11:40, Krutika Dhananjay ha scritto:
>>>>     Also to help isolate the component, could you answer these:
>>>>
>>>>     1. on a different volume with shard not enabled, do you see
>>>>     this issue?
>>>>     2. on a plain 3-way replicated volume (no arbiter), do you
see
>>>>     this issue?
>>>>
>>>>
>>>>
>>>>     On Tue, Jan 16, 2018 at 4:03 PM, Krutika Dhananjay
>>>>     <kdhananj at redhat.com <mailto:kdhananj at
redhat.com>> wrote:
>>>>
>>>>         Please share the volume-info output and the logs under
>>>>         /var/log/glusterfs/ from all your nodes. for
investigating
>>>>         the issue.
>>>>
>>>>         -Krutika
>>>>
>>>>         On Tue, Jan 16, 2018 at 1:30 PM, Ing. Luca Lazzeroni -
>>>>         Trend Servizi Srl <luca at gvnet.it <mailto:luca
at gvnet.it>> wrote:
>>>>
>>>>             Hi to everyone.
>>>>
>>>>             I've got a strange problem with a gluster
setup: 3
>>>>             nodes with Centos 7.4, Gluster 3.12.4 from
>>>>             Centos/Gluster repositories, QEMU-KVM version 2.9.0
>>>>             (compiled from RHEL sources).
>>>>
>>>>             I'm running volumes in replica 3 arbiter 1 mode
(but
>>>>             I've got a volume in "pure" replica 3
mode too). I've
>>>>             applied the "virt" group settings to my
volumes since
>>>>             they host VM images.
>>>>
>>>>             If I try to install something (eg: Ubuntu Server
>>>>             16.04.3) on a VM (and so I generate a bit of I/O
inside
>>>>             it) and configure KVM to access gluster volume
directly
>>>>             (via libvirt), install fails after a while because
the
>>>>             disk content is corrupted. If I inspect the block
>>>>             inside the disk (by accessing the image directly
from
>>>>             outside) I can found many files filled with
"^@".
>>>>
>>>>
>>>>     Also, what exactly do you mean by accessing the image
directly
>>>>     from outside? Was it from the brick directories directly?
Was
>>>>     it from the mount point of the volume? Could you elaborate?
>>>>     Which files exactly did you check?
>>>>
>>>>     -Krutika
>>>>
>>>>
>>>>             If, instead, I configure KVM to access VM images
via a
>>>>             FUSE mount, everything seems to work correctly.
>>>>
>>>>             Note that the problem with install is verified 100%
>>>>             time with QCOW2 image, while it appears only after
with
>>>>             RAW disk images.
>>>>
>>>>             Is there anyone who experienced the same problem ?
>>>>
>>>>             Thank you,
>>>>
>>>>
>>>>             -- 
>>>>             Ing. Luca Lazzeroni
>>>>             Responsabile Ricerca e Sviluppo
>>>>             Trend Servizi Srl
>>>>             Tel: 0376/631761
>>>>             Web: https://www.trendservizi.it
>>>>
>>>>             _______________________________________________
>>>>             Gluster-users mailing list
>>>>             Gluster-users at gluster.org
>>>>             <mailto:Gluster-users at gluster.org>
>>>>            
http://lists.gluster.org/mailman/listinfo/gluster-users
>>>>            
<http://lists.gluster.org/mailman/listinfo/gluster-users>
>>>>
>>>>
>>>>
>>>
>>>     -- 
>>>     Ing. Luca Lazzeroni
>>>     Responsabile Ricerca e Sviluppo
>>>     Trend Servizi Srl
>>>     Tel: 0376/631761
>>>     Web:https://www.trendservizi.it
>>>
>>>
>>>     _______________________________________________
>>>     Gluster-users mailing list
>>>     Gluster-users at gluster.org <mailto:Gluster-users at
gluster.org>
>>>     http://lists.gluster.org/mailman/listinfo/gluster-users
>>>     <http://lists.gluster.org/mailman/listinfo/gluster-users>
>>
>>     -- 
>>     Ing. Luca Lazzeroni
>>     Responsabile Ricerca e Sviluppo
>>     Trend Servizi Srl
>>     Tel: 0376/631761
>>     Web:https://www.trendservizi.it
>>
>>
>>     _______________________________________________
>>     Gluster-users mailing list
>>     Gluster-users at gluster.org <mailto:Gluster-users at
gluster.org>
>>     http://lists.gluster.org/mailman/listinfo/gluster-users
>>     <http://lists.gluster.org/mailman/listinfo/gluster-users>
>
>     -- 
>     Ing. Luca Lazzeroni
>     Responsabile Ricerca e Sviluppo
>     Trend Servizi Srl
>     Tel: 0376/631761
>     Web:https://www.trendservizi.it
>
>
>     _______________________________________________
>     Gluster-users mailing list
>     Gluster-users at gluster.org <mailto:Gluster-users at
gluster.org>
>     http://lists.gluster.org/mailman/listinfo/gluster-users
>     <http://lists.gluster.org/mailman/listinfo/gluster-users>
>
>
-- 
Ing. Luca Lazzeroni
Responsabile Ricerca e Sviluppo
Trend Servizi Srl
Tel: 0376/631761
Web: https://www.trendservizi.it

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180117/9109e5f3/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mnt-gv2a2.log
Type: text/x-log
Size: 12292 bytes
Desc: not available
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180117/9109e5f3/attachment-0001.bin>

Possibly Parallel Threads

Search for more possibly parallel threads

Gluster users - Jan 2018 - [Possibile SPAM] Re: Problem with Gluster 3.12.4, VM and sharding

[Gluster-users] [Possibile SPAM] Re: Problem with Gluster 3.12.4, VM and sharding

[Gluster-users] [Possibile SPAM] Re: Problem with Gluster 3.12.4, VM and sharding

Possibly Parallel Threads