thr3ads.net - Gluster users - [Gluster-users] [Possibile SPAM] Re: Problem with Gluster 3.12.4, VM and sharding [Jan 2018]

If this information is useful, please help other people find it:
Share via:

Ing. Luca Lazzeroni - Trend Servizi Srl

2018-Jan-16 15:31 UTC

[Gluster-users] [Possibile SPAM] Re: Problem with Gluster 3.12.4, VM and sharding

I've just done all the steps to reproduce the problem.

Tha VM volume has been created via "qemu-img create -f qcow2 
Test-vda2.qcow2 20G" on the gluster volume mounted via FUSE. I've tried
also to create the volume with preallocated metadata, which moves the 
problem a bit far away (in time). The volume is a replice 3 arbiter 1 
volume hosted on XFS bricks.

Here are the informations:

[root at ovh-ov1 bricks]# gluster volume info gv2a2

Volume Name: gv2a2
Type: Replicate
Volume ID: 83c84774-2068-4bfc-b0b9-3e6b93705b9f
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (2 + 1) = 3
Transport-type: tcp
Bricks:
Brick1: gluster1:/bricks/brick2/gv2a2
Brick2: gluster3:/bricks/brick3/gv2a2
Brick3: gluster2:/bricks/arbiter_brick_gv2a2/gv2a2 (arbiter)
Options Reconfigured:
storage.owner-gid: 107
storage.owner-uid: 107
user.cifs: off
features.shard: on
cluster.shd-wait-qlength: 10000
cluster.shd-max-threads: 8
cluster.locking-scheme: granular
cluster.data-self-heal-algorithm: full
cluster.server-quorum-type: server
cluster.quorum-type: auto
cluster.eager-lock: enable
network.remote-dio: enable
performance.low-prio-threads: 32
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
transport.address-family: inet
nfs.disable: off
performance.client-io-threads: off

/var/log/glusterfs/glusterd.log:

[2018-01-15 14:17:50.196228] I [MSGID: 106488] 
[glusterd-handler.c:1548:__glusterd_handle_cli_get_volume] 0-management: 
Received get vol req
[2018-01-15 14:25:09.555214] I [MSGID: 106488] 
[glusterd-handler.c:1548:__glusterd_handle_cli_get_volume] 0-management: 
Received get vol req

(empty because today it's 2018-01-16)

/var/log/glusterfs/glustershd.log:

[2018-01-14 02:23:02.731245] I [glusterfsd-mgmt.c:1821:mgmt_getspec_cbk] 
0-glusterfs: No change in volfile,continuing

(empty too)

/var/log/glusterfs/bricks/brick-brick2-gv2a2.log (the interested volume):

[2018-01-16 15:14:37.809965] I [MSGID: 115029] 
[server-handshake.c:793:server_setvolume] 0-gv2a2-server: accepted 
client from ovh-ov1-10302-2018/01/16-15:14:37:790306-gv2a2-client-0-0-0 
(version: 3.12.4)
[2018-01-16 15:16:41.471751] E [MSGID: 113020] 
[posix.c:1485:posix_mknod] 0-gv2a2-posix: setting gfid on 
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.4 failed
[2018-01-16 15:16:41.471745] W [MSGID: 113096] 
[posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link 
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.4 -> 
/bricks/brick2/gv2a2/.glusterfs/a0/14/a0144df3-8d89-4aed-872e-5fef141e9e1efailed
[File exists]
[2018-01-16 15:16:42.593392] W [MSGID: 113096] 
[posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link 
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.5 -> 
/bricks/brick2/gv2a2/.glusterfs/eb/04/eb044e6e-3a23-40a4-9ce1-f13af148eb67failed
[File exists]
[2018-01-16 15:16:42.593426] E [MSGID: 113020] 
[posix.c:1485:posix_mknod] 0-gv2a2-posix: setting gfid on 
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.5 failed
[2018-01-16 15:17:04.129593] W [MSGID: 113096] 
[posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link 
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.8 -> 
/bricks/brick2/gv2a2/.glusterfs/dc/92/dc92bd0a-0d46-4826-a4c9-d073a924dd8dfailed
[File exists]
The message "W [MSGID: 113096] [posix-handle.c:770:posix_handle_hard] 
0-gv2a2-posix: link 
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.8 -> 
/bricks/brick2/gv2a2/.glusterfs/dc/92/dc92bd0a-0d46-4826-a4c9-d073a924dd8dfailed
[File exists]" repeated 5 times between [2018-01-16 15:17:04.129593] and 
[2018-01-16 15:17:04.129593]
[2018-01-16 15:17:04.129661] E [MSGID: 113020] 
[posix.c:1485:posix_mknod] 0-gv2a2-posix: setting gfid on 
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.8 failed
[2018-01-16 15:17:08.279162] W [MSGID: 113096] 
[posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link 
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.9 -> 
/bricks/brick2/gv2a2/.glusterfs/c9/b7/c9b71b00-a09f-4df1-b874-041820ca8241failed
[File exists]
[2018-01-16 15:17:08.279162] W [MSGID: 113096] 
[posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link 
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.9 -> 
/bricks/brick2/gv2a2/.glusterfs/c9/b7/c9b71b00-a09f-4df1-b874-041820ca8241failed
[File exists]
The message "W [MSGID: 113096] [posix-handle.c:770:posix_handle_hard] 
0-gv2a2-posix: link 
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.9 -> 
/bricks/brick2/gv2a2/.glusterfs/c9/b7/c9b71b00-a09f-4df1-b874-041820ca8241failed
[File exists]" repeated 2 times between [2018-01-16 15:17:08.279162] and 
[2018-01-16 15:17:08.279162]

[2018-01-16 15:17:08.279177] E [MSGID: 113020] 
[posix.c:1485:posix_mknod] 0-gv2a2-posix: setting gfid on 
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.9 failed
The message "W [MSGID: 113096] [posix-handle.c:770:posix_handle_hard] 
0-gv2a2-posix: link 
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.4 -> 
/bricks/brick2/gv2a2/.glusterfs/a0/14/a0144df3-8d89-4aed-872e-5fef141e9e1efailed
[File exists]" repeated 6 times between [2018-01-16 15:16:41.471745] and 
[2018-01-16 15:16:41.471807]
The message "W [MSGID: 113096] [posix-handle.c:770:posix_handle_hard] 
0-gv2a2-posix: link 
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.5 -> 
/bricks/brick2/gv2a2/.glusterfs/eb/04/eb044e6e-3a23-40a4-9ce1-f13af148eb67failed
[File exists]" repeated 2 times between [2018-01-16 15:16:42.593392] and 
[2018-01-16 15:16:42.593430]
[2018-01-16 15:17:32.229689] W [MSGID: 113096] 
[posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link 
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.14 -> 
/bricks/brick2/gv2a2/.glusterfs/53/04/530449fa-d698-4928-a262-9a0234232323failed
[File exists]
[2018-01-16 15:17:32.229720] E [MSGID: 113020] 
[posix.c:1485:posix_mknod] 0-gv2a2-posix: setting gfid on 
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.14 failed
[2018-01-16 15:18:07.154330] W [MSGID: 113096] 
[posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link 
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.17 -> 
/bricks/brick2/gv2a2/.glusterfs/81/96/8196dd19-84bc-4c3d-909f-8792e9b4929dfailed
[File exists]
[2018-01-16 15:18:07.154375] E [MSGID: 113020] 
[posix.c:1485:posix_mknod] 0-gv2a2-posix: setting gfid on 
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.17 failed
The message "W [MSGID: 113096] [posix-handle.c:770:posix_handle_hard] 
0-gv2a2-posix: link 
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.14 -> 
/bricks/brick2/gv2a2/.glusterfs/53/04/530449fa-d698-4928-a262-9a0234232323failed
[File exists]" repeated 7 times between [2018-01-16 15:17:32.229689] and 
[2018-01-16 15:17:32.229806]
The message "W [MSGID: 113096] [posix-handle.c:770:posix_handle_hard] 
0-gv2a2-posix: link 
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.17 -> 
/bricks/brick2/gv2a2/.glusterfs/81/96/8196dd19-84bc-4c3d-909f-8792e9b4929dfailed
[File exists]" repeated 3 times between [2018-01-16 15:18:07.154330] and 
[2018-01-16 15:18:07.154357]
[2018-01-16 15:19:23.618794] W [MSGID: 113096] 
[posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link 
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.21 -> 
/bricks/brick2/gv2a2/.glusterfs/6d/02/6d02bd98-83de-43e8-a7af-b1d5f5160403failed
[File exists]
[2018-01-16 15:19:23.618827] E [MSGID: 113020] 
[posix.c:1485:posix_mknod] 0-gv2a2-posix: setting gfid on 
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.21 failed
The message "W [MSGID: 113096] [posix-handle.c:770:posix_handle_hard] 
0-gv2a2-posix: link 
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.21 -> 
/bricks/brick2/gv2a2/.glusterfs/6d/02/6d02bd98-83de-43e8-a7af-b1d5f5160403failed
[File exists]" repeated 3 times between [2018-01-16 15:19:23.618794] and 
[2018-01-16 15:19:23.618794]

Thank you,


Il 16/01/2018 11:40, Krutika Dhananjay ha scritto:> Also to help isolate the component, could you answer these:
>
> 1. on a different volume with shard not enabled, do you see this issue?
> 2. on a plain 3-way replicated volume (no arbiter), do you see this issue?
>
>
>
> On Tue, Jan 16, 2018 at 4:03 PM, Krutika Dhananjay 
> <kdhananj at redhat.com <mailto:kdhananj at redhat.com>> wrote:
>
>     Please share the volume-info output and the logs under
>     /var/log/glusterfs/ from all your nodes. for investigating the issue.
>
>     -Krutika
>
>     On Tue, Jan 16, 2018 at 1:30 PM, Ing. Luca Lazzeroni - Trend
>     Servizi Srl <luca at gvnet.it <mailto:luca at gvnet.it>>
wrote:
>
>         Hi to everyone.
>
>         I've got a strange problem with a gluster setup: 3 nodes with
>         Centos 7.4, Gluster 3.12.4 from Centos/Gluster repositories,
>         QEMU-KVM version 2.9.0 (compiled from RHEL sources).
>
>         I'm running volumes in replica 3 arbiter 1 mode (but I've
got
>         a volume in "pure" replica 3 mode too). I've applied
the
>         "virt" group settings to my volumes since they host VM
images.
>
>         If I try to install something (eg: Ubuntu Server 16.04.3) on a
>         VM (and so I generate a bit of I/O inside it) and configure
>         KVM to access gluster volume directly (via libvirt), install
>         fails after a while because the disk content is corrupted. If
>         I inspect the block inside the disk (by accessing the image
>         directly from outside) I can found many files filled with
"^@".
>
>
> Also, what exactly do you mean by accessing the image directly from 
> outside? Was it from the brick directories directly? Was it from the 
> mount point of the volume? Could you elaborate? Which files exactly 
> did you check?
>
> -Krutika
>
>
>         If, instead, I configure KVM to access VM images via a FUSE
>         mount, everything seems to work correctly.
>
>         Note that the problem with install is verified 100% time with
>         QCOW2 image, while it appears only after with RAW disk images.
>
>         Is there anyone who experienced the same problem ?
>
>         Thank you,
>
>
>         -- 
>         Ing. Luca Lazzeroni
>         Responsabile Ricerca e Sviluppo
>         Trend Servizi Srl
>         Tel: 0376/631761
>         Web: https://www.trendservizi.it
>
>         _______________________________________________
>         Gluster-users mailing list
>         Gluster-users at gluster.org <mailto:Gluster-users at
gluster.org>
>         http://lists.gluster.org/mailman/listinfo/gluster-users
>         <http://lists.gluster.org/mailman/listinfo/gluster-users>
>
>
>
-- 
Ing. Luca Lazzeroni
Responsabile Ricerca e Sviluppo
Trend Servizi Srl
Tel: 0376/631761
Web: https://www.trendservizi.it

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180116/6c26eae9/attachment.html>

Ing. Luca Lazzeroni - Trend Servizi Srl

2018-Jan-16 16:39 UTC

head link

[Gluster-users] [Possibile SPAM] Re: Problem with Gluster 3.12.4, VM and sharding

An update:

I've tried, for my tests, to create the vm volume as

qemu-img create -f qcow2 -o preallocation=full 
gluster://gluster1/Test/Test-vda.img 20G

et voila !

No errors at all, neither in bricks' log file (the "link failed"
message
disappeared), neither in VM (no corruption and installed succesfully).

I'll do another test with a fully preallocated raw image.



Il 16/01/2018 16:31, Ing. Luca Lazzeroni - Trend Servizi Srl ha
scritto:>
> I've just done all the steps to reproduce the problem.
>
> Tha VM volume has been created via "qemu-img create -f qcow2 
> Test-vda2.qcow2 20G" on the gluster volume mounted via FUSE. I've 
> tried also to create the volume with preallocated metadata, which 
> moves the problem a bit far away (in time). The volume is a replice 3 
> arbiter 1 volume hosted on XFS bricks.
>
> Here are the informations:
>
> [root at ovh-ov1 bricks]# gluster volume info gv2a2
>
> Volume Name: gv2a2
> Type: Replicate
> Volume ID: 83c84774-2068-4bfc-b0b9-3e6b93705b9f
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x (2 + 1) = 3
> Transport-type: tcp
> Bricks:
> Brick1: gluster1:/bricks/brick2/gv2a2
> Brick2: gluster3:/bricks/brick3/gv2a2
> Brick3: gluster2:/bricks/arbiter_brick_gv2a2/gv2a2 (arbiter)
> Options Reconfigured:
> storage.owner-gid: 107
> storage.owner-uid: 107
> user.cifs: off
> features.shard: on
> cluster.shd-wait-qlength: 10000
> cluster.shd-max-threads: 8
> cluster.locking-scheme: granular
> cluster.data-self-heal-algorithm: full
> cluster.server-quorum-type: server
> cluster.quorum-type: auto
> cluster.eager-lock: enable
> network.remote-dio: enable
> performance.low-prio-threads: 32
> performance.io-cache: off
> performance.read-ahead: off
> performance.quick-read: off
> transport.address-family: inet
> nfs.disable: off
> performance.client-io-threads: off
>
> /var/log/glusterfs/glusterd.log:
>
> [2018-01-15 14:17:50.196228] I [MSGID: 106488] 
> [glusterd-handler.c:1548:__glusterd_handle_cli_get_volume] 
> 0-management: Received get vol req
> [2018-01-15 14:25:09.555214] I [MSGID: 106488] 
> [glusterd-handler.c:1548:__glusterd_handle_cli_get_volume] 
> 0-management: Received get vol req
>
> (empty because today it's 2018-01-16)
>
> /var/log/glusterfs/glustershd.log:
>
> [2018-01-14 02:23:02.731245] I 
> [glusterfsd-mgmt.c:1821:mgmt_getspec_cbk] 0-glusterfs: No change in 
> volfile,continuing
>
> (empty too)
>
> /var/log/glusterfs/bricks/brick-brick2-gv2a2.log (the interested volume):
>
> [2018-01-16 15:14:37.809965] I [MSGID: 115029] 
> [server-handshake.c:793:server_setvolume] 0-gv2a2-server: accepted 
> client from 
> ovh-ov1-10302-2018/01/16-15:14:37:790306-gv2a2-client-0-0-0 (version: 
> 3.12.4)
> [2018-01-16 15:16:41.471751] E [MSGID: 113020] 
> [posix.c:1485:posix_mknod] 0-gv2a2-posix: setting gfid on 
> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.4 failed
> [2018-01-16 15:16:41.471745] W [MSGID: 113096] 
> [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link 
> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.4 -> 
>
/bricks/brick2/gv2a2/.glusterfs/a0/14/a0144df3-8d89-4aed-872e-5fef141e9e1efailed
> [File exists]
> [2018-01-16 15:16:42.593392] W [MSGID: 113096] 
> [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link 
> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.5 -> 
>
/bricks/brick2/gv2a2/.glusterfs/eb/04/eb044e6e-3a23-40a4-9ce1-f13af148eb67failed
> [File exists]
> [2018-01-16 15:16:42.593426] E [MSGID: 113020] 
> [posix.c:1485:posix_mknod] 0-gv2a2-posix: setting gfid on 
> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.5 failed
> [2018-01-16 15:17:04.129593] W [MSGID: 113096] 
> [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link 
> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.8 -> 
>
/bricks/brick2/gv2a2/.glusterfs/dc/92/dc92bd0a-0d46-4826-a4c9-d073a924dd8dfailed
> [File exists]
> The message "W [MSGID: 113096] [posix-handle.c:770:posix_handle_hard] 
> 0-gv2a2-posix: link 
> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.8 -> 
>
/bricks/brick2/gv2a2/.glusterfs/dc/92/dc92bd0a-0d46-4826-a4c9-d073a924dd8dfailed
> [File exists]" repeated 5 times between [2018-01-16 15:17:04.129593] 
> and [2018-01-16 15:17:04.129593]
> [2018-01-16 15:17:04.129661] E [MSGID: 113020] 
> [posix.c:1485:posix_mknod] 0-gv2a2-posix: setting gfid on 
> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.8 failed
> [2018-01-16 15:17:08.279162] W [MSGID: 113096] 
> [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link 
> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.9 -> 
>
/bricks/brick2/gv2a2/.glusterfs/c9/b7/c9b71b00-a09f-4df1-b874-041820ca8241failed
> [File exists]
> [2018-01-16 15:17:08.279162] W [MSGID: 113096] 
> [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link 
> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.9 -> 
>
/bricks/brick2/gv2a2/.glusterfs/c9/b7/c9b71b00-a09f-4df1-b874-041820ca8241failed
> [File exists]
> The message "W [MSGID: 113096] [posix-handle.c:770:posix_handle_hard] 
> 0-gv2a2-posix: link 
> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.9 -> 
>
/bricks/brick2/gv2a2/.glusterfs/c9/b7/c9b71b00-a09f-4df1-b874-041820ca8241failed
> [File exists]" repeated 2 times between [2018-01-16 15:17:08.279162] 
> and [2018-01-16 15:17:08.279162]
>
> [2018-01-16 15:17:08.279177] E [MSGID: 113020] 
> [posix.c:1485:posix_mknod] 0-gv2a2-posix: setting gfid on 
> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.9 failed
> The message "W [MSGID: 113096] [posix-handle.c:770:posix_handle_hard] 
> 0-gv2a2-posix: link 
> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.4 -> 
>
/bricks/brick2/gv2a2/.glusterfs/a0/14/a0144df3-8d89-4aed-872e-5fef141e9e1efailed
> [File exists]" repeated 6 times between [2018-01-16 15:16:41.471745] 
> and [2018-01-16 15:16:41.471807]
> The message "W [MSGID: 113096] [posix-handle.c:770:posix_handle_hard] 
> 0-gv2a2-posix: link 
> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.5 -> 
>
/bricks/brick2/gv2a2/.glusterfs/eb/04/eb044e6e-3a23-40a4-9ce1-f13af148eb67failed
> [File exists]" repeated 2 times between [2018-01-16 15:16:42.593392] 
> and [2018-01-16 15:16:42.593430]
> [2018-01-16 15:17:32.229689] W [MSGID: 113096] 
> [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link 
> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.14 -> 
>
/bricks/brick2/gv2a2/.glusterfs/53/04/530449fa-d698-4928-a262-9a0234232323failed
> [File exists]
> [2018-01-16 15:17:32.229720] E [MSGID: 113020] 
> [posix.c:1485:posix_mknod] 0-gv2a2-posix: setting gfid on 
> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.14 failed
> [2018-01-16 15:18:07.154330] W [MSGID: 113096] 
> [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link 
> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.17 -> 
>
/bricks/brick2/gv2a2/.glusterfs/81/96/8196dd19-84bc-4c3d-909f-8792e9b4929dfailed
> [File exists]
> [2018-01-16 15:18:07.154375] E [MSGID: 113020] 
> [posix.c:1485:posix_mknod] 0-gv2a2-posix: setting gfid on 
> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.17 failed
> The message "W [MSGID: 113096] [posix-handle.c:770:posix_handle_hard] 
> 0-gv2a2-posix: link 
> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.14 -> 
>
/bricks/brick2/gv2a2/.glusterfs/53/04/530449fa-d698-4928-a262-9a0234232323failed
> [File exists]" repeated 7 times between [2018-01-16 15:17:32.229689] 
> and [2018-01-16 15:17:32.229806]
> The message "W [MSGID: 113096] [posix-handle.c:770:posix_handle_hard] 
> 0-gv2a2-posix: link 
> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.17 -> 
>
/bricks/brick2/gv2a2/.glusterfs/81/96/8196dd19-84bc-4c3d-909f-8792e9b4929dfailed
> [File exists]" repeated 3 times between [2018-01-16 15:18:07.154330] 
> and [2018-01-16 15:18:07.154357]
> [2018-01-16 15:19:23.618794] W [MSGID: 113096] 
> [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link 
> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.21 -> 
>
/bricks/brick2/gv2a2/.glusterfs/6d/02/6d02bd98-83de-43e8-a7af-b1d5f5160403failed
> [File exists]
> [2018-01-16 15:19:23.618827] E [MSGID: 113020] 
> [posix.c:1485:posix_mknod] 0-gv2a2-posix: setting gfid on 
> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.21 failed
> The message "W [MSGID: 113096] [posix-handle.c:770:posix_handle_hard] 
> 0-gv2a2-posix: link 
> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.21 -> 
>
/bricks/brick2/gv2a2/.glusterfs/6d/02/6d02bd98-83de-43e8-a7af-b1d5f5160403failed
> [File exists]" repeated 3 times between [2018-01-16 15:19:23.618794] 
> and [2018-01-16 15:19:23.618794]
>
> Thank you,
>
>
> Il 16/01/2018 11:40, Krutika Dhananjay ha scritto:
>> Also to help isolate the component, could you answer these:
>>
>> 1. on a different volume with shard not enabled, do you see this issue?
>> 2. on a plain 3-way replicated volume (no arbiter), do you see this 
>> issue?
>>
>>
>>
>> On Tue, Jan 16, 2018 at 4:03 PM, Krutika Dhananjay 
>> <kdhananj at redhat.com <mailto:kdhananj at redhat.com>>
wrote:
>>
>>     Please share the volume-info output and the logs under
>>     /var/log/glusterfs/ from all your nodes. for investigating the
issue.
>>
>>     -Krutika
>>
>>     On Tue, Jan 16, 2018 at 1:30 PM, Ing. Luca Lazzeroni - Trend
>>     Servizi Srl <luca at gvnet.it <mailto:luca at
gvnet.it>> wrote:
>>
>>         Hi to everyone.
>>
>>         I've got a strange problem with a gluster setup: 3 nodes
with
>>         Centos 7.4, Gluster 3.12.4 from Centos/Gluster repositories,
>>         QEMU-KVM version 2.9.0 (compiled from RHEL sources).
>>
>>         I'm running volumes in replica 3 arbiter 1 mode (but
I've got
>>         a volume in "pure" replica 3 mode too). I've
applied the
>>         "virt" group settings to my volumes since they host
VM images.
>>
>>         If I try to install something (eg: Ubuntu Server 16.04.3) on
>>         a VM (and so I generate a bit of I/O inside it) and configure
>>         KVM to access gluster volume directly (via libvirt), install
>>         fails after a while because the disk content is corrupted. If
>>         I inspect the block inside the disk (by accessing the image
>>         directly from outside) I can found many files filled with
"^@".
>>
>>
>> Also, what exactly do you mean by accessing the image directly from 
>> outside? Was it from the brick directories directly? Was it from the 
>> mount point of the volume? Could you elaborate? Which files exactly 
>> did you check?
>>
>> -Krutika
>>
>>
>>         If, instead, I configure KVM to access VM images via a FUSE
>>         mount, everything seems to work correctly.
>>
>>         Note that the problem with install is verified 100% time with
>>         QCOW2 image, while it appears only after with RAW disk images.
>>
>>         Is there anyone who experienced the same problem ?
>>
>>         Thank you,
>>
>>
>>         -- 
>>         Ing. Luca Lazzeroni
>>         Responsabile Ricerca e Sviluppo
>>         Trend Servizi Srl
>>         Tel: 0376/631761
>>         Web: https://www.trendservizi.it
>>
>>         _______________________________________________
>>         Gluster-users mailing list
>>         Gluster-users at gluster.org <mailto:Gluster-users at
gluster.org>
>>         http://lists.gluster.org/mailman/listinfo/gluster-users
>>         <http://lists.gluster.org/mailman/listinfo/gluster-users>
>>
>>
>>
>
> -- 
> Ing. Luca Lazzeroni
> Responsabile Ricerca e Sviluppo
> Trend Servizi Srl
> Tel: 0376/631761
> Web:https://www.trendservizi.it
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
-- 
Ing. Luca Lazzeroni
Responsabile Ricerca e Sviluppo
Trend Servizi Srl
Tel: 0376/631761
Web: https://www.trendservizi.it

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180116/51bed482/attachment.html>

Reasonably Related Threads

Search for more reasonably related threads

Gluster users - Jan 2018 - [Possibile SPAM] Re: Problem with Gluster 3.12.4, VM and sharding

[Gluster-users] [Possibile SPAM] Re: Problem with Gluster 3.12.4, VM and sharding

[Gluster-users] [Possibile SPAM] Re: Problem with Gluster 3.12.4, VM and sharding

Reasonably Related Threads