Ing. Luca Lazzeroni - Trend Servizi Srl
2018-Jan-17 13:51 UTC
[Gluster-users] [Possibile SPAM] Re: Problem with Gluster 3.12.4, VM and sharding
Hi, after our IRC chat I've rebuilt a virtual machine with FUSE based virtual disk. Everything worked flawlessly. Now I'm sending you the output of the requested getfattr command on the disk image: # file: TestFUSE-vda.qcow2 trusted.afr.dirty=0x000000000000000000000000 trusted.gfid=0x40ffafbbe987445692bb31295fa40105 trusted.gfid2path.dc9dde61f0b77eab=0x31326533323631662d373839332d346262302d383738632d3966623765306232336263652f54657374465553452d7664612e71636f7732 trusted.glusterfs.shard.block-size=0x0000000004000000 trusted.glusterfs.shard.file-size=0x00000000c15300000000000000000000000000000060be900000000000000000 Hope this helps. Il 17/01/2018 11:37, Ing. Luca Lazzeroni - Trend Servizi Srl ha scritto:> > I actually use FUSE and it works. If i try to use "libgfapi" direct > interface to gluster in qemu-kvm, the problem appears. > > > > Il 17/01/2018 11:35, Krutika Dhananjay ha scritto: >> Really? Then which protocol exactly do you see this issue with? >> libgfapi? NFS? >> >> -Krutika >> >> On Wed, Jan 17, 2018 at 3:59 PM, Ing. Luca Lazzeroni - Trend Servizi >> Srl <luca at gvnet.it <mailto:luca at gvnet.it>> wrote: >> >> Of course. Here's the full log. Please, note that in FUSE mode >> everything works apparently without problems. I've installed 4 vm >> and updated them without problems. >> >> >> >> Il 17/01/2018 11:00, Krutika Dhananjay ha scritto: >>> >>> >>> On Tue, Jan 16, 2018 at 10:47 PM, Ing. Luca Lazzeroni - Trend >>> Servizi Srl <luca at gvnet.it <mailto:luca at gvnet.it>> wrote: >>> >>> I've made the test with raw image format (preallocated too) >>> and the corruption problem is still there (but without >>> errors in bricks' log file). >>> >>> What does the "link" error in bricks log files means ? >>> >>> I've seen the source code looking for the lines where it >>> happens and it seems a warning (it doesn't imply a failure). >>> >>> >>> Indeed, it only represents a transient state when the shards are >>> created for the first time and does not indicate a failure. >>> Could you also get the logs of the gluster fuse mount process? >>> It should be under /var/log/glusterfs of your client machine >>> with the filename as a hyphenated mount point path. >>> >>> For example, if your volume was mounted at /mnt/glusterfs, then >>> your log file would be named mnt-glusterfs.log. >>> >>> -Krutika >>> >>> >>> >>> Il 16/01/2018 17:39, Ing. Luca Lazzeroni - Trend Servizi Srl >>> ha scritto: >>>> >>>> An update: >>>> >>>> I've tried, for my tests, to create the vm volume as >>>> >>>> qemu-img create -f qcow2 -o preallocation=full >>>> gluster://gluster1/Test/Test-vda.img 20G >>>> >>>> et voila ! >>>> >>>> No errors at all, neither in bricks' log file (the "link >>>> failed" message disappeared), neither in VM (no corruption >>>> and installed succesfully). >>>> >>>> I'll do another test with a fully preallocated raw image. >>>> >>>> >>>> >>>> Il 16/01/2018 16:31, Ing. Luca Lazzeroni - Trend Servizi >>>> Srl ha scritto: >>>>> >>>>> I've just done all the steps to reproduce the problem. >>>>> >>>>> Tha VM volume has been created via "qemu-img create -f >>>>> qcow2 Test-vda2.qcow2 20G" on the gluster volume mounted >>>>> via FUSE. I've tried also to create the volume with >>>>> preallocated metadata, which moves the problem a bit far >>>>> away (in time). The volume is a replice 3 arbiter 1 volume >>>>> hosted on XFS bricks. >>>>> >>>>> Here are the informations: >>>>> >>>>> [root at ovh-ov1 bricks]# gluster volume info gv2a2 >>>>> >>>>> Volume Name: gv2a2 >>>>> Type: Replicate >>>>> Volume ID: 83c84774-2068-4bfc-b0b9-3e6b93705b9f >>>>> Status: Started >>>>> Snapshot Count: 0 >>>>> Number of Bricks: 1 x (2 + 1) = 3 >>>>> Transport-type: tcp >>>>> Bricks: >>>>> Brick1: gluster1:/bricks/brick2/gv2a2 >>>>> Brick2: gluster3:/bricks/brick3/gv2a2 >>>>> Brick3: gluster2:/bricks/arbiter_brick_gv2a2/gv2a2 (arbiter) >>>>> Options Reconfigured: >>>>> storage.owner-gid: 107 >>>>> storage.owner-uid: 107 >>>>> user.cifs: off >>>>> features.shard: on >>>>> cluster.shd-wait-qlength: 10000 >>>>> cluster.shd-max-threads: 8 >>>>> cluster.locking-scheme: granular >>>>> cluster.data-self-heal-algorithm: full >>>>> cluster.server-quorum-type: server >>>>> cluster.quorum-type: auto >>>>> cluster.eager-lock: enable >>>>> network.remote-dio: enable >>>>> performance.low-prio-threads: 32 >>>>> performance.io-cache: off >>>>> performance.read-ahead: off >>>>> performance.quick-read: off >>>>> transport.address-family: inet >>>>> nfs.disable: off >>>>> performance.client-io-threads: off >>>>> >>>>> /var/log/glusterfs/glusterd.log: >>>>> >>>>> [2018-01-15 14:17:50.196228] I [MSGID: 106488] >>>>> [glusterd-handler.c:1548:__glusterd_handle_cli_get_volume] >>>>> 0-management: Received get vol req >>>>> [2018-01-15 14:25:09.555214] I [MSGID: 106488] >>>>> [glusterd-handler.c:1548:__glusterd_handle_cli_get_volume] >>>>> 0-management: Received get vol req >>>>> >>>>> (empty because today it's 2018-01-16) >>>>> >>>>> /var/log/glusterfs/glustershd.log: >>>>> >>>>> [2018-01-14 02:23:02.731245] I >>>>> [glusterfsd-mgmt.c:1821:mgmt_getspec_cbk] 0-glusterfs: No >>>>> change in volfile,continuing >>>>> >>>>> (empty too) >>>>> >>>>> /var/log/glusterfs/bricks/brick-brick2-gv2a2.log (the >>>>> interested volume): >>>>> >>>>> [2018-01-16 15:14:37.809965] I [MSGID: 115029] >>>>> [server-handshake.c:793:server_setvolume] 0-gv2a2-server: >>>>> accepted client from >>>>> ovh-ov1-10302-2018/01/16-15:14:37:790306-gv2a2-client-0-0-0 >>>>> (version: 3.12.4) >>>>> [2018-01-16 15:16:41.471751] E [MSGID: 113020] >>>>> [posix.c:1485:posix_mknod] 0-gv2a2-posix: setting gfid on >>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.4 >>>>> failed >>>>> [2018-01-16 15:16:41.471745] W [MSGID: 113096] >>>>> [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link >>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.4 >>>>> -> >>>>> /bricks/brick2/gv2a2/.glusterfs/a0/14/a0144df3-8d89-4aed-872e-5fef141e9e1efailed >>>>> [File exists] >>>>> [2018-01-16 15:16:42.593392] W [MSGID: 113096] >>>>> [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link >>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.5 >>>>> -> >>>>> /bricks/brick2/gv2a2/.glusterfs/eb/04/eb044e6e-3a23-40a4-9ce1-f13af148eb67failed >>>>> [File exists] >>>>> [2018-01-16 15:16:42.593426] E [MSGID: 113020] >>>>> [posix.c:1485:posix_mknod] 0-gv2a2-posix: setting gfid on >>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.5 >>>>> failed >>>>> [2018-01-16 15:17:04.129593] W [MSGID: 113096] >>>>> [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link >>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.8 >>>>> -> >>>>> /bricks/brick2/gv2a2/.glusterfs/dc/92/dc92bd0a-0d46-4826-a4c9-d073a924dd8dfailed >>>>> [File exists] >>>>> The message "W [MSGID: 113096] >>>>> [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link >>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.8 >>>>> -> >>>>> /bricks/brick2/gv2a2/.glusterfs/dc/92/dc92bd0a-0d46-4826-a4c9-d073a924dd8dfailed >>>>> [File exists]" repeated 5 times between [2018-01-16 >>>>> 15:17:04.129593] and [2018-01-16 15:17:04.129593] >>>>> [2018-01-16 15:17:04.129661] E [MSGID: 113020] >>>>> [posix.c:1485:posix_mknod] 0-gv2a2-posix: setting gfid on >>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.8 >>>>> failed >>>>> [2018-01-16 15:17:08.279162] W [MSGID: 113096] >>>>> [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link >>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.9 >>>>> -> >>>>> /bricks/brick2/gv2a2/.glusterfs/c9/b7/c9b71b00-a09f-4df1-b874-041820ca8241failed >>>>> [File exists] >>>>> [2018-01-16 15:17:08.279162] W [MSGID: 113096] >>>>> [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link >>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.9 >>>>> -> >>>>> /bricks/brick2/gv2a2/.glusterfs/c9/b7/c9b71b00-a09f-4df1-b874-041820ca8241failed >>>>> [File exists] >>>>> The message "W [MSGID: 113096] >>>>> [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link >>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.9 >>>>> -> >>>>> /bricks/brick2/gv2a2/.glusterfs/c9/b7/c9b71b00-a09f-4df1-b874-041820ca8241failed >>>>> [File exists]" repeated 2 times between [2018-01-16 >>>>> 15:17:08.279162] and [2018-01-16 15:17:08.279162] >>>>> >>>>> [2018-01-16 15:17:08.279177] E [MSGID: 113020] >>>>> [posix.c:1485:posix_mknod] 0-gv2a2-posix: setting gfid on >>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.9 >>>>> failed >>>>> The message "W [MSGID: 113096] >>>>> [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link >>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.4 >>>>> -> >>>>> /bricks/brick2/gv2a2/.glusterfs/a0/14/a0144df3-8d89-4aed-872e-5fef141e9e1efailed >>>>> [File exists]" repeated 6 times between [2018-01-16 >>>>> 15:16:41.471745] and [2018-01-16 15:16:41.471807] >>>>> The message "W [MSGID: 113096] >>>>> [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link >>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.5 >>>>> -> >>>>> /bricks/brick2/gv2a2/.glusterfs/eb/04/eb044e6e-3a23-40a4-9ce1-f13af148eb67failed >>>>> [File exists]" repeated 2 times between [2018-01-16 >>>>> 15:16:42.593392] and [2018-01-16 15:16:42.593430] >>>>> [2018-01-16 15:17:32.229689] W [MSGID: 113096] >>>>> [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link >>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.14 >>>>> -> >>>>> /bricks/brick2/gv2a2/.glusterfs/53/04/530449fa-d698-4928-a262-9a0234232323failed >>>>> [File exists] >>>>> [2018-01-16 15:17:32.229720] E [MSGID: 113020] >>>>> [posix.c:1485:posix_mknod] 0-gv2a2-posix: setting gfid on >>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.14 >>>>> failed >>>>> [2018-01-16 15:18:07.154330] W [MSGID: 113096] >>>>> [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link >>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.17 >>>>> -> >>>>> /bricks/brick2/gv2a2/.glusterfs/81/96/8196dd19-84bc-4c3d-909f-8792e9b4929dfailed >>>>> [File exists] >>>>> [2018-01-16 15:18:07.154375] E [MSGID: 113020] >>>>> [posix.c:1485:posix_mknod] 0-gv2a2-posix: setting gfid on >>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.17 >>>>> failed >>>>> The message "W [MSGID: 113096] >>>>> [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link >>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.14 >>>>> -> >>>>> /bricks/brick2/gv2a2/.glusterfs/53/04/530449fa-d698-4928-a262-9a0234232323failed >>>>> [File exists]" repeated 7 times between [2018-01-16 >>>>> 15:17:32.229689] and [2018-01-16 15:17:32.229806] >>>>> The message "W [MSGID: 113096] >>>>> [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link >>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.17 >>>>> -> >>>>> /bricks/brick2/gv2a2/.glusterfs/81/96/8196dd19-84bc-4c3d-909f-8792e9b4929dfailed >>>>> [File exists]" repeated 3 times between [2018-01-16 >>>>> 15:18:07.154330] and [2018-01-16 15:18:07.154357] >>>>> [2018-01-16 15:19:23.618794] W [MSGID: 113096] >>>>> [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link >>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.21 >>>>> -> >>>>> /bricks/brick2/gv2a2/.glusterfs/6d/02/6d02bd98-83de-43e8-a7af-b1d5f5160403failed >>>>> [File exists] >>>>> [2018-01-16 15:19:23.618827] E [MSGID: 113020] >>>>> [posix.c:1485:posix_mknod] 0-gv2a2-posix: setting gfid on >>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.21 >>>>> failed >>>>> The message "W [MSGID: 113096] >>>>> [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: link >>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.21 >>>>> -> >>>>> /bricks/brick2/gv2a2/.glusterfs/6d/02/6d02bd98-83de-43e8-a7af-b1d5f5160403failed >>>>> [File exists]" repeated 3 times between [2018-01-16 >>>>> 15:19:23.618794] and [2018-01-16 15:19:23.618794] >>>>> >>>>> Thank you, >>>>> >>>>> >>>>> Il 16/01/2018 11:40, Krutika Dhananjay ha scritto: >>>>>> Also to help isolate the component, could you answer these: >>>>>> >>>>>> 1. on a different volume with shard not enabled, do you >>>>>> see this issue? >>>>>> 2. on a plain 3-way replicated volume (no arbiter), do >>>>>> you see this issue? >>>>>> >>>>>> >>>>>> >>>>>> On Tue, Jan 16, 2018 at 4:03 PM, Krutika Dhananjay >>>>>> <kdhananj at redhat.com <mailto:kdhananj at redhat.com>> wrote: >>>>>> >>>>>> Please share the volume-info output and the logs >>>>>> under /var/log/glusterfs/ from all your nodes. for >>>>>> investigating the issue. >>>>>> >>>>>> -Krutika >>>>>> >>>>>> On Tue, Jan 16, 2018 at 1:30 PM, Ing. Luca Lazzeroni >>>>>> - Trend Servizi Srl <luca at gvnet.it >>>>>> <mailto:luca at gvnet.it>> wrote: >>>>>> >>>>>> Hi to everyone. >>>>>> >>>>>> I've got a strange problem with a gluster setup: >>>>>> 3 nodes with Centos 7.4, Gluster 3.12.4 from >>>>>> Centos/Gluster repositories, QEMU-KVM version >>>>>> 2.9.0 (compiled from RHEL sources). >>>>>> >>>>>> I'm running volumes in replica 3 arbiter 1 mode >>>>>> (but I've got a volume in "pure" replica 3 mode >>>>>> too). I've applied the "virt" group settings to >>>>>> my volumes since they host VM images. >>>>>> >>>>>> If I try to install something (eg: Ubuntu Server >>>>>> 16.04.3) on a VM (and so I generate a bit of I/O >>>>>> inside it) and configure KVM to access gluster >>>>>> volume directly (via libvirt), install fails >>>>>> after a while because the disk content is >>>>>> corrupted. If I inspect the block inside the disk >>>>>> (by accessing the image directly from outside) I >>>>>> can found many files filled with "^@". >>>>>> >>>>>> >>>>>> Also, what exactly do you mean by accessing the image >>>>>> directly from outside? Was it from the brick directories >>>>>> directly? Was it from the mount point of the volume? >>>>>> Could you elaborate? Which files exactly did you check? >>>>>> >>>>>> -Krutika >>>>>> >>>>>> >>>>>> If, instead, I configure KVM to access VM images >>>>>> via a FUSE mount, everything seems to work correctly. >>>>>> >>>>>> Note that the problem with install is verified >>>>>> 100% time with QCOW2 image, while it appears only >>>>>> after with RAW disk images. >>>>>> >>>>>> Is there anyone who experienced the same problem ? >>>>>> >>>>>> Thank you, >>>>>> >>>>>> >>>>>> -- >>>>>> Ing. Luca Lazzeroni >>>>>> Responsabile Ricerca e Sviluppo >>>>>> Trend Servizi Srl >>>>>> Tel: 0376/631761 >>>>>> Web: https://www.trendservizi.it >>>>>> >>>>>> _______________________________________________ >>>>>> Gluster-users mailing list >>>>>> Gluster-users at gluster.org >>>>>> <mailto:Gluster-users at gluster.org> >>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>>> <http://lists.gluster.org/mailman/listinfo/gluster-users> >>>>>> >>>>>> >>>>>> >>>>> >>>>> -- >>>>> Ing. Luca Lazzeroni >>>>> Responsabile Ricerca e Sviluppo >>>>> Trend Servizi Srl >>>>> Tel: 0376/631761 >>>>> Web:https://www.trendservizi.it >>>>> >>>>> >>>>> _______________________________________________ >>>>> Gluster-users mailing list >>>>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> >>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>> <http://lists.gluster.org/mailman/listinfo/gluster-users> >>>> >>>> -- >>>> Ing. Luca Lazzeroni >>>> Responsabile Ricerca e Sviluppo >>>> Trend Servizi Srl >>>> Tel: 0376/631761 >>>> Web:https://www.trendservizi.it >>>> >>>> >>>> _______________________________________________ >>>> Gluster-users mailing list >>>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> >>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>> <http://lists.gluster.org/mailman/listinfo/gluster-users> >>> >>> -- >>> Ing. Luca Lazzeroni >>> Responsabile Ricerca e Sviluppo >>> Trend Servizi Srl >>> Tel: 0376/631761 >>> Web:https://www.trendservizi.it >>> >>> >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> >>> http://lists.gluster.org/mailman/listinfo/gluster-users >>> <http://lists.gluster.org/mailman/listinfo/gluster-users> >>> >>> >> >> -- >> Ing. Luca Lazzeroni >> Responsabile Ricerca e Sviluppo >> Trend Servizi Srl >> Tel: 0376/631761 >> Web:https://www.trendservizi.it >> >> > > -- > Ing. Luca Lazzeroni > Responsabile Ricerca e Sviluppo > Trend Servizi Srl > Tel: 0376/631761 > Web:https://www.trendservizi.it > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users-- Ing. Luca Lazzeroni Responsabile Ricerca e Sviluppo Trend Servizi Srl Tel: 0376/631761 Web: https://www.trendservizi.it -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180117/a617a0cd/attachment.html>
Ing. Luca Lazzeroni - Trend Servizi Srl
2018-Jan-18 08:09 UTC
[Gluster-users] [Possibile SPAM] Re: Problem with Gluster 3.12.4, VM and sharding
Another update. I've setup a replica 3 volume without sharding and tried to install a VM on a qcow2 volume on that device; however the result is the same and the vm image has been corrupted, exactly at the same point. Here's the volume info of the create volume: Volume Name: gvtest Type: Replicate Volume ID: e2ddf694-ba46-4bc7-bc9c-e30803374e9d Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: gluster1:/bricks/brick1/gvtest Brick2: gluster2:/bricks/brick1/gvtest Brick3: gluster3:/bricks/brick1/gvtest Options Reconfigured: user.cifs: off features.shard: off cluster.shd-wait-qlength: 10000 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: server cluster.quorum-type: auto cluster.eager-lock: enable network.remote-dio: enable performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off transport.address-family: inet nfs.disable: on performance.client-io-threads: off Il 17/01/2018 14:51, Ing. Luca Lazzeroni - Trend Servizi Srl ha scritto:> > Hi, > > after our IRC chat I've rebuilt a virtual machine with FUSE based > virtual disk. Everything worked flawlessly. > > Now I'm sending you the output of the requested getfattr command on > the disk image: > > # file: TestFUSE-vda.qcow2 > trusted.afr.dirty=0x000000000000000000000000 > trusted.gfid=0x40ffafbbe987445692bb31295fa40105 > trusted.gfid2path.dc9dde61f0b77eab=0x31326533323631662d373839332d346262302d383738632d3966623765306232336263652f54657374465553452d7664612e71636f7732 > trusted.glusterfs.shard.block-size=0x0000000004000000 > trusted.glusterfs.shard.file-size=0x00000000c15300000000000000000000000000000060be900000000000000000 > > Hope this helps. > > > > Il 17/01/2018 11:37, Ing. Luca Lazzeroni - Trend Servizi Srl ha scritto: >> >> I actually use FUSE and it works. If i try to use "libgfapi" direct >> interface to gluster in qemu-kvm, the problem appears. >> >> >> >> Il 17/01/2018 11:35, Krutika Dhananjay ha scritto: >>> Really? Then which protocol exactly do you see this issue with? >>> libgfapi? NFS? >>> >>> -Krutika >>> >>> On Wed, Jan 17, 2018 at 3:59 PM, Ing. Luca Lazzeroni - Trend Servizi >>> Srl <luca at gvnet.it <mailto:luca at gvnet.it>> wrote: >>> >>> Of course. Here's the full log. Please, note that in FUSE mode >>> everything works apparently without problems. I've installed 4 >>> vm and updated them without problems. >>> >>> >>> >>> Il 17/01/2018 11:00, Krutika Dhananjay ha scritto: >>>> >>>> >>>> On Tue, Jan 16, 2018 at 10:47 PM, Ing. Luca Lazzeroni - Trend >>>> Servizi Srl <luca at gvnet.it <mailto:luca at gvnet.it>> wrote: >>>> >>>> I've made the test with raw image format (preallocated too) >>>> and the corruption problem is still there (but without >>>> errors in bricks' log file). >>>> >>>> What does the "link" error in bricks log files means ? >>>> >>>> I've seen the source code looking for the lines where it >>>> happens and it seems a warning (it doesn't imply a failure). >>>> >>>> >>>> Indeed, it only represents a transient state when the shards >>>> are created for the first time and does not indicate a failure. >>>> Could you also get the logs of the gluster fuse mount process? >>>> It should be under /var/log/glusterfs of your client machine >>>> with the filename as a hyphenated mount point path. >>>> >>>> For example, if your volume was mounted at /mnt/glusterfs, then >>>> your log file would be named mnt-glusterfs.log. >>>> >>>> -Krutika >>>> >>>> >>>> >>>> Il 16/01/2018 17:39, Ing. Luca Lazzeroni - Trend Servizi >>>> Srl ha scritto: >>>>> >>>>> An update: >>>>> >>>>> I've tried, for my tests, to create the vm volume as >>>>> >>>>> qemu-img create -f qcow2 -o preallocation=full >>>>> gluster://gluster1/Test/Test-vda.img 20G >>>>> >>>>> et voila ! >>>>> >>>>> No errors at all, neither in bricks' log file (the "link >>>>> failed" message disappeared), neither in VM (no corruption >>>>> and installed succesfully). >>>>> >>>>> I'll do another test with a fully preallocated raw image. >>>>> >>>>> >>>>> >>>>> Il 16/01/2018 16:31, Ing. Luca Lazzeroni - Trend Servizi >>>>> Srl ha scritto: >>>>>> >>>>>> I've just done all the steps to reproduce the problem. >>>>>> >>>>>> Tha VM volume has been created via "qemu-img create -f >>>>>> qcow2 Test-vda2.qcow2 20G" on the gluster volume mounted >>>>>> via FUSE. I've tried also to create the volume with >>>>>> preallocated metadata, which moves the problem a bit far >>>>>> away (in time). The volume is a replice 3 arbiter 1 >>>>>> volume hosted on XFS bricks. >>>>>> >>>>>> Here are the informations: >>>>>> >>>>>> [root at ovh-ov1 bricks]# gluster volume info gv2a2 >>>>>> >>>>>> Volume Name: gv2a2 >>>>>> Type: Replicate >>>>>> Volume ID: 83c84774-2068-4bfc-b0b9-3e6b93705b9f >>>>>> Status: Started >>>>>> Snapshot Count: 0 >>>>>> Number of Bricks: 1 x (2 + 1) = 3 >>>>>> Transport-type: tcp >>>>>> Bricks: >>>>>> Brick1: gluster1:/bricks/brick2/gv2a2 >>>>>> Brick2: gluster3:/bricks/brick3/gv2a2 >>>>>> Brick3: gluster2:/bricks/arbiter_brick_gv2a2/gv2a2 (arbiter) >>>>>> Options Reconfigured: >>>>>> storage.owner-gid: 107 >>>>>> storage.owner-uid: 107 >>>>>> user.cifs: off >>>>>> features.shard: on >>>>>> cluster.shd-wait-qlength: 10000 >>>>>> cluster.shd-max-threads: 8 >>>>>> cluster.locking-scheme: granular >>>>>> cluster.data-self-heal-algorithm: full >>>>>> cluster.server-quorum-type: server >>>>>> cluster.quorum-type: auto >>>>>> cluster.eager-lock: enable >>>>>> network.remote-dio: enable >>>>>> performance.low-prio-threads: 32 >>>>>> performance.io-cache: off >>>>>> performance.read-ahead: off >>>>>> performance.quick-read: off >>>>>> transport.address-family: inet >>>>>> nfs.disable: off >>>>>> performance.client-io-threads: off >>>>>> >>>>>> /var/log/glusterfs/glusterd.log: >>>>>> >>>>>> [2018-01-15 14:17:50.196228] I [MSGID: 106488] >>>>>> [glusterd-handler.c:1548:__glusterd_handle_cli_get_volume] >>>>>> 0-management: Received get vol req >>>>>> [2018-01-15 14:25:09.555214] I [MSGID: 106488] >>>>>> [glusterd-handler.c:1548:__glusterd_handle_cli_get_volume] >>>>>> 0-management: Received get vol req >>>>>> >>>>>> (empty because today it's 2018-01-16) >>>>>> >>>>>> /var/log/glusterfs/glustershd.log: >>>>>> >>>>>> [2018-01-14 02:23:02.731245] I >>>>>> [glusterfsd-mgmt.c:1821:mgmt_getspec_cbk] 0-glusterfs: No >>>>>> change in volfile,continuing >>>>>> >>>>>> (empty too) >>>>>> >>>>>> /var/log/glusterfs/bricks/brick-brick2-gv2a2.log (the >>>>>> interested volume): >>>>>> >>>>>> [2018-01-16 15:14:37.809965] I [MSGID: 115029] >>>>>> [server-handshake.c:793:server_setvolume] 0-gv2a2-server: >>>>>> accepted client from >>>>>> ovh-ov1-10302-2018/01/16-15:14:37:790306-gv2a2-client-0-0-0 >>>>>> (version: 3.12.4) >>>>>> [2018-01-16 15:16:41.471751] E [MSGID: 113020] >>>>>> [posix.c:1485:posix_mknod] 0-gv2a2-posix: setting gfid on >>>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.4 >>>>>> failed >>>>>> [2018-01-16 15:16:41.471745] W [MSGID: 113096] >>>>>> [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: >>>>>> link >>>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.4 >>>>>> -> >>>>>> /bricks/brick2/gv2a2/.glusterfs/a0/14/a0144df3-8d89-4aed-872e-5fef141e9e1efailed >>>>>> [File exists] >>>>>> [2018-01-16 15:16:42.593392] W [MSGID: 113096] >>>>>> [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: >>>>>> link >>>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.5 >>>>>> -> >>>>>> /bricks/brick2/gv2a2/.glusterfs/eb/04/eb044e6e-3a23-40a4-9ce1-f13af148eb67failed >>>>>> [File exists] >>>>>> [2018-01-16 15:16:42.593426] E [MSGID: 113020] >>>>>> [posix.c:1485:posix_mknod] 0-gv2a2-posix: setting gfid on >>>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.5 >>>>>> failed >>>>>> [2018-01-16 15:17:04.129593] W [MSGID: 113096] >>>>>> [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: >>>>>> link >>>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.8 >>>>>> -> >>>>>> /bricks/brick2/gv2a2/.glusterfs/dc/92/dc92bd0a-0d46-4826-a4c9-d073a924dd8dfailed >>>>>> [File exists] >>>>>> The message "W [MSGID: 113096] >>>>>> [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: >>>>>> link >>>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.8 >>>>>> -> >>>>>> /bricks/brick2/gv2a2/.glusterfs/dc/92/dc92bd0a-0d46-4826-a4c9-d073a924dd8dfailed >>>>>> [File exists]" repeated 5 times between [2018-01-16 >>>>>> 15:17:04.129593] and [2018-01-16 15:17:04.129593] >>>>>> [2018-01-16 15:17:04.129661] E [MSGID: 113020] >>>>>> [posix.c:1485:posix_mknod] 0-gv2a2-posix: setting gfid on >>>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.8 >>>>>> failed >>>>>> [2018-01-16 15:17:08.279162] W [MSGID: 113096] >>>>>> [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: >>>>>> link >>>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.9 >>>>>> -> >>>>>> /bricks/brick2/gv2a2/.glusterfs/c9/b7/c9b71b00-a09f-4df1-b874-041820ca8241failed >>>>>> [File exists] >>>>>> [2018-01-16 15:17:08.279162] W [MSGID: 113096] >>>>>> [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: >>>>>> link >>>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.9 >>>>>> -> >>>>>> /bricks/brick2/gv2a2/.glusterfs/c9/b7/c9b71b00-a09f-4df1-b874-041820ca8241failed >>>>>> [File exists] >>>>>> The message "W [MSGID: 113096] >>>>>> [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: >>>>>> link >>>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.9 >>>>>> -> >>>>>> /bricks/brick2/gv2a2/.glusterfs/c9/b7/c9b71b00-a09f-4df1-b874-041820ca8241failed >>>>>> [File exists]" repeated 2 times between [2018-01-16 >>>>>> 15:17:08.279162] and [2018-01-16 15:17:08.279162] >>>>>> >>>>>> [2018-01-16 15:17:08.279177] E [MSGID: 113020] >>>>>> [posix.c:1485:posix_mknod] 0-gv2a2-posix: setting gfid on >>>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.9 >>>>>> failed >>>>>> The message "W [MSGID: 113096] >>>>>> [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: >>>>>> link >>>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.4 >>>>>> -> >>>>>> /bricks/brick2/gv2a2/.glusterfs/a0/14/a0144df3-8d89-4aed-872e-5fef141e9e1efailed >>>>>> [File exists]" repeated 6 times between [2018-01-16 >>>>>> 15:16:41.471745] and [2018-01-16 15:16:41.471807] >>>>>> The message "W [MSGID: 113096] >>>>>> [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: >>>>>> link >>>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.5 >>>>>> -> >>>>>> /bricks/brick2/gv2a2/.glusterfs/eb/04/eb044e6e-3a23-40a4-9ce1-f13af148eb67failed >>>>>> [File exists]" repeated 2 times between [2018-01-16 >>>>>> 15:16:42.593392] and [2018-01-16 15:16:42.593430] >>>>>> [2018-01-16 15:17:32.229689] W [MSGID: 113096] >>>>>> [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: >>>>>> link >>>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.14 >>>>>> -> >>>>>> /bricks/brick2/gv2a2/.glusterfs/53/04/530449fa-d698-4928-a262-9a0234232323failed >>>>>> [File exists] >>>>>> [2018-01-16 15:17:32.229720] E [MSGID: 113020] >>>>>> [posix.c:1485:posix_mknod] 0-gv2a2-posix: setting gfid on >>>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.14 >>>>>> failed >>>>>> [2018-01-16 15:18:07.154330] W [MSGID: 113096] >>>>>> [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: >>>>>> link >>>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.17 >>>>>> -> >>>>>> /bricks/brick2/gv2a2/.glusterfs/81/96/8196dd19-84bc-4c3d-909f-8792e9b4929dfailed >>>>>> [File exists] >>>>>> [2018-01-16 15:18:07.154375] E [MSGID: 113020] >>>>>> [posix.c:1485:posix_mknod] 0-gv2a2-posix: setting gfid on >>>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.17 >>>>>> failed >>>>>> The message "W [MSGID: 113096] >>>>>> [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: >>>>>> link >>>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.14 >>>>>> -> >>>>>> /bricks/brick2/gv2a2/.glusterfs/53/04/530449fa-d698-4928-a262-9a0234232323failed >>>>>> [File exists]" repeated 7 times between [2018-01-16 >>>>>> 15:17:32.229689] and [2018-01-16 15:17:32.229806] >>>>>> The message "W [MSGID: 113096] >>>>>> [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: >>>>>> link >>>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.17 >>>>>> -> >>>>>> /bricks/brick2/gv2a2/.glusterfs/81/96/8196dd19-84bc-4c3d-909f-8792e9b4929dfailed >>>>>> [File exists]" repeated 3 times between [2018-01-16 >>>>>> 15:18:07.154330] and [2018-01-16 15:18:07.154357] >>>>>> [2018-01-16 15:19:23.618794] W [MSGID: 113096] >>>>>> [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: >>>>>> link >>>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.21 >>>>>> -> >>>>>> /bricks/brick2/gv2a2/.glusterfs/6d/02/6d02bd98-83de-43e8-a7af-b1d5f5160403failed >>>>>> [File exists] >>>>>> [2018-01-16 15:19:23.618827] E [MSGID: 113020] >>>>>> [posix.c:1485:posix_mknod] 0-gv2a2-posix: setting gfid on >>>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.21 >>>>>> failed >>>>>> The message "W [MSGID: 113096] >>>>>> [posix-handle.c:770:posix_handle_hard] 0-gv2a2-posix: >>>>>> link >>>>>> /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.21 >>>>>> -> >>>>>> /bricks/brick2/gv2a2/.glusterfs/6d/02/6d02bd98-83de-43e8-a7af-b1d5f5160403failed >>>>>> [File exists]" repeated 3 times between [2018-01-16 >>>>>> 15:19:23.618794] and [2018-01-16 15:19:23.618794] >>>>>> >>>>>> Thank you, >>>>>> >>>>>> >>>>>> Il 16/01/2018 11:40, Krutika Dhananjay ha scritto: >>>>>>> Also to help isolate the component, could you answer these: >>>>>>> >>>>>>> 1. on a different volume with shard not enabled, do you >>>>>>> see this issue? >>>>>>> 2. on a plain 3-way replicated volume (no arbiter), do >>>>>>> you see this issue? >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Tue, Jan 16, 2018 at 4:03 PM, Krutika Dhananjay >>>>>>> <kdhananj at redhat.com <mailto:kdhananj at redhat.com>> wrote: >>>>>>> >>>>>>> Please share the volume-info output and the logs >>>>>>> under /var/log/glusterfs/ from all your nodes. for >>>>>>> investigating the issue. >>>>>>> >>>>>>> -Krutika >>>>>>> >>>>>>> On Tue, Jan 16, 2018 at 1:30 PM, Ing. Luca Lazzeroni >>>>>>> - Trend Servizi Srl <luca at gvnet.it >>>>>>> <mailto:luca at gvnet.it>> wrote: >>>>>>> >>>>>>> Hi to everyone. >>>>>>> >>>>>>> I've got a strange problem with a gluster setup: >>>>>>> 3 nodes with Centos 7.4, Gluster 3.12.4 from >>>>>>> Centos/Gluster repositories, QEMU-KVM version >>>>>>> 2.9.0 (compiled from RHEL sources). >>>>>>> >>>>>>> I'm running volumes in replica 3 arbiter 1 mode >>>>>>> (but I've got a volume in "pure" replica 3 mode >>>>>>> too). I've applied the "virt" group settings to >>>>>>> my volumes since they host VM images. >>>>>>> >>>>>>> If I try to install something (eg: Ubuntu Server >>>>>>> 16.04.3) on a VM (and so I generate a bit of I/O >>>>>>> inside it) and configure KVM to access gluster >>>>>>> volume directly (via libvirt), install fails >>>>>>> after a while because the disk content is >>>>>>> corrupted. If I inspect the block inside the >>>>>>> disk (by accessing the image directly from >>>>>>> outside) I can found many files filled with "^@". >>>>>>> >>>>>>> >>>>>>> Also, what exactly do you mean by accessing the image >>>>>>> directly from outside? Was it from the brick directories >>>>>>> directly? Was it from the mount point of the volume? >>>>>>> Could you elaborate? Which files exactly did you check? >>>>>>> >>>>>>> -Krutika >>>>>>> >>>>>>> >>>>>>> If, instead, I configure KVM to access VM images >>>>>>> via a FUSE mount, everything seems to work >>>>>>> correctly. >>>>>>> >>>>>>> Note that the problem with install is verified >>>>>>> 100% time with QCOW2 image, while it appears >>>>>>> only after with RAW disk images. >>>>>>> >>>>>>> Is there anyone who experienced the same problem ? >>>>>>> >>>>>>> Thank you, >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Ing. Luca Lazzeroni >>>>>>> Responsabile Ricerca e Sviluppo >>>>>>> Trend Servizi Srl >>>>>>> Tel: 0376/631761 >>>>>>> Web: https://www.trendservizi.it >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Gluster-users mailing list >>>>>>> Gluster-users at gluster.org >>>>>>> <mailto:Gluster-users at gluster.org> >>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>> <http://lists.gluster.org/mailman/listinfo/gluster-users> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> Ing. Luca Lazzeroni >>>>>> Responsabile Ricerca e Sviluppo >>>>>> Trend Servizi Srl >>>>>> Tel: 0376/631761 >>>>>> Web:https://www.trendservizi.it >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Gluster-users mailing list >>>>>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> >>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>>> <http://lists.gluster.org/mailman/listinfo/gluster-users> >>>>> >>>>> -- >>>>> Ing. Luca Lazzeroni >>>>> Responsabile Ricerca e Sviluppo >>>>> Trend Servizi Srl >>>>> Tel: 0376/631761 >>>>> Web:https://www.trendservizi.it >>>>> >>>>> >>>>> _______________________________________________ >>>>> Gluster-users mailing list >>>>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> >>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>> <http://lists.gluster.org/mailman/listinfo/gluster-users> >>>> >>>> -- >>>> Ing. Luca Lazzeroni >>>> Responsabile Ricerca e Sviluppo >>>> Trend Servizi Srl >>>> Tel: 0376/631761 >>>> Web:https://www.trendservizi.it >>>> >>>> >>>> _______________________________________________ >>>> Gluster-users mailing list >>>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> >>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>> <http://lists.gluster.org/mailman/listinfo/gluster-users> >>>> >>>> >>> >>> -- >>> Ing. Luca Lazzeroni >>> Responsabile Ricerca e Sviluppo >>> Trend Servizi Srl >>> Tel: 0376/631761 >>> Web:https://www.trendservizi.it >>> >>> >> >> -- >> Ing. Luca Lazzeroni >> Responsabile Ricerca e Sviluppo >> Trend Servizi Srl >> Tel: 0376/631761 >> Web:https://www.trendservizi.it >> >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://lists.gluster.org/mailman/listinfo/gluster-users > > -- > Ing. Luca Lazzeroni > Responsabile Ricerca e Sviluppo > Trend Servizi Srl > Tel: 0376/631761 > Web:https://www.trendservizi.it > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users-- Ing. Luca Lazzeroni Responsabile Ricerca e Sviluppo Trend Servizi Srl Tel: 0376/631761 Web: https://www.trendservizi.it -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180118/772bbacb/attachment.html>
Apparently Analagous Threads
- [Possibile SPAM] Re: Problem with Gluster 3.12.4, VM and sharding
- [Possibile SPAM] Re: Problem with Gluster 3.12.4, VM and sharding
- [Possibile SPAM] Re: Problem with Gluster 3.12.4, VM and sharding
- [Possibile SPAM] Re: Problem with Gluster 3.12.4, VM and sharding
- [Possibile SPAM] Re: Problem with Gluster 3.12.4, VM and sharding