Krutika Dhananjay
2015-Nov-14 03:45 UTC
[Gluster-users] File Corruption with shards - 100% reproducable
The logs are at /var/log/glusterfs/<hyphenated-path-to-the-mountpoint>.log OK. So what do you observe when you set group virt to on? # gluster volume set <VOL> group virt -Krutika ----- Original Message -----> From: "Lindsay Mathieson" <lindsay.mathieson at gmail.com> > To: "Krutika Dhananjay" <kdhananj at redhat.com> > Cc: "gluster-users" <gluster-users at gluster.org> > Sent: Friday, November 13, 2015 11:57:15 AM > Subject: Re: [Gluster-users] File Corruption with shards - 100% reproducable> On 12 November 2015 at 15:46, Krutika Dhananjay < kdhananj at redhat.com > > wrote:> > OK. What do the client logs say? >> Dumb question - Which logs are those?> > Could you share the exact steps to recreate this, and I will try it locally > > on my setup? >> I'm running this on a 3 node proxmox cluster, which makes the vm creation & > migration easy to test.> Steps: > - Create 3 node gluster datastore using proxmox vm host nodes> - Add gluster datastore as a storage dvice to proxmox > * qemu vms use the gfapi to access the datastore > * proxmox also adds a fuse mount for easy acces> - create a VM on the gluster storage, QCOW2 format. I just created a simple > debain Mate vm> - start the vm, open a console to it.> - live migrate the VM to a another node> - It will rapdily barf itself with disk errors> - stop the VM> - qemu will show file corruption (many many errors) > * qemu-img check <vm disk image> > * qemu-img info <vm disk image>> Repeating the process with sharding off has no errors.> > Also, want to see the output of 'gluster volume info'. >> I've trimmed settings down to a bare minimum. This is a test gluster cluster > so I can do with it as I wish.> gluster volume info> Volume Name: datastore1 > Type: Replicate > Volume ID: 238fddd0-a88c-4edb-8ac5-ef87c58682bf > Status: Started > Number of Bricks: 1 x 3 = 3 > Transport-type: tcp > Bricks: > Brick1: vnb.proxmox.softlog:/mnt/ext4 > Brick2: vng.proxmox.softlog:/mnt/ext4 > Brick3: vna.proxmox.softlog:/mnt/ext4 > Options Reconfigured: > performance.strict-write-ordering: on > performance.readdir-ahead: off > cluster.quorum-type: auto > features.shard: on> -- > Lindsay-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20151113/49ad398d/attachment.html>
Lindsay Mathieson
2015-Nov-14 05:21 UTC
[Gluster-users] File Corruption with shards - 100% reproducable
gluster volume set datastore1 group virt Unable to open file '/var/lib/glusterd/groups/virt'. Error: No such file or directory Not sure I understand this one ? couldn?t find any docs for it. Sent from Mail for Windows 10 From: Krutika Dhananjay Sent: Saturday, 14 November 2015 1:45 PM To: Lindsay Mathieson Cc: gluster-users Subject: Re: [Gluster-users] File Corruption with shards - 100% reproducable The logs are at /var/log/glusterfs/<hyphenated-path-to-the-mountpoint>.log OK. So what do you observe when you set group virt to on? # gluster volume set <VOL> group virt -Krutika From: "Lindsay Mathieson" <lindsay.mathieson at gmail.com> To: "Krutika Dhananjay" <kdhananj at redhat.com> Cc: "gluster-users" <gluster-users at gluster.org> Sent: Friday, November 13, 2015 11:57:15 AM Subject: Re: [Gluster-users] File Corruption with shards - 100% reproducable On 12 November 2015 at 15:46, Krutika Dhananjay <kdhananj at redhat.com> wrote: OK. What do the client logs say? Dumb question - Which logs are those?? Could you share the exact steps to recreate this, and I will try it locally on my setup? I'm running this on a 3 node proxmox cluster, which makes the vm creation & migration easy to test. Steps: - Create 3 node gluster datastore using proxmox vm host nodes - Add gluster datastore as a storage dvice to proxmox ? * qemu vms use the gfapi to access the datastore ? * proxmox also adds a fuse mount for easy acces - create a VM on the gluster storage, QCOW2 format. I just created a simple debain Mate vm - start the vm, open a console to it. - live migrate the VM to a another node - It will rapdily barf itself with disk errors - stop the VM - qemu will show file corruption (many many errors) ? * qemu-img check <vm disk image> ? * qemu-img info <vm disk image> Repeating the process with sharding off has no errors. ? Also, want to see the output of 'gluster volume info'. I've trimmed settings down to a bare minimum. This is a test gluster cluster so I can do with it as I wish. gluster volume info ? Volume Name: datastore1 Type: Replicate Volume ID: 238fddd0-a88c-4edb-8ac5-ef87c58682bf Status: Started Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: vnb.proxmox.softlog:/mnt/ext4 Brick2: vng.proxmox.softlog:/mnt/ext4 Brick3: vna.proxmox.softlog:/mnt/ext4 Options Reconfigured: performance.strict-write-ordering: on performance.readdir-ahead: off cluster.quorum-type: auto features.shard: on -- Lindsay -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20151114/eb1bf38d/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: 087031D8F78147B19543CB04F52FB3EB.png Type: image/png Size: 148 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20151114/eb1bf38d/attachment.png>
Lindsay Mathieson
2015-Nov-14 10:09 UTC
[Gluster-users] File Corruption with shards - 100% reproducable
On 14 November 2015 at 13:45, Krutika Dhananjay <kdhananj at redhat.com> wrote:> The logs are at /var/log/glusterfs/<hyphenated-path-to-the-mountpoint>.log >Attached are the logs for node vnb & vna. I started the VM on vnb and migrated it to vng vnb => vng NB: The actual access of the VM image is not done via the fuse mount, but using gfapi direct, so I'm not sure what log is relevant for that. Have included them all. Thanks. -- Lindsay -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20151114/8823cf0c/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: vnb.tgz Type: application/x-gzip Size: 5724 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20151114/8823cf0c/attachment.tgz> -------------- next part -------------- A non-text attachment was scrubbed... Name: vng.tgz Type: application/x-gzip Size: 5113 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20151114/8823cf0c/attachment-0001.tgz>