Hi, I have upgraded to Gluster 3.8.10 today and ran the add-brick procedure in a volume contains few VMs. After the completion of rebalance, i have rebooted the VMs, some of ran just fine, and others just crashed. Windows boot to recovery mode and Linux throw xfs errors and does not boot. I ran the test again and it happened just as the first one, but i have noticed only VMs doing disk IOs are affected by this bug. The VMs in power off mode started fine and even md5 of the disk file did not change after the rebalance. anyone else can confirm this ? Volume info: Volume Name: vmware2 Type: Distributed-Replicate Volume ID: 02328d46-a285-4533-aa3a-fb9bfeb688bf Status: Started Snapshot Count: 0 Number of Bricks: 22 x 2 = 44 Transport-type: tcp Bricks: Brick1: gluster01:/mnt/disk1/vmware2 Brick2: gluster03:/mnt/disk1/vmware2 Brick3: gluster02:/mnt/disk1/vmware2 Brick4: gluster04:/mnt/disk1/vmware2 Brick5: gluster01:/mnt/disk2/vmware2 Brick6: gluster03:/mnt/disk2/vmware2 Brick7: gluster02:/mnt/disk2/vmware2 Brick8: gluster04:/mnt/disk2/vmware2 Brick9: gluster01:/mnt/disk3/vmware2 Brick10: gluster03:/mnt/disk3/vmware2 Brick11: gluster02:/mnt/disk3/vmware2 Brick12: gluster04:/mnt/disk3/vmware2 Brick13: gluster01:/mnt/disk4/vmware2 Brick14: gluster03:/mnt/disk4/vmware2 Brick15: gluster02:/mnt/disk4/vmware2 Brick16: gluster04:/mnt/disk4/vmware2 Brick17: gluster01:/mnt/disk5/vmware2 Brick18: gluster03:/mnt/disk5/vmware2 Brick19: gluster02:/mnt/disk5/vmware2 Brick20: gluster04:/mnt/disk5/vmware2 Brick21: gluster01:/mnt/disk6/vmware2 Brick22: gluster03:/mnt/disk6/vmware2 Brick23: gluster02:/mnt/disk6/vmware2 Brick24: gluster04:/mnt/disk6/vmware2 Brick25: gluster01:/mnt/disk7/vmware2 Brick26: gluster03:/mnt/disk7/vmware2 Brick27: gluster02:/mnt/disk7/vmware2 Brick28: gluster04:/mnt/disk7/vmware2 Brick29: gluster01:/mnt/disk8/vmware2 Brick30: gluster03:/mnt/disk8/vmware2 Brick31: gluster02:/mnt/disk8/vmware2 Brick32: gluster04:/mnt/disk8/vmware2 Brick33: gluster01:/mnt/disk9/vmware2 Brick34: gluster03:/mnt/disk9/vmware2 Brick35: gluster02:/mnt/disk9/vmware2 Brick36: gluster04:/mnt/disk9/vmware2 Brick37: gluster01:/mnt/disk10/vmware2 Brick38: gluster03:/mnt/disk10/vmware2 Brick39: gluster02:/mnt/disk10/vmware2 Brick40: gluster04:/mnt/disk10/vmware2 Brick41: gluster01:/mnt/disk11/vmware2 Brick42: gluster03:/mnt/disk11/vmware2 Brick43: gluster02:/mnt/disk11/vmware2 Brick44: gluster04:/mnt/disk11/vmware2 Options Reconfigured: cluster.server-quorum-type: server nfs.disable: on performance.readdir-ahead: on transport.address-family: inet performance.quick-read: off performance.read-ahead: off performance.io-cache: off performance.stat-prefetch: off cluster.eager-lock: enable network.remote-dio: enable features.shard: on cluster.data-self-heal-algorithm: full features.cache-invalidation: on ganesha.enable: on features.shard-block-size: 256MB client.event-threads: 2 server.event-threads: 2 cluster.favorite-child-policy: size storage.build-pgfid: off network.ping-timeout: 5 cluster.enable-shared-storage: enable nfs-ganesha: enable cluster.server-quorum-ratio: 51% Adding bricks: gluster volume add-brick vmware2 replica 2 gluster01:/mnt/disk11/vmware2 gluster03:/mnt/disk11/vmware2 gluster02:/mnt/disk11/vmware2 gluster04:/mnt/disk11/vmware2 starting fix layout: gluster volume rebalance vmware2 fix-layout start Starting rebalance: gluster volume rebalance vmware2 start -- Respectfully Mahdi A. Mahdi -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170317/9bd6fd20/attachment.html>
Lindsay Mathieson
2017-Mar-18 01:09 UTC
[Gluster-users] Gluster 3.8.10 rebalance VMs corruption
On 18/03/2017 12:20 AM, Mahdi Adnan wrote:> > I have upgraded to Gluster 3.8.10 today and ran the add-brick > procedure in a volume contains few VMs. > > After the completion of rebalance, i have rebooted the VMs, some of > ran just fine, and others just crashed. > > Windows boot to recovery mode and Linux throw xfs errors and does not > boot. > > I ran the test again and it happened just as the first one, but i have > noticed only VMs doing disk IOs are affected by this bug. > > The VMs in power off mode started fine and even md5 of the disk file > did not change after the rebalance. > >Concerning, this was supposed to be fixed in 3.8.10 -- Lindsay Mathieson