Ashish Pandey
2016-Jul-08 05:03 UTC
[Gluster-users] replace brick in distributed-dispersed setup
Hi I?aki The steps you are following don't have any issue. I would like to have more information to debug this further. 1 - gluster v info <volname> 2 - gluster v status <volname> before and after running replace-brick 3 - Brick logs (for this volume only) from /var/log/glusterfs/bricks/ 4 - glusterd logs /var/log/glusterfs/ starts with "usr-local-etc-glusterfs-glusterd-<volname>" Although it should not matter, could you also try to replace a brick without killing that brick process? Ashish ----- Original Message ----- From: "itlinux_team" <itlinux at imppc.org> To: gluster-users at gluster.org Sent: Wednesday, July 6, 2016 4:33:54 PM Subject: [Gluster-users] replace brick in distributed-dispersed setup Hi all, I'm doing some testings with glusterfs in a virtualized environment running a 3 x (8 + 4) distributed-dispersed volume simulating a 3 node cluster with 12 drives per node configuration. The system versions are: OS: Debian jessie kernel 3.16 Gluster: 3.8.0-2 installed from the gluster.org debian repository I have tested the node failure scenario while some clients are running some read/write operations and the setup works as expected. Now I'm trying to test how to replace a faulty drive on this setup, however I'm not able to replace a brick. To test it I have: 1: Find the pid of the brick I'd like to 'fail' and kill the process. (tried removing the drive from the host but that would make the whole guest unresponsive) 2: Attach a new virtual drive, format and mount it 3: Try the gluster volume replace-brick command And I'm getting the following error: gluster volume replace-brick vol_1 glusterserver1:/ext/bricks/brick-1 glusterserver1:/ext/bricks/brick-13 commit force volume replace-brick: failed: Fuse unavailable Replace-brick failed I assume I'm doing something wrong but don't know what exactly. Looking in the documentation I have not found information about brick replacement in distributed-dispersed setups. Thanks! I?aki _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://www.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160708/4211003c/attachment.html>
itlinux_team
2016-Jul-08 13:10 UTC
[Gluster-users] replace brick in distributed-dispersed setup
Hi Ashish, It was an error on my side, nothing gluster related. The kernel version I was running had a bug that prevented the fuse module from loading and causing the brick replacement errors. After upgrading I can confirm that the process of replacing the brick works fine both with the brick to be replaced online or after killing the brick process. I have a question though. In a real (non virtual) server, if I setup the server in JBOD mode, and a drive fails, does gluster kill the brick pid? Regards, I?aki. On 07/08/2016 07:03 AM, Ashish Pandey wrote:> > Hi I?aki > > The steps you are following don't have any issue. > I would like to have more information to debug this further. > > 1 - gluster v info <volname> > 2 - gluster v status <volname> before and after running replace-brick > 3 - Brick logs (for this volume only) from /var/log/glusterfs/bricks/ > 4 - glusterd logs /var/log/glusterfs/ starts with > "usr-local-etc-glusterfs-glusterd-<volname>" > > Although it should not matter, could you also try to replace a brick > without killing that brick process? > > Ashish > > ------------------------------------------------------------------------ > *From: *"itlinux_team" <itlinux at imppc.org> > *To: *gluster-users at gluster.org > *Sent: *Wednesday, July 6, 2016 4:33:54 PM > *Subject: *[Gluster-users] replace brick in distributed-dispersed setup > > Hi all, > > I'm doing some testings with glusterfs in a virtualized environment > running a 3 x (8 + 4) distributed-dispersed volume simulating a 3 node > cluster with 12 drives per node configuration. The system versions are: > > OS: Debian jessie kernel 3.16 > > Gluster: 3.8.0-2 installed from the gluster.org debian repository > > I have tested the node failure scenario while some clients are running > some read/write operations and the setup works as expected. Now I'm > trying to test how to replace a faulty drive on this setup, however I'm > not able to replace a brick. To test it I have: > > 1: Find the pid of the brick I'd like to 'fail' and kill the process. > (tried removing the drive from the host but that would make the whole > guest unresponsive) > > 2: Attach a new virtual drive, format and mount it > > 3: Try the gluster volume replace-brick command > > And I'm getting the following error: > > gluster volume replace-brick vol_1 glusterserver1:/ext/bricks/brick-1 > glusterserver1:/ext/bricks/brick-13 commit force > volume replace-brick: failed: Fuse unavailable > Replace-brick failed > > I assume I'm doing something wrong but don't know what exactly. Looking > in the documentation I have not found information about brick > replacement in distributed-dispersed setups. > > > Thanks! > > I?aki > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160708/21e172bf/attachment.html>