Apologies for the delay!>From the logs:[2017-05-31 13:06:35.839780] E [MSGID: 106005] [glusterd-utils.c:4877:glusterd_brick_start] 0-management: Unable to start brick 156.125.102.103:/data/br0/vm0 [2017-05-31 13:06:35.839869] E [MSGID: 106074] [glusterd-brick-ops.c:2493:glusterd_op_add_brick] 0-glusterd: Unable to add bricks This looks like in serv3 the newly added brick failed to come up. You'd have to look at the same brick log file on serv3 to see what was the reason. As a workaround, does gluster v start <volname> force bring up this brick now? As far as glusterd configuration data is concerned, everything seems fine, no inconsistency. On Wed, May 31, 2017 at 6:53 PM, Merwan Ouddane <merwan19 at msn.com> wrote:> Hello, > > > Here is a tar gz with the log, and the output of "gluster volume status > && gluster volume info && gluster peer status" for each server > > > Thank you, > > Merwan > ------------------------------ > *De :* Atin Mukherjee <amukherj at redhat.com> > *Envoy? :* mercredi 31 mai 2017 12:20:49 > > *? :* Merwan Ouddane > *Cc :* gluster-users at gluster.org > *Objet :* Re: [Gluster-users] Adding a new replica to the cluster > > > On Tue, May 30, 2017 at 2:14 PM, Merwan Ouddane <merwan19 at msn.com> wrote: > >> Gluster peer status tells me that the peer is connected >> >> >> Here is the log from the servers in the cluster (they are the same for >> both of them): >> >> https://pastebin.com/4aM94PJ6 >> >> The log from the server i'm trying to add to the cluster: >> >> https://pastebin.com/YGTcKRyw >> > I'm sorry but these log files are for brick and client. What I am looking > for is glusterd log ( ls -lrt /var/log/glusterfs/*glusterd* ) from all the > nodes. Also please paste the output of gluster volume status, gluster > volume info, gluster peer status from all the nodes. > >> >> >> Merwan >> >> ------------------------------ >> *De :* Atin Mukherjee <amukherj at redhat.com> >> *Envoy? :* mardi 30 mai 2017 04:51 >> *? :* Merwan Ouddane >> *Cc :* gluster-users at gluster.org >> *Objet :* Re: [Gluster-users] Adding a new replica to the cluster >> >> >> >> On Mon, May 29, 2017 at 9:52 PM, Merwan Ouddane <merwan19 at msn.com> wrote: >> >>> Hello, >>> >>> I wanted to play around with gluster and I made a 2 nodes cluster >>> replicated, then I wanted to add a third replica "on the fly". >>> >>> I manage to probe my third server from the cluster, but when I try to >>> add the new brick to the volume, I get a "Request timed out" >>> >>> My command: >>> gluster volume add-brick vol0 replica 3 192.168.0.4:/data/br0/vol0 >>> >> >> This might happen if the originator glusterd lost connection with other >> peers while this command was in process. Can you check if peer status is >> healthy by gluster peer status command? If yes, you'd need to share the >> glusterd log files from all the nodes with us to further analyse the issue. >> >> >> >>> >>> Does anyone have an idea ? >>> >>> Regards, >>> Merwan >>> >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> http://lists.gluster.org/mailman/listinfo/gluster-users >>> >> >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://lists.gluster.org/mailman/listinfo/gluster-users >> > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170604/e2966778/attachment.html>
I get a timeout when trying to force start the volume from serv3, but I can stop the volume ... Isn't the error linked to "0-glusterfs: failed to submit rpc-request" ? (from the log i send in the first place) Merwan ________________________________ De : Atin Mukherjee <amukherj at redhat.com> Envoy? : dimanche 4 juin 2017 15:31:34 ? : Merwan Ouddane Cc : gluster-users Objet : Re: [Gluster-users] Adding a new replica to the cluster Apologies for the delay!>From the logs:[2017-05-31 13:06:35.839780] E [MSGID: 106005] [glusterd-utils.c:4877:glusterd_brick_start] 0-management: Unable to start brick 156.125.102.103:/data/br0/vm0 [2017-05-31 13:06:35.839869] E [MSGID: 106074] [glusterd-brick-ops.c:2493:glusterd_op_add_brick] 0-glusterd: Unable to add bricks This looks like in serv3 the newly added brick failed to come up. You'd have to look at the same brick log file on serv3 to see what was the reason. As a workaround, does gluster v start <volname> force bring up this brick now? As far as glusterd configuration data is concerned, everything seems fine, no inconsistency. On Wed, May 31, 2017 at 6:53 PM, Merwan Ouddane <merwan19 at msn.com<mailto:merwan19 at msn.com>> wrote: Hello, Here is a tar gz with the log, and the output of "gluster volume status && gluster volume info && gluster peer status" for each server Thank you, Merwan ________________________________ De : Atin Mukherjee <amukherj at redhat.com<mailto:amukherj at redhat.com>> Envoy? : mercredi 31 mai 2017 12:20:49 ? : Merwan Ouddane Cc : gluster-users at gluster.org<mailto:gluster-users at gluster.org> Objet : Re: [Gluster-users] Adding a new replica to the cluster On Tue, May 30, 2017 at 2:14 PM, Merwan Ouddane <merwan19 at msn.com<mailto:merwan19 at msn.com>> wrote: Gluster peer status tells me that the peer is connected Here is the log from the servers in the cluster (they are the same for both of them): https://pastebin.com/4aM94PJ6 The log from the server i'm trying to add to the cluster: https://pastebin.com/YGTcKRyw I'm sorry but these log files are for brick and client. What I am looking for is glusterd log ( ls -lrt /var/log/glusterfs/*glusterd* ) from all the nodes. Also please paste the output of gluster volume status, gluster volume info, gluster peer status from all the nodes. Merwan ________________________________ De : Atin Mukherjee <amukherj at redhat.com<mailto:amukherj at redhat.com>> Envoy? : mardi 30 mai 2017 04:51 ? : Merwan Ouddane Cc : gluster-users at gluster.org<mailto:gluster-users at gluster.org> Objet : Re: [Gluster-users] Adding a new replica to the cluster On Mon, May 29, 2017 at 9:52 PM, Merwan Ouddane <merwan19 at msn.com<mailto:merwan19 at msn.com>> wrote: Hello, I wanted to play around with gluster and I made a 2 nodes cluster replicated, then I wanted to add a third replica "on the fly". I manage to probe my third server from the cluster, but when I try to add the new brick to the volume, I get a "Request timed out" My command: gluster volume add-brick vol0 replica 3 192.168.0.4<http://192.168.0.4>:/data/br0/vol0 This might happen if the originator glusterd lost connection with other peers while this command was in process. Can you check if peer status is healthy by gluster peer status command? If yes, you'd need to share the glusterd log files from all the nodes with us to further analyse the issue. Does anyone have an idea ? Regards, Merwan _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org<mailto:Gluster-users at gluster.org> http://lists.gluster.org/mailman/listinfo/gluster-users _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org<mailto:Gluster-users at gluster.org> http://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170606/f0ca7fa1/attachment.html>
Can you please send the brick log files from serv3? On Tue, Jun 6, 2017 at 7:56 PM, Merwan Ouddane <merwan19 at msn.com> wrote:> I get a timeout when trying to force start the volume from serv3, but I > can stop the volume ... > > Isn't the error linked to "0-glusterfs: failed to submit rpc-request" ? > (from the log i send in the first place) > > Merwan > > ------------------------------ > *De :* Atin Mukherjee <amukherj at redhat.com> > *Envoy? :* dimanche 4 juin 2017 15:31:34 > *? :* Merwan Ouddane > *Cc :* gluster-users > > *Objet :* Re: [Gluster-users] Adding a new replica to the cluster > > Apologies for the delay! > > From the logs: > > [2017-05-31 13:06:35.839780] E [MSGID: 106005] [glusterd-utils.c:4877:glusterd_brick_start] > 0-management: Unable to start brick 156.125.102.103:/data/br0/vm0 > > > [2017-05-31 13:06:35.839869] E [MSGID: 106074] [glusterd-brick-ops.c:2493:glusterd_op_add_brick] > 0-glusterd: Unable to add bricks > > This looks like in serv3 the newly added brick failed to come up. You'd > have to look at the same brick log file on serv3 to see what was the > reason. As a workaround, does gluster v start <volname> force bring up this > brick now? As far as glusterd configuration data is concerned, everything > seems fine, no inconsistency. > > On Wed, May 31, 2017 at 6:53 PM, Merwan Ouddane <merwan19 at msn.com> wrote: > >> Hello, >> >> >> Here is a tar gz with the log, and the output of "gluster volume status >> && gluster volume info && gluster peer status" for each server >> >> >> Thank you, >> >> Merwan >> ------------------------------ >> *De :* Atin Mukherjee <amukherj at redhat.com> >> *Envoy? :* mercredi 31 mai 2017 12:20:49 >> >> *? :* Merwan Ouddane >> *Cc :* gluster-users at gluster.org >> *Objet :* Re: [Gluster-users] Adding a new replica to the cluster >> >> >> On Tue, May 30, 2017 at 2:14 PM, Merwan Ouddane <merwan19 at msn.com> wrote: >> >>> Gluster peer status tells me that the peer is connected >>> >>> >>> Here is the log from the servers in the cluster (they are the same for >>> both of them): >>> >>> https://pastebin.com/4aM94PJ6 >>> >>> The log from the server i'm trying to add to the cluster: >>> >>> https://pastebin.com/YGTcKRyw >>> >> I'm sorry but these log files are for brick and client. What I am looking >> for is glusterd log ( ls -lrt /var/log/glusterfs/*glusterd* ) from all the >> nodes. Also please paste the output of gluster volume status, gluster >> volume info, gluster peer status from all the nodes. >> >>> >>> >>> Merwan >>> >>> ------------------------------ >>> *De :* Atin Mukherjee <amukherj at redhat.com> >>> *Envoy? :* mardi 30 mai 2017 04:51 >>> *? :* Merwan Ouddane >>> *Cc :* gluster-users at gluster.org >>> *Objet :* Re: [Gluster-users] Adding a new replica to the cluster >>> >>> >>> >>> On Mon, May 29, 2017 at 9:52 PM, Merwan Ouddane <merwan19 at msn.com> >>> wrote: >>> >>>> Hello, >>>> >>>> I wanted to play around with gluster and I made a 2 nodes cluster >>>> replicated, then I wanted to add a third replica "on the fly". >>>> >>>> I manage to probe my third server from the cluster, but when I try to >>>> add the new brick to the volume, I get a "Request timed out" >>>> >>>> My command: >>>> gluster volume add-brick vol0 replica 3 192.168.0.4:/data/br0/vol0 >>>> >>> >>> This might happen if the originator glusterd lost connection with other >>> peers while this command was in process. Can you check if peer status is >>> healthy by gluster peer status command? If yes, you'd need to share the >>> glusterd log files from all the nodes with us to further analyse the issue. >>> >>> >>> >>>> >>>> Does anyone have an idea ? >>>> >>>> Regards, >>>> Merwan >>>> >>>> _______________________________________________ >>>> Gluster-users mailing list >>>> Gluster-users at gluster.org >>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>> >>> >>> >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> http://lists.gluster.org/mailman/listinfo/gluster-users >>> >> >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170606/06b2e399/attachment.html>