Kaushal M
2015-May-13 07:41 UTC
[Gluster-users] Adding a 3.6.3 peer to a 3.6.2 trusted pool fails
Hey Adrian, Can you list your volume infos? Most of the times the reason for peer state being stuck is because of some volume information being incorrect and the volume infos can help debug the problem. Could you also start glusterd in debug mode (`glusterd -LDEBUG`) and give logs? ~kaushal On May 13, 2015 9:13 AM, "Adri?n Santos Marrero" <asmarre at ull.edu.es> wrote:> Hi again, > > I upgraded to 3.6.3 all my gluster servers but peer probes still get stuck > in "Accepted peer request". > > Any hint? > > Regards. > > 2015-05-11 13:55 GMT+01:00 Adri?n Santos Marrero <asmarre at ull.edu.es>: > >> Hi, I've the same error with gluster03 (first installed with 3.6.2 >> version, but upgraded to 3.6.3 before any peer probe) and gluster04 (fresh >> install with Gluster 3.6.3). I attach the whole content of >> /var/log/glusterfs/ directory from gluster03. >> >> Regards. >> >> 2015-05-11 12:06 GMT+01:00 Gaurav Garg <ggarg at redhat.com>: >> >>> Hi Adrian, >>> >>> Did you try to upgrade/downgrade gluster03 before performing any >>> operation. If you attach fresh node (3.6.3 version) to (3.6.2) glusterfs >>> version then this problems is not coming. we need to know what operation >>> you performed on gluster03 node before peer probe. if possible could you >>> attach cmd_history.log and glusterd log. >>> >>> ~ Gaurav >>> >>> ----- Original Message ----- >>> From: "Adri?n Santos Marrero" <asmarre at ull.edu.es> >>> To: "Gluster-users" <gluster-users at gluster.org> >>> Sent: Monday, May 11, 2015 12:39:33 PM >>> Subject: [Gluster-users] Adding a 3.6.3 peer to a 3.6.2 trusted pool >>> fails >>> >>> Hi, >>> >>> I've been trying to add a third node to my two-node trusted pool with >>> gluster peer probe: >>> >>> >>> >>> gluster02:~# gluster peer probe gluster03.stic.ull.es >>> peer probe: success. >>> >>> The problem is that this node remains in "Accepted peer request" state: >>> >>> >>> >>> Hostname: gluster03.stic.ull.es >>> Uuid: 9ca23fb2-cd76-4d1c-8a5f-f93306119539 >>> State: Accepted peer request (Connected) >>> >>> If I restart glusterd in gluster02, the state changes to "Sent and >>> Received peer request": >>> >>> >>> >>> Hostname: gluster03.stic.ull.es >>> Uuid: 9ca23fb2-cd76-4d1c-8a5f-f93306119539 >>> State: Sent and Received peer request (Connected) >>> >>> And if I restart glusterd in gluster03 it doesn't start because is >>> "Unable to find friend: gluster01.stic.ull.es ": >>> >>> >>> >>> [2015-05-11 06:59:39.035184] D >>> [glusterd-peer-utils.c:121:glusterd_peerinfo_find_by_hostname] >>> 0-management: Unable to find friend: gluster01.stic.ull.es >>> [2015-05-11 06:59:39.037942] D [common-utils.c:2897:gf_is_local_addr] >>> 0-management: 10.107.34.17 >>> [2015-05-11 06:59:39.039392] D [common-utils.c:2897:gf_is_local_addr] >>> 0-management: 10.107.34.17 >>> [2015-05-11 06:59:39.040584] D [common-utils.c:2897:gf_is_local_addr] >>> 0-management: 10.107.34.17 >>> [2015-05-11 06:59:39.041790] D [common-utils.c:2913:gf_is_local_addr] >>> 0-management: gluster01.stic.ull.es is not local >>> >>> In fact, the only peer that gluster03 knows is gluster02: >>> >>> >>> >>> gluster03:~# ls -l /var/lib/glusterd/peers/ >>> total 4 >>> -rw-------. 1 root root 73 may 11 07:48 >>> 739d7c3f-99d5-4c8b-ac28-f85163e26322 >>> gluster03:~# cat >>> /var/lib/glusterd/peers/739d7c3f-99d5-4c8b-ac28-f85163e26322 >>> uuid=739d7c3f-99d5-4c8b-ac28-f85163e26322 >>> state=5 >>> hostname1=10.107.34.18 >>> >>> 10.107.34.18 == gluster02 >>> >>> >>> >>> gluster03:~# gluster --version >>> glusterfs 3.6.3 built on Apr 23 2015 16:11:45 >>> Repository revision: git:// git.gluster.com/glusterfs.git >>> Copyright (c) 2006-2011 Gluster Inc. < http://www.gluster.com > >>> GlusterFS comes with ABSOLUTELY NO WARRANTY. >>> You may redistribute copies of GlusterFS under the terms of the GNU >>> General Public License. >>> >>> All servers are running Centos 7, gluster01 and gluster02 have Gluster >>> 3.6.2 version and gluster03 3.6.3. >>> >>> Am I forced to upgrade gluster01 and gluster02 to 3.6.3? >>> >>> Regards. >>> >>> -- >>> ---------------------------------------------------------------- >>> Adri?n Santos Marrero >>> T?cnico de Sistemas - ?rea de Infraestructuras TIC >>> Servicios de Tecnolog?as de la Informaci?n y Comunicaci?n (STIC) >>> Universidad de La Laguna (ULL) >>> Tel?fono/Phone: +34 922 845089 >>> ---------------------------------------------------------------- >>> >>> Este mensaje puede contener informaci?n confidencial y/o privilegiada. >>> Si usted no es el destinatario o lo ha recibido por error debe borrarlo >>> inmediatamente. >>> Est? estrictamente prohibido por la legislaci?n vigente realizar >>> cualquier copia, revelaci?n o distribuci?n del contenido de este mensaje >>> sin autorizaci?n expresa. >>> >>> This e-mail may contain confidential and/or privileged information. >>> If you are not the intended recipient or have received this e-mail in >>> error you must destroy it. >>> Any unauthorised copying, disclosure or distribution of the material in >>> this e-mail is strictly forbidden by current legislation. >>> >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> http://www.gluster.org/mailman/listinfo/gluster-users >>> >> >> >> >> -- >> ---------------------------------------------------------------- >> Adri?n Santos Marrero >> T?cnico de Sistemas - ?rea de Infraestructuras TIC >> Servicios de Tecnolog?as de la Informaci?n y Comunicaci?n (STIC) >> Universidad de La Laguna (ULL) >> Tel?fono/Phone: +34 922 845089 >> ---------------------------------------------------------------- >> >> Este mensaje puede contener informaci?n confidencial y/o privilegiada. >> Si usted no es el destinatario o lo ha recibido por error debe borrarlo >> inmediatamente. >> Est? estrictamente prohibido por la legislaci?n vigente realizar >> cualquier copia, revelaci?n o distribuci?n del contenido de este mensaje >> sin autorizaci?n expresa. >> >> This e-mail may contain confidential and/or privileged information. >> If you are not the intended recipient or have received this e-mail in >> error you must destroy it. >> Any unauthorised copying, disclosure or distribution of the material in >> this e-mail is strictly forbidden by current legislation. >> > > > > -- > ---------------------------------------------------------------- > Adri?n Santos Marrero > T?cnico de Sistemas - ?rea de Infraestructuras TIC > Servicios de Tecnolog?as de la Informaci?n y Comunicaci?n (STIC) > Universidad de La Laguna (ULL) > Tel?fono/Phone: +34 922 845089 > ---------------------------------------------------------------- > > Este mensaje puede contener informaci?n confidencial y/o privilegiada. > Si usted no es el destinatario o lo ha recibido por error debe borrarlo > inmediatamente. > Est? estrictamente prohibido por la legislaci?n vigente realizar > cualquier copia, revelaci?n o distribuci?n del contenido de este mensaje > sin autorizaci?n expresa. > > This e-mail may contain confidential and/or privileged information. > If you are not the intended recipient or have received this e-mail in > error you must destroy it. > Any unauthorised copying, disclosure or distribution of the material in > this e-mail is strictly forbidden by current legislation. > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150513/6d8ecf49/attachment.html>
Adrián Santos Marrero
2015-May-13 08:07 UTC
[Gluster-users] Adding a 3.6.3 peer to a 3.6.2 trusted pool fails
Hi Kaushal, Volume info:> gluster02:~# gluster volume info all > > Volume Name: ovirt1 > Type: Replicate > Volume ID: 05171231-a4a5-48ae-bcfe-63408bc0fc0c > Status: Started > Number of Bricks: 1 x 2 = 2 > Transport-type: tcp > Bricks: > Brick1: gluster01.stic.ull.es:/gluster/ovirt1_brick_01/brick > Brick2: gluster02.stic.ull.es:/gluster/ovirt1_brick_01/brick > Options Reconfigured: > storage.owner-gid: 36 > storage.owner-uid: 36 > cluster.server-quorum-type: none > network.remote-dio: enable > cluster.eager-lock: enable > performance.stat-prefetch: off > performance.io-cache: off > performance.read-ahead: off > performance.quick-read: off > > Volume Name: ovirt2 > Type: Replicate > Volume ID: 0c0be558-9d98-4403-82d4-faafa2a3630a > Status: Started > Number of Bricks: 1 x 2 = 2 > Transport-type: tcp > Bricks: > Brick1: gluster01.stic.ull.es:/gluster/ovirt2_brick_01/brick > Brick2: gluster02.stic.ull.es:/gluster/ovirt2_brick_01/brick > Options Reconfigured: > storage.owner-gid: 36 > storage.owner-uid: 36 > cluster.server-quorum-type: none > network.remote-dio: enable > cluster.eager-lock: enable > performance.stat-prefetch: off > performance.io-cache: off > performance.read-ahead: off > performance.quick-read: off >And attached you can find the debug log for gluster03. Gluster01 is still re-healing so I am not sure if it is safe to restart glusterd in DEBUG mode to obtain logs.... Regards. 2015-05-13 8:41 GMT+01:00 Kaushal M <kshlmster at gmail.com>:> Hey Adrian, > Can you list your volume infos? Most of the times the reason for peer > state being stuck is because of some volume information being incorrect and > the volume infos can help debug the problem. Could you also start glusterd > in debug mode (`glusterd -LDEBUG`) and give logs? > > ~kaushal > On May 13, 2015 9:13 AM, "Adri?n Santos Marrero" <asmarre at ull.edu.es> > wrote: > >> Hi again, >> >> I upgraded to 3.6.3 all my gluster servers but peer probes still get >> stuck in "Accepted peer request". >> >> Any hint? >> >> Regards. >> >> 2015-05-11 13:55 GMT+01:00 Adri?n Santos Marrero <asmarre at ull.edu.es>: >> >>> Hi, I've the same error with gluster03 (first installed with 3.6.2 >>> version, but upgraded to 3.6.3 before any peer probe) and gluster04 (fresh >>> install with Gluster 3.6.3). I attach the whole content of >>> /var/log/glusterfs/ directory from gluster03. >>> >>> Regards. >>> >>> 2015-05-11 12:06 GMT+01:00 Gaurav Garg <ggarg at redhat.com>: >>> >>>> Hi Adrian, >>>> >>>> Did you try to upgrade/downgrade gluster03 before performing any >>>> operation. If you attach fresh node (3.6.3 version) to (3.6.2) glusterfs >>>> version then this problems is not coming. we need to know what operation >>>> you performed on gluster03 node before peer probe. if possible could you >>>> attach cmd_history.log and glusterd log. >>>> >>>> ~ Gaurav >>>> >>>> ----- Original Message ----- >>>> From: "Adri?n Santos Marrero" <asmarre at ull.edu.es> >>>> To: "Gluster-users" <gluster-users at gluster.org> >>>> Sent: Monday, May 11, 2015 12:39:33 PM >>>> Subject: [Gluster-users] Adding a 3.6.3 peer to a 3.6.2 trusted pool >>>> fails >>>> >>>> Hi, >>>> >>>> I've been trying to add a third node to my two-node trusted pool with >>>> gluster peer probe: >>>> >>>> >>>> >>>> gluster02:~# gluster peer probe gluster03.stic.ull.es >>>> peer probe: success. >>>> >>>> The problem is that this node remains in "Accepted peer request" state: >>>> >>>> >>>> >>>> Hostname: gluster03.stic.ull.es >>>> Uuid: 9ca23fb2-cd76-4d1c-8a5f-f93306119539 >>>> State: Accepted peer request (Connected) >>>> >>>> If I restart glusterd in gluster02, the state changes to "Sent and >>>> Received peer request": >>>> >>>> >>>> >>>> Hostname: gluster03.stic.ull.es >>>> Uuid: 9ca23fb2-cd76-4d1c-8a5f-f93306119539 >>>> State: Sent and Received peer request (Connected) >>>> >>>> And if I restart glusterd in gluster03 it doesn't start because is >>>> "Unable to find friend: gluster01.stic.ull.es ": >>>> >>>> >>>> >>>> [2015-05-11 06:59:39.035184] D >>>> [glusterd-peer-utils.c:121:glusterd_peerinfo_find_by_hostname] >>>> 0-management: Unable to find friend: gluster01.stic.ull.es >>>> [2015-05-11 06:59:39.037942] D [common-utils.c:2897:gf_is_local_addr] >>>> 0-management: 10.107.34.17 >>>> [2015-05-11 06:59:39.039392] D [common-utils.c:2897:gf_is_local_addr] >>>> 0-management: 10.107.34.17 >>>> [2015-05-11 06:59:39.040584] D [common-utils.c:2897:gf_is_local_addr] >>>> 0-management: 10.107.34.17 >>>> [2015-05-11 06:59:39.041790] D [common-utils.c:2913:gf_is_local_addr] >>>> 0-management: gluster01.stic.ull.es is not local >>>> >>>> In fact, the only peer that gluster03 knows is gluster02: >>>> >>>> >>>> >>>> gluster03:~# ls -l /var/lib/glusterd/peers/ >>>> total 4 >>>> -rw-------. 1 root root 73 may 11 07:48 >>>> 739d7c3f-99d5-4c8b-ac28-f85163e26322 >>>> gluster03:~# cat >>>> /var/lib/glusterd/peers/739d7c3f-99d5-4c8b-ac28-f85163e26322 >>>> uuid=739d7c3f-99d5-4c8b-ac28-f85163e26322 >>>> state=5 >>>> hostname1=10.107.34.18 >>>> >>>> 10.107.34.18 == gluster02 >>>> >>>> >>>> >>>> gluster03:~# gluster --version >>>> glusterfs 3.6.3 built on Apr 23 2015 16:11:45 >>>> Repository revision: git:// git.gluster.com/glusterfs.git >>>> Copyright (c) 2006-2011 Gluster Inc. < http://www.gluster.com > >>>> GlusterFS comes with ABSOLUTELY NO WARRANTY. >>>> You may redistribute copies of GlusterFS under the terms of the GNU >>>> General Public License. >>>> >>>> All servers are running Centos 7, gluster01 and gluster02 have Gluster >>>> 3.6.2 version and gluster03 3.6.3. >>>> >>>> Am I forced to upgrade gluster01 and gluster02 to 3.6.3? >>>> >>>> Regards. >>>> >>>> -- >>>> ---------------------------------------------------------------- >>>> Adri?n Santos Marrero >>>> T?cnico de Sistemas - ?rea de Infraestructuras TIC >>>> Servicios de Tecnolog?as de la Informaci?n y Comunicaci?n (STIC) >>>> Universidad de La Laguna (ULL) >>>> Tel?fono/Phone: +34 922 845089 >>>> ---------------------------------------------------------------- >>>> >>>> Este mensaje puede contener informaci?n confidencial y/o privilegiada. >>>> Si usted no es el destinatario o lo ha recibido por error debe borrarlo >>>> inmediatamente. >>>> Est? estrictamente prohibido por la legislaci?n vigente realizar >>>> cualquier copia, revelaci?n o distribuci?n del contenido de este mensaje >>>> sin autorizaci?n expresa. >>>> >>>> This e-mail may contain confidential and/or privileged information. >>>> If you are not the intended recipient or have received this e-mail in >>>> error you must destroy it. >>>> Any unauthorised copying, disclosure or distribution of the material in >>>> this e-mail is strictly forbidden by current legislation. >>>> >>>> _______________________________________________ >>>> Gluster-users mailing list >>>> Gluster-users at gluster.org >>>> http://www.gluster.org/mailman/listinfo/gluster-users >>>> >>> >>> >>> >>> -- >>> ---------------------------------------------------------------- >>> Adri?n Santos Marrero >>> T?cnico de Sistemas - ?rea de Infraestructuras TIC >>> Servicios de Tecnolog?as de la Informaci?n y Comunicaci?n (STIC) >>> Universidad de La Laguna (ULL) >>> Tel?fono/Phone: +34 922 845089 >>> ---------------------------------------------------------------- >>> >>> Este mensaje puede contener informaci?n confidencial y/o privilegiada. >>> Si usted no es el destinatario o lo ha recibido por error debe borrarlo >>> inmediatamente. >>> Est? estrictamente prohibido por la legislaci?n vigente realizar >>> cualquier copia, revelaci?n o distribuci?n del contenido de este mensaje >>> sin autorizaci?n expresa. >>> >>> This e-mail may contain confidential and/or privileged information. >>> If you are not the intended recipient or have received this e-mail in >>> error you must destroy it. >>> Any unauthorised copying, disclosure or distribution of the material in >>> this e-mail is strictly forbidden by current legislation. >>> >> >> >> >> -- >> ---------------------------------------------------------------- >> Adri?n Santos Marrero >> T?cnico de Sistemas - ?rea de Infraestructuras TIC >> Servicios de Tecnolog?as de la Informaci?n y Comunicaci?n (STIC) >> Universidad de La Laguna (ULL) >> Tel?fono/Phone: +34 922 845089 >> ---------------------------------------------------------------- >> >> Este mensaje puede contener informaci?n confidencial y/o privilegiada. >> Si usted no es el destinatario o lo ha recibido por error debe borrarlo >> inmediatamente. >> Est? estrictamente prohibido por la legislaci?n vigente realizar >> cualquier copia, revelaci?n o distribuci?n del contenido de este mensaje >> sin autorizaci?n expresa. >> >> This e-mail may contain confidential and/or privileged information. >> If you are not the intended recipient or have received this e-mail in >> error you must destroy it. >> Any unauthorised copying, disclosure or distribution of the material in >> this e-mail is strictly forbidden by current legislation. >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://www.gluster.org/mailman/listinfo/gluster-users >> >-- ---------------------------------------------------------------- Adri?n Santos Marrero T?cnico de Sistemas - ?rea de Infraestructuras TIC Servicios de Tecnolog?as de la Informaci?n y Comunicaci?n (STIC) Universidad de La Laguna (ULL) Tel?fono/Phone: +34 922 845089 ---------------------------------------------------------------- Este mensaje puede contener informaci?n confidencial y/o privilegiada. Si usted no es el destinatario o lo ha recibido por error debe borrarlo inmediatamente. Est? estrictamente prohibido por la legislaci?n vigente realizar cualquier copia, revelaci?n o distribuci?n del contenido de este mensaje sin autorizaci?n expresa. This e-mail may contain confidential and/or privileged information. If you are not the intended recipient or have received this e-mail in error you must destroy it. Any unauthorised copying, disclosure or distribution of the material in this e-mail is strictly forbidden by current legislation. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150513/54ec683f/attachment-0001.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: etc-glusterfs-glusterd.vol.log Type: text/x-log Size: 309033 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150513/54ec683f/attachment-0001.bin>