Laurent DOUCHY
2011-Aug-31 16:49 UTC
[Gluster-users] add bricks on distributed replicated volume failed
Hi, I'm using gluster 3.2.2 on 10 nodes. Each node have 2x2 TB disk for gluster. I manage to configure a distributed and replicated volume on 4 nodes : [root at node07 ~]# gluster volume info cluster Volume Name: cluster Type: Distributed-Replicate Status: Started Number of Bricks: 4 x 2 = 8 Transport-type: tcp Bricks: Brick1: node09:/gluster1 Brick2: node10:/gluster1 Brick3: node09:/gluster2 Brick4: node10:/gluster2 Brick5: node07:/gluster1 Brick6: node08:/gluster1 Brick7: node07:/gluster2 Brick8: node08:/gluster2 But I can't add new nodes to this volume [root at node07 ~]# gluster peer status Number of Peers: 5 Hostname: node10 Uuid: 212ce5a0-de51-4a98-9262-ae071c2d63a0 State: Peer in Cluster (Connected) Hostname: node08 Uuid: dd114546-5b94-4a62-9301-260703bf5707 State: Peer in Cluster (Connected) Hostname: node09 Uuid: f73fee83-8d47-4f07-bfac-b8a8592eff04 State: Peer in Cluster (Connected) Hostname: node06 Uuid: 3142fb9a-0a6b-46ec-9262-ede95e8f798a State: Peer in Cluster (Connected) Hostname: node05 Uuid: 13ffcf87-6e8d-4c6b-814a-cbc14d15d88b State: Peer in Cluster (Connected) [root at node07 ~]# gluster volume add-brick cluster node05:/gluster1 node06:/gluster1 node05:/gluster2 node06:/gluster2 Operation failed on node05 I try to detach nodes 5 and 6, restart glusterd do the probe and the add-brick but still nothing ... Did some one have any idea to fix this ? Thanks in advance, Laurent.
Burnash, James
2011-Aug-31 18:07 UTC
[Gluster-users] add bricks on distributed replicated volume failed
Hi Laurent. Since your configuration specifies replication, you must add bricks in the same number as your number of replicas. For instance - if you have 2 replicas (most normal case), you would need to do something like this: gluster volume add-brick cluster node05:/gluster1 node06:/gluster1 James Burnash Unix Engineer Knight Capital Group -----Original Message----- From: gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at gluster.org] On Behalf Of Laurent DOUCHY Sent: Wednesday, August 31, 2011 12:49 PM To: gluster-users at gluster.org Subject: [Gluster-users] add bricks on distributed replicated volume failed Hi, I'm using gluster 3.2.2 on 10 nodes. Each node have 2x2 TB disk for gluster. I manage to configure a distributed and replicated volume on 4 nodes : [root at node07 ~]# gluster volume info cluster Volume Name: cluster Type: Distributed-Replicate Status: Started Number of Bricks: 4 x 2 = 8 Transport-type: tcp Bricks: Brick1: node09:/gluster1 Brick2: node10:/gluster1 Brick3: node09:/gluster2 Brick4: node10:/gluster2 Brick5: node07:/gluster1 Brick6: node08:/gluster1 Brick7: node07:/gluster2 Brick8: node08:/gluster2 But I can't add new nodes to this volume [root at node07 ~]# gluster peer status Number of Peers: 5 Hostname: node10 Uuid: 212ce5a0-de51-4a98-9262-ae071c2d63a0 State: Peer in Cluster (Connected) Hostname: node08 Uuid: dd114546-5b94-4a62-9301-260703bf5707 State: Peer in Cluster (Connected) Hostname: node09 Uuid: f73fee83-8d47-4f07-bfac-b8a8592eff04 State: Peer in Cluster (Connected) Hostname: node06 Uuid: 3142fb9a-0a6b-46ec-9262-ede95e8f798a State: Peer in Cluster (Connected) Hostname: node05 Uuid: 13ffcf87-6e8d-4c6b-814a-cbc14d15d88b State: Peer in Cluster (Connected) [root at node07 ~]# gluster volume add-brick cluster node05:/gluster1 node06:/gluster1 node05:/gluster2 node06:/gluster2 Operation failed on node05 I try to detach nodes 5 and 6, restart glusterd do the probe and the add-brick but still nothing ... Did some one have any idea to fix this ? Thanks in advance, Laurent. _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users DISCLAIMER: This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this in error, please immediately notify me and permanently delete the original and any copy of any e-mail and any printout thereof. E-mail transmission cannot be guaranteed to be secure or error-free. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission. NOTICE REGARDING PRIVACY AND CONFIDENTIALITY Knight Capital Group may, at its discretion, monitor and review the content of all e-mail communications. http://www.knight.com
Laurent DOUCHY
2011-Sep-01 15:57 UTC
[Gluster-users] add bricks on distributed replicated volume failed
It works ... [root at node07 ~]# gluster volume add-brick cluster node09:/gluster3 node10:/gluster3 Add Brick successful On 9/1/11 5:39 PM, Mohit Anchlia wrote:> Can you try with node09:/gluster3 and node10:gluster3 instead? > > On Thu, Sep 1, 2011 at 2:49 AM, Laurent DOUCHY<Laurent.Douchy at unige.ch> wrote: >> Hi, >> >> I work on the node7 so it doesn't appear on the list. >> >> I create a folder /gluster3 on node5 and node6 and try to add them to my >> volume but it failed with the same message :( >> >> [root at node07 ~]# gluster volume add-brick cluster node05:/gluster3 >> node06:/gluster3 >> Operation failed on node05 >> >> next step is to reinstall from scratch the node I hope I can avoid this. >> >> On 8/31/11 9:08 PM, Mohit Anchlia wrote: >>> I don't see node07 in above output of gluster peer status. >>> >>> Can you try to add bricks on the hosts that gluster1, gluster2? So add >>> gluster3 and see if that works. >>> >>> On Wed, Aug 31, 2011 at 11:56 AM, Laurent DOUCHY >>> <Laurent.Douchy at unige.ch> wrote: >>>> Hi, >>>> >>>> I try to add 2 bricks or 4 bricks for the same effect. >>>> >>>> I try to reinstall gluster without success. >>>> >>>> Cheers, >>>> Laurent DOUCHY. >>>> >>>> >>>> On 8/31/11 8:07 PM, Burnash, James wrote: >>>>> Hi Laurent. >>>>> >>>>> Since your configuration specifies replication, you must add bricks in >>>>> the >>>>> same number as your number of replicas. >>>>> >>>>> For instance - if you have 2 replicas (most normal case), you would need >>>>> to do something like this: >>>>> >>>>> gluster volume add-brick cluster node05:/gluster1 node06:/gluster1 >>>>> >>>>> James Burnash >>>>> Unix Engineer >>>>> Knight Capital Group >>>>> >>>>> >>>>> -----Original Message----- >>>>> From: gluster-users-bounces at gluster.org >>>>> [mailto:gluster-users-bounces at gluster.org] On Behalf Of Laurent DOUCHY >>>>> Sent: Wednesday, August 31, 2011 12:49 PM >>>>> To: gluster-users at gluster.org >>>>> Subject: [Gluster-users] add bricks on distributed replicated volume >>>>> failed >>>>> >>>>> Hi, >>>>> >>>>> I'm using gluster 3.2.2 on 10 nodes. Each node have 2x2 TB disk for >>>>> gluster. >>>>> >>>>> I manage to configure a distributed and replicated volume on 4 nodes : >>>>> >>>>> [root at node07 ~]# gluster volume info cluster >>>>> >>>>> Volume Name: cluster >>>>> Type: Distributed-Replicate >>>>> Status: Started >>>>> Number of Bricks: 4 x 2 = 8 >>>>> Transport-type: tcp >>>>> Bricks: >>>>> Brick1: node09:/gluster1 >>>>> Brick2: node10:/gluster1 >>>>> Brick3: node09:/gluster2 >>>>> Brick4: node10:/gluster2 >>>>> Brick5: node07:/gluster1 >>>>> Brick6: node08:/gluster1 >>>>> Brick7: node07:/gluster2 >>>>> Brick8: node08:/gluster2 >>>>> >>>>> But I can't add new nodes to this volume >>>>> >>>>> [root at node07 ~]# gluster peer status >>>>> Number of Peers: 5 >>>>> >>>>> Hostname: node10 >>>>> Uuid: 212ce5a0-de51-4a98-9262-ae071c2d63a0 >>>>> State: Peer in Cluster (Connected) >>>>> >>>>> Hostname: node08 >>>>> Uuid: dd114546-5b94-4a62-9301-260703bf5707 >>>>> State: Peer in Cluster (Connected) >>>>> >>>>> Hostname: node09 >>>>> Uuid: f73fee83-8d47-4f07-bfac-b8a8592eff04 >>>>> State: Peer in Cluster (Connected) >>>>> >>>>> Hostname: node06 >>>>> Uuid: 3142fb9a-0a6b-46ec-9262-ede95e8f798a >>>>> State: Peer in Cluster (Connected) >>>>> >>>>> Hostname: node05 >>>>> Uuid: 13ffcf87-6e8d-4c6b-814a-cbc14d15d88b >>>>> State: Peer in Cluster (Connected) >>>>> [root at node07 ~]# gluster volume add-brick cluster node05:/gluster1 >>>>> node06:/gluster1 node05:/gluster2 node06:/gluster2 Operation failed on >>>>> node05 >>>>> >>>>> I try to detach nodes 5 and 6, restart glusterd do the probe and the >>>>> add-brick but still nothing ... >>>>> >>>>> Did some one have any idea to fix this ? >>>>> >>>>> Thanks in advance, >>>>> Laurent. >>>>> >>>>> _______________________________________________ >>>>> Gluster-users mailing list >>>>> Gluster-users at gluster.org >>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >>>>> >>>>> >>>>> DISCLAIMER: >>>>> This e-mail, and any attachments thereto, is intended only for use by >>>>> the >>>>> addressee(s) named herein and may contain legally privileged and/or >>>>> confidential information. If you are not the intended recipient of this >>>>> e-mail, you are hereby notified that any dissemination, distribution or >>>>> copying of this e-mail, and any attachments thereto, is strictly >>>>> prohibited. >>>>> If you have received this in error, please immediately notify me and >>>>> permanently delete the original and any copy of any e-mail and any >>>>> printout >>>>> thereof. E-mail transmission cannot be guaranteed to be secure or >>>>> error-free. The sender therefore does not accept liability for any >>>>> errors or >>>>> omissions in the contents of this message which arise as a result of >>>>> e-mail >>>>> transmission. >>>>> NOTICE REGARDING PRIVACY AND CONFIDENTIALITY Knight Capital Group may, >>>>> at >>>>> its discretion, monitor and review the content of all e-mail >>>>> communications. >>>>> http://www.knight.com >>>>> _______________________________________________ >>>>> Gluster-users mailing list >>>>> Gluster-users at gluster.org >>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >>>> _______________________________________________ >>>> Gluster-users mailing list >>>> Gluster-users at gluster.org >>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >>>>
Laurent DOUCHY
2011-Sep-01 16:39 UTC
[Gluster-users] add bricks on distributed replicated volume failed
see below Cheers, Laurent DOUCHY. On 9/1/11 6:01 PM, Mohit Anchlia wrote:> You can check few things on 5 and 6: > > 1) gluster processes are running on node5 and 6yes: node05 root 4902 1 0 Aug31 ? 00:00:00 /opt/glusterfs/3.2.2/sbin/glusterd root 9626 1 0 19:55 ? 00:00:00 /opt/glusterfs/3.2.2/sbin/glusterfs -f /etc/glusterd/nfs/nfs-server.vol -p /etc/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log root 9690 9686 0 20:04 ? 00:00:00 bash -c ps -edf | grep gluster root 9704 9690 0 20:04 ? 00:00:00 grep gluster node06 root 4441 1 0 Aug31 ? 00:00:00 /opt/glusterfs/3.2.2/sbin/glusterd root 9178 1 0 19:55 ? 00:00:00 /opt/glusterfs/3.2.2/sbin/glusterfs -f /etc/glusterd/nfs/nfs-server.vol -p /etc/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log root 9242 9238 0 20:04 ? 00:00:00 bash -c ps -edf | grep gluster root 9256 9242 0 20:04 ? 00:00:00 grep gluster> 2) both nodes are able to see each otheryes: [root at node05 ~]# ping node06 PING node06.isdc.unige.ch (129.194.168.70) 56(84) bytes of data. 64 bytes from node06.isdc.unige.ch (129.194.168.70): icmp_seq=1 ttl=64 time=0.376 ms [root at node06 ~]# ping node05 PING node05.isdc.unige.ch (129.194.168.69) 56(84) bytes of data. 64 bytes from node05.isdc.unige.ch (129.194.168.69): icmp_seq=1 ttl=64 time=0.337 ms> 3) do gluster peer status on both the nodes and see what you seenode 5 trust node 6 and node 6 trust node 5 [root at node05 ~]# gluster peer status Number of Peers: 5 Hostname: node08 Uuid: dd114546-5b94-4a62-9301-260703bf5707 State: Peer in Cluster (Connected) Hostname: node06 Uuid: 3142fb9a-0a6b-46ec-9262-ede95e8f798a State: Peer in Cluster (Connected) Hostname: node10 Uuid: 212ce5a0-de51-4a98-9262-ae071c2d63a0 State: Peer in Cluster (Connected) Hostname: 129.194.168.71 Uuid: a35fb0a1-af35-4a04-b38a-434f68369508 State: Peer in Cluster (Connected) Hostname: node09 Uuid: f73fee83-8d47-4f07-bfac-b8a8592eff04 State: Peer in Cluster (Connected) [root at node06 ~]# gluster peer status Number of Peers: 5 Hostname: node08 Uuid: dd114546-5b94-4a62-9301-260703bf5707 State: Peer in Cluster (Connected) Hostname: node09 Uuid: f73fee83-8d47-4f07-bfac-b8a8592eff04 State: Peer in Cluster (Connected) Hostname: node05 Uuid: 13ffcf87-6e8d-4c6b-814a-cbc14d15d88b State: Peer in Cluster (Connected) Hostname: node10 Uuid: 212ce5a0-de51-4a98-9262-ae071c2d63a0 State: Peer in Cluster (Connected) Hostname: 129.194.168.71 Uuid: a35fb0a1-af35-4a04-b38a-434f68369508 State: Peer in Cluster (Connected)> 4) check iptablessame file on each node (the installation is manage by puppet)> > On Thu, Sep 1, 2011 at 8:57 AM, Laurent DOUCHY<Laurent.Douchy at unige.ch> wrote: >> It works ... >> >> [root at node07 ~]# gluster volume add-brick cluster node09:/gluster3 >> node10:/gluster3 >> Add Brick successful >> >> >> On 9/1/11 5:39 PM, Mohit Anchlia wrote: >>> Can you try with node09:/gluster3 and node10:gluster3 instead? >>> >>> On Thu, Sep 1, 2011 at 2:49 AM, Laurent DOUCHY<Laurent.Douchy at unige.ch> >>> wrote: >>>> Hi, >>>> >>>> I work on the node7 so it doesn't appear on the list. >>>> >>>> I create a folder /gluster3 on node5 and node6 and try to add them to my >>>> volume but it failed with the same message :( >>>> >>>> [root at node07 ~]# gluster volume add-brick cluster node05:/gluster3 >>>> node06:/gluster3 >>>> Operation failed on node05 >>>> >>>> next step is to reinstall from scratch the node I hope I can avoid this. >>>> >>>> On 8/31/11 9:08 PM, Mohit Anchlia wrote: >>>>> I don't see node07 in above output of gluster peer status. >>>>> >>>>> Can you try to add bricks on the hosts that gluster1, gluster2? So add >>>>> gluster3 and see if that works. >>>>> >>>>> On Wed, Aug 31, 2011 at 11:56 AM, Laurent DOUCHY >>>>> <Laurent.Douchy at unige.ch> wrote: >>>>>> Hi, >>>>>> >>>>>> I try to add 2 bricks or 4 bricks for the same effect. >>>>>> >>>>>> I try to reinstall gluster without success. >>>>>> >>>>>> Cheers, >>>>>> Laurent DOUCHY. >>>>>> >>>>>> >>>>>> On 8/31/11 8:07 PM, Burnash, James wrote: >>>>>>> Hi Laurent. >>>>>>> >>>>>>> Since your configuration specifies replication, you must add bricks in >>>>>>> the >>>>>>> same number as your number of replicas. >>>>>>> >>>>>>> For instance - if you have 2 replicas (most normal case), you would >>>>>>> need >>>>>>> to do something like this: >>>>>>> >>>>>>> gluster volume add-brick cluster node05:/gluster1 node06:/gluster1 >>>>>>> >>>>>>> James Burnash >>>>>>> Unix Engineer >>>>>>> Knight Capital Group >>>>>>> >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: gluster-users-bounces at gluster.org >>>>>>> [mailto:gluster-users-bounces at gluster.org] On Behalf Of Laurent DOUCHY >>>>>>> Sent: Wednesday, August 31, 2011 12:49 PM >>>>>>> To: gluster-users at gluster.org >>>>>>> Subject: [Gluster-users] add bricks on distributed replicated volume >>>>>>> failed >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I'm using gluster 3.2.2 on 10 nodes. Each node have 2x2 TB disk for >>>>>>> gluster. >>>>>>> >>>>>>> I manage to configure a distributed and replicated volume on 4 nodes : >>>>>>> >>>>>>> [root at node07 ~]# gluster volume info cluster >>>>>>> >>>>>>> Volume Name: cluster >>>>>>> Type: Distributed-Replicate >>>>>>> Status: Started >>>>>>> Number of Bricks: 4 x 2 = 8 >>>>>>> Transport-type: tcp >>>>>>> Bricks: >>>>>>> Brick1: node09:/gluster1 >>>>>>> Brick2: node10:/gluster1 >>>>>>> Brick3: node09:/gluster2 >>>>>>> Brick4: node10:/gluster2 >>>>>>> Brick5: node07:/gluster1 >>>>>>> Brick6: node08:/gluster1 >>>>>>> Brick7: node07:/gluster2 >>>>>>> Brick8: node08:/gluster2 >>>>>>> >>>>>>> But I can't add new nodes to this volume >>>>>>> >>>>>>> [root at node07 ~]# gluster peer status >>>>>>> Number of Peers: 5 >>>>>>> >>>>>>> Hostname: node10 >>>>>>> Uuid: 212ce5a0-de51-4a98-9262-ae071c2d63a0 >>>>>>> State: Peer in Cluster (Connected) >>>>>>> >>>>>>> Hostname: node08 >>>>>>> Uuid: dd114546-5b94-4a62-9301-260703bf5707 >>>>>>> State: Peer in Cluster (Connected) >>>>>>> >>>>>>> Hostname: node09 >>>>>>> Uuid: f73fee83-8d47-4f07-bfac-b8a8592eff04 >>>>>>> State: Peer in Cluster (Connected) >>>>>>> >>>>>>> Hostname: node06 >>>>>>> Uuid: 3142fb9a-0a6b-46ec-9262-ede95e8f798a >>>>>>> State: Peer in Cluster (Connected) >>>>>>> >>>>>>> Hostname: node05 >>>>>>> Uuid: 13ffcf87-6e8d-4c6b-814a-cbc14d15d88b >>>>>>> State: Peer in Cluster (Connected) >>>>>>> [root at node07 ~]# gluster volume add-brick cluster node05:/gluster1 >>>>>>> node06:/gluster1 node05:/gluster2 node06:/gluster2 Operation failed on >>>>>>> node05 >>>>>>> >>>>>>> I try to detach nodes 5 and 6, restart glusterd do the probe and the >>>>>>> add-brick but still nothing ... >>>>>>> >>>>>>> Did some one have any idea to fix this ? >>>>>>> >>>>>>> Thanks in advance, >>>>>>> Laurent. >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Gluster-users mailing list >>>>>>> Gluster-users at gluster.org >>>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >>>>>>> >>>>>>> >>>>>>> DISCLAIMER: >>>>>>> This e-mail, and any attachments thereto, is intended only for use by >>>>>>> the >>>>>>> addressee(s) named herein and may contain legally privileged and/or >>>>>>> confidential information. If you are not the intended recipient of >>>>>>> this >>>>>>> e-mail, you are hereby notified that any dissemination, distribution >>>>>>> or >>>>>>> copying of this e-mail, and any attachments thereto, is strictly >>>>>>> prohibited. >>>>>>> If you have received this in error, please immediately notify me and >>>>>>> permanently delete the original and any copy of any e-mail and any >>>>>>> printout >>>>>>> thereof. E-mail transmission cannot be guaranteed to be secure or >>>>>>> error-free. The sender therefore does not accept liability for any >>>>>>> errors or >>>>>>> omissions in the contents of this message which arise as a result of >>>>>>> e-mail >>>>>>> transmission. >>>>>>> NOTICE REGARDING PRIVACY AND CONFIDENTIALITY Knight Capital Group may, >>>>>>> at >>>>>>> its discretion, monitor and review the content of all e-mail >>>>>>> communications. >>>>>>> http://www.knight.com >>>>>>> _______________________________________________ >>>>>>> Gluster-users mailing list >>>>>>> Gluster-users at gluster.org >>>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >>>>>> _______________________________________________ >>>>>> Gluster-users mailing list >>>>>> Gluster-users at gluster.org >>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >>>>>>