Atin Mukherjee
2016-Dec-13 15:49 UTC
[Gluster-users] Pre Validation failed when adding bricks
And 3. In case 2 doesn't work, please provide the glusterd log files from gl1 & gl5 On Tue, Dec 13, 2016 at 9:16 PM, Atin Mukherjee <amukherj at redhat.com> wrote:> 1. Could you mention which gluster version are you running with? > 2. Does restarting glusterd instance on gl1 & gl5 solves the issue (after > removing the volume-id xattr from the bricks) ? > > On Tue, Dec 13, 2016 at 8:56 PM, Cedric Lemarchand <yipikai7 at gmail.com> > wrote: > >> Hello, >> >> When I try to add 3 bricks to a working cluster composed of 3 nodes / 3 >> bricks in dispersed mode 2+1, it fails like this : >> >> root at gl1:~# gluster volume add-brick vol1 gl4:/data/br1 gl5:/data/br1 >> gl6:/data/br1 >> volume add-brick: failed: Pre Validation failed on gl4. Host gl5 not >> connected >> >> However all peers are connected and there aren't networking issues : >> >> root at gl1:~# gluster peer status >> Number of Peers: 5 >> >> Hostname: gl2 >> Uuid: 616f100f-a3f4-46e4-b161-ee5db5a60e26 >> State: Peer in Cluster (Connected) >> >> Hostname: gl3 >> Uuid: acb828b8-f4b3-42ab-a9d2-b3e7b917dc9a >> State: Peer in Cluster (Connected) >> >> Hostname: gl4 >> Uuid: 813ad056-5e84-4fdb-ac13-38d24c748bc4 >> State: Peer in Cluster (Connected) >> >> Hostname: gl5 >> Uuid: a7933aeb-b08b-4ebb-a797-b8ecbe5a03c6 >> State: Peer in Cluster (Connected) >> >> Hostname: gl6 >> Uuid: 63c9a6c1-0adf-4cf5-af7b-b28a60911c99 >> State: Peer in Cluster (Connected) >> : >> >> When I try a second time, the error is different : >> >> root at gl1:~# gluster volume add-brick vol1 gl4:/data/br1 gl5:/data/br1 >> gl6:/data/br1 >> volume add-brick: failed: Pre Validation failed on gl5. /data/br1 is >> already part of a volume >> Pre Validation failed on gl6. /data/br1 is already part of a volume >> Pre Validation failed on gl4. /data/br1 is already part of a volume >> >> It seems the previous try, even if it has failed, have created the >> gluster attributes on file system as shown by attr on gl4/5/6 : >> >> Attribute "glusterfs.volume-id" has a 16 byte value for /data/br1 >> >> I already purge gluster and reformat brick on gl4/5/6 but the issue >> persist, any ideas ? did I miss something ? >> >> >> Some informations : >> >> root at gl1:~# gluster volume info >> >> Volume Name: vol1 >> Type: Disperse >> Volume ID: bb563884-0e2a-4757-9fd5-cb851ba113c3 >> Status: Started >> Snapshot Count: 0 >> Number of Bricks: 1 x (2 + 1) = 3 >> Transport-type: tcp >> Bricks: >> Brick1: gl1:/data/br1 >> Brick2: gl2:/data/br1 >> Brick3: gl3:/data/br1 >> Options Reconfigured: >> features.scrub-freq: hourly >> features.scrub: Inactive >> features.bitrot: off >> cluster.disperse-self-heal-daemon: enable >> transport.address-family: inet >> performance.readdir-ahead: on >> nfs.disable: on, I have the following error : >> >> root at gl1:~# gluster volume status >> Status of volume: vol1 >> Gluster process TCP Port RDMA Port Online >> Pid >> ------------------------------------------------------------ >> ------------------ >> Brick gl1:/data/br1 49152 0 Y >> 23403 >> Brick gl2:/data/br1 49152 0 Y >> 14545 >> Brick gl3:/data/br1 49152 0 Y >> 11348 >> Self-heal Daemon on localhost N/A N/A Y >> 24766 >> Self-heal Daemon on gl4 N/A N/A Y >> 1087 >> Self-heal Daemon on gl5 N/A N/A Y >> 1080 >> Self-heal Daemon on gl3 N/A N/A Y >> 12321 >> Self-heal Daemon on gl2 N/A N/A Y >> 15496 >> Self-heal Daemon on gl6 N/A N/A Y >> 1091 >> >> Task Status of Volume vol1 >> ------------------------------------------------------------ >> ------------------ >> There are no active volume tasks >> >> root at gl1:~# gluster volume info >> >> Volume Name: vol1 >> Type: Disperse >> Volume ID: bb563884-0e2a-4757-9fd5-cb851ba113c3 >> Status: Started >> Snapshot Count: 0 >> Number of Bricks: 1 x (2 + 1) = 3 >> Transport-type: tcp >> Bricks: >> Brick1: gl1:/data/br1 >> Brick2: gl2:/data/br1 >> Brick3: gl3:/data/br1 >> Options Reconfigured: >> features.scrub-freq: hourly >> features.scrub: Inactive >> features.bitrot: off >> cluster.disperse-self-heal-daemon: enable >> transport.address-family: inet >> performance.readdir-ahead: on >> nfs.disable: on >> >> root at gl1:~# gluster peer status >> Number of Peers: 5 >> >> Hostname: gl2 >> Uuid: 616f100f-a3f4-46e4-b161-ee5db5a60e26 >> State: Peer in Cluster (Connected) >> >> Hostname: gl3 >> Uuid: acb828b8-f4b3-42ab-a9d2-b3e7b917dc9a >> State: Peer in Cluster (Connected) >> >> Hostname: gl4 >> Uuid: 813ad056-5e84-4fdb-ac13-38d24c748bc4 >> State: Peer in Cluster (Connected) >> >> Hostname: gl5 >> Uuid: a7933aeb-b08b-4ebb-a797-b8ecbe5a03c6 >> State: Peer in Cluster (Connected) >> >> Hostname: gl6 >> Uuid: 63c9a6c1-0adf-4cf5-af7b-b28a60911c99 >> State: Peer in Cluster (Connected) >> >> >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://www.gluster.org/mailman/listinfo/gluster-users >> > > > > -- > > ~ Atin (atinm) >-- ~ Atin (atinm) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20161213/47b3ad06/attachment.html>
Cedric Lemarchand
2016-Dec-13 16:41 UTC
[Gluster-users] Pre Validation failed when adding bricks
1. sorry, 3.9.0-1 2. no it does nothing 3. here they are, from gl1 to gl6 : https://we.tl/EPaMs6geoR <https://we.tl/EPaMs6geoR>> On 13 Dec 2016, at 16:49, Atin Mukherjee <amukherj at redhat.com> wrote: > > And 3. In case 2 doesn't work, please provide the glusterd log files from gl1 & gl5 > > On Tue, Dec 13, 2016 at 9:16 PM, Atin Mukherjee <amukherj at redhat.com <mailto:amukherj at redhat.com>> wrote: > 1. Could you mention which gluster version are you running with? > 2. Does restarting glusterd instance on gl1 & gl5 solves the issue (after removing the volume-id xattr from the bricks) ? > > On Tue, Dec 13, 2016 at 8:56 PM, Cedric Lemarchand <yipikai7 at gmail.com <mailto:yipikai7 at gmail.com>> wrote: > Hello, > > When I try to add 3 bricks to a working cluster composed of 3 nodes / 3 bricks in dispersed mode 2+1, it fails like this : > > root at gl1:~# gluster volume add-brick vol1 gl4:/data/br1 gl5:/data/br1 gl6:/data/br1 > volume add-brick: failed: Pre Validation failed on gl4. Host gl5 not connected > > However all peers are connected and there aren't networking issues : > > root at gl1:~# gluster peer status > Number of Peers: 5 > > Hostname: gl2 > Uuid: 616f100f-a3f4-46e4-b161-ee5db5a60e26 > State: Peer in Cluster (Connected) > > Hostname: gl3 > Uuid: acb828b8-f4b3-42ab-a9d2-b3e7b917dc9a > State: Peer in Cluster (Connected) > > Hostname: gl4 > Uuid: 813ad056-5e84-4fdb-ac13-38d24c748bc4 > State: Peer in Cluster (Connected) > > Hostname: gl5 > Uuid: a7933aeb-b08b-4ebb-a797-b8ecbe5a03c6 > State: Peer in Cluster (Connected) > > Hostname: gl6 > Uuid: 63c9a6c1-0adf-4cf5-af7b-b28a60911c99 > State: Peer in Cluster (Connected) > : > > When I try a second time, the error is different : > > root at gl1:~# gluster volume add-brick vol1 gl4:/data/br1 gl5:/data/br1 gl6:/data/br1 > volume add-brick: failed: Pre Validation failed on gl5. /data/br1 is already part of a volume > Pre Validation failed on gl6. /data/br1 is already part of a volume > Pre Validation failed on gl4. /data/br1 is already part of a volume > > It seems the previous try, even if it has failed, have created the gluster attributes on file system as shown by attr on gl4/5/6 : > > Attribute "glusterfs.volume-id" has a 16 byte value for /data/br1 > > I already purge gluster and reformat brick on gl4/5/6 but the issue persist, any ideas ? did I miss something ? > > > Some informations : > > root at gl1:~# gluster volume info > > Volume Name: vol1 > Type: Disperse > Volume ID: bb563884-0e2a-4757-9fd5-cb851ba113c3 > Status: Started > Snapshot Count: 0 > Number of Bricks: 1 x (2 + 1) = 3 > Transport-type: tcp > Bricks: > Brick1: gl1:/data/br1 > Brick2: gl2:/data/br1 > Brick3: gl3:/data/br1 > Options Reconfigured: > features.scrub-freq: hourly > features.scrub: Inactive > features.bitrot: off > cluster.disperse-self-heal-daemon: enable > transport.address-family: inet > performance.readdir-ahead: on > nfs.disable: on, I have the following error : > > root at gl1:~# gluster volume status > Status of volume: vol1 > Gluster process TCP Port RDMA Port Online Pid > ------------------------------------------------------------------------------ > Brick gl1:/data/br1 49152 0 Y 23403 > Brick gl2:/data/br1 49152 0 Y 14545 > Brick gl3:/data/br1 49152 0 Y 11348 > Self-heal Daemon on localhost N/A N/A Y 24766 > Self-heal Daemon on gl4 N/A N/A Y 1087 > Self-heal Daemon on gl5 N/A N/A Y 1080 > Self-heal Daemon on gl3 N/A N/A Y 12321 > Self-heal Daemon on gl2 N/A N/A Y 15496 > Self-heal Daemon on gl6 N/A N/A Y 1091 > > Task Status of Volume vol1 > ------------------------------------------------------------------------------ > There are no active volume tasks > > root at gl1:~# gluster volume info > > Volume Name: vol1 > Type: Disperse > Volume ID: bb563884-0e2a-4757-9fd5-cb851ba113c3 > Status: Started > Snapshot Count: 0 > Number of Bricks: 1 x (2 + 1) = 3 > Transport-type: tcp > Bricks: > Brick1: gl1:/data/br1 > Brick2: gl2:/data/br1 > Brick3: gl3:/data/br1 > Options Reconfigured: > features.scrub-freq: hourly > features.scrub: Inactive > features.bitrot: off > cluster.disperse-self-heal-daemon: enable > transport.address-family: inet > performance.readdir-ahead: on > nfs.disable: on > > root at gl1:~# gluster peer status > Number of Peers: 5 > > Hostname: gl2 > Uuid: 616f100f-a3f4-46e4-b161-ee5db5a60e26 > State: Peer in Cluster (Connected) > > Hostname: gl3 > Uuid: acb828b8-f4b3-42ab-a9d2-b3e7b917dc9a > State: Peer in Cluster (Connected) > > Hostname: gl4 > Uuid: 813ad056-5e84-4fdb-ac13-38d24c748bc4 > State: Peer in Cluster (Connected) > > Hostname: gl5 > Uuid: a7933aeb-b08b-4ebb-a797-b8ecbe5a03c6 > State: Peer in Cluster (Connected) > > Hostname: gl6 > Uuid: 63c9a6c1-0adf-4cf5-af7b-b28a60911c99 > State: Peer in Cluster (Connected) > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> > http://www.gluster.org/mailman/listinfo/gluster-users <http://www.gluster.org/mailman/listinfo/gluster-users> > > > > -- > > ~ Atin (atinm) > > > > -- > > ~ Atin (atinm)-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20161213/3c02523a/attachment.html>