Hello, We've set up geo-replication but it isn't actually syncing. Scenario is that we have two GFS clusters. Cluster A has nodes cafs10, cafs20, and cafs30, replicating with each other over a LAN. Cluster B has nodes nvfs10, nvfs20, and nvfs30 also replicating with each other over a LAN. We are geo-replicating data from the A cluster to the B cluster over the internet. SSH key access is set up, allowing all the A nodes password-less access to root on nvfs10 Geo-replication was set up using these commands, run on cafs10: gluster volume geo-replication gvol0 nvfs10.example.com::gvol0 create ssh-port 8822 no-verify gluster volume geo-replication gvol0 nvfs10.example.com::gvol0 config remote-gsyncd /usr/lib/x86_64-linux-gnu/glusterfs/gsyncd gluster volume geo-replication gvol0 nvfs10.example.com::gvol0 start However after a very short period of the status being "Initializing..." the status then sits on "Passive": # gluster volume geo-replication gvol0 nvfs10.example.com::gvol0 status MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ------------------------------------------------------------------------------------------------------------------------------------------------------------------ cafs10 gvol0 /nodirectwritedata/gluster/gvol0 root nvfs10.example.com::gvol0 nvfs30.local Passive N/A N/A cafs30 gvol0 /nodirectwritedata/gluster/gvol0 root nvfs10.example.com::gvol0 N/A Created N/A N/A cafs20 gvol0 /nodirectwritedata/gluster/gvol0 root nvfs10.example.com::gvol0 N/A Created N/A N/A So my questions are: 1. Why does the status on cafs10 mention "nvfs30.local"? That's the LAN address that nvfs10 replicates with nvfs30 using. It's not accessible from the A cluster, and I didn't use it when configuring geo-replication. 2. Why does geo-replication sit in Passive status? Thanks very much for any assistance. On Tue, 25 Feb 2020 at 15:46, David Cunningham <dcunningham at voisonics.com> wrote:> Hi Aravinda and Sunny, > > Thank you for the replies. We have 3 replicating nodes on the master side, > and want to geo-replicate their data to the remote slave side. As I > understand it if the master node which had the geo-replication create > command run goes down then another node will take over pushing updates to > the remote slave. Is that right? > > We have already taken care of adding all master node's SSH keys to the > remote slave's authorized_keys externally, so won't include the push-pem > part of the create command. > > Mostly I wanted to confirm the geo-replication behaviour on the > replicating master nodes if one of them goes down. > > Thank you! > > > On Tue, 25 Feb 2020 at 14:32, Aravinda VK <aravinda at kadalu.io> wrote: > >> Hi David, >> >> >> On 25-Feb-2020, at 3:45 AM, David Cunningham <dcunningham at voisonics.com> >> wrote: >> >> Hello, >> >> I've a couple of questions on geo-replication that hopefully someone can >> help with: >> >> 1. If there are multiple nodes in a cluster on the master side (pushing >> updates to the geo-replication slave), which node actually does the >> pushing? Does GlusterFS decide itself automatically? >> >> >> Once Geo-replication session is started, one worker will be started >> corresponding to each Master bricks. Each worker identifies the changes >> happened in respective brick and sync those changes via Mount. This way >> load is distributed among Master nodes. In case of Replica sub volume, one >> worker among the Replica group will become active and participate in the >> syncing. Other bricks in that Replica group will remain Passive. Passive >> worker will become Active if the previously Active brick goes down (This is >> because all Replica bricks will have the same set of changes, syncing from >> each worker is redundant). >> >> >> 2.With regard to copying SSH keys, presumably the SSH key of all master >> nodes should be authorized on the geo-replication client side? >> >> >> Geo-replication session is established between one master node and one >> remote node. If Geo-rep create command is successful then, >> >> - SSH keys generated in all master nodes >> - Public keys from all master nodes are copied to initiator Master node >> - Public keys copied to the Remote node specified in the create command >> - Master public keys are distributed to all nodes of remote Cluster and >> added to respective ~/.ssh/authorized_keys >> >> After successful Geo-rep create command, any Master node can connect to >> any remote node via ssh. >> >> Security: Command prefix is added while adding public key to remote >> node?s authorized_keys file, So that if anyone gain access using this key >> can access only gsyncd command. >> >> ``` >> command=gsyncd ssh-key?. >> ``` >> >> >> >> Thanks for your help. >> >> -- >> David Cunningham, Voisonics Limited >> http://voisonics.com/ >> USA: +1 213 221 1092 >> New Zealand: +64 (0)28 2558 3782 >> ________ >> >> >> >> Community Meeting Calendar: >> >> Schedule - >> Every Tuesday at 14:30 IST / 09:00 UTC >> Bridge: https://bluejeans.com/441850968 >> >> Gluster-users mailing list >> Gluster-users at gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users >> >> >> >> ? >> regards >> Aravinda Vishwanathapura >> https://kadalu.io >> >> > > -- > David Cunningham, Voisonics Limited > http://voisonics.com/ > USA: +1 213 221 1092 > New Zealand: +64 (0)28 2558 3782 >-- David Cunningham, Voisonics Limited http://voisonics.com/ USA: +1 213 221 1092 New Zealand: +64 (0)28 2558 3782 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200302/509d7752/attachment.html>
On March 2, 2020 2:33:10 AM GMT+02:00, David Cunningham <dcunningham at voisonics.com> wrote:>Hello, > >We've set up geo-replication but it isn't actually syncing. Scenario is >that we have two GFS clusters. Cluster A has nodes cafs10, cafs20, and >cafs30, replicating with each other over a LAN. Cluster B has nodes >nvfs10, >nvfs20, and nvfs30 also replicating with each other over a LAN. We are >geo-replicating data from the A cluster to the B cluster over the >internet. >SSH key access is set up, allowing all the A nodes password-less access >to >root on nvfs10 > >Geo-replication was set up using these commands, run on cafs10: > >gluster volume geo-replication gvol0 nvfs10.example.com::gvol0 create >ssh-port 8822 no-verify >gluster volume geo-replication gvol0 nvfs10.example.com::gvol0 config >remote-gsyncd /usr/lib/x86_64-linux-gnu/glusterfs/gsyncd >gluster volume geo-replication gvol0 nvfs10.example.com::gvol0 start > >However after a very short period of the status being "Initializing..." >the >status then sits on "Passive": > ># gluster volume geo-replication gvol0 nvfs10.example.com::gvol0 status >MASTER NODE MASTER VOL MASTER BRICK SLAVE >USER > SLAVE SLAVE NODE STATUS CRAWL STATUS > LAST_SYNCED >------------------------------------------------------------------------------------------------------------------------------------------------------------------ >cafs10 gvol0 /nodirectwritedata/gluster/gvol0 root > nvfs10.example.com::gvol0 nvfs30.local Passive N/A >N/A >cafs30 gvol0 /nodirectwritedata/gluster/gvol0 root > nvfs10.example.com::gvol0 N/A Created N/A >N/A >cafs20 gvol0 /nodirectwritedata/gluster/gvol0 root > nvfs10.example.com::gvol0 N/A Created N/A >N/A > >So my questions are: >1. Why does the status on cafs10 mention "nvfs30.local"? That's the LAN >address that nvfs10 replicates with nvfs30 using. It's not accessible >from >the A cluster, and I didn't use it when configuring geo-replication. >2. Why does geo-replication sit in Passive status? > >Thanks very much for any assistance. > > >On Tue, 25 Feb 2020 at 15:46, David Cunningham ><dcunningham at voisonics.com> >wrote: > >> Hi Aravinda and Sunny, >> >> Thank you for the replies. We have 3 replicating nodes on the master >side, >> and want to geo-replicate their data to the remote slave side. As I >> understand it if the master node which had the geo-replication create >> command run goes down then another node will take over pushing >updates to >> the remote slave. Is that right? >> >> We have already taken care of adding all master node's SSH keys to >the >> remote slave's authorized_keys externally, so won't include the >push-pem >> part of the create command. >> >> Mostly I wanted to confirm the geo-replication behaviour on the >> replicating master nodes if one of them goes down. >> >> Thank you! >> >> >> On Tue, 25 Feb 2020 at 14:32, Aravinda VK <aravinda at kadalu.io> wrote: >> >>> Hi David, >>> >>> >>> On 25-Feb-2020, at 3:45 AM, David Cunningham ><dcunningham at voisonics.com> >>> wrote: >>> >>> Hello, >>> >>> I've a couple of questions on geo-replication that hopefully someone >can >>> help with: >>> >>> 1. If there are multiple nodes in a cluster on the master side >(pushing >>> updates to the geo-replication slave), which node actually does the >>> pushing? Does GlusterFS decide itself automatically? >>> >>> >>> Once Geo-replication session is started, one worker will be started >>> corresponding to each Master bricks. Each worker identifies the >changes >>> happened in respective brick and sync those changes via Mount. This >way >>> load is distributed among Master nodes. In case of Replica sub >volume, one >>> worker among the Replica group will become active and participate in >the >>> syncing. Other bricks in that Replica group will remain Passive. >Passive >>> worker will become Active if the previously Active brick goes down >(This is >>> because all Replica bricks will have the same set of changes, >syncing from >>> each worker is redundant). >>> >>> >>> 2.With regard to copying SSH keys, presumably the SSH key of all >master >>> nodes should be authorized on the geo-replication client side? >>> >>> >>> Geo-replication session is established between one master node and >one >>> remote node. If Geo-rep create command is successful then, >>> >>> - SSH keys generated in all master nodes >>> - Public keys from all master nodes are copied to initiator Master >node >>> - Public keys copied to the Remote node specified in the create >command >>> - Master public keys are distributed to all nodes of remote Cluster >and >>> added to respective ~/.ssh/authorized_keys >>> >>> After successful Geo-rep create command, any Master node can connect >to >>> any remote node via ssh. >>> >>> Security: Command prefix is added while adding public key to remote >>> node?s authorized_keys file, So that if anyone gain access using >this key >>> can access only gsyncd command. >>> >>> ``` >>> command=gsyncd ssh-key?. >>> ``` >>> >>> >>> >>> Thanks for your help. >>> >>> -- >>> David Cunningham, Voisonics Limited >>> http://voisonics.com/ >>> USA: +1 213 221 1092 >>> New Zealand: +64 (0)28 2558 3782 >>> ________ >>> >>> >>> >>> Community Meeting Calendar: >>> >>> Schedule - >>> Every Tuesday at 14:30 IST / 09:00 UTC >>> Bridge: https://bluejeans.com/441850968 >>> >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> https://lists.gluster.org/mailman/listinfo/gluster-users >>> >>> >>> >>> ? >>> regards >>> Aravinda Vishwanathapura >>> https://kadalu.io >>> >>> >> >> -- >> David Cunningham, Voisonics Limited >> http://voisonics.com/ >> USA: +1 213 221 1092 >> New Zealand: +64 (0)28 2558 3782 >>Hey David, What are the logs saying ? Also, doublecheck access from the master to the slave node via ssh. Best Regards, Strahil Nikolov
Looks like setup issue to me. Copying SSH keys manually is not required. Command prefix is required while adding to authorized_keys file in each remote nodes. That will not be available if ssh keys are added manually. Geo-rep specifies /nonexisting/gsyncd in the command to make sure it connects via the actual command specified in authorized_keys file, in your case Geo-replication is actually looking for gsyncd command in /nonexisting/gsyncd path. Please try with push-pem option during Geo-rep create command. ? regards Aravinda Vishwanathapura https://kadalu <https://kadalu/>.io> On 02-Mar-2020, at 6:03 AM, David Cunningham <dcunningham at voisonics.com> wrote: > > Hello, > > We've set up geo-replication but it isn't actually syncing. Scenario is that we have two GFS clusters. Cluster A has nodes cafs10, cafs20, and cafs30, replicating with each other over a LAN. Cluster B has nodes nvfs10, nvfs20, and nvfs30 also replicating with each other over a LAN. We are geo-replicating data from the A cluster to the B cluster over the internet. SSH key access is set up, allowing all the A nodes password-less access to root on nvfs10 > > Geo-replication was set up using these commands, run on cafs10: > > gluster volume geo-replication gvol0 nvfs10.example.com::gvol0 create ssh-port 8822 no-verify > gluster volume geo-replication gvol0 nvfs10.example.com::gvol0 config remote-gsyncd /usr/lib/x86_64-linux-gnu/glusterfs/gsyncd > gluster volume geo-replication gvol0 nvfs10.example.com::gvol0 start > > However after a very short period of the status being "Initializing..." the status then sits on "Passive": > > # gluster volume geo-replication gvol0 nvfs10.example.com::gvol0 status > MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED > ------------------------------------------------------------------------------------------------------------------------------------------------------------------ > cafs10 gvol0 /nodirectwritedata/gluster/gvol0 root nvfs10.example.com::gvol0 nvfs30.local Passive N/A N/A > cafs30 gvol0 /nodirectwritedata/gluster/gvol0 root nvfs10.example.com::gvol0 N/A Created N/A N/A > cafs20 gvol0 /nodirectwritedata/gluster/gvol0 root nvfs10.example.com::gvol0 N/A Created N/A N/A > > So my questions are: > 1. Why does the status on cafs10 mention "nvfs30.local"? That's the LAN address that nvfs10 replicates with nvfs30 using. It's not accessible from the A cluster, and I didn't use it when configuring geo-replication. > 2. Why does geo-replication sit in Passive status? > > Thanks very much for any assistance. > > > On Tue, 25 Feb 2020 at 15:46, David Cunningham <dcunningham at voisonics.com <mailto:dcunningham at voisonics.com>> wrote: > Hi Aravinda and Sunny, > > Thank you for the replies. We have 3 replicating nodes on the master side, and want to geo-replicate their data to the remote slave side. As I understand it if the master node which had the geo-replication create command run goes down then another node will take over pushing updates to the remote slave. Is that right? > > We have already taken care of adding all master node's SSH keys to the remote slave's authorized_keys externally, so won't include the push-pem part of the create command. > > Mostly I wanted to confirm the geo-replication behaviour on the replicating master nodes if one of them goes down. > > Thank you! > > > On Tue, 25 Feb 2020 at 14:32, Aravinda VK <aravinda at kadalu.io <mailto:aravinda at kadalu.io>> wrote: > Hi David, > > >> On 25-Feb-2020, at 3:45 AM, David Cunningham <dcunningham at voisonics.com <mailto:dcunningham at voisonics.com>> wrote: >> >> Hello, >> >> I've a couple of questions on geo-replication that hopefully someone can help with: >> >> 1. If there are multiple nodes in a cluster on the master side (pushing updates to the geo-replication slave), which node actually does the pushing? Does GlusterFS decide itself automatically? > > Once Geo-replication session is started, one worker will be started corresponding to each Master bricks. Each worker identifies the changes happened in respective brick and sync those changes via Mount. This way load is distributed among Master nodes. In case of Replica sub volume, one worker among the Replica group will become active and participate in the syncing. Other bricks in that Replica group will remain Passive. Passive worker will become Active if the previously Active brick goes down (This is because all Replica bricks will have the same set of changes, syncing from each worker is redundant). > >> >> 2.With regard to copying SSH keys, presumably the SSH key of all master nodes should be authorized on the geo-replication client side? > > Geo-replication session is established between one master node and one remote node. If Geo-rep create command is successful then, > > - SSH keys generated in all master nodes > - Public keys from all master nodes are copied to initiator Master node > - Public keys copied to the Remote node specified in the create command > - Master public keys are distributed to all nodes of remote Cluster and added to respective ~/.ssh/authorized_keys > > After successful Geo-rep create command, any Master node can connect to any remote node via ssh. > > Security: Command prefix is added while adding public key to remote node?s authorized_keys file, So that if anyone gain access using this key can access only gsyncd command. > > ``` > command=gsyncd ssh-key?. > ``` > > >> >> Thanks for your help. >> >> -- >> David Cunningham, Voisonics Limited >> http://voisonics.com/ <http://voisonics.com/> >> USA: +1 213 221 1092 >> New Zealand: +64 (0)28 2558 3782 >> ________ >> >> >> >> Community Meeting Calendar: >> >> Schedule - >> Every Tuesday at 14:30 IST / 09:00 UTC >> Bridge: https://bluejeans.com/441850968 <https://bluejeans.com/441850968> >> >> Gluster-users mailing list >> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> >> https://lists.gluster.org/mailman/listinfo/gluster-users <https://lists.gluster.org/mailman/listinfo/gluster-users> > > > ? > regards > Aravinda Vishwanathapura > https://kadalu <https://kadalu/>.io > > > > -- > David Cunningham, Voisonics Limited > http://voisonics.com/ <http://voisonics.com/> > USA: +1 213 221 1092 > New Zealand: +64 (0)28 2558 3782 > > > -- > David Cunningham, Voisonics Limited > http://voisonics.com/ <http://voisonics.com/> > USA: +1 213 221 1092 > New Zealand: +64 (0)28 2558 3782-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200302/a68a22d5/attachment.html>