Hi all, I am planning my new gluster system and tested things out in a bunch of virtual machines. I need a bit of help to understand how geo-replication behaves. I have a master gluster cluster replica 2 (in production I will use an arbiter and replicatied/distributed) and the geo cluster is distributed with 2 machines. (in production I will have the geo cluster distributed) Everything is up and running and creating files from client both replicates and is distributed in the geo cluster. The thing I am wondering about is: When I run: gluster volume geo-replication status I see both slave nodes one is active and the other is passive. MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- gluster1 interbullfs /interbullfs geouser ssh://geouser at gluster-geo1::interbullfs-geo gluster-geo2 Active Changelog Crawl 2018-02-06 11:46:08 gluster2 interbullfs /interbullfs geouser ssh://geouser at gluster-geo1::interbullfs-geo gluster-geo1 Passive N/A N/A If I shutdown the active slave the status changes to faulty and the other one continues to be passive. MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ---------------------------------------------------------------------------------------------------------------------------------------------------------------- gluster1 interbullfs /interbullfs geouser ssh://geouser at gluster-geo1::interbullfs-geo N/A Faulty N/A N/A gluster2 interbullfs /interbullfs geouser ssh://geouser at gluster-geo1::interbullfs-geo gluster-geo1 Passive N/A N/A In my understanding I thought that if the active slave stopped working the passive slave should become active and should continue to replicate from master. Am I wrong? Is there just one active slave if it is setup as a distributed system? What I use: Centos 7, gluster 3.12 I have followed the geo instructions: http://docs.gluster.org/en/latest/Administrator%20Guide/Geo%20Replication/ Many thanks in advance! Bets regards Marcus -- ************************************************** * Marcus Peders?n * * System administrator * ************************************************** * Interbull Centre * * ================ * * Department of Animal Breeding & Genetics ? SLU * * Box 7023, SE-750 07 * * Uppsala, Sweden * ************************************************** * Visiting address: * * Room 55614, Ulls v?g 26, Ultuna * * Uppsala * * Sweden * * * * Tel: +46-(0)18-67 1962 * * * ************************************************** * ISO 9001 Bureau Veritas No SE004561-1 * **************************************************
Hi again, I made some more tests and the behavior I get is that if any of the slaves are down the geo-replication stops working. It this the way distributed volumes work, if one server goes down the entire system stops to work? The servers that are online do not continue to work? Sorry, for asking stupid questions. Best regards Marcus On Tue, Feb 06, 2018 at 12:09:40PM +0100, Marcus Peders?n wrote:> Hi all, > > I am planning my new gluster system and tested things out in > a bunch of virtual machines. > I need a bit of help to understand how geo-replication behaves. > > I have a master gluster cluster replica 2 > (in production I will use an arbiter and replicatied/distributed) > and the geo cluster is distributed with 2 machines. > (in production I will have the geo cluster distributed) > > Everything is up and running and creating files from client both > replicates and is distributed in the geo cluster. > > The thing I am wondering about is: > When I run: gluster volume geo-replication status > I see both slave nodes one is active and the other is passive. > > MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED > --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > gluster1 interbullfs /interbullfs geouser ssh://geouser at gluster-geo1::interbullfs-geo gluster-geo2 Active Changelog Crawl 2018-02-06 11:46:08 > gluster2 interbullfs /interbullfs geouser ssh://geouser at gluster-geo1::interbullfs-geo gluster-geo1 Passive N/A N/A > > > If I shutdown the active slave the status changes to faulty > and the other one continues to be passive. > > MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED > ---------------------------------------------------------------------------------------------------------------------------------------------------------------- > gluster1 interbullfs /interbullfs geouser ssh://geouser at gluster-geo1::interbullfs-geo N/A Faulty N/A N/A > gluster2 interbullfs /interbullfs geouser ssh://geouser at gluster-geo1::interbullfs-geo gluster-geo1 Passive N/A N/A > > > In my understanding I thought that if the active slave stopped > working the passive slave should become active and should > continue to replicate from master. > > Am I wrong? Is there just one active slave if it is setup as > a distributed system? > > What I use: > Centos 7, gluster 3.12 > I have followed the geo instructions: > http://docs.gluster.org/en/latest/Administrator%20Guide/Geo%20Replication/ > > Many thanks in advance! > > Bets regards > Marcus > > -- > ************************************************** > * Marcus Peders?n * > * System administrator * > ************************************************** > * Interbull Centre * > * ================ * > * Department of Animal Breeding & Genetics ? SLU * > * Box 7023, SE-750 07 * > * Uppsala, Sweden * > ************************************************** > * Visiting address: * > * Room 55614, Ulls v?g 26, Ultuna * > * Uppsala * > * Sweden * > * * > * Tel: +46-(0)18-67 1962 * > * * > ************************************************** > * ISO 9001 Bureau Veritas No SE004561-1 * > ************************************************** > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users-- ************************************************** * Marcus Peders?n * * System administrator * ************************************************** * Interbull Centre * * ================ * * Department of Animal Breeding & Genetics ? SLU * * Box 7023, SE-750 07 * * Uppsala, Sweden * ************************************************** * Visiting address: * * Room 55614, Ulls v?g 26, Ultuna * * Uppsala * * Sweden * * * * Tel: +46-(0)18-67 1962 * * * ************************************************** * ISO 9001 Bureau Veritas No SE004561-1 * **************************************************
We are happy to help you out. Please find the answers inline. On Tue, Feb 6, 2018 at 4:39 PM, Marcus Peders?n <marcus.pedersen at slu.se> wrote:> Hi all, > > I am planning my new gluster system and tested things out in > a bunch of virtual machines. > I need a bit of help to understand how geo-replication behaves. > > I have a master gluster cluster replica 2 > (in production I will use an arbiter and replicatied/distributed) > and the geo cluster is distributed with 2 machines. > (in production I will have the geo cluster distributed) >It's recommended to use slave also to be distribute replicate/aribiter/ec. Choosing only distribute will cause issues when of the slave node is down and a file is being synced which belongs to that node. It would not sync later.> Everything is up and running and creating files from client both > replicates and is distributed in the geo cluster. > > The thing I am wondering about is: > When I run: gluster volume geo-replication status > I see both slave nodes one is active and the other is passive. > > MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE > SLAVE NODE STATUS CRAWL STATUS > LAST_SYNCED > ------------------------------------------------------------ > ------------------------------------------------------------ > --------------------------------------------------- > gluster1 interbullfs /interbullfs geouser > ssh://geouser at gluster-geo1::interbullfs-geo gluster-geo2 Active > Changelog Crawl 2018-02-06 11:46:08 > gluster2 interbullfs /interbullfs geouser > ssh://geouser at gluster-geo1::interbullfs-geo gluster-geo1 Passive > N/A N/A > > > If I shutdown the active slave the status changes to faulty > and the other one continues to be passive. >> MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE > SLAVE NODE STATUS CRAWL STATUS > LAST_SYNCED > ------------------------------------------------------------ > ------------------------------------------------------------ > ---------------------------------------- > gluster1 interbullfs /interbullfs geouser > ssh://geouser at gluster-geo1::interbullfs-geo N/A Faulty > N/A N/A > gluster2 interbullfs /interbullfs geouser > ssh://geouser at gluster-geo1::interbullfs-geo gluster-geo1 Passive > N/A N/A > > > In my understanding I thought that if the active slave stopped > working the passive slave should become active and should > continue to replicate from master. > > Am I wrong? Is there just one active slave if it is setup as > a distributed system? >The Active/Passive notion is for master node. If gluster1 master node is down glusterd2 master node will become Active. It's not for slave node.> > What I use: > Centos 7, gluster 3.12 > I have followed the geo instructions: > http://docs.gluster.org/en/latest/Administrator%20Guide/Geo%20Replication/ > > Many thanks in advance! > > Bets regards > Marcus > > -- > ************************************************** > * Marcus Peders?n * > * System administrator * > ************************************************** > * Interbull Centre * > * ================ * > * Department of Animal Breeding & Genetics ? SLU * > * Box 7023, SE-750 07 * > * Uppsala, Sweden * > ************************************************** > * Visiting address: * > * Room 55614, Ulls v?g 26, Ultuna * > * Uppsala * > * Sweden * > * * > * Tel: +46-(0)18-67 1962 * > * * > ************************************************** > * ISO 9001 Bureau Veritas No SE004561-1 * > ************************************************** > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users-- Thanks and Regards, Kotresh H R -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180207/c4a9f694/attachment.html>
Answers in line. On Tue, Feb 6, 2018 at 6:24 PM, Marcus Peders?n <marcus.pedersen at slu.se> wrote:> Hi again, > I made some more tests and the behavior I get is that if any of > the slaves are down the geo-replication stops working. > It this the way distributed volumes work, if one server goes down > the entire system stops to work? > The servers that are online do not continue to work? >If one of the slave node is down, the corresponding master node does connect to other slave node which is up. But if the primary slave node (the one used to create geo-rep session) is down, that connection remains Faulty until it is brought back up.> > Sorry, for asking stupid questions. > > Best regards > Marcus > > > On Tue, Feb 06, 2018 at 12:09:40PM +0100, Marcus Peders?n wrote: > > Hi all, > > > > I am planning my new gluster system and tested things out in > > a bunch of virtual machines. > > I need a bit of help to understand how geo-replication behaves. > > > > I have a master gluster cluster replica 2 > > (in production I will use an arbiter and replicatied/distributed) > > and the geo cluster is distributed with 2 machines. > > (in production I will have the geo cluster distributed) > > > > Everything is up and running and creating files from client both > > replicates and is distributed in the geo cluster. > > > > The thing I am wondering about is: > > When I run: gluster volume geo-replication status > > I see both slave nodes one is active and the other is passive. > > > > MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE > SLAVE NODE STATUS CRAWL STATUS > LAST_SYNCED > > ------------------------------------------------------------ > ------------------------------------------------------------ > --------------------------------------------------- > > gluster1 interbullfs /interbullfs geouser > ssh://geouser at gluster-geo1::interbullfs-geo gluster-geo2 Active > Changelog Crawl 2018-02-06 11:46:08 > > gluster2 interbullfs /interbullfs geouser > ssh://geouser at gluster-geo1::interbullfs-geo gluster-geo1 Passive > N/A N/A > > > > > > If I shutdown the active slave the status changes to faulty > > and the other one continues to be passive. > > > > MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE > SLAVE NODE STATUS CRAWL STATUS > LAST_SYNCED > > ------------------------------------------------------------ > ------------------------------------------------------------ > ---------------------------------------- > > gluster1 interbullfs /interbullfs geouser > ssh://geouser at gluster-geo1::interbullfs-geo N/A Faulty > N/A N/A > > gluster2 interbullfs /interbullfs geouser > ssh://geouser at gluster-geo1::interbullfs-geo gluster-geo1 Passive > N/A N/A > > > > > > In my understanding I thought that if the active slave stopped > > working the passive slave should become active and should > > continue to replicate from master. > > > > Am I wrong? Is there just one active slave if it is setup as > > a distributed system? > > > > What I use: > > Centos 7, gluster 3.12 > > I have followed the geo instructions: > > http://docs.gluster.org/en/latest/Administrator%20Guide/ > Geo%20Replication/ > > > > Many thanks in advance! > > > > Bets regards > > Marcus > > > > -- > > ************************************************** > > * Marcus Peders?n * > > * System administrator * > > ************************************************** > > * Interbull Centre * > > * ================ * > > * Department of Animal Breeding & Genetics ? SLU * > > * Box 7023, SE-750 07 * > > * Uppsala, Sweden * > > ************************************************** > > * Visiting address: * > > * Room 55614, Ulls v?g 26, Ultuna * > > * Uppsala * > > * Sweden * > > * * > > * Tel: +46-(0)18-67 1962 * > > * * > > ************************************************** > > * ISO 9001 Bureau Veritas No SE004561-1 * > > ************************************************** > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users at gluster.org > > http://lists.gluster.org/mailman/listinfo/gluster-users > > -- > ************************************************** > * Marcus Peders?n * > * System administrator * > ************************************************** > * Interbull Centre * > * ================ * > * Department of Animal Breeding & Genetics ? SLU * > * Box 7023, SE-750 07 * > * Uppsala, Sweden * > ************************************************** > * Visiting address: * > * Room 55614, Ulls v?g 26, Ultuna * > * Uppsala * > * Sweden * > * * > * Tel: +46-(0)18-67 1962 * > * * > ************************************************** > * ISO 9001 Bureau Veritas No SE004561-1 * > ************************************************** > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users >-- Thanks and Regards, Kotresh H R -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180207/4fd32b99/attachment.html>
Thank you for your help! Just to make things clear to me (and get a better understanding of gluster): So, if I make the slave cluster just distributed and node 1 goes down, data (say file.txt) that belongs to node 1 will not be synced. When node 1 comes back up does the master not realize that file.txt has not been synced and makes sure that it is synced when it has contact with node 1 again? So file.txt will not exist on node 1 at all? I did a small test on my testing machines. Turned one of the geo machines off and created 10000 files containing one short string in the master nodes. Nothing became synced with the geo slaves. When I turned on the geo machine again all 10000 files were synced to the geo slaves. Ofcause devided between the two machines. Is this the right/expected behavior of geo-replication with a distributed cluster? Many thanks in advance! Regards Marcus On Wed, Feb 07, 2018 at 06:39:20PM +0530, Kotresh Hiremath Ravishankar wrote:> We are happy to help you out. Please find the answers inline. > > On Tue, Feb 6, 2018 at 4:39 PM, Marcus Peders?n <marcus.pedersen at slu.se> > wrote: > > > Hi all, > > > > I am planning my new gluster system and tested things out in > > a bunch of virtual machines. > > I need a bit of help to understand how geo-replication behaves. > > > > I have a master gluster cluster replica 2 > > (in production I will use an arbiter and replicatied/distributed) > > and the geo cluster is distributed with 2 machines. > > (in production I will have the geo cluster distributed) > > > > It's recommended to use slave also to be distribute replicate/aribiter/ec. > Choosing only distribute will cause issues > when of the slave node is down and a file is being synced which belongs to > that node. It would not sync > later. > > > > Everything is up and running and creating files from client both > > replicates and is distributed in the geo cluster. > > > > The thing I am wondering about is: > > When I run: gluster volume geo-replication status > > I see both slave nodes one is active and the other is passive. > > > > MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE > > SLAVE NODE STATUS CRAWL STATUS > > LAST_SYNCED > > ------------------------------------------------------------ > > ------------------------------------------------------------ > > --------------------------------------------------- > > gluster1 interbullfs /interbullfs geouser > > ssh://geouser at gluster-geo1::interbullfs-geo gluster-geo2 Active > > Changelog Crawl 2018-02-06 11:46:08 > > gluster2 interbullfs /interbullfs geouser > > ssh://geouser at gluster-geo1::interbullfs-geo gluster-geo1 Passive > > N/A N/A > > > > > > If I shutdown the active slave the status changes to faulty > > and the other one continues to be passive. > > > > > MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE > > SLAVE NODE STATUS CRAWL STATUS > > LAST_SYNCED > > ------------------------------------------------------------ > > ------------------------------------------------------------ > > ---------------------------------------- > > gluster1 interbullfs /interbullfs geouser > > ssh://geouser at gluster-geo1::interbullfs-geo N/A Faulty > > N/A N/A > > gluster2 interbullfs /interbullfs geouser > > ssh://geouser at gluster-geo1::interbullfs-geo gluster-geo1 Passive > > N/A N/A > > > > > > In my understanding I thought that if the active slave stopped > > working the passive slave should become active and should > > continue to replicate from master. > > > > Am I wrong? Is there just one active slave if it is setup as > > a distributed system? > > > > The Active/Passive notion is for master node. If gluster1 master node is > down glusterd2 master node will become Active. > It's not for slave node. > > > > > > > What I use: > > Centos 7, gluster 3.12 > > I have followed the geo instructions: > > http://docs.gluster.org/en/latest/Administrator%20Guide/Geo%20Replication/ > > > > Many thanks in advance! > > > > Bets regards > > Marcus > > > > -- > > ************************************************** > > * Marcus Peders?n * > > * System administrator * > > ************************************************** > > * Interbull Centre * > > * ================ * > > * Department of Animal Breeding & Genetics ? SLU * > > * Box 7023, SE-750 07 * > > * Uppsala, Sweden * > > ************************************************** > > * Visiting address: * > > * Room 55614, Ulls v?g 26, Ultuna * > > * Uppsala * > > * Sweden * > > * * > > * Tel: +46-(0)18-67 1962 * > > * * > > ************************************************** > > * ISO 9001 Bureau Veritas No SE004561-1 * > > ************************************************** > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users at gluster.org > > http://lists.gluster.org/mailman/listinfo/gluster-users > > > > > -- > Thanks and Regards, > Kotresh H R-- ************************************************** * Marcus Peders?n * * System administrator * ************************************************** * Interbull Centre * * ================ * * Department of Animal Breeding & Genetics ? SLU * * Box 7023, SE-750 07 * * Uppsala, Sweden * ************************************************** * Visiting address: * * Room 55614, Ulls v?g 26, Ultuna * * Uppsala * * Sweden * * * * Tel: +46-(0)18-67 1962 * * * ************************************************** * ISO 9001 Bureau Veritas No SE004561-1 * **************************************************