Strahil Nikolov
2022-Mar-05 21:22 UTC
[Gluster-users] proper way to temporarily remove brick server from replica cluster to avoid kvm guest disruption
Hey Todd,
can you provide 'gluster volume info <VOLUME>' ?
Best Regards,Strahil Nikolov
On Sat, Mar 5, 2022 at 18:17, Todd Pfaff<pfaff at rhpcs.mcmaster.ca>
wrote: I have a replica volume created as:
gluster volume create vol1 replica 4 \
? host{1,2,3,4}:/mnt/gluster/brick1/data \
? force
All hosts host{1,2,3,4} mount this volume as:
localhost:/vol1 /mnt/gluster/vol1 glusterfs defaults
Some other hosts are trusted peers but do not contribute bricks, and they
also mount vol1 in the same way:
localhost:/vol1 /mnt/gluster/vol1 glusterfs defaults
All hosts run CentOS 7.9, and all are running glusterfs 9.4 or 9.5 from
centos-release-gluster9-1.0-1.el7.noarch.
All hosts run kvm guests that use qcow2 files for root filesystems that
are stored on gluster volume vol1.
This is all working well, as long as none of host{1,2,3,4} go offline.
I want to take one of host{1,2,3,4} offline temporarily for maintenance.
I'll refer to this as hostX.
I understand that hostX will need to be healed when it comes back online.
I would, of course, migrate guests from hostX to another host, in which
case hostX would then only be participating as a gluster replica brick
provider and serving gluster client requests.
What I've experienced is that if I take one of host{1,2,3,4} offline, this
can disrupt some of the VM guests on various other hosts such that their
root filesystems go to read-only.
What I'm looking for here are suggestions as to how to properly take one
of host{1,2,3,4} offline to avoid such disruption or how to tune the
libvirt kvm hosts and guests to be sufficiently resilient in the face of
taking one gluster replica node offline.
Thanks,
Todd
________
Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users at gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20220305/798c2b3e/attachment.html>
Strahil Nikolov
2022-Mar-05 21:22 UTC
[Gluster-users] proper way to temporarily remove brick server from replica cluster to avoid kvm guest disruption
Also, 'gluster pool list' .
Best Regards,Strahil Nikolov
On Sat, Mar 5, 2022 at 23:22, Strahil Nikolov<hunter86_bg at yahoo.com>
wrote: Hey Todd,
can you provide 'gluster volume info <VOLUME>' ?
Best Regards,Strahil Nikolov
On Sat, Mar 5, 2022 at 18:17, Todd Pfaff<pfaff at rhpcs.mcmaster.ca>
wrote: I have a replica volume created as:
gluster volume create vol1 replica 4 \
? host{1,2,3,4}:/mnt/gluster/brick1/data \
? force
All hosts host{1,2,3,4} mount this volume as:
localhost:/vol1 /mnt/gluster/vol1 glusterfs defaults
Some other hosts are trusted peers but do not contribute bricks, and they
also mount vol1 in the same way:
localhost:/vol1 /mnt/gluster/vol1 glusterfs defaults
All hosts run CentOS 7.9, and all are running glusterfs 9.4 or 9.5 from
centos-release-gluster9-1.0-1.el7.noarch.
All hosts run kvm guests that use qcow2 files for root filesystems that
are stored on gluster volume vol1.
This is all working well, as long as none of host{1,2,3,4} go offline.
I want to take one of host{1,2,3,4} offline temporarily for maintenance.
I'll refer to this as hostX.
I understand that hostX will need to be healed when it comes back online.
I would, of course, migrate guests from hostX to another host, in which
case hostX would then only be participating as a gluster replica brick
provider and serving gluster client requests.
What I've experienced is that if I take one of host{1,2,3,4} offline, this
can disrupt some of the VM guests on various other hosts such that their
root filesystems go to read-only.
What I'm looking for here are suggestions as to how to properly take one
of host{1,2,3,4} offline to avoid such disruption or how to tune the
libvirt kvm hosts and guests to be sufficiently resilient in the face of
taking one gluster replica node offline.
Thanks,
Todd
________
Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users at gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20220305/30eb584a/attachment.html>
Todd Pfaff
2022-Mar-06 02:06 UTC
[Gluster-users] proper way to temporarily remove brick server from replica cluster to avoid kvm guest disruption
[root at vh1 ~]# gluster volume info vol1 Volume Name: vol1 Type: Replicate Volume ID: dfd681bb-5b68-4831-9863-e13f9f027620 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 4 = 4 Transport-type: tcp Bricks: Brick1: vh1:/pool/gluster/brick1/data Brick2: vh2:/pool/gluster/brick1/data Brick3: vh3:/pool/gluster/brick1/data Brick4: vh4:/pool/gluster/brick1/data Options Reconfigured: transport.address-family: inet nfs.disable: on performance.client-io-threads: off [root at vh1 ~]# gluster pool list UUID Hostname State 75fc4258-fabd-47c9-8198-bbe6e6a906fb vh2 Connected 00697e28-96c0-4534-a314-e878070b653d vh3 Connected 2a9b891b-35d0-496c-bb06-f5dab4feb6bf vh4 Connected 8ba6fb80-3b13-4379-94cf-22662cbb48a2 vh5 Disconnected 1298d334-3500-4b40-a8bd-cc781f7349d0 vh6 Connected 79a533ac-3d89-44b9-b0ce-823cfec8cf75 vh7 Connected 4141cd74-9c13-404c-a02c-f553fa19bc22 vh8 Connected On Sat, 5 Mar 2022, Strahil Nikolov wrote:> Hey Todd, > > can you provide 'gluster volume info <VOLUME>' ? > > Best Regards, > Strahil Nikolov > > On Sat, Mar 5, 2022 at 18:17, Todd Pfaff > <pfaff at rhpcs.mcmaster.ca> wrote: > I have a replica volume created as: > > gluster volume create vol1 replica 4 \ > ? host{1,2,3,4}:/mnt/gluster/brick1/data \ > ? force > > > All hosts host{1,2,3,4} mount this volume as: > > localhost:/vol1 /mnt/gluster/vol1 glusterfs defaults > > > Some other hosts are trusted peers but do not contribute bricks, and > they > also mount vol1 in the same way: > > localhost:/vol1 /mnt/gluster/vol1 glusterfs defaults > > > All hosts run CentOS 7.9, and all are running glusterfs 9.4 or 9.5 from > centos-release-gluster9-1.0-1.el7.noarch. > > > All hosts run kvm guests that use qcow2 files for root filesystems that > are stored on gluster volume vol1. > > > This is all working well, as long as none of host{1,2,3,4} go offline. > > > I want to take one of host{1,2,3,4} offline temporarily for > maintenance. > I'll refer to this as hostX. > > I understand that hostX will need to be healed when it comes back > online. > > I would, of course, migrate guests from hostX to another host, in which > case hostX would then only be participating as a gluster replica brick > provider and serving gluster client requests. > > What I've experienced is that if I take one of host{1,2,3,4} offline, > this > can disrupt some of the VM guests on various other hosts such that > their > root filesystems go to read-only. > > What I'm looking for here are suggestions as to how to properly take > one > of host{1,2,3,4} offline to avoid such disruption or how to tune the > libvirt kvm hosts and guests to be sufficiently resilient in the face > of > taking one gluster replica node offline. > > Thanks, > Todd > ________ > > > > Community Meeting Calendar: > > Schedule - > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > Bridge: https://meet.google.com/cpu-eiue-hvk > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users > > >