Hi all, I hope this message finds you well.
I've been running a Gluster volume (32 bricks in distributed,
replicated mode) on my 2 home servers for about 1,5 years now. I'm
generally very happy with it!
Disks are distributed across 4 enclosures (2 enclosures per server). At
one point one of these enclosures failed (8 bricks down on 1 server)
but due to the awesomeness of Gluster (and my lack of monitoring :-( )
I only noticed this after about 6 weeks... This left me with A LOT of
pending heals, about 40k per brick if I remember correctly.
Well, I brought back online the failed bricks and let the the gluster
heal. And it did, mostly... It left about 1 to 4 pending heals on
multiple bricks that won't heal, no matter what I've tried. I just let
them be for some time until I had time to figure out what to do with
them.
Also, because some disks were about to fail, I started replacing
bricks. And taking advantage of this, I replaced them with a larger
disk (4TB -> 8TB). ?Healing took care of copying all data to the new
brick and finished succesfully. However, for some reason I do not see
an increase in total space on the systems were I have mounted the
Gluster volume.
So in short, I have 2 issues:
1. Some Pending Heals that I cannot get rid of
2. Gluster total space being reported incorrect
Ideally, I'd like to address issue 2 first (seems the easiest fix), and
then focus on issue 1. Should that be ok?
I hope you guys can help me with these 2 issues. Thanks in advance!
Added as attachent:
* Commands used to replace the brick
* "gluster volume status data detail" output
* "grep -n "shared-brick-count"
/var/lib/glusterd/vols/data/*" output,
as I read somewhere that this may be relevant...
--
groet / cheers,
Patrick Dijkgraaf
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20220416/786f1025/attachment.html>
-------------- next part --------------
[root at dorus ~]# gluster volume status data detail
Status of volume: data
------------------------------------------------------------------------------
Brick : Brick dorus:/mnt/bricks/data/01/VGK8VEXG
TCP Port : 50816
RDMA Port : 0
Online : Y
Pid : 134061
File System : xfs
Device : /dev/sda2
Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
Inode Size : 512
Disk Space Free : 4.1TB
Total Disk Space : 7.3TB
Inode Count : 781402176
Free Inodes : 781049061
------------------------------------------------------------------------------
Brick : Brick cornelis:/mnt/bricks/data/01/VRK4L7MK
TCP Port : 54462
RDMA Port : 0
Online : Y
Pid : 146792
File System : xfs
Device : /dev/sda2
Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
Inode Size : 512
Disk Space Free : 4.1TB
Total Disk Space : 7.3TB
Inode Count : 781402176
Free Inodes : 781049061
------------------------------------------------------------------------------
Brick : Brick dorus:/mnt/bricks/data/02/VKKP6LAY
TCP Port : 53275
RDMA Port : 0
Online : Y
Pid : 134066
File System : xfs
Device : /dev/sdb2
Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
Inode Size : 512
Disk Space Free : 1.4TB
Total Disk Space : 7.3TB
Inode Count : 781402176
Free Inodes : 780984850
------------------------------------------------------------------------------
Brick : Brick cornelis:/mnt/bricks/data/02/VAG4UNTL
TCP Port : 52386
RDMA Port : 0
Online : Y
Pid : 146797
File System : xfs
Device : /dev/sdb2
Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
Inode Size : 512
Disk Space Free : 1.4TB
Total Disk Space : 7.3TB
Inode Count : 781402176
Free Inodes : 780984847
------------------------------------------------------------------------------
Brick : Brick dorus:/mnt/bricks/data/03/WD-WCC7K7PCN1FH
TCP Port : 56176
RDMA Port : 0
Online : Y
Pid : 134090
File System : xfs
Device : /dev/sdc2
Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
Inode Size : 512
Disk Space Free : 793.4GB
Total Disk Space : 3.6TB
Inode Count : 390701440
Free Inodes : 390345631
------------------------------------------------------------------------------
Brick : Brick cornelis:/mnt/bricks/data/03/WD-WCC7K2ZF967P
TCP Port : 51349
RDMA Port : 0
Online : Y
Pid : 146821
File System : xfs
Device : /dev/sdc2
Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
Inode Size : 512
Disk Space Free : 793.4GB
Total Disk Space : 3.6TB
Inode Count : 390701440
Free Inodes : 390345632
------------------------------------------------------------------------------
Brick : Brick dorus:/mnt/bricks/data/04/WD-WCC4E6XXFZK5
TCP Port : 59307
RDMA Port : 0
Online : Y
Pid : 134114
File System : xfs
Device : /dev/sdd2
Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
Inode Size : 512
Disk Space Free : 700.0GB
Total Disk Space : 3.6TB
Inode Count : 390701440
Free Inodes : 390344717
------------------------------------------------------------------------------
Brick : Brick cornelis:/mnt/bricks/data/04/WD-WCC4E4DEL2R7
TCP Port : 50401
RDMA Port : 0
Online : Y
Pid : 146845
File System : xfs
Device : /dev/sdd2
Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
Inode Size : 512
Disk Space Free : 700.0GB
Total Disk Space : 3.6TB
Inode Count : 390701440
Free Inodes : 390344716
------------------------------------------------------------------------------
Brick : Brick dorus:/mnt/bricks/data/05/VK0RBEUY
TCP Port : 52860
RDMA Port : 0
Online : Y
Pid : 134137
File System : xfs
Device : /dev/sde2
Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
Inode Size : 512
Disk Space Free : 1.2TB
Total Disk Space : 7.3TB
Inode Count : 781402176
Free Inodes : 780986042
------------------------------------------------------------------------------
Brick : Brick cornelis:/mnt/bricks/data/05/VKKRGDRY
TCP Port : 49846
RDMA Port : 0
Online : Y
Pid : 146869
File System : xfs
Device : /dev/sde2
Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
Inode Size : 512
Disk Space Free : 1.2TB
Total Disk Space : 7.3TB
Inode Count : 781402176
Free Inodes : 780986043
------------------------------------------------------------------------------
Brick : Brick dorus:/mnt/bricks/data/06/R6GLYPZY
TCP Port : 57091
RDMA Port : 0
Online : Y
Pid : 134160
File System : xfs
Device : /dev/sdf2
Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
Inode Size : 512
Disk Space Free : 1.1TB
Total Disk Space : 7.3TB
Inode Count : 781402176
Free Inodes : 780985689
------------------------------------------------------------------------------
Brick : Brick cornelis:/mnt/bricks/data/06/R6GUJRGY
TCP Port : 53391
RDMA Port : 0
Online : Y
Pid : 146891
File System : xfs
Device : /dev/sdf2
Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
Inode Size : 512
Disk Space Free : 1.1TB
Total Disk Space : 7.3TB
Inode Count : 781402176
Free Inodes : 780985687
------------------------------------------------------------------------------
Brick : Brick dorus:/mnt/bricks/data/07/WD-WCC7K3KZAAY7
TCP Port : 60264
RDMA Port : 0
Online : Y
Pid : 134184
File System : xfs
Device : /dev/sdg2
Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
Inode Size : 512
Disk Space Free : 738.4GB
Total Disk Space : 3.6TB
Inode Count : 390701440
Free Inodes : 390345263
------------------------------------------------------------------------------
Brick : Brick cornelis:/mnt/bricks/data/07/WD-WCC4E2FRNVLS
TCP Port : 56653
RDMA Port : 0
Online : Y
Pid : 146933
File System : xfs
Device : /dev/sdg2
Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
Inode Size : 512
Disk Space Free : 738.4GB
Total Disk Space : 3.6TB
Inode Count : 390701440
Free Inodes : 390345267
------------------------------------------------------------------------------
Brick : Brick dorus:/mnt/bricks/data/08/VKKNNE7Y
TCP Port : 49829
RDMA Port : 0
Online : Y
Pid : 134231
File System : xfs
Device : /dev/sdh2
Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
Inode Size : 512
Disk Space Free : 1.2TB
Total Disk Space : 7.3TB
Inode Count : 781402176
Free Inodes : 780984701
------------------------------------------------------------------------------
Brick : Brick cornelis:/mnt/bricks/data/08/VK0PTDWY
TCP Port : 56208
RDMA Port : 0
Online : Y
Pid : 146962
File System : xfs
Device : /dev/sdh2
Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
Inode Size : 512
Disk Space Free : 1.2TB
Total Disk Space : 7.3TB
Inode Count : 781402176
Free Inodes : 780984676
------------------------------------------------------------------------------
Brick : Brick dorus:/mnt/bricks/data/09/CA0ZRMEK
TCP Port : 52365
RDMA Port : 0
Online : Y
Pid : 134259
File System : xfs
Device : /dev/sdj2
Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
Inode Size : 512
Disk Space Free : 4.1TB
Total Disk Space : 7.3TB
Inode Count : 781402176
Free Inodes : 781049832
------------------------------------------------------------------------------
Brick : Brick cornelis:/mnt/bricks/data/09/VRK4L9PK
TCP Port : 56532
RDMA Port : 0
Online : Y
Pid : 146992
File System : xfs
Device : /dev/sdj2
Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
Inode Size : 512
Disk Space Free : 4.1TB
Total Disk Space : 7.3TB
Inode Count : 781402176
Free Inodes : 781049832
------------------------------------------------------------------------------
Brick : Brick dorus:/mnt/bricks/data/10/WD-WCC7K2ZF95D4
TCP Port : 60470
RDMA Port : 0
Online : Y
Pid : 134265
File System : xfs
Device : /dev/sdi2
Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
Inode Size : 512
Disk Space Free : 772.2GB
Total Disk Space : 3.6TB
Inode Count : 390701440
Free Inodes : 390344692
------------------------------------------------------------------------------
Brick : Brick cornelis:/mnt/bricks/data/10/CA0ZYD5K
TCP Port : 60153
RDMA Port : 0
Online : Y
Pid : 147482
File System : xfs
Device : /dev/sdi2
Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
Inode Size : 512
Disk Space Free : 6.3TB
Total Disk Space : 7.3TB
Inode Count : 781402176
Free Inodes : 781224223
------------------------------------------------------------------------------
Brick : Brick dorus:/mnt/bricks/data/11/VK0N9YMY
TCP Port : 54556
RDMA Port : 0
Online : Y
Pid : 134313
File System : xfs
Device : /dev/sdk2
Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
Inode Size : 512
Disk Space Free : 1.4TB
Total Disk Space : 7.3TB
Inode Count : 781402176
Free Inodes : 780985141
------------------------------------------------------------------------------
Brick : Brick cornelis:/mnt/bricks/data/11/R6GSPRKY
TCP Port : 60423
RDMA Port : 0
Online : Y
Pid : 146997
File System : xfs
Device : /dev/sdk2
Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
Inode Size : 512
Disk Space Free : 1.4TB
Total Disk Space : 7.3TB
Inode Count : 781402176
Free Inodes : 780985144
------------------------------------------------------------------------------
Brick : Brick dorus:/mnt/bricks/data/12/WD-WCC4E4UREPNH
TCP Port : 59959
RDMA Port : 0
Online : Y
Pid : 134336
File System : xfs
Device : /dev/sdl2
Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
Inode Size : 512
Disk Space Free : 505.3GB
Total Disk Space : 3.6TB
Inode Count : 390701440
Free Inodes : 390344901
------------------------------------------------------------------------------
Brick : Brick cornelis:/mnt/bricks/data/12/WD-WCC7K7PCNUDT
TCP Port : 51963
RDMA Port : 0
Online : Y
Pid : 147025
File System : xfs
Device : /dev/sdl2
Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
Inode Size : 512
Disk Space Free : 505.3GB
Total Disk Space : 3.6TB
Inode Count : 390701440
Free Inodes : 390344897
------------------------------------------------------------------------------
Brick : Brick dorus:/mnt/bricks/data/13/R6GV252Y
TCP Port : 50713
RDMA Port : 0
Online : Y
Pid : 134364
File System : xfs
Device : /dev/sdm2
Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
Inode Size : 512
Disk Space Free : 1.2TB
Total Disk Space : 7.3TB
Inode Count : 781402176
Free Inodes : 780984671
------------------------------------------------------------------------------
Brick : Brick cornelis:/mnt/bricks/data/13/VKKMP42Y
TCP Port : 49773
RDMA Port : 0
Online : Y
Pid : 147067
File System : xfs
Device : /dev/sdm2
Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
Inode Size : 512
Disk Space Free : 1.3TB
Total Disk Space : 7.3TB
Inode Count : 781402176
Free Inodes : 780984814
------------------------------------------------------------------------------
Brick : Brick dorus:/mnt/bricks/data/14/VGJYYJXG
TCP Port : 58434
RDMA Port : 0
Online : Y
Pid : 134388
File System : xfs
Device : /dev/sdn2
Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
Inode Size : 512
Disk Space Free : 4.0TB
Total Disk Space : 7.3TB
Inode Count : 781402176
Free Inodes : 781046270
------------------------------------------------------------------------------
Brick : Brick cornelis:/mnt/bricks/data/14/CA0VWVTK
TCP Port : 54407
RDMA Port : 0
Online : Y
Pid : 147075
File System : xfs
Device : /dev/sdn2
Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
Inode Size : 512
Disk Space Free : 4.0TB
Total Disk Space : 7.3TB
Inode Count : 781402176
Free Inodes : 781050455
------------------------------------------------------------------------------
Brick : Brick dorus:/mnt/bricks/data/15/WD-WCC7K1HU54PX
TCP Port : 60452
RDMA Port : 0
Online : Y
Pid : 134415
File System : xfs
Device : /dev/sdo2
Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
Inode Size : 512
Disk Space Free : 579.2GB
Total Disk Space : 3.6TB
Inode Count : 390701440
Free Inodes : 390345128
------------------------------------------------------------------------------
Brick : Brick cornelis:/mnt/bricks/data/15/WD-WCC4E0UP9SXY
TCP Port : 50382
RDMA Port : 0
Online : Y
Pid : 147097
File System : xfs
Device : /dev/sdo2
Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
Inode Size : 512
Disk Space Free : 599.8GB
Total Disk Space : 3.6TB
Inode Count : 390701440
Free Inodes : 390345230
------------------------------------------------------------------------------
Brick : Brick dorus:/mnt/bricks/data/16/R6GV83HY
TCP Port : 50703
RDMA Port : 0
Online : Y
Pid : 134439
File System : xfs
Device : /dev/sdp2
Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
Inode Size : 512
Disk Space Free : 1.2TB
Total Disk Space : 7.3TB
Inode Count : 781402176
Free Inodes : 780984648
------------------------------------------------------------------------------
Brick : Brick cornelis:/mnt/bricks/data/16/VK0N9UBY
TCP Port : 56717
RDMA Port : 0
Online : Y
Pid : 147142
File System : xfs
Device : /dev/sdp2
Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
Inode Size : 512
Disk Space Free : 1.2TB
Total Disk Space : 7.3TB
Inode Count : 781402176
Free Inodes : 780984644
-------------- next part --------------
[root at dorus ~]# grep -n "shared-brick-count"
/var/lib/glusterd/vols/data/*
grep: /var/lib/glusterd/vols/data/bricks: Is a directory
/var/lib/glusterd/vols/data/data.cornelis.mnt-bricks-data-01-VRK4L7MK.vol:3:
option shared-brick-count 0
/var/lib/glusterd/vols/data/data.cornelis.mnt-bricks-data-02-VAG4UNTL.vol:3:
option shared-brick-count 0
/var/lib/glusterd/vols/data/data.cornelis.mnt-bricks-data-03-WD-WCC7K2ZF967P.vol:3:
option shared-brick-count 0
/var/lib/glusterd/vols/data/data.cornelis.mnt-bricks-data-04-WD-WCC4E4DEL2R7.vol:3:
option shared-brick-count 0
/var/lib/glusterd/vols/data/data.cornelis.mnt-bricks-data-05-VKKRGDRY.vol:3:
option shared-brick-count 0
/var/lib/glusterd/vols/data/data.cornelis.mnt-bricks-data-06-R6GUJRGY.vol:3:
option shared-brick-count 0
/var/lib/glusterd/vols/data/data.cornelis.mnt-bricks-data-07-WD-WCC4E2FRNVLS.vol:3:
option shared-brick-count 0
/var/lib/glusterd/vols/data/data.cornelis.mnt-bricks-data-08-VK0PTDWY.vol:3:
option shared-brick-count 0
/var/lib/glusterd/vols/data/data.cornelis.mnt-bricks-data-09-VRK4L9PK.vol:3:
option shared-brick-count 0
/var/lib/glusterd/vols/data/data.cornelis.mnt-bricks-data-10-CA0ZYD5K.vol:3:
option shared-brick-count 0
/var/lib/glusterd/vols/data/data.cornelis.mnt-bricks-data-11-R6GSPRKY.vol:3:
option shared-brick-count 0
/var/lib/glusterd/vols/data/data.cornelis.mnt-bricks-data-12-WD-WCC7K7PCNUDT.vol:3:
option shared-brick-count 0
/var/lib/glusterd/vols/data/data.cornelis.mnt-bricks-data-13-VKKMP42Y.vol:3:
option shared-brick-count 0
/var/lib/glusterd/vols/data/data.cornelis.mnt-bricks-data-14-CA0VWVTK.vol:3:
option shared-brick-count 0
/var/lib/glusterd/vols/data/data.cornelis.mnt-bricks-data-15-WD-WCC4E0UP9SXY.vol:3:
option shared-brick-count 0
/var/lib/glusterd/vols/data/data.cornelis.mnt-bricks-data-16-VK0N9UBY.vol:3:
option shared-brick-count 0
/var/lib/glusterd/vols/data/data.dorus.mnt-bricks-data-01-VGK8VEXG.vol:3:
option shared-brick-count 2
/var/lib/glusterd/vols/data/data.dorus.mnt-bricks-data-02-VKKP6LAY.vol:3:
option shared-brick-count 1
/var/lib/glusterd/vols/data/data.dorus.mnt-bricks-data-03-WD-WCC7K7PCN1FH.vol:3:
option shared-brick-count 1
/var/lib/glusterd/vols/data/data.dorus.mnt-bricks-data-04-WD-WCC4E6XXFZK5.vol:3:
option shared-brick-count 1
/var/lib/glusterd/vols/data/data.dorus.mnt-bricks-data-05-VK0RBEUY.vol:3:
option shared-brick-count 1
/var/lib/glusterd/vols/data/data.dorus.mnt-bricks-data-06-R6GLYPZY.vol:3:
option shared-brick-count 1
/var/lib/glusterd/vols/data/data.dorus.mnt-bricks-data-07-WD-WCC7K3KZAAY7.vol:3:
option shared-brick-count 1
/var/lib/glusterd/vols/data/data.dorus.mnt-bricks-data-08-VKKNNE7Y.vol:3:
option shared-brick-count 1
/var/lib/glusterd/vols/data/data.dorus.mnt-bricks-data-09-CA0ZRMEK.vol:3:
option shared-brick-count 1
/var/lib/glusterd/vols/data/data.dorus.mnt-bricks-data-10-WD-WCC7K2ZF95D4.vol:3:
option shared-brick-count 1
/var/lib/glusterd/vols/data/data.dorus.mnt-bricks-data-11-VK0N9YMY.vol:3:
option shared-brick-count 1
/var/lib/glusterd/vols/data/data.dorus.mnt-bricks-data-12-WD-WCC4E4UREPNH.vol:3:
option shared-brick-count 2
/var/lib/glusterd/vols/data/data.dorus.mnt-bricks-data-13-R6GV252Y.vol:3:
option shared-brick-count 1
/var/lib/glusterd/vols/data/data.dorus.mnt-bricks-data-14-VGJYYJXG.vol:3:
option shared-brick-count 1
/var/lib/glusterd/vols/data/data.dorus.mnt-bricks-data-15-WD-WCC7K1HU54PX.vol:3:
option shared-brick-count 1
/var/lib/glusterd/vols/data/data.dorus.mnt-bricks-data-16-R6GV83HY.vol:3:
option shared-brick-count 1
-------------- next part --------------
VOLUME=data
DISKNR=10
NODE=${HOSTNAME}
NODELETTER=${NODE:0:1}
OLDSERIAL=`basename \`/usr/bin/gluster volume status all detail |grep -P
"^(?=.*${NODE})(?=.*${VOLUME})(?=.*${DISKNR})" | awk '{print
$4}'\``
## Unmount old disk
kill -15 `ps aux | grep glusterfsd |grep ${VOLUME} |grep ${DISKNR} |awk
'{print $2}'`
sleep 10
umount /mnt/bricks/${VOLUME}/${DISKNR}
## Check that the gluster volume is online with 1 brick offline
gluster volume status ${VOLUME}
<replace disk>
NEWDISK=sdi
NEWSERIAL=`smartctl -a /dev/${NEWDISK} |grep Serial | awk '{print $3}' |
sed -e 's/WD-//g'`
echo ${NEWSERIAL}
## Create partitions on the new disk
parted -a optimal /dev/${NEWDISK} 'mklabel gpt' 'mkpart primary 1MiB
3MiB' 'name 1 grub' 'set 1 bios_grub on' 'mkpart primary
3MiB -1' "name 2 ${NODELETTER}-${VOLUME}-${DISKNR}"
## Create File System on the new disks
mkfs.xfs -L ${NODELETTER}-${VOLUME}-${DISKNR} /dev/${NEWDISK}2
## Mount new disk
mount -a
## Check that all 16 disks are mounted
mount |grep data |wc -l
## Create the new brick base directory
mkdir -p /mnt/bricks/${VOLUME}/${DISKNR}/${NEWSERIAL}
## Replace the faulty brick
gluster volume replace-brick ${VOLUME}
${NODE}:/mnt/bricks/${VOLUME}/${DISKNR}/${OLDSERIAL}
${NODE}:/mnt/bricks/${VOLUME}/${DISKNR}/${NEWSERIAL} commit force
## Check that the new brick is now online
gluster volume status ${VOLUME}
## Check the pending heals and see then decrease over time
gluster volume heal ${VOLUME} info