thr3ads.net - Gluster users - [Gluster-users] Gluster 9.3 not distributing data evenly across bricks [Dec 2023]

If this information is useful, please help other people find it:
Share via:

Shreyansh Shah

2023-Dec-01 12:37 UTC

[Gluster-users] Gluster 9.3 not distributing data evenly across bricks

Hi,
We are running a gluster 9.3 volume with 47 bricks across 13 nodes. We are
noticing uneven data distribution across the bricks. Some bricks are 100%
utilised whereas some are < 85%. Due to this, data write randomly fails
with error OSError(28, 'No space left on device') even when the volume
has
sufficient space on other bricks. We expect the glusterfs process to
automatically sync and maintain equal data distribution across the bricks.

This is the volume info:

root at nas-5:~# gluster v info

Volume Name: datavol
Type: Distribute
Volume ID: b4b52b4b-0ea0-4eeb-a359-5a0573d4f83a
Status: Started
Snapshot Count: 0
Number of Bricks: 47
Transport-type: tcp
Bricks:
Brick1: 10.132.2.101:/data/data
Brick2: 10.132.2.101:/data1/data
Brick3: 10.132.2.101:/data2/data
Brick4: 10.132.2.101:/data3/data
Brick5: 10.132.2.102:/data/data
Brick6: 10.132.2.102:/data1/data
Brick7: 10.132.2.102:/data2/data
Brick8: 10.132.2.102:/data3/data
Brick9: 10.132.2.103:/data/data
Brick10: 10.132.2.103:/data1/data
Brick11: 10.132.2.103:/data2/data
Brick12: 10.132.2.103:/data3/data
Brick13: 10.132.2.104:/data/data
Brick14: 10.132.2.104:/data3/data
Brick15: 10.132.2.105:/data1/data
Brick16: 10.132.2.105:/data2/data
Brick17: 10.132.2.106:/data/data
Brick18: 10.132.2.106:/data1/data
Brick19: 10.132.2.106:/data2/data
Brick20: 10.132.2.107:/data/data
Brick21: 10.132.2.107:/data1/data
Brick22: 10.132.2.107:/data2/data
Brick23: 10.132.2.108:/data/data
Brick24: 10.132.2.108:/data1/data
Brick25: 10.132.2.108:/data2/data
Brick26: 10.132.2.109:/data/data
Brick27: 10.132.2.109:/data1/data
Brick28: 10.132.2.109:/data2/data
Brick29: 10.132.2.110:/data/data
Brick30: 10.132.2.110:/data1/data
Brick31: 10.132.2.111:/data/data
Brick32: 10.132.2.111:/data1/data
Brick33: 10.132.2.111:/data2/data
Brick34: 10.132.2.112:/data/data
Brick35: 10.132.2.112:/data1/data
Brick36: 10.132.2.112:/data2/data
Brick37: 10.132.2.113:/data/data
Brick38: 10.132.2.113:/data1/data
Brick39: 10.132.2.113:/data2/data
Brick40: 10.132.2.108:/data3/data
Brick41: 10.132.2.107:/data3/data
Brick42: 10.132.2.106:/data3/data
Brick43: 10.132.2.105:/data3/data
Brick44: 10.132.2.110:/data2/data
Brick45: 10.132.2.105:/data/data
Brick46: 10.132.2.104:/data1/data
Brick47: 10.132.2.104:/data2/data
Options Reconfigured:
performance.client-io-threads: on
cluster.min-free-disk: 2%
performance.cache-refresh-timeout: 60
client.event-threads: 4
server.event-threads: 4
network.ping-timeout: 90
storage.health-check-interval: 60
storage.health-check-timeout: 60
performance.io-cache-size: 8GB
storage.fips-mode-rchecksum: on
transport.address-family: inet
nfs.disable: on
root at nas-5:~# glusterfs --version
glusterfs 9.3
Repository revision: git://git.gluster.org/glusterfs.git
Copyright (c) 2006-2016 Red Hat, Inc. <https://www.gluster.org/>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
It is licensed to you under your choice of the GNU Lesser
General Public License, version 3 or any later version (LGPLv3
or later), or the GNU General Public License, version 2 (GPLv2),
in all cases as published by the Free Software Foundation.
root at nas-5:~#



This is the disk usage distribution when the write failed:
*Node 1:*
/dev/bcache1    9.7T  8.5T  1.2T  88% /data
/dev/bcache3    3.9T  3.3T  563G  86% /data1
/dev/bcache0    3.9T  3.3T  560G  86% /data2
/dev/bcache2    3.9T  3.5T  395G  90% /data3
*Node 2:*
/dev/bcache0    9.7T  8.5T  1.2T  88% /data
/dev/bcache1    3.9T  3.5T  421G  90% /data1
/dev/bcache2    3.9T  3.6T  330G  92% /data2
/dev/bcache3    3.9T  3.4T  513G  87% /data3
*Node 3:*
/dev/bcache0    9.7T  8.8T  882G  92% /data
/dev/bcache1    3.9T  3.6T  335G  92% /data1
/dev/bcache2    3.9T  3.4T  532G  87% /data2
/dev/bcache3    3.9T  3.3T  564G  86% /data3
*Node 4:*
/dev/bcache0    9.7T  8.7T  982G  91% /data
/dev/bcache1    3.9T  3.5T  424G  90% /data1
/dev/bcache2    3.9T  3.4T  549G  87% /data2
/dev/bcache3    3.9T  3.6T  344G  92% /data3
*Node 5:*
/dev/bcache0    9.7T  8.5T  1.3T  88% /data
/dev/bcache1    3.9T  3.6T  288G  93% /data1
/dev/bcache2    3.9T  3.4T  470G  89% /data2
/dev/bcache3    9.9T  9.8T  101G 100% /data3
*Node 6:*
/dev/bcache0    9.7T  8.2T  1.5T  86% /data
/dev/bcache1    3.9T  3.4T  526G  87% /data1
/dev/bcache2    3.9T  3.5T  431G  90% /data2
/dev/bcache3    9.9T  8.9T  1.1T  90% /data3
*Node 7:*
/dev/bcache0    9.7T  8.9T  783G  93% /data
/dev/bcache1    3.9T  3.3T  561G  86% /data1
/dev/bcache2    3.9T  3.5T  360G  91% /data2
/dev/bcache3    9.9T  8.7T  1.2T  89% /data3
*Node 8:*
/dev/bcache0    9.7T  8.7T  994G  90% /data
/dev/bcache1    3.9T  3.3T  645G  84% /data1
/dev/bcache2    3.9T  3.4T  519G  87% /data2
/dev/bcache3    9.9T  9.0T  868G  92% /data3
*Node 9:*
/dev/bcache0     10T  8.6T  1.4T  87% /data
/dev/bcache1    8.0T  6.7T  1.4T  84% /data1
/dev/bcache2    8.0T  6.8T  1.3T  85% /data2
*Node 10:*
/dev/bcache0     10T  8.8T  1.3T  88% /data
/dev/bcache1    8.0T  6.6T  1.4T  83% /data1
/dev/bcache2    8.0T  7.0T  990G  88% /data2
*Node 11:*
/dev/bcache0     10T  8.1T  1.9T  82% /data
/dev/bcache1     10T  8.5T  1.5T  86% /data1
/dev/bcache2     10T  8.4T  1.6T  85% /data2
*Node 12:*
/dev/bcache0     10T  8.4T  1.6T  85% /data
/dev/bcache1     10T  8.4T  1.6T  85% /data1
/dev/bcache2     10T  8.2T  1.8T  83% /data2
*Node 13:*
/dev/bcache1     10T  8.7T  1.3T  88% /data1
/dev/bcache2     10T  8.8T  1.2T  88% /data2
/dev/bcache0     10T  8.6T  1.5T  86% /data




-- 
Regards,
Shreyansh Shah
AlphaGrep* Securities Pvt. Ltd.*
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20231201/3f9e1850/attachment.html>

Diego Zuccato

2023-Dec-01 13:04 UTC

head link

[Gluster-users] Gluster 9.3 not distributing data evenly across bricks

IIUC you have to manually run a rebalance
gluster v rebalance datavol start

Diego

Il 01/12/2023 13:37, Shreyansh Shah ha scritto:> Hi,
> We are running a gluster 9.3 volume with 47 bricks across 13 nodes. We 
> are noticing uneven data distribution across?the bricks. Some bricks are 
> 100% utilised whereas some are < 85%. Due to this, data write randomly 
> fails with error OSError(28, 'No space left on device') even when
the
> volume has sufficient space on other bricks. We expect the glusterfs 
> process to automatically?sync and maintain equal data distribution 
> across the bricks.
> 
> This is the volume info:
> 
>     root at nas-5:~# gluster v info
> 
>     Volume Name: datavol
>     Type: Distribute
>     Volume ID: b4b52b4b-0ea0-4eeb-a359-5a0573d4f83a
>     Status: Started
>     Snapshot Count: 0
>     Number of Bricks: 47
>     Transport-type: tcp
>     Bricks:
>     Brick1: 10.132.2.101:/data/data
>     Brick2: 10.132.2.101:/data1/data
>     Brick3: 10.132.2.101:/data2/data
>     Brick4: 10.132.2.101:/data3/data
>     Brick5: 10.132.2.102:/data/data
>     Brick6: 10.132.2.102:/data1/data
>     Brick7: 10.132.2.102:/data2/data
>     Brick8: 10.132.2.102:/data3/data
>     Brick9: 10.132.2.103:/data/data
>     Brick10: 10.132.2.103:/data1/data
>     Brick11: 10.132.2.103:/data2/data
>     Brick12: 10.132.2.103:/data3/data
>     Brick13: 10.132.2.104:/data/data
>     Brick14: 10.132.2.104:/data3/data
>     Brick15: 10.132.2.105:/data1/data
>     Brick16: 10.132.2.105:/data2/data
>     Brick17: 10.132.2.106:/data/data
>     Brick18: 10.132.2.106:/data1/data
>     Brick19: 10.132.2.106:/data2/data
>     Brick20: 10.132.2.107:/data/data
>     Brick21: 10.132.2.107:/data1/data
>     Brick22: 10.132.2.107:/data2/data
>     Brick23: 10.132.2.108:/data/data
>     Brick24: 10.132.2.108:/data1/data
>     Brick25: 10.132.2.108:/data2/data
>     Brick26: 10.132.2.109:/data/data
>     Brick27: 10.132.2.109:/data1/data
>     Brick28: 10.132.2.109:/data2/data
>     Brick29: 10.132.2.110:/data/data
>     Brick30: 10.132.2.110:/data1/data
>     Brick31: 10.132.2.111:/data/data
>     Brick32: 10.132.2.111:/data1/data
>     Brick33: 10.132.2.111:/data2/data
>     Brick34: 10.132.2.112:/data/data
>     Brick35: 10.132.2.112:/data1/data
>     Brick36: 10.132.2.112:/data2/data
>     Brick37: 10.132.2.113:/data/data
>     Brick38: 10.132.2.113:/data1/data
>     Brick39: 10.132.2.113:/data2/data
>     Brick40: 10.132.2.108:/data3/data
>     Brick41: 10.132.2.107:/data3/data
>     Brick42: 10.132.2.106:/data3/data
>     Brick43: 10.132.2.105:/data3/data
>     Brick44: 10.132.2.110:/data2/data
>     Brick45: 10.132.2.105:/data/data
>     Brick46: 10.132.2.104:/data1/data
>     Brick47: 10.132.2.104:/data2/data
>     Options Reconfigured:
>     performance.client-io-threads: on
>     cluster.min-free-disk: 2%
>     performance.cache-refresh-timeout: 60
>     client.event-threads: 4
>     server.event-threads: 4
>     network.ping-timeout: 90
>     storage.health-check-interval: 60
>     storage.health-check-timeout: 60
>     performance.io-cache-size: 8GB
>     storage.fips-mode-rchecksum: on
>     transport.address-family: inet
>     nfs.disable: on
>     root at nas-5:~# glusterfs --version
>     glusterfs 9.3
>     Repository revision: git://git.gluster.org/glusterfs.git
>     <http://git.gluster.org/glusterfs.git>
>     Copyright (c) 2006-2016 Red Hat, Inc. <https://www.gluster.org/
>     <https://www.gluster.org/>>
>     GlusterFS comes with ABSOLUTELY NO WARRANTY.
>     It is licensed to you under your choice of the GNU Lesser
>     General Public License, version 3 or any later version (LGPLv3
>     or later), or the GNU General Public License, version 2 (GPLv2),
>     in all cases as published by the Free Software Foundation.
>     root at nas-5:~# 
> 
> 
> 
> This is the disk usage distribution when the write failed:
> *Node 1:*
> /dev/bcache1 ? ?9.7T ?8.5T ?1.2T ?88% /data
> /dev/bcache3 ? ?3.9T ?3.3T ?563G ?86% /data1
> /dev/bcache0 ? ?3.9T ?3.3T ?560G ?86% /data2
> /dev/bcache2 ? ?3.9T ?3.5T ?395G ?90% /data3
> *Node 2:*
> /dev/bcache0 ? ?9.7T ?8.5T ?1.2T ?88% /data
> /dev/bcache1 ? ?3.9T ?3.5T ?421G ?90% /data1
> /dev/bcache2 ? ?3.9T ?3.6T ?330G ?92% /data2
> /dev/bcache3 ? ?3.9T ?3.4T ?513G ?87% /data3
> *Node 3:*
> /dev/bcache0 ? ?9.7T ?8.8T ?882G ?92% /data
> /dev/bcache1 ? ?3.9T ?3.6T ?335G ?92% /data1
> /dev/bcache2 ? ?3.9T ?3.4T ?532G ?87% /data2
> /dev/bcache3 ? ?3.9T ?3.3T ?564G ?86% /data3
> *Node 4:*
> /dev/bcache0 ? ?9.7T ?8.7T ?982G ?91% /data
> /dev/bcache1 ? ?3.9T ?3.5T ?424G ?90% /data1
> /dev/bcache2 ? ?3.9T ?3.4T ?549G ?87% /data2
> /dev/bcache3 ? ?3.9T ?3.6T ?344G ?92% /data3
> *Node 5:*
> /dev/bcache0 ? ?9.7T ?8.5T ?1.3T ?88% /data
> /dev/bcache1 ? ?3.9T ?3.6T ?288G ?93% /data1
> /dev/bcache2 ? ?3.9T ?3.4T ?470G ?89% /data2
> /dev/bcache3 ? ?9.9T ?9.8T ?101G 100% /data3
> *Node 6:*
> /dev/bcache0 ? ?9.7T ?8.2T ?1.5T ?86% /data
> /dev/bcache1 ? ?3.9T ?3.4T ?526G ?87% /data1
> /dev/bcache2 ? ?3.9T ?3.5T ?431G ?90% /data2
> /dev/bcache3 ? ?9.9T ?8.9T ?1.1T ?90% /data3
> *Node 7:*
> /dev/bcache0 ? ?9.7T ?8.9T ?783G ?93% /data
> /dev/bcache1 ? ?3.9T ?3.3T ?561G ?86% /data1
> /dev/bcache2 ? ?3.9T ?3.5T ?360G ?91% /data2
> /dev/bcache3 ? ?9.9T ?8.7T ?1.2T ?89% /data3
> *Node 8:*
> /dev/bcache0 ? ?9.7T ?8.7T ?994G ?90% /data
> /dev/bcache1 ? ?3.9T ?3.3T ?645G ?84% /data1
> /dev/bcache2 ? ?3.9T ?3.4T ?519G ?87% /data2
> /dev/bcache3 ? ?9.9T ?9.0T ?868G ?92% /data3
> *Node 9:*
> /dev/bcache0 ? ? 10T ?8.6T ?1.4T ?87% /data
> /dev/bcache1 ? ?8.0T ?6.7T ?1.4T ?84% /data1
> /dev/bcache2 ? ?8.0T ?6.8T ?1.3T ?85% /data2
> *Node 10:*
> /dev/bcache0 ? ? 10T ?8.8T ?1.3T ?88% /data
> /dev/bcache1 ? ?8.0T ?6.6T ?1.4T ?83% /data1
> /dev/bcache2 ? ?8.0T ?7.0T ?990G ?88% /data2
> *Node 11:*
> /dev/bcache0 ? ? 10T ?8.1T ?1.9T ?82% /data
> /dev/bcache1 ? ? 10T ?8.5T ?1.5T ?86% /data1
> /dev/bcache2 ? ? 10T ?8.4T ?1.6T ?85% /data2
> *Node 12:*
> /dev/bcache0 ? ? 10T ?8.4T ?1.6T ?85% /data
> /dev/bcache1 ? ? 10T ?8.4T ?1.6T ?85% /data1
> /dev/bcache2 ? ? 10T ?8.2T ?1.8T ?83% /data2
> *Node 13:*
> /dev/bcache1 ? ? 10T ?8.7T ?1.3T ?88% /data1
> /dev/bcache2 ? ? 10T ?8.8T ?1.2T ?88% /data2
> /dev/bcache0 ? ? 10T ?8.6T ?1.5T ?86% /data
> 
> 
> 
> 
> -- 
> Regards,
> Shreyansh Shah
> AlphaGrep*?Securities Pvt. Ltd.*
> 
> ________
> 
> 
> 
> Community Meeting Calendar:
> 
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://meet.google.com/cpu-eiue-hvk
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
-- 
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Universit? di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786

Gluster users - Dec 2023 - Gluster 9.3 not distributing data evenly across bricks

[Gluster-users] Gluster 9.3 not distributing data evenly across bricks

[Gluster-users] Gluster 9.3 not distributing data evenly across bricks