thr3ads.net - Gluster users - [Gluster-users] How to configure? [Apr 2023]

If this information is useful, please help other people find it:
Share via:

Strahil Nikolov

2023-Mar-24 18:21 UTC

[Gluster-users] How to configure?

Try finding if any of them is missing on one of the systems.
Best Regards,Strahil Nikolov?
 
 
  On Fri, Mar 24, 2023 at 15:59, Diego Zuccato<diego.zuccato at unibo.it>
wrote:   There are 285 files in /var/lib/glusterd/vols/cluster_data ...
including
many files with names related to quorum bricks already moved to a 
different path (like cluster_data.client.clustor02.srv-quorum-00-d.vol 
that should already have been replaced by 
cluster_data.clustor02.srv-bricks-00-q.vol -- and both vol files exist).

Is there something I should check inside the volfiles?

Diego

Il 24/03/2023 13:05, Strahil Nikolov ha scritto:> Can you check your volume file contents?
> Maybe it really can't find (or access) a specific volfile ?
> 
> Best Regards,
> Strahil Nikolov
> 
>? ? On Fri, Mar 24, 2023 at 8:07, Diego Zuccato
>? ? <diego.zuccato at unibo.it> wrote:
>? ? In glfsheal-Connection.log I see many lines like:
>? ? [2023-03-13 23:04:40.241481 +0000] E [MSGID: 104021]
>? ? [glfs-mgmt.c:586:glfs_mgmt_getspec_cbk] 0-gfapi: failed to get the
>? ? volume file [{from server}, {errno=2}, {error=File o directory non
>? ? esistente}]
> 
>? ? And *lots* of gfid-mismatch errors in glustershd.log .
> 
>? ? Couldn't find anything that would prevent heal to start. :(
> 
>? ? Diego
> 
>? ? Il 21/03/2023 20:39, Strahil Nikolov ha scritto:
>? ? ? > I have no clue. Have you checked for errors in the logs ? Maybe
you
>? ? ? > might find something useful.
>? ? ? >
>? ? ? > Best Regards,
>? ? ? > Strahil Nikolov
>? ? ? >
>? ? ? >? ? On Tue, Mar 21, 2023 at 9:56, Diego Zuccato
>? ? ? >? ? <diego.zuccato at unibo.it <mailto:diego.zuccato at
unibo.it>> wrote:
>? ? ? >? ? Killed glfsheal, after a day there were 218 processes, then
>? ? they got
>? ? ? >? ? killed by OOM during the weekend. Now there are no processes
>? ? active.
>? ? ? >? ? Trying to run "heal info" reports lots of files
quite quickly
>? ? but does
>? ? ? >? ? not spawn any glfsheal process. And neither does restarting
>? ? glusterd.
>? ? ? >? ? Is there some way to selectively run glfsheal to fix one brick
>? ? at a
>? ? ? >? ? time?
>? ? ? >
>? ? ? >? ? Diego
>? ? ? >
>? ? ? >? ? Il 21/03/2023 01:21, Strahil Nikolov ha scritto:
>? ? ? >? ? ? > Theoretically it might help.
>? ? ? >? ? ? > If possible, try to resolve any pending heals.
>? ? ? >? ? ? >
>? ? ? >? ? ? > Best Regards,
>? ? ? >? ? ? > Strahil Nikolov
>? ? ? >? ? ? >
>? ? ? >? ? ? >? ? On Thu, Mar 16, 2023 at 15:29, Diego Zuccato
>? ? ? >? ? ? >? ? <diego.zuccato at unibo.it
<mailto:diego.zuccato at unibo.it>
>? ? <mailto:diego.zuccato at unibo.it>> wrote:
>? ? ? >? ? ? >? ? In Debian stopping glusterd does not stop brick
>? ? processes: to stop
>? ? ? >? ? ? >? ? everything (and free the memory) I have to
>? ? ? >? ? ? >? ? systemctl stop glusterd
>? ? ? >? ? ? >? ? ? ? killall glusterfs{,d}
>? ? ? >? ? ? >? ? ? ? killall glfsheal
>? ? ? >? ? ? >? ? ? ? systemctl start glusterd
>? ? ? >? ? ? >? ? [this behaviour hangs a simple reboot of a machine
running
>? ? ? >? ? glusterd...
>? ? ? >? ? ? >? ? not nice]
>? ? ? >? ? ? >
>? ? ? >? ? ? >? ? For now I just restarted glusterd w/o killing the
bricks:
>? ? ? >? ? ? >
>? ? ? >? ? ? >? ? root at str957-clustor00:~# ps aux|grep glfsheal|wc
-l ;
>? ? ? >? ? systemctl restart
>? ? ? >? ? ? >? ? glusterd ; ps aux|grep glfsheal|wc -l
>? ? ? >? ? ? >? ? 618
>? ? ? >? ? ? >? ? 618
>? ? ? >? ? ? >
>? ? ? >? ? ? >? ? No change neither in glfsheal processes nor in free
>? ? memory :(
>? ? ? >? ? ? >? ? Should I "killall glfsheal" before OOK
kicks in?
>? ? ? >? ? ? >
>? ? ? >? ? ? >? ? Diego
>? ? ? >? ? ? >
>? ? ? >? ? ? >? ? Il 16/03/2023 12:37, Strahil Nikolov ha scritto:
>? ? ? >? ? ? >? ? ? > Can you restart glusterd service (first check
that
>? ? it was not
>? ? ? >? ? ? >? ? modified
>? ? ? >? ? ? >? ? ? > to kill the bricks)?
>? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? ? ? > Best Regards,
>? ? ? >? ? ? >? ? ? > Strahil Nikolov
>? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? ? ? >? ? On Thu, Mar 16, 2023 at 8:26, Diego
Zuccato
>? ? ? >? ? ? >? ? ? >? ? <diego.zuccato at unibo.it
>? ? <mailto:diego.zuccato at unibo.it> <mailto:diego.zuccato at
unibo.it>
>? ? ? >? ? <mailto:diego.zuccato at unibo.it>> wrote:
>? ? ? >? ? ? >? ? ? >? ? OOM is just just a matter of time.
>? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? ? ? >? ? Today mem use is up to 177G/187 and:
>? ? ? >? ? ? >? ? ? >? ? # ps aux|grep glfsheal|wc -l
>? ? ? >? ? ? >? ? ? >? ? 551
>? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? ? ? >? ? (well, one is actually the grep process,
so
>? ? "only" 550
>? ? ? >? ? glfsheal
>? ? ? >? ? ? >? ? ? >? ? processes.
>? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? ? ? >? ? I'll take the last 5:
>? ? ? >? ? ? >? ? ? >? ? root? ? 3266352? 0.5? 0.0 600292 93044 ??
? ? ? Sl
>? ? ? >? ? 06:55? 0:07
>? ? ? >? ? ? >? ? ? >? ? /usr/libexec/glusterfs/glfsheal
cluster_data
>? ? ? >? ? info-summary --xml
>? ? ? >? ? ? >? ? ? >? ? root? ? 3267220? 0.7? 0.0 600292 91964 ??
? ? ? Sl
>? ? ? >? ? 07:00? 0:07
>? ? ? >? ? ? >? ? ? >? ? /usr/libexec/glusterfs/glfsheal
cluster_data
>? ? ? >? ? info-summary --xml
>? ? ? >? ? ? >? ? ? >? ? root? ? 3268076? 1.0? 0.0 600160 88216 ??
? ? ? Sl
>? ? ? >? ? 07:05? 0:08
>? ? ? >? ? ? >? ? ? >? ? /usr/libexec/glusterfs/glfsheal
cluster_data
>? ? ? >? ? info-summary --xml
>? ? ? >? ? ? >? ? ? >? ? root? ? 3269492? 1.6? 0.0 600292 91248 ??
? ? ? Sl
>? ? ? >? ? 07:10? 0:07
>? ? ? >? ? ? >? ? ? >? ? /usr/libexec/glusterfs/glfsheal
cluster_data
>? ? ? >? ? info-summary --xml
>? ? ? >? ? ? >? ? ? >? ? root? ? 3270354? 4.4? 0.0 600292 93260 ??
? ? ? Sl
>? ? ? >? ? 07:15? 0:07
>? ? ? >? ? ? >? ? ? >? ? /usr/libexec/glusterfs/glfsheal
cluster_data
>? ? ? >? ? info-summary --xml
>? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? ? ? >? ? -8<--
>? ? ? >? ? ? >? ? ? >? ? root at str957-clustor00:~# ps -o ppid=
3266352
>? ? ? >? ? ? >? ? ? >? ? 3266345
>? ? ? >? ? ? >? ? ? >? ? root at str957-clustor00:~# ps -o ppid=
3267220
>? ? ? >? ? ? >? ? ? >? ? 3267213
>? ? ? >? ? ? >? ? ? >? ? root at str957-clustor00:~# ps -o ppid=
3268076
>? ? ? >? ? ? >? ? ? >? ? 3268069
>? ? ? >? ? ? >? ? ? >? ? root at str957-clustor00:~# ps -o ppid=
3269492
>? ? ? >? ? ? >? ? ? >? ? 3269485
>? ? ? >? ? ? >? ? ? >? ? root at str957-clustor00:~# ps -o ppid=
3270354
>? ? ? >? ? ? >? ? ? >? ? 3270347
>? ? ? >? ? ? >? ? ? >? ? root at str957-clustor00:~# ps aux|grep
3266345
>? ? ? >? ? ? >? ? ? >? ? root? ? 3266345? 0.0? 0.0 430536 10764 ??
? ? ? Sl
>? ? ? >? ? 06:55? 0:00
>? ? ? >? ? ? >? ? ? >? ? gluster volume heal cluster_data info
summary --xml
>? ? ? >? ? ? >? ? ? >? ? root? ? 3271532? 0.0? 0.0? 6260? 2500
pts/1? ? S+
>? ? ? >? ? 07:21? 0:00
>? ? ? >? ? ? >? ? grep
>? ? ? >? ? ? >? ? ? >? ? 3266345
>? ? ? >? ? ? >? ? ? >? ? root at str957-clustor00:~# ps aux|grep
3267213
>? ? ? >? ? ? >? ? ? >? ? root? ? 3267213? 0.0? 0.0 430536 10644 ??
? ? ? Sl
>? ? ? >? ? 07:00? 0:00
>? ? ? >? ? ? >? ? ? >? ? gluster volume heal cluster_data info
summary --xml
>? ? ? >? ? ? >? ? ? >? ? root? ? 3271599? 0.0? 0.0? 6260? 2480
pts/1? ? S+
>? ? ? >? ? 07:22? 0:00
>? ? ? >? ? ? >? ? grep
>? ? ? >? ? ? >? ? ? >? ? 3267213
>? ? ? >? ? ? >? ? ? >? ? root at str957-clustor00:~# ps aux|grep
3268069
>? ? ? >? ? ? >? ? ? >? ? root? ? 3268069? 0.0? 0.0 430536 10704 ??
? ? ? Sl
>? ? ? >? ? 07:05? 0:00
>? ? ? >? ? ? >? ? ? >? ? gluster volume heal cluster_data info
summary --xml
>? ? ? >? ? ? >? ? ? >? ? root? ? 3271626? 0.0? 0.0? 6260? 2516
pts/1? ? S+
>? ? ? >? ? 07:22? 0:00
>? ? ? >? ? ? >? ? grep
>? ? ? >? ? ? >? ? ? >? ? 3268069
>? ? ? >? ? ? >? ? ? >? ? root at str957-clustor00:~# ps aux|grep
3269485
>? ? ? >? ? ? >? ? ? >? ? root? ? 3269485? 0.0? 0.0 430536 10756 ??
? ? ? Sl
>? ? ? >? ? 07:10? 0:00
>? ? ? >? ? ? >? ? ? >? ? gluster volume heal cluster_data info
summary --xml
>? ? ? >? ? ? >? ? ? >? ? root? ? 3271647? 0.0? 0.0? 6260? 2480
pts/1? ? S+
>? ? ? >? ? 07:22? 0:00
>? ? ? >? ? ? >? ? grep
>? ? ? >? ? ? >? ? ? >? ? 3269485
>? ? ? >? ? ? >? ? ? >? ? root at str957-clustor00:~# ps aux|grep
3270347
>? ? ? >? ? ? >? ? ? >? ? root? ? 3270347? 0.0? 0.0 430536 10672 ??
? ? ? Sl
>? ? ? >? ? 07:15? 0:00
>? ? ? >? ? ? >? ? ? >? ? gluster volume heal cluster_data info
summary --xml
>? ? ? >? ? ? >? ? ? >? ? root? ? 3271666? 0.0? 0.0? 6260? 2568
pts/1? ? S+
>? ? ? >? ? 07:22? 0:00
>? ? ? >? ? ? >? ? grep
>? ? ? >? ? ? >? ? ? >? ? 3270347
>? ? ? >? ? ? >? ? ? >? ? -8<--
>? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? ? ? >? ? Seems glfsheal is spawning more processes.
>? ? ? >? ? ? >? ? ? >? ? I can't rule out a metadata corruption
(or at
>? ? least a
>? ? ? >? ? desync),
>? ? ? >? ? ? >? ? but it
>? ? ? >? ? ? >? ? ? >? ? shouldn't happen...
>? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? ? ? >? ? Diego
>? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? ? ? >? ? Il 15/03/2023 20:11, Strahil Nikolov ha
scritto:
>? ? ? >? ? ? >? ? ? >? ? ? > If you don't experience any OOM
, you can
>? ? focus on
>? ? ? >? ? the heals.
>? ? ? >? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? ? ? >? ? ? > 284 processes of glfsheal seems
odd.
>? ? ? >? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? ? ? >? ? ? > Can you check the ppid for 2-3
randomly picked ?
>? ? ? >? ? ? >? ? ? >? ? ? > ps -o ppid= <pid>
>? ? ? >? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? ? ? >? ? ? > Best Regards,
>? ? ? >? ? ? >? ? ? >? ? ? > Strahil Nikolov
>? ? ? >? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? ? ? >? ? ? >? ? On Wed, Mar 15, 2023 at 9:54,
Diego Zuccato
>? ? ? >? ? ? >? ? ? >? ? ? >? ? <diego.zuccato at unibo.it
>? ? <mailto:diego.zuccato at unibo.it>
>? ? ? >? ? <mailto:diego.zuccato at unibo.it>
<mailto:diego.zuccato at unibo.it>
>? ? ? >? ? ? >? ? <mailto:diego.zuccato at unibo.it>> wrote:
>? ? ? >? ? ? >? ? ? >? ? ? >? ? I enabled it yesterday and that
greatly
>? ? reduced
>? ? ? >? ? memory
>? ? ? >? ? ? >? ? pressure.
>? ? ? >? ? ? >? ? ? >? ? ? >? ? Current volume info:
>? ? ? >? ? ? >? ? ? >? ? ? >? ? -8<--
>? ? ? >? ? ? >? ? ? >? ? ? >? ? Volume Name: cluster_data
>? ? ? >? ? ? >? ? ? >? ? ? >? ? Type: Distributed-Replicate
>? ? ? >? ? ? >? ? ? >? ? ? >? ? Volume ID:
>? ? a8caaa90-d161-45bb-a68c-278263a8531a
>? ? ? >? ? ? >? ? ? >? ? ? >? ? Status: Started
>? ? ? >? ? ? >? ? ? >? ? ? >? ? Snapshot Count: 0
>? ? ? >? ? ? >? ? ? >? ? ? >? ? Number of Bricks: 45 x (2 + 1) =
135
>? ? ? >? ? ? >? ? ? >? ? ? >? ? Transport-type: tcp
>? ? ? >? ? ? >? ? ? >? ? ? >? ? Bricks:
>? ? ? >? ? ? >? ? ? >? ? ? >? ? Brick1:
clustor00:/srv/bricks/00/d
>? ? ? >? ? ? >? ? ? >? ? ? >? ? Brick2:
clustor01:/srv/bricks/00/d
>? ? ? >? ? ? >? ? ? >? ? ? >? ? Brick3:
clustor02:/srv/bricks/00/q (arbiter)
>? ? ? >? ? ? >? ? ? >? ? ? >? ? [...]
>? ? ? >? ? ? >? ? ? >? ? ? >? ? Brick133:
clustor01:/srv/bricks/29/d
>? ? ? >? ? ? >? ? ? >? ? ? >? ? Brick134:
clustor02:/srv/bricks/29/d
>? ? ? >? ? ? >? ? ? >? ? ? >? ? Brick135:
clustor00:/srv/bricks/14/q
>? ? (arbiter)
>? ? ? >? ? ? >? ? ? >? ? ? >? ? Options Reconfigured:
>? ? ? >? ? ? >? ? ? >? ? ? >? ? performance.quick-read: off
>? ? ? >? ? ? >? ? ? >? ? ? >? ? cluster.entry-self-heal: on
>? ? ? >? ? ? >? ? ? >? ? ? >? ?
cluster.data-self-heal-algorithm: full
>? ? ? >? ? ? >? ? ? >? ? ? >? ? cluster.metadata-self-heal: on
>? ? ? >? ? ? >? ? ? >? ? ? >? ? cluster.shd-max-threads: 2
>? ? ? >? ? ? >? ? ? >? ? ? >? ? network.inode-lru-limit: 500000
>? ? ? >? ? ? >? ? ? >? ? ? >? ? performance.md-cache-timeout:
600
>? ? ? >? ? ? >? ? ? >? ? ? >? ? performance.cache-invalidation:
on
>? ? ? >? ? ? >? ? ? >? ? ? >? ?
features.cache-invalidation-timeout: 600
>? ? ? >? ? ? >? ? ? >? ? ? >? ? features.cache-invalidation: on
>? ? ? >? ? ? >? ? ? >? ? ? >? ? features.quota-deem-statfs: on
>? ? ? >? ? ? >? ? ? >? ? ? >? ? performance.readdir-ahead: on
>? ? ? >? ? ? >? ? ? >? ? ? >? ? cluster.granular-entry-heal:
enable
>? ? ? >? ? ? >? ? ? >? ? ? >? ? features.scrub: Active
>? ? ? >? ? ? >? ? ? >? ? ? >? ? features.bitrot: on
>? ? ? >? ? ? >? ? ? >? ? ? >? ? cluster.lookup-optimize: on
>? ? ? >? ? ? >? ? ? >? ? ? >? ? performance.stat-prefetch: on
>? ? ? >? ? ? >? ? ? >? ? ? >? ?
performance.cache-refresh-timeout: 60
>? ? ? >? ? ? >? ? ? >? ? ? >? ? performance.parallel-readdir: on
>? ? ? >? ? ? >? ? ? >? ? ? >? ?
performance.write-behind-window-size: 128MB
>? ? ? >? ? ? >? ? ? >? ? ? >? ? cluster.self-heal-daemon: enable
>? ? ? >? ? ? >? ? ? >? ? ? >? ? features.inode-quota: on
>? ? ? >? ? ? >? ? ? >? ? ? >? ? features.quota: on
>? ? ? >? ? ? >? ? ? >? ? ? >? ? transport.address-family: inet
>? ? ? >? ? ? >? ? ? >? ? ? >? ? nfs.disable: on
>? ? ? >? ? ? >? ? ? >? ? ? >? ? performance.client-io-threads:
off
>? ? ? >? ? ? >? ? ? >? ? ? >? ? client.event-threads: 1
>? ? ? >? ? ? >? ? ? >? ? ? >? ? features.scrub-throttle: normal
>? ? ? >? ? ? >? ? ? >? ? ? >? ? diagnostics.brick-log-level:
ERROR
>? ? ? >? ? ? >? ? ? >? ? ? >? ? diagnostics.client-log-level:
ERROR
>? ? ? >? ? ? >? ? ? >? ? ? >? ? config.brick-threads: 0
>? ? ? >? ? ? >? ? ? >? ? ? >? ? cluster.lookup-unhashed: on
>? ? ? >? ? ? >? ? ? >? ? ? >? ? config.client-threads: 1
>? ? ? >? ? ? >? ? ? >? ? ? >? ? cluster.use-anonymous-inode: off
>? ? ? >? ? ? >? ? ? >? ? ? >? ? diagnostics.brick-sys-log-level:
CRITICAL
>? ? ? >? ? ? >? ? ? >? ? ? >? ? features.scrub-freq: monthly
>? ? ? >? ? ? >? ? ? >? ? ? >? ? cluster.data-self-heal: on
>? ? ? >? ? ? >? ? ? >? ? ? >? ? cluster.brick-multiplex: on
>? ? ? >? ? ? >? ? ? >? ? ? >? ? cluster.daemon-log-level: ERROR
>? ? ? >? ? ? >? ? ? >? ? ? >? ? -8<--
>? ? ? >? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? ? ? >? ? ? >? ? htop reports that memory usage
is up to 143G,
>? ? ? >? ? there are 602
>? ? ? >? ? ? >? ? ? >? ? tasks and
>? ? ? >? ? ? >? ? ? >? ? ? >? ? 5232 threads (~20 running) on
clustor00,
>? ? 117G/49
>? ? ? >? ? tasks/1565
>? ? ? >? ? ? >? ? ? >? ? threads on
>? ? ? >? ? ? >? ? ? >? ? ? >? ? clustor01 and 126G/45 tasks/1574
threads on
>? ? ? >? ? clustor02.
>? ? ? >? ? ? >? ? ? >? ? ? >? ? I see quite a lot (284!) of
glfsheal
>? ? processes
>? ? ? >? ? running on
>? ? ? >? ? ? >? ? ? >? ? clustor00 (a
>? ? ? >? ? ? >? ? ? >? ? ? >? ? "gluster v heal
cluster_data info summary" is
>? ? ? >? ? running
>? ? ? >? ? ? >? ? on clustor02
>? ? ? >? ? ? >? ? ? >? ? ? >? ? since
>? ? ? >? ? ? >? ? ? >? ? ? >? ? yesterday, still no output).
Shouldn't be
>? ? just
>? ? ? >? ? one per
>? ? ? >? ? ? >? ? brick?
>? ? ? >? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? ? ? >? ? ? >? ? Diego
>? ? ? >? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? ? ? >? ? ? >? ? Il 15/03/2023 08:30, Strahil
Nikolov ha
>? ? scritto:
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? > Do you use brick
multiplexing ?
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? > Best Regards,
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? > Strahil Nikolov
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? On Tue, Mar 14, 2023
at 16:44,
>? ? Diego Zuccato
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? <diego.zuccato at
unibo.it
>? ? <mailto:diego.zuccato at unibo.it>
>? ? ? >? ? <mailto:diego.zuccato at unibo.it>
>? ? ? >? ? ? >? ? <mailto:diego.zuccato at unibo.it>
>? ? <mailto:diego.zuccato at unibo.it>
>? ? ? >? ? ? >? ? ? >? ? <mailto:diego.zuccato at
unibo.it>> wrote:
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? Hello all.
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? Our Gluster 9.6
cluster is showing
>? ? increasing
>? ? ? >? ? ? >? ? problems.
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? Currently it's
composed of 3
>? ? servers (2x
>? ? ? >? ? Intel Xeon
>? ? ? >? ? ? >? ? ? >? ? 4210 [20
>? ? ? >? ? ? >? ? ? >? ? ? >? ? cores dual
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? thread, total 40
threads], 192GB
>? ? RAM, 30x
>? ? ? >? ? HGST
>? ? ? >? ? ? >? ? ? >? ? HUH721212AL5200
>? ? ? >? ? ? >? ? ? >? ? ? >? ? [12TB]),
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? configured in replica
3 arbiter 1.
>? ? Using
>? ? ? >? ? Debian
>? ? ? >? ? ? >? ? ? >? ? packages from
>? ? ? >? ? ? >? ? ? >? ? ? >? ? Gluster
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? 9.x latest repository.
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? Seems 192G RAM are not
enough to
>? ? handle
>? ? ? >? ? 30 data
>? ? ? >? ? ? >? ? bricks + 15
>? ? ? >? ? ? >? ? ? >? ? ? >? ? arbiters
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? and
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? I often had to reload
glusterfsd
>? ? because
>? ? ? >? ? glusterfs
>? ? ? >? ? ? >? ? ? >? ? processed
>? ? ? >? ? ? >? ? ? >? ? ? >? ? got killed
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? for OOM.
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? On top of that,
performance have been
>? ? ? >? ? quite bad,
>? ? ? >? ? ? >? ? especially
>? ? ? >? ? ? >? ? ? >? ? ? >? ? when we
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? reached about 20M
files. On top of
>? ? that,
>? ? ? >? ? one of
>? ? ? >? ? ? >? ? the servers
>? ? ? >? ? ? >? ? ? >? ? ? >? ? have had
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? mobo issues that
resulted in memory
>? ? ? >? ? errors that
>? ? ? >? ? ? >? ? ? >? ? corrupted some
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? bricks fs
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? (XFS, it required
"xfs_reparir -L"
>? ? to fix).
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? Now I'm getting
lots of "stale
>? ? file handle"
>? ? ? >? ? ? >? ? errors and
>? ? ? >? ? ? >? ? ? >? ? other
>? ? ? >? ? ? >? ? ? >? ? ? >? ? errors
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? (like directories that
seem empty
>? ? from the
>? ? ? >? ? ? >? ? client but still
>? ? ? >? ? ? >? ? ? >? ? ? >? ? containing
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? files in some bricks)
and auto
>? ? healing seems
>? ? ? >? ? ? >? ? unable to
>? ? ? >? ? ? >? ? ? >? ? complete.
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? Since I can't keep
up continuing to
>? ? ? >? ? manually fix
>? ? ? >? ? ? >? ? all the
>? ? ? >? ? ? >? ? ? >? ? ? >? ? issues, I'm
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? thinking about
backup+destroy+recreate
>? ? ? >? ? strategy.
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? I think that if I
reduce the number of
>? ? ? >? ? bricks per
>? ? ? >? ? ? >? ? ? >? ? server to just 5
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? (RAID1 of 6x12TB
disks) I might
>? ? resolve RAM
>? ? ? >? ? ? >? ? issues - at the
>? ? ? >? ? ? >? ? ? >? ? ? >? ? cost of
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? longer heal times in
case a disk
>? ? fails. Am I
>? ? ? >? ? ? >? ? right or it's
>? ? ? >? ? ? >? ? ? >? ? ? >? ? useless?
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? Other recommendations?
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? Servers have space for
another 6
>? ? disks.
>? ? ? >? ? Maybe those
>? ? ? >? ? ? >? ? ? >? ? could be
>? ? ? >? ? ? >? ? ? >? ? ? >? ? used for
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? some SSDs to speed up
access?
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? TIA.
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? --
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? Diego Zuccato
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? DIFA - Dip. di Fisica
e Astronomia
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? Servizi Informatici
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? Alma Mater Studiorum -
Universit?
>? ? di Bologna
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? V.le Berti-Pichat 6/2
- 40127
>? ? Bologna - Italy
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? tel.: +39 051 20 95786
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? ________
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? Community Meeting
Calendar:
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? Schedule -
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? Every 2nd and 4th
Tuesday at 14:30
>? ? IST /
>? ? ? >? ? 09:00 UTC
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? Bridge:
>? ? ? > https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>
>? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>>
>? ? ? >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>
>? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>>>
>? ? ? >? ? ? >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>
>? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>>
>? ? ? >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>
>? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>>>>
>? ? ? >? ? ? >? ? ? >? ? ? >? ?
<https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>
>? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>>
>? ? ? >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>
>? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>>>
>? ? ? >? ? ? >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>
>? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>>
>? ? ? >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>
>? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>>>>>
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? 
>? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>
>? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>>
>? ? ? >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>
>? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>>>
>? ? ? >? ? ? >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>
>? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>>
>? ? ? >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>
>? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>>>>
>? ? ? >? ? ? >? ? ? >? ? ? >? ?
<https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>
>? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>>
>? ? ? >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>
>? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>>>
>? ? ? >? ? ? >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>
>? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>>
>? ? ? >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>
>? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>>>>>>
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? Gluster-users mailing
list
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? > Gluster-users at
gluster.org
>? ? <mailto:Gluster-users at gluster.org>
>? ? ? >? ? <mailto:Gluster-users at gluster.org>
>? ? ? >? ? ? >? ? <mailto:Gluster-users at gluster.org>
>? ? ? >? ? ? >? ? ? >? ? <mailto:Gluster-users at
gluster.org>
>? ? ? >? ? ? >? ? <mailto:Gluster-users at gluster.org>
>? ? ? >? ? ? >? ? ? >? ? ? >? ? <mailto:Gluster-users at
gluster.org>
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >
https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>
>? ? ? >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>>
>? ? ? >? ? ? >? 
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>
>? ? ? >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>>>
>? ? ? >? ? ? >? ? ? >
>? ? ? >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>
>? ? ? >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>>
>? ? ? >? ? ? >? 
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>
>? ? ? >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ?
<https://lists.gluster.org/mailman/listinfo/gluster-users>>>>
>? ? ? >? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? 
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>
>? ? ? >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>>
>? ? ? >? ? ? >? 
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>
>? ? ? >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>>>
>? ? ? >? ? ? >? ? ? >
>? ? ? >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>
>? ? ? >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>>
>? ? ? >? ? ? >? 
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>
>? ? ? >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ?
<https://lists.gluster.org/mailman/listinfo/gluster-users>>>>>
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? ? ? >
>? ? ? >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>
>? ? ? >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>>
>? ? ? >? ? ? >? 
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>
>? ? ? >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>>>
>? ? ? >? ? ? >? ? ? >
>? ? ? >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>
>? ? ? >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>>
>? ? ? >? ? ? >? 
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>
>? ? ? >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ?
<https://lists.gluster.org/mailman/listinfo/gluster-users>>>>
>? ? ? >? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? 
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>
>? ? ? >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>>
>? ? ? >? ? ? >? 
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>
>? ? ? >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>>>
>? ? ? >? ? ? >? ? ? >
>? ? ? >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>
>? ? ? >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>>
>? ? ? >? ? ? >? 
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>
>? ? ? >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ?
<https://lists.gluster.org/mailman/listinfo/gluster-users>>>>>>
>? ? ? >
>? ? ? >? ? ? >
>? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? ? ? >? ? ? >? ? --
>? ? ? >? ? ? >? ? ? >? ? ? >? ? Diego Zuccato
>? ? ? >? ? ? >? ? ? >? ? ? >? ? DIFA - Dip. di Fisica e
Astronomia
>? ? ? >? ? ? >? ? ? >? ? ? >? ? Servizi Informatici
>? ? ? >? ? ? >? ? ? >? ? ? >? ? Alma Mater Studiorum -
Universit? di Bologna
>? ? ? >? ? ? >? ? ? >? ? ? >? ? V.le Berti-Pichat 6/2 - 40127
Bologna - Italy
>? ? ? >? ? ? >? ? ? >? ? ? >? ? tel.: +39 051 20 95786
>? ? ? >? ? ? >? ? ? >? ? ? >? ? ________
>? ? ? >? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? ? ? >? ? ? >? ? Community Meeting Calendar:
>? ? ? >? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? ? ? >? ? ? >? ? Schedule -
>? ? ? >? ? ? >? ? ? >? ? ? >? ? Every 2nd and 4th Tuesday at
14:30 IST /
>? ? 09:00 UTC
>? ? ? >? ? ? >? ? ? >? ? ? >? ? Bridge:
>? ? https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>
>? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>>
>? ? ? >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>
>? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>>>
>? ? ? >? ? ? >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>
>? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>>
>? ? ? >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>
>? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>>>>
>? ? ? >? ? ? >? ? ? >? ? ? >? ?
<https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>
>? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>>
>? ? ? >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>
>? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>>>
>? ? ? >? ? ? >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>
>? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>>
>? ? ? >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>
>? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>? ? <https://meet.google.com/cpu-eiue-hvk>>>>>
>? ? ? >? ? ? >? ? ? >? ? ? >? ? Gluster-users mailing list
>? ? ? >? ? ? >? ? ? >? ? ? > Gluster-users at gluster.org
>? ? <mailto:Gluster-users at gluster.org>
>? ? ? >? ? <mailto:Gluster-users at gluster.org>
>? ? ? >? ? ? >? ? <mailto:Gluster-users at gluster.org>
>? ? ? >? ? <mailto:Gluster-users at gluster.org>
>? ? ? >? ? ? >? ? ? >? ? <mailto:Gluster-users at
gluster.org>
>? ? ? >? ? ? >? ? ? >? ? ? >
>? ? ? > https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>
>? ? ? >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>>
>? ? ? >? ? ? >? 
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>
>? ? ? >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>>>
>? ? ? >? ? ? >? ? ? >
>? ? ? >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>
>? ? ? >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>>
>? ? ? >? ? ? >? 
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>
>? ? ? >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ?
<https://lists.gluster.org/mailman/listinfo/gluster-users>>>>
>? ? ? >? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? 
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>
>? ? ? >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>>
>? ? ? >? ? ? >? 
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>
>? ? ? >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>>>
>? ? ? >? ? ? >? ? ? >
>? ? ? >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>
>? ? ? >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>>
>? ? ? >? ? ? >? 
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>
>? ? ? >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>? ?
<https://lists.gluster.org/mailman/listinfo/gluster-users>>>>>
>? ? ? >? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >? ? ? >? ? --
>? ? ? >? ? ? >? ? ? >? ? Diego Zuccato
>? ? ? >? ? ? >? ? ? >? ? DIFA - Dip. di Fisica e Astronomia
>? ? ? >? ? ? >? ? ? >? ? Servizi Informatici
>? ? ? >? ? ? >? ? ? >? ? Alma Mater Studiorum - Universit? di
Bologna
>? ? ? >? ? ? >? ? ? >? ? V.le Berti-Pichat 6/2 - 40127 Bologna -
Italy
>? ? ? >? ? ? >? ? ? >? ? tel.: +39 051 20 95786
>? ? ? >? ? ? >? ? ? >
>? ? ? >? ? ? >
>? ? ? >? ? ? >? ? --
>? ? ? >? ? ? >? ? Diego Zuccato
>? ? ? >? ? ? >? ? DIFA - Dip. di Fisica e Astronomia
>? ? ? >? ? ? >? ? Servizi Informatici
>? ? ? >? ? ? >? ? Alma Mater Studiorum - Universit? di Bologna
>? ? ? >? ? ? >? ? V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
>? ? ? >? ? ? >? ? tel.: +39 051 20 95786
>? ? ? >? ? ? >
>? ? ? >
>? ? ? >? ? --
>? ? ? >? ? Diego Zuccato
>? ? ? >? ? DIFA - Dip. di Fisica e Astronomia
>? ? ? >? ? Servizi Informatici
>? ? ? >? ? Alma Mater Studiorum - Universit? di Bologna
>? ? ? >? ? V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
>? ? ? >? ? tel.: +39 051 20 95786
>? ? ? >
> 
>? ? -- 
>? ? Diego Zuccato
>? ? DIFA - Dip. di Fisica e Astronomia
>? ? Servizi Informatici
>? ? Alma Mater Studiorum - Universit? di Bologna
>? ? V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
>? ? tel.: +39 051 20 95786
> 
-- 
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Universit? di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786
  
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20230324/c876f852/attachment.html>

Diego Zuccato

2023-Apr-23 07:14 UTC

head link

[Gluster-users] How to configure?

After a lot of tests and unsuccessful searching, I decided to start from 
scratch: I'm going to ditch the old volume and create a new one.

I have 3 servers with 30 12TB disks each. Since I'm going to start a new 
volume, could it be better to group disks in 10 3-disk (or 6 5-disk) 
RAID-0 volumes to reduce the number of bricks? Redundancy would be given 
by replica 2 (still undecided about arbiter vs thin-arbiter...).

Current configuration is:
root at str957-clustor00:~# gluster v info cluster_data

Volume Name: cluster_data
Type: Distributed-Replicate
Volume ID: a8caaa90-d161-45bb-a68c-278263a8531a
Status: Started
Snapshot Count: 0
Number of Bricks: 45 x (2 + 1) = 135
Transport-type: tcp
Bricks:
Brick1: clustor00:/srv/bricks/00/d
Brick2: clustor01:/srv/bricks/00/d
Brick3: clustor02:/srv/bricks/00/q (arbiter)
...
Brick133: clustor01:/srv/bricks/29/d
Brick134: clustor02:/srv/bricks/29/d
Brick135: clustor00:/srv/bricks/14/q (arbiter)
Options Reconfigured:
cluster.background-self-heal-count: 256
cluster.heal-wait-queue-length: 10000
performance.quick-read: off
cluster.entry-self-heal: on
cluster.data-self-heal-algorithm: full
cluster.metadata-self-heal: on
cluster.shd-max-threads: 2
network.inode-lru-limit: 500000
performance.md-cache-timeout: 600
performance.cache-invalidation: on
features.cache-invalidation-timeout: 600
features.cache-invalidation: on
features.quota-deem-statfs: on
performance.readdir-ahead: on
cluster.granular-entry-heal: enable
features.scrub: Active
features.bitrot: on
cluster.lookup-optimize: on
performance.stat-prefetch: on
performance.cache-refresh-timeout: 60
performance.parallel-readdir: on
performance.write-behind-window-size: 128MB
cluster.self-heal-daemon: enable
features.inode-quota: on
features.quota: on
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
client.event-threads: 1
features.scrub-throttle: normal
diagnostics.brick-log-level: ERROR
diagnostics.client-log-level: ERROR
config.brick-threads: 0
cluster.lookup-unhashed: on
config.client-threads: 1
cluster.use-anonymous-inode: off
diagnostics.brick-sys-log-level: CRITICAL
features.scrub-freq: monthly
cluster.data-self-heal: on
cluster.brick-multiplex: on
cluster.daemon-log-level: ERROR

Each node is a dual-Xeon 4210 (for a total of 20 cores, 40 threads) 
equipped with 192GB RAM (that got exhausted quite often, before enabling 
brick-multiplex).

Diego

Il 24/03/2023 19:21, Strahil Nikolov ha scritto:> Try finding if any of them is missing on one of the systems.
> 
> Best Regards,
> Strahil Nikolov
> 
>     On Fri, Mar 24, 2023 at 15:59, Diego Zuccato
>     <diego.zuccato at unibo.it> wrote:
>     There are 285 files in /var/lib/glusterd/vols/cluster_data ...
>     including
>     many files with names related to quorum bricks already moved to a
>     different path (like cluster_data.client.clustor02.srv-quorum-00-d.vol
>     that should already have been replaced by
>     cluster_data.clustor02.srv-bricks-00-q.vol -- and both vol files
exist).
> 
>     Is there something I should check inside the volfiles?
> 
>     Diego
> 
>     Il 24/03/2023 13:05, Strahil Nikolov ha scritto:
>      > Can you check your volume file contents?
>      > Maybe it really can't find (or access) a specific volfile ?
>      >
>      > Best Regards,
>      > Strahil Nikolov
>      >
>      >? ? On Fri, Mar 24, 2023 at 8:07, Diego Zuccato
>      >? ? <diego.zuccato at unibo.it <mailto:diego.zuccato at
unibo.it>> wrote:
>      >? ? In glfsheal-Connection.log I see many lines like:
>      >? ? [2023-03-13 23:04:40.241481 +0000] E [MSGID: 104021]
>      >? ? [glfs-mgmt.c:586:glfs_mgmt_getspec_cbk] 0-gfapi: failed to get
the
>      >? ? volume file [{from server}, {errno=2}, {error=File o directory
non
>      >? ? esistente}]
>      >
>      >? ? And *lots* of gfid-mismatch errors in glustershd.log .
>      >
>      >? ? Couldn't find anything that would prevent heal to start.
:(
>      >
>      >? ? Diego
>      >
>      >? ? Il 21/03/2023 20:39, Strahil Nikolov ha scritto:
>      >? ? ? > I have no clue. Have you checked for errors in the logs
?
>     Maybe you
>      >? ? ? > might find something useful.
>      >? ? ? >
>      >? ? ? > Best Regards,
>      >? ? ? > Strahil Nikolov
>      >? ? ? >
>      >? ? ? >? ? On Tue, Mar 21, 2023 at 9:56, Diego Zuccato
>      >? ? ? >? ? <diego.zuccato at unibo.it
<mailto:diego.zuccato at unibo.it>
>     <mailto:diego.zuccato at unibo.it>> wrote:
>      >? ? ? >? ? Killed glfsheal, after a day there were 218
processes, then
>      >? ? they got
>      >? ? ? >? ? killed by OOM during the weekend. Now there are no
>     processes
>      >? ? active.
>      >? ? ? >? ? Trying to run "heal info" reports lots of
files quite
>     quickly
>      >? ? but does
>      >? ? ? >? ? not spawn any glfsheal process. And neither does
restarting
>      >? ? glusterd.
>      >? ? ? >? ? Is there some way to selectively run glfsheal to fix
>     one brick
>      >? ? at a
>      >? ? ? >? ? time?
>      >? ? ? >
>      >? ? ? >? ? Diego
>      >? ? ? >
>      >? ? ? >? ? Il 21/03/2023 01:21, Strahil Nikolov ha scritto:
>      >? ? ? >? ? ? > Theoretically it might help.
>      >? ? ? >? ? ? > If possible, try to resolve any pending
heals.
>      >? ? ? >? ? ? >
>      >? ? ? >? ? ? > Best Regards,
>      >? ? ? >? ? ? > Strahil Nikolov
>      >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? On Thu, Mar 16, 2023 at 15:29, Diego
Zuccato
>      >? ? ? >? ? ? >? ? <diego.zuccato at unibo.it
>     <mailto:diego.zuccato at unibo.it> <mailto:diego.zuccato at
unibo.it>
>      >? ? <mailto:diego.zuccato at unibo.it>> wrote:
>      >? ? ? >? ? ? >? ? In Debian stopping glusterd does not stop
brick
>      >? ? processes: to stop
>      >? ? ? >? ? ? >? ? everything (and free the memory) I have to
>      >? ? ? >? ? ? >? ? systemctl stop glusterd
>      >? ? ? >? ? ? >? ? ? ? killall glusterfs{,d}
>      >? ? ? >? ? ? >? ? ? ? killall glfsheal
>      >? ? ? >? ? ? >? ? ? ? systemctl start glusterd
>      >? ? ? >? ? ? >? ? [this behaviour hangs a simple reboot of a
>     machine running
>      >? ? ? >? ? glusterd...
>      >? ? ? >? ? ? >? ? not nice]
>      >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? For now I just restarted glusterd w/o
killing
>     the bricks:
>      >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? root at str957-clustor00:~# ps aux|grep
glfsheal|wc
>     -l ;
>      >? ? ? >? ? systemctl restart
>      >? ? ? >? ? ? >? ? glusterd ; ps aux|grep glfsheal|wc -l
>      >? ? ? >? ? ? >? ? 618
>      >? ? ? >? ? ? >? ? 618
>      >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? No change neither in glfsheal processes
nor in free
>      >? ? memory :(
>      >? ? ? >? ? ? >? ? Should I "killall glfsheal"
before OOK kicks in?
>      >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? Diego
>      >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? Il 16/03/2023 12:37, Strahil Nikolov ha
scritto:
>      >? ? ? >? ? ? >? ? ? > Can you restart glusterd service
(first
>     check that
>      >? ? it was not
>      >? ? ? >? ? ? >? ? modified
>      >? ? ? >? ? ? >? ? ? > to kill the bricks)?
>      >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? ? > Best Regards,
>      >? ? ? >? ? ? >? ? ? > Strahil Nikolov
>      >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? ? >? ? On Thu, Mar 16, 2023 at 8:26,
Diego Zuccato
>      >? ? ? >? ? ? >? ? ? >? ? <diego.zuccato at unibo.it
>     <mailto:diego.zuccato at unibo.it>
>      >? ? <mailto:diego.zuccato at unibo.it>
<mailto:diego.zuccato at unibo.it>
>      >? ? ? >? ? <mailto:diego.zuccato at unibo.it>> wrote:
>      >? ? ? >? ? ? >? ? ? >? ? OOM is just just a matter of
time.
>      >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? ? >? ? Today mem use is up to 177G/187
and:
>      >? ? ? >? ? ? >? ? ? >? ? # ps aux|grep glfsheal|wc -l
>      >? ? ? >? ? ? >? ? ? >? ? 551
>      >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? ? >? ? (well, one is actually the grep
process, so
>      >? ? "only" 550
>      >? ? ? >? ? glfsheal
>      >? ? ? >? ? ? >? ? ? >? ? processes.
>      >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? ? >? ? I'll take the last 5:
>      >? ? ? >? ? ? >? ? ? >? ? root? ? 3266352? 0.5? 0.0 600292
93044 ?
>      ? ? ? Sl
>      >? ? ? >? ? 06:55? 0:07
>      >? ? ? >? ? ? >? ? ? >? ? /usr/libexec/glusterfs/glfsheal
cluster_data
>      >? ? ? >? ? info-summary --xml
>      >? ? ? >? ? ? >? ? ? >? ? root? ? 3267220? 0.7? 0.0 600292
91964 ?
>      ? ? ? Sl
>      >? ? ? >? ? 07:00? 0:07
>      >? ? ? >? ? ? >? ? ? >? ? /usr/libexec/glusterfs/glfsheal
cluster_data
>      >? ? ? >? ? info-summary --xml
>      >? ? ? >? ? ? >? ? ? >? ? root? ? 3268076? 1.0? 0.0 600160
88216 ?
>      ? ? ? Sl
>      >? ? ? >? ? 07:05? 0:08
>      >? ? ? >? ? ? >? ? ? >? ? /usr/libexec/glusterfs/glfsheal
cluster_data
>      >? ? ? >? ? info-summary --xml
>      >? ? ? >? ? ? >? ? ? >? ? root? ? 3269492? 1.6? 0.0 600292
91248 ?
>      ? ? ? Sl
>      >? ? ? >? ? 07:10? 0:07
>      >? ? ? >? ? ? >? ? ? >? ? /usr/libexec/glusterfs/glfsheal
cluster_data
>      >? ? ? >? ? info-summary --xml
>      >? ? ? >? ? ? >? ? ? >? ? root? ? 3270354? 4.4? 0.0 600292
93260 ?
>      ? ? ? Sl
>      >? ? ? >? ? 07:15? 0:07
>      >? ? ? >? ? ? >? ? ? >? ? /usr/libexec/glusterfs/glfsheal
cluster_data
>      >? ? ? >? ? info-summary --xml
>      >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? ? >? ? -8<--
>      >? ? ? >? ? ? >? ? ? >? ? root at str957-clustor00:~# ps
-o ppid= 3266352
>      >? ? ? >? ? ? >? ? ? >? ? 3266345
>      >? ? ? >? ? ? >? ? ? >? ? root at str957-clustor00:~# ps
-o ppid= 3267220
>      >? ? ? >? ? ? >? ? ? >? ? 3267213
>      >? ? ? >? ? ? >? ? ? >? ? root at str957-clustor00:~# ps
-o ppid= 3268076
>      >? ? ? >? ? ? >? ? ? >? ? 3268069
>      >? ? ? >? ? ? >? ? ? >? ? root at str957-clustor00:~# ps
-o ppid= 3269492
>      >? ? ? >? ? ? >? ? ? >? ? 3269485
>      >? ? ? >? ? ? >? ? ? >? ? root at str957-clustor00:~# ps
-o ppid= 3270354
>      >? ? ? >? ? ? >? ? ? >? ? 3270347
>      >? ? ? >? ? ? >? ? ? >? ? root at str957-clustor00:~# ps
aux|grep 3266345
>      >? ? ? >? ? ? >? ? ? >? ? root? ? 3266345? 0.0? 0.0 430536
10764 ?
>      ? ? ? Sl
>      >? ? ? >? ? 06:55? 0:00
>      >? ? ? >? ? ? >? ? ? >? ? gluster volume heal cluster_data
info
>     summary --xml
>      >? ? ? >? ? ? >? ? ? >? ? root? ? 3271532? 0.0? 0.0? 6260?
2500
>     pts/1? ? S+
>      >? ? ? >? ? 07:21? 0:00
>      >? ? ? >? ? ? >? ? grep
>      >? ? ? >? ? ? >? ? ? >? ? 3266345
>      >? ? ? >? ? ? >? ? ? >? ? root at str957-clustor00:~# ps
aux|grep 3267213
>      >? ? ? >? ? ? >? ? ? >? ? root? ? 3267213? 0.0? 0.0 430536
10644 ?
>      ? ? ? Sl
>      >? ? ? >? ? 07:00? 0:00
>      >? ? ? >? ? ? >? ? ? >? ? gluster volume heal cluster_data
info
>     summary --xml
>      >? ? ? >? ? ? >? ? ? >? ? root? ? 3271599? 0.0? 0.0? 6260?
2480
>     pts/1? ? S+
>      >? ? ? >? ? 07:22? 0:00
>      >? ? ? >? ? ? >? ? grep
>      >? ? ? >? ? ? >? ? ? >? ? 3267213
>      >? ? ? >? ? ? >? ? ? >? ? root at str957-clustor00:~# ps
aux|grep 3268069
>      >? ? ? >? ? ? >? ? ? >? ? root? ? 3268069? 0.0? 0.0 430536
10704 ?
>      ? ? ? Sl
>      >? ? ? >? ? 07:05? 0:00
>      >? ? ? >? ? ? >? ? ? >? ? gluster volume heal cluster_data
info
>     summary --xml
>      >? ? ? >? ? ? >? ? ? >? ? root? ? 3271626? 0.0? 0.0? 6260?
2516
>     pts/1? ? S+
>      >? ? ? >? ? 07:22? 0:00
>      >? ? ? >? ? ? >? ? grep
>      >? ? ? >? ? ? >? ? ? >? ? 3268069
>      >? ? ? >? ? ? >? ? ? >? ? root at str957-clustor00:~# ps
aux|grep 3269485
>      >? ? ? >? ? ? >? ? ? >? ? root? ? 3269485? 0.0? 0.0 430536
10756 ?
>      ? ? ? Sl
>      >? ? ? >? ? 07:10? 0:00
>      >? ? ? >? ? ? >? ? ? >? ? gluster volume heal cluster_data
info
>     summary --xml
>      >? ? ? >? ? ? >? ? ? >? ? root? ? 3271647? 0.0? 0.0? 6260?
2480
>     pts/1? ? S+
>      >? ? ? >? ? 07:22? 0:00
>      >? ? ? >? ? ? >? ? grep
>      >? ? ? >? ? ? >? ? ? >? ? 3269485
>      >? ? ? >? ? ? >? ? ? >? ? root at str957-clustor00:~# ps
aux|grep 3270347
>      >? ? ? >? ? ? >? ? ? >? ? root? ? 3270347? 0.0? 0.0 430536
10672 ?
>      ? ? ? Sl
>      >? ? ? >? ? 07:15? 0:00
>      >? ? ? >? ? ? >? ? ? >? ? gluster volume heal cluster_data
info
>     summary --xml
>      >? ? ? >? ? ? >? ? ? >? ? root? ? 3271666? 0.0? 0.0? 6260?
2568
>     pts/1? ? S+
>      >? ? ? >? ? 07:22? 0:00
>      >? ? ? >? ? ? >? ? grep
>      >? ? ? >? ? ? >? ? ? >? ? 3270347
>      >? ? ? >? ? ? >? ? ? >? ? -8<--
>      >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? ? >? ? Seems glfsheal is spawning more
processes.
>      >? ? ? >? ? ? >? ? ? >? ? I can't rule out a metadata
corruption (or at
>      >? ? least a
>      >? ? ? >? ? desync),
>      >? ? ? >? ? ? >? ? but it
>      >? ? ? >? ? ? >? ? ? >? ? shouldn't happen...
>      >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? ? >? ? Diego
>      >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? ? >? ? Il 15/03/2023 20:11, Strahil
Nikolov ha
>     scritto:
>      >? ? ? >? ? ? >? ? ? >? ? ? > If you don't
experience any OOM , you can
>      >? ? focus on
>      >? ? ? >? ? the heals.
>      >? ? ? >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? ? >? ? ? > 284 processes of glfsheal
seems odd.
>      >? ? ? >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? ? >? ? ? > Can you check the ppid
for 2-3
>     randomly picked ?
>      >? ? ? >? ? ? >? ? ? >? ? ? > ps -o ppid= <pid>
>      >? ? ? >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? ? >? ? ? > Best Regards,
>      >? ? ? >? ? ? >? ? ? >? ? ? > Strahil Nikolov
>      >? ? ? >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? On Wed, Mar 15, 2023
at 9:54,
>     Diego Zuccato
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? <diego.zuccato at
unibo.it
>     <mailto:diego.zuccato at unibo.it>
>      >? ? <mailto:diego.zuccato at unibo.it>
>      >? ? ? >? ? <mailto:diego.zuccato at unibo.it>
>     <mailto:diego.zuccato at unibo.it>
>      >? ? ? >? ? ? >? ? <mailto:diego.zuccato at
unibo.it>> wrote:
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? I enabled it yesterday
and that
>     greatly
>      >? ? reduced
>      >? ? ? >? ? memory
>      >? ? ? >? ? ? >? ? pressure.
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? Current volume info:
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? -8<--
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? Volume Name:
cluster_data
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? Type:
Distributed-Replicate
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? Volume ID:
>      >? ? a8caaa90-d161-45bb-a68c-278263a8531a
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? Status: Started
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? Snapshot Count: 0
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? Number of Bricks: 45 x
(2 + 1) = 135
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? Transport-type: tcp
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? Bricks:
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? Brick1:
clustor00:/srv/bricks/00/d
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? Brick2:
clustor01:/srv/bricks/00/d
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? Brick3:
clustor02:/srv/bricks/00/q
>     (arbiter)
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? [...]
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? Brick133:
clustor01:/srv/bricks/29/d
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? Brick134:
clustor02:/srv/bricks/29/d
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? Brick135:
clustor00:/srv/bricks/14/q
>      >? ? (arbiter)
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? Options Reconfigured:
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ?
performance.quick-read: off
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ?
cluster.entry-self-heal: on
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ?
cluster.data-self-heal-algorithm: full
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ?
cluster.metadata-self-heal: on
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ?
cluster.shd-max-threads: 2
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ?
network.inode-lru-limit: 500000
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ?
performance.md-cache-timeout: 600
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ?
performance.cache-invalidation: on
>      >? ? ? >? ? ? >? ? ? >? ? ? >   
>     features.cache-invalidation-timeout: 600
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ?
features.cache-invalidation: on
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ?
features.quota-deem-statfs: on
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ?
performance.readdir-ahead: on
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ?
cluster.granular-entry-heal: enable
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? features.scrub: Active
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? features.bitrot: on
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ?
cluster.lookup-optimize: on
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ?
performance.stat-prefetch: on
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ?
performance.cache-refresh-timeout: 60
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ?
performance.parallel-readdir: on
>      >? ? ? >? ? ? >? ? ? >? ? ? >   
>     performance.write-behind-window-size: 128MB
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ?
cluster.self-heal-daemon: enable
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? features.inode-quota:
on
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? features.quota: on
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ?
transport.address-family: inet
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? nfs.disable: on
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ?
performance.client-io-threads: off
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? client.event-threads:
1
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ?
features.scrub-throttle: normal
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ?
diagnostics.brick-log-level: ERROR
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ?
diagnostics.client-log-level: ERROR
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? config.brick-threads:
0
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ?
cluster.lookup-unhashed: on
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? config.client-threads:
1
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ?
cluster.use-anonymous-inode: off
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ?
diagnostics.brick-sys-log-level:
>     CRITICAL
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? features.scrub-freq:
monthly
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ?
cluster.data-self-heal: on
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ?
cluster.brick-multiplex: on
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ?
cluster.daemon-log-level: ERROR
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? -8<--
>      >? ? ? >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? htop reports that
memory usage is
>     up to 143G,
>      >? ? ? >? ? there are 602
>      >? ? ? >? ? ? >? ? ? >? ? tasks and
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? 5232 threads (~20
running) on
>     clustor00,
>      >? ? 117G/49
>      >? ? ? >? ? tasks/1565
>      >? ? ? >? ? ? >? ? ? >? ? threads on
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? clustor01 and 126G/45
tasks/1574
>     threads on
>      >? ? ? >? ? clustor02.
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? I see quite a lot
(284!) of glfsheal
>      >? ? processes
>      >? ? ? >? ? running on
>      >? ? ? >? ? ? >? ? ? >? ? clustor00 (a
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? "gluster v heal
cluster_data info
>     summary" is
>      >? ? ? >? ? running
>      >? ? ? >? ? ? >? ? on clustor02
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? since
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? yesterday, still no
output).
>     Shouldn't be
>      >? ? just
>      >? ? ? >? ? one per
>      >? ? ? >? ? ? >? ? brick?
>      >? ? ? >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? Diego
>      >? ? ? >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? Il 15/03/2023 08:30,
Strahil
>     Nikolov ha
>      >? ? scritto:
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? > Do you use
brick multiplexing ?
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? > Best Regards,
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? > Strahil Nikolov
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? On Tue, Mar
14, 2023 at 16:44,
>      >? ? Diego Zuccato
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ?
<diego.zuccato at unibo.it
>     <mailto:diego.zuccato at unibo.it>
>      >? ? <mailto:diego.zuccato at unibo.it>
>      >? ? ? >? ? <mailto:diego.zuccato at unibo.it>
>      >? ? ? >? ? ? >? ? <mailto:diego.zuccato at unibo.it>
>      >? ? <mailto:diego.zuccato at unibo.it>
>      >? ? ? >? ? ? >? ? ? >? ? <mailto:diego.zuccato at
unibo.it>> wrote:
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? Hello all.
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? Our Gluster
9.6 cluster is
>     showing
>      >? ? increasing
>      >? ? ? >? ? ? >? ? problems.
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? Currently
it's composed of 3
>      >? ? servers (2x
>      >? ? ? >? ? Intel Xeon
>      >? ? ? >? ? ? >? ? ? >? ? 4210 [20
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? cores dual
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? thread,
total 40 threads],
>     192GB
>      >? ? RAM, 30x
>      >? ? ? >? ? HGST
>      >? ? ? >? ? ? >? ? ? >? ? HUH721212AL5200
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? [12TB]),
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? configured
in replica 3
>     arbiter 1.
>      >? ? Using
>      >? ? ? >? ? Debian
>      >? ? ? >? ? ? >? ? ? >? ? packages from
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? Gluster
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? 9.x latest
repository.
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? Seems 192G
RAM are not
>     enough to
>      >? ? handle
>      >? ? ? >? ? 30 data
>      >? ? ? >? ? ? >? ? bricks + 15
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? arbiters
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? and
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? I often had
to reload
>     glusterfsd
>      >? ? because
>      >? ? ? >? ? glusterfs
>      >? ? ? >? ? ? >? ? ? >? ? processed
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? got killed
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? for OOM.
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? On top of
that, performance
>     have been
>      >? ? ? >? ? quite bad,
>      >? ? ? >? ? ? >? ? especially
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? when we
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? reached
about 20M files. On
>     top of
>      >? ? that,
>      >? ? ? >? ? one of
>      >? ? ? >? ? ? >? ? the servers
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? have had
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? mobo issues
that resulted
>     in memory
>      >? ? ? >? ? errors that
>      >? ? ? >? ? ? >? ? ? >? ? corrupted some
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? bricks fs
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? (XFS, it
required
>     "xfs_reparir -L"
>      >? ? to fix).
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? Now I'm
getting lots of "stale
>      >? ? file handle"
>      >? ? ? >? ? ? >? ? errors and
>      >? ? ? >? ? ? >? ? ? >? ? other
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? errors
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? (like
directories that seem
>     empty
>      >? ? from the
>      >? ? ? >? ? ? >? ? client but still
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? containing
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? files in
some bricks) and auto
>      >? ? healing seems
>      >? ? ? >? ? ? >? ? unable to
>      >? ? ? >? ? ? >? ? ? >? ? complete.
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? Since I
can't keep up
>     continuing to
>      >? ? ? >? ? manually fix
>      >? ? ? >? ? ? >? ? all the
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? issues, I'm
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? thinking
about
>     backup+destroy+recreate
>      >? ? ? >? ? strategy.
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? I think that
if I reduce
>     the number of
>      >? ? ? >? ? bricks per
>      >? ? ? >? ? ? >? ? ? >? ? server to just 5
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? (RAID1 of
6x12TB disks) I might
>      >? ? resolve RAM
>      >? ? ? >? ? ? >? ? issues - at the
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? cost of
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? longer heal
times in case a
>     disk
>      >? ? fails. Am I
>      >? ? ? >? ? ? >? ? right or it's
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? useless?
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? Other
recommendations?
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? Servers have
space for
>     another 6
>      >? ? disks.
>      >? ? ? >? ? Maybe those
>      >? ? ? >? ? ? >? ? ? >? ? could be
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? used for
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? some SSDs to
speed up access?
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? TIA.
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? --
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? Diego
Zuccato
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? DIFA - Dip.
di Fisica e
>     Astronomia
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? Servizi
Informatici
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? Alma Mater
Studiorum -
>     Universit?
>      >? ? di Bologna
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? V.le
Berti-Pichat 6/2 - 40127
>      >? ? Bologna - Italy
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? tel.: +39
051 20 95786
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? ________
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? Community
Meeting Calendar:
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? Schedule -
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? Every 2nd
and 4th Tuesday
>     at 14:30
>      >? ? IST /
>      >? ? ? >? ? 09:00 UTC
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ? Bridge:
>      >? ? ? > https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>
>      >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>>
>      >? ? ? >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>
>      >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>>>
>      >? ? ? >? ? ? >? ? ? >? ?
<https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>
>      >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>>
>      >? ? ? >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>
>      >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>>>>
>      >? ? ? >? ? ? >? ? ? >? ? ? >   
>     <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>
>      >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>>
>      >? ? ? >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>
>      >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>>>
>      >? ? ? >? ? ? >? ? ? >? ?
<https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>
>      >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>>
>      >? ? ? >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>
>      >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>>>>>
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>
>      >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>>
>      >? ? ? >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>
>      >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>>>
>      >? ? ? >? ? ? >? ? ? >? ?
<https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>
>      >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>>
>      >? ? ? >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>
>      >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>>>>
>      >? ? ? >? ? ? >? ? ? >? ? ? >   
>     <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>
>      >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>>
>      >? ? ? >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>
>      >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>>>
>      >? ? ? >? ? ? >? ? ? >? ?
<https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>
>      >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>>
>      >? ? ? >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>
>      >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>>>>>>
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >? ?
Gluster-users mailing list
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? > Gluster-users
at gluster.org
>     <mailto:Gluster-users at gluster.org>
>      >? ? <mailto:Gluster-users at gluster.org>
>      >? ? ? >? ? <mailto:Gluster-users at gluster.org>
>      >? ? ? >? ? ? >? ? <mailto:Gluster-users at
gluster.org>
>      >? ? ? >? ? ? >? ? ? >? ? <mailto:Gluster-users at
gluster.org>
>      >? ? ? >? ? ? >? ? <mailto:Gluster-users at
gluster.org>
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ?
<mailto:Gluster-users at gluster.org>
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >
>     https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>>
>      >? ? ? >   
>     <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>    
<https://lists.gluster.org/mailman/listinfo/gluster-users>>>
>      >? ? ? >? ? ? >
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>>
>      >? ? ? >   
>     <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>    
<https://lists.gluster.org/mailman/listinfo/gluster-users>>>>
>      >? ? ? >? ? ? >? ? ? >
>      >? ? ? >   
>     <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>>
>      >? ? ? >   
>     <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>    
<https://lists.gluster.org/mailman/listinfo/gluster-users>>>
>      >? ? ? >? ? ? >
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>>
>      >? ? ? >   
>     <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>    
<https://lists.gluster.org/mailman/listinfo/gluster-users>>>>>
>      >? ? ? >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>>
>      >? ? ? >   
>     <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>    
<https://lists.gluster.org/mailman/listinfo/gluster-users>>>
>      >? ? ? >? ? ? >
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>>
>      >? ? ? >   
>     <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>    
<https://lists.gluster.org/mailman/listinfo/gluster-users>>>>
>      >? ? ? >? ? ? >? ? ? >
>      >? ? ? >   
>     <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>>
>      >? ? ? >   
>     <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>    
<https://lists.gluster.org/mailman/listinfo/gluster-users>>>
>      >? ? ? >? ? ? >
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>>
>      >? ? ? >   
>     <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>    
<https://lists.gluster.org/mailman/listinfo/gluster-users>>>>>>
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? ? >
>      >? ? ? >   
>     <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>>
>      >? ? ? >   
>     <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>    
<https://lists.gluster.org/mailman/listinfo/gluster-users>>>
>      >? ? ? >? ? ? >
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>>
>      >? ? ? >   
>     <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>    
<https://lists.gluster.org/mailman/listinfo/gluster-users>>>>
>      >? ? ? >? ? ? >? ? ? >
>      >? ? ? >   
>     <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>>
>      >? ? ? >   
>     <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>    
<https://lists.gluster.org/mailman/listinfo/gluster-users>>>
>      >? ? ? >? ? ? >
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>>
>      >? ? ? >   
>     <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>    
<https://lists.gluster.org/mailman/listinfo/gluster-users>>>>>
>      >? ? ? >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>>
>      >? ? ? >   
>     <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>    
<https://lists.gluster.org/mailman/listinfo/gluster-users>>>
>      >? ? ? >? ? ? >
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>>
>      >? ? ? >   
>     <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>    
<https://lists.gluster.org/mailman/listinfo/gluster-users>>>>
>      >? ? ? >? ? ? >? ? ? >
>      >? ? ? >   
>     <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>>
>      >? ? ? >   
>     <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>    
<https://lists.gluster.org/mailman/listinfo/gluster-users>>>
>      >? ? ? >? ? ? >
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>>
>      >? ? ? >   
>     <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>    
<https://lists.gluster.org/mailman/listinfo/gluster-users>>>>>>>
>      >? ? ? >
>      >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? --
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? Diego Zuccato
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? DIFA - Dip. di Fisica
e Astronomia
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? Servizi Informatici
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? Alma Mater Studiorum -
Universit?
>     di Bologna
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? V.le Berti-Pichat 6/2
- 40127
>     Bologna - Italy
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? tel.: +39 051 20 95786
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? ________
>      >? ? ? >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? Community Meeting
Calendar:
>      >? ? ? >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? Schedule -
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? Every 2nd and 4th
Tuesday at 14:30
>     IST /
>      >? ? 09:00 UTC
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? Bridge:
>      > https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>
>      >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>>
>      >? ? ? >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>
>      >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>>>
>      >? ? ? >? ? ? >? ? ? >? ?
<https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>
>      >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>>
>      >? ? ? >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>
>      >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>>>>
>      >? ? ? >? ? ? >? ? ? >? ? ? >   
>     <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>
>      >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>>
>      >? ? ? >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>
>      >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>>>
>      >? ? ? >? ? ? >? ? ? >? ?
<https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>
>      >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>>
>      >? ? ? >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>
>      >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>
>      >? ? <https://meet.google.com/cpu-eiue-hvk
>     <https://meet.google.com/cpu-eiue-hvk>>>>>>
>      >? ? ? >? ? ? >? ? ? >? ? ? >? ? Gluster-users mailing
list
>      >? ? ? >? ? ? >? ? ? >? ? ? > Gluster-users at
gluster.org
>     <mailto:Gluster-users at gluster.org>
>      >? ? <mailto:Gluster-users at gluster.org>
>      >? ? ? >? ? <mailto:Gluster-users at gluster.org>
>      >? ? ? >? ? ? >? ? <mailto:Gluster-users at
gluster.org>
>      >? ? ? >? ? <mailto:Gluster-users at gluster.org>
>      >? ? ? >? ? ? >? ? ? >? ? <mailto:Gluster-users at
gluster.org>
>      >? ? ? >? ? ? >? ? ? >? ? ? >
>      >? ? ? >
https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>>
>      >? ? ? >   
>     <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>    
<https://lists.gluster.org/mailman/listinfo/gluster-users>>>
>      >? ? ? >? ? ? >
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>>
>      >? ? ? >   
>     <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>    
<https://lists.gluster.org/mailman/listinfo/gluster-users>>>>
>      >? ? ? >? ? ? >? ? ? >
>      >? ? ? >   
>     <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>>
>      >? ? ? >   
>     <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>    
<https://lists.gluster.org/mailman/listinfo/gluster-users>>>
>      >? ? ? >? ? ? >
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>>
>      >? ? ? >   
>     <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>    
<https://lists.gluster.org/mailman/listinfo/gluster-users>>>>>
>      >? ? ? >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>>
>      >? ? ? >   
>     <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>    
<https://lists.gluster.org/mailman/listinfo/gluster-users>>>
>      >? ? ? >? ? ? >
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>>
>      >? ? ? >   
>     <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>    
<https://lists.gluster.org/mailman/listinfo/gluster-users>>>>
>      >? ? ? >? ? ? >? ? ? >
>      >? ? ? >   
>     <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>>
>      >? ? ? >   
>     <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>    
<https://lists.gluster.org/mailman/listinfo/gluster-users>>>
>      >? ? ? >? ? ? >
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>>
>      >? ? ? >   
>     <https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>      >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users
>    
<https://lists.gluster.org/mailman/listinfo/gluster-users>>>>>>
>      >? ? ? >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? ? >? ? --
>      >? ? ? >? ? ? >? ? ? >? ? Diego Zuccato
>      >? ? ? >? ? ? >? ? ? >? ? DIFA - Dip. di Fisica e
Astronomia
>      >? ? ? >? ? ? >? ? ? >? ? Servizi Informatici
>      >? ? ? >? ? ? >? ? ? >? ? Alma Mater Studiorum -
Universit? di Bologna
>      >? ? ? >? ? ? >? ? ? >? ? V.le Berti-Pichat 6/2 - 40127
Bologna - Italy
>      >? ? ? >? ? ? >? ? ? >? ? tel.: +39 051 20 95786
>      >? ? ? >? ? ? >? ? ? >
>      >? ? ? >? ? ? >
>      >? ? ? >? ? ? >? ? --
>      >? ? ? >? ? ? >? ? Diego Zuccato
>      >? ? ? >? ? ? >? ? DIFA - Dip. di Fisica e Astronomia
>      >? ? ? >? ? ? >? ? Servizi Informatici
>      >? ? ? >? ? ? >? ? Alma Mater Studiorum - Universit? di
Bologna
>      >? ? ? >? ? ? >? ? V.le Berti-Pichat 6/2 - 40127 Bologna -
Italy
>      >? ? ? >? ? ? >? ? tel.: +39 051 20 95786
>      >? ? ? >? ? ? >
>      >? ? ? >
>      >? ? ? >? ? --
>      >? ? ? >? ? Diego Zuccato
>      >? ? ? >? ? DIFA - Dip. di Fisica e Astronomia
>      >? ? ? >? ? Servizi Informatici
>      >? ? ? >? ? Alma Mater Studiorum - Universit? di Bologna
>      >? ? ? >? ? V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
>      >? ? ? >? ? tel.: +39 051 20 95786
>      >? ? ? >
>      >
>      >? ? --
>      >? ? Diego Zuccato
>      >? ? DIFA - Dip. di Fisica e Astronomia
>      >? ? Servizi Informatici
>      >? ? Alma Mater Studiorum - Universit? di Bologna
>      >? ? V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
>      >? ? tel.: +39 051 20 95786
>      >
> 
>     -- 
>     Diego Zuccato
>     DIFA - Dip. di Fisica e Astronomia
>     Servizi Informatici
>     Alma Mater Studiorum - Universit? di Bologna
>     V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
>     tel.: +39 051 20 95786
> 
-- 
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Universit? di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786

Seemingly Similar Threads

Search for more seemingly similar threads

Gluster users - Apr 2023 - How to configure?

[Gluster-users] How to configure?

[Gluster-users] How to configure?

Seemingly Similar Threads