Gluster users - Jul 2020 - Gluster linear scale-out performance

Hi,

I am trying to test Gluster linear scale-out performance by adding more
storage server/bricks, and measure the storage I/O performance. To vary the
storage server number, I create several "stripe" volumes that contain
2
brick servers, 3 brick servers, 4 brick servers, and so on. On gluster
client side, I used "dd if=/dev/zero of=/mnt/glusterfs/dns_test_data_26g
bs=1M count=26000" to create 26G data (or larger size), and those data will
be distributed to the corresponding gluster servers (each has gluster brick
on it) and "dd" returns the final I/O throughput. The Internet is 40G
infiniband, although I didn't do any specific configurations to use
advanced features.

What confuses me is that the storage I/O seems not to relate to the number
of storage nodes, but Gluster documents said it should be linear scaling.
For example, when "write-behind" is on, and when Infiniband
"jumbo frame"
(connected mode) is on, I can get ~800 MB/sec reported by "dd", no
matter I
have 2 brick servers or 8 brick servers -- for 2 server case, each server
can have ~400 MB/sec; for 4 server case, each server can have ~200MB/sec.
That said, each server I/O does aggregate to the final storage I/O (800
MB/sec), but this is not "linear scale-out".

Can somebody help me to understand why this is the case? I certainly can
have some misunderstanding/misconfiguration here. Please correct me if I
do, thanks!

Best,
Qing
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20200721/22f376c9/attachment.html>

On Tue, 21 Jul 2020, 21:30 Qing Wang <qw at g.clemson.edu> wrote:
> Hi,
>
> I am trying to test Gluster linear scale-out performance by adding more
> storage server/bricks, and measure the storage I/O performance. To vary the
> storage server number, I create several "stripe" volumes that
contain 2
> brick servers, 3 brick servers, 4 brick servers, and so on. On gluster
> client side, I used "dd if=/dev/zero
of=/mnt/glusterfs/dns_test_data_26g
> bs=1M count=26000" to create 26G data (or larger size), and those data
will
> be distributed to the corresponding gluster servers (each has gluster brick
> on it) and "dd" returns the final I/O throughput. The Internet is
40G
> infiniband, although I didn't do any specific configurations to use
> advanced features.
>
Your dd command is inaccurate, as it'll hit the client cache. It is also
single threaded. I suggest switching to fio.
Y.

> What confuses me is that the storage I/O seems not to relate to the number
> of storage nodes, but Gluster documents said it should be linear scaling.
> For example, when "write-behind" is on, and when Infiniband
"jumbo frame"
> (connected mode) is on, I can get ~800 MB/sec reported by "dd",
no matter I
> have 2 brick servers or 8 brick servers -- for 2 server case, each server
> can have ~400 MB/sec; for 4 server case, each server can have ~200MB/sec.
> That said, each server I/O does aggregate to the final storage I/O (800
> MB/sec), but this is not "linear scale-out".
>
> Can somebody help me to understand why this is the case? I certainly can
> have some misunderstanding/misconfiguration here. Please correct me if I
> do, thanks!
>
> Best,
> Qing
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20200721/1524ac67/attachment.html>