venky evr
2020-Jan-16 09:10 UTC
[Gluster-users] glusterfs performance issue with fio fdatasync
Description of problem:
I am getting bad performance with glusterfs 7 with fio.
Any help is greatly appreicated.
With glusterfs :
1 node
1 brick
1 volume (ssd volume) formated with ext4
volume created and mounted with default values.
fio --rw=write --ioengine=sync --fdatasync=1 --directory=/mnt/1
--size=64m --bs=1024 --name=mytest
fio-3.16
Starting 1 process
mytest2: Laying out IO file (1 file / 64MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=2759KiB/s][w=2759 IOPS][eta 00m:00s]
mytest2: (groupid=0, jobs=1): err= 0: pid=11970: Thu Jan 16 13:06:50 2020
write: IOPS=2632, BW=2633KiB/s (2696kB/s)(64.0MiB/24892msec)
clat (usec): min=50, max=1433, avg=61.43, stdev=11.78
lat (usec): min=50, max=1435, avg=61.68, stdev=11.82
clat percentiles (usec):
| 1.00th=[ 54], 5.00th=[ 55], 10.00th=[ 55], 20.00th=[ 56],
| 30.00th=[ 56], 40.00th=[ 57], 50.00th=[ 58], 60.00th=[ 59],
| 70.00th=[ 64], 80.00th=[ 69], 90.00th=[ 72], 95.00th=[ 79],
| 99.00th=[ 103], 99.50th=[ 121], 99.90th=[ 133], 99.95th=[ 137],
| 99.99th=[ 155]
bw ( KiB/s): min= 1834, max= 2752, per=99.94%, avg=2630.37,
stdev=146.81, samples=49
iops : min= 1834, max= 2752, avg=2630.37, stdev=146.81, samples=49
lat (usec) : 100=98.72%, 250=1.28%, 750=0.01%
lat (msec) : 2=0.01%
fsync/fdatasync/sync_file_range:
sync (usec): min=245, max=13622, avg=315.54, stdev=84.93
sync percentiles (usec):
| 1.00th=[ 255], 5.00th=[ 262], 10.00th=[ 265], 20.00th=[ 269],
| 30.00th=[ 273], 40.00th=[ 277], 50.00th=[ 281], 60.00th=[ 289],
| 70.00th=[ 310], 80.00th=[ 400], 90.00th=[ 416], 95.00th=[ 433],
| 99.00th=[ 486], 99.50th=[ 519], 99.90th=[ 676], 99.95th=[ 848],
| 99.99th=[ 914]
cpu : usr=1.41%, sys=3.50%, ctx=196761, majf=0, minf=13
IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwts: total=0,65536,0,0 short=65535,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=2633KiB/s (2696kB/s), 2633KiB/s-2633KiB/s
(2696kB/s-2696kB/s), io=64.0MiB (67.1MB), run=24892-24892msec
Without Glusterfs:
Ext4 filesystem created and mounted locally, there is no gluster
involved during this test
mount /dev/sdb1 /mnt/1
Starting 1 process
mytest: Laying out IO file (1 file / 64MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=10.2MiB/s][w=10.5k IOPS][eta 00m:00s]
mytest: (groupid=0, jobs=1): err= 0: pid=9308: Thu Jan 16 13:00:20 2020
write: IOPS=10.2k, BW=9.99MiB/s (10.5MB/s)(64.0MiB/6407msec)
clat (nsec): min=1257, max=831943, avg=2674.65, stdev=10327.82
lat (nsec): min=1323, max=832038, avg=2783.73, stdev=10330.58
clat percentiles (nsec):
| 1.00th=[ 1352], 5.00th=[ 1400], 10.00th=[ 1448], 20.00th=[ 1496],
| 30.00th=[ 1560], 40.00th=[ 1656], 50.00th=[ 2320], 60.00th=[ 2448],
| 70.00th=[ 2864], 80.00th=[ 2992], 90.00th=[ 3280], 95.00th=[ 4384],
| 99.00th=[ 8896], 99.50th=[ 11328], 99.90th=[ 20608], 99.95th=[222208],
| 99.99th=[464896]
bw ( KiB/s): min= 8946, max=10536, per=99.92%, avg=10219.75,
stdev=441.53, samples=12
iops : min= 8946, max=10536, avg=10219.75, stdev=441.53, samples=12
lat (usec) : 2=42.86%, 4=50.26%, 10=6.19%, 20=0.57%, 50=0.06%
lat (usec) : 100=0.01%, 250=0.01%, 500=0.04%, 750=0.01%, 1000=0.01%
fsync/fdatasync/sync_file_range:
sync (usec): min=50, max=1174, avg=93.61, stdev=62.99
sync percentiles (usec):
| 1.00th=[ 52], 5.00th=[ 53], 10.00th=[ 54], 20.00th=[ 55],
| 30.00th=[ 55], 40.00th=[ 56], 50.00th=[ 57], 60.00th=[ 61],
| 70.00th=[ 82], 80.00th=[ 178], 90.00th=[ 194], 95.00th=[ 215],
| 99.00th=[ 253], 99.50th=[ 265], 99.90th=[ 334], 99.95th=[ 627],
| 99.99th=[ 758]
cpu : usr=2.68%, sys=16.98%, ctx=81932, majf=0, minf=16
IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwts: total=0,65536,0,0 short=65535,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=9.99MiB/s (10.5MB/s), 9.99MiB/s-9.99MiB/s
(10.5MB/s-10.5MB/s), io=64.0MiB (67.1MB), run=6407-6407msec
Disk stats (read/write):
sdb: ios=0/98184, merge=0/16369, ticks=0/4804, in_queue=4788, util=73.71%
we tried following tuning parameters, but performance didnt increase.
performance. Client-io-threads on
performance.io-thread-count 64
server. Event-threads 4 or 8
client. Event-threads 4 or 8
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20200116/69d8126a/attachment.html>
Strahil Nikolov
2020-Jan-16 19:43 UTC
[Gluster-users] glusterfs performance issue with fio fdatasync
On January 16, 2020 11:10:14 AM GMT+02:00, venky evr <venky.evr at gmail.com> wrote:>Description of problem: > >I am getting bad performance with glusterfs 7 with fio. >Any help is greatly appreicated. > >With glusterfs : > >1 node >1 brick >1 volume (ssd volume) formated with ext4 > >volume created and mounted with default values. > > > >fio --rw=write --ioengine=sync --fdatasync=1 --directory=/mnt/1 >--size=64m --bs=1024 --name=mytest > > > >fio-3.16 >Starting 1 process >mytest2: Laying out IO file (1 file / 64MiB) >Jobs: 1 (f=1): [W(1)][100.0%][w=2759KiB/s][w=2759 IOPS][eta 00m:00s] >mytest2: (groupid=0, jobs=1): err= 0: pid=11970: Thu Jan 16 13:06:50 >2020 > write: IOPS=2632, BW=2633KiB/s (2696kB/s)(64.0MiB/24892msec) > clat (usec): min=50, max=1433, avg=61.43, stdev=11.78 > lat (usec): min=50, max=1435, avg=61.68, stdev=11.82 > clat percentiles (usec): > | 1.00th=[ 54], 5.00th=[ 55], 10.00th=[ 55], 20.00th=[ 56], > | 30.00th=[ 56], 40.00th=[ 57], 50.00th=[ 58], 60.00th=[ 59], > | 70.00th=[ 64], 80.00th=[ 69], 90.00th=[ 72], 95.00th=[ 79], > | 99.00th=[ 103], 99.50th=[ 121], 99.90th=[ 133], 99.95th=[ 137], > | 99.99th=[ 155] > bw ( KiB/s): min= 1834, max= 2752, per=99.94%, avg=2630.37, >stdev=146.81, samples=49 >iops : min= 1834, max= 2752, avg=2630.37, stdev=146.81, >samples=49 > lat (usec) : 100=98.72%, 250=1.28%, 750=0.01% > lat (msec) : 2=0.01% > fsync/fdatasync/sync_file_range: > sync (usec): min=245, max=13622, avg=315.54, stdev=84.93 > sync percentiles (usec): > | 1.00th=[ 255], 5.00th=[ 262], 10.00th=[ 265], 20.00th=[ 269], > | 30.00th=[ 273], 40.00th=[ 277], 50.00th=[ 281], 60.00th=[ 289], > | 70.00th=[ 310], 80.00th=[ 400], 90.00th=[ 416], 95.00th=[ 433], > | 99.00th=[ 486], 99.50th=[ 519], 99.90th=[ 676], 99.95th=[ 848], > | 99.99th=[ 914] > cpu : usr=1.41%, sys=3.50%, ctx=196761, majf=0, minf=13 >IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >>=64=0.0% >submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >>=64=0.0% >complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >>=64=0.0% > issued rwts: total=0,65536,0,0 short=65535,0,0,0 dropped=0,0,0,0 > latency : target=0, window=0, percentile=100.00%, depth=1 > >Run status group 0 (all jobs): > WRITE: bw=2633KiB/s (2696kB/s), 2633KiB/s-2633KiB/s >(2696kB/s-2696kB/s), io=64.0MiB (67.1MB), run=24892-24892msec > > >Without Glusterfs: > > >Ext4 filesystem created and mounted locally, there is no gluster >involved during this test > > mount /dev/sdb1 /mnt/1 > > >Starting 1 process >mytest: Laying out IO file (1 file / 64MiB) >Jobs: 1 (f=1): [W(1)][100.0%][w=10.2MiB/s][w=10.5k IOPS][eta 00m:00s] >mytest: (groupid=0, jobs=1): err= 0: pid=9308: Thu Jan 16 13:00:20 2020 > write: IOPS=10.2k, BW=9.99MiB/s (10.5MB/s)(64.0MiB/6407msec) > clat (nsec): min=1257, max=831943, avg=2674.65, stdev=10327.82 > lat (nsec): min=1323, max=832038, avg=2783.73, stdev=10330.58 > clat percentiles (nsec): >| 1.00th=[ 1352], 5.00th=[ 1400], 10.00th=[ 1448], 20.00th=[ >1496], >| 30.00th=[ 1560], 40.00th=[ 1656], 50.00th=[ 2320], 60.00th=[ >2448], >| 70.00th=[ 2864], 80.00th=[ 2992], 90.00th=[ 3280], 95.00th=[ >4384], >| 99.00th=[ 8896], 99.50th=[ 11328], 99.90th=[ 20608], >99.95th=[222208], > | 99.99th=[464896] > bw ( KiB/s): min= 8946, max=10536, per=99.92%, avg=10219.75, >stdev=441.53, samples=12 >iops : min= 8946, max=10536, avg=10219.75, stdev=441.53, >samples=12 > lat (usec) : 2=42.86%, 4=50.26%, 10=6.19%, 20=0.57%, 50=0.06% > lat (usec) : 100=0.01%, 250=0.01%, 500=0.04%, 750=0.01%, 1000=0.01% > fsync/fdatasync/sync_file_range: > sync (usec): min=50, max=1174, avg=93.61, stdev=62.99 > sync percentiles (usec): > | 1.00th=[ 52], 5.00th=[ 53], 10.00th=[ 54], 20.00th=[ 55], > | 30.00th=[ 55], 40.00th=[ 56], 50.00th=[ 57], 60.00th=[ 61], > | 70.00th=[ 82], 80.00th=[ 178], 90.00th=[ 194], 95.00th=[ 215], > | 99.00th=[ 253], 99.50th=[ 265], 99.90th=[ 334], 99.95th=[ 627], > | 99.99th=[ 758] > cpu : usr=2.68%, sys=16.98%, ctx=81932, majf=0, minf=16 >IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >>=64=0.0% >submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >>=64=0.0% >complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >>=64=0.0% > issued rwts: total=0,65536,0,0 short=65535,0,0,0 dropped=0,0,0,0 > latency : target=0, window=0, percentile=100.00%, depth=1 > >Run status group 0 (all jobs): > WRITE: bw=9.99MiB/s (10.5MB/s), 9.99MiB/s-9.99MiB/s >(10.5MB/s-10.5MB/s), io=64.0MiB (67.1MB), run=6407-6407msec > >Disk stats (read/write): >sdb: ios=0/98184, merge=0/16369, ticks=0/4804, in_queue=4788, >util=73.71% > > > > > >we tried following tuning parameters, but performance didnt increase. > >performance. Client-io-threads on >performance.io-thread-count 64 >server. Event-threads 4 or 8 >client. Event-threads 4 or 8Something that come to my mind: 1. Switch to XFS 2. Set I/O scheduler of the brick to none/noop 3. Mount options of the brick : 'noatime' and if there is battery-backed cache 'nobarrier' 4. Set reasonable value for 'vm.dirty_background_bytes' & 'vm.dirty_bytes' to a reasonable values 5. Disable gluster logs or raise them at ERR or higher. Check if any of the described 'profiles' in /var/lib/gluster/groups' match your workload. Best Regards, Strahil Nikolov