But when I'm inside of VM, I'm getting 50-53 MB/s for replica and 120 MB for distributed volumes... whats the logic? that slow write speed directly to mounted gfs volume slows down backup and restore process of VMs by VE. 2015-07-18 22:47 GMT+03:00 Roman <romeo.r at gmail.com>:> Hi, > > Looked a lot from forums, but didn't find much.. > So I've got glusterfs 3.6.4 > 1 Gbps network > replicate and distributed volumes > > so when i read from any of them, I get maxed-out network performance > (around 80-120 MB/sec) > If i write to distributed volume, I get 16 MB/sec > If i write to replicated volume, I get around half of it (which is > logical) 8-10MB/sec > > i use XFS (with all set recommended options) and EXT4 for gluster. no > difference. > Disks are in HW RAID5 for Distributed volume and RAID0 for Replicated > volume > > volume options: > > performance.write-behind: off > server.allow-insecure: on > cluster.server-quorum-type: server > cluster.quorum-type: auto > network.remote-dio: enable > cluster.eager-lock: enable > performance.stat-prefetch: off > performance.io-cache: off > performance.read-ahead: off > performance.quick-read: off > > Am I missing something? > > -- > Best regards, > Roman. >-- Best regards, Roman. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150718/b4bfc3ed/attachment.html>
Hello, you can try tuning your TCP settings to be more aggressives (both on 1 client and all gluster bricks) in /etc/systctl.conf: net.core.rmem_max=67108864 net.core.wmem_max=67108864 # increase Linux autotuning TCP buffer limit to 32MB net.ipv4.tcp_rmem="4096 87380 33554432" net.ipv4.tcp_wmem="4096 65536 33554432" # increase the length of the processor input queue net.core.netdev_max_backlog=30000 # recommended default congestion control is htcp net.ipv4.tcp_congestion_control=htcp Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-07-18 22:11 GMT+02:00 Roman <romeo.r at gmail.com>:> But when I'm inside of VM, I'm getting 50-53 MB/s for replica and 120 MB > for distributed volumes... > whats the logic? that slow write speed directly to mounted gfs volume > slows down backup and restore process of VMs by VE. > > 2015-07-18 22:47 GMT+03:00 Roman <romeo.r at gmail.com>: > >> Hi, >> >> Looked a lot from forums, but didn't find much.. >> So I've got glusterfs 3.6.4 >> 1 Gbps network >> replicate and distributed volumes >> >> so when i read from any of them, I get maxed-out network performance >> (around 80-120 MB/sec) >> If i write to distributed volume, I get 16 MB/sec >> If i write to replicated volume, I get around half of it (which is >> logical) 8-10MB/sec >> >> i use XFS (with all set recommended options) and EXT4 for gluster. no >> difference. >> Disks are in HW RAID5 for Distributed volume and RAID0 for Replicated >> volume >> >> volume options: >> >> performance.write-behind: off >> server.allow-insecure: on >> cluster.server-quorum-type: server >> cluster.quorum-type: auto >> network.remote-dio: enable >> cluster.eager-lock: enable >> performance.stat-prefetch: off >> performance.io-cache: off >> performance.read-ahead: off >> performance.quick-read: off >> >> Am I missing something? >> >> -- >> Best regards, >> Roman. >> > > > > -- > Best regards, > Roman. > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150718/d5e3534a/attachment.html>
These numbers are not helpfull without context : - Raid level - Number of disks - nature & size of files What do you get for this sample? dd if=/dev/zero of=sample.txt bs=50M count=1 Test it accross differents cases to get a baseline: - Local disk on client - Locak disk on each brick - through gluster from client Once you have your baseline, you can start to tune system and see changes based on it. Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-07-18 22:11 GMT+02:00 Roman <romeo.r at gmail.com>:> But when I'm inside of VM, I'm getting 50-53 MB/s for replica and 120 MB > for distributed volumes... > whats the logic? that slow write speed directly to mounted gfs volume > slows down backup and restore process of VMs by VE. > > 2015-07-18 22:47 GMT+03:00 Roman <romeo.r at gmail.com>: > >> Hi, >> >> Looked a lot from forums, but didn't find much.. >> So I've got glusterfs 3.6.4 >> 1 Gbps network >> replicate and distributed volumes >> >> so when i read from any of them, I get maxed-out network performance >> (around 80-120 MB/sec) >> If i write to distributed volume, I get 16 MB/sec >> If i write to replicated volume, I get around half of it (which is >> logical) 8-10MB/sec >> >> i use XFS (with all set recommended options) and EXT4 for gluster. no >> difference. >> Disks are in HW RAID5 for Distributed volume and RAID0 for Replicated >> volume >> >> volume options: >> >> performance.write-behind: off >> server.allow-insecure: on >> cluster.server-quorum-type: server >> cluster.quorum-type: auto >> network.remote-dio: enable >> cluster.eager-lock: enable >> performance.stat-prefetch: off >> performance.io-cache: off >> performance.read-ahead: off >> performance.quick-read: off >> >> Am I missing something? >> >> -- >> Best regards, >> Roman. >> > > > > -- > Best regards, > Roman. > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150718/5d728f88/attachment.html>