Pranith Kumar Karampuri
2017-Apr-24 10:17 UTC
[Gluster-users] slow write perf for disperse volume
We were able to saturate hardware with EC as well. Could you check 'top' in threaded mode to see if fuse thread is saturated when you run dd? On Mon, Apr 24, 2017 at 3:27 PM, Ingard Mev?g <ingard at jotta.no> wrote:> Hi > I've been playing with disperse volumes the past week, and so far i can > not get more than 12MB/s when i do a write test. I've tried a distributed > volume on the same bricks and gotten close to gigabit speeds. iperf > confirms gigabit speeds to all three servers in the storage pool. > > The three storage servers have 10gig nics (connected to the same switch). > The client is for a now a docker container in a 2nd DC (latency roughly 8-9 > ms). > > dpkg -l|grep -i gluster > ii glusterfs-client 3.10.1-ubuntu1~xenial1 amd64 > clustered file-system (client package) > ii glusterfs-common 3.10.1-ubuntu1~xenial1 amd64 > GlusterFS common libraries and translator modules > ii glusterfs-server 3.10.1-ubuntu1~xenial1 amd64 > clustered file-system (server package) > > $ gluster volume info > > Volume Name: DFS-ARCHIVE-001 > Type: Disperse > Volume ID: 1497bc85-cb47-4123-8f91-a07f55c11dcc > Status: Started > Snapshot Count: 0 > Number of Bricks: 1 x (4 + 2) = 6 > Transport-type: tcp > Bricks: > Brick1: dna-001:/mnt/data01/brick > Brick2: dna-001:/mnt/data02/brick > Brick3: dna-002:/mnt/data01/brick > Brick4: dna-002:/mnt/data02/brick > Brick5: dna-003:/mnt/data01/brick > Brick6: dna-003:/mnt/data02/brick > Options Reconfigured: > transport.address-family: inet > nfs.disable: on > > Anyone know the reason for the slow speeds on disperse vs distribute? > > kind regards > ingard > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users >-- Pranith -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170424/fc56e6da/attachment.html>
Pranith Kumar Karampuri
2017-Apr-24 10:17 UTC
[Gluster-users] slow write perf for disperse volume
On Mon, Apr 24, 2017 at 3:47 PM, Pranith Kumar Karampuri < pkarampu at redhat.com> wrote:> We were able to saturate hardware with EC as well. Could you check 'top' > in threaded mode to see if fuse thread is saturated when you run dd? >This is for mount process by the way.> > On Mon, Apr 24, 2017 at 3:27 PM, Ingard Mev?g <ingard at jotta.no> wrote: > >> Hi >> I've been playing with disperse volumes the past week, and so far i can >> not get more than 12MB/s when i do a write test. I've tried a distributed >> volume on the same bricks and gotten close to gigabit speeds. iperf >> confirms gigabit speeds to all three servers in the storage pool. >> >> The three storage servers have 10gig nics (connected to the same switch). >> The client is for a now a docker container in a 2nd DC (latency roughly 8-9 >> ms). >> >> dpkg -l|grep -i gluster >> ii glusterfs-client 3.10.1-ubuntu1~xenial1 amd64 >> clustered file-system (client package) >> ii glusterfs-common 3.10.1-ubuntu1~xenial1 amd64 >> GlusterFS common libraries and translator modules >> ii glusterfs-server 3.10.1-ubuntu1~xenial1 amd64 >> clustered file-system (server package) >> >> $ gluster volume info >> >> Volume Name: DFS-ARCHIVE-001 >> Type: Disperse >> Volume ID: 1497bc85-cb47-4123-8f91-a07f55c11dcc >> Status: Started >> Snapshot Count: 0 >> Number of Bricks: 1 x (4 + 2) = 6 >> Transport-type: tcp >> Bricks: >> Brick1: dna-001:/mnt/data01/brick >> Brick2: dna-001:/mnt/data02/brick >> Brick3: dna-002:/mnt/data01/brick >> Brick4: dna-002:/mnt/data02/brick >> Brick5: dna-003:/mnt/data01/brick >> Brick6: dna-003:/mnt/data02/brick >> Options Reconfigured: >> transport.address-family: inet >> nfs.disable: on >> >> Anyone know the reason for the slow speeds on disperse vs distribute? >> >> kind regards >> ingard >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://lists.gluster.org/mailman/listinfo/gluster-users >> > > > > -- > Pranith >-- Pranith -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170424/e8af8e1e/attachment.html>
Hi. I can't see a fuse thread at all. Please see attached screenshot of top process with threads. Keep in mind this is from inside the container. 2017-04-24 12:17 GMT+02:00 Pranith Kumar Karampuri <pkarampu at redhat.com>:> We were able to saturate hardware with EC as well. Could you check 'top' > in threaded mode to see if fuse thread is saturated when you run dd? > > On Mon, Apr 24, 2017 at 3:27 PM, Ingard Mev?g <ingard at jotta.no> wrote: > >> Hi >> I've been playing with disperse volumes the past week, and so far i can >> not get more than 12MB/s when i do a write test. I've tried a distributed >> volume on the same bricks and gotten close to gigabit speeds. iperf >> confirms gigabit speeds to all three servers in the storage pool. >> >> The three storage servers have 10gig nics (connected to the same switch). >> The client is for a now a docker container in a 2nd DC (latency roughly 8-9 >> ms). >> >> dpkg -l|grep -i gluster >> ii glusterfs-client 3.10.1-ubuntu1~xenial1 amd64 >> clustered file-system (client package) >> ii glusterfs-common 3.10.1-ubuntu1~xenial1 amd64 >> GlusterFS common libraries and translator modules >> ii glusterfs-server 3.10.1-ubuntu1~xenial1 amd64 >> clustered file-system (server package) >> >> $ gluster volume info >> >> Volume Name: DFS-ARCHIVE-001 >> Type: Disperse >> Volume ID: 1497bc85-cb47-4123-8f91-a07f55c11dcc >> Status: Started >> Snapshot Count: 0 >> Number of Bricks: 1 x (4 + 2) = 6 >> Transport-type: tcp >> Bricks: >> Brick1: dna-001:/mnt/data01/brick >> Brick2: dna-001:/mnt/data02/brick >> Brick3: dna-002:/mnt/data01/brick >> Brick4: dna-002:/mnt/data02/brick >> Brick5: dna-003:/mnt/data01/brick >> Brick6: dna-003:/mnt/data02/brick >> Options Reconfigured: >> transport.address-family: inet >> nfs.disable: on >> >> Anyone know the reason for the slow speeds on disperse vs distribute? >> >> kind regards >> ingard >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://lists.gluster.org/mailman/listinfo/gluster-users >> > > > > -- > Pranith >-- Ingard Mev?g Driftssjef Jottacloud Mobil: +47 450 22 834 E-post: ingard at jottacloud.com Webside: www.jottacloud.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170424/5ffef174/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2017-04-24 at 12.59.44.png Type: image/png Size: 115838 bytes Desc: not available URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170424/5ffef174/attachment.png>