Hi, I have a distributed and replica=3 volume (not to use stripe ) in a cluster. I used dd to write 120 files to test. I foundthe write performane of some files are much lower than others. all these "BAD" files are stored in the same three brick servers for replication (I called node1 node2 node3) e.g the bad write performance could be 10MBps while good performance could be 150Mbps+ there are no problems about raid and networks. If i stopped node1 & node2, the write performance of "BAD" files are the similar to (even better) GOOD ones. One thing I must metion is the raids of node1 and node2 are reformated for some reason, there are many self-heal activities to restore files in node1 and node2. Is the BAD write performance caused by aggresive self-heal? How could I slow down the self-heal? Any advise? -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140128/871c8875/attachment.html>
P.S. we use single 10000Mbps NIC. I found the networks bandwidth is not a issue. On Tue, Jan 28, 2014 at 4:49 PM, Mingfan Lu <mingfan.lu at gmail.com> wrote:> > Hi, > I have a distributed and replica=3 volume (not to use stripe ) in a > cluster. I used dd to write 120 files to test. I foundthe write performane > of some files are much lower than others. all these "BAD" files are stored > in the same three brick servers for replication (I called node1 node2 node3) > > e.g the bad write performance could be 10MBps while good performance could > be 150Mbps+ > > there are no problems about raid and networks. > If i stopped node1 & node2, the write performance of "BAD" files are the > similar to (even better) GOOD ones. > > One thing I must metion is the raids of node1 and node2 are reformated > for some reason, there are many self-heal activities to restore files in > node1 and node2. > Is the BAD write performance caused by aggresive self-heal? > How could I slow down the self-heal? > Any advise? > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140128/ec743d9e/attachment.html>
Is your write single or multi-threaded? If it's single threaded, try writing your files across as many threads as possible, and see what the performance improvement is like. -Dan ---------------- Dan Mons Skunk Works Cutting Edge http://cuttingedge.com.au On 28 January 2014 18:49, Mingfan Lu <mingfan.lu at gmail.com> wrote:> > Hi, > I have a distributed and replica=3 volume (not to use stripe ) in a > cluster. I used dd to write 120 files to test. I foundthe write performane > of some files are much lower than others. all these "BAD" files are stored > in the same three brick servers for replication (I called node1 node2 node3) > > e.g the bad write performance could be 10MBps while good performance could > be 150Mbps+ > > there are no problems about raid and networks. > If i stopped node1 & node2, the write performance of "BAD" files are the > similar to (even better) GOOD ones. > > One thing I must metion is the raids of node1 and node2 are reformated for > some reason, there are many self-heal activities to restore files in node1 > and node2. > Is the BAD write performance caused by aggresive self-heal? > How could I slow down the self-heal? > Any advise? > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-users