David Cunningham
2019-Dec-17 21:19 UTC
[Gluster-users] GFS performance under heavy traffic
Hello, We switched a production system to using GFS instead of NFS at the weekend, however it didn't go well on Monday when full load hit. The application started crashing regularly and we had to revert to NFS. It seems that the problem was high network traffic used by GFS. We've two GFS nodes plus one arbiter node, each about 1.3ms latency from each other. Attached is a chart of network traffic on one of the GFS nodes. We see that it saturated the 1Gbps link before we reverted to NFS at 15:10. The question is, why does GFS use so much network traffic and is there anything we can do about it? NFS traffic doesn't exceed 4MBps, so 120MBps for GFS seems awfully high. It would also be good to have faster read performance from GFS, but that's another issue. Thanks in advance for any assistance. -- David Cunningham, Voisonics Limited http://voisonics.com/ USA: +1 213 221 1092 New Zealand: +64 (0)28 2558 3782 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20191218/4105ca86/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: traffic.png Type: image/png Size: 59079 bytes Desc: not available URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20191218/4105ca86/attachment.png>
Raghavendra Gowdappa
2019-Dec-18 03:39 UTC
[Gluster-users] GFS performance under heavy traffic
What version of Glusterfs are you using? Though, not sure what's the root cause of your problem, just wanted to point out a bug with read-ahead which would cause read-amplification over network [1][2], which should be fixed in recent versions. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1214489 [2] https://bugzilla.redhat.com/show_bug.cgi?id=1393419 On Wed, Dec 18, 2019 at 2:50 AM David Cunningham <dcunningham at voisonics.com> wrote:> Hello, > > We switched a production system to using GFS instead of NFS at the > weekend, however it didn't go well on Monday when full load hit. The > application started crashing regularly and we had to revert to NFS. It > seems that the problem was high network traffic used by GFS. > > We've two GFS nodes plus one arbiter node, each about 1.3ms latency from > each other. Attached is a chart of network traffic on one of the GFS nodes. > We see that it saturated the 1Gbps link before we reverted to NFS at 15:10. > > The question is, why does GFS use so much network traffic and is there > anything we can do about it? NFS traffic doesn't exceed 4MBps, so 120MBps > for GFS seems awfully high. > > It would also be good to have faster read performance from GFS, but that's > another issue. > > Thanks in advance for any assistance. > > -- > David Cunningham, Voisonics Limited > http://voisonics.com/ > USA: +1 213 221 1092 > New Zealand: +64 (0)28 2558 3782 > ________ > > Community Meeting Calendar: > > APAC Schedule - > Every 2nd and 4th Tuesday at 11:30 AM IST > Bridge: https://bluejeans.com/441850968 > > NA/EMEA Schedule - > Every 1st and 3rd Tuesday at 01:00 PM EDT > Bridge: https://bluejeans.com/441850968 > > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20191218/80fbf783/attachment.html>
Hi David, I guess you are using FUSE mounts?Does you app is capable of using libgfapi ? Best Regards,Strahil Nikolov Hello, We switched a production system to using GFS instead of NFS at the weekend, however it didn't go well on Monday when full load hit. The application started crashing regularly and we had to revert to NFS. It seems that the problem was high network traffic used by GFS. We've two GFS nodes plus one arbiter node, each about 1.3ms latency from each other. Attached is a chart of network traffic on one of the GFS nodes. We see that it saturated the 1Gbps link before we reverted to NFS at 15:10. The question is, why does GFS use so much network traffic and is there anything we can do about it? NFS traffic doesn't exceed 4MBps, so 120MBps for GFS seems awfully high. It would also be good to have faster read performance from GFS, but that's another issue. Thanks in advance for any assistance. -- David Cunningham, Voisonics Limited http://voisonics.com/ USA: +1 213 221 1092 New Zealand: +64 (0)28 2558 3782________ Community Meeting Calendar: APAC Schedule - Every 2nd and 4th Tuesday at 11:30 AM IST Bridge: https://bluejeans.com/441850968 NA/EMEA Schedule - Every 1st and 3rd Tuesday at 01:00 PM EDT Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users at gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20191218/c824ff3b/attachment.html>