Felix Kölzow
2020-May-27 10:44 UTC
[Gluster-users] Readdirp (ls -l) Performance Improvement
Dear Rafi KC, lets suppose I going to spend some time for testing. How would I install glusterfs-server including your feature? Maybe this is an easy procedure, but actually I am not familiar with it. Regards, Felix On 27/05/2020 07:56, RAFI KC wrote:> > Hi All, > > I have been working on POC to improve readdirp performance > improvement. At the end of the experiment, The results are showing > promising result in performance, overall there is a 104% improvement > for full filesystem crawl compared to the existing solution. Here is > the short test numbers. The tests were carried out in 16*3 setup with > 1.5 Million dentries (Both files and dir). The system also contains > some empty directories. *In the result the proposed solution is 287% > faster than the plane volume and 104% faster than the parallel-readdir > based solution.* > > * > * > > Configuration > > > > Plain volume > > > > Parallel-readdir > > > > Proposed Solution > > FS Crawl Time in Seconds > > > > 16497.523 > > > > 8717.872 > > > > 4261.401 > > In short, the basic idea behind the proposal is the efficient managing > of readdir buffer in gluster along with prefetching the dentries for > intelligent switch-over to the next buffer. The detailed problem > description, deign description and results are available in the > doc.https://docs.google.com/document/d/10z4T5Sd_-wCFrmDrzyQtlWOGLang1_g17wO8VUxSiJ8/edit > > > > If anybody can help with the testing on a different kind of workloads, > I would be very happy to assist. If wanted to test the patch and run a > performance test on your setup, I could help with back-porting the > patch to the version of your choice. > > > https://review.gluster.org/24469 > > https://review.gluster.org/24470 > > > Regards > > Rafi KC > > > ________ > > > > Community Meeting Calendar: > > Schedule - > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > Bridge: https://bluejeans.com/441850968 > > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200527/1b84a2ce/attachment.html>
Hi Felix, If you already have a test cluster, please give me the gluster version. I will try to build rpm then you can just do an upgrade. If you don't have a test cluster, and you are creating a new one, then I will build a new rpm against the latest head. This is more easier to me as I don't have to backport to a previous version. May I ask you the access protocol that you use (FUSE, SMB , NFS, gfapi). I did the testing on a fuse protocol. Rafi KC On 27/05/20 4:14 pm, Felix K?lzow wrote:> > Dear Rafi KC, > > > lets suppose I going to spend some time for testing. How would I > install glusterfs-server including your feature? > > Maybe this is an easy procedure, but actually I am not familiar with it. > > Regards, > > Felix > > On 27/05/2020 07:56, RAFI KC wrote: >> >> Hi All, >> >> I have been working on POC to improve readdirp performance >> improvement. At the end of the experiment, The results are showing >> promising result in performance, overall there is a 104% improvement >> for full filesystem crawl compared to the existing solution. Here is >> the short test numbers. The tests were carried out in 16*3 setup with >> 1.5 Million dentries (Both files and dir). The system also contains >> some empty directories. *In the result the proposed solution is 287% >> faster than the plane volume and 104% faster than the >> parallel-readdir based solution.* >> >> * >> * >> >> Configuration >> >> >> >> Plain volume >> >> >> >> Parallel-readdir >> >> >> >> Proposed Solution >> >> FS Crawl Time in Seconds >> >> >> >> 16497.523 >> >> >> >> 8717.872 >> >> >> >> 4261.401 >> >> In short, the basic idea behind the proposal is the efficient >> managing of readdir buffer in gluster along with prefetching the >> dentries for intelligent switch-over to the next buffer. The detailed >> problem description, deign description and results are available in >> the >> doc.https://docs.google.com/document/d/10z4T5Sd_-wCFrmDrzyQtlWOGLang1_g17wO8VUxSiJ8/edit >> >> >> >> If anybody can help with the testing on a different kind of >> workloads, I would be very happy to assist. If wanted to test the >> patch and run a performance test on your setup, I could help with >> back-porting the patch to the version of your choice. >> >> >> https://review.gluster.org/24469 >> >> https://review.gluster.org/24470 >> >> >> Regards >> >> Rafi KC >> >> >> ________ >> >> >> >> Community Meeting Calendar: >> >> Schedule - >> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC >> Bridge:https://bluejeans.com/441850968 >> >> Gluster-users mailing list >> Gluster-users at gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users > > ________ > > > > Community Meeting Calendar: > > Schedule - > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > Bridge: https://bluejeans.com/441850968 > > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200527/6317edc5/attachment.html>