Hi Felix, If you already have a test cluster, please give me the gluster version. I will try to build rpm then you can just do an upgrade. If you don't have a test cluster, and you are creating a new one, then I will build a new rpm against the latest head. This is more easier to me as I don't have to backport to a previous version. May I ask you the access protocol that you use (FUSE, SMB , NFS, gfapi). I did the testing on a fuse protocol. Rafi KC On 27/05/20 4:14 pm, Felix K?lzow wrote:> > Dear Rafi KC, > > > lets suppose I going to spend some time for testing. How would I > install glusterfs-server including your feature? > > Maybe this is an easy procedure, but actually I am not familiar with it. > > Regards, > > Felix > > On 27/05/2020 07:56, RAFI KC wrote: >> >> Hi All, >> >> I have been working on POC to improve readdirp performance >> improvement. At the end of the experiment, The results are showing >> promising result in performance, overall there is a 104% improvement >> for full filesystem crawl compared to the existing solution. Here is >> the short test numbers. The tests were carried out in 16*3 setup with >> 1.5 Million dentries (Both files and dir). The system also contains >> some empty directories. *In the result the proposed solution is 287% >> faster than the plane volume and 104% faster than the >> parallel-readdir based solution.* >> >> * >> * >> >> Configuration >> >> >> >> Plain volume >> >> >> >> Parallel-readdir >> >> >> >> Proposed Solution >> >> FS Crawl Time in Seconds >> >> >> >> 16497.523 >> >> >> >> 8717.872 >> >> >> >> 4261.401 >> >> In short, the basic idea behind the proposal is the efficient >> managing of readdir buffer in gluster along with prefetching the >> dentries for intelligent switch-over to the next buffer. The detailed >> problem description, deign description and results are available in >> the >> doc.https://docs.google.com/document/d/10z4T5Sd_-wCFrmDrzyQtlWOGLang1_g17wO8VUxSiJ8/edit >> >> >> >> If anybody can help with the testing on a different kind of >> workloads, I would be very happy to assist. If wanted to test the >> patch and run a performance test on your setup, I could help with >> back-porting the patch to the version of your choice. >> >> >> https://review.gluster.org/24469 >> >> https://review.gluster.org/24470 >> >> >> Regards >> >> Rafi KC >> >> >> ________ >> >> >> >> Community Meeting Calendar: >> >> Schedule - >> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC >> Bridge:https://bluejeans.com/441850968 >> >> Gluster-users mailing list >> Gluster-users at gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users > > ________ > > > > Community Meeting Calendar: > > Schedule - > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > Bridge: https://bluejeans.com/441850968 > > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200527/6317edc5/attachment.html>
Strahil Nikolov
2020-May-27 12:19 UTC
[Gluster-users] Readdirp (ls -l) Performance Improvement
Hi Rafi, I have a test oVirt 4.3.9 cluster with Gluster v7.5 on CentOS7. Can you provide the rpms and I will try to test. Also, please share the switch that disables this behaviour (in case something goes wrong). Best Regards, Strahil Nikolov ?? 27 ??? 2020 ?. 14:54:34 GMT+03:00, RAFI KC <rkavunga at redhat.com> ??????:>Hi Felix, > >If you already have a test cluster, please give me the gluster version. > >I will try to build rpm then you can just do an upgrade. If you don't >have a test cluster, and you are creating a new one, then I will build >a >new rpm against the latest head. This is more easier to me as I don't >have to backport to a previous version. > >May I ask you the access protocol that you use (FUSE, SMB , NFS, >gfapi). >I did the testing on a fuse protocol. > > >Rafi KC > >On 27/05/20 4:14 pm, Felix K?lzow wrote: >> >> Dear Rafi KC, >> >> >> lets suppose I going to spend some time for testing. How would I >> install glusterfs-server including your feature? >> >> Maybe this is an easy procedure, but actually I am not familiar with >it. >> >> Regards, >> >> Felix >> >> On 27/05/2020 07:56, RAFI KC wrote: >>> >>> Hi All, >>> >>> I have been working on POC to improve readdirp performance >>> improvement. At the end of the experiment, The results are showing >>> promising result in performance, overall there is a 104% improvement > >>> for full filesystem crawl compared to the existing solution. Here is > >>> the short test numbers. The tests were carried out in 16*3 setup >with >>> 1.5 Million dentries (Both files and dir). The system also contains >>> some empty directories. *In the result the proposed solution is 287% > >>> faster than the plane volume and 104% faster than the >>> parallel-readdir based solution.* >>> >>> * >>> * >>> >>> Configuration >>> >>> >>> >>> Plain volume >>> >>> >>> >>> Parallel-readdir >>> >>> >>> >>> Proposed Solution >>> >>> FS Crawl Time in Seconds >>> >>> >>> >>> 16497.523 >>> >>> >>> >>> 8717.872 >>> >>> >>> >>> 4261.401 >>> >>> In short, the basic idea behind the proposal is the efficient >>> managing of readdir buffer in gluster along with prefetching the >>> dentries for intelligent switch-over to the next buffer. The >detailed >>> problem description, deign description and results are available in >>> the >>> >doc.https://docs.google.com/document/d/10z4T5Sd_-wCFrmDrzyQtlWOGLang1_g17wO8VUxSiJ8/edit > >>> >>> >>> >>> If anybody can help with the testing on a different kind of >>> workloads, I would be very happy to assist. If wanted to test the >>> patch and run a performance test on your setup, I could help with >>> back-porting the patch to the version of your choice. >>> >>> >>> https://review.gluster.org/24469 >>> >>> https://review.gluster.org/24470 >>> >>> >>> Regards >>> >>> Rafi KC >>> >>> >>> ________ >>> >>> >>> >>> Community Meeting Calendar: >>> >>> Schedule - >>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC >>> Bridge:https://bluejeans.com/441850968 >>> >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> https://lists.gluster.org/mailman/listinfo/gluster-users >> >> ________ >> >> >> >> Community Meeting Calendar: >> >> Schedule - >> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC >> Bridge: https://bluejeans.com/441850968 >> >> Gluster-users mailing list >> Gluster-users at gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users