Hi I have just moved our 400 TB HPC storage from lustre to gluster. It is part of a research institute and users have very small files to big files ( few KB to 20GB) . Our setup consists of 5 servers, each with 96TB RAID 6 disks. All servers are connected through 10G ethernet but not all clients. Gluster volumes are distributed without any replication. There are approximately 80 million files in file system. I am mounting using glusterfs on clients. I have copied everything from lustre to gluster but old file system exist so I can compare. The problem, I am facing is extremely slow du on even a small directory. Also the time taken is substantially different each time. I tried du from same client on a particular directory twice and got these results. time du -sh /data/aa/bb/cc 3.7G /data/aa/bb/cc real 7m29.243s user 0m1.448s sys 0m7.067s time du -sh /data/aa/bb/cc 3.7G /data/aa/bb/cc real 16m43.735s user 0m1.097s sys 0m5.802s 16m and 7m is too long for a 3.7 G directory. I must mention that the directory contains huge number of files (208736) but running du on same directory on old data gives this result time du -sh /olddata/aa/bb/cc 4.0G /olddata/aa/bb/cc real 3m1.255s user 0m0.755s sys 0m38.099s much better if I run same command again time du -sh /olddata/aa/bb/cc 4.0G /olddata/aa/bb/cc real 0m8.309s user 0m0.313s sys 0m7.755s Is there anything I can do to improve this performance? I would also like hear from some one who is running same kind of setup. Thanks Kashif -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170609/0f49b5c2/attachment.html>
Can you please provide more details about your volume configuration and the version of gluster that you are using? Regards, Vijay On Fri, Jun 9, 2017 at 5:35 PM, mohammad kashif <kashif.alig at gmail.com> wrote:> Hi > > I have just moved our 400 TB HPC storage from lustre to gluster. It is > part of a research institute and users have very small files to big files > ( few KB to 20GB) . Our setup consists of 5 servers, each with 96TB RAID 6 > disks. All servers are connected through 10G ethernet but not all clients. > Gluster volumes are distributed without any replication. There are > approximately 80 million files in file system. > I am mounting using glusterfs on clients. > > I have copied everything from lustre to gluster but old file system exist > so I can compare. > > The problem, I am facing is extremely slow du on even a small directory. > Also the time taken is substantially different each time. > I tried du from same client on a particular directory twice and got these > results. > > time du -sh /data/aa/bb/cc > 3.7G /data/aa/bb/cc > real 7m29.243s > user 0m1.448s > sys 0m7.067s > > time du -sh /data/aa/bb/cc > 3.7G /data/aa/bb/cc > real 16m43.735s > user 0m1.097s > sys 0m5.802s > > 16m and 7m is too long for a 3.7 G directory. I must mention that the > directory contains huge number of files (208736) > > but running du on same directory on old data gives this result > > time du -sh /olddata/aa/bb/cc > 4.0G /olddata/aa/bb/cc > real 3m1.255s > user 0m0.755s > sys 0m38.099s > > much better if I run same command again > > time du -sh /olddata/aa/bb/cc > 4.0G /olddata/aa/bb/cc > real 0m8.309s > user 0m0.313s > sys 0m7.755s > > Is there anything I can do to improve this performance? I would also like > hear from some one who is running same kind of setup. > > Thanks > > Kashif > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170609/69db101b/attachment.html>
Hi Vijay Thanks for your quick response. I am using gluster 3.8.11 on Centos 7 servers glusterfs-3.8.11-1.el7.x86_64 clients are centos 6 but I tested with a centos 7 client as well and results didn't change gluster volume info Volume Name: atlasglust Type: Distribute Volume ID: fbf0ebb8-deab-4388-9d8a-f722618a624b Status: Started Snapshot Count: 0 Number of Bricks: 5 Transport-type: tcp Bricks: Brick1: pplxgluster01.x.y.z:/glusteratlas/brick001/gv0 Brick2: pplxgluster02..x.y.z:/glusteratlas/brick002/gv0 Brick3: pplxgluster03.x.y.z:/glusteratlas/brick003/gv0 Brick4: pplxgluster04.x.y.z:/glusteratlas/brick004/gv0 Brick5: pplxgluster05.x.y.z:/glusteratlas/brick005/gv0 Options Reconfigured: nfs.disable: on performance.readdir-ahead: on transport.address-family: inet auth.allow: x.y.z I am not using directory quota. Please let me know if you require some more info Thanks Kashif On Fri, Jun 9, 2017 at 2:34 PM, Vijay Bellur <vbellur at redhat.com> wrote:> Can you please provide more details about your volume configuration and > the version of gluster that you are using? > > Regards, > Vijay > > On Fri, Jun 9, 2017 at 5:35 PM, mohammad kashif <kashif.alig at gmail.com> > wrote: > >> Hi >> >> I have just moved our 400 TB HPC storage from lustre to gluster. It is >> part of a research institute and users have very small files to big files >> ( few KB to 20GB) . Our setup consists of 5 servers, each with 96TB RAID 6 >> disks. All servers are connected through 10G ethernet but not all clients. >> Gluster volumes are distributed without any replication. There are >> approximately 80 million files in file system. >> I am mounting using glusterfs on clients. >> >> I have copied everything from lustre to gluster but old file system exist >> so I can compare. >> >> The problem, I am facing is extremely slow du on even a small directory. >> Also the time taken is substantially different each time. >> I tried du from same client on a particular directory twice and got >> these results. >> >> time du -sh /data/aa/bb/cc >> 3.7G /data/aa/bb/cc >> real 7m29.243s >> user 0m1.448s >> sys 0m7.067s >> >> time du -sh /data/aa/bb/cc >> 3.7G /data/aa/bb/cc >> real 16m43.735s >> user 0m1.097s >> sys 0m5.802s >> >> 16m and 7m is too long for a 3.7 G directory. I must mention that the >> directory contains huge number of files (208736) >> >> but running du on same directory on old data gives this result >> >> time du -sh /olddata/aa/bb/cc >> 4.0G /olddata/aa/bb/cc >> real 3m1.255s >> user 0m0.755s >> sys 0m38.099s >> >> much better if I run same command again >> >> time du -sh /olddata/aa/bb/cc >> 4.0G /olddata/aa/bb/cc >> real 0m8.309s >> user 0m0.313s >> sys 0m7.755s >> >> Is there anything I can do to improve this performance? I would also like >> hear from some one who is running same kind of setup. >> >> Thanks >> >> Kashif >> >> >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://lists.gluster.org/mailman/listinfo/gluster-users >> > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170609/b444b99e/attachment.html>