On 17/09/17 18:03, Niklas Hamb?chen wrote:> So far the only difference between `ls` and `bup index` I could observe > is that `bup index` chdir()s into the directory to index, ls doesn't. > > But when I `cd` into the dir and run `ls` without directory argument, it > is still much faster than bup index for each stat().Hmm, bup uses the fchdir() syscall to go into the target directory. Might that make a difference for gluster?
I found the reason now, at least for this set of lstat()s I was looking at. bup first does all getdents(), obtaining all file names in the directory, and then stat()s them. Apparently this destroys some of gluster's caching, making stat()s ~100x slower. What caching could this be, and how could I convince gluster to serve these stat()s as fast as if a getdents() had been done just before them?
Note, I have now put out a patch for bup that makes it 50x faster on my 1M files on gluster, exploiting the fact that stat() is fast after getdents() on it: https://groups.google.com/forum/#!topic/bup-list/6CM7Jr7b_Ek I'd still be interested in answers to this question though:> What caching could this be, and how could I convince gluster to serve > these stat()s as fast as if a getdents() had been done just before them?
----- Original Message -----> From: "Niklas Hamb?chen" <mail at nh2.me> > To: "Sam McLeod" <mailinglists at smcleod.net> > Cc: gluster-users at gluster.org > Sent: Sunday, September 17, 2017 7:34:15 PM > Subject: Re: [Gluster-users] Confusing lstat() performance > > I found the reason now, at least for this set of lstat()s I was looking at. > > bup first does all getdents(), obtaining all file names in the > directory, and then stat()s them.+1 I was thinking of a case just like this. -b> > Apparently this destroys some of gluster's caching, making stat()s ~100x > slower. > > What caching could this be, and how could I convince gluster to serve > these stat()s as fast as if a getdents() had been done just before them? > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users >