similar to: Very slow rsync to gluster volume UNLESS `ls` or `find` scan dir on gluster volume first

You mounting it to the local bricks? struggling with same performance issues try using this volume setting http://lists.gluster.org/pipermail/gluster-users/2018-January/033397.html performance.stat-prefetch: on might be it seems like when it gets to cache it is fast - those stat fetch which seem to come from .gluster are slow On Sun, Feb 4, 2018 at 3:45 AM, Artem Russakovskii <archon810 at

Thanks for the report Artem, Looks like the issue is about cache warming up. Specially, I suspect rsync doing a 'readdir(), stat(), file operations' loop, where as when a find or ls is issued, we get 'readdirp()' request, which contains the stat information along with entries, which also makes sure cache is up-to-date (at md-cache layer). Note that this is just a off-the memory

Hi all, I have seen this issue as well, on Gluster 3.12.1. (3 bricks per box, 2 boxes, distributed-replicate) My testing shows the same thing -- running a find on a directory dramatically increases lstat performance. To add another clue, the performance degrades again after issuing a call to reset the system's cache of dentries and inodes: # sync; echo 2 > /proc/sys/vm/drop_caches I

Any updates on this one? On Mon, Feb 5, 2018 at 8:18 AM, Tom Fite <tomfite at gmail.com> wrote: > Hi all, > > I have seen this issue as well, on Gluster 3.12.1. (3 bricks per box, 2 > boxes, distributed-replicate) My testing shows the same thing -- running a > find on a directory dramatically increases lstat performance. To add > another clue, the performance degrades

Nithya, Amar, Any movement here? There could be a significant performance gain here that may also affect other bottlenecks that I'm experiencing which make gluster close to unusable at times. Sincerely, Artem -- Founder, Android Police <http://www.androidpolice.com>, APK Mirror <http://www.apkmirror.com/>, Illogical Robot LLC beerpla.net | +ArtemRussakovskii

I calculate an anova test in the following way: expdata<-read.table("/home/dorien/UA/meta-music/optimuse/optimuse1-build-desktop/results/results_processedCP", header=TRUE)

Hi, I have a gluster 3.10 volume with a dir with ~1 million small files in them, say mounted at /mnt/dir with FUSE, and I'm observing something weird: When I list and stat them all using rsync, then the lstat() calls that rsync does are incredibly fast (23 microseconds per call on average, definitely faster than a network roundtrip between my 3-machine bricks connected via Ethernet). But

On 17/09/17 18:03, Niklas Hamb?chen wrote: > So far the only difference between `ls` and `bup index` I could observe > is that `bup index` chdir()s into the directory to index, ls doesn't. > > But when I `cd` into the dir and run `ls` without directory argument, it > is still much faster than bup index for each stat(). Hmm, bup uses the fchdir() syscall to go into the target

Hi Ben, do you know if the smallfile benchmark also does interleaved getdents() and lstat, which is what I found as being the key difference that creates the performance gap (further down this thread)? Also, wouldn't `--threads 8` change the performance numbers by factor 8 versus the plain `ls` and `rsync` that I did? Would you mind running those commands directly/plainly on your cluster to

Hi, I have been working on setting up a 4 replica gluster with over a million files (~250GB total), and I've seen some really weird stuff happen, even after trying to optimize for small files. I've set up a 4-brick replicate volume (gluster 3.13.2). It took almost 2 days to rsync the data from the local drive to the gluster volume, and now I'm running a 2nd rsync that just looks for

On 18/09/17 17:23, Ben Turner wrote: > Do you want tuned or untuned? If tuned I'd like to try one of my tunings for metadata, but I will use yours if you want. (Re-CC'd list) I would be interested in both, if possible: To confirm that it's not only my machines that exhibit this behaviour given my settings, and to see what can be achieved with your tuned settings. Thank you!

I found the reason now, at least for this set of lstat()s I was looking at. bup first does all getdents(), obtaining all file names in the directory, and then stat()s them. Apparently this destroys some of gluster's caching, making stat()s ~100x slower. What caching could this be, and how could I convince gluster to serve these stat()s as fast as if a getdents() had been done just before

Thanks for the link. Looking at the status of that doc, it isn't quite ready yet, and there's no mention of the option. Does it mean that whatever is ready now in 4.0.1 is incomplete but can be enabled via granular-entry-heal=on, and when it is complete, it'll become the default and the flag will simply go away? Is there any risk enabling the option now in 4.0.1? Sincerely, Artem

That would probably work, too; it's harder to configure though and easier to get wrong. If I could avoid having the tun0, that would trivially solve the problem. On 27/01/17 09:41, Azul wrote: > Why not just firewall incoming traffic on the clients? > > > On 27 Jan 2017 8:37 am, "Niklas Hambüchen" <mail at nh2.me > <mailto:mail at nh2.me>> wrote: >

I did a quick test on one of my lab clusters with no tuning except for quota being enabled: [root at dell-per730-03 ~]# gluster v info Volume Name: vmstore Type: Replicate Volume ID: 0d2e4c49-334b-47c9-8e72-86a4c040a7bd Status: Started Snapshot Count: 0 Number of Bricks: 1 x (2 + 1) = 3 Transport-type: tcp Bricks: Brick1: 192.168.50.1:/rhgs/brick1/vmstore Brick2:

Hi Ravi, Could you please expand on how these would help? By forcing full here, we move the logic from the CPU to network, thus decreasing CPU utilization, is that right? This is assuming the CPU and disk utilization are caused by the differ and not by lstat and other calls or something. > Option: cluster.data-self-heal-algorithm > Default Value: (null) > Description: Select between

Hi, I am sorry for this simple question, but... How do I extract something from a data.frame? The following is my Problem: I have got a dataframe "a" with various columns. One of those columns is called V3 and contains elements of the following levels: > levels(a$V3) [1] "C" "CA" "CB" "CD" "CD1" "CD2"

Following up here on a related and very serious for us issue. I took down one of the 4 replicate gluster servers for maintenance today. There are 2 gluster volumes totaling about 600GB. Not that much data. After the server comes back online, it starts auto healing and pretty much all operations on gluster freeze for many minutes. For example, I was trying to run an ls -alrt in a folder with 7300

Btw, I've now noticed at least 5 variations in toggling binary option values. Are they all interchangeable, or will using the wrong value not work in some cases? yes/no true/false True/False on/off enable/disable It's quite a confusing/inconsistent practice, especially given that many options will accept any value without erroring out/validation. Sincerely, Artem -- Founder, Android

pylon:/var/lib/glusterd/vols/dev_apkmirror_data # ack shared-brick-count dev_apkmirror_data.pylon.mnt-pylon_block3-dev_apkmirror_data.vol 3: option shared-brick-count 3 dev_apkmirror_data.pylon.mnt-pylon_block2-dev_apkmirror_data.vol 3: option shared-brick-count 3 dev_apkmirror_data.pylon.mnt-pylon_block1-dev_apkmirror_data.vol 3: option shared-brick-count 3 Sincerely, Artem --