Gluster users - Feb 2018 - Very slow rsync to gluster volume UNLESS `ls` or `find` scan dir on gluster volume first

An update, and a very interesting one!

After I started stracing rsync, all I could see was lstat calls, quite slow
ones, over and over, which is expected.

For example:
lstat("uploads/2016/10/nexus2cee_DSC05339_thumb-161x107.jpg",
{st_mode=S_IFREG|0664, st_size=4043, ...}) = 0

I googled around and found
https://gist.github.com/nh2/1836415489e2132cf85ed3832105fcc1, which is seeing
this exact issue with gluster, rsync and xfs.

Here's the craziest finding so far. If while rsync is running (or right
before), I run /bin/ls or find on the same gluster dirs, it immediately
speeds up rsync by a factor of 100 or maybe even 1000. It's absolutely
insane.

I'm stracing the rsync run, and the slow lstat calls flood in at an
incredible speed as soon as ls or find run. Several hundred of files per
minute (excruciatingly slow) becomes thousands or even tens of thousands of
files a second.

What do you make of this?
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180204/8929a2a9/attachment.html>

You mounting it to the local bricks?

struggling with same performance issues
try using this volume setting
http://lists.gluster.org/pipermail/gluster-users/2018-January/033397.html
performance.stat-prefetch: on might be it

seems like when it gets to cache it is fast - those stat fetch which
seem to come from .gluster are slow

On Sun, Feb 4, 2018 at 3:45 AM, Artem Russakovskii <archon810 at
gmail.com> wrote:
> An update, and a very interesting one!
>
> After I started stracing rsync, all I could see was lstat calls, quite slow
> ones, over and over, which is expected.
>
> For example:
lstat("uploads/2016/10/nexus2cee_DSC05339_thumb-161x107.jpg",
> {st_mode=S_IFREG|0664, st_size=4043, ...}) = 0
>
> I googled around and found
> https://gist.github.com/nh2/1836415489e2132cf85ed3832105fcc1, which is
> seeing this exact issue with gluster, rsync and xfs.
>
> Here's the craziest finding so far. If while rsync is running (or right
> before), I run /bin/ls or find on the same gluster dirs, it immediately
> speeds up rsync by a factor of 100 or maybe even 1000. It's absolutely
> insane.
>
> I'm stracing the rsync run, and the slow lstat calls flood in at an
> incredible speed as soon as ls or find run. Several hundred of files per
> minute (excruciatingly slow) becomes thousands or even tens of thousands of
> files a second.
>
> What do you make of this?
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users

Thanks for the report Artem,

Looks like the issue is about cache warming up. Specially, I suspect rsync
doing a 'readdir(), stat(), file operations' loop, where as when a find
or
ls is issued, we get 'readdirp()' request, which contains the stat
information along with entries, which also makes sure cache is up-to-date
(at md-cache layer).

Note that this is just a off-the memory hypothesis, We surely need to
analyse and debug more thoroughly for a proper explanation.  Some one in my
team would look at it soon.

Regards,
Amar

On Mon, Feb 5, 2018 at 7:25 AM, Vlad Kopylov <vladkopy at gmail.com>
wrote:
> You mounting it to the local bricks?
>
> struggling with same performance issues
> try using this volume setting
> http://lists.gluster.org/pipermail/gluster-users/2018-January/033397.html
> performance.stat-prefetch: on might be it
>
> seems like when it gets to cache it is fast - those stat fetch which
> seem to come from .gluster are slow
>
> On Sun, Feb 4, 2018 at 3:45 AM, Artem Russakovskii <archon810 at
gmail.com>
> wrote:
> > An update, and a very interesting one!
> >
> > After I started stracing rsync, all I could see was lstat calls, quite
> slow
> > ones, over and over, which is expected.
> >
> > For example: lstat("uploads/2016/10/nexus2cee_DSC05339_thumb-
> 161x107.jpg",
> > {st_mode=S_IFREG|0664, st_size=4043, ...}) = 0
> >
> > I googled around and found
> > https://gist.github.com/nh2/1836415489e2132cf85ed3832105fcc1, which is
> > seeing this exact issue with gluster, rsync and xfs.
> >
> > Here's the craziest finding so far. If while rsync is running (or
right
> > before), I run /bin/ls or find on the same gluster dirs, it
immediately
> > speeds up rsync by a factor of 100 or maybe even 1000. It's
absolutely
> > insane.
> >
> > I'm stracing the rsync run, and the slow lstat calls flood in at
an
> > incredible speed as soon as ls or find run. Several hundred of files
per
> > minute (excruciatingly slow) becomes thousands or even tens of
thousands
> of
> > files a second.
> >
> > What do you make of this?
> >
> >
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > http://lists.gluster.org/mailman/listinfo/gluster-users
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>


-- 
Amar Tumballi (amarts)
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180205/206efebb/attachment.html>