Gluster users - Oct 2020 - high load when copy directory with many files

hi,
we use glusterfs on a php application that have many small php files 
images etc...
We use glusterfs in replication mode.
We have 2 nodes connected in fiber with 100MBps and less than 1 ms latency.
We have also an arbiter on slower network (but the issue is there also 
without the arbiter).
When we copy a directory (cp command) with many files, cpu usage and 
load explode raplidly,
our application become inaccessible until the copy ends.

I wonder if is that normal or we have done something wrong.
I know that glusterfs is not indicated with many small files, and I know 
that it slow down,
but I want to avoid that a simple copy of a directory will put down out 
application.

Any suggestion?

Thanks a lot

Hi,
Have you tried tuned profiles? random-io
what type of storage are you using, spinning or SSD?


Just guessing here,  if you can provide a bit more details maybe someone
has already ran into this and has found a solution


regards,

Adrian Quinter

On Mon, Oct 5, 2020, 1:55 PM Marco Lerda - FOREACH S.R.L. <
marco.lerda at foreach.it> wrote:
> hi,
> we use glusterfs on a php application that have many small php files
> images etc...
> We use glusterfs in replication mode.
> We have 2 nodes connected in fiber with 100MBps and less than 1 ms latency.
> We have also an arbiter on slower network (but the issue is there also
> without the arbiter).
> When we copy a directory (cp command) with many files, cpu usage and
> load explode raplidly,
> our application become inaccessible until the copy ends.
>
> I wonder if is that normal or we have done something wrong.
> I know that glusterfs is not indicated with many small files, and I know
> that it slow down,
> but I want to avoid that a simple copy of a directory will put down out
> application.
>
> Any suggestion?
>
> Thanks a lot
>
>
>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20201005/73ddf9ff/attachment.html>

hi,
we have isolated the problem (meanwhile some hardware upgrade and code 
optimization helped to limit the problem).
it happens when many request (HTTP over apache) comes to a non existent 
file.
With 30 concurrent request to the same non existing file cause the load 
go high without limit.
Same requests on existing files works fine.
I have tried to simulate che apache access to file excluding apache with 
repeated command on files with the same parallelism (30):
- with ls works fine, file exists or not
- with stat works fine, file exists or not
- with xattr load go up, file exists or not

thank you


Il 05/10/2020 19.45, Marco Lerda - FOREACH S.R.L. ha
scritto:
> hi,
> we use glusterfs on a php application that have many small php files 
> images etc...
> We use glusterfs in replication mode.
> We have 2 nodes connected in fiber with 100MBps and less than 1 ms 
> latency.
> We have also an arbiter on slower network (but the issue is there also 
> without the arbiter).
> When we copy a directory (cp command) with many files, cpu usage and 
> load explode raplidly,
> our application become inaccessible until the copy ends.
>
> I wonder if is that normal or we have done something wrong.
> I know that glusterfs is not indicated with many small files, and I 
> know that it slow down,
> but I want to avoid that a simple copy of a directory will put down 
> out application.
>
> Any suggestion?
>
> Thanks a lot
>
>
>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
-- 

------------------------------------------------------
Marco Lerda
FOREACH S.R.L.
Via Laghi di Avigliana 115, 12022 - Busca (CN)
Telefono: 0171-1984102
Centralino/Fax: 0171-1984100
Email:  marco.lerda at foreach.it
Web: http://www.foreach.it