Gluster users - Apr 2021 - high load when copy directory with many files

hi,
we have isolated the problem (meanwhile some hardware upgrade and code 
optimization helped to limit the problem).
it happens when many request (HTTP over apache) comes to a non existent 
file.
With 30 concurrent request to the same non existing file cause the load 
go high without limit.
Same requests on existing files works fine.
I have tried to simulate che apache access to file excluding apache with 
repeated command on files with the same parallelism (30):
- with ls works fine, file exists or not
- with stat works fine, file exists or not
- with xattr load go up, file exists or not

thank you


Il 05/10/2020 19.45, Marco Lerda - FOREACH S.R.L. ha
scritto:
> hi,
> we use glusterfs on a php application that have many small php files 
> images etc...
> We use glusterfs in replication mode.
> We have 2 nodes connected in fiber with 100MBps and less than 1 ms 
> latency.
> We have also an arbiter on slower network (but the issue is there also 
> without the arbiter).
> When we copy a directory (cp command) with many files, cpu usage and 
> load explode raplidly,
> our application become inaccessible until the copy ends.
>
> I wonder if is that normal or we have done something wrong.
> I know that glusterfs is not indicated with many small files, and I 
> know that it slow down,
> but I want to avoid that a simple copy of a directory will put down 
> out application.
>
> Any suggestion?
>
> Thanks a lot
>
>
>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
-- 

------------------------------------------------------
Marco Lerda
FOREACH S.R.L.
Via Laghi di Avigliana 115, 12022 - Busca (CN)
Telefono: 0171-1984102
Centralino/Fax: 0171-1984100
Email:  marco.lerda at foreach.it
Web: http://www.foreach.it

Hi Marco, this is really good test/info. Thanks.

One more thing to observe is you are running such tests is 'gluster profile
info', so the bottleneck fop is listed.

Mohit, Xavi, in this parallel operations, the load may be high due to
inodelk used in mds xattr update in dht? Or you guys suspect something else?

Regards
Amar

On Sat, 10 Apr, 2021, 11:45 pm Marco Lerda - FOREACH S.R.L., <
marco.lerda at foreach.it> wrote:
> hi,
> we have isolated the problem (meanwhile some hardware upgrade and code
> optimization helped to limit the problem).
> it happens when many request (HTTP over apache) comes to a non existent
> file.
> With 30 concurrent request to the same non existing file cause the load
> go high without limit.
> Same requests on existing files works fine.
> I have tried to simulate che apache access to file excluding apache with
> repeated command on files with the same parallelism (30):
> - with ls works fine, file exists or not
> - with stat works fine, file exists or not
> - with xattr load go up, file exists or not
>
> thank you
>
>
> Il 05/10/2020 19.45, Marco Lerda - FOREACH S.R.L. ha scritto:
> > hi,
> > we use glusterfs on a php application that have many small php files
> > images etc...
> > We use glusterfs in replication mode.
> > We have 2 nodes connected in fiber with 100MBps and less than 1 ms
> > latency.
> > We have also an arbiter on slower network (but the issue is there also
> > without the arbiter).
> > When we copy a directory (cp command) with many files, cpu usage and
> > load explode raplidly,
> > our application become inaccessible until the copy ends.
> >
> > I wonder if is that normal or we have done something wrong.
> > I know that glusterfs is not indicated with many small files, and I
> > know that it slow down,
> > but I want to avoid that a simple copy of a directory will put down
> > out application.
> >
> > Any suggestion?
> >
> > Thanks a lot
> >
> >
> >
> > ________
> >
> >
> >
> > Community Meeting Calendar:
> >
> > Schedule -
> > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> > Bridge: https://bluejeans.com/441850968
> >
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > https://lists.gluster.org/mailman/listinfo/gluster-users
>
> --
>
> ------------------------------------------------------
> Marco Lerda
> FOREACH S.R.L.
> Via Laghi di Avigliana 115, 12022 - Busca (CN)
> Telefono: 0171-1984102
> Centralino/Fax: 0171-1984100
> Email:  marco.lerda at foreach.it
> Web: http://www.foreach.it
>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://meet.google.com/cpu-eiue-hvk
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20210411/25a96da3/attachment.html>