Simon Matter
2021-Mar-12 08:35 UTC
[CentOS] Disk read io very high, but no process perform io read
Hi, You said that you have multiple systems running this same application. But, do they work with the same data on disk or are there big differences?>From how I understand the figures below, your buff/cache seems a bit lowif you read a lot of data from disk. If you read a lot of data and filesystem caches are slow, it will result in heavy reading from disk directly. One more thing comes to mind: is your hardware behaving fine? I mean, if your storag has difficulties to read data from disk due to hardware issues, this could also lead to such problems you're facing. Regards, Simon> > yes. I suspect it has something to do with swapping. but swap is turned > off on this server. > here is the result of free -m. > total used free shared buff/cache available > Mem: 128174 97449 24400 4158 6325 > 25232 > Swap: 0 0 0 > > > We have other servers. The processes running on these servers are same. > but on other servers, the size of buff/cache is larger than the size on > the server which experienced the problem and the size of "free" is smaller > than the size on the server which experienced the problem. > > > > > > > > > > > > > > > ?? 2021-03-10 14:30:00??"Thomas Stephen Lee" <lee.iitb at gmail.com> ????? >>On Sat, Mar 6, 2021 at 6:37 AM yf chu <cyflhn at 163.com> wrote: >>> >>> no, not vm >>> >>> >>> >>> >>> | | >>> yf chu >>> | >>> | >>> ????cyflhn at 163.com >>> | >>> >>> ????? ?????????? ???? >>> >>> On 03/05/2021 23:37, Matthew Miller wrote: >>> On Fri, Mar 05, 2021 at 03:54:14PM +0800, yf chu wrote: >>> > We have experienced a very weird problem. The load of the server >>> machine is very high. We use "pidstat" and find that the disk read io >>> is very high. >>> >>> Is this system a VM? >>> >>> -- >>> Matthew Miller >>> <mattdm at fedoraproject.org> >>> Fedora Project Leader >> >>Hi, >> >>Is this due to swapping? >> >>what is the output of >> >>free -m >> >>--- >>Lee >>_______________________________________________ >>CentOS mailing list >>CentOS at centos.org >>https://lists.centos.org/mailman/listinfo/centos > _______________________________________________ > CentOS mailing list > CentOS at centos.org > https://lists.centos.org/mailman/listinfo/centos >
yf chu
2021-Mar-12 10:25 UTC
[CentOS] Disk read io very high, but no process perform io read
The applications on all those servers are same. They are working on same data. I still don't know why the size of buff/cache is different between different servers. At 2021-03-12 16:35:09, "Simon Matter" <simon.matter at invoca.ch> wrote:>Hi, > >You said that you have multiple systems running this same application. >But, do they work with the same data on disk or are there big differences? > >From how I understand the figures below, your buff/cache seems a bit low >if you read a lot of data from disk. If you read a lot of data and >filesystem caches are slow, it will result in heavy reading from disk >directly. > >One more thing comes to mind: is your hardware behaving fine? I mean, if >your storag has difficulties to read data from disk due to hardware >issues, this could also lead to such problems you're facing. > >Regards, >Simon > >> >> yes. I suspect it has something to do with swapping. but swap is turned >> off on this server. >> here is the result of free -m. >> total used free shared buff/cache available >> Mem: 128174 97449 24400 4158 6325 >> 25232 >> Swap: 0 0 0 >> >> >> We have other servers. The processes running on these servers are same. >> but on other servers, the size of buff/cache is larger than the size on >> the server which experienced the problem and the size of "free" is smaller >> than the size on the server which experienced the problem. >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> ?? 2021-03-10 14:30:00??"Thomas Stephen Lee" <lee.iitb at gmail.com> ????? >>>On Sat, Mar 6, 2021 at 6:37 AM yf chu <cyflhn at 163.com> wrote: >>>> >>>> no, not vm >>>> >>>> >>>> >>>> >>>> | | >>>> yf chu >>>> | >>>> | >>>> ????cyflhn at 163.com >>>> | >>>> >>>> ????? ?????????? ???? >>>> >>>> On 03/05/2021 23:37, Matthew Miller wrote: >>>> On Fri, Mar 05, 2021 at 03:54:14PM +0800, yf chu wrote: >>>> > We have experienced a very weird problem. The load of the server >>>> machine is very high. We use "pidstat" and find that the disk read io >>>> is very high. >>>> >>>> Is this system a VM? >>>> >>>> -- >>>> Matthew Miller >>>> <mattdm at fedoraproject.org> >>>> Fedora Project Leader >>> >>>Hi, >>> >>>Is this due to swapping? >>> >>>what is the output of >>> >>>free -m >>> >>>--- >>>Lee >>>_______________________________________________ >>>CentOS mailing list >>>CentOS at centos.org >>>https://lists.centos.org/mailman/listinfo/centos >> _______________________________________________ >> CentOS mailing list >> CentOS at centos.org >> https://lists.centos.org/mailman/listinfo/centos >> > > >_______________________________________________ >CentOS mailing list >CentOS at centos.org >https://lists.centos.org/mailman/listinfo/centos