Erick Perez - Quadrian Enterprises
2020-Jul-03 15:12 UTC
[CentOS] Slow terminal response Centos 7.7 1908
Hey! I have a strange condition in one of the servers that I don't where to start looking. I login to the server via SSH (cant doit any other way) and anything that I type is slow HTTP sessions timeout waiting for screen redraw. So, the server is acting "slow". server is bare metal. no virtual services. no alarms in the disk raid note: server was restarted because of power failure. Some outputs from this server that is a mail server: [root at correo ~]# top top - 09:54:43 up 23:51, 2 users, load average: 0.18, 0.23, 0.28 Tasks: 210 total, 1 running, 209 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.2 us, 0.1 sy, 0.0 ni, 99.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 32606084 total, 25106412 free, 5932244 used, 1567428 buff/cache KiB Swap: 16449532 total, 16449532 free, 0 used. 26282624 avail Mem **iostat** [root at correo ~]# iostat -y 5 Linux 3.10.0-1062.12.1.el7.x86_64 (correo.binal.ac.pa) 07/03/2020 _x86_64_ (4 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 0.05 0.00 0.05 0.05 0.00 99.85 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn sda 0.00 0.00 0.00 0 0 dm-0 0.00 0.00 0.00 0 0 dm-1 0.00 0.00 0.00 0 0 avg-cpu: %user %nice %system %iowait %steal %idle 0.05 0.00 0.05 0.05 0.00 99.85 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn sda 21.40 0.00 169.60 0 848 dm-0 21.40 0.00 169.60 0 848 dm-1 0.00 0.00 0.00 0 0 avg-cpu: %user %nice %system %iowait %steal %idle 0.60 0.00 0.05 0.45 0.00 98.90 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn sda 1.20 16.80 0.00 84 0 dm-0 1.20 16.80 0.00 84 0 dm-1 0.00 0.00 0.00 0 0 avg-cpu: %user %nice %system %iowait %steal %idle 0.05 0.00 0.00 0.05 0.00 99.90 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn sda 8.00 0.00 100.20 0 501 dm-0 9.00 0.00 100.20 0 501 dm-1 0.00 0.00 0.00 0 0 avg-cpu: %user %nice %system %iowait %steal %idle 0.45 0.00 0.35 0.05 0.00 99.15 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn sda 1.00 0.80 3.20 4 16 dm-0 1.00 0.80 3.20 4 16 dm-1 0.00 0.00 0.00 0 0 **dstop** [root at correo ~]# dstat -cd --disk-util --disk-tps ----total-cpu-usage---- -dsk/total- sda- -dsk/total- usr sys idl wai hiq siq| read writ|util|reads writs 1 0 99 0 0 0| 20k 17k|0.14| 1 1 0 0 100 0 0 0| 0 0 | 0| 0 0 0 0 100 0 0 0| 0 0 | 0| 0 0 0 0 100 0 0 0| 0 0 | 0| 0 0 0 0 100 0 0 0| 0 0 | 0| 0 0 0 0 100 0 0 0| 0 0 | 0| 0 0 4 0 84 11 0 0|2512k 228k|52.3| 123 2 31 4 58 7 0 0|1912k 1026k|38.1| 132 23 0 0 99 0 0 0| 0 0 | 0| 0 0 1 0 99 1 0 0|4096B 3819k|22.5| 1 270 0 0 100 0 0 0| 0 0 | 0| 0 0 13 1 83 4 0 0| 148k 2304k|15.3| 18 214 1 0 98 1 0 0| 140k 499k|9.70| 14 8 26 5 69 0 0 0| 0 1260k|1.30| 0 46 56 7 38 0 0 0| 0 204k|0.30| 0 12 14 11 75 0 0 0| 0 0 | 0| 0 0 22 10 68 0 0 0| 0 0 | 0| 0 0 16 10 71 3 0 0| 192k 37k|14.0| 12 2 0 0 100 0 0 0| 0 0 | 0| 0 0 0 0 100 0 0 0| 0 152k| 0| 0 2 0 0 100 0 0 0| 0 0 | 0| 0 0 1 1 98 1 0 0| 16k 2569k|14.8| 1 207 1 1 98 0 0 0|4096B 0 |1.10| 1 0 1 0 99 0 0 0| 0 0 | 0| 0 0 0 0 100 0 0 0| 0 600k|1.30| 0 51 2 0 98 0 0 0| 0 0 | 0| 0 0 4 0 96 0 0 0| 0 0 | 0| 0 0 0 0 100 0 0 0| 0 0 | 0| 0 0 -- --------------------- Erick Perez ---------------------
Erick Perez - Quadrian Enterprises
2020-Jul-03 15:48 UTC
[CentOS] Slow terminal response Centos 7.7 1908
It was found that the software NIC team created in Centos was having issues due to a failing network cable. The team was going berserk with up/down changes. On Fri, Jul 3, 2020 at 10:12 AM Erick Perez - Quadrian Enterprises < eperez at quadrianweb.com> wrote:> Hey! > I have a strange condition in one of the servers that I don't where to > start looking. > I login to the server via SSH (cant doit any other way) and anything that > I type is slow > HTTP sessions timeout waiting for screen redraw. So, the server is acting > "slow". > > server is bare metal. no virtual services. > no alarms in the disk raid > > note: server was restarted because of power failure. > > Some outputs from this server that is a mail server: > [root at correo ~]# top > top - 09:54:43 up 23:51, 2 users, load average: 0.18, 0.23, 0.28 > Tasks: 210 total, 1 running, 209 sleeping, 0 stopped, 0 zombie > %Cpu(s): 0.2 us, 0.1 sy, 0.0 ni, 99.8 id, 0.0 wa, 0.0 hi, 0.0 si, > 0.0 st > KiB Mem : 32606084 total, 25106412 free, 5932244 used, 1567428 buff/cache > KiB Swap: 16449532 total, 16449532 free, 0 used. 26282624 avail Mem > > **iostat** > [root at correo ~]# iostat -y 5 > Linux 3.10.0-1062.12.1.el7.x86_64 (correo.binal.ac.pa) 07/03/2020 > _x86_64_ (4 CPU) > > avg-cpu: %user %nice %system %iowait %steal %idle > 0.05 0.00 0.05 0.05 0.00 99.85 > > Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn > sda 0.00 0.00 0.00 0 0 > dm-0 0.00 0.00 0.00 0 0 > dm-1 0.00 0.00 0.00 0 0 > > avg-cpu: %user %nice %system %iowait %steal %idle > 0.05 0.00 0.05 0.05 0.00 99.85 > > Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn > sda 21.40 0.00 169.60 0 848 > dm-0 21.40 0.00 169.60 0 848 > dm-1 0.00 0.00 0.00 0 0 > > avg-cpu: %user %nice %system %iowait %steal %idle > 0.60 0.00 0.05 0.45 0.00 98.90 > > Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn > sda 1.20 16.80 0.00 84 0 > dm-0 1.20 16.80 0.00 84 0 > dm-1 0.00 0.00 0.00 0 0 > > avg-cpu: %user %nice %system %iowait %steal %idle > 0.05 0.00 0.00 0.05 0.00 99.90 > > Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn > sda 8.00 0.00 100.20 0 501 > dm-0 9.00 0.00 100.20 0 501 > dm-1 0.00 0.00 0.00 0 0 > > avg-cpu: %user %nice %system %iowait %steal %idle > 0.45 0.00 0.35 0.05 0.00 99.15 > > Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn > sda 1.00 0.80 3.20 4 16 > dm-0 1.00 0.80 3.20 4 16 > dm-1 0.00 0.00 0.00 0 0 > > > **dstop** > [root at correo ~]# dstat -cd --disk-util --disk-tps > ----total-cpu-usage---- -dsk/total- sda- -dsk/total- > usr sys idl wai hiq siq| read writ|util|reads writs > 1 0 99 0 0 0| 20k 17k|0.14| 1 1 > 0 0 100 0 0 0| 0 0 | 0| 0 0 > 0 0 100 0 0 0| 0 0 | 0| 0 0 > 0 0 100 0 0 0| 0 0 | 0| 0 0 > 0 0 100 0 0 0| 0 0 | 0| 0 0 > 0 0 100 0 0 0| 0 0 | 0| 0 0 > 4 0 84 11 0 0|2512k 228k|52.3| 123 2 > 31 4 58 7 0 0|1912k 1026k|38.1| 132 23 > 0 0 99 0 0 0| 0 0 | 0| 0 0 > 1 0 99 1 0 0|4096B 3819k|22.5| 1 270 > 0 0 100 0 0 0| 0 0 | 0| 0 0 > 13 1 83 4 0 0| 148k 2304k|15.3| 18 214 > 1 0 98 1 0 0| 140k 499k|9.70| 14 8 > 26 5 69 0 0 0| 0 1260k|1.30| 0 46 > 56 7 38 0 0 0| 0 204k|0.30| 0 12 > 14 11 75 0 0 0| 0 0 | 0| 0 0 > 22 10 68 0 0 0| 0 0 | 0| 0 0 > 16 10 71 3 0 0| 192k 37k|14.0| 12 2 > 0 0 100 0 0 0| 0 0 | 0| 0 0 > 0 0 100 0 0 0| 0 152k| 0| 0 2 > 0 0 100 0 0 0| 0 0 | 0| 0 0 > 1 1 98 1 0 0| 16k 2569k|14.8| 1 207 > 1 1 98 0 0 0|4096B 0 |1.10| 1 0 > 1 0 99 0 0 0| 0 0 | 0| 0 0 > 0 0 100 0 0 0| 0 600k|1.30| 0 51 > 2 0 98 0 0 0| 0 0 | 0| 0 0 > 4 0 96 0 0 0| 0 0 | 0| 0 0 > 0 0 100 0 0 0| 0 0 | 0| 0 0 > > > > -- > > --------------------- > Erick Perez > --------------------- >-- --------------------- Erick Perez Quadrian Enterprises S.A. - Panama, Republica de Panama Skype chat: eaperezh WhatsApp IM: +507-6675-5083 ---------------------
Hi Erick, what was the value of 'si' in top ? Best Regards, Strahil Nikolov ?? 3 ??? 2020 ?. 18:48:30 GMT+03:00, Erick Perez - Quadrian Enterprises <eperez at quadrianweb.com> ??????:>It was found that the software NIC team created in Centos was having >issues due to a failing network cable. The team was going berserk with >up/down changes. > > >On Fri, Jul 3, 2020 at 10:12 AM Erick Perez - Quadrian Enterprises < >eperez at quadrianweb.com> wrote: > >> Hey! >> I have a strange condition in one of the servers that I don't where >to >> start looking. >> I login to the server via SSH (cant doit any other way) and anything >that >> I type is slow >> HTTP sessions timeout waiting for screen redraw. So, the server is >acting >> "slow". >> >> server is bare metal. no virtual services. >> no alarms in the disk raid >> >> note: server was restarted because of power failure. >> >> Some outputs from this server that is a mail server: >> [root at correo ~]# top >> top - 09:54:43 up 23:51, 2 users, load average: 0.18, 0.23, 0.28 >> Tasks: 210 total, 1 running, 209 sleeping, 0 stopped, 0 zombie >> %Cpu(s): 0.2 us, 0.1 sy, 0.0 ni, 99.8 id, 0.0 wa, 0.0 hi, 0.0 >si, >> 0.0 st >> KiB Mem : 32606084 total, 25106412 free, 5932244 used, 1567428 >buff/cache >> KiB Swap: 16449532 total, 16449532 free, 0 used. 26282624 >avail Mem >> >> **iostat** >> [root at correo ~]# iostat -y 5 >> Linux 3.10.0-1062.12.1.el7.x86_64 (correo.binal.ac.pa) 07/03/2020 >> _x86_64_ (4 CPU) >> >> avg-cpu: %user %nice %system %iowait %steal %idle >> 0.05 0.00 0.05 0.05 0.00 99.85 >> >> Device: tps kB_read/s kB_wrtn/s kB_read >kB_wrtn >> sda 0.00 0.00 0.00 0 >0 >> dm-0 0.00 0.00 0.00 0 >0 >> dm-1 0.00 0.00 0.00 0 >0 >> >> avg-cpu: %user %nice %system %iowait %steal %idle >> 0.05 0.00 0.05 0.05 0.00 99.85 >> >> Device: tps kB_read/s kB_wrtn/s kB_read >kB_wrtn >> sda 21.40 0.00 169.60 0 >848 >> dm-0 21.40 0.00 169.60 0 >848 >> dm-1 0.00 0.00 0.00 0 >0 >> >> avg-cpu: %user %nice %system %iowait %steal %idle >> 0.60 0.00 0.05 0.45 0.00 98.90 >> >> Device: tps kB_read/s kB_wrtn/s kB_read >kB_wrtn >> sda 1.20 16.80 0.00 84 >0 >> dm-0 1.20 16.80 0.00 84 >0 >> dm-1 0.00 0.00 0.00 0 >0 >> >> avg-cpu: %user %nice %system %iowait %steal %idle >> 0.05 0.00 0.00 0.05 0.00 99.90 >> >> Device: tps kB_read/s kB_wrtn/s kB_read >kB_wrtn >> sda 8.00 0.00 100.20 0 >501 >> dm-0 9.00 0.00 100.20 0 >501 >> dm-1 0.00 0.00 0.00 0 >0 >> >> avg-cpu: %user %nice %system %iowait %steal %idle >> 0.45 0.00 0.35 0.05 0.00 99.15 >> >> Device: tps kB_read/s kB_wrtn/s kB_read >kB_wrtn >> sda 1.00 0.80 3.20 4 >16 >> dm-0 1.00 0.80 3.20 4 >16 >> dm-1 0.00 0.00 0.00 0 >0 >> >> >> **dstop** >> [root at correo ~]# dstat -cd --disk-util --disk-tps >> ----total-cpu-usage---- -dsk/total- sda- -dsk/total- >> usr sys idl wai hiq siq| read writ|util|reads writs >> 1 0 99 0 0 0| 20k 17k|0.14| 1 1 >> 0 0 100 0 0 0| 0 0 | 0| 0 0 >> 0 0 100 0 0 0| 0 0 | 0| 0 0 >> 0 0 100 0 0 0| 0 0 | 0| 0 0 >> 0 0 100 0 0 0| 0 0 | 0| 0 0 >> 0 0 100 0 0 0| 0 0 | 0| 0 0 >> 4 0 84 11 0 0|2512k 228k|52.3| 123 2 >> 31 4 58 7 0 0|1912k 1026k|38.1| 132 23 >> 0 0 99 0 0 0| 0 0 | 0| 0 0 >> 1 0 99 1 0 0|4096B 3819k|22.5| 1 270 >> 0 0 100 0 0 0| 0 0 | 0| 0 0 >> 13 1 83 4 0 0| 148k 2304k|15.3| 18 214 >> 1 0 98 1 0 0| 140k 499k|9.70| 14 8 >> 26 5 69 0 0 0| 0 1260k|1.30| 0 46 >> 56 7 38 0 0 0| 0 204k|0.30| 0 12 >> 14 11 75 0 0 0| 0 0 | 0| 0 0 >> 22 10 68 0 0 0| 0 0 | 0| 0 0 >> 16 10 71 3 0 0| 192k 37k|14.0| 12 2 >> 0 0 100 0 0 0| 0 0 | 0| 0 0 >> 0 0 100 0 0 0| 0 152k| 0| 0 2 >> 0 0 100 0 0 0| 0 0 | 0| 0 0 >> 1 1 98 1 0 0| 16k 2569k|14.8| 1 207 >> 1 1 98 0 0 0|4096B 0 |1.10| 1 0 >> 1 0 99 0 0 0| 0 0 | 0| 0 0 >> 0 0 100 0 0 0| 0 600k|1.30| 0 51 >> 2 0 98 0 0 0| 0 0 | 0| 0 0 >> 4 0 96 0 0 0| 0 0 | 0| 0 0 >> 0 0 100 0 0 0| 0 0 | 0| 0 0 >> >> >> >> -- >> >> --------------------- >> Erick Perez >> --------------------- >>