We have a very old Dell desk top machine that has been running CentOS 6 for the past five years.? It received a new, 1 TB disk and additional memory before the OS installation.? It has been the primary Linux machine in our smallest and most remote field office.? It has been updated at least once a week and has all current dates installed. Boot-up this morning lasted about six times as long as usual. Disk access, as indicated by the disk activity light, is almost continuous and for extended periods of time when ever something is done that requires the disk.? Everything observed happens whether or not the machine is connected to our network. All of our files appear to be accessible if one is patient. One theory put forward is that some application is running that uses up CPU and disk bandwidth.? Another theory is that thereare disk errors, mostly corrected by EDCS features. We do not see any rogue applications and error logs show no disk issues. This is a mysterious issue that we hope to circumvent by putting a new disk and installing CentOS 7 from DVD.? Our hope is that the current disk can be mounted externally on the new CentOS system using a USB to SATA adapter and that data can be moved off of the old disk. Advice regarding this issue and any possible diagnostic methods will be greatly appreciated. df -k output: +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Filesystem??????? 1K-blocks????? Used Available Use% Mounted on /dev/mapper/vg_delle520-lv_root ?????????????????? 51475068? 12110896? 36742732? 25% / tmpfs?????????????? 1928152?????? 176?? 1927976?? 1% /dev/shm /dev/sda1??????????? 487652??? 211073??? 250979? 46% /boot /dev/mapper/vg_delle520-lv_home ????????????????? 905124888 246856176 612284356? 29% /home +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Olson via CentOS wrote:> We have a very old Dell desk top machine that has been running > CentOS 6 for the past five years.? It received a new, 1 TB disk > and additional memory before the OS installation.? It has been the primary > Linux machine in our smallest and most remote field > office.? It has been updated at least once a week and has all current dates > installed. > > Boot-up this morning lasted about six times as long as usual. > Disk access, as indicated by the disk activity light, is almost > continuous and for extended periods of time when ever something is done > that requires the disk.? Everything observed happens whether or not the > machine is connected to our network. All of our files appear to be > accessible if one is patient. > > One theory put forward is that some application is running that > uses up CPU and disk bandwidth.? Another theory is that thereare disk > errors, mostly corrected by EDCS features. We do not see any rogue > applications and error logs show no disk issues. > > This is a mysterious issue that we hope to circumvent by putting > a new disk and installing CentOS 7 from DVD.? Our hope is that the current > disk can be mounted externally on the new CentOS system using a USB to > SATA adapter and that data can be moved > off of the old disk. > > Advice regarding this issue and any possible diagnostic methods > will be greatly appreciated. > > df -k output: > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > Filesystem??????? 1K-blocks????? Used Available Use% Mounted on > /dev/mapper/vg_delle520-lv_root > ?????????????????? 51475068? 12110896? 36742732? 25% / > tmpfs?????????????? 1928152?????? 176?? 1927976?? 1% /dev/shm > /dev/sda1??????????? 487652??? 211073??? 250979? 46% /boot > /dev/mapper/vg_delle520-lv_home > ????????????????? 905124888 246856176 612284356? 29% /home > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >If there's nothing in /var/log/messages, then my next trick would be to do a smartctl -t short, and wait for the results. Also, have you done an ipmitool sel list? mark
On Mon, Jun 24, 2019 at 10:42 AM Chris Olson via CentOS <centos at centos.org> wrote:> Advice regarding this issue and any possible diagnostic methods > will be greatly appreciated. >Install smartmontools. You can use the smartctl tool to get all of the S.M.A.R.T. data for the drive. You can also set up the config to email you (or whoever) when a S.M.A.R.T. error occurs.
Chris Olson <chris_e_olson at yahoo.com> wrote:> One theory put forward is that some application is running that > uses up CPU and disk bandwidth.1. Check the cable. I'm getting convinced that half of all problems are caused by cables. 2. Use iotop to see what's using the disk. 3. Check the disk using smartmontools and badblocks (not in destructive mode). -- Yves Bellefeuille <yan at storm.ca>
On Mon, Jun 24, 2019 at 03:42:24PM +0000, Chris Olson via CentOS wrote:> We have a very old Dell desk top machine that has been running > CentOS 6 for the past five years.? It received a new, 1 TB disk > and additional memory before the OS installation.? It has been > the primary Linux machine in our smallest and most remote field > office.? It has been updated at least once a week and has all > current dates installed. > > Boot-up this morning lasted about six times as long as usual. > Disk access, as indicated by the disk activity light, is almost > continuous and for extended periods of time when ever something > is done that requires the disk.? Everything observed happens > whether or not the machine is connected to our network. All of > our files appear to be accessible if one is patient. > > One theory put forward is that some application is running that > uses up CPU and disk bandwidth.? Another theory is that thereare disk errors, mostly corrected by EDCS features. We do not > see any rogue applications and error logs show no disk issues. > > This is a mysterious issue that we hope to circumvent by putting > a new disk and installing CentOS 7 from DVD.? Our hope is that > the current disk can be mounted externally on the new CentOS > system using a USB to SATA adapter and that data can be moved > off of the old disk. > > Advice regarding this issue and any possible diagnostic methods > will be greatly appreciated. > > df -k output: > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > Filesystem??????? 1K-blocks????? Used Available Use% Mounted on > /dev/mapper/vg_delle520-lv_root > ?????????????????? 51475068? 12110896? 36742732? 25% / > tmpfs?????????????? 1928152?????? 176?? 1927976?? 1% /dev/shm > /dev/sda1??????????? 487652??? 211073??? 250979? 46% /boot > /dev/mapper/vg_delle520-lv_home > ????????????????? 905124888 246856176 612284356? 29% /home > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >If it had not been booted in a while, it may have been running the file system check (fsck) program. I think if fsck has not been run in 6 months, it automatically happens at boot. jl -- Jon H. LaBadie jon at jgcomp.com 11226 South Shore Rd. (703) 787-0688 (H) Reston, VA 20190 (703) 935-6720 (C)
On 6/24/19 8:42 AM, Chris Olson via CentOS wrote:> We have a very old Dell desk top machine that has been running > CentOS 6 for the past five years.? It received a new, 1 TB disk > ... > Boot-up this morning lasted about six times as long as usual. > Disk access, as indicated by the disk activity light, is almost > continuousIs it running software raid?? Check the output of "cat /proc/mdstat" to see if it's re-syncing an array.