search for: r_await

...1 ~] # iostat -xdmc 1 10 Linux 3.10.0-327.13.1.el7.x86_64 (r2k1) 05/27/16 _x86_64_ (32 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 8.87 0.02 1.28 0.21 0.00 89.62 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sdd 0.02 0.55 0.15 27.06 0.03 11.40 859.89 1.02 37.40 36.13 37.41 6.86 18.65 sdf 0.02 0.48 0.15 26.99 0.03 11.40 862.17 0.15 5.56 40.94 5.37 7.27 19.73 sdk 0.03 0.58...

On 2016-05-25 19:13, Kelly Lesperance wrote: > Hdparm didn?t get far: > > [root at r1k1 ~] # hdparm -tT /dev/sda > > /dev/sda: > Timing cached reads: Alarm clock > [root at r1k1 ~] # Hi Kelly, Try running 'iostat -xdmc 1'. Look for a single drive that has substantially greater await than ~10msec. If all the drives except one are taking 6-8msec, but one is very

...>Linux 3.10.0-327.13.1.el7.x86_64 (r2k1) 05/27/16 _x86_64_ (32 CPU) > >avg-cpu: %user %nice %system %iowait %steal %idle > 8.87 0.02 1.28 0.21 0.00 89.62 > >Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util >sdd 0.02 0.55 0.15 27.06 0.03 11.40 859.89 1.02 37.40 36.13 37.41 6.86 18.65 >sdf 0.02 0.48 0.15 26.99 0.03 11.40 862.17 0.15 5.56 40.94 5.37 7.27 19.73 >sdk 0.0...

Hi, I have high IO-Wait on the ods (ceph), the osd are running a v3.1-rc9 kernel. I also experience high IO-rates, around 500IO/s reported via iostat. Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 0.00 0.00 6.80 0.00 62.40 18.35 0.04 5.29 0.00 5.29 5.29 3.60 sdb 0.00 249.80 0.40 669.60 1.60 4118.40 12.30 87.47 130.56 15.00 130.63 1.01 67.40 In comparison, the same workload, bu...

I?ve posted this on the forums at https://www.centos.org/forums/viewtopic.php?f=47&t=57926&p=244614#p244614 - posting to the list in the hopes of getting more eyeballs on it. We have a cluster of 23 HP DL380p Gen8 hosts running Kafka. Basic specs: 2x E5-2650 128 GB RAM 12 x 4 TB 7200 RPM SATA drives connected to an HP H220 HBA Dual port 10 GB NIC The drives are configured as one large

.../cache KiB Swap: 1048572 total, 990776 free, 57796 used. 353468 avail Mem iostat: avg-cpu: %user %nice %system %iowait %steal %idle 0,00 0,00 0,00 50,00 0,00 50,00 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util xvda 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 xvdb 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 xvdc 0,00 0,00...

...random IO, ca 10 - 20 G) it effectively turns in to a writethrough cache although there's much space left on the cachedlv. When working as expected on KVM host all writes go to SSDs iostat -x -m 2 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 324.50 0.00 22.00 0.00 14.94 1390.57 1.90 86.39 0.00 86.39 5.32 11.70 sdb 0.00 324.50 0.00 22.00 0.00 14.94 1390.57 2.03 92.45 0.00 92.45 5.48 12.05 sdc 0.00 3932.00...

...), and so far everything I tried results in sub-optimal performance. There are many files - mostly images, several million - and many operations take minutes, copying multiple files (even if they're small) suddenly freezes up for seconds at a time, then continues, iostat frequently shows large r_await and w_awaits with 100% utilization for the attached block device, etc. But anyway, there are many guides out there for small-file performance improvements, but more explanation is needed, and I think more tweaks should be possible. My question today is about performance.cache-size. Is this a size...

...turns in to a writethrough cache although > there's much space left on the cachedlv. > > > When working as expected on KVM host all writes go to SSDs > > iostat -x -m 2 > > Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz > avgqu-sz await r_await w_await svctm %util > sda 0.00 324.50 0.00 22.00 0.00 14.94 > 1390.57 1.90 86.39 0.00 86.39 5.32 11.70 > sdb 0.00 324.50 0.00 22.00 0.00 14.94 > 1390.57 2.03 92.45 0.00 92.45 5.48 12.05 > sdc...

...ied results in sub-optimal > performance. > > There are many files - mostly images, several million - and many > operations take minutes, copying multiple files (even if they're small) > suddenly freezes up for seconds at a time, then continues, iostat > frequently shows large r_await and w_awaits with 100% utilization for the > attached block device, etc. > > But anyway, there are many guides out there for small-file performance > improvements, but more explanation is needed, and I think more tweaks > should be possible. > > My question today is about perfo...

...al >> performance. >> >> There are many files - mostly images, several million - and many >> operations take minutes, copying multiple files (even if they're small) >> suddenly freezes up for seconds at a time, then continues, iostat >> frequently shows large r_await and w_awaits with 100% utilization for the >> attached block device, etc. >> >> But anyway, there are many guides out there for small-file performance >> improvements, but more explanation is needed, and I think more tweaks >> should be possible. >> >> My q...

...nce. >>> >>> There are many files - mostly images, several million - and many >>> operations take minutes, copying multiple files (even if they're small) >>> suddenly freezes up for seconds at a time, then continues, iostat >>> frequently shows large r_await and w_awaits with 100% utilization for the >>> attached block device, etc. >>> >>> But anyway, there are many guides out there for small-file performance >>> improvements, but more explanation is needed, and I think more tweaks >>> should be possible. &g...

...itethrough cache although there's much space left on the > cachedlv. > > > When working as expected on KVM host all writes go to SSDs > > iostat -x -m 2 > > Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s > avgrq-sz avgqu-sz await r_await w_await svctm %util > sda 0.00 324.50 0.00 22.00 0.00 14.94 > 1390.57 1.90 86.39 0.00 86.39 5.32 11.70 > sdb 0.00 324.50 0.00 22.00 0.00 14.94 > 1390.57 2.03 92.45 0.00 92.45 5.48 12.05 >...

...; >>>> There are many files - mostly images, several million - and many >>>> operations take minutes, copying multiple files (even if they're small) >>>> suddenly freezes up for seconds at a time, then continues, iostat >>>> frequently shows large r_await and w_awaits with 100% utilization for the >>>> attached block device, etc. >>>> >>>> But anyway, there are many guides out there for small-file performance >>>> improvements, but more explanation is needed, and I think more tweaks >>>> sh...

...al >> performance. >> >> There are many files - mostly images, several million - and many >> operations take minutes, copying multiple files (even if they're small) >> suddenly freezes up for seconds at a time, then continues, iostat >> frequently shows large r_await and w_awaits with 100% utilization for the >> attached block device, etc. >> >> But anyway, there are many guides out there for small-file performance >> improvements, but more explanation is needed, and I think more tweaks >> should be possible. >> >> My q...

...t;> There are many files - mostly images, several million - and many >>>>> operations take minutes, copying multiple files (even if they're small) >>>>> suddenly freezes up for seconds at a time, then continues, iostat >>>>> frequently shows large r_await and w_awaits with 100% utilization for the >>>>> attached block device, etc. >>>>> >>>>> But anyway, there are many guides out there for small-file performance >>>>> improvements, but more explanation is needed, and I think more tweaks &gt...

...are many files - mostly images, several million - and many >>>>>> operations take minutes, copying multiple files (even if they're small) >>>>>> suddenly freezes up for seconds at a time, then continues, iostat >>>>>> frequently shows large r_await and w_awaits with 100% utilization for the >>>>>> attached block device, etc. >>>>>> >>>>>> But anyway, there are many guides out there for small-file >>>>>> performance improvements, but more explanation is needed, and I think...

...million - and many operations take minutes, > copying multiple files (even if they're small) > suddenly freezes up for seconds at a time, > then continues, iostat frequently shows large > r_await and w_awaits with 100% utilization for > the attached block device, etc. > > But anyway, there are many guides out there > for small-file performance improvements, but > more explanation i...

...les - mostly images, several million - and many >>>>>>> operations take minutes, copying multiple files (even if they're small) >>>>>>> suddenly freezes up for seconds at a time, then continues, iostat >>>>>>> frequently shows large r_await and w_awaits with 100% utilization for the >>>>>>> attached block device, etc. >>>>>>> >>>>>>> But anyway, there are many guides out there for small-file >>>>>>> performance improvements, but more explanation is nee...

...images, several million - and many >>>>>>>> operations take minutes, copying multiple files (even if they're small) >>>>>>>> suddenly freezes up for seconds at a time, then continues, iostat >>>>>>>> frequently shows large r_await and w_awaits with 100% utilization for the >>>>>>>> attached block device, etc. >>>>>>>> >>>>>>>> But anyway, there are many guides out there for small-file >>>>>>>> performance improvements, but more ex...