Hi Users I am using RHEL 6.5 on my server. From top command i can see that the processors in my server are spending a lot of time on wait for I/O. I can see high percentage in terms of 30-50% on "wa" time. Here is the df output about the disk space in my system: ********** [root at localhost images]# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda1 1915071844 103227908 1714563936 6% / tmpfs 32931472 0 32931472 0% /dev/shm ********** Could someone point me on how to improve the disk I/O on my server and reduce the wait time on I/O. Thanks Jatin
On 01/28/2015 01:32 PM, Jatin Davey wrote:> Hi Users > > I am using RHEL 6.5 on my server. > > From top command i can see that the processors in my server are > spending a lot of time on wait for I/O. > I can see high percentage in terms of 30-50% on "wa" time. > > Here is the df output about the disk space in my system: > > ********** > [root at localhost images]# df > Filesystem 1K-blocks Used Available Use% Mounted on > /dev/sda1 1915071844 103227908 1714563936 6% / > tmpfs 32931472 0 32931472 0% /dev/shm > ********** > > Could someone point me on how to improve the disk I/O on my server and > reduce the wait time on I/O.you could use iotop or vmstat to see what processes are causing the IO.
+1 And remember that I/O is more than just disk. The atop monitor gives you information like top and htop, but also provides a lot of I/O information as well including network. Perhaps your server is the target of a network-based DDOS attack which can cause lots of I/O wait time. Also look at top to see what numbers are in the si and hi columns. THese stand for software and hardware interrupts. If one of those is high you can also narrow it down. Hope this helps. On 01/28/2015 07:51 AM, Nicolas Thierry-Mieg wrote:> On 01/28/2015 01:32 PM, Jatin Davey wrote: >> Hi Users >> >> I am using RHEL 6.5 on my server. >> >> From top command i can see that the processors in my server are >> spending a lot of time on wait for I/O. >> I can see high percentage in terms of 30-50% on "wa" time. >> >> Here is the df output about the disk space in my system: >> >> ********** >> [root at localhost images]# df >> Filesystem 1K-blocks Used Available Use% Mounted on >> /dev/sda1 1915071844 103227908 1714563936 6% / >> tmpfs 32931472 0 32931472 0% /dev/shm >> ********** >> >> Could someone point me on how to improve the disk I/O on my server and >> reduce the wait time on I/O. > > you could use iotop or vmstat to see what processes are causing the IO. > > _______________________________________________ > CentOS mailing list > CentOS at centos.org > http://lists.centos.org/mailman/listinfo/centos > > -- > > > ********************************************************* > David P. Both, RHCE > Millennium Technology Consulting LLC > Raleigh, NC, USA > 919-389-8678 > > dboth at millennium-technology.com > > www.millennium-technology.com > www.databook.bz - Home of the DataBook for Linux > DataBook is a Registered Trademark of David Both > ********************************************************* > This communication may be unlawfully collected and stored by the National > Security Agency (NSA) in secret. The parties to this email do not consent to the > retrieving or storing of this communication and any related metadata, as well as > printing, copying, re-transmitting, disseminating, or otherwise using it. If you > believe you have received this communication in error, please delete it > immediately. >
On 01/28/2015 03:32 PM, Jatin Davey wrote:> > Could someone point me on how to improve the disk I/O on my server and > reduce the wait time on I/O.Fisrt of all, you should ensure how fast are your disks. There are several methods to check that: https://www.thomas-krenn.com/en/wiki/Linux_I/O_Performance_Tests_using_dd But there are several aother way to achieve measurement. If you notice some poor performance, then you can assume your disks are slow.
On 1/28/2015 4:32 AM, Jatin Davey wrote:> I am using RHEL 6.5 on my server. > > From top command i can see that the processors in my server are > spending a lot of time on wait for I/O. > I can see high percentage in terms of 30-50% on "wa" time. > > Here is the df output about the disk space in my system: > > ********** > [root at localhost images]# df > Filesystem 1K-blocks Used Available Use% Mounted on > /dev/sda1 1915071844 103227908 1714563936 6% / > tmpfs 32931472 0 32931472 0% /dev/shm > ********** > > Could someone point me on how to improve the disk I/O on my server and > reduce the wait time on I/O.it appears you have everything on a single file system on a single 2TB drive? I'm going to guess thats a desktop grade SATA disk spinning at most at 7200rpm (many are slower, 'green' disks are often 5400rpm). what sort of workload are you running? our high disk IO servers, doing things like transactional databases, have many 10000rpm or 15000 rpm drives on a SAS2 raid card with 1GB or 2GB of write-back-cache (battery or flash backed). -- john r pierce 37N 122W somewhere on the middle of the left coast
On 1/28/2015 9:37 AM, John R Pierce wrote:> On 1/28/2015 4:32 AM, Jatin Davey wrote: >> .... >> >> Could someone point me on how to improve the disk I/O on my server >> and reduce the wait time on I/O. > ...oops, left off the last part. To improve disk IO performance, use more/faster disks, or do less disk IO. Since you've given us no clues about your hardware configuration OR software workload, its hard to be more specific. -- john r pierce 37N 122W somewhere on the middle of the left coast
On 01/28/2015 04:32 AM, Jatin Davey wrote:> Could someone point me on how to improve the disk I/O on my server and > reduce the wait time on I/O.Start by identifying your disk and controller. Assuming that this is a single SATA disk: # smartctl -a /dev/sda | egrep 'Model:|Rate:|SATA Version' # lspci | grep SATA Next install and run iotop. If there's something on your system that's particularly active and unnecessary, disable it. If everything contending for your disk *is* necessary, then you may need to take steps to improve disk performance. There are a number of possibilities. 1: Use an SSD instead of a disk. 2: Use an SSD in addition to a disk. If a purely SSD storage system is too expensive, you might choose to use an SSD for one or more filesytems that you can mount where the high volume IO is occurring, and continue to use the disk for filesystems that need more data capacity and fewer IOPS. 3: Use a fast RAID array. A RAID1 array can provide better read performance than a single disk. For higher write volumes, use RAID10 on four or more disks. Avoid RAID 5 or 6, as they'll be slower than a single disk. 4: Replace poorly supported hardware. It might be that your SATA controller isn't well supported, and performs badly as a result. I used to have a desktop system with NVidia motherboard chipset. Its SATA controller was garbage. Performance with the same disk was noticeably better with a cheap PCI-E SATA controller.
On 1/29/2015 4:50 AM, Gordon Messmer wrote:> On 01/28/2015 04:32 AM, Jatin Davey wrote: >> Could someone point me on how to improve the disk I/O on my server and >> reduce the wait time on I/O. > > Start by identifying your disk and controller. Assuming that this is > a single SATA disk: > > # smartctl -a /dev/sda | egrep 'Model:|Rate:|SATA Version'[Jatin] [root at localhost images]# smartctl -a /dev/sda | egrep 'Model:|Rate:|SATA Version' [root at localhost images]#> # lspci | grep SATA[Jatin] [root at localhost images]# lspci | grep SATA 00:11.4 SATA controller: Intel Corporation Wellsburg sSATA Controller [AHCI mode] (rev 05) 00:1f.2 SATA controller: Intel Corporation Wellsburg 6-Port SATA Controller [AHCI mode] (rev 05)> Next install and run iotop. If there's something on your system > that's particularly active and unnecessary, disable it. > > If everything contending for your disk *is* necessary, then you may > need to take steps to improve disk performance. There are a number of > possibilities. > > 1: Use an SSD instead of a disk. > > 2: Use an SSD in addition to a disk. If a purely SSD storage system > is too expensive, you might choose to use an SSD for one or more > filesytems that you can mount where the high volume IO is occurring, > and continue to use the disk for filesystems that need more data > capacity and fewer IOPS.[Jatin] Thanks for these inputs. I will surely take it up with my team to see if we can incorporate new hardware for our product.> > 3: Use a fast RAID array. A RAID1 array can provide better read > performance than a single disk. For higher write volumes, use RAID10 > on four or more disks. Avoid RAID 5 or 6, as they'll be slower than a > single disk.[Jatin] We are using RAID 10 in our server with 4 disks.> > 4: Replace poorly supported hardware. It might be that your SATA > controller isn't well supported, and performs badly as a result. I > used to have a desktop system with NVidia motherboard chipset. Its > SATA controller was garbage. Performance with the same disk was > noticeably better with a cheap PCI-E SATA controller. > > > _______________________________________________ > CentOS mailing list > CentOS at centos.org > http://lists.centos.org/mailman/listinfo/centos >Hi Gordon Please see my responses inline to your questions. Appreciate your inputs in this direction. Thanks Jatin
On 1/28/2015 11:07 PM, John R Pierce wrote:> On 1/28/2015 4:32 AM, Jatin Davey wrote: >> I am using RHEL 6.5 on my server. >> >> From top command i can see that the processors in my server are >> spending a lot of time on wait for I/O. >> I can see high percentage in terms of 30-50% on "wa" time. >> >> Here is the df output about the disk space in my system: >> >> ********** >> [root at localhost images]# df >> Filesystem 1K-blocks Used Available Use% Mounted on >> /dev/sda1 1915071844 103227908 1714563936 6% / >> tmpfs 32931472 0 32931472 0% /dev/shm >> ********** >> >> Could someone point me on how to improve the disk I/O on my server >> and reduce the wait time on I/O. > > it appears you have everything on a single file system on a single 2TB > drive?[Jatin] Yes , entire disk is mounted on a single file system. Will this cause high wait time for I/O operations ? Do you think splitting it into multiple partitions would help ? Appreciate if you can enlighten me in this regard or provide pointers to it.> I'm going to guess thats a desktop grade SATA disk spinning at most at > 7200rpm (many are slower, 'green' disks are often 5400rpm).[Jatin] Yes , it is a SATA disk. I am not sure of the speed. Can you tell me how to find out this information ? Additionally we are using RAID 10 configuration with 4 disks.> > what sort of workload are you running?[Jatin] Our workload is a bit high on disk reads / writes and hence i am seeing slowness in it when using the top command. We basically run VMs on the RHEL 6.5 host. Our target is to house 5-6 application VMs on this host and ship this as a product on a rack server.> our high disk IO servers, doing things like transactional databases, > have many 10000rpm or 15000 rpm drives on a SAS2 raid card with 1GB or > 2GB of write-back-cache (battery or flash backed). > >Hi John Please see my responses inline. Appreciate your inputs in this regard. Thanks Jatin