Vanja Hrustic
2006-Mar-23 14:37 UTC
[CentOS] Extremely slow disk transfers with SATA disks
I have installed CentOS 4.2, recently, on new computer. Everything seemed to work fine. However, we''ve found out that disk transfers are incredibly slow, and I just can''t figure out what to do in order to fix it. Computer is now running CentOS 4.3 (updated it 1 hour ago), and same thing is still present. I hoped kernel upgrade might fix it, but it didn''t. Computer has 2 SATA disks, 200GB (Maxtor 6Y200M0) and 80GB (Maxtor 6L080M0). There are no errors or any other indications of problems in logs/dmesg. Copying 500MB file takes around 210 seconds. Around 2.3MB/sec. Doesn''t matter if I am copying from one hard disk to another, or onto the same hard disk. When copying, vmstat shows: r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 160 525928 1060 278044 0 0 1609 1112 1387 1146 9 4 39 47 0 0 160 525928 1060 278044 0 0 0 0 1084 745 0 0 100 0 0 0 160 525844 1060 278044 0 0 0 0 1067 795 2 0 98 0 0 0 160 525844 1060 278044 0 0 0 0 1084 745 1 0 99 0 0 0 160 525844 1072 278044 0 0 0 156 1073 757 0 0 100 0 0 1 160 525564 1148 278228 0 0 240 0 1129 1136 2 1 77 20 0 1 160 522708 1176 281256 0 0 1600 0 1283 1155 0 2 0 98 0 1 160 519380 1184 284568 0 0 1672 0 1327 1231 0 2 0 98 3 1 160 516404 1196 287368 0 0 1412 0 1264 1126 1 1 0 98 0 1 160 513524 1196 290424 0 0 1536 0 1353 1926 11 2 0 87 0 3 160 510132 1208 293544 0 0 1544 7348 1407 2163 24 6 0 71 0 1 160 513780 1216 289972 0 0 1412 140 1360 1536 9 3 0 88 2 0 160 515316 1220 288464 0 0 1104 0 1341 2067 47 4 0 49 0 1 160 518996 1216 284608 0 0 1712 0 1333 1676 34 7 0 59 0 1 160 516820 1224 286832 0 0 2052 0 1318 2025 25 4 0 71 1 2 160 512388 1228 291024 0 0 2952 8736 1433 1431 6 4 0 90 2 1 160 504772 1244 295304 0 0 3076 76 1382 1453 15 5 0 80 0 1 160 502340 1252 301096 0 0 3332 0 1358 1624 7 5 0 88 0 1 160 495236 1260 308216 0 0 3588 0 1323 1631 2 3 0 95 1 1 160 487620 1268 315640 0 0 3716 0 1355 1175 7 2 0 91 0 2 160 480708 1276 322120 0 0 3204 16732 1423 1354 2 5 0 93 You can see text version here (in case this got wrapped): http://www.vanja.com/vmstat.txt I''ve tried vmstat on my machine, with IDE disks, and I''ve noticed that I ave much less ''cs'' (context switches), although it might be because I''m not having as many processes in the background (problematic computer is running KDE :). On my computer, number of blocks in (bi) gets into tens of thousands, while problematic computer has 1500-3000 bs/second. Does anyone know what I might do in order to find what the problem might be? I couldn''t find any SATA specific tools, which might help with troubleshooting. hdparm -tT tests show: /dev/sda: Timing cached reads: 2408 MB in 2.00 seconds = 1202.98 MB/sec Timing buffered disk reads: 150 MB in 3.02 seconds = 49.61 MB/sec /dev/sdb: Timing cached reads: 2628 MB in 2.00 seconds = 1313.54 MB/sec Timing buffered disk reads: 166 MB in 3.02 seconds = 54.88 MB/sec All filesystems are ext3. This "problematic" computer has nforce3 motherboard, I have nforce2. Same CentOS, but no SATA on my computer. This really affects complete performance of computer, and any help or ideas are welcome. Btw, no problems with transfer under Windows :( Thanks. Vanja
Chris Mauritz
2006-Mar-23 14:44 UTC
[CentOS] Extremely slow disk transfers with SATA disks
Vanja Hrustic wrote:> I have installed CentOS 4.2, recently, on new computer. > > Everything seemed to work fine. > > However, we''ve found out that disk transfers are incredibly slow, and I > just can''t figure out what to do in order to fix it. > > Computer is now running CentOS 4.3 (updated it 1 hour ago), and same > thing is still present. I hoped kernel upgrade might fix it, but it > didn''t. >I am having similar issues with an Intel 845 based board. It sounds like your disks are running in PIO rather than in UDMA mode. Have you tried: hdparm -d1 /dev/sdX substituting the actual device names for each drive? I was unable to remotely fix mine, but plan to visit the machine soon and fiddle with bios settings to see if I can get the drives out of PIO mode. Cheers,
Vanja Hrustic
2006-Mar-23 14:53 UTC
[CentOS] Extremely slow disk transfers with SATA disks
On Thu, 23 Mar 2006 09:44:09 -0500 Chris Mauritz <chrism@imntv.com> wrote:> Vanja Hrustic wrote: > > I have installed CentOS 4.2, recently, on new computer. > > > > Everything seemed to work fine. > > > > However, we''ve found out that disk transfers are incredibly slow, > > and I just can''t figure out what to do in order to fix it. > > > > Computer is now running CentOS 4.3 (updated it 1 hour ago), and same > > thing is still present. I hoped kernel upgrade might fix it, but it > > didn''t. > > > > I am having similar issues with an Intel 845 based board. It sounds > like your disks are running in PIO rather than in UDMA mode.Erm, if I''m not wrong, SATA uses DMA ''by default'', it''s not something we can change. Thanks anyway :) Vanja
Chris Mauritz
2006-Mar-23 14:56 UTC
[CentOS] Extremely slow disk transfers with SATA disks
Vanja Hrustic wrote:> On Thu, 23 Mar 2006 09:44:09 -0500 > Chris Mauritz <chrism@imntv.com> wrote: > > >> Vanja Hrustic wrote: >> >>> I have installed CentOS 4.2, recently, on new computer. >>> >>> Everything seemed to work fine. >>> >>> However, we''ve found out that disk transfers are incredibly slow, >>> and I just can''t figure out what to do in order to fix it. >>> >>> Computer is now running CentOS 4.3 (updated it 1 hour ago), and same >>> thing is still present. I hoped kernel upgrade might fix it, but it >>> didn''t. >>> >>> >> I am having similar issues with an Intel 845 based board. It sounds >> like your disks are running in PIO rather than in UDMA mode. >> > > Erm, if I''m not wrong, SATA uses DMA ''by default'', it''s not something > we can change. > > Thanks anyway :) >Could be. I''m certainly not an expert on the subject. If/when you get it fixed, I''d appreciate it if you''d post your solution. Maybe it will help others (like me!) out. Cheers,
On Thu, 2006-03-23 at 15:37 +0100, Vanja Hrustic wrote:> I have installed CentOS 4.2, recently, on new computer. > > Everything seemed to work fine. > > However, we''ve found out that disk transfers are incredibly slow, and I > just can''t figure out what to do in order to fix it. > > Computer is now running CentOS 4.3 (updated it 1 hour ago), and same > thing is still present. I hoped kernel upgrade might fix it, but it > didn''t. > > Computer has 2 SATA disks, 200GB (Maxtor 6Y200M0) and 80GB (Maxtor > 6L080M0). > > There are no errors or any other indications of problems in logs/dmesg. > > Copying 500MB file takes around 210 seconds. Around 2.3MB/sec. > > Doesn''t matter if I am copying from one hard disk to another, or onto > the same hard disk. > > When copying, vmstat shows: > > r b swpd free buff cache si so bi bo in cs us sy id wa > 0 0 160 525928 1060 278044 0 0 1609 1112 1387 1146 9 4 39 47 > 0 0 160 525928 1060 278044 0 0 0 0 1084 745 0 0 100 0 > 0 0 160 525844 1060 278044 0 0 0 0 1067 795 2 0 98 0 > 0 0 160 525844 1060 278044 0 0 0 0 1084 745 1 0 99 0 > 0 0 160 525844 1072 278044 0 0 0 156 1073 757 0 0 100 0 > 0 1 160 525564 1148 278228 0 0 240 0 1129 1136 2 1 77 20 > 0 1 160 522708 1176 281256 0 0 1600 0 1283 1155 0 2 0 98 > 0 1 160 519380 1184 284568 0 0 1672 0 1327 1231 0 2 0 98 > 3 1 160 516404 1196 287368 0 0 1412 0 1264 1126 1 1 0 98 > 0 1 160 513524 1196 290424 0 0 1536 0 1353 1926 11 2 0 87 > 0 3 160 510132 1208 293544 0 0 1544 7348 1407 2163 24 6 0 71 > 0 1 160 513780 1216 289972 0 0 1412 140 1360 1536 9 3 0 88 > 2 0 160 515316 1220 288464 0 0 1104 0 1341 2067 47 4 0 49 > 0 1 160 518996 1216 284608 0 0 1712 0 1333 1676 34 7 0 59 > 0 1 160 516820 1224 286832 0 0 2052 0 1318 2025 25 4 0 71 > 1 2 160 512388 1228 291024 0 0 2952 8736 1433 1431 6 4 0 90 > 2 1 160 504772 1244 295304 0 0 3076 76 1382 1453 15 5 0 80 > 0 1 160 502340 1252 301096 0 0 3332 0 1358 1624 7 5 0 88 > 0 1 160 495236 1260 308216 0 0 3588 0 1323 1631 2 3 0 95 > 1 1 160 487620 1268 315640 0 0 3716 0 1355 1175 7 2 0 91 > 0 2 160 480708 1276 322120 0 0 3204 16732 1423 1354 2 5 0 93 > > You can see text version here (in case this got wrapped): > > http://www.vanja.com/vmstat.txt > > I''ve tried vmstat on my machine, with IDE disks, and I''ve noticed that I > ave much less ''cs'' (context switches), although it might be because I''m > not having as many processes in the background (problematic computer > is running KDE :). > > On my computer, number of blocks in (bi) gets into tens of thousands, > while problematic computer has 1500-3000 bs/second. > > Does anyone know what I might do in order to find what the problem > might be? > > I couldn''t find any SATA specific tools, which might help with > troubleshooting. hdparm -tT tests show: > > /dev/sda: > Timing cached reads: 2408 MB in 2.00 seconds = 1202.98 MB/sec > Timing buffered disk reads: 150 MB in 3.02 seconds = 49.61 MB/sec > > /dev/sdb: > Timing cached reads: 2628 MB in 2.00 seconds = 1313.54 MB/sec > Timing buffered disk reads: 166 MB in 3.02 seconds = 54.88 MB/sec > > All filesystems are ext3. > > This "problematic" computer has nforce3 motherboard, I have nforce2. > Same CentOS, but no SATA on my computer. > > This really affects complete performance of computer, and any help or > ideas are welcome. > > Btw, no problems with transfer under Windows :( > > Thanks. > > VanjaYour hdparm numbers are slightly better than mine for the same drive. For me: copying a 3.2GB file achieved 18.4MB/s I guess hdparm is not a real-world indicator. What does top tell you? Is something hogging the CPU. Bob...
Peter Kjellström
2006-Mar-23 15:54 UTC
[CentOS] Extremely slow disk transfers with SATA disks
On Thursday 23 March 2006 15:37, Vanja Hrustic wrote:> I have installed CentOS 4.2, recently, on new computer. > > Everything seemed to work fine. > > However, we''ve found out that disk transfers are incredibly slow, and I > just can''t figure out what to do in order to fix it. > > Computer is now running CentOS 4.3 (updated it 1 hour ago), and same > thing is still present. I hoped kernel upgrade might fix it, but it > didn''t.Very interesting, you do have working sata drivers (your device is /dev/sd*) and your hdparm -t gives expected number and yet you see really bad performance... wow. Have you tried to figure out if it''s reading or writing or both that suck? try this: time dd if=/dev/zero of=testfile bs=1M count=1000 time dd if=testfile of=/dev/null bs=1M If you have 1 gig ram or more adjust count... /Peter -- ------------------------------------------------------------ Peter Kjellstr?m | E-mail: cap@nsc.liu.se National Supercomputer Centre | Sweden | http://www.nsc.liu.se -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 191 bytes Desc: not available Url : http://lists.centos.org/pipermail/centos/attachments/20060323/136765c2/attachment.bin
vanja@pobox.com
2006-Mar-24 07:31 UTC
[CentOS] Extremely slow disk transfers with SATA disks
On Thu, Mar 23, 2006 at 04:54:44PM +0100, Peter Kjellstr?m wrote:> On Thursday 23 March 2006 15:37, Vanja Hrustic wrote: > > I have installed CentOS 4.2, recently, on new computer. > > > > Everything seemed to work fine. > > > > However, we''ve found out that disk transfers are incredibly slow, and I > > just can''t figure out what to do in order to fix it. > > > > Computer is now running CentOS 4.3 (updated it 1 hour ago), and same > > thing is still present. I hoped kernel upgrade might fix it, but it > > didn''t. > > Very interesting, you do have working sata drivers (your device is /dev/sd*) > and your hdparm -t gives expected number and yet you see really bad > performance... wow. > > Have you tried to figure out if it''s reading or writing or both that suck? try > this: > > time dd if=/dev/zero of=testfile bs=1M count=1000 > > time dd if=testfile of=/dev/null bs=1M > > If you have 1 gig ram or more adjust count...As usual, I get to test more things only after I make a post to the mailing list :) What I found out is that 1 disk is creating problems (out of those 2), and only when reading. So, that 200GB Maxtor has no problems writing (300MB file gets written in few seconds, when it''s read from another 80GB SATA disk), but when I read something from that disk - I get 2MB/sec. I have no trouble reading from 80GB SATA disk, so I guess it will be cable/hardware related. Also, I said it works ok in Windows, but Windows partition is located on 80GB drive. 200GB drive only holds Linux partitions. I have no idea why hdparm reports 50MB/sec rates on 200GB disk, but I''ll just assume that hdparm doesn''t use "real life" way of testing speed :) Thanks for feedback. I will post again when/if I fix this, and figure out if it was indeed a hardware problem. Take care. Vanja
Peter Kjellström
2006-Mar-24 11:13 UTC
[CentOS] Extremely slow disk transfers with SATA disks
On Friday 24 March 2006 08:31, vanja@pobox.com wrote:> On Thu, Mar 23, 2006 at 04:54:44PM +0100, Peter Kjellstr?m wrote: > > On Thursday 23 March 2006 15:37, Vanja Hrustic wrote: > > > ... > > Have you tried to figure out if it''s reading or writing or both that > > suck? try this: > > > > time dd if=/dev/zero of=testfile bs=1M count=1000 > > > > time dd if=testfile of=/dev/null bs=1M > > > > If you have 1 gig ram or more adjust count... > > As usual, I get to test more things only after I make a post to the mailing > list :)fwiw, it was a very good bugreport making progress easy.> What I found out is that 1 disk is creating problems (out of those 2), and > only when reading. > > So, that 200GB Maxtor has no problems writing (300MB file gets written in > few seconds, when it''s read from another 80GB SATA disk), but when I read > something from that disk - I get 2MB/sec. I have no trouble reading from > 80GB SATA disk, so I guess it will be cable/hardware related. > > Also, I said it works ok in Windows, but Windows partition is located on > 80GB drive. 200GB drive only holds Linux partitions. > > I have no idea why hdparm reports 50MB/sec rates on 200GB disk, but I''ll > just assume that hdparm doesn''t use "real life" way of testing speed :)It just reads, and only at the beginning of the device. /Peter> > Thanks for feedback. I will post again when/if I fix this, and figure out > if it was indeed a hardware problem. > > Take care. > > Vanja-- ------------------------------------------------------------ Peter Kjellstr?m | National Supercomputer Centre | Sweden | http://www.nsc.liu.se -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 191 bytes Desc: not available Url : http://lists.centos.org/pipermail/centos/attachments/20060324/f3856805/attachment.bin
On Fri, 2006-03-24 at 01:31, vanja@pobox.com wrote:> So, that 200GB Maxtor has no problems writing (300MB file gets written in few seconds, when it''s read from another 80GB SATA > disk), but when I read something from that disk - I get 2MB/sec. I have no trouble reading from 80GB SATA disk, so I guess > it will be cable/hardware related.Writing is always buffered and unless you do some kind of fsync you don''t know when it is committed back to disk. Your timing will stop when it goes to the RAM buffer. Reading is buffered too, but the first read obviously has to wait for the disk.> I have no idea why hdparm reports 50MB/sec rates on 200GB disk, but I''ll just assume that hdparm doesn''t use "real life" way > of testing speed :)Change your tests to use large enough samples that you can''t buffer a significant part in RAM. Anyway the usual problem with IDE/SATA vs. SCSI is not sustained transfer in a single operation, it is that scsi will accept and buffer multiple commands while most ide/sata drives (the WD raptor being an exception) can only have a single command waiting for completion - and with IDE that is per controller. -- Les Mikesell lesmikesell@gmail.com
Scott Silva
2006-Mar-28 00:44 UTC
[CentOS] Re: Extremely slow disk transfers with SATA disks
Vanja Hrustic spake the following on 3/23/2006 6:53 AM:> On Thu, 23 Mar 2006 09:44:09 -0500 > Chris Mauritz <chrism@imntv.com> wrote: > >> Vanja Hrustic wrote: >>> I have installed CentOS 4.2, recently, on new computer. >>> >>> Everything seemed to work fine. >>> >>> However, we''ve found out that disk transfers are incredibly slow, >>> and I just can''t figure out what to do in order to fix it. >>> >>> Computer is now running CentOS 4.3 (updated it 1 hour ago), and same >>> thing is still present. I hoped kernel upgrade might fix it, but it >>> didn''t. >>> >> I am having similar issues with an Intel 845 based board. It sounds >> like your disks are running in PIO rather than in UDMA mode. > > Erm, if I''m not wrong, SATA uses DMA ''by default'', it''s not something > we can change. > > Thanks anyway :) > > VanjaBut many bios''s have a pata/sata mode that can confuse the hell out of Linux. I can''t remember the specifics, but the setting should be noticeable in the bios settings, -- MailScanner is like deodorant... You hope everybody uses it, and you notice quickly if they don''t!!!!