I am seeing what seems to be a notable limit on read performance of an ext3 filesystem. If anyone could offer some insight it would be helpful. Background: 12 x 500G SATA disks in a Hardware RAID enclosure connected via 2Gb/s FC to a 4 x 2.6 Ghz system with 4GB ram running RHEL4.5. Initially the enclosure was configured RAID5 10+1 parity, although I've also tried RAID 50 and currently RAID 0. I've varied chunk sizes from 64-256K. Problem: No matter what I do I cannot get the ext3 read performance above ~90MB/s. Under virtually every configuration listed above the write performance is greater than the read performance. I've run a large number of Bonnie++ and IOzone tests, but for the sake of simplicity in this email I'll just refer to simple dd's with /dev/zero. Details: Under the current RAID0 setup I see the following when dd'ing. DD 4G from /dev/zero to /dev/sdd disk (no filesystem) & sync 28 seconds DD 4G from /dev/sdd to /dev/null 32 seconds DD 4G to ext3 on /dev/sdd & sync 32 seconds DD 4G from ext3 file to /dev/null 48 seconds. I've been watching the port usage on the FC switch and it verifies what I am seeing, Writes max out near 2Gb/s but reads hit some artificial limit around 90 MB/s and never ever exceed it with the filesystem, regardless of they underlying RAID configuration. Without a filesystem the reads are atleast 50% faster, and it can be seen on the FC switch graphs as well. Any help or thoughts would be appreciated. Thanks, ~Glen
Hi, Did you see any difference when different block size is used (for example, dd with bs=64k or 128k)? Try also change the read-ahead cache. blockdev --getra /dev/sdd to see what is the current value, and blockdev --setra 8192 /dev/sdd to change it. 8192 is a good number that has been working well for me for the similar size setup. ... ling Glen Dosey wrote:> I am seeing what seems to be a notable limit on read performance of an > ext3 filesystem. If anyone could offer some insight it would be helpful. > > Background: > 12 x 500G SATA disks in a Hardware RAID enclosure connected via 2Gb/s FC > to a 4 x 2.6 Ghz system with 4GB ram running RHEL4.5. Initially the > enclosure was configured RAID5 10+1 parity, although I've also tried > RAID 50 and currently RAID 0. I've varied chunk sizes from 64-256K. > > Problem: > No matter what I do I cannot get the ext3 read performance above > ~90MB/s. Under virtually every configuration listed above the write > performance is greater than the read performance. I've run a large > number of Bonnie++ and IOzone tests, but for the sake of simplicity in > this email I'll just refer to simple dd's with /dev/zero. > > Details: > Under the current RAID0 setup I see the following when dd'ing. > > DD 4G from /dev/zero to /dev/sdd disk (no filesystem) & sync > 28 seconds > DD 4G from /dev/sdd to /dev/null 32 seconds > DD 4G to ext3 on /dev/sdd & sync 32 seconds > DD 4G from ext3 file to /dev/null 48 seconds. > > I've been watching the port usage on the FC switch and it verifies what > I am seeing, Writes max out near 2Gb/s but reads hit some artificial > limit around 90 MB/s and never ever exceed it with the filesystem, > regardless of they underlying RAID configuration. Without a filesystem > the reads are atleast 50% faster, and it can be seen on the FC switch > graphs as well. > > Any help or thoughts would be appreciated. > > Thanks, > ~Glen > > > _______________________________________________ > Ext3-users mailing list > Ext3-users at redhat.com > https://www.redhat.com/mailman/listinfo/ext3-users >
ling, Wow, Thanks ! That made an enormous difference. Now I'd like to try and understand why. I had initially tried 256, 512 and 1024 block read ahead settings in the OS with no difference in performance and went no further. 8192 made a huge difference, but 16384 brings the performance back to the same at 256. Why is 8192 such a magic number ? The RAID enclosure itself also has a read ahead setting , which I've tried at 512K , 1M and 2M, also with no difference. I also want to make sure I understand how read-ahead is working in my setup. If I request some data A, the OS will request 8192 blocks (4MB) past the end of data A. Now the controller will see the OS request for A + 4M and read an additional 2MB past that, such that the disks have read 6MB beyond the end of what was initially requested, with 2MB being in the controller cache, 4MB in the OS cache, and data A passed to the application. Is this correct ? Thanks, Glen On Sat, 2007-06-30 at 00:18 -0500, Ling C. Ho wrote:> Hi, > > Did you see any difference when different block size is used (for > example, dd with bs=64k or 128k)? Try also change the read-ahead cache. > blockdev --getra /dev/sdd to see what is the current value, and blockdev > --setra 8192 /dev/sdd to change it. 8192 is a good number that has been > working well for me for the similar size setup. > > ... > ling > > Glen Dosey wrote: > > I am seeing what seems to be a notable limit on read performance of an > > ext3 filesystem. If anyone could offer some insight it would be helpful. > > > > Background: > > 12 x 500G SATA disks in a Hardware RAID enclosure connected via 2Gb/s FC > > to a 4 x 2.6 Ghz system with 4GB ram running RHEL4.5. Initially the > > enclosure was configured RAID5 10+1 parity, although I've also tried > > RAID 50 and currently RAID 0. I've varied chunk sizes from 64-256K. > > > > Problem: > > No matter what I do I cannot get the ext3 read performance above > > ~90MB/s. Under virtually every configuration listed above the write > > performance is greater than the read performance. I've run a large > > number of Bonnie++ and IOzone tests, but for the sake of simplicity in > > this email I'll just refer to simple dd's with /dev/zero. > > > > Details: > > Under the current RAID0 setup I see the following when dd'ing. > > > > DD 4G from /dev/zero to /dev/sdd disk (no filesystem) & sync > > 28 seconds > > DD 4G from /dev/sdd to /dev/null 32 seconds > > DD 4G to ext3 on /dev/sdd & sync 32 seconds > > DD 4G from ext3 file to /dev/null 48 seconds. > > > > I've been watching the port usage on the FC switch and it verifies what > > I am seeing, Writes max out near 2Gb/s but reads hit some artificial > > limit around 90 MB/s and never ever exceed it with the filesystem, > > regardless of they underlying RAID configuration. Without a filesystem > > the reads are atleast 50% faster, and it can be seen on the FC switch > > graphs as well. > > > > Any help or thoughts would be appreciated. > > > > Thanks, > > ~Glen > > > > > > _______________________________________________ > > Ext3-users mailing list > > Ext3-users at redhat.com > > https://www.redhat.com/mailman/listinfo/ext3-users > > >