We are running a benchmark that does single threaded 512 KB writes to a LUN on a CLARiiON storage array. The dual Xeon host (Dell 2650) with 4 GB of memory runs RHEL 4U3 We measured the write bandwidth for writes to the block device corresponding to the lun (e.g. /dev/sdb), a file in an ext2 filesystem and to a file in an ext3 file system. Write b/w for 512 KB writes Block device 312 MBps Ext2 file 247 MBps Ext3 file 130 MBps We are looking for ways to improve the ext3 file write bandwidth. Tracing of I/Os at the storage array shows that in the case of ext3 experiment, the workload does not keep the lun busy enough. Every 5 seconds there is an increase in I/O activity that lasts for 2-3 seconds. The lun then has very low activity for 2-3 seconds. It appears that the buffers at the host are flushed every 5 seconds and the flushing takes 2-3 seconds. To maximize write bandwidth, we would like to be in a situation where the buffers are flushed continuously to keep the lun constanly busy. That is what we see in the case of the ext2 file. In the case of ext2 we also see the host do quite a few ver large writes (up to 7784 KB). Thanks Himanshu Sinha
On Mon, Jun 19, 2006 at 02:18:12PM -0400, Sinha_Himanshu at emc.com wrote:> We are running a benchmark that does single threaded 512 KB writes to a > LUN on a CLARiiON storage array. The dual Xeon host (Dell 2650) with 4 > GB of memory runs RHEL 4U3 > > We measured the write bandwidth for writes to the block device > corresponding to the lun (e.g. /dev/sdb), a file in an ext2 filesystem > and to a file in an ext3 file system. > Write b/w for 512 KB writes > Block device 312 MBps > Ext2 file 247 MBps > Ext3 file 130 MBps > > We are looking for ways to improve the ext3 file write bandwidth. > > Tracing of I/Os at the storage array shows that in the case of ext3 > experiment, the workload does not keep the lun busy enough. Every 5 > seconds there is an increase in I/O activity that lasts for 2-3 seconds. > The lun then has very low activity for 2-3 seconds. It appears that the > buffers at the host are flushed every 5 seconds and the flushing takes > 2-3 seconds. To maximize write bandwidth, we would like to be in a > situation where the buffers are flushed continuously to keep the lun > constanly busy. > That is what we see in the case of the ext2 file. > > In the case of ext2 we also see the host do quite a few ver large writes > (up to 7784 KB).Hmm, is the new kernel 2.6.17 addressing such issues? I understand that the new multibuffered implementation of ext3 may enhance the performance considerably. What kernel did you benchmark on? best regards keld
I misspoke earlier - our data is from a RHEL 4U1 system - 2.6.9 kernel. We will try RHEL 4U3 soon. That will give us a later kernel. Thanks Himanshu -----Original Message----- From: Keld J?rn Simonsen [mailto:keld at dkuug.dk] Sent: Wednesday, June 21, 2006 10:12 AM To: Sinha, Himanshu Cc: ext3-users at redhat.com Subject: Re: Limited write bandwidth from ext3 On Mon, Jun 19, 2006 at 02:18:12PM -0400, Sinha_Himanshu at emc.com wrote:> We are running a benchmark that does single threaded 512 KB writes to a > LUN on a CLARiiON storage array. The dual Xeon host (Dell 2650) with 4 > GB of memory runs RHEL 4U3 > > We measured the write bandwidth for writes to the block device > corresponding to the lun (e.g. /dev/sdb), a file in an ext2 filesystem > and to a file in an ext3 file system. > Write b/w for 512 KB writes > Block device 312 MBps > Ext2 file 247 MBps > Ext3 file 130 MBps > > We are looking for ways to improve the ext3 file write bandwidth. > > Tracing of I/Os at the storage array shows that in the case of ext3 > experiment, the workload does not keep the lun busy enough. Every 5 > seconds there is an increase in I/O activity that lasts for 2-3 seconds. > The lun then has very low activity for 2-3 seconds. It appears that the > buffers at the host are flushed every 5 seconds and the flushing takes > 2-3 seconds. To maximize write bandwidth, we would like to be in a > situation where the buffers are flushed continuously to keep the lun > constanly busy. > That is what we see in the case of the ext2 file. > > In the case of ext2 we also see the host do quite a few ver large writes > (up to 7784 KB).Hmm, is the new kernel 2.6.17 addressing such issues? I understand that the new multibuffered implementation of ext3 may enhance the performance considerably. What kernel did you benchmark on? best regards keld
Sinha_Himanshu at emc.com wrote:> We are running a benchmark that does single threaded 512 KB writes to a > LUN on a CLARiiON storage array. The dual Xeon host (Dell 2650) with 4 > GB of memory runs RHEL 4U3 > > We measured the write bandwidth for writes to the block device > corresponding to the lun (e.g. /dev/sdb), a file in an ext2 filesystem > and to a file in an ext3 file system. > Write b/w for 512 KB writes > Block device 312 MBps > Ext2 file 247 MBps > Ext3 file 130 MBps > > We are looking for ways to improve the ext3 file write bandwidth. > > Tracing of I/Os at the storage array shows that in the case of ext3 > experiment, the workload does not keep the lun busy enough. Every 5 > seconds there is an increase in I/O activity that lasts for 2-3 seconds. > The lun then has very low activity for 2-3 seconds. It appears that the > buffers at the host are flushed every 5 seconds and the flushing takes > 2-3 seconds. To maximize write bandwidth, we would like to be in a > situation where the buffers are flushed continuously to keep the lun > constanly busy. > That is what we see in the case of the ext2 file. > > In the case of ext2 we also see the host do quite a few ver large writes > (up to 7784 KB). > > Thanks > Himanshu Sinha > > _______________________________________________ > Ext3-users mailing list > Ext3-users at redhat.com > https://www.redhat.com/mailman/listinfo/ext3-users >Sounds like you are describing the default ext3 sync'ing feature. ext3 has a mount option "commit", with default value 5 seconds. It causes all data and metadata to be sync'ed every 5. (Cf. 'man mount') I'm not an expert, so I'm not sure what the potential impact might be (other than delaying sync times) if you change this value. Kind regards, Herta Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm
On Jun 19, 2006 14:18 -0400, Sinha_Himanshu at emc.com wrote:> We measured the write bandwidth for writes to the block device > corresponding to the lun (e.g. /dev/sdb), a file in an ext2 filesystem > and to a file in an ext3 file system. > Write b/w for 512 KB writes > Block device 312 MBps > Ext2 file 247 MBps > Ext3 file 130 MBps > > We are looking for ways to improve the ext3 file write bandwidth.Have a look at the extents+mballoc+delalloc patches from Alex Tomas: ftp://ftp.lustre.org/pub/people/alex/2.6.16.8/ Mount the filesystem with "-o extents,mballoc,delalloc" to enable this. They noticably improve IO performance while also reducing the CPU load for ext3. The extent patches are approved by all of the ext3 developers and will be supported upstream fairly soon (in the kernel and e2fsprogs), and mballoc+delalloc will follow on afterward. NOTE: the extents on-disk format is incompatible with older kernels, so at this stage consider it "for benchmarking only". Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc.
We tried the extents+mballoc+delalloc patches suggested by Andreas and found that it made a significant improvement in our benchmark - write bandwidth increased from 144 MBps to 214 MBps. We are at about 85% of the bandwidth that one can get writing to an ext2 file which in turn is about 82% of the bandwidth one can get writing to the block device. We are analyzing our traces to determine the cause of these differences. So far, we see that during writes to the ext3 file lun writes periodically wait for 5 reads while in the case of writes to ext2 file lun writes periodically wait for only one read. Workload: Single threaded 512 KB writes to a new file. RedHat 4 U1 2.6.16.8 kernel (2.6.9 based kernel) Block Device 308 MBps 306 MBps Ext2 file 267 255 Ext3 138 144 Ext3 with patches N/A 216 Ext3 with patches, journal on separate LUN 215 Himanshu -----Original Message----- From: Andreas Dilger [mailto:adilger at clusterfs.com] Sent: Wednesday, June 21, 2006 4:54 PM To: Sinha, Himanshu Cc: ext3-users at redhat.com Subject: Re: Limited write bandwidth from ext3 On Jun 19, 2006 14:18 -0400, Sinha_Himanshu at emc.com wrote:> We measured the write bandwidth for writes to the block device > corresponding to the lun (e.g. /dev/sdb), a file in an ext2 filesystem > and to a file in an ext3 file system. > Write b/w for 512 KB writes > Block device 312 MBps > Ext2 file 247 MBps > Ext3 file 130 MBps > > We are looking for ways to improve the ext3 file write bandwidth.Have a look at the extents+mballoc+delalloc patches from Alex Tomas: ftp://ftp.lustre.org/pub/people/alex/2.6.16.8/ Mount the filesystem with "-o extents,mballoc,delalloc" to enable this. They noticably improve IO performance while also reducing the CPU load for ext3. The extent patches are approved by all of the ext3 developers and will be supported upstream fairly soon (in the kernel and e2fsprogs), and mballoc+delalloc will follow on afterward. NOTE: the extents on-disk format is incompatible with older kernels, so at this stage consider it "for benchmarking only". Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc.