Hello. I'm experiencing solid I/O performance drop when using software raid 5 from PV DomU. I can get about 420M/s for sequential reads and ~220 M/s for sequential writes when using it from Dom0. But it's only ~170 M/s for read and ~80 M/s for write when using it from DomU. DomU performance for single drive is close to native — ~160 M/s for reads and ~160 for writes. There is no filesystem or LVM, just raw data. Debian wheezy x86_64 for both Dom0 and DomU. And "phy" backend is used to attach drive to DomU. Is this a bug or something wrong with my setup? What should I check? Kirill _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
> > Hello. I'm experiencing solid I/O performance drop when using software > raid 5 from PV DomU. I can get about 420M/s for sequential reads and > ~220 M/s for sequential writes when using it from Dom0. But it's only > ~170 M/s for read and ~80 M/s for write when using it from DomU. > > DomU performance for single drive is close to native — ~160 M/s for > reads and ~160 for writes. > > There is no filesystem or LVM, just raw data. Debian wheezy x86_64 for > both Dom0 and DomU. And "phy" backend is used to attach drive to DomU. > Is this a bug or something wrong with my setup? What should I check? >How are you measuring this performance? James _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
26.05.2013 14:43, James Harper пишет:>> Hello. I'm experiencing solid I/O performance drop when using software >> raid 5 from PV DomU. I can get about 420M/s for sequential reads and >> ~220 M/s for sequential writes when using it from Dom0. But it's only >> ~170 M/s for read and ~80 M/s for write when using it from DomU. >> >> DomU performance for single drive is close to native — ~160 M/s for >> reads and ~160 for writes. >> >> There is no filesystem or LVM, just raw data. Debian wheezy x86_64 for >> both Dom0 and DomU. And "phy" backend is used to attach drive to DomU. >> Is this a bug or something wrong with my setup? What should I check? >> > How are you measuring this performance? > > JamesI running dd if=/dev/zero of=/dev/xvdb bs=1M for several minutes. Also tried "cat /dev/zero | pv -r > /dev/xvdb " which gave me similar results. _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
> > 26.05.2013 14:43, James Harper пишет: > >> Hello. I'm experiencing solid I/O performance drop when using software > >> raid 5 from PV DomU. I can get about 420M/s for sequential reads and > >> ~220 M/s for sequential writes when using it from Dom0. But it's only > >> ~170 M/s for read and ~80 M/s for write when using it from DomU. > >> > >> DomU performance for single drive is close to native — ~160 M/s for > >> reads and ~160 for writes. > >> > >> There is no filesystem or LVM, just raw data. Debian wheezy x86_64 for > >> both Dom0 and DomU. And "phy" backend is used to attach drive to > DomU. > >> Is this a bug or something wrong with my setup? What should I check? > >> > > How are you measuring this performance? > > > > James > I running dd if=/dev/zero of=/dev/xvdb bs=1M for several minutes. > > Also tried "cat /dev/zero | pv -r > /dev/xvdb " which gave me similar > results.Add oflag=direct to the dd command so that no caching is in effect and then compare. James _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
26.05.2013 15:16, James Harper пишет:>> 26.05.2013 14:43, James Harper пишет: >>>> Hello. I'm experiencing solid I/O performance drop when using software >>>> raid 5 from PV DomU. I can get about 420M/s for sequential reads and >>>> ~220 M/s for sequential writes when using it from Dom0. But it's only >>>> ~170 M/s for read and ~80 M/s for write when using it from DomU. >>>> >>>> DomU performance for single drive is close to native — ~160 M/s for >>>> reads and ~160 for writes. >>>> >>>> There is no filesystem or LVM, just raw data. Debian wheezy x86_64 for >>>> both Dom0 and DomU. And "phy" backend is used to attach drive to >> DomU. >>>> Is this a bug or something wrong with my setup? What should I check? >>>> >>> How are you measuring this performance? >>> >>> James >> I running dd if=/dev/zero of=/dev/xvdb bs=1M for several minutes. >> >> Also tried "cat /dev/zero | pv -r > /dev/xvdb " which gave me similar >> results. > Add oflag=direct to the dd command so that no caching is in effect and then compare. > > JamesJames, it's even more dramatic without caching. Dom0: Reading: dd if=/dev/md0 of=/dev/null bs=1M iflag=direct скопировано 11659116544 байта (12 GB), 27,4614 c, 425 MB/c Writing: dd if=/dev/zero of=/dev/md0 bs=1M oflag=direct скопировано 10108272640 байт (10 GB), 135,859 c, 74,4 MB/c Domu: Reading: dd if=/dev/xvdb of=/dev/null iflag=direct скопировано 229615104 байта (230 MB), 75,9394 c, 3,0 MB/c Writing: dd if=/dev/zero of=/dev/xvdb oflag=direct скопировано 231818240 байт (232 MB), 158,283 c, 1,5 MB/c _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
> -----Original Message----- > From: braintorch [mailto:kkabardin@gmail.com] > Sent: Sunday, 26 May 2013 9:49 PM > To: James Harper > Cc: xen-users@lists.xen.org > Subject: Re: [Xen-users] Software Raid 5 domu performance drop > > 26.05.2013 15:16, James Harper пишет: > >> 26.05.2013 14:43, James Harper пишет: > >>>> Hello. I'm experiencing solid I/O performance drop when using > software > >>>> raid 5 from PV DomU. I can get about 420M/s for sequential reads and > >>>> ~220 M/s for sequential writes when using it from Dom0. But it's only > >>>> ~170 M/s for read and ~80 M/s for write when using it from DomU. > >>>> > >>>> DomU performance for single drive is close to native — ~160 M/s for > >>>> reads and ~160 for writes. > >>>> > >>>> There is no filesystem or LVM, just raw data. Debian wheezy x86_64 for > >>>> both Dom0 and DomU. And "phy" backend is used to attach drive to > >> DomU. > >>>> Is this a bug or something wrong with my setup? What should I check? > >>>> > >>> How are you measuring this performance? > >>> > >>> James > >> I running dd if=/dev/zero of=/dev/xvdb bs=1M for several minutes. > >> > >> Also tried "cat /dev/zero | pv -r > /dev/xvdb " which gave me similar > >> results. > > Add oflag=direct to the dd command so that no caching is in effect and > then compare. > > > > James > James, it's even more dramatic without caching. > > > Dom0: > > Reading: > dd if=/dev/md0 of=/dev/null bs=1M iflag=direct > скопировано 11659116544 байта (12 GB), 27,4614 c, 425 MB/c > > Writing: > dd if=/dev/zero of=/dev/md0 bs=1M oflag=direct > скопировано 10108272640 байт (10 GB), 135,859 c, 74,4 MB/c > > Domu: > > Reading: > dd if=/dev/xvdb of=/dev/null iflag=direct > скопировано 229615104 байта (230 MB), 75,9394 c, 3,0 MB/c > > Writing: > dd if=/dev/zero of=/dev/xvdb oflag=direct > скопировано 231818240 байт (232 MB), 158,283 c, 1,5 MB/cI don't see a block size on the domu measurements... did you just copy and paste it wrong or did you really leave it at default 512 byte block size? James _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
26.05.2013 15:51, James Harper пишет:> >> -----Original Message----- >> From: braintorch [mailto:kkabardin@gmail.com] >> Sent: Sunday, 26 May 2013 9:49 PM >> To: James Harper >> Cc: xen-users@lists.xen.org >> Subject: Re: [Xen-users] Software Raid 5 domu performance drop >> >> 26.05.2013 15:16, James Harper пишет: >>>> 26.05.2013 14:43, James Harper пишет: >>>>>> Hello. I'm experiencing solid I/O performance drop when using >> software >>>>>> raid 5 from PV DomU. I can get about 420M/s for sequential reads and >>>>>> ~220 M/s for sequential writes when using it from Dom0. But it's only >>>>>> ~170 M/s for read and ~80 M/s for write when using it from DomU. >>>>>> >>>>>> DomU performance for single drive is close to native — ~160 M/s for >>>>>> reads and ~160 for writes. >>>>>> >>>>>> There is no filesystem or LVM, just raw data. Debian wheezy x86_64 for >>>>>> both Dom0 and DomU. And "phy" backend is used to attach drive to >>>> DomU. >>>>>> Is this a bug or something wrong with my setup? What should I check? >>>>>> >>>>> How are you measuring this performance? >>>>> >>>>> James >>>> I running dd if=/dev/zero of=/dev/xvdb bs=1M for several minutes. >>>> >>>> Also tried "cat /dev/zero | pv -r > /dev/xvdb " which gave me similar >>>> results. >>> Add oflag=direct to the dd command so that no caching is in effect and >> then compare. >>> James >> James, it's even more dramatic without caching. >> >> >> Dom0: >> >> Reading: >> dd if=/dev/md0 of=/dev/null bs=1M iflag=direct >> скопировано 11659116544 байта (12 GB), 27,4614 c, 425 MB/c >> >> Writing: >> dd if=/dev/zero of=/dev/md0 bs=1M oflag=direct >> скопировано 10108272640 байт (10 GB), 135,859 c, 74,4 MB/c >> >> Domu: >> >> Reading: >> dd if=/dev/xvdb of=/dev/null iflag=direct >> скопировано 229615104 байта (230 MB), 75,9394 c, 3,0 MB/c >> >> Writing: >> dd if=/dev/zero of=/dev/xvdb oflag=direct >> скопировано 231818240 байт (232 MB), 158,283 c, 1,5 MB/c > I don't see a block size on the domu measurements... did you just copy and paste it wrong or did you really leave it at default 512 byte block size? > > JamesAh, my mistake. I'm sorry. :( Reading: dd if=/dev/xvdb of=/dev/null bs=1M iflag=direct скопировано 13060014080 байт (13 GB), 58,949 c, 222 MB/c Writing: dd if=/dev/zero of=/dev/xvdb bs=1M oflag=direct скопировано 2241855488 байт (2,2 GB), 29,6292 c, 75,7 MB/c So, writing without caching is almost the same. Reading is halfed in compare to dom0, but this is not really an issue to me. Is there a way to optimize DomU caching to boost write speed? Kirill. _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Software raids by design have huge performance hits. I won't really use dmraid in production. On Sun, May 26, 2013 at 5:48 PM, braintorch <kkabardin@gmail.com> wrote:> 26.05.2013 15:51, James Harper пишет: > >> >>> -----Original Message----- >>> From: braintorch [mailto:kkabardin@gmail.com] >>> Sent: Sunday, 26 May 2013 9:49 PM >>> To: James Harper >>> Cc: xen-users@lists.xen.org >>> Subject: Re: [Xen-users] Software Raid 5 domu performance drop >>> >>> 26.05.2013 15:16, James Harper пишет: >>>>> >>>>> 26.05.2013 14:43, James Harper пишет: >>>>>>> >>>>>>> Hello. I'm experiencing solid I/O performance drop when using >>> >>> software >>>>>>> >>>>>>> raid 5 from PV DomU. I can get about 420M/s for sequential reads and >>>>>>> ~220 M/s for sequential writes when using it from Dom0. But it's only >>>>>>> ~170 M/s for read and ~80 M/s for write when using it from DomU. >>>>>>> >>>>>>> DomU performance for single drive is close to native -- ~160 M/s for >>>>>>> reads and ~160 for writes. >>>>>>> >>>>>>> There is no filesystem or LVM, just raw data. Debian wheezy x86_64 >>>>>>> for >>>>>>> both Dom0 and DomU. And "phy" backend is used to attach drive to >>>>> >>>>> DomU. >>>>>>> >>>>>>> Is this a bug or something wrong with my setup? What should I check? >>>>>>> >>>>>> How are you measuring this performance? >>>>>> >>>>>> James >>>>> >>>>> I running dd if=/dev/zero of=/dev/xvdb bs=1M for several minutes. >>>>> >>>>> Also tried "cat /dev/zero | pv -r > /dev/xvdb " which gave me similar >>>>> results. >>>> >>>> Add oflag=direct to the dd command so that no caching is in effect and >>> >>> then compare. >>>> >>>> James >>> >>> James, it's even more dramatic without caching. >>> >>> >>> Dom0: >>> >>> Reading: >>> dd if=/dev/md0 of=/dev/null bs=1M iflag=direct >>> скопировано 11659116544 байта (12 GB), 27,4614 c, 425 MB/c >>> >>> Writing: >>> dd if=/dev/zero of=/dev/md0 bs=1M oflag=direct >>> скопировано 10108272640 байт (10 GB), 135,859 c, 74,4 MB/c >>> >>> Domu: >>> >>> Reading: >>> dd if=/dev/xvdb of=/dev/null iflag=direct >>> скопировано 229615104 байта (230 MB), 75,9394 c, 3,0 MB/c >>> >>> Writing: >>> dd if=/dev/zero of=/dev/xvdb oflag=direct >>> скопировано 231818240 байт (232 MB), 158,283 c, 1,5 MB/c >> >> I don't see a block size on the domu measurements... did you just copy and >> paste it wrong or did you really leave it at default 512 byte block size? >> >> James > > Ah, my mistake. I'm sorry. :( > > Reading: > dd if=/dev/xvdb of=/dev/null bs=1M iflag=direct > скопировано 13060014080 байт (13 GB), 58,949 c, 222 MB/c > > Writing: > dd if=/dev/zero of=/dev/xvdb bs=1M oflag=direct > скопировано 2241855488 байт (2,2 GB), 29,6292 c, 75,7 MB/c > > So, writing without caching is almost the same. Reading is halfed in compare > to dom0, but this is not really an issue to me. > Is there a way to optimize DomU caching to boost write speed? > > Kirill. > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xen.org > http://lists.xen.org/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
> > So, writing without caching is almost the same. Reading is halfed in > compare to dom0, but this is not really an issue to me. > Is there a way to optimize DomU caching to boost write speed? >I think you wouldn''t want to do this if you value your data. Maybe dom0 just has more memory than domu? Are you saying that raid5 has a specific problem or is this just what you happen to be using? Can you do some other tests with just a single harddisk passed through to domu? And maybe a raid1 too? James
On 5/26/2013 8:39 AM, Micky wrote:> Software raids by design have huge performance hits. I won''t really > use dmraid in production.Raid 5 also has significantly more more I/O delays & performance hits since a single Write usually causes two Reads, then processing, finally two Writes. It gets even worse when sync''ing or if something waits for the the operation to complete before continuing. Anyone using S/W raid (and sometimes even with h/w raid) is better off using Raid 1 or Raid 10 if you are concerned with performance. S.W.
> > On 5/26/2013 8:39 AM, Micky wrote: > > Software raids by design have huge performance hits. I won''t really > > use dmraid in production. > > Raid 5 also has significantly more more I/O delays & performance hits > since a single Write usually causes two Reads, then processing, finally > two Writes. It gets even worse when sync''ing or if something waits for > the the operation to complete before continuing. > > Anyone using S/W raid (and sometimes even with h/w raid) is better off > using Raid 1 or Raid 10 if you are concerned with performance. >According to a recent post - http://comments.gmane.org/gmane.linux.kernel.bcache.devel/1713 - bcache now has some smarts for raid5 where it tries to write out an entire stripe where possible. If you have a fast flash disk and can compile your own kernel then bcache is the only way I would recommend using software raid5 where performance matters. Also in theory it might close the raid5 write hole. James
26.05.2013 17:47, James Harper пишет:>> So, writing without caching is almost the same. Reading is halfed in >> compare to dom0, but this is not really an issue to me. >> Is there a way to optimize DomU caching to boost write speed? >> > I think you wouldn't want to do this if you value your data. Maybe dom0 just has more memory than domu? > > Are you saying that raid5 has a specific problem or is this just what you happen to be using? Can you do some other tests with just a single harddisk passed through to domu? And maybe a raid1 too? > > JamesActually, dom0 have 512 Mb memory vs DomU with 1024 Mb. I have done some tests with single drive, raid1 and raid10. Results in DomU almost identical to those in Dom0. Only raid 5 setup have the difference. _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
On 05/26/2013 02:39 PM, Micky wrote:> Software raids by design have huge performance hits.Who told you that? The ARM ASIC on a typical RAID controller has a lot less throughput for XOR-ing data than the main CPU in the system, and also has a lot less RAM for caching. If anything software RAID is going to be faster if you are operating at saturation point. With hardware RAID you are also suffering the problem that alignment of blocks is unknowable, which is relevant when you are trying to align your entire block storage stack (especially when the disk is reporting 512b sectors when they are emulated from 4KB physical sectors). Gordan
> > On 05/26/2013 02:39 PM, Micky wrote: > > Software raids by design have huge performance hits. > > Who told you that? The ARM ASIC on a typical RAID controller has a lot > less throughput for XOR-ing data than the main CPU in the system, and > also has a lot less RAM for caching. If anything software RAID is going > to be faster if you are operating at saturation point.What spoils it is that hardware raid normally has nv write cache which can be a huge boost for performance, especially for raid5. Bcache fixes that to some extent but I''m not aware of any cheap battery backed memory cards. SSD is the next best thing but that''s often attached via SATA or SAS. James
On 05/27/2013 12:22 AM, James Harper wrote:>> >> On 05/26/2013 02:39 PM, Micky wrote: >>> Software raids by design have huge performance hits. >> >> Who told you that? The ARM ASIC on a typical RAID controller has a lot >> less throughput for XOR-ing data than the main CPU in the system, and >> also has a lot less RAM for caching. If anything software RAID is going >> to be faster if you are operating at saturation point. > > What spoils it is that hardware raid normally has nv write cache which > can be a huge boost for performance, especially for raid5.That only helps if your load is very bursty and you aren''t operating anywhere near the saturation point for any length of time. The write hole can largely be avoided by making sure your FS stack geometry is optimally alighted with what your application does. As a random example, if you are using MySQL and InnoDB which has the default page size of 16KB, if you have a 5-disk RAID5, you can make the RAID block size 4KB. That means your 16KB commits (assuming everything else is also aligned correctly) will typically write out a whole stripe at the same time, which avoids the write-read-modify cycle. But - this doesn''t help you in terms of overall throughput because you are still performing an operation on every disk in the array, so your overall write IOPS are the same as that of a single disk whatever you do. In the non-aligned case you get: 1.1) Write on subset of x disks 1.2) Read on subset of n-x-1 disks 1.3) Write on the 1 parity disk Total: 1 operation per disk Where n is the number of disks in the array, and x is the number of disks such that operation size / RAID block size = x. In the aligned case you get: 2.1) Write on all disks Total: 1 operation per disk So it doesn''t actually gain you any throughput in the saturation case whatever you do. RAID5 generally sucks for both performance and reliability regardless of what kind of RAID you use. But this is getting fairly off-topic for Xen... Gordan