I''m having terrible IO trouble with an image based domu This is the performance on the Dom0 [root@localhost /]# hdparm -tT /dev/sda /dev/sda: Timing cached reads: 3228 MB in 2.00 seconds = 1617.22 MB/sec Timing buffered disk reads: 174 MB in 3.02 seconds = 57.60 MB/sec And this is the performance on the DomU [root@localhost ~]# hdparm -tT /dev/xvda /dev/xvda: Timing cached reads: 3336 MB in 2.00 seconds = 1669.98 MB/sec Timing buffered disk reads: 26 MB in 3.13 seconds = 8.30 MB/sec Is it normal for the performance difference to be so great? I''m running CentOS 5 and the DomU image is stored on a LVM ReiserFS formatted partition. Here is the config for the DomU name = "dddevdb" uuid = "927f1e915813c2dec60c8f76c2716783" maxmem = 3072 memory = 2042 vcpus = 2 bootloader = "/usr/bin/pygrub" on_poweroff = "destroy" on_reboot = "restart" on_crash = "restart" vfb = [ "type=vnc,vncunused=1,keymap=en-us" ] disk = [ "tap:aio:/var/lib/xen/images/dddevdb.img,xvda,w"] vif = [ "mac=00:16:3e:62:7a:d1,bridge=xenbr0" ] _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Greg Hellings wrote on Thu, 01 May 2008 16:32:56 -0700:> Is it normal for the performance difference to be so great?No, not at all. I actually did similar tests recently and found that the performance is almost identical (usually something like 10% less on the first test). I also found that there must be occurring some caching when you use images as the speed of the disk reads increases dramatically (it skyrockets) with a few repeated hdparm tests. I tested with CentOS 5 on ext3 filesystems, not with Reiser. Kai -- Kai Schätzl, Berlin, Germany _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Are you using a pre-allocated image or a sparse file? If the latter it''s likely to have performance problems like you found? What if you test with dd or iozone or bonnie++? --Sadique Greg Hellings wrote:> I''m having terrible IO trouble with an image based domu > > > This is the performance on the Dom0 > [root@localhost /]# hdparm -tT /dev/sda > > /dev/sda: > Timing cached reads: 3228 MB in 2.00 seconds = 1617.22 MB/sec > Timing buffered disk reads: 174 MB in 3.02 seconds = 57.60 MB/sec > > > And this is the performance on the DomU > [root@localhost ~]# hdparm -tT /dev/xvda > > /dev/xvda: > Timing cached reads: 3336 MB in 2.00 seconds = 1669.98 MB/sec > Timing buffered disk reads: 26 MB in 3.13 seconds = 8.30 MB/sec > > > Is it normal for the performance difference to be so great? I''m running > CentOS 5 and the DomU image is stored on a LVM ReiserFS formatted partition. > > Here is the config for the DomU > name = "dddevdb" > uuid = "927f1e915813c2dec60c8f76c2716783" > maxmem = 3072 > memory = 2042 > vcpus = 2 > bootloader = "/usr/bin/pygrub" > on_poweroff = "destroy" > on_reboot = "restart" > on_crash = "restart" > vfb = [ "type=vnc,vncunused=1,keymap=en-us" ] > disk = [ "tap:aio:/var/lib/xen/images/dddevdb.img,xvda,w"] > vif = [ "mac=00:16:3e:62:7a:d1,bridge=xenbr0" ] > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Sadique Puthen wrote on Fri, 02 May 2008 16:34:55 +0530:> Are you using a pre-allocated image or a sparse file? If the latter it''s > likely to have performance problems like you found?At least for hdparm tests there is no difference between a fully allocated image and a sparse file. I expect a difference for writes, though. Kai -- Kai Schätzl, Berlin, Germany Get your web at Conactive Internet Services: http://www.conactive.com _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On 5/2/08 4:04 AM, "Sadique Puthen" <sputhenp@redhat.com> wrote:> Are you using a pre-allocated image or a sparse file? If the latter it''s > likely to have performance problems like you found? What if you test > with dd or iozone or bonnie++? > > --Sadique > > Greg Hellings wrote: >> I''m having terrible IO trouble with an image based domu >> >> >> This is the performance on the Dom0 >> [root@localhost /]# hdparm -tT /dev/sda >> >> /dev/sda: >> Timing cached reads: 3228 MB in 2.00 seconds = 1617.22 MB/sec >> Timing buffered disk reads: 174 MB in 3.02 seconds = 57.60 MB/sec >> >> >> And this is the performance on the DomU >> [root@localhost ~]# hdparm -tT /dev/xvda >> >> /dev/xvda: >> Timing cached reads: 3336 MB in 2.00 seconds = 1669.98 MB/sec >> Timing buffered disk reads: 26 MB in 3.13 seconds = 8.30 MB/sec >> >> >> Is it normal for the performance difference to be so great? I''m running >> CentOS 5 and the DomU image is stored on a LVM ReiserFS formatted partition. >> >> Here is the config for the DomU >> name = "dddevdb" >> uuid = "927f1e915813c2dec60c8f76c2716783" >> maxmem = 3072 >> memory = 2042 >> vcpus = 2 >> bootloader = "/usr/bin/pygrub" >> on_poweroff = "destroy" >> on_reboot = "restart" >> on_crash = "restart" >> vfb = [ "type=vnc,vncunused=1,keymap=en-us" ] >> disk = [ "tap:aio:/var/lib/xen/images/dddevdb.img,xvda,w"] >> vif = [ "mac=00:16:3e:62:7a:d1,bridge=xenbr0" ] >> >> >> _______________________________________________ >> Xen-users mailing list >> Xen-users@lists.xensource.com >> http://lists.xensource.com/xen-users >> >> >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
I started this thinking I was having an IO problem. Now, I''m not so sure. I am using a pre-allocated disk, not a sparse file. Write performance seems about the same for both Dom0 [root@localhost ~]# dd if=/dev/zero of=/tmp/dummy count=2097152 2097152+0 records in 2097152+0 records out 1073741824 bytes (1.1 GB) copied, 29.8368 seconds, 36.0 MB/s DomU [root@localhost ~]# dd if=/dev/zero of=/tmp/dummy count=2097152 2097152+0 records in 2097152+0 records out 1073741824 bytes (1.1 GB) copied, 30.4716 seconds, 35.2 MB/s Iozone shows very similar numbers on both the Dom0 and DomU. The DomU seems to freeze periodically and drop network connections. I assumed this was IO related because everything else seems fine and those hdparm numbers are so far off. Also heavy IO related tasks seem slow. I''m fairly baffled now. -- Greg On 5/2/08 4:04 AM, "Sadique Puthen" <sputhenp@redhat.com> wrote:> Are you using a pre-allocated image or a sparse file? If the latter it''s > likely to have performance problems like you found? What if you test > with dd or iozone or bonnie++? > > --Sadique > > Greg Hellings wrote: >> I''m having terrible IO trouble with an image based domu >> >> >> This is the performance on the Dom0 >> [root@localhost /]# hdparm -tT /dev/sda >> >> /dev/sda: >> Timing cached reads: 3228 MB in 2.00 seconds = 1617.22 MB/sec >> Timing buffered disk reads: 174 MB in 3.02 seconds = 57.60 MB/sec >> >> >> And this is the performance on the DomU >> [root@localhost ~]# hdparm -tT /dev/xvda >> >> /dev/xvda: >> Timing cached reads: 3336 MB in 2.00 seconds = 1669.98 MB/sec >> Timing buffered disk reads: 26 MB in 3.13 seconds = 8.30 MB/sec >> >> >> Is it normal for the performance difference to be so great? I''m running >> CentOS 5 and the DomU image is stored on a LVM ReiserFS formatted partition. >> >> Here is the config for the DomU >> name = "dddevdb" >> uuid = "927f1e915813c2dec60c8f76c2716783" >> maxmem = 3072 >> memory = 2042 >> vcpus = 2 >> bootloader = "/usr/bin/pygrub" >> on_poweroff = "destroy" >> on_reboot = "restart" >> on_crash = "restart" >> vfb = [ "type=vnc,vncunused=1,keymap=en-us" ] >> disk = [ "tap:aio:/var/lib/xen/images/dddevdb.img,xvda,w"] >> vif = [ "mac=00:16:3e:62:7a:d1,bridge=xenbr0" ] >> >> >> _______________________________________________ >> Xen-users mailing list >> Xen-users@lists.xensource.com >> http://lists.xensource.com/xen-users >> >> >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Greg Hellings wrote:> > I started this thinking I was having an IO problem. Now, I''m > not so sure. > I am using a pre-allocated disk, not a sparse file. > > Write performance seems about the same for both > > Dom0 > [root@localhost ~]# dd if=/dev/zero of=/tmp/dummy count=2097152 > 2097152+0 records in > 2097152+0 records out > 1073741824 bytes (1.1 GB) copied, 29.8368 seconds, 36.0 MB/s > > DomU > [root@localhost ~]# dd if=/dev/zero of=/tmp/dummy count=2097152 > 2097152+0 records in > 2097152+0 records out > 1073741824 bytes (1.1 GB) copied, 30.4716 seconds, 35.2 MB/s > > Iozone shows very similar numbers on both the Dom0 and DomU. The DomU seems > to freeze periodically and drop network connections. I assumed this was IO > related because everything else seems fine and those hdparm numbers are so > far off. Also heavy IO related tasks seem slow. I''m fairly baffled now. >Run a ''vmstat 1'' and take a look at the interrupts and io waits. Maybe the network is generating a high load? Performance, and performance under load are two different things.> On 5/2/08 4:04 AM, "Sadique Puthen" <sputhenp@redhat.com> wrote: > > > Are you using a pre-allocated image or a sparse file? If > the latter it''s > > likely to have performance problems like you found? What > if you test > > with dd or iozone or bonnie++? > > > > --Sadique > > > > Greg Hellings wrote: > >> I''m having terrible IO trouble with an image based domu > >> > >> > >> This is the performance on the Dom0 > >> [root@localhost /]# hdparm -tT /dev/sda > >> > >> /dev/sda: > >> Timing cached reads: 3228 MB in 2.00 seconds = 1617.22 MB/sec > >> Timing buffered disk reads: 174 MB in 3.02 seconds = > 57.60 MB/sec > >> > >> > >> And this is the performance on the DomU > >> [root@localhost ~]# hdparm -tT /dev/xvda > >> > >> /dev/xvda: > >> Timing cached reads: 3336 MB in 2.00 seconds = 1669.98 MB/sec > >> Timing buffered disk reads: 26 MB in 3.13 seconds = > 8.30 MB/sec > >> > >> > >> Is it normal for the performance difference to be so > great? I''m running > >> CentOS 5 and the DomU image is stored on a LVM ReiserFS > formatted partition. > >> > >> Here is the config for the DomU > >> name = "dddevdb" > >> uuid = "927f1e915813c2dec60c8f76c2716783" > >> maxmem = 3072 > >> memory = 2042 > >> vcpus = 2 > >> bootloader = "/usr/bin/pygrub" > >> on_poweroff = "destroy" > >> on_reboot = "restart" > >> on_crash = "restart" > >> vfb = [ "type=vnc,vncunused=1,keymap=en-us" ] > >> disk = [ "tap:aio:/var/lib/xen/images/dddevdb.img,xvda,w"] > >> vif = [ "mac=00:16:3e:62:7a:d1,bridge=xenbr0" ] > >> > >> > >> _______________________________________________ > >> Xen-users mailing list > >> Xen-users@lists.xensource.com > >> http://lists.xensource.com/xen-users > >> > >> > > > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >______________________________________________________________________ This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender and permanently delete the original and any copy or printout thereof. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Mon, 5 May 2008, Greg Hellings wrote:> Iozone shows very similar numbers on both the Dom0 and DomU. The DomU seems > to freeze periodically and drop network connections. I assumed this was IO > related because everything else seems fine and those hdparm numbers are so > far off. Also heavy IO related tasks seem slow. I''m fairly baffled now....to put my 2cts again. It is probably block cache related. And I really ask for any ''real'' Xen developer of Citrix to do this simple experiment and report their findings: Allocate a nice partition, for example on NFS. Run bonnie++ with two times the memory size of the host system. Report findings :) Stefan _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Stefan de Konink wrote:> On Mon, 5 May 2008, Greg Hellings wrote: > > >> Iozone shows very similar numbers on both the Dom0 and DomU. The DomU seems >> to freeze periodically and drop network connections. I assumed this was IO >> related because everything else seems fine and those hdparm numbers are so >> far off. Also heavy IO related tasks seem slow. I''m fairly baffled now. >> > > ...to put my 2cts again. It is probably block cache related. And I really > ask for any ''real'' Xen developer of Citrix to do this simple experiment > and report their findings: >Why don''t u do this and report ur findings to developers and the list rather them doing this and report?> Allocate a nice partition, for example on NFS. Run bonnie++ with two times > the memory size of the host system. Report findings :) > > > Stefan > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, 6 May 2008, Sadique Puthen wrote:> Stefan de Konink wrote: > > On Mon, 5 May 2008, Greg Hellings wrote: > > > > > >> Iozone shows very similar numbers on both the Dom0 and DomU. The DomU seems > >> to freeze periodically and drop network connections. I assumed this was IO > >> related because everything else seems fine and those hdparm numbers are so > >> far off. Also heavy IO related tasks seem slow. I''m fairly baffled now. > >> > > > > ...to put my 2cts again. It is probably block cache related. And I really > > ask for any ''real'' Xen developer of Citrix to do this simple experiment > > and report their findings: > > > > Why don''t u do this and report ur findings to developers and the list > rather them doing this and report?Because I already did it... lets count FIVE times... and nothing happens... Stefan _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
So I tracked down my network freezing issues. I have a few xen servers all connected to the same network I''m working with here. All of them I used the virt-manager tool to install. One of the other hypervisors had a domu running on it that virt-manager had chosen the same mac address as the domu on the hypervisor I was having problems with. I still see the same performance difference between the dom0 and domu when using hdparm -tT, but maybe that isn''t a good test. I do notice that if I run hdparm over and over on the domu''s the numbers improve, suggesting that caching is effecting the out come. Under iozone, the number between domu and the dom0 are almost exactly the same. -- Greg On 5/5/08 2:58 PM, "Ross S. W. Walker" <rwalker@medallion.com> wrote:> Greg Hellings wrote: >> >> I started this thinking I was having an IO problem. Now, I''m >> not so sure. >> I am using a pre-allocated disk, not a sparse file. >> >> Write performance seems about the same for both >> >> Dom0 >> [root@localhost ~]# dd if=/dev/zero of=/tmp/dummy count=2097152 >> 2097152+0 records in >> 2097152+0 records out >> 1073741824 bytes (1.1 GB) copied, 29.8368 seconds, 36.0 MB/s >> >> DomU >> [root@localhost ~]# dd if=/dev/zero of=/tmp/dummy count=2097152 >> 2097152+0 records in >> 2097152+0 records out >> 1073741824 bytes (1.1 GB) copied, 30.4716 seconds, 35.2 MB/s >> >> Iozone shows very similar numbers on both the Dom0 and DomU. The DomU seems >> to freeze periodically and drop network connections. I assumed this was IO >> related because everything else seems fine and those hdparm numbers are so >> far off. Also heavy IO related tasks seem slow. I''m fairly baffled now. >> > > Run a ''vmstat 1'' and take a look at the interrupts and io waits. > > Maybe the network is generating a high load? > > Performance, and performance under load are two different things. > > >> On 5/2/08 4:04 AM, "Sadique Puthen" <sputhenp@redhat.com> wrote: >> >>> Are you using a pre-allocated image or a sparse file? If >> the latter it''s >>> likely to have performance problems like you found? What >> if you test >>> with dd or iozone or bonnie++? >>> >>> --Sadique >>> >>> Greg Hellings wrote: >>>> I''m having terrible IO trouble with an image based domu >>>> >>>> >>>> This is the performance on the Dom0 >>>> [root@localhost /]# hdparm -tT /dev/sda >>>> >>>> /dev/sda: >>>> Timing cached reads: 3228 MB in 2.00 seconds = 1617.22 MB/sec >>>> Timing buffered disk reads: 174 MB in 3.02 seconds >> 57.60 MB/sec >>>> >>>> >>>> And this is the performance on the DomU >>>> [root@localhost ~]# hdparm -tT /dev/xvda >>>> >>>> /dev/xvda: >>>> Timing cached reads: 3336 MB in 2.00 seconds = 1669.98 MB/sec >>>> Timing buffered disk reads: 26 MB in 3.13 seconds >> 8.30 MB/sec >>>> >>>> >>>> Is it normal for the performance difference to be so >> great? I''m running >>>> CentOS 5 and the DomU image is stored on a LVM ReiserFS >> formatted partition. >>>> >>>> Here is the config for the DomU >>>> name = "dddevdb" >>>> uuid = "927f1e915813c2dec60c8f76c2716783" >>>> maxmem = 3072 >>>> memory = 2042 >>>> vcpus = 2 >>>> bootloader = "/usr/bin/pygrub" >>>> on_poweroff = "destroy" >>>> on_reboot = "restart" >>>> on_crash = "restart" >>>> vfb = [ "type=vnc,vncunused=1,keymap=en-us" ] >>>> disk = [ "tap:aio:/var/lib/xen/images/dddevdb.img,xvda,w"] >>>> vif = [ "mac=00:16:3e:62:7a:d1,bridge=xenbr0" ] >>>> >>>> >>>> _______________________________________________ >>>> Xen-users mailing list >>>> Xen-users@lists.xensource.com >>>> http://lists.xensource.com/xen-users >>>> >>>> >>> >> >> >> _______________________________________________ >> Xen-users mailing list >> Xen-users@lists.xensource.com >> http://lists.xensource.com/xen-users >> > > ______________________________________________________________________ > This e-mail, and any attachments thereto, is intended only for use by > the addressee(s) named herein and may contain legally privileged > and/or confidential information. If you are not the intended recipient > of this e-mail, you are hereby notified that any dissemination, > distribution or copying of this e-mail, and any attachments thereto, > is strictly prohibited. If you have received this e-mail in error, > please immediately notify the sender and permanently delete the > original and any copy or printout thereof. >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Greg Hellings wrote:> > So I tracked down my network freezing issues. I have a few xen servers all > connected to the same network I''m working with here. All of them I used the > virt-manager tool to install. One of the other hypervisors had a domu > running on it that virt-manager had chosen the same mac address as the domu > on the hypervisor I was having problems with.Good news. It''s always the last place you look...> I still see the same performance difference between the dom0 and domu when > using hdparm -tT, but maybe that isn''t a good test. I do notice that if I > run hdparm over and over on the domu''s the numbers improve, suggesting that > caching is effecting the out come. Under iozone, the number between domu > and the dom0 are almost exactly the same.Well when doing aio through the VFS layer to an image file you will loose some performance. It is the sacrifice you make in order to gain flexibility. Everything''s a trade-off. -Ross ______________________________________________________________________ This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender and permanently delete the original and any copy or printout thereof. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users