Hi again, just today an issue I've thought to be resolved popped up again. We backup the machine by doing: virsh snapshot-create-as --domain domain --name backup --no-metadata --atomic --disk-only --diskspec hda,snapshot=external # backup hda.qcow2 virsh blockcommit domain hda --active --pivot Every now and then this process fails with the following error message: error: failed to pivot job for disk hda error: block copy still active: disk 'hda' not ready for pivot yet Could not merge changes for disk hda of domain. VM may be in invalid state. I expect live backups are a great asset and should work. Is this a bug that may relates also to the virtual drive performance issues we observe? Cheers 2017-07-02 10:10 GMT+02:00 Dominik Psenner <dpsenner@gmail.com>:> Hi > > a small update on this. I just migrated the vm from the site to my laptop > and fired it up. The exact same xml configuration (except file paths and > such) starts up and bursts with 50Mb/s to 115Mb/s in the guest. This allows > only one reasonable answer: the cpu on my laptop is somehow better suited > to emulate IO than the CPU built into the host on site. The host there is a > HP proliant microserver gen8 with xeon processor. But the processor there > is also never capped at 100% when the guest copies files. > > I just ran another test by copying a 3Gb large file on the guest. What I > can observe on my computer is that the copy process is not at a constant > rate but rather starts with 90Mb/s, then drops down to 30Mb/s, goes up to > 70Mb/s, drops down to 1Mb/s, goes up to 75Mb/s, drops to 1Mb/s, goes up to > 55Mb/s and the pattern continues. Please note that the drive is still > configured as: > > <driver name='qemu' type='qcow2' cache='none' io='threads'/> > > and I would expect a constant rate that is either high or low since there > is no caching involved and the underlying hard drive is a samsung ssd evo > 850. To have an idea how fast that drive is on my laptop: > > $ dd if=/dev/zero of=testfile bs=1M count=1000 oflag=direct > 1000+0 records in > 1000+0 records out > 1048576000 bytes (1.0 GB, 1000 MiB) copied, 2.47301 s, 424 MB/s > > I can further observe that the smaller the saved chunks are the slower the > overall performance is: > > dd if=/dev/zero of=testfile bs=512K count=1000 oflag=direct > 1000+0 records in > 1000+0 records out > 524288000 bytes (524 MB, 500 MiB) copied, 1.34874 s, 389 MB/s > > $ dd if=/dev/zero of=testfile bs=5K count=1000 oflag=direct > 1000+0 records in > 1000+0 records out > 5120000 bytes (5.1 MB, 4.9 MiB) copied, 0.105109 s, 48.7 MB/s > > $ dd if=/dev/zero of=testfile bs=1K count=10000 oflag=direct > 10000+0 records in > 10000+0 records out > 10240000 bytes (10 MB, 9.8 MiB) copied, 0.668438 s, 15.3 MB/s > > $ dd if=/dev/zero of=testfile bs=512 count=20000 oflag=direct > 20000+0 records in > 20000+0 records out > 10240000 bytes (10 MB, 9.8 MiB) copied, 1.10964 s, 9.2 MB/s > > Could this be a limiting factor? Does qemu/kvm do many many writes of just > a few bytes? > > Ideas, anyone? > > Cheers > > 2017-06-21 20:46 GMT+02:00 Dan <srwx4096@gmail.com>: > >> On Tue, Jun 20, 2017 at 04:24:32PM +0200, Gianluca Cecchi wrote: >> > On Tue, Jun 20, 2017 at 3:38 PM, Dominik Psenner <dpsenner@gmail.com> >> wrote: >> > >> > > >> > > to the following: >> > > >> > > <disk type='file' device='disk'> >> > > <driver name='qemu' type='qcow2' cache='none'/> >> > > <source file='/var/data/virtuals/machines/windows-server-2016- >> > > x64/image.qcow2'/> >> > > <backingStore/> >> > > <target dev='hda' bus='scsi'/> >> > > <address type='drive' controller='0' bus='0' target='0' unit='0'/> >> > > </disk> >> > > >> > > Do you see any gotchas in this configuration that could prevent the >> > > virtualized guest to power on and boot up? >> > > >> > > >> > When I configure like this, from a linux guest point of view I get this >> > Symbios Logic SCSI Controller: >> > 00:08.0 SCSI storage controller: LSI Logic / Symbios Logic 53c895a >> > >> > But htis is true only if you add the SCSI controller too, not only the >> disk >> > definition. >> > In my case >> > >> > <controller type='scsi' index='0'> >> > <address type='pci' domain='0x0000' bus='0x00' slot='0x08' >> > function='0x0'/> >> > </controller> >> > >> > Note the slot='0x08' that is reflected into the first field of lspci >> inside >> > my linux guest. >> > So between your controllers you have to add the SCSI one >> > >> > In my case (Fedora 25 with virt-manager-1.4.1-2.fc25.noarch, >> > qemu-kvm-2.7.1-6.fc25.x86_64, libvirt-2.2.1-2.fc25.x86_64) with "Disk >> bus" >> > set as SCSI in virt-manager, the xml defintiion for the guest is >> > automatically updated with the controller if not existent yet. >> > And the disk definition sections is like this: >> > >> > <disk type='file' device='disk'> >> > <driver name='qemu' type='qcow2'/> >> > <source file='/var/lib/libvirt/images/slaxsmall.qcow2'/> >> > <target dev='sda' bus='scsi'/> >> > <boot order='1'/> >> > <address type='drive' controller='0' bus='0' target='0' unit='0'/> >> > </disk> >> > >> > So I think you should set dev='sda' and not 'hda' in your xml for it >> > >> I am actually very curious to know if that would make a difference. I >> don't have a such windows vm images ready to test at present. >> >> Dan >> > I don't kknow if w2016 contains the symbios logic drivers already >> > installed, so that a "simple" reboot could imply an automatic >> > reconfiguration of the guest.... >> > Note also that in Windows when the hw configuration is considered >> heavily >> > changed, you could be asked to register again (I don't think that the >> IDE >> > --> SCSI should imply it...) >> > >> > Gianluca >> >> > _______________________________________________ >> > libvirt-users mailing list >> > libvirt-users@redhat.com >> > https://www.redhat.com/mailman/listinfo/libvirt-users >> >> > > > -- > Dominik Psenner >-- Dominik Psenner
mybe this is because you physic host memory is small then this will Causing instability of the virtual machine But I'm just guessing You can try to increase your memory Wang Liming 发件人: libvirt-users-bounces@redhat.com [mailto:libvirt-users-bounces@redhat.com] 代表 Dominik Psenner 发送时间: 2017年7月2日 16:22 收件人: libvirt-users@redhat.com 主题: Re: [libvirt-users] virtual drive performance Hi again, just today an issue I've thought to be resolved popped up again. We backup the machine by doing: virsh snapshot-create-as --domain domain --name backup --no-metadata --atomic --disk-only --diskspec hda,snapshot=external # backup hda.qcow2 virsh blockcommit domain hda --active --pivot Every now and then this process fails with the following error message: error: failed to pivot job for disk hda error: block copy still active: disk 'hda' not ready for pivot yet Could not merge changes for disk hda of domain. VM may be in invalid state. I expect live backups are a great asset and should work. Is this a bug that may relates also to the virtual drive performance issues we observe? Cheers 2017-07-02 10:10 GMT+02:00 Dominik Psenner <dpsenner@gmail.com>: Hi a small update on this. I just migrated the vm from the site to my laptop and fired it up. The exact same xml configuration (except file paths and such) starts up and bursts with 50Mb/s to 115Mb/s in the guest. This allows only one reasonable answer: the cpu on my laptop is somehow better suited to emulate IO than the CPU built into the host on site. The host there is a HP proliant microserver gen8 with xeon processor. But the processor there is also never capped at 100% when the guest copies files. I just ran another test by copying a 3Gb large file on the guest. What I can observe on my computer is that the copy process is not at a constant rate but rather starts with 90Mb/s, then drops down to 30Mb/s, goes up to 70Mb/s, drops down to 1Mb/s, goes up to 75Mb/s, drops to 1Mb/s, goes up to 55Mb/s and the pattern continues. Please note that the drive is still configured as: <driver name='qemu' type='qcow2' cache='none' io='threads'/> and I would expect a constant rate that is either high or low since there is no caching involved and the underlying hard drive is a samsung ssd evo 850. To have an idea how fast that drive is on my laptop: $ dd if=/dev/zero of=testfile bs=1M count=1000 oflag=direct 1000+0 records in 1000+0 records out 1048576000 bytes (1.0 GB, 1000 MiB) copied, 2.47301 s, 424 MB/s I can further observe that the smaller the saved chunks are the slower the overall performance is: dd if=/dev/zero of=testfile bs=512K count=1000 oflag=direct 1000+0 records in 1000+0 records out 524288000 bytes (524 MB, 500 MiB) copied, 1.34874 s, 389 MB/s $ dd if=/dev/zero of=testfile bs=5K count=1000 oflag=direct 1000+0 records in 1000+0 records out 5120000 bytes (5.1 MB, 4.9 MiB) copied, 0.105109 s, 48.7 MB/s $ dd if=/dev/zero of=testfile bs=1K count=10000 oflag=direct 10000+0 records in 10000+0 records out 10240000 bytes (10 MB, 9.8 MiB) copied, 0.668438 s, 15.3 MB/s $ dd if=/dev/zero of=testfile bs=512 count=20000 oflag=direct 20000+0 records in 20000+0 records out 10240000 bytes (10 MB, 9.8 MiB) copied, 1.10964 s, 9.2 MB/s Could this be a limiting factor? Does qemu/kvm do many many writes of just a few bytes? Ideas, anyone? Cheers 2017-06-21 20:46 GMT+02:00 Dan <srwx4096@gmail.com>: On Tue, Jun 20, 2017 at 04:24:32PM +0200, Gianluca Cecchi wrote:> On Tue, Jun 20, 2017 at 3:38 PM, Dominik Psenner <dpsenner@gmail.com> wrote: > > > > > to the following: > > > > <disk type='file' device='disk'> > > <driver name='qemu' type='qcow2' cache='none'/> > > <source file='/var/data/virtuals/machines/windows-server-2016- > > x64/image.qcow2'/> > > <backingStore/> > > <target dev='hda' bus='scsi'/> > > <address type='drive' controller='0' bus='0' target='0' unit='0'/> > > </disk> > > > > Do you see any gotchas in this configuration that could prevent the > > virtualized guest to power on and boot up? > > > > > When I configure like this, from a linux guest point of view I get this > Symbios Logic SCSI Controller: > 00:08.0 SCSI storage controller: LSI Logic / Symbios Logic 53c895a > > But htis is true only if you add the SCSI controller too, not only the disk > definition. > In my case > > <controller type='scsi' index='0'> > <address type='pci' domain='0x0000' bus='0x00' slot='0x08' > function='0x0'/> > </controller> > > Note the slot='0x08' that is reflected into the first field of lspci inside > my linux guest. > So between your controllers you have to add the SCSI one > > In my case (Fedora 25 with virt-manager-1.4.1-2.fc25.noarch, > qemu-kvm-2.7.1-6.fc25.x86_64, libvirt-2.2.1-2.fc25.x86_64) with "Disk bus" > set as SCSI in virt-manager, the xml defintiion for the guest is > automatically updated with the controller if not existent yet. > And the disk definition sections is like this: > > <disk type='file' device='disk'> > <driver name='qemu' type='qcow2'/> > <source file='/var/lib/libvirt/images/slaxsmall.qcow2'/> > <target dev='sda' bus='scsi'/> > <boot order='1'/> > <address type='drive' controller='0' bus='0' target='0' unit='0'/> > </disk> > > So I think you should set dev='sda' and not 'hda' in your xml for it >I am actually very curious to know if that would make a difference. I don't have a such windows vm images ready to test at present. Dan> I don't kknow if w2016 contains the symbios logic drivers already > installed, so that a "simple" reboot could imply an automatic > reconfiguration of the guest.... > Note also that in Windows when the hw configuration is considered heavily > changed, you could be asked to register again (I don't think that the IDE > --> SCSI should imply it...) > > Gianluca> _______________________________________________ > libvirt-users mailing list > libvirt-users@redhat.com > https://www.redhat.com/mailman/listinfo/libvirt-users-- Dominik Psenner -- Dominik Psenner
Just a little catch-up. This time I was able to resolve the issue by doing:
virsh blockjob domain hda --abort
virsh blockcommit domain hda --active --pivot
Last time I had to shut down the virtual machine and do this while being
offline.
Thanks Wang for your valuable input. As far as the memory goes, there's
plenty of head room:
$ free -h
total used free shared buff/cache
available
Mem: 7.8G 1.8G 407M 9.7M 5.5G
5.5G
Swap: 8.0G 619M 7.4G
2017-07-02 10:26 GMT+02:00 王李明 <wanglm@certusnet.com.cn>:
> mybe this is because you physic host memory is small
>
> then this will Causing instability of the virtual machine
>
> But I'm just guessing
>
> You can try to increase your memory
>
>
>
>
>
> Wang Liming
>
>
>
>
>
> *发件人:* libvirt-users-bounces@redhat.com [mailto:libvirt-users-bounces@
> redhat.com] *代表 *Dominik Psenner
> *发送时间:* 2017年7月2日 16:22
> *收件人:* libvirt-users@redhat.com
> *主题:* Re: [libvirt-users] virtual drive performance
>
>
>
> Hi again,
>
> just today an issue I've thought to be resolved popped up again. We
backup
> the machine by doing:
>
> virsh snapshot-create-as --domain domain --name backup --no-metadata
> --atomic --disk-only --diskspec hda,snapshot=external
>
> # backup hda.qcow2
>
> virsh blockcommit domain hda --active --pivot
>
> Every now and then this process fails with the following error message:
>
> error: failed to pivot job for disk hda
> error: block copy still active: disk 'hda' not ready for pivot yet
> Could not merge changes for disk hda of domain. VM may be in invalid state.
>
> I expect live backups are a great asset and should work. Is this a bug
> that may relates also to the virtual drive performance issues we observe?
>
> Cheers
>
>
>
> 2017-07-02 10:10 GMT+02:00 Dominik Psenner <dpsenner@gmail.com>:
>
> Hi
>
> a small update on this. I just migrated the vm from the site to my laptop
> and fired it up. The exact same xml configuration (except file paths and
> such) starts up and bursts with 50Mb/s to 115Mb/s in the guest. This allows
> only one reasonable answer: the cpu on my laptop is somehow better suited
> to emulate IO than the CPU built into the host on site. The host there is a
> HP proliant microserver gen8 with xeon processor. But the processor there
> is also never capped at 100% when the guest copies files.
>
> I just ran another test by copying a 3Gb large file on the guest. What I
> can observe on my computer is that the copy process is not at a constant
> rate but rather starts with 90Mb/s, then drops down to 30Mb/s, goes up to
> 70Mb/s, drops down to 1Mb/s, goes up to 75Mb/s, drops to 1Mb/s, goes up to
> 55Mb/s and the pattern continues. Please note that the drive is still
> configured as:
>
> <driver name='qemu' type='qcow2' cache='none'
io='threads'/>
>
> and I would expect a constant rate that is either high or low since there
> is no caching involved and the underlying hard drive is a samsung ssd evo
> 850. To have an idea how fast that drive is on my laptop:
>
> $ dd if=/dev/zero of=testfile bs=1M count=1000 oflag=direct
> 1000+0 records in
> 1000+0 records out
> 1048576000 bytes (1.0 GB, 1000 MiB) copied, 2.47301 s, 424 MB/s
>
>
>
> I can further observe that the smaller the saved chunks are the slower the
> overall performance is:
>
> dd if=/dev/zero of=testfile bs=512K count=1000 oflag=direct
> 1000+0 records in
> 1000+0 records out
> 524288000 bytes (524 MB, 500 MiB) copied, 1.34874 s, 389 MB/s
>
> $ dd if=/dev/zero of=testfile bs=5K count=1000 oflag=direct
> 1000+0 records in
> 1000+0 records out
> 5120000 bytes (5.1 MB, 4.9 MiB) copied, 0.105109 s, 48.7 MB/s
>
> $ dd if=/dev/zero of=testfile bs=1K count=10000 oflag=direct
> 10000+0 records in
> 10000+0 records out
> 10240000 bytes (10 MB, 9.8 MiB) copied, 0.668438 s, 15.3 MB/s
>
> $ dd if=/dev/zero of=testfile bs=512 count=20000 oflag=direct
> 20000+0 records in
> 20000+0 records out
> 10240000 bytes (10 MB, 9.8 MiB) copied, 1.10964 s, 9.2 MB/s
>
> Could this be a limiting factor? Does qemu/kvm do many many writes of just
> a few bytes?
>
>
> Ideas, anyone?
>
> Cheers
>
>
>
> 2017-06-21 20:46 GMT+02:00 Dan <srwx4096@gmail.com>:
>
> On Tue, Jun 20, 2017 at 04:24:32PM +0200, Gianluca Cecchi wrote:
> > On Tue, Jun 20, 2017 at 3:38 PM, Dominik Psenner
<dpsenner@gmail.com>
> wrote:
> >
> > >
> > > to the following:
> > >
> > > <disk type='file' device='disk'>
> > > <driver name='qemu' type='qcow2'
cache='none'/>
> > > <source
file='/var/data/virtuals/machines/windows-server-2016-
> > > x64/image.qcow2'/>
> > > <backingStore/>
> > > <target dev='hda' bus='scsi'/>
> > > <address type='drive' controller='0'
bus='0' target='0' unit='0'/>
> > > </disk>
> > >
> > > Do you see any gotchas in this configuration that could prevent
the
> > > virtualized guest to power on and boot up?
> > >
> > >
> > When I configure like this, from a linux guest point of view I get
this
> > Symbios Logic SCSI Controller:
> > 00:08.0 SCSI storage controller: LSI Logic / Symbios Logic 53c895a
> >
> > But htis is true only if you add the SCSI controller too, not only the
> disk
> > definition.
> > In my case
> >
> > <controller type='scsi' index='0'>
> > <address type='pci' domain='0x0000'
bus='0x00' slot='0x08'
> > function='0x0'/>
> > </controller>
> >
> > Note the slot='0x08' that is reflected into the first field of
lspci
> inside
> > my linux guest.
> > So between your controllers you have to add the SCSI one
> >
> > In my case (Fedora 25 with virt-manager-1.4.1-2.fc25.noarch,
> > qemu-kvm-2.7.1-6.fc25.x86_64, libvirt-2.2.1-2.fc25.x86_64) with
"Disk
> bus"
> > set as SCSI in virt-manager, the xml defintiion for the guest is
> > automatically updated with the controller if not existent yet.
> > And the disk definition sections is like this:
> >
> > <disk type='file' device='disk'>
> > <driver name='qemu' type='qcow2'/>
> > <source
file='/var/lib/libvirt/images/slaxsmall.qcow2'/>
> > <target dev='sda' bus='scsi'/>
> > <boot order='1'/>
> > <address type='drive' controller='0'
bus='0' target='0' unit='0'/>
> > </disk>
> >
> > So I think you should set dev='sda' and not 'hda' in
your xml for it
> >
>
> I am actually very curious to know if that would make a difference. I
> don't have a such windows vm images ready to test at present.
>
> Dan
> > I don't kknow if w2016 contains the symbios logic drivers already
> > installed, so that a "simple" reboot could imply an
automatic
> > reconfiguration of the guest....
> > Note also that in Windows when the hw configuration is considered
heavily
> > changed, you could be asked to register again (I don't think that
the IDE
> > --> SCSI should imply it...)
> >
> > Gianluca
>
> > _______________________________________________
> > libvirt-users mailing list
> > libvirt-users@redhat.com
> > https://www.redhat.com/mailman/listinfo/libvirt-users
>
>
>
>
> --
>
> Dominik Psenner
>
>
>
>
> --
>
> Dominik Psenner
>
--
Dominik Psenner
王李明
2017-Jul-04 08:45 UTC
[libvirt-users] The emergence of virtual machines, probabilistic copy large file rate is very low
hi all:
I used openstack+ceph to create a virtual machine for the window 2008
Enterprise Edition sp2. Now I find that almost every day I copy a large 60G file
for the first time under two different folders on the same disk. Replication
rate occurs only 2MB/s. But after this copy is finished, it is normal to copy
again, and it can reach 70MB/s. But the first time the second day of the same
operation, it will appear again. Not necessarily every day, of course
any help will be appricate
Wang Liming