libvirt users - Sep 2011 - Problem with allocation of big files

Hello,

I found a strange problem with allocationg big files on drive.

For example in guestfish I allocate large disk image
for example:

$ guestfish
><fs> allocate test.img 20G
When an image allocating, there is a big slowdown of guest OSes,
launched on host machine, and on different drives.

For Linux guests with virtio drivers, there is no so big performance
penalty, but for FreeBSD guests with ide it seems like the OS locked.

Manipulating with ionice and niceness do not affect at all.

Manipulating with cfq, deadline schedulers also do not have any results.

OS Debian Squeeze
libvirtd (libvirt) 0.9.2
QEMU emulator version 0.14.0 (Debian 0.14.0+dfsg-5.1), Copyright (c)
2003-2008 Fabrice Bellard
guestfish 1.10.3 with some local patches, backported from main tree

Does anyone have ideas how to fix it?

-- 
Nikita A Menkovich
http://libc6.org/
JID: menkovich at gmail.com

On Tue, Sep 20, 2011 at 11:38:15AM +0400, Nikita A Menkovich
wrote:
> Hello,
> 
> I found a strange problem with allocationg big files on drive.
> 
> For example in guestfish I allocate large disk image
> for example:
> 
> $ guestfish
> ><fs> allocate test.img 20G
> 
> When an image allocating, there is a big slowdown of guest OSes,
> launched on host machine, and on different drives.
I'm guessing this is a problem with the host kernel (Linux?).
Specifically you may be saturating SATA, your Southbridge, or PCI if
it's using that.

Here is the code anyway:

http://git.annexia.org/?p=libguestfs.git;a=blob;f=fish/alloc.c;h=7799e4e1c5f0b34ce079934eea7d7c0a13ecdcd3;hb=refs/heads/stable-1.10#l85

Notice that the 'allocate' command takes the non-sparse code path.

What happens next depends on:

 - does your operating system have the posix_fallocate call?

 - does your filesystem support efficient fallocate? (ext3: no, ext4: yes,
   xfs: yes)

If the answer to either question is 'no' then you end up in a loop
which writes zeroes.  For a 20 GB file this is going to be slow and is
going to saturate SATA.

If the answer to *both* questions is 'yes', then ext4/xfs should be
able to allocate the required extents and lazily initialize it.  This
is typically very fast, almost instantaneous.
> For Linux guests with virtio drivers, there is no so big performance
> penalty, but for FreeBSD guests with ide it seems like the OS locked.
> 
> Manipulating with ionice and niceness do not affect at all.
> 
> Manipulating with cfq, deadline schedulers also do not have any results.
> 
> OS Debian Squeeze
> libvirtd (libvirt) 0.9.2
> QEMU emulator version 0.14.0 (Debian 0.14.0+dfsg-5.1), Copyright (c)
> 2003-2008 Fabrice Bellard
> guestfish 1.10.3 with some local patches, backported from main tree
> 
> Does anyone have ideas how to fix it?
Probably the best idea is to use ext4, or use 'sparse' instead of
'allocate' (but with 'sparse' you pay the same penalty, just you
pay
for it later on).

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into Xen guests.
http://et.redhat.com/~rjones/virt-p2v