On Tue, Dec 01, 2009 at 04:08:38PM +0100, Pierre Riteau wrote:> For a project of mine I would like to create VMs from tarballs of > root filesystems. I would like to use libguestfs since this is > exactly what the tar2vm.sh guestfish recipe does. > > However, I need to extract tarballs that are in the hundreds of > megabytes, but my tests showed that performance is very low, both > with tgz-in and with guestmount (sorry I don't have the numbers any > more, but it was something like more than one hour to extract a > tarball containing a Debian Lenny installation). > > Is there any way to drastically improve libguestfs' performance?First of all, read: http://libguestfs.org/FAQ.html#slow Using tar-in/tgz-in to copy the tarball into the appliance is always going to be slow. I would think you should get better performance if you write the tarball into an ISO first, and attach that. Something like: mkisofs -JR -o cd.iso the_tarball.tar.gz guestfish -a guest.img -a cd.iso [...]><fs> mkmountpoint /guest ><fs> mkmountpoint /cd[...]><fs> mount /dev/sdb /cd[...]><fs> debug sh "tar xcf /cd/the_tarball.tar.gz /guest"However we found experimentally that the fastest method of all for creating distributions was to prebuild an ext2/3/4 image, and 'dd' it over to the target partition, then resize it to the final (larger) size. This method is extremely quick, enabling us to build Fedora images in just a minute or two: http://rwmj.wordpress.com/2009/11/23/prebuilt-distributions-part-2/#content Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://et.redhat.com/~rjones/libguestfs/ See what it can do: http://et.redhat.com/~rjones/libguestfs/recipes.html
On Mon, Dec 07, 2009 at 09:57:08AM +0000, Richard W.M. Jones wrote:> ><fs> debug sh "tar xcf /cd/the_tarball.tar.gz /guest"xcf? I mean zxf. Using debug sh is a hack. If you want to do it this way we should add a structured API call to pack and unpack local tarballs. However I would still recommend the 'dd+resize' approach, as it will always be much faster. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://et.redhat.com/~rjones/virt-top
On 7 d?c. 2009, at 10:57, Richard W.M. Jones wrote:> On Tue, Dec 01, 2009 at 04:08:38PM +0100, Pierre Riteau wrote: >> For a project of mine I would like to create VMs from tarballs of >> root filesystems. I would like to use libguestfs since this is >> exactly what the tar2vm.sh guestfish recipe does. >> >> However, I need to extract tarballs that are in the hundreds of >> megabytes, but my tests showed that performance is very low, both >> with tgz-in and with guestmount (sorry I don't have the numbers any >> more, but it was something like more than one hour to extract a >> tarball containing a Debian Lenny installation). >> >> Is there any way to drastically improve libguestfs' performance? > > First of all, read: > > http://libguestfs.org/FAQ.html#slow > > Using tar-in/tgz-in to copy the tarball into the appliance is always > going to be slow. I would think you should get better performance if > you write the tarball into an ISO first, and attach that. Something > like: > > mkisofs -JR -o cd.iso the_tarball.tar.gz > > guestfish -a guest.img -a cd.iso > [...] >> <fs> mkmountpoint /guest >> <fs> mkmountpoint /cd > [...] >> <fs> mount /dev/sdb /cd > [...] >> <fs> debug sh "tar xcf /cd/the_tarball.tar.gz /guest"I got it to work using the following sequence: $ guestfish -a guest.img -a cd.iso run sfdiskM /dev/sda "," mkfs ext3 /dev/sda1 mkmountpoint /guest mkmountpoint /cd mount /dev/sda1 /guest mount /dev/sdb /cd debug sh "tar -C /sysroot/guest -xzf /sysroot/cd/the_tarball.tar.gz" This is much better (on my test system it writes at about 15 MB/s). However, it is still far from what the hardware can do (more than 60 MB/s). Now the last problem is that I need to install a bootloader. Can I easily install grub in the appliance, or do I need to recreate a new one?> However we found experimentally that the fastest method of all for > creating distributions was to prebuild an ext2/3/4 image, and 'dd' it > over to the target partition, then resize it to the final (larger) > size. This method is extremely quick, enabling us to build Fedora > images in just a minute or two: > > http://rwmj.wordpress.com/2009/11/23/prebuilt-distributions-part-2/#contentI've read your blog, this is interesting. I will see how I can leverage that. -- Pierre Riteau -- http://perso.univ-rennes1.fr/pierre.riteau/