Libguestfs - Mar 2016 - Improving supermin appliance startup time (lkvm/qboot)

Hi,

I remembered reading about Intel Clear Containers [1], and Hyper/Qboot [2]
that support startup times measured in the hundreds of miliseconds range.

On an AMD FX(tm)-8350 'guestfish -a /dev/null run' takes ~4s when run
the 2nd/3rd time:
real	0m4.152s
user	0m2.120s
sys	0m0.564s

Are there any plans on applying these improvements to the supermin appliance?

I did some quick tests on what would be possible, see below:

If I try to use virtio 9P instead it becomes a little faster (instead of root fs
on block device):
real	0m3.299s
user	0m2.084s
sys	0m1.828s

To do this we need to load just the virtio and 9p modules:
let kmods = [
"virtio*.ko*";
"9p*.ko*";
]
Modify init.c to mount the 9p root:
https://gist.github.com/anonymous/cb32986fd9b85c315ae4
And wrap qemu to create the 9p device (the mount is just there for a quick test,
in reality supermin itself should probably leave the unpacker dir around):
https://gist.github.com/anonymous/5cdbc18974f88d6ea9e0

Next step: kvmtool [3]. Avoids some delays introduced by legacy BIOS boot, and
device probing, however
lkvm currently lacks support for additional virtio-serial channels, so -
although the supermin appliance would  boot - guestfsd wouldn't be able to
communicate with the host:
https://gist.github.com/anonymous/e6b0a12e1da811f24e5b

I tried QBoot [4], which requires that you first strip down your kernel+initrd
to fit into 8128k,
and uses an unmodified Qemu otherwise.
But then I haven't been able to get it to actually launch guestfsd, it hangs
somewhere earlier:
https://gist.github.com/anonymous/52fc2890a0884230e4f8
Maybe I removed too many modules here?

Another possibility would be to run guestfsd with rkt+lkvm stage1, or hyper.sh;
but I don't see a way
to forward a unix socket or a virtio-serial channel through them, would guestfsd
be able to work over a TCP socket
or the main virtio console? (its probably preferable to avoid having any
networking in the guestfsd appliance)

[1] https://lwn.net/Articles/644675/
[2] https://hyper.sh/faq.html
[3] https://git.kernel.org/cgit/linux/kernel/git/will/kvmtool.git/tree/README
[4] https://github.com/bonzini/qboot

Best regards,
-- 
Edwin Török | Co-founder and Lead Developer

Skylable open-source object storage: reliable, fast, secure
http://www.skylable.com

On Sun, Mar 13, 2016 at 04:24:53PM +0200, Török Edwin
wrote:
> Hi,
> 
> I remembered reading about Intel Clear Containers [1], and Hyper/Qboot [2]
> that support startup times measured in the hundreds of miliseconds range.
> 
> On an AMD FX(tm)-8350 'guestfish -a /dev/null run' takes ~4s when
run the 2nd/3rd time:
> real	0m4.152s
> user	0m2.120s
> sys	0m0.564s
> 
> Are there any plans on applying these improvements to the supermin
appliance?
I was looking at Clear Containers last week.
> I did some quick tests on what would be possible, see below:
> 
> If I try to use virtio 9P instead it becomes a little faster (instead of
root fs on block device):
> real	0m3.299s
> user	0m2.084s
> sys	0m1.828s
> 
> To do this we need to load just the virtio and 9p modules:
> let kmods = [
> "virtio*.ko*";
> "9p*.ko*";
> ]
> Modify init.c to mount the 9p root:
https://gist.github.com/anonymous/cb32986fd9b85c315ae4
> And wrap qemu to create the 9p device (the mount is just there for a quick
test, in reality supermin itself should probably leave the unpacker dir around):
> https://gist.github.com/anonymous/5cdbc18974f88d6ea9e0
> 
> Next step: kvmtool [3]. Avoids some delays introduced by legacy BIOS boot,
and device probing, however
> lkvm currently lacks support for additional virtio-serial channels, so -
although the supermin appliance would  boot - guestfsd wouldn't be able to
communicate with the host:
> https://gist.github.com/anonymous/e6b0a12e1da811f24e5b
> 
> I tried QBoot [4], which requires that you first strip down your
kernel+initrd to fit into 8128k,
> and uses an unmodified Qemu otherwise.
> But then I haven't been able to get it to actually launch guestfsd, it
hangs somewhere earlier:
> https://gist.github.com/anonymous/52fc2890a0884230e4f8
> Maybe I removed too many modules here?
> 
> Another possibility would be to run guestfsd with rkt+lkvm stage1, or
hyper.sh; but I don't see a way
> to forward a unix socket or a virtio-serial channel through them, would
guestfsd be able to work over a TCP socket
> or the main virtio console? (its probably preferable to avoid having any
networking in the guestfsd appliance)
> 
> [1] https://lwn.net/Articles/644675/
> [2] https://hyper.sh/faq.html
> [3]
https://git.kernel.org/cgit/linux/kernel/git/will/kvmtool.git/tree/README
> [4] https://github.com/bonzini/qboot
This is all very good analysis.

The issues that I had in brief were:

(1) We could run kvmtool, perhaps by adding a new backend, but it
seems a better idea to add the required features to qemu.  Anything
based on kvmtool wouldn't support qcow2 and the myriad other features
of qemu (see also:
https://rwmj.wordpress.com/2015/11/07/linux-kernel-library-backend-for-libguestfs/)

(2) On the kernel side, the Intel kernel contains lots of little
hacks, and many baremetal CONFIG_* options are disabled.  Hacks can be
upstreamed once we massage them into shape.  The real issue is keeping
a single baremetal + virt kernel image, since separate kernel images
would never fly with Fedora or RHEL.  That means we cannot just
disable slow drivers/subsystems by having an alternate kernel with
lots of CONFIG_* options disabled, we need to address those problems
properly.

(3) DAX / NVDIMM / etc - love them.  Not supported upstream (either
kernel or qemu) yet.

(4) Passthrough (eg 9p) filesystems.  You touched on that above.
Red Hat doesn't much like 9pfs for several reasons, yet we also don't
have a plausible alternative at the moment.  This is mainly a problem
for running fast Docker containers, rather than libguestfs though.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

On 03/15/2016 21:34, Richard W.M. Jones wrote:
> I was looking at Clear Containers last week.
> [...]
> 
> This is all very good analysis.
Thanks, looks like I raised the question at a good time :)
> 
> The issues that I had in brief were:
> 
> (1) We could run kvmtool, perhaps by adding a new backend, but it
> seems a better idea to add the required features to qemu.  Anything
> based on kvmtool wouldn't support qcow2 and the myriad other features
> of qemu (see also:
>
https://rwmj.wordpress.com/2015/11/07/linux-kernel-library-backend-for-libguestfs/)
Could qemu-nbd from inside the guest be used? (as a performance/security
tradeoff)
> 
> (2) On the kernel side, the Intel kernel contains lots of little
> hacks, and many baremetal CONFIG_* options are disabled.  Hacks can be
> upstreamed once we massage them into shape.  The real issue is keeping
> a single baremetal + virt kernel image, since separate kernel images
> would never fly with Fedora or RHEL.  That means we cannot just
> disable slow drivers/subsystems by having an alternate kernel with
> lots of CONFIG_* options disabled, we need to address those problems
> properly.
> 
> (3) DAX / NVDIMM / etc - love them.  Not supported upstream (either
> kernel or qemu) yet.
> 
> (4) Passthrough (eg 9p) filesystems.  You touched on that above.
> Red Hat doesn't much like 9pfs for several reasons, yet we also
don't
> have a plausible alternative at the moment.  This is mainly a problem
> for running fast Docker containers, rather than libguestfs though.
Another thought: why does guestfish need to boot the appliance more than once?
Could virsh save/restore or managedsave/stop/start be used?

The guestfs appliance seems to be around 80MB saved [*] (perhaps ballooning
could help shrink this, or it could be compressed with lz4/lzop).

[*] I copied the XML and changed some things in it, cause the original failed to
save with: error: Requested operation is not valid: domain is marked for auto
destroy

Best regards,
-- 
Edwin Török | Co-founder and Lead Developer

Skylable open-source object storage: reliable, fast, secure
http://www.skylable.com