Török Edwin
2016-Mar-13 14:24 UTC
[Libguestfs] Improving supermin appliance startup time (lkvm/qboot)
Hi, I remembered reading about Intel Clear Containers [1], and Hyper/Qboot [2] that support startup times measured in the hundreds of miliseconds range. On an AMD FX(tm)-8350 'guestfish -a /dev/null run' takes ~4s when run the 2nd/3rd time: real 0m4.152s user 0m2.120s sys 0m0.564s Are there any plans on applying these improvements to the supermin appliance? I did some quick tests on what would be possible, see below: If I try to use virtio 9P instead it becomes a little faster (instead of root fs on block device): real 0m3.299s user 0m2.084s sys 0m1.828s To do this we need to load just the virtio and 9p modules: let kmods = [ "virtio*.ko*"; "9p*.ko*"; ] Modify init.c to mount the 9p root: https://gist.github.com/anonymous/cb32986fd9b85c315ae4 And wrap qemu to create the 9p device (the mount is just there for a quick test, in reality supermin itself should probably leave the unpacker dir around): https://gist.github.com/anonymous/5cdbc18974f88d6ea9e0 Next step: kvmtool [3]. Avoids some delays introduced by legacy BIOS boot, and device probing, however lkvm currently lacks support for additional virtio-serial channels, so - although the supermin appliance would boot - guestfsd wouldn't be able to communicate with the host: https://gist.github.com/anonymous/e6b0a12e1da811f24e5b I tried QBoot [4], which requires that you first strip down your kernel+initrd to fit into 8128k, and uses an unmodified Qemu otherwise. But then I haven't been able to get it to actually launch guestfsd, it hangs somewhere earlier: https://gist.github.com/anonymous/52fc2890a0884230e4f8 Maybe I removed too many modules here? Another possibility would be to run guestfsd with rkt+lkvm stage1, or hyper.sh; but I don't see a way to forward a unix socket or a virtio-serial channel through them, would guestfsd be able to work over a TCP socket or the main virtio console? (its probably preferable to avoid having any networking in the guestfsd appliance) [1] https://lwn.net/Articles/644675/ [2] https://hyper.sh/faq.html [3] https://git.kernel.org/cgit/linux/kernel/git/will/kvmtool.git/tree/README [4] https://github.com/bonzini/qboot Best regards, -- Edwin Török | Co-founder and Lead Developer Skylable open-source object storage: reliable, fast, secure http://www.skylable.com
Richard W.M. Jones
2016-Mar-15 19:34 UTC
Re: [Libguestfs] Improving supermin appliance startup time (lkvm/qboot)
On Sun, Mar 13, 2016 at 04:24:53PM +0200, Török Edwin wrote:> Hi, > > I remembered reading about Intel Clear Containers [1], and Hyper/Qboot [2] > that support startup times measured in the hundreds of miliseconds range. > > On an AMD FX(tm)-8350 'guestfish -a /dev/null run' takes ~4s when run the 2nd/3rd time: > real 0m4.152s > user 0m2.120s > sys 0m0.564s > > Are there any plans on applying these improvements to the supermin appliance?I was looking at Clear Containers last week.> I did some quick tests on what would be possible, see below: > > If I try to use virtio 9P instead it becomes a little faster (instead of root fs on block device): > real 0m3.299s > user 0m2.084s > sys 0m1.828s > > To do this we need to load just the virtio and 9p modules: > let kmods = [ > "virtio*.ko*"; > "9p*.ko*"; > ] > Modify init.c to mount the 9p root: https://gist.github.com/anonymous/cb32986fd9b85c315ae4 > And wrap qemu to create the 9p device (the mount is just there for a quick test, in reality supermin itself should probably leave the unpacker dir around): > https://gist.github.com/anonymous/5cdbc18974f88d6ea9e0 > > Next step: kvmtool [3]. Avoids some delays introduced by legacy BIOS boot, and device probing, however > lkvm currently lacks support for additional virtio-serial channels, so - although the supermin appliance would boot - guestfsd wouldn't be able to communicate with the host: > https://gist.github.com/anonymous/e6b0a12e1da811f24e5b > > I tried QBoot [4], which requires that you first strip down your kernel+initrd to fit into 8128k, > and uses an unmodified Qemu otherwise. > But then I haven't been able to get it to actually launch guestfsd, it hangs somewhere earlier: > https://gist.github.com/anonymous/52fc2890a0884230e4f8 > Maybe I removed too many modules here? > > Another possibility would be to run guestfsd with rkt+lkvm stage1, or hyper.sh; but I don't see a way > to forward a unix socket or a virtio-serial channel through them, would guestfsd be able to work over a TCP socket > or the main virtio console? (its probably preferable to avoid having any networking in the guestfsd appliance) > > [1] https://lwn.net/Articles/644675/ > [2] https://hyper.sh/faq.html > [3] https://git.kernel.org/cgit/linux/kernel/git/will/kvmtool.git/tree/README > [4] https://github.com/bonzini/qbootThis is all very good analysis. The issues that I had in brief were: (1) We could run kvmtool, perhaps by adding a new backend, but it seems a better idea to add the required features to qemu. Anything based on kvmtool wouldn't support qcow2 and the myriad other features of qemu (see also: https://rwmj.wordpress.com/2015/11/07/linux-kernel-library-backend-for-libguestfs/) (2) On the kernel side, the Intel kernel contains lots of little hacks, and many baremetal CONFIG_* options are disabled. Hacks can be upstreamed once we massage them into shape. The real issue is keeping a single baremetal + virt kernel image, since separate kernel images would never fly with Fedora or RHEL. That means we cannot just disable slow drivers/subsystems by having an alternate kernel with lots of CONFIG_* options disabled, we need to address those problems properly. (3) DAX / NVDIMM / etc - love them. Not supported upstream (either kernel or qemu) yet. (4) Passthrough (eg 9p) filesystems. You touched on that above. Red Hat doesn't much like 9pfs for several reasons, yet we also don't have a plausible alternative at the moment. This is mainly a problem for running fast Docker containers, rather than libguestfs though. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
Török Edwin
2016-Mar-16 17:44 UTC
Re: [Libguestfs] Improving supermin appliance startup time (lkvm/qboot)
On 03/15/2016 21:34, Richard W.M. Jones wrote:> I was looking at Clear Containers last week. > [...] > > This is all very good analysis.Thanks, looks like I raised the question at a good time :)> > The issues that I had in brief were: > > (1) We could run kvmtool, perhaps by adding a new backend, but it > seems a better idea to add the required features to qemu. Anything > based on kvmtool wouldn't support qcow2 and the myriad other features > of qemu (see also: > https://rwmj.wordpress.com/2015/11/07/linux-kernel-library-backend-for-libguestfs/)Could qemu-nbd from inside the guest be used? (as a performance/security tradeoff)> > (2) On the kernel side, the Intel kernel contains lots of little > hacks, and many baremetal CONFIG_* options are disabled. Hacks can be > upstreamed once we massage them into shape. The real issue is keeping > a single baremetal + virt kernel image, since separate kernel images > would never fly with Fedora or RHEL. That means we cannot just > disable slow drivers/subsystems by having an alternate kernel with > lots of CONFIG_* options disabled, we need to address those problems > properly. > > (3) DAX / NVDIMM / etc - love them. Not supported upstream (either > kernel or qemu) yet. > > (4) Passthrough (eg 9p) filesystems. You touched on that above. > Red Hat doesn't much like 9pfs for several reasons, yet we also don't > have a plausible alternative at the moment. This is mainly a problem > for running fast Docker containers, rather than libguestfs though.Another thought: why does guestfish need to boot the appliance more than once? Could virsh save/restore or managedsave/stop/start be used? The guestfs appliance seems to be around 80MB saved [*] (perhaps ballooning could help shrink this, or it could be compressed with lz4/lzop). [*] I copied the XML and changed some things in it, cause the original failed to save with: error: Requested operation is not valid: domain is marked for auto destroy Best regards, -- Edwin Török | Co-founder and Lead Developer Skylable open-source object storage: reliable, fast, secure http://www.skylable.com
Seemingly Similar Threads
- Improving supermin appliance startup time (lkvm/qboot)
- Re: Improving supermin appliance startup time (lkvm/qboot)
- Re: [PATCH supermin 5/5] init: Drop SCSI modules.
- [RFC] kvmtool: add support for modern virtio-pci
- [RFC] kvmtool: add support for modern virtio-pci