Libguestfs - May 2020 - Re: Provide NBD via Browser over Websockets

On Mon, 15 Oct 2018, Nir Soffer wrote:
> On Sat, Oct 13, 2018 at 9:45 PM Eric Wheeler <nbd@lists.ewheeler.net>
wrote:
>       Hello all,
> 
>       It might be neat to attach ISOs to KVM guests via websockets. 
Basically
>       the  browser would be the NBD "server" and an NBD client
would run on the
>       hypervisor, then use `virsh change-media vm1 hdc --insert /dev/nbd0`
could
>       use an ISO from my desk to boot from.
> 
>       Here's an HTML5 open file example:
>      
https://stackoverflow.com/questions/3582671/how-to-open-a-local-disk-file-with-javascript
> 
>       and the NBD protocol looks simple enough to implement in javascript:
>      
https://stackoverflow.com/questions/17295140/where-is-the-network-block-device-format-describled
> 
>       What do you think? Does anyone have an interest in doing this?
> 
> 
> HTML File API is very limited:
> - you cannot access any file except file provided by the user interactively
> - no support for sparseness or underlying disk block size
> 
> So it will be a pretty bad backend for NBD server.
> 
> What are you trying to do?
Hi Nir and Eric,

I hope you are well!  

(I'm resurecting this old thread, not sure how I missed the replies.)

We are interested in attaching a local ISO to a remote VM over http (maybe 
.qcow's, but ISO is the primary interest).

This is common for remote KVM (iDRAC/iLO/iKVM/CIMC), so wondering about 
an http front-end for qemu to do the same.

Combining that with a spice JS client or noVNC for VM console access would 
be neat.

I also like Eric Blake's idea of direct NBD client integration with qemu 
instead of using /dev/nbd0.

-Eric
> 
> Nir
> 
>

On Thu, May 28, 2020 at 12:24:22AM +0000, Eric Wheeler
wrote:
> On Mon, 15 Oct 2018, Nir Soffer wrote:
> > On Sat, Oct 13, 2018 at 9:45 PM Eric Wheeler
<nbd@lists.ewheeler.net> wrote:
> >       Hello all,
> > 
> >       It might be neat to attach ISOs to KVM guests via websockets. 
Basically
> >       the  browser would be the NBD "server" and an NBD
client would run on the
> >       hypervisor, then use `virsh change-media vm1 hdc --insert
/dev/nbd0` could
> >       use an ISO from my desk to boot from.
> > 
> >       Here's an HTML5 open file example:
> >      
https://stackoverflow.com/questions/3582671/how-to-open-a-local-disk-file-with-javascript
> > 
> >       and the NBD protocol looks simple enough to implement in
javascript:
> >      
https://stackoverflow.com/questions/17295140/where-is-the-network-block-device-format-describled
> > 
> >       What do you think? Does anyone have an interest in doing this?
> > 
> > 
> > HTML File API is very limited:
> > - you cannot access any file except file provided by the user
interactively
> > - no support for sparseness or underlying disk block size
> > 
> > So it will be a pretty bad backend for NBD server.
> > 
> > What are you trying to do?
> 
> Hi Nir and Eric,
> 
> I hope you are well!  
> 
> (I'm resurecting this old thread, not sure how I missed the replies.)
> 
> We are interested in attaching a local ISO to a remote VM over http (maybe 
> .qcow's, but ISO is the primary interest).
So this bit is simple, and something we do regularly.  Assuming the
"remote VM" is using qemu as a hypervisor, then qemu has a curl driver
integrated so you can mount remote http:// or https:// resources as
disks on the VM directly.  eg:

  $ qemu-system-x86_64 -m 2048 -cdrom
https://download.fedoraproject.org/pub/fedora/linux/releases/32/Server/x86_64/iso/Fedora-Server-dvd-x86_64-32-1.6.iso

(A more realistic example would use libvirt, see the https protocol
in: https://libvirt.org/formatdomain.html#elementsDisks )

You can also bounce the connection through nbdkit-curl-plugin which
gives you a bit more flexibility, eg the ability to transparently
uncompress the remote ISO.  You would be using the NBD client in qemu
to connect (eg. over a Unix domain socket) to a local nbdkit, and
nbdkit's curl plugin to connect to the remote ISO.  More info and a
worked example in these pages:

  http://libguestfs.org/nbdkit-curl-plugin.1.html
  https://rwmj.wordpress.com/2018/11/23/nbdkit-xz-curl/
> This is common for remote KVM (iDRAC/iLO/iKVM/CIMC), so wondering about 
> an http front-end for qemu to do the same.
I'm a bit confused by what you mean by "http front-end for qemu".
> Combining that with a spice JS client or noVNC for VM console access would 
> be neat.
These are separate issues.  qemu has a VNC or SPICE client built in
for the console.
> I also like Eric Blake's idea of direct NBD client integration with
qemu
> instead of using /dev/nbd0.
You shouldn't need to use a loop device for this scenario, assuming
I've understood what you want.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v

On Thu, 28 May 2020, Richard W.M. Jones wrote:
> On Thu, May 28, 2020 at 12:24:22AM +0000, Eric Wheeler wrote:
> > On Mon, 15 Oct 2018, Nir Soffer wrote:
> > > On Sat, Oct 13, 2018 at 9:45 PM Eric Wheeler
<nbd@lists.ewheeler.net> wrote:
> > >       Hello all,
> > > 
> > >       It might be neat to attach ISOs to KVM guests via
websockets.  Basically
> > >       the  browser would be the NBD "server" and an
NBD client would run on the
> > >       hypervisor, then use `virsh change-media vm1 hdc --insert
/dev/nbd0` could
> > >       use an ISO from my desk to boot from.
> > > 
> > >       Here's an HTML5 open file example:
> > >      
https://stackoverflow.com/questions/3582671/how-to-open-a-local-disk-file-with-javascript
> > > 
> > >       and the NBD protocol looks simple enough to implement in
javascript:
> > >      
https://stackoverflow.com/questions/17295140/where-is-the-network-block-device-format-describled
> > > 
> > >       What do you think? Does anyone have an interest in doing
this?
> > > 
> > > 
> > > HTML File API is very limited:
> > > - you cannot access any file except file provided by the user
interactively
> > > - no support for sparseness or underlying disk block size
> > > 
> > > So it will be a pretty bad backend for NBD server.
> > > 
> > > What are you trying to do?
> > 
> > Hi Nir and Eric,
> > 
> > I hope you are well!  
> > 
> > (I'm resurecting this old thread, not sure how I missed the
replies.)
> > 
> > We are interested in attaching a local ISO to a remote VM over http
(maybe
> > .qcow's, but ISO is the primary interest).
> 
> So this bit is simple, and something we do regularly.  Assuming the
> "remote VM" is using qemu as a hypervisor, then qemu has a curl
driver
> integrated so you can mount remote http:// or https:// resources as
> disks on the VM directly.  eg:
> 
>   $ qemu-system-x86_64 -m 2048 -cdrom
https://download.fedoraproject.org/pub/fedora/linux/releases/32/Server/x86_64/iso/Fedora-Server-dvd-x86_64-32-1.6.iso
> 
> (A more realistic example would use libvirt, see the https protocol
> in: https://libvirt.org/formatdomain.html#elementsDisks )
Cool feature, I hadn't realized this was an option.
> You can also bounce the connection through nbdkit-curl-plugin which
> gives you a bit more flexibility, eg the ability to transparently
> uncompress the remote ISO.  You would be using the NBD client in qemu
> to connect (eg. over a Unix domain socket) to a local nbdkit, and
> nbdkit's curl plugin to connect to the remote ISO.  More info and a
> worked example in these pages:
> 
>   http://libguestfs.org/nbdkit-curl-plugin.1.html
>   https://rwmj.wordpress.com/2018/11/23/nbdkit-xz-curl/
Looks neat, perhaps we would write a plugin to integrate with a 
webserver (cgi, php, etc) for the JS client instead of implementing the 
whole protocol over websockets.
 
> > This is common for remote KVM (iDRAC/iLO/iKVM/CIMC), so wondering
about
> > an http front-end for qemu to do the same.
> > Combining that with a spice JS client or noVNC for VM console access
would
> > be neat.
> 
> I'm a bit confused by what you mean by "http front-end for
qemu".
When qemu is running headless using a VNC or Spice display we can access 
the display of https+websockets using things like noVNC---which is out of 
scope to this converstation---but I'm just saying that such an existing 
web front-end for the display could be extended to have an "Insert
CDROM"
button and use the javascript file IO for the user to reference a local 
file and pass it to qemu over nbd, perhaps via nbdkit.
> > I also like Eric Blake's idea of direct NBD client integration
with qemu
> > instead of using /dev/nbd0.
> 
> You shouldn't need to use a loop device for this scenario, assuming
> I've understood what you want.
True.

-Eric
> Rich.
> 
> -- 
> Richard Jones, Virtualization Group, Red Hat
http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-p2v converts physical machines to virtual machines.  Boot with a
> live CD or over the network (PXE) and turn machines into KVM guests.
> http://libguestfs.org/virt-v2v
> 
>