Libguestfs - Nov 2019 - Re: Shebang sh plugins (was: Re: [PATCH nbdkit v2 03/10] python: Implement nbdkit API version 2.)

On Fri, Nov 22, 2019 at 11:07 PM Richard W.M. Jones <rjones@redhat.com>
wrote:
>
> On Fri, Nov 22, 2019 at 02:55:31PM -0600, Eric Blake wrote:
> > Unrelated side topic: in your recent addition of eval.sh, you
> > wondered if we should promote it to a full-blown plugin rather than
> > just an example script.  But reading 'man nbdkit-sh-plugin',
there
> > is no mention of turning an executable script into a full-blown
> > plugin via a shebang, the way that python documents it.  [I guess
> > 'man nbdkit' sort of mentions it under Shebang scripts]
>
> I believe it's not possible to do it for sh plugins.
>
> For (eg) python plugins it works like this:
>
>   #!/usr/sbin/nbdkit python
>   -> runs nbdkit python <script name>
>   -> <script name> is interpreted as magic script= parameter
>   -> the python plugin works by loading the script using
>      PyRun_SimpleFileEx which interprets the contents of the
>      script as python code
>
> However for shell it doesn't work:
>
>   #!/usr/sbin/nbdkit sh
>   -> runs nbdkit sh <script name>
>   -> <script name> is interpreted as magic script= parameter
>   -> the sh plugin works by actually executing the script
>   -> executing the script repeats the steps over from the top,
>      causing an infinite loop
>
> If you can think of a way to make this work it would be a useful
> feature IMO.
Even if we can find a way, it sounds too complicated, and I don't see
the use case.

Nir

On Sat, Nov 23, 2019 at 01:58:02AM +0200, Nir Soffer
wrote:
> On Fri, Nov 22, 2019 at 11:07 PM Richard W.M. Jones
<rjones@redhat.com> wrote:
> >
> > On Fri, Nov 22, 2019 at 02:55:31PM -0600, Eric Blake wrote:
> > > Unrelated side topic: in your recent addition of eval.sh, you
> > > wondered if we should promote it to a full-blown plugin rather
than
> > > just an example script.  But reading 'man
nbdkit-sh-plugin', there
> > > is no mention of turning an executable script into a full-blown
> > > plugin via a shebang, the way that python documents it.  [I guess
> > > 'man nbdkit' sort of mentions it under Shebang scripts]
> >
> > I believe it's not possible to do it for sh plugins.
> >
> > For (eg) python plugins it works like this:
> >
> >   #!/usr/sbin/nbdkit python
> >   -> runs nbdkit python <script name>
> >   -> <script name> is interpreted as magic script= parameter
> >   -> the python plugin works by loading the script using
> >      PyRun_SimpleFileEx which interprets the contents of the
> >      script as python code
> >
> > However for shell it doesn't work:
> >
> >   #!/usr/sbin/nbdkit sh
> >   -> runs nbdkit sh <script name>
> >   -> <script name> is interpreted as magic script= parameter
> >   -> the sh plugin works by actually executing the script
> >   -> executing the script repeats the steps over from the top,
> >      causing an infinite loop
> >
> > If you can think of a way to make this work it would be a useful
> > feature IMO.
> 
> Even if we can find a way, it sounds too complicated, and I don't see
> the use case.
I think an aim of nbdkit is to make it as widely usable as possible.
That's for example why we have apparent esoterica like plugins written
in shell script.

We want to in the future ship the (or "an") imageio plugin separately
from virt-v2v.  A possibility is it might be shipped as a single
Python file which starts with ‘#!/usr/bin/env nbdkit python’ which
would allow anyone to run it directly.  They wouldn't need to
understand anything about nbdkit or its command line.

So that's the use case (which to be fair is not actually used much
today at all).  The context above is I can't think of a way to extend
this feature to nbdkit-sh-plugin.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/

On Sat, Nov 23, 2019 at 5:46 PM Richard W.M. Jones <rjones@redhat.com>
wrote:
>
> On Sat, Nov 23, 2019 at 01:58:02AM +0200, Nir Soffer wrote:
> > On Fri, Nov 22, 2019 at 11:07 PM Richard W.M. Jones
<rjones@redhat.com> wrote:
> > >
> > > On Fri, Nov 22, 2019 at 02:55:31PM -0600, Eric Blake wrote:
> > > > Unrelated side topic: in your recent addition of eval.sh,
you
> > > > wondered if we should promote it to a full-blown plugin
rather than
> > > > just an example script.  But reading 'man
nbdkit-sh-plugin', there
> > > > is no mention of turning an executable script into a
full-blown
> > > > plugin via a shebang, the way that python documents it.  [I
guess
> > > > 'man nbdkit' sort of mentions it under Shebang
scripts]
> > >
> > > I believe it's not possible to do it for sh plugins.
> > >
> > > For (eg) python plugins it works like this:
> > >
> > >   #!/usr/sbin/nbdkit python
> > >   -> runs nbdkit python <script name>
> > >   -> <script name> is interpreted as magic script=
parameter
> > >   -> the python plugin works by loading the script using
> > >      PyRun_SimpleFileEx which interprets the contents of the
> > >      script as python code
> > >
> > > However for shell it doesn't work:
> > >
> > >   #!/usr/sbin/nbdkit sh
> > >   -> runs nbdkit sh <script name>
> > >   -> <script name> is interpreted as magic script=
parameter
> > >   -> the sh plugin works by actually executing the script
> > >   -> executing the script repeats the steps over from the top,
> > >      causing an infinite loop
> > >
> > > If you can think of a way to make this work it would be a useful
> > > feature IMO.
> >
> > Even if we can find a way, it sounds too complicated, and I don't
see
> > the use case.
>
> I think an aim of nbdkit is to make it as widely usable as possible.
> That's for example why we have apparent esoterica like plugins written
> in shell script.
>
> We want to in the future ship the (or "an") imageio plugin
separately
> from virt-v2v.  A possibility is it might be shipped as a single
> Python file which starts with ‘#!/usr/bin/env nbdkit python’ which
> would allow anyone to run it directly.  They wouldn't need to
> understand anything about nbdkit or its command line.
But the imageio plugin is not a script you can run, and if you want to
make it a standalone tool it needs a lot of work that is not related to virt-v2v
like adding command line parsing or dealing with file formats.

Such tool mostly already exists in ovirt sdk examples:
https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/examples/upload_disk.py

So for rhv-upload there is no use case for standalone script.
> So that's the use case (which to be fair is not actually used much
> today at all).  The context above is I can't think of a way to extend
> this feature to nbdkit-sh-plugin.
But today we require the plugin name in nbdkit command line:

    ./nbdkit -f -v python zero.py

It could be nice to be able to do:

    ./nbdkit -f -v zero.py

Detect that the filename is a python/lua/shell/perl,... file and then
load the right plugin.

We can use special comments in the plugin to provide more info about the plugin,
for example arguments that can be used with the plugin.

$ cat example.py
# example python plugin
# nbdkit-plugin: python
# nbdkit-argument: filename

code...

$ ./nbdkit -f -v example.py --help

show help about example.py usage...

$ ./nbdkit -f -v example.py filename=/dev/zero

Internally nbdkit will replace this command with:

    nbdkit -f -v python example.py filename=/dev/zero

And continue the normal flow.

So the actual nbdkit command is:

   nbdkit plugin-name|script ...

Nir

On 11/23/19 9:46 AM, Richard W.M. Jones wrote:
> I think an aim of nbdkit is to make it as widely usable as possible.
> That's for example why we have apparent esoterica like plugins written
> in shell script.
> 
> We want to in the future ship the (or "an") imageio plugin
separately
> from virt-v2v.  A possibility is it might be shipped as a single
> Python file which starts with ‘#!/usr/bin/env nbdkit python’ which
That is not portable; not all kernels permit more than one argument in 
#!.  You can work around it with modern env (both BSD and Coreutils have 
recently added -S), as:

#!/usr/bin/env -S nbdkit python

which will then invoke the desired 'nbdkit' 'python'
'file' regardless
of whether the OS invoked 'env' '-S nbdkit python' (Linux) or
'env' '-S'
'nbdkit' 'python' (others).  But that in turn assumes env -S is
portable
(it's fairly new, so not yet).  Otherwise, interpreted plugin shebangs 
really do have to hard-code the expected path of nbdkit rather than 
relying on env, unless we find a way to make:

nbdkit file

determine that when 'nbdkit-plugin-file.so' does not exist but
'file'
does, then inspect file to determine if 'nbdkit-plugin-interpreter.so' 
exists based on the first line of file.

Except that would mean the first line changes to:

#!/usr/bin/env nbdkit

which lacks the desired interpreter, so we'd still have to add something 
like documenting that if the first line of the #! contains 'nbdkit', 
then the second line is some form of magic comment that includes the 
substring 'PLUGIN=...' (we have to allow arbitrary slop as prefix or 
suffix after a plugin name, to allow the second line to be a comment or 
otherwise innocuous interpreted line in whatever desired target 
language), so we end up with:

#!/usr/bin/env nbdkit
# PLUGIN=python

where the initial invocation of '/usr/bin/env' 'nbdkit'
'file'
then passes on to '/path/to/nbdkit' 'file'
which sees that 'file' is not a plugin, but does have magic, so it then 
retries as if invoked as '/path/to/nbdkit' 'python'
'file'

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org