Nir Soffer
2019-Nov-22 23:58 UTC
Re: [Libguestfs] Shebang sh plugins (was: Re: [PATCH nbdkit v2 03/10] python: Implement nbdkit API version 2.)
On Fri, Nov 22, 2019 at 11:07 PM Richard W.M. Jones <rjones@redhat.com> wrote:> > On Fri, Nov 22, 2019 at 02:55:31PM -0600, Eric Blake wrote: > > Unrelated side topic: in your recent addition of eval.sh, you > > wondered if we should promote it to a full-blown plugin rather than > > just an example script. But reading 'man nbdkit-sh-plugin', there > > is no mention of turning an executable script into a full-blown > > plugin via a shebang, the way that python documents it. [I guess > > 'man nbdkit' sort of mentions it under Shebang scripts] > > I believe it's not possible to do it for sh plugins. > > For (eg) python plugins it works like this: > > #!/usr/sbin/nbdkit python > -> runs nbdkit python <script name> > -> <script name> is interpreted as magic script= parameter > -> the python plugin works by loading the script using > PyRun_SimpleFileEx which interprets the contents of the > script as python code > > However for shell it doesn't work: > > #!/usr/sbin/nbdkit sh > -> runs nbdkit sh <script name> > -> <script name> is interpreted as magic script= parameter > -> the sh plugin works by actually executing the script > -> executing the script repeats the steps over from the top, > causing an infinite loop > > If you can think of a way to make this work it would be a useful > feature IMO.Even if we can find a way, it sounds too complicated, and I don't see the use case. Nir
Richard W.M. Jones
2019-Nov-23 15:46 UTC
Re: [Libguestfs] Shebang sh plugins (was: Re: [PATCH nbdkit v2 03/10] python: Implement nbdkit API version 2.)
On Sat, Nov 23, 2019 at 01:58:02AM +0200, Nir Soffer wrote:> On Fri, Nov 22, 2019 at 11:07 PM Richard W.M. Jones <rjones@redhat.com> wrote: > > > > On Fri, Nov 22, 2019 at 02:55:31PM -0600, Eric Blake wrote: > > > Unrelated side topic: in your recent addition of eval.sh, you > > > wondered if we should promote it to a full-blown plugin rather than > > > just an example script. But reading 'man nbdkit-sh-plugin', there > > > is no mention of turning an executable script into a full-blown > > > plugin via a shebang, the way that python documents it. [I guess > > > 'man nbdkit' sort of mentions it under Shebang scripts] > > > > I believe it's not possible to do it for sh plugins. > > > > For (eg) python plugins it works like this: > > > > #!/usr/sbin/nbdkit python > > -> runs nbdkit python <script name> > > -> <script name> is interpreted as magic script= parameter > > -> the python plugin works by loading the script using > > PyRun_SimpleFileEx which interprets the contents of the > > script as python code > > > > However for shell it doesn't work: > > > > #!/usr/sbin/nbdkit sh > > -> runs nbdkit sh <script name> > > -> <script name> is interpreted as magic script= parameter > > -> the sh plugin works by actually executing the script > > -> executing the script repeats the steps over from the top, > > causing an infinite loop > > > > If you can think of a way to make this work it would be a useful > > feature IMO. > > Even if we can find a way, it sounds too complicated, and I don't see > the use case.I think an aim of nbdkit is to make it as widely usable as possible. That's for example why we have apparent esoterica like plugins written in shell script. We want to in the future ship the (or "an") imageio plugin separately from virt-v2v. A possibility is it might be shipped as a single Python file which starts with ‘#!/usr/bin/env nbdkit python’ which would allow anyone to run it directly. They wouldn't need to understand anything about nbdkit or its command line. So that's the use case (which to be fair is not actually used much today at all). The context above is I can't think of a way to extend this feature to nbdkit-sh-plugin. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/
Nir Soffer
2019-Nov-24 13:31 UTC
Re: [Libguestfs] Shebang sh plugins (was: Re: [PATCH nbdkit v2 03/10] python: Implement nbdkit API version 2.)
On Sat, Nov 23, 2019 at 5:46 PM Richard W.M. Jones <rjones@redhat.com> wrote:> > On Sat, Nov 23, 2019 at 01:58:02AM +0200, Nir Soffer wrote: > > On Fri, Nov 22, 2019 at 11:07 PM Richard W.M. Jones <rjones@redhat.com> wrote: > > > > > > On Fri, Nov 22, 2019 at 02:55:31PM -0600, Eric Blake wrote: > > > > Unrelated side topic: in your recent addition of eval.sh, you > > > > wondered if we should promote it to a full-blown plugin rather than > > > > just an example script. But reading 'man nbdkit-sh-plugin', there > > > > is no mention of turning an executable script into a full-blown > > > > plugin via a shebang, the way that python documents it. [I guess > > > > 'man nbdkit' sort of mentions it under Shebang scripts] > > > > > > I believe it's not possible to do it for sh plugins. > > > > > > For (eg) python plugins it works like this: > > > > > > #!/usr/sbin/nbdkit python > > > -> runs nbdkit python <script name> > > > -> <script name> is interpreted as magic script= parameter > > > -> the python plugin works by loading the script using > > > PyRun_SimpleFileEx which interprets the contents of the > > > script as python code > > > > > > However for shell it doesn't work: > > > > > > #!/usr/sbin/nbdkit sh > > > -> runs nbdkit sh <script name> > > > -> <script name> is interpreted as magic script= parameter > > > -> the sh plugin works by actually executing the script > > > -> executing the script repeats the steps over from the top, > > > causing an infinite loop > > > > > > If you can think of a way to make this work it would be a useful > > > feature IMO. > > > > Even if we can find a way, it sounds too complicated, and I don't see > > the use case. > > I think an aim of nbdkit is to make it as widely usable as possible. > That's for example why we have apparent esoterica like plugins written > in shell script. > > We want to in the future ship the (or "an") imageio plugin separately > from virt-v2v. A possibility is it might be shipped as a single > Python file which starts with ‘#!/usr/bin/env nbdkit python’ which > would allow anyone to run it directly. They wouldn't need to > understand anything about nbdkit or its command line.But the imageio plugin is not a script you can run, and if you want to make it a standalone tool it needs a lot of work that is not related to virt-v2v like adding command line parsing or dealing with file formats. Such tool mostly already exists in ovirt sdk examples: https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/examples/upload_disk.py So for rhv-upload there is no use case for standalone script.> So that's the use case (which to be fair is not actually used much > today at all). The context above is I can't think of a way to extend > this feature to nbdkit-sh-plugin.But today we require the plugin name in nbdkit command line: ./nbdkit -f -v python zero.py It could be nice to be able to do: ./nbdkit -f -v zero.py Detect that the filename is a python/lua/shell/perl,... file and then load the right plugin. We can use special comments in the plugin to provide more info about the plugin, for example arguments that can be used with the plugin. $ cat example.py # example python plugin # nbdkit-plugin: python # nbdkit-argument: filename code... $ ./nbdkit -f -v example.py --help show help about example.py usage... $ ./nbdkit -f -v example.py filename=/dev/zero Internally nbdkit will replace this command with: nbdkit -f -v python example.py filename=/dev/zero And continue the normal flow. So the actual nbdkit command is: nbdkit plugin-name|script ... Nir
On 11/23/19 9:46 AM, Richard W.M. Jones wrote:> I think an aim of nbdkit is to make it as widely usable as possible. > That's for example why we have apparent esoterica like plugins written > in shell script. > > We want to in the future ship the (or "an") imageio plugin separately > from virt-v2v. A possibility is it might be shipped as a single > Python file which starts with ‘#!/usr/bin/env nbdkit python’ whichThat is not portable; not all kernels permit more than one argument in #!. You can work around it with modern env (both BSD and Coreutils have recently added -S), as: #!/usr/bin/env -S nbdkit python which will then invoke the desired 'nbdkit' 'python' 'file' regardless of whether the OS invoked 'env' '-S nbdkit python' (Linux) or 'env' '-S' 'nbdkit' 'python' (others). But that in turn assumes env -S is portable (it's fairly new, so not yet). Otherwise, interpreted plugin shebangs really do have to hard-code the expected path of nbdkit rather than relying on env, unless we find a way to make: nbdkit file determine that when 'nbdkit-plugin-file.so' does not exist but 'file' does, then inspect file to determine if 'nbdkit-plugin-interpreter.so' exists based on the first line of file. Except that would mean the first line changes to: #!/usr/bin/env nbdkit which lacks the desired interpreter, so we'd still have to add something like documenting that if the first line of the #! contains 'nbdkit', then the second line is some form of magic comment that includes the substring 'PLUGIN=...' (we have to allow arbitrary slop as prefix or suffix after a plugin name, to allow the second line to be a comment or otherwise innocuous interpreted line in whatever desired target language), so we end up with: #!/usr/bin/env nbdkit # PLUGIN=python where the initial invocation of '/usr/bin/env' 'nbdkit' 'file' then passes on to '/path/to/nbdkit' 'file' which sees that 'file' is not a plugin, but does have magic, so it then retries as if invoked as '/path/to/nbdkit' 'python' 'file' -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Seemingly Similar Threads
- Re: Shebang sh plugins (was: Re: [PATCH nbdkit v2 03/10] python: Implement nbdkit API version 2.)
- Re: Shebang sh plugins (was: Re: [PATCH nbdkit v2 03/10] python: Implement nbdkit API version 2.)
- Shebang sh plugins (was: Re: [PATCH nbdkit v2 03/10] python: Implement nbdkit API version 2.)
- [PATCH nbdkit] plugins: python: Add imageio plugin example
- Re: [PATCH nbdkit v2 03/10] python: Implement nbdkit API version 2.