Libguestfs - Jul 2019 - Re: 1.39 proposal: Let's split up the libguestfs git repo and tarballs

On Mon, Jul 01, 2019 at 06:10:48PM +0200, Pino Toscano
wrote:
> On Monday, 10 June 2019 17:35:52 CEST Richard W.M. Jones wrote:
> > So while I'm not a massive fan of git submodules, now that I have
used
> > them a few times with riscv stuff, they do solve a certain problem as
> > long as they are managed carefully.  I think the common code and the
> > generator are cases where a submodule or two would work.
> 
> TBH I've always found submodules tricky and problematic to use:
> - they are fixed to a certain revision (so no way to dynamically follow
>   the branch of another repo)
> - the URL is the same for all the users, meaning you cannot reuse the
>   same authenticated/secure protocols that your repo has
> - they create a certain burden when switching to a tag/branch/commit
>   whose revision of a submodule is different than what is at the current
>   branch
> - even more problematic when switching commit, and in the old commit
>   a subdirectory is a real directory while in the latest HEAD is a
>   submodule (or viceversa)
I mean, I don't disagree with any of this :-)

For riscv we pinned Linux kernel and various toolchains at precise
commits, and then only moved those forwards as we tested new
combinations.

Anyhow, whatever works.
> > Does this mean we need to move immediately to a submodule if just
> > splitting virt-p2v, or copy code as you suggest?  Maybe not, because
> > you can imagine for just this project copying the code needed from the
> > common/ directory, and creating a new "mini-generator" for
the project
> > which handles the little bits that need to be generated in virt-p2v.
> 
> I'm actually solving in a different way, i.e. avoiding altogether the
> generator for p2v stuff.
Hmm.  There are parts of the current generator that apply to virt-p2v.

Can we split those parts of the generator out to have a new generator
that only applies to p2v?  I find the generated config stuff useful,
and in fact have a non-upstream patch to enhance it some more.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

On Monday, 1 July 2019 22:47:32 CEST Richard W.M. Jones
wrote:
> > > Does this mean we need to move immediately to a submodule if just
> > > splitting virt-p2v, or copy code as you suggest?  Maybe not,
because
> > > you can imagine for just this project copying the code needed
from the
> > > common/ directory, and creating a new "mini-generator"
for the project
> > > which handles the little bits that need to be generated in
virt-p2v.
> > 
> > I'm actually solving in a different way, i.e. avoiding altogether
the
> > generator for p2v stuff.
> 
> Hmm.  There are parts of the current generator that apply to virt-p2v.
Which ones? As I explained already, I found only two:
- p2v_config.ml, which is entirely p2v-specific; when converted to
  another way, it can be dropped without affecting the rest of the
  generated sources of libguestfs
- authors.ml, shared with libguestfs to generate the top-level AUTHORS
  file; I did not work on that yet -- my idea for it is to simply
  provide a static AUTHORS file for p2v, and generate the C source with
  the authors (for the GTK about dialog) using a simple Perl script
> Can we split those parts of the generator out to have a new generator
> that only applies to p2v?  I find the generated config stuff useful,
> and in fact have a non-upstream patch to enhance it some more.
While I find the generator great for all the repetitive stuff related
to bindings and actions, I'd rather not statically generate files at
"dist" time if possible.  For libguestfs still makes sense because:
- it allows users to build even the C library without OCaml installed
- generating the doc texts of the APIs takes "a lot", even 2 minutes
  on fast machines, because of all the pod2text invocations
- few files (AUTHORS, gobject/Makefile.inc) are needed to create a dist
  tarball

OTOH, to me this static generation has few drawbacks:
- generated files in dist tarballs, which makes it more tricky to patch
  them
- a slightly different workflow than other generated files (say C
  sources from flex/bison/gperf)

In the case of p2v, I simply do not see the need for OCaml, neither at
development time nor at simple tarball building time, and not even for
a static generator in general.  The generated files (with the notable
exception of the p2v-config.h file) can be quickly generated at build
time, with no drawback needed.

-- 
Pino Toscano

On Tue, Jul 02, 2019 at 09:32:26AM +0200, Pino Toscano
wrote:
> On Monday, 1 July 2019 22:47:32 CEST Richard W.M. Jones wrote:
> > > > Does this mean we need to move immediately to a submodule if
just
> > > > splitting virt-p2v, or copy code as you suggest?  Maybe not,
because
> > > > you can imagine for just this project copying the code
needed from the
> > > > common/ directory, and creating a new
"mini-generator" for the project
> > > > which handles the little bits that need to be generated in
virt-p2v.
> > > 
> > > I'm actually solving in a different way, i.e. avoiding
altogether the
> > > generator for p2v stuff.
> > 
> > Hmm.  There are parts of the current generator that apply to virt-p2v.
> 
> Which ones? As I explained already, I found only two:
> - p2v_config.ml, which is entirely p2v-specific; when converted to
>   another way, it can be dropped without affecting the rest of the
>   generated sources of libguestfs
> - authors.ml, shared with libguestfs to generate the top-level AUTHORS
>   file; I did not work on that yet -- my idea for it is to simply
>   provide a static AUTHORS file for p2v, and generate the C source with
>   the authors (for the GTK about dialog) using a simple Perl script
Yes, those are the bits I mean.
> > Can we split those parts of the generator out to have a new generator
> > that only applies to p2v?  I find the generated config stuff useful,
> > and in fact have a non-upstream patch to enhance it some more.
> 
> While I find the generator great for all the repetitive stuff related
> to bindings and actions, I'd rather not statically generate files at
> "dist" time if possible.  For libguestfs still makes sense
because:
> - it allows users to build even the C library without OCaml installed
> - generating the doc texts of the APIs takes "a lot", even 2
minutes
>   on fast machines, because of all the pod2text invocations
> - few files (AUTHORS, gobject/Makefile.inc) are needed to create a dist
>   tarball
> 
> OTOH, to me this static generation has few drawbacks:
> - generated files in dist tarballs, which makes it more tricky to patch
>   them
> - a slightly different workflow than other generated files (say C
>   sources from flex/bison/gperf)
> 
> In the case of p2v, I simply do not see the need for OCaml, neither at
> development time nor at simple tarball building time, and not even for
> a static generator in general.  The generated files (with the notable
> exception of the p2v-config.h file) can be quickly generated at build
> time, with no drawback needed.
If you've come up with a way to deal with it I'm sure it'll be fine.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html