Libguestfs - Oct 2019 - Re: Splitting the large libguestfs repo

On Tue, Oct 15, 2019 at 03:59:04PM +0200, Pino Toscano
wrote:
> On Tuesday, 15 October 2019 10:01:28 CEST Richard W.M. Jones wrote:
> > I got a little way into this.  The two attached patches are
> > preliminary work.
> 
> I see the work was done already, so I guess providing alternative ideas
> (or opinions, apparently) is of no use now...
It's always valued.
> > My proposed split is:
> > 
> >       libguestfs.git
> >           common -> git submodule libguestfs-common.git
> >           generator/
> >           lib/
> >           all language bindings
> >           C based tools (eg. virt-df, virt-edit, guestfish)
> >     
> >       guestfs-tools.git
> >           common -> git submodule libguestfs-common.git
> >           virt-builder, virt-customize, virt-sparsify, virt-sparsify,
etc
> 
> I do not think splitting these tools in an own repository makes much
> sense.  What is the goal/advantage you get by splitting them in an own
> repository, compared to the ones left in libguestfs.git?  Users do not
> care about virt-customize written in OCaml rather than C, and it makes
> harder to eventually rewrite a C tool in OCaml.
This part of the split isn't absolutely necessary, I really wanted to
concentrate on virt-v2v and get that done, partly just because dealing
with virt-v2v inside the bigger repo is such a pain.

You're right that people don't care about what a tool is written in,
so another idea might be to put the C _and_ OCaml tools into the
guestfs-tools.git repo (leaving lib + daemon + language bindings only
in libguestfs.git).  In fact I like this better now I think about it.
> > The current common/ subdirectory would become a git submodule.  While
> > git submodules are awkward, they do solve this particular problem with
> > having common code shared across the repositories, there's only
one
> > git submodule and it's under our control.  It does mean that any
time
> > there's a change to common/, we would need to add a commit to the
> > other 3 repos updating the submodule hash.
> 
> The current common/ subdirectory is a giant mixup of different
> components needed by some or just one tool each; few examples:
> - common/mlaugeas -> only for the daemon
> - common/mllibvirt -> only for v2v
> - common/mlxml -> only for v2v
> - some of the ml modules are need by any OCaml stuff
> - some of the ml modules are used by 1/2 tools
> - etc
It's a step on the path rather than the end point.  We can definitely
move mllibvirt & mlxml to virt-v2v in future.  We can also try to
organize it better or split it in future.

The alternatives to the submodule are somehow packaging everything up
into a library, which would then become a build dep of the tools.
This isn't particularly nice because (as you say) it's a big mix of
miscellaneous "stuff", essentially a core library of missing C and
OCaml functionality that we happened to need.  So therefore not useful
as an independent library.

I'm not overjoyed by git submodules, but in this particular instance I
think it can make sense, and there's only one of them (at the moment,
we could change that).

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v

On Tuesday, 15 October 2019 18:39:32 CEST Richard W.M. Jones
wrote:
> On Tue, Oct 15, 2019 at 03:59:04PM +0200, Pino Toscano wrote:
> > On Tuesday, 15 October 2019 10:01:28 CEST Richard W.M. Jones wrote:
> > > I got a little way into this.  The two attached patches are
> > > preliminary work.
> > 
> > I see the work was done already, so I guess providing alternative
ideas
> > (or opinions, apparently) is of no use now...
> 
> It's always valued.
Of limited use though, if things are already rolling toward a specific
direction.
> > > My proposed split is:
> > > 
> > >       libguestfs.git
> > >           common -> git submodule libguestfs-common.git
> > >           generator/
> > >           lib/
> > >           all language bindings
> > >           C based tools (eg. virt-df, virt-edit, guestfish)
> > >     
> > >       guestfs-tools.git
> > >           common -> git submodule libguestfs-common.git
> > >           virt-builder, virt-customize, virt-sparsify,
virt-sparsify, etc
> > 
> > I do not think splitting these tools in an own repository makes much
> > sense.  What is the goal/advantage you get by splitting them in an own
> > repository, compared to the ones left in libguestfs.git?  Users do not
> > care about virt-customize written in OCaml rather than C, and it makes
> > harder to eventually rewrite a C tool in OCaml.
> 
> This part of the split isn't absolutely necessary, I really wanted to
> concentrate on virt-v2v and get that done, partly just because dealing
> with virt-v2v inside the bigger repo is such a pain.
TBH I would have concentrated on virt-v2v only, just like I handled
virt-p2v on its own.
> You're right that people don't care about what a tool is written
in,
> so another idea might be to put the C _and_ OCaml tools into the
> guestfs-tools.git repo (leaving lib + daemon + language bindings only
> in libguestfs.git).  In fact I like this better now I think about it.
Still some of the questions I wrote above apply: what do we gain from
splitting the tools in an own repository? Having them in an own
repository means that, unles you use the API yourself (in C/Python/etc)
then you need to build another repository to do image
manipulation/customization.

The tools (other than v2v) are quite stable, and they do not have many
changes these days. Looking at the release notes for libguestfs 1.38,
and 1.40 (the latest two stable series), the biggest changes were
- add --key in all the tools
- output selection for --machine-readable in OCaml tools
whose main implementations were in common code anyway.

This also crosses another topic: how is virt-v2v going to be versioned,
and released? Similar question for the potential split of the tools.
If the answer is "packed together in a single tarball like now", then
splitting makes the work more complex only to developers, i.e. us.
> > > The current common/ subdirectory would become a git submodule. 
While
> > > git submodules are awkward, they do solve this particular problem
with
> > > having common code shared across the repositories, there's
only one
> > > git submodule and it's under our control.  It does mean that
any time
> > > there's a change to common/, we would need to add a commit to
the
> > > other 3 repos updating the submodule hash.
> > 
> > The current common/ subdirectory is a giant mixup of different
> > components needed by some or just one tool each; few examples:
> > - common/mlaugeas -> only for the daemon
> > - common/mllibvirt -> only for v2v
> > - common/mlxml -> only for v2v
> > - some of the ml modules are need by any OCaml stuff
> > - some of the ml modules are used by 1/2 tools
> > - etc
> 
> It's a step on the path rather than the end point.  We can definitely
> move mllibvirt & mlxml to virt-v2v in future.  We can also try to
> organize it better or split it in future.
Considering that the changes are deep in the structure of the
libguestfs sources, having a bit more light on the proposed path would
certainly help to understand it better, and check whether the direction
is sound.
> The alternatives to the submodule are somehow packaging everything up
> into a library, which would then become a build dep of the tools.
> This isn't particularly nice because (as you say) it's a big mix of
> miscellaneous "stuff", essentially a core library of missing C
and
> OCaml functionality that we happened to need.  So therefore not useful
> as an independent library.
There is also the fact that this set of common code has no concept of
compatibility, so this potentially needs branches for each stable
series. Also, this crosses the topic above of releasing, w.r.t. bundled
or independent releases.
> I'm not overjoyed by git submodules, but in this particular instance I
> think it can make sense, and there's only one of them (at the moment,
> we could change that).
Right, I'm not thrilled by git submodules either, and so far I have a
bad experience with them.

-- 
Pino Toscano

On Wed, Oct 16, 2019 at 01:57:44PM +0200, Pino Toscano
wrote:
> TBH I would have concentrated on virt-v2v only, just like I handled
> virt-p2v on its own.
I've only done virt-v2v.  I've actually now got it working, including
the make check and make check-slow (although there are absolutely
bound to be latent bugs).  The new repo is here:

  https://github.com/libguestfs/virt-v2v

I'm going to add all the contributors from libguestfs.git so they have
permissions on this repo in a minute.  I've not removed the v2v/
subdirectory from the old repo.
> > You're right that people don't care about what a tool is
written in,
> > so another idea might be to put the C _and_ OCaml tools into the
> > guestfs-tools.git repo (leaving lib + daemon + language bindings only
> > in libguestfs.git).  In fact I like this better now I think about it.
> 
> Still some of the questions I wrote above apply: what do we gain from
> splitting the tools in an own repository? Having them in an own
> repository means that, unles you use the API yourself (in C/Python/etc)
> then you need to build another repository to do image
> manipulation/customization.
> 
> The tools (other than v2v) are quite stable, and they do not have many
> changes these days. Looking at the release notes for libguestfs 1.38,
> and 1.40 (the latest two stable series), the biggest changes were
> - add --key in all the tools
> - output selection for --machine-readable in OCaml tools
> whose main implementations were in common code anyway.
Well we can park this for a while.  I'd still value a smaller
libguestfs.git repo in the long term.  However getting virt-v2v out
was the main priority because as you say it moves much faster than the
tools and we want to make bigger changes.
> This also crosses another topic: how is virt-v2v going to be versioned,
> and released? Similar question for the potential split of the tools.
So at the moment, virt-v2v has the same version (1.41.6).  It should
build a separate virt-v2v-1.41.X.tar.gz tarball (not actually tested
this yet).  It requires libguestfs >= 1.41.5, although in fact it
would work with libguestfs 1.40, except for literally a single test:

 
https://github.com/libguestfs/virt-v2v/blob/5f355c2952729581f1b988297fe8cabe756c01e2/v2v/test-v2v-o-json.sh#L54

(I might modify that test so it is conditional on the version of
libguestfs found, and then make virt-v2v depend on libguestfs >= 1.40)

Future versioning -- up in the air.  What did we decide about virt-p2v?
> If the answer is "packed together in a single tarball like now",
then
> splitting makes the work more complex only to developers, i.e. us.
I wasn't sure if you meant the common/ subdirectory here, but I'll
answer about the common/ subdirectory: It is packed into both the
libguestfs and virt-v2v tarballs.  The submodule concept doesn't exist
outside git.
> > The alternatives to the submodule are somehow packaging everything up
> > into a library, which would then become a build dep of the tools.
> > This isn't particularly nice because (as you say) it's a big
mix of
> > miscellaneous "stuff", essentially a core library of missing
C and
> > OCaml functionality that we happened to need.  So therefore not useful
> > as an independent library.
> 
> There is also the fact that this set of common code has no concept of
> compatibility, so this potentially needs branches for each stable
> series. Also, this crosses the topic above of releasing, w.r.t. bundled
> or independent releases.
So several things here:

Every time we significantly change libguestfs-common.git we will need
to update libguestfs.git and virt-v2v.git.  This isn't great.

Common isn't versioned, and I don't believe it needs to be because of
the previous point.

common/ subdirectory is bundled in the tarballs, so this shouldn't
affect anyone building tarballs and downstream packagers, except that
there will be a new virt-v2v package.
> > I'm not overjoyed by git submodules, but in this particular
instance I
> > think it can make sense, and there's only one of them (at the
moment,
> > we could change that).
> 
> Right, I'm not thrilled by git submodules either, and so far I have a
> bad experience with them.
Oh yes, me too for sure.  I think when they can work is when you have
full control over all the git modules.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW