Libguestfs - Nov 2019 - Continuing the split (was: Let's split up the libguestfs git repo and tarballs)

On Tue, Jul 02, 2019 at 09:32:26AM +0200, Pino Toscano
wrote:
> On Monday, 1 July 2019 22:47:32 CEST Richard W.M. Jones wrote:
> > > > Does this mean we need to move immediately to a submodule if
just
> > > > splitting virt-p2v, or copy code as you suggest?  Maybe not,
because
> > > > you can imagine for just this project copying the code
needed from the
> > > > common/ directory, and creating a new
"mini-generator" for the project
> > > > which handles the little bits that need to be generated in
virt-p2v.
> > > 
> > > I'm actually solving in a different way, i.e. avoiding
altogether the
> > > generator for p2v stuff.
> > 
> > Hmm.  There are parts of the current generator that apply to virt-p2v.
> 
> Which ones? As I explained already, I found only two:
> - p2v_config.ml, which is entirely p2v-specific; when converted to
>   another way, it can be dropped without affecting the rest of the
>   generated sources of libguestfs
> - authors.ml, shared with libguestfs to generate the top-level AUTHORS
>   file; I did not work on that yet -- my idea for it is to simply
>   provide a static AUTHORS file for p2v, and generate the C source with
>   the authors (for the GTK about dialog) using a simple Perl script
Yes, those are the bits I mean.
> > Can we split those parts of the generator out to have a new generator
> > that only applies to p2v?  I find the generated config stuff useful,
> > and in fact have a non-upstream patch to enhance it some more.
> 
> While I find the generator great for all the repetitive stuff related
> to bindings and actions, I'd rather not statically generate files at
> "dist" time if possible.  For libguestfs still makes sense
because:
> - it allows users to build even the C library without OCaml installed
> - generating the doc texts of the APIs takes "a lot", even 2
minutes
>   on fast machines, because of all the pod2text invocations
> - few files (AUTHORS, gobject/Makefile.inc) are needed to create a dist
>   tarball
> 
> OTOH, to me this static generation has few drawbacks:
> - generated files in dist tarballs, which makes it more tricky to patch
>   them
> - a slightly different workflow than other generated files (say C
>   sources from flex/bison/gperf)
> 
> In the case of p2v, I simply do not see the need for OCaml, neither at
> development time nor at simple tarball building time, and not even for
> a static generator in general.  The generated files (with the notable
> exception of the p2v-config.h file) can be quickly generated at build
> time, with no drawback needed.
If you've come up with a way to deal with it I'm sure it'll be fine.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html

So, the difficulty of git submodules aside, we have now split off
virt-v2v and virt-p2v into separate projects.

I also yesterday split off the boot analysis tools into a repo which
is likely to be rarely used and which I'll probably not bother to
package in Fedora.
https://github.com/libguestfs/libguestfs-analysis-tools

Where do we go from here?

I think one of three ways.  Either:

* (1) * All libguestfs tools, whether written in C or OCaml, are
sufficiently similar to each other and so are moved into a new
libguestfs-tools project.

libguestfs.git would end up containing only the daemon, core library
and language bindings.  (Splitting out the language bindings is
technically difficult because of their interdependence with the
generator, and the language bindings are closely related to the API,
so I don't see the point of splitting them out anyway.)

Or:

* (2) * Libguestfs tools in OCaml are different, and heavier-weight,
from the C ones so we put those in a new project (named ...?)  The
libguestfs tools in C are "lightweight" in some sense so we keep those
in libguestfs.git.

Note that the OCaml tools are quite interdependent on each other,
eg. most of them use virt-customize code.  So splitting them further
is difficult.

Or:

* (3) * Libguestfs tools in C and OCaml are different from both
libguestfs and each other, and so we should move them into two new
projects (again, naming suggestions welcome).


I'm not especially wedded to a particular approach, but I would like
to get something going on this sooner rather than later.

Rich.


PS.
If you want to follow the rest of this lengthy thread then:
https://www.redhat.com/archives/libguestfs/2018-February/thread.html#00067
https://www.redhat.com/archives/libguestfs/2019-April/thread.html#00260
https://www.redhat.com/archives/libguestfs/2019-June/thread.html#00095

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html

On Fri, Nov 29, 2019 at 12:09:47PM +0000, Richard W.M. Jones
wrote:
> So, the difficulty of git submodules aside, we have now split off
> virt-v2v and virt-p2v into separate projects.
> 
> I also yesterday split off the boot analysis tools into a repo which
> is likely to be rarely used and which I'll probably not bother to
> package in Fedora.
> https://github.com/libguestfs/libguestfs-analysis-tools
> 
> Where do we go from here?
> 
> I think one of three ways.  Either:
> 
> * (1) * All libguestfs tools, whether written in C or OCaml, are
> sufficiently similar to each other and so are moved into a new
> libguestfs-tools project.
>From your original mail I see this
       small tools written in C
          (virt-cat, virt-filesystems, virt-log, virt-ls, virt-tail,
	   virt-diff, virt-edit, virt-format, guestmount, virt-inspector,
	   virt-make-fs, virt-rescue)

       guestfish

       virt-alignment-scan and virt-df

       virt-dib	   

       virt-get-kernel

       virt-resize

       virt-sparsify

       virt-v2v and virt-p2v

       virt-win-reg

Most of these are fairly low level core tools just exposing some
libguestfs APIs / functionality in a slightly more convenient way
than guestfish, whether in C or OCaml.

v2v & p2v are full applications in their own right & have split
already which makes alot of sense.

The other one here that feels like an application, as opposed to
a convience wrapper, is virt-dib. So I think perhaps that one
could be a separate repo. Everything else feels fine to be bundled
as one to me.
> libguestfs.git would end up containing only the daemon, core library
> and language bindings.  (Splitting out the language bindings is
> technically difficult because of their interdependence with the
> generator, and the language bindings are closely related to the API,
> so I don't see the point of splitting them out anyway.)
What about "guestfish" ?  Do you consider that "core" or an
add-on tool ? To me it feels like a core thing you'd expect
to always get when you have libguestfs, so perhaps that should
be in the main repo rather than tools repo.
> Or:
> 
> * (2) * Libguestfs tools in OCaml are different, and heavier-weight,
> from the C ones so we put those in a new project (named ...?)  The
> libguestfs tools in C are "lightweight" in some sense so we keep
those
> in libguestfs.git.
Users shouldn't know or care what language the tool uses,
so exposing an arbitrary split of tools into 2 groups based
on language feels undesirable. Doubly undesirable if there's
a chance some of those C tools will get ported to OCaml.

> * (3) * Libguestfs tools in C and OCaml are different from both
> libguestfs and each other, and so we should move them into two new
> projects (again, naming suggestions welcome).
Same thoughts as option (2).
> I'm not especially wedded to a particular approach, but I would like
> to get something going on this sooner rather than later.
To me it looks like something close to (1) is most sensible.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On Friday, 29 November 2019 13:09:47 CET Richard W.M. Jones
wrote:
> So, the difficulty of git submodules aside, we have now split off
> virt-v2v and virt-p2v into separate projects.
> 
> I also yesterday split off the boot analysis tools into a repo which
> is likely to be rarely used and which I'll probably not bother to
> package in Fedora.
> https://github.com/libguestfs/libguestfs-analysis-tools
> 
> Where do we go from here?
Before going on with splitting, I'd take a "pause" to review the
status
of the projects, after their split. Considering the work that was
already done to split them, and various difficulties we faced, there is
enough material to check what still need to be improved/fixed, as most
probably the same issues will happen in case of further splits.

The biggest issue I see so far is the impossibility to build virt-v2v
(the only split project that requires libguestfs) with an uninstalled
version of libguestfs. I see (at least) two problems.

1) building C programs.
Usually this means having a .pc file for pkg-config in PKG_CONFIG_PATH,
with headers and library in the location of that file.  An approach
I've seen done is to create a -uninstalled.pc file with its paths
pointing to the source/build directories, and making sure the path of
that -uninstalled.pc file is in PKG_CONFIG_PATH (which ./run can do).
I never tried to do this, I can give it a try on Monday to see how it
works.

2) building OCaml programs.
It seems like ocamlfind.findlib has the possibility to locate modules
in non-standard directories (see findlib.conf(5)), however I never
tried that.  Most probably a certain file structure is expected, which
is definitely not what we have right now.

IMHO solving (1) is definitely a must, otherwise there is no way to
test new APIs outside of the repository with libguestfs (the library).
Possibly also (2) is a must -- for example, earlier I added to the OCaml
binding this:
https://github.com/libguestfs/libguestfs/commit/4aa4edf972bc8df8869cdcaa4ab4f40b63a626cf
the idea is to use it in virt-v2v (not for an important change anyway),
however right now it is not possible.

Thanks,
-- 
Pino Toscano