Richard W.M. Jones
2019-Jul-02 07:42 UTC
Re: [Libguestfs] 1.39 proposal: Let's split up the libguestfs git repo and tarballs
On Tue, Jul 02, 2019 at 09:32:26AM +0200, Pino Toscano wrote:> On Monday, 1 July 2019 22:47:32 CEST Richard W.M. Jones wrote: > > > > Does this mean we need to move immediately to a submodule if just > > > > splitting virt-p2v, or copy code as you suggest? Maybe not, because > > > > you can imagine for just this project copying the code needed from the > > > > common/ directory, and creating a new "mini-generator" for the project > > > > which handles the little bits that need to be generated in virt-p2v. > > > > > > I'm actually solving in a different way, i.e. avoiding altogether the > > > generator for p2v stuff. > > > > Hmm. There are parts of the current generator that apply to virt-p2v. > > Which ones? As I explained already, I found only two: > - p2v_config.ml, which is entirely p2v-specific; when converted to > another way, it can be dropped without affecting the rest of the > generated sources of libguestfs > - authors.ml, shared with libguestfs to generate the top-level AUTHORS > file; I did not work on that yet -- my idea for it is to simply > provide a static AUTHORS file for p2v, and generate the C source with > the authors (for the GTK about dialog) using a simple Perl scriptYes, those are the bits I mean.> > Can we split those parts of the generator out to have a new generator > > that only applies to p2v? I find the generated config stuff useful, > > and in fact have a non-upstream patch to enhance it some more. > > While I find the generator great for all the repetitive stuff related > to bindings and actions, I'd rather not statically generate files at > "dist" time if possible. For libguestfs still makes sense because: > - it allows users to build even the C library without OCaml installed > - generating the doc texts of the APIs takes "a lot", even 2 minutes > on fast machines, because of all the pod2text invocations > - few files (AUTHORS, gobject/Makefile.inc) are needed to create a dist > tarball > > OTOH, to me this static generation has few drawbacks: > - generated files in dist tarballs, which makes it more tricky to patch > them > - a slightly different workflow than other generated files (say C > sources from flex/bison/gperf) > > In the case of p2v, I simply do not see the need for OCaml, neither at > development time nor at simple tarball building time, and not even for > a static generator in general. The generated files (with the notable > exception of the p2v-config.h file) can be quickly generated at build > time, with no drawback needed.If you've come up with a way to deal with it I'm sure it'll be fine. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
Richard W.M. Jones
2019-Nov-29 12:09 UTC
[Libguestfs] Continuing the split (was: Let's split up the libguestfs git repo and tarballs)
So, the difficulty of git submodules aside, we have now split off virt-v2v and virt-p2v into separate projects. I also yesterday split off the boot analysis tools into a repo which is likely to be rarely used and which I'll probably not bother to package in Fedora. https://github.com/libguestfs/libguestfs-analysis-tools Where do we go from here? I think one of three ways. Either: * (1) * All libguestfs tools, whether written in C or OCaml, are sufficiently similar to each other and so are moved into a new libguestfs-tools project. libguestfs.git would end up containing only the daemon, core library and language bindings. (Splitting out the language bindings is technically difficult because of their interdependence with the generator, and the language bindings are closely related to the API, so I don't see the point of splitting them out anyway.) Or: * (2) * Libguestfs tools in OCaml are different, and heavier-weight, from the C ones so we put those in a new project (named ...?) The libguestfs tools in C are "lightweight" in some sense so we keep those in libguestfs.git. Note that the OCaml tools are quite interdependent on each other, eg. most of them use virt-customize code. So splitting them further is difficult. Or: * (3) * Libguestfs tools in C and OCaml are different from both libguestfs and each other, and so we should move them into two new projects (again, naming suggestions welcome). I'm not especially wedded to a particular approach, but I would like to get something going on this sooner rather than later. Rich. PS. If you want to follow the rest of this lengthy thread then: https://www.redhat.com/archives/libguestfs/2018-February/thread.html#00067 https://www.redhat.com/archives/libguestfs/2019-April/thread.html#00260 https://www.redhat.com/archives/libguestfs/2019-June/thread.html#00095 -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
Daniel P. Berrangé
2019-Nov-29 12:24 UTC
Re: [Libguestfs] Continuing the split (was: Let's split up the libguestfs git repo and tarballs)
On Fri, Nov 29, 2019 at 12:09:47PM +0000, Richard W.M. Jones wrote:> So, the difficulty of git submodules aside, we have now split off > virt-v2v and virt-p2v into separate projects. > > I also yesterday split off the boot analysis tools into a repo which > is likely to be rarely used and which I'll probably not bother to > package in Fedora. > https://github.com/libguestfs/libguestfs-analysis-tools > > Where do we go from here? > > I think one of three ways. Either: > > * (1) * All libguestfs tools, whether written in C or OCaml, are > sufficiently similar to each other and so are moved into a new > libguestfs-tools project.>From your original mail I see thissmall tools written in C (virt-cat, virt-filesystems, virt-log, virt-ls, virt-tail, virt-diff, virt-edit, virt-format, guestmount, virt-inspector, virt-make-fs, virt-rescue) guestfish virt-alignment-scan and virt-df virt-dib virt-get-kernel virt-resize virt-sparsify virt-v2v and virt-p2v virt-win-reg Most of these are fairly low level core tools just exposing some libguestfs APIs / functionality in a slightly more convenient way than guestfish, whether in C or OCaml. v2v & p2v are full applications in their own right & have split already which makes alot of sense. The other one here that feels like an application, as opposed to a convience wrapper, is virt-dib. So I think perhaps that one could be a separate repo. Everything else feels fine to be bundled as one to me.> libguestfs.git would end up containing only the daemon, core library > and language bindings. (Splitting out the language bindings is > technically difficult because of their interdependence with the > generator, and the language bindings are closely related to the API, > so I don't see the point of splitting them out anyway.)What about "guestfish" ? Do you consider that "core" or an add-on tool ? To me it feels like a core thing you'd expect to always get when you have libguestfs, so perhaps that should be in the main repo rather than tools repo.> Or: > > * (2) * Libguestfs tools in OCaml are different, and heavier-weight, > from the C ones so we put those in a new project (named ...?) The > libguestfs tools in C are "lightweight" in some sense so we keep those > in libguestfs.git.Users shouldn't know or care what language the tool uses, so exposing an arbitrary split of tools into 2 groups based on language feels undesirable. Doubly undesirable if there's a chance some of those C tools will get ported to OCaml.> * (3) * Libguestfs tools in C and OCaml are different from both > libguestfs and each other, and so we should move them into two new > projects (again, naming suggestions welcome).Same thoughts as option (2).> I'm not especially wedded to a particular approach, but I would like > to get something going on this sooner rather than later.To me it looks like something close to (1) is most sensible. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Pino Toscano
2019-Nov-29 16:04 UTC
Re: [Libguestfs] Continuing the split (was: Let's split up the libguestfs git repo and tarballs)
On Friday, 29 November 2019 13:09:47 CET Richard W.M. Jones wrote:> So, the difficulty of git submodules aside, we have now split off > virt-v2v and virt-p2v into separate projects. > > I also yesterday split off the boot analysis tools into a repo which > is likely to be rarely used and which I'll probably not bother to > package in Fedora. > https://github.com/libguestfs/libguestfs-analysis-tools > > Where do we go from here?Before going on with splitting, I'd take a "pause" to review the status of the projects, after their split. Considering the work that was already done to split them, and various difficulties we faced, there is enough material to check what still need to be improved/fixed, as most probably the same issues will happen in case of further splits. The biggest issue I see so far is the impossibility to build virt-v2v (the only split project that requires libguestfs) with an uninstalled version of libguestfs. I see (at least) two problems. 1) building C programs. Usually this means having a .pc file for pkg-config in PKG_CONFIG_PATH, with headers and library in the location of that file. An approach I've seen done is to create a -uninstalled.pc file with its paths pointing to the source/build directories, and making sure the path of that -uninstalled.pc file is in PKG_CONFIG_PATH (which ./run can do). I never tried to do this, I can give it a try on Monday to see how it works. 2) building OCaml programs. It seems like ocamlfind.findlib has the possibility to locate modules in non-standard directories (see findlib.conf(5)), however I never tried that. Most probably a certain file structure is expected, which is definitely not what we have right now. IMHO solving (1) is definitely a must, otherwise there is no way to test new APIs outside of the repository with libguestfs (the library). Possibly also (2) is a must -- for example, earlier I added to the OCaml binding this: https://github.com/libguestfs/libguestfs/commit/4aa4edf972bc8df8869cdcaa4ab4f40b63a626cf the idea is to use it in virt-v2v (not for an important change anyway), however right now it is not possible. Thanks, -- Pino Toscano
Reasonably Related Threads
- Re: 1.39 proposal: Let's split up the libguestfs git repo and tarballs
- Re: Continuing the split (was: Let's split up the libguestfs git repo and tarballs)
- Continuing the split (was: Let's split up the libguestfs git repo and tarballs)
- Re: 1.39 proposal: Let's split up the libguestfs git repo and tarballs
- 1.39 proposal: Let's split up the libguestfs git repo and tarballs