Libguestfs - Oct 2021 - Publishing the libnbd go module

I'm playing with libnbd go module, planning to use it in a new command[1]

The biggest obstacle for me is that the module is not published in a way that
Go developers expect.

The module is listed in:
https://pkg.go.dev/github.com/libguestfs/libnbd/golang

But the module actually lives in:
https://github.com/libguestfs/libnbd/tree/master/golang/src/libguestfs.org/libnbd

So the pkg.go.dev page is broken, .e.g no there is no documation or license, and
the suggested import is wrong.

The module name is "libguestfs.org/libnbd". But if you try to use it,
for example
in the (improved) example from libnbd-golang.pod:

$ cat test.go
package main

import "fmt"
import "libguestfs.org/libnbd"

func main() {
    h, err := libnbd.Create()
    if err != nil {
        panic(err)
    }
    defer h.Close()
    uri := "nbd://localhost"
    err = h.ConnectUri(uri)
    if err != nil {
        panic(err)
    }
    size, err := h.GetSize()
    if err != nil {
        panic(err)
    }
    fmt.Printf("size of %s = %d\n", uri, size)
}

$ go mod init example/test
go: creating new go.mod: module example/test
go: to add module requirements and sums:
        go mod tidy

$ go mod tidy
go: finding module for package libguestfs.org/libnbd
example/test imports
        libguestfs.org/libnbd: cannot find module providing package
libguestfs.org/libnbd: unrecognized import path
"libguestfs.org/libnbd": reading
https://libguestfs.org/libnbd?go-get=1: 404 Not Found

If we use libguestfs.org, https://libguestfs.org/libnbd?go-get=1
should return the expected
metadata instead of 404.

But even if "libguestfs.org/libnbd" would work, we cannot use the
module from the source
since the source is missing the generated files (wrappers.go, binding.go, ...).

It looks like the only ways to use the module are:

- Vendor the go code from the tarball.

I did not try this since I don't want to copy libnbd into source into
my project.

- Clone and build libnbd locally, and replace libguestfs.org with the
path to your local source

$ go mod edit -replace
libguestfs.org/libnbd=../../src/libnbd/golang/src/libguestfs.org/libnbd
$ go mod tidy
go: found libguestfs.org/libnbd in libguestfs.org/libnbd
v0.0.0-00010101000000-000000000000

$ cat go.mod
module example/test

go 1.16

replace libguestfs.org/libnbd =>
../../src/libnbd/golang/src/libguestfs.org/libnbd

require libguestfs.org/libnbd v0.0.0-00010101000000-000000000000


But the version is wrong - it should be v1.10.0.
I think the issue is missing tag:
https://golang.org/ref/mod#vcs-version

    If a module is defined in a subdirectory within the repository,
that is, the module subdirectory
    portion of the module path is not empty, then each tag name must
be prefixed with the module
    subdirectory, followed by a slash. For example, the module
golang.org/x/tools/gopls is defined
    in the gopls subdirectory of the repository with root path
golang.org/x/tools. The version v0.4.0
    of that module must have the tag named gopls/v0.4.0 in that repository.

So the linbd project needs a tag like:
golang/src/libguestfs.org/libnbd/v1.10.0

Removing the "src/libguestfs.org" directories will clean things up:
golang/libnbd/v1.10.0

I hope we can solve this issue. Making the go binding easy to use for
developers is
important for making libnbd more popular in the Go community.

[1] https://gerrit.ovirt.org/c/ovirt-imageio/+/117277

Nir

[Adding Matthew]

On Mon, Oct 25, 2021 at 04:45:03PM +0300, Nir Soffer
wrote:
> I'm playing with libnbd go module, planning to use it in a new
command[1]
> 
> The biggest obstacle for me is that the module is not published in a way
that
> Go developers expect.
> 
> The module is listed in:
> https://pkg.go.dev/github.com/libguestfs/libnbd/golang
> 
> But the module actually lives in:
>
https://github.com/libguestfs/libnbd/tree/master/golang/src/libguestfs.org/libnbd
> 
> So the pkg.go.dev page is broken, .e.g no there is no documation or
license, and
> the suggested import is wrong.
> 
> The module name is "libguestfs.org/libnbd". But if you try to use
it,
> for example
> in the (improved) example from libnbd-golang.pod:
> 
> $ cat test.go
> package main
> 
> import "fmt"
> import "libguestfs.org/libnbd"
> 
> func main() {
>     h, err := libnbd.Create()
>     if err != nil {
>         panic(err)
>     }
>     defer h.Close()
>     uri := "nbd://localhost"
>     err = h.ConnectUri(uri)
>     if err != nil {
>         panic(err)
>     }
>     size, err := h.GetSize()
>     if err != nil {
>         panic(err)
>     }
>     fmt.Printf("size of %s = %d\n", uri, size)
> }
> 
> $ go mod init example/test
> go: creating new go.mod: module example/test
> go: to add module requirements and sums:
>         go mod tidy
> 
> $ go mod tidy
> go: finding module for package libguestfs.org/libnbd
> example/test imports
>         libguestfs.org/libnbd: cannot find module providing package
> libguestfs.org/libnbd: unrecognized import path
> "libguestfs.org/libnbd": reading
> https://libguestfs.org/libnbd?go-get=1: 404 Not Found
That website is entirely static so if it involves fetching stuff from
there it's probably not going to work.
> If we use libguestfs.org, https://libguestfs.org/libnbd?go-get=1
> should return the expected
> metadata instead of 404.
> 
> But even if "libguestfs.org/libnbd" would work, we cannot use the
> module from the source
> since the source is missing the generated files (wrappers.go, binding.go,
...).
I believe that Matthew hit this problem too ...
> It looks like the only ways to use the module are:
> 
> - Vendor the go code from the tarball.
> 
> I did not try this since I don't want to copy libnbd into source into
> my project.
> 
> - Clone and build libnbd locally, and replace libguestfs.org with the
> path to your local source
> 
> $ go mod edit -replace
> libguestfs.org/libnbd=../../src/libnbd/golang/src/libguestfs.org/libnbd
> $ go mod tidy
> go: found libguestfs.org/libnbd in libguestfs.org/libnbd
> v0.0.0-00010101000000-000000000000
> 
> $ cat go.mod
> module example/test
> 
> go 1.16
> 
> replace libguestfs.org/libnbd =>
> ../../src/libnbd/golang/src/libguestfs.org/libnbd
> 
> require libguestfs.org/libnbd v0.0.0-00010101000000-000000000000
> 
> 
> But the version is wrong - it should be v1.10.0.
> I think the issue is missing tag:
> https://golang.org/ref/mod#vcs-version
> 
>     If a module is defined in a subdirectory within the repository,
> that is, the module subdirectory
>     portion of the module path is not empty, then each tag name must
> be prefixed with the module
>     subdirectory, followed by a slash. For example, the module
> golang.org/x/tools/gopls is defined
>     in the gopls subdirectory of the repository with root path
> golang.org/x/tools. The version v0.4.0
>     of that module must have the tag named gopls/v0.4.0 in that repository.
> 
> So the linbd project needs a tag like:
> golang/src/libguestfs.org/libnbd/v1.10.0
I'm not sure I understand what this all means.  Is that the literal
name of the git tag (ie.
git tag golang/src/libguestfs.org/libnbd/v1.10.0 )?
> Removing the "src/libguestfs.org" directories will clean things
up:
> golang/libnbd/v1.10.0
> 
> I hope we can solve this issue. Making the go binding easy to use for
> developers is
> important for making libnbd more popular in the Go community.
> 
> [1] https://gerrit.ovirt.org/c/ovirt-imageio/+/117277
Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

On Mon, Oct 25, 2021 at 04:45:03PM +0300, Nir Soffer
wrote:
> I'm playing with libnbd go module, planning to use it in a new
command[1]
> 
> The biggest obstacle for me is that the module is not published in a way
that
> Go developers expect.
> 
> The module is listed in:
> https://pkg.go.dev/github.com/libguestfs/libnbd/golang
> 
> But the module actually lives in:
>
https://github.com/libguestfs/libnbd/tree/master/golang/src/libguestfs.org/libnbd
> 
> So the pkg.go.dev page is broken, .e.g no there is no documation or
license, and
> the suggested import is wrong.
> 
> The module name is "libguestfs.org/libnbd". But if you try to use
it,
> for example
> in the (improved) example from libnbd-golang.pod:
> 
> $ cat test.go
> package main
> 
> import "fmt"
> import "libguestfs.org/libnbd"
> 
> func main() {
>     h, err := libnbd.Create()
>     if err != nil {
>         panic(err)
>     }
>     defer h.Close()
>     uri := "nbd://localhost"
>     err = h.ConnectUri(uri)
>     if err != nil {
>         panic(err)
>     }
>     size, err := h.GetSize()
>     if err != nil {
>         panic(err)
>     }
>     fmt.Printf("size of %s = %d\n", uri, size)
> }
> 
> $ go mod init example/test
> go: creating new go.mod: module example/test
> go: to add module requirements and sums:
>         go mod tidy
> 
> $ go mod tidy
> go: finding module for package libguestfs.org/libnbd
> example/test imports
>         libguestfs.org/libnbd: cannot find module providing package
> libguestfs.org/libnbd: unrecognized import path
> "libguestfs.org/libnbd": reading
> https://libguestfs.org/libnbd?go-get=1: 404 Not Found
> 
> If we use libguestfs.org, https://libguestfs.org/libnbd?go-get=1
> should return the expected
> metadata instead of 404.
> 
> But even if "libguestfs.org/libnbd" would work, we cannot use the
> module from the source
> since the source is missing the generated files (wrappers.go, binding.go,
...).
> 
> It looks like the only ways to use the module are:
> 
> - Vendor the go code from the tarball.
> 
> I did not try this since I don't want to copy libnbd into source into
> my project.
> 
> - Clone and build libnbd locally, and replace libguestfs.org with the
> path to your local source
> 
> $ go mod edit -replace
> libguestfs.org/libnbd=../../src/libnbd/golang/src/libguestfs.org/libnbd
> $ go mod tidy
> go: found libguestfs.org/libnbd in libguestfs.org/libnbd
> v0.0.0-00010101000000-000000000000
> 
> $ cat go.mod
> module example/test
> 
> go 1.16
> 
> replace libguestfs.org/libnbd =>
> ../../src/libnbd/golang/src/libguestfs.org/libnbd
> 
> require libguestfs.org/libnbd v0.0.0-00010101000000-000000000000
> 
> 
> But the version is wrong - it should be v1.10.0.
> I think the issue is missing tag:
> https://golang.org/ref/mod#vcs-version
> 
>     If a module is defined in a subdirectory within the repository,
> that is, the module subdirectory
>     portion of the module path is not empty, then each tag name must
> be prefixed with the module
>     subdirectory, followed by a slash. For example, the module
> golang.org/x/tools/gopls is defined
>     in the gopls subdirectory of the repository with root path
> golang.org/x/tools. The version v0.4.0
>     of that module must have the tag named gopls/v0.4.0 in that repository.
> 
> So the linbd project needs a tag like:
> golang/src/libguestfs.org/libnbd/v1.10.0
> 
> Removing the "src/libguestfs.org" directories will clean things
up:
> golang/libnbd/v1.10.0
Go modules are a bit annoying in that they're effectively designed
around the use of semver for versioning. Anything from v2.xxxx
onwards means you need to put the module code in sub-directory
named after the major version number. This means all your apps
need to update their imports any time you change major version.

In libvirt's go modules we had to re-create the git repos with
new tags based on semver instead of calver, to get out of the
ever changing sub-directory trap.

libnbd is lucky that it is still on v1.xxxx, so isn't in the
sub-dir naming trap yet.

I feel like as a general rule life is simpler if the Go code is
all in a dedicated repo, completely separate from any other
non-Go code, and any auto-generated code is committed too, and
wired up such that "go gen" will trigger whatever command is
needed to re-generate.

Assuming you don't want to change libnbd build system at all
though, you could have a separate git repo where you import
the generated Go binding at time of each release. It'll still
get a little odd when people want to submit bugs/patches, as
they will need to know to submit to the main repo, not this
import-only repo.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Couple of months back I was playing with GitHub Actions so that the
libnbd-go is regenerated after each commit and the changes pushed into a
separate repository, tagging it if tag was pushed as well.

Unfortunately more pressing things came my way and I did not finish this
work.  Now I am making terribly slow progress on the CI in GitLab, but
nevertheless the aforementioned approach could still work.  AFAIU we
would just need to figure out the versioning.

If anyone wants to take a stab at it, it might help, or is my
understanding of go modules flawed in this respect?  Thanks for any
corrections.

Have a nice day
Martin

On Mon, Oct 25, 2021 at 04:45:03PM +0300, Nir Soffer
wrote:
>I'm playing with libnbd go module, planning to use it in a new
command[1]
>
>The biggest obstacle for me is that the module is not published in a way
that
>Go developers expect.
>
>The module is listed in:
>https://pkg.go.dev/github.com/libguestfs/libnbd/golang
>
>But the module actually lives in:
>https://github.com/libguestfs/libnbd/tree/master/golang/src/libguestfs.org/libnbd
>
>So the pkg.go.dev page is broken, .e.g no there is no documation or license,
and
>the suggested import is wrong.
>
>The module name is "libguestfs.org/libnbd". But if you try to use
it,
>for example
>in the (improved) example from libnbd-golang.pod:
>
>$ cat test.go
>package main
>
>import "fmt"
>import "libguestfs.org/libnbd"
>
>func main() {
>    h, err := libnbd.Create()
>    if err != nil {
>        panic(err)
>    }
>    defer h.Close()
>    uri := "nbd://localhost"
>    err = h.ConnectUri(uri)
>    if err != nil {
>        panic(err)
>    }
>    size, err := h.GetSize()
>    if err != nil {
>        panic(err)
>    }
>    fmt.Printf("size of %s = %d\n", uri, size)
>}
>
>$ go mod init example/test
>go: creating new go.mod: module example/test
>go: to add module requirements and sums:
>        go mod tidy
>
>$ go mod tidy
>go: finding module for package libguestfs.org/libnbd
>example/test imports
>        libguestfs.org/libnbd: cannot find module providing package
>libguestfs.org/libnbd: unrecognized import path
>"libguestfs.org/libnbd": reading
>https://libguestfs.org/libnbd?go-get=1: 404 Not Found
>
>If we use libguestfs.org, https://libguestfs.org/libnbd?go-get=1
>should return the expected
>metadata instead of 404.
>
>But even if "libguestfs.org/libnbd" would work, we cannot use the
>module from the source
>since the source is missing the generated files (wrappers.go, binding.go,
...).
>
>It looks like the only ways to use the module are:
>
>- Vendor the go code from the tarball.
>
>I did not try this since I don't want to copy libnbd into source into
>my project.
>
>- Clone and build libnbd locally, and replace libguestfs.org with the
>path to your local source
>
>$ go mod edit -replace
>libguestfs.org/libnbd=../../src/libnbd/golang/src/libguestfs.org/libnbd
>$ go mod tidy
>go: found libguestfs.org/libnbd in libguestfs.org/libnbd
>v0.0.0-00010101000000-000000000000
>
>$ cat go.mod
>module example/test
>
>go 1.16
>
>replace libguestfs.org/libnbd =>
>../../src/libnbd/golang/src/libguestfs.org/libnbd
>
>require libguestfs.org/libnbd v0.0.0-00010101000000-000000000000
>
>
>But the version is wrong - it should be v1.10.0.
>I think the issue is missing tag:
>https://golang.org/ref/mod#vcs-version
>
>    If a module is defined in a subdirectory within the repository,
>that is, the module subdirectory
>    portion of the module path is not empty, then each tag name must
>be prefixed with the module
>    subdirectory, followed by a slash. For example, the module
>golang.org/x/tools/gopls is defined
>    in the gopls subdirectory of the repository with root path
>golang.org/x/tools. The version v0.4.0
>    of that module must have the tag named gopls/v0.4.0 in that repository.
>
>So the linbd project needs a tag like:
>golang/src/libguestfs.org/libnbd/v1.10.0
>
>Removing the "src/libguestfs.org" directories will clean things
up:
>golang/libnbd/v1.10.0
>
>I hope we can solve this issue. Making the go binding easy to use for
>developers is
>important for making libnbd more popular in the Go community.
>
>[1] https://gerrit.ovirt.org/c/ovirt-imageio/+/117277
>
>Nir
>
>_______________________________________________
>Libguestfs mailing list
>Libguestfs at redhat.com
>https://listman.redhat.com/mailman/listinfo/libguestfs
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL:
<http://listman.redhat.com/archives/libguestfs/attachments/20211025/dbb8a10e/attachment.sig>

On Mon, Oct 25, 2021 at 4:45 PM Nir Soffer <nsoffer at redhat.com>
wrote:
>
> I'm playing with libnbd go module, planning to use it in a new
command[1]
>
> The biggest obstacle for me is that the module is not published in a way
that
> Go developers expect.
>
> The module is listed in:
> https://pkg.go.dev/github.com/libguestfs/libnbd/golang
>
> But the module actually lives in:
>
https://github.com/libguestfs/libnbd/tree/master/golang/src/libguestfs.org/libnbd
>
> So the pkg.go.dev page is broken, .e.g no there is no documation or
license, and
> the suggested import is wrong.
>
> The module name is "libguestfs.org/libnbd". But if you try to use
it,
> for example
> in the (improved) example from libnbd-golang.pod:
>
> $ cat test.go
> package main
>
> import "fmt"
> import "libguestfs.org/libnbd"
>
> func main() {
>     h, err := libnbd.Create()
>     if err != nil {
>         panic(err)
>     }
>     defer h.Close()
>     uri := "nbd://localhost"
>     err = h.ConnectUri(uri)
>     if err != nil {
>         panic(err)
>     }
>     size, err := h.GetSize()
>     if err != nil {
>         panic(err)
>     }
>     fmt.Printf("size of %s = %d\n", uri, size)
> }
>
> $ go mod init example/test
> go: creating new go.mod: module example/test
> go: to add module requirements and sums:
>         go mod tidy
>
> $ go mod tidy
> go: finding module for package libguestfs.org/libnbd
> example/test imports
>         libguestfs.org/libnbd: cannot find module providing package
> libguestfs.org/libnbd: unrecognized import path
> "libguestfs.org/libnbd": reading
> https://libguestfs.org/libnbd?go-get=1: 404 Not Found
>
> If we use libguestfs.org, https://libguestfs.org/libnbd?go-get=1
> should return the expected
> metadata instead of 404.
>
> But even if "libguestfs.org/libnbd" would work, we cannot use the
> module from the source
> since the source is missing the generated files (wrappers.go, binding.go,
...).
>
> It looks like the only ways to use the module are:
>
> - Vendor the go code from the tarball.
>
> I did not try this since I don't want to copy libnbd into source into
> my project.
>
> - Clone and build libnbd locally, and replace libguestfs.org with the
> path to your local source
>
> $ go mod edit -replace
> libguestfs.org/libnbd=../../src/libnbd/golang/src/libguestfs.org/libnbd
> $ go mod tidy
> go: found libguestfs.org/libnbd in libguestfs.org/libnbd
> v0.0.0-00010101000000-000000000000
>
> $ cat go.mod
> module example/test
>
> go 1.16
>
> replace libguestfs.org/libnbd =>
> ../../src/libnbd/golang/src/libguestfs.org/libnbd
>
> require libguestfs.org/libnbd v0.0.0-00010101000000-000000000000
>
>
> But the version is wrong - it should be v1.10.0.
> I think the issue is missing tag:
> https://golang.org/ref/mod#vcs-version
>
>     If a module is defined in a subdirectory within the repository,
> that is, the module subdirectory
>     portion of the module path is not empty, then each tag name must
> be prefixed with the module
>     subdirectory, followed by a slash. For example, the module
> golang.org/x/tools/gopls is defined
>     in the gopls subdirectory of the repository with root path
> golang.org/x/tools. The version v0.4.0
>     of that module must have the tag named gopls/v0.4.0 in that repository.
>
> So the linbd project needs a tag like:
> golang/src/libguestfs.org/libnbd/v1.10.0
>
> Removing the "src/libguestfs.org" directories will clean things
up:
> golang/libnbd/v1.10.0
>
> I hope we can solve this issue. Making the go binding easy to use for
> developers is
> important for making libnbd more popular in the Go community.
Another issue, what is the license of the libnbd Go module?

The project license is LGPL 2.1, but how can tools find the
license when we point them to a subdirectory?, and the license
is not using a standard file name?

Some modules have a GPL header, and some LGPL:

$ git grep -n 'GNU General Public' golang | wc -l
66
$ git grep -n 'GNU Lesser General Public' golang | wc -l
6

I guess using LGPL is the right license?

I think the directory with the module should have a LICENSE file to
make this clear, and libnbd-golang.pod should have a LICENSE section.

Nir