Libguestfs - Oct 2022 - distutils use in hivex libdnet libnbd (and more) [was: Re: Help needed triaging build failures without distutils]

On Tue, Oct 18, 2022 at 11:27:47AM +0200, Miro Hron?ok
wrote:
> Hey Pythonistas.
> 
> The Python standard library distutils module will be removed from Python
3.12+
> 
> https://peps.python.org/pep-0632/
Annoying, but OK ...
> As preparatory work, we build all python packages in a Copr repository
> with Python 3.11 sans distutils:
> 
> https://copr.fedorainfracloud.org/coprs/g/python/python-without-distutils/
> 
> I've rebuilt all the failed builds again and also in a control-group
copr:
> 
> https://copr.fedorainfracloud.org/coprs/g/python/python-with-distutils/
>
> 250 packages that failed to build without distuils but succeeded
> with distutils need to be examined and categorized into various
> different groups:
> 
>  - package uses distutils only if sys.version_info < (3, 12)
>      -- this is OK but still fails here
>  - package uses distutils unconditionally and the package needs to be fixed
>  - package uses another package that uses distutils unconditionally
>       and the dependency needs to be fixed
> 
> I suspect most of the packages will fail to build with Python 3.12
> (planned for Fedora 39, change proposal TBD). The python3-setutpools
> package provides a distutils module [^1], so sometimes "simply"
> adding BuildRequires: python3-setuptools might workaround the
> problem.
...
> rjones     hivex libdnet libnbd
I think your testing methodology might have been wrong because we use
distutils in other packages that I maintain, notably:

  https://github.com/libguestfs/libguestfs/blob/master/m4/guestfs-python.m4

I'm not sure what could have happened here.  The RPM would have failed
to build if the Python bindings had been ./configure-d out.  The link
shows libguestfs as "not build yet / Disabled":

 
https://copr.fedorainfracloud.org/coprs/g/python/python-without-distutils/packages/?page=13

but then there's a build which succeeded:

 
https://copr.fedorainfracloud.org/coprs/g/python/python-without-distutils/build/4906970/

and it has Python bindings.  (The logs of the build don't seem to be
available.)

Anyway I'll see if I can fix this upstream as requested.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

On Tue, Oct 18, 2022 at 10:53:12AM +0100, Richard W.M. Jones
wrote:
>   https://github.com/libguestfs/libguestfs/blob/master/m4/guestfs-python.m4
Actually some help for how to replace it would be useful too as
sysconfig is not an exact replacement for distutils.sysconfig.  In
particular get_python_lib doesn't exist, and I can't find an exact
replacement.

Original code:

  $ python3 -c 'import distutils.sysconfig;
print(distutils.sysconfig.get_python_lib(1,0));'
  /usr/lib64/python3.11/site-packages

Potential replacement:

  $ python3 -c 'import sysconfig;
print(sysconfig.get_path("platlib"));'
  /usr/local/lib64/python3.11/site-packages

Maybe the original code was wrong, but I guess we don't want to
install site packages in /usr/local when running under RPM at least.
Note it's used here:

https://github.com/libguestfs/libguestfs/blob/master/python/Makefile.am#L50-L54

(We do not -- and cannot -- use ordinary Python install mechanisms.)

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
nbdkit - Flexible, fast NBD server with plugins
https://gitlab.com/nbdkit/nbdkit

[Sending again since it didn't reach the Python-devel list first time]

On Tue, Oct 18, 2022 at 10:53:12AM +0100, Richard W.M. Jones
wrote:
> On Tue, Oct 18, 2022 at 11:27:47AM +0200, Miro Hron?ok wrote:
> > Hey Pythonistas.
> > 
> > The Python standard library distutils module will be removed from
Python 3.12+
> > 
> > https://peps.python.org/pep-0632/
> 
> Annoying, but OK ...
> 
> > As preparatory work, we build all python packages in a Copr repository
> > with Python 3.11 sans distutils:
> > 
> >
https://copr.fedorainfracloud.org/coprs/g/python/python-without-distutils/
> > 
> > I've rebuilt all the failed builds again and also in a
control-group copr:
> > 
> >
https://copr.fedorainfracloud.org/coprs/g/python/python-with-distutils/
> >
> > 250 packages that failed to build without distuils but succeeded
> > with distutils need to be examined and categorized into various
> > different groups:
> > 
> >  - package uses distutils only if sys.version_info < (3, 12)
> >      -- this is OK but still fails here
> >  - package uses distutils unconditionally and the package needs to be
fixed
> >  - package uses another package that uses distutils unconditionally
> >       and the dependency needs to be fixed
> > 
> > I suspect most of the packages will fail to build with Python 3.12
> > (planned for Fedora 39, change proposal TBD). The python3-setutpools
> > package provides a distutils module [^1], so sometimes
"simply"
> > adding BuildRequires: python3-setuptools might workaround the
> > problem.
> 
> ...
> 
> > rjones     hivex libdnet libnbd
> 
> I think your testing methodology might have been wrong because we use
> distutils in other packages that I maintain, notably:
> 
>   https://github.com/libguestfs/libguestfs/blob/master/m4/guestfs-python.m4
> 
> I'm not sure what could have happened here.  The RPM would have failed
> to build if the Python bindings had been ./configure-d out.  The link
> shows libguestfs as "not build yet / Disabled":
> 
>  
https://copr.fedorainfracloud.org/coprs/g/python/python-without-distutils/packages/?page=13
> 
> but then there's a build which succeeded:
> 
>  
https://copr.fedorainfracloud.org/coprs/g/python/python-without-distutils/build/4906970/
> 
> and it has Python bindings.  (The logs of the build don't seem to be
> available.)
> 
> Anyway I'll see if I can fix this upstream as requested.
> 
> Rich.
> 
> -- 
> Richard Jones, Virtualization Group, Red Hat
http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-top is 'top' for virtual machines.  Tiny program with many
> powerful monitoring features, net stats, disk stats, logging, etc.
> http://people.redhat.com/~rjones/virt-top
-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top