Danny Milosavljevic
2018-Nov-26 11:36 UTC
Reproducible build of xapian-bindings for python - patch included
Hi, we're trying to make all builds reproducible, aiming to provide a verifiable path from source code to binary. In the course of that we found that the output produced by the python xapian-bindings is not reproducible. The reason is that in python3/Makefile.am there's a "cp" statement without "-p" - which means it will not preserve timestamps but rather generate a new timestamp every time - making every build output different. (We do have a general snippet to reset timestamps, but it comes too late - by that time, the pyc file already contains the ever-changing source timestamp and thus is also ever-changing) So we patch xapian-bindings to include the "-p" option on our side. Would you be willing to apply the patch below? --- xapian-bindings-1.4.9/python/Makefile.am.orig 2018-11-26 12:31:21.484852123 +0100 +++ xapian-bindings-1.4.9/python/Makefile.am 2018-11-26 12:31:46.252580636 +0100 @@ -73,7 +73,7 @@ _xapian_la_LIBADD = $(XAPIAN_LIBS) $(PYT # as a package. xapian/__init__.py: xapian.py $(MKDIR_P) xapian - cp `test -f xapian.py || echo '$(srcdir)/'`xapian.py xapian/__init__.py + cp -p `test -f xapian.py || echo '$(srcdir)/'`xapian.py xapian/__init__.py # We "import _xapian" first so that if we fail to import the glue library # we don't generate a broken .pyc or .pyo, and we do it with the current --- xapian-bindings-1.4.9/python3/Makefile.am.orig 2018-11-26 12:31:26.068801861 +0100 +++ xapian-bindings-1.4.9/python3/Makefile.am 2018-11-26 12:31:39.416655546 +0100 @@ -72,7 +72,7 @@ _xapian_la_LIBADD = $(XAPIAN_LIBS) $(PYT # as a package. xapian/__init__.py: xapian.py $(MKDIR_P) xapian - cp `test -f xapian.py || echo '$(srcdir)/'`xapian.py xapian/__init__.py + cp -p `test -f xapian.py || echo '$(srcdir)/'`xapian.py xapian/__init__.py # We "import _xapian" first so that if we fail to import the glue library # we don't generate a broken .pyc or .pyo, and we do it with the current -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 488 bytes Desc: OpenPGP digital signature URL: <http://lists.xapian.org/pipermail/xapian-devel/attachments/20181126/20d5d990/attachment.sig>
Olly Betts
2018-Dec-03 01:38 UTC
Reproducible build of xapian-bindings for python - patch included
On Mon, Nov 26, 2018 at 12:36:32PM +0100, Danny Milosavljevic wrote:> The reason is that in python3/Makefile.am there's a "cp" statement > without "-p" - which means it will not preserve timestamps but rather > generate a new timestamp every time - making every build output > different. > > (We do have a general snippet to reset timestamps, but it comes too > late - by that time, the pyc file already contains the ever-changing > source timestamp and thus is also ever-changing) > > So we patch xapian-bindings to include the "-p" option on our side. > > Would you be willing to apply the patch below?The patch looks OK (though this should be addressed in git master first). However it seems it only solves the problem of .pyc files embedding timestamps because we currently ship the SWIG-generated .py files (which we mostly do because there was a period when it was hard to find a SWIG release which worked for all the languages we use it for - SWIG now has extensive testsuites which are monitored by CI for most of its target languages so that's probably no longer a valid concern). A package which runs SWIG at build time would presumably also have arbitrary timestamps on the SWIG-generated .py files. Maybe https://www.python.org/dev/peps/pep-0552/ is better general solution? Cheers, Olly