Hi, I'm on a Ubuntu 18.04 server, trying to use django-haystack with xapian in a python3.6 virtualenv. The virtualenv is set up not to use system packages, meaning that I can't just install python3-xapian-haystack with apt, and instead have to manually build xapian-core and xapian-bindings within the virtualenv. This works with xapian-core and the --prefix argument to configure. When I try the same with xapian-bindings, using: ./configure --prefix=$venv --with-python3 The configure process looks fine: xapian-config is found correctly, as is my virtualenv python installation. Then I export LD_LIBRARY_PATH as $venv/lib. Then "make" fails with: make all-recursive make[1]: Entering directory '/home/<me>/.local/share/virtualenvs/<venv>/src/xapian-bindings-1.4.7' Making all in . make[2]: Entering directory '/home/<me>/.local/share/virtualenvs/<venv>/src/xapian-bindings-1.4.7' make[2]: Leaving directory '/home/<me>/.local/share/virtualenvs/<venv>/src/xapian-bindings-1.4.7' Making all in python3 make[2]: Entering directory '/home/<me>/.local/share/virtualenvs/<venv>/src/xapian-bindings-1.4.7/python3' make all-am make[3]: Entering directory '/home/<me>/.local/share/virtualenvs/<venv>/src/xapian-bindings-1.4.7/python3' depbase=`echo xapian_wrap.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;\ /bin/bash ../libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H -I. -I.. -I/usr/include/python3.6m -fno-strict-aliasing -Wall -Wno-unused -Wno-uninitialized -fvisibility=hidden -I/home/<me>/.local/share/virtualenvs/<venv>/include -g -O2 -MT xapian_wrap.lo -MD -MP -MF $depbase.Tpo -c -o xapian_wrap.lo xapian_wrap.cc &&\ mv -f $depbase.Tpo $depbase.Plo libtool: compile: g++ -DHAVE_CONFIG_H -I. -I.. -I/usr/include/python3.6m -fno-strict-aliasing -Wall -Wno-unused -Wno-uninitialized -fvisibility=hidden -I/home/<me>/.local/share/virtualenvs/<venv>/include -g -O2 -MT xapian_wrap.lo -MD -MP -MF .deps/xapian_wrap.Tpo -c xapian_wrap.cc -fPIC -DPIC -o .libs/xapian_wrap.o g++: internal compiler error: Killed (program cc1plus) Please submit a full bug report, with preprocessed source if appropriate. See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions. Makefile:787: recipe for target 'xapian_wrap.lo' failed make[3]: *** [xapian_wrap.lo] Error 1 make[3]: Leaving directory '/home/<me>/.local/share/virtualenvs/<venv>/src/xapian-bindings-1.4.7/python3' Makefile:687: recipe for target 'all' failed make[2]: *** [all] Error 2 make[2]: Leaving directory '/home/<me>/.local/share/virtualenvs/<venv>/src/xapian-bindings-1.4.7/python3' Makefile:521: recipe for target 'all-recursive' failed make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory '/home/<me>/.local/share/virtualenvs/<venv>/src/xapian-bindings-1.4.7' Makefile:443: recipe for target 'all' failed make: *** [all] Error 2 Can anyone tell me how to get this package configured and built correctly? Thanks! Eric
On Tue, Oct 02, 2018 at 04:13:14PM -0700, Eric Abrahamsen wrote:> ./configure --prefix=$venv --with-python3 > > The configure process looks fine: xapian-config is found correctly, as > is my virtualenv python installation. Then I export LD_LIBRARY_PATH as > $venv/lib.You shouldn't need to set LD_LIBRARY_PATH - libtool should set rpath when linking against a library outside the standard search path. In general it's probably harmless to set it though, and it shouldn't result in the compiler getting killed.> Then "make" fails with:[...]> libtool: compile: g++ -DHAVE_CONFIG_H -I. -I.. -I/usr/include/python3.6m -fno-strict-aliasing -Wall -Wno-unused -Wno-uninitialized -fvisibility=hidden -I/home/<me>/.local/share/virtualenvs/<venv>/include -g -O2 -MT xapian_wrap.lo -MD -MP -MF .deps/xapian_wrap.Tpo -c xapian_wrap.cc -fPIC -DPIC -o .libs/xapian_wrap.o > g++: internal compiler error: Killed (program cc1plus)This seems to indicate the compiler received SIGKILL, which smells to me like the kernel's OOM (Out Of Memory) killer kicked in. The bindings are quite resource intensive to compile, so on a low-memory system you may need to turn down or disable optimisation to get them to build. Also check that the amount memory a process can use isn't limited by ulimit. The INSTALL file shows how to disable optimisation: https://trac.xapian.org/browser/git/xapian-bindings/INSTALL#L14 The change way back in 0.9.3 mentioned there was that we factored out the exception mapping into a helper function. Cheers, Olly
On 10/03/18 23:25 PM, Olly Betts wrote:> On Tue, Oct 02, 2018 at 04:13:14PM -0700, Eric Abrahamsen wrote: >> ./configure --prefix=$venv --with-python3 >> >> The configure process looks fine: xapian-config is found correctly, as >> is my virtualenv python installation. Then I export LD_LIBRARY_PATH as >> $venv/lib. > > You shouldn't need to set LD_LIBRARY_PATH - libtool should set rpath > when linking against a library outside the standard search path. > > In general it's probably harmless to set it though, and it shouldn't > result in the compiler getting killed.I set it anyway -- belt and suspenders :)>> Then "make" fails with: > [...] >> libtool: compile: g++ -DHAVE_CONFIG_H -I. -I.. >> -I/usr/include/python3.6m -fno-strict-aliasing -Wall -Wno-unused >> -Wno-uninitialized -fvisibility=hidden >> -I/home/<me>/.local/share/virtualenvs/<venv>/include -g -O2 -MT >> xapian_wrap.lo -MD -MP -MF .deps/xapian_wrap.Tpo -c xapian_wrap.cc >> -fPIC -DPIC -o .libs/xapian_wrap.o >> g++: internal compiler error: Killed (program cc1plus) > > This seems to indicate the compiler received SIGKILL, which smells to me > like the kernel's OOM (Out Of Memory) killer kicked in. > > The bindings are quite resource intensive to compile, so on a low-memory > system you may need to turn down or disable optimisation to get them to > build. Also check that the amount memory a process can use isn't > limited by ulimit. > > The INSTALL file shows how to disable optimisation: > > https://trac.xapian.org/browser/git/xapian-bindings/INSTALL#L14 > > The change way back in 0.9.3 mentioned there was that we factored out > the exception mapping into a helper function.Thanks for this -- the server is resource-constrained, and that was indeed the problem. I assumed that the optimization would produce smaller binaries, so rather than disabling that I instead temporarily shut off some other non-critical services, and freed up enough memory to do the build. All seems to have worked out fine! Thanks very much for the pointer. Eric