Libguestfs - May 2015 - [nbdkit python]: unable to locate modules

I'm working on a python plugin for nbdkit that treats Google Cloud
Storage as a block device. I have a library, gcs.py, that implements
simple commands: list, get, put, delete; source is here:
https://github.com/wspeirs/cldblkdev/blob/master/gcs.py

When I attempt to run nbdkit I get the following error:

nbdkit python script=/home/wspeirs/src/cldblkdev/cldblkdev.py
Traceback (most recent call last):
 File "/home/wspeirs/src/cldblkdev/cldblkdev.py", line 5, in
<module>
   import gcs
 File "/home/wspeirs/src/cldblkdev/gcs.py", line 2, in <module>
   import httplib2
 File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py",
line 930, in <module>
   class HTTPSConnectionWithTimeout(httplib.HTTPSConnection):
AttributeError: 'module' object has no attribute
'HTTPSConnection'
Error in sys.excepthook:
Traceback (most recent call last):
 File "/usr/lib/python2.7/dist-packages/apport_python_hook.py", line
63, in apport_excepthook
   from apport.fileutils import likely_packaged, get_recent_crashes
 File "/usr/lib/python2.7/dist-packages/apport/__init__.py", line 5,
in <module>
   from apport.report import Report
 File "/usr/lib/python2.7/dist-packages/apport/report.py", line 16, in
<module>
   from xml.parsers.expat import ExpatError
 File "/usr/lib/python2.7/xml/parsers/expat.py", line 4, in
<module>
   from pyexpat import *
ImportError: /usr/lib/python2.7/lib-dynload/pyexpat.x86_64-linux-gnu.so:
undefined symbol: _Py_ZeroStruct

Original exception was:
Traceback (most recent call last):
 File "/home/wspeirs/src/cldblkdev/cldblkdev.py", line 5, in
<module>
   import gcs
 File "/home/wspeirs/src/cldblkdev/gcs.py", line 2, in <module>
   import httplib2
 File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py",
line 930, in <module>
   class HTTPSConnectionWithTimeout(httplib.HTTPSConnection):
AttributeError: 'module' object has no attribute
'HTTPSConnection'
nbdkit: error: /home/wspeirs/src/cldblkdev/cldblkdev.py: error running
this script

Everything seems to work outside of nbdkit. I'm assuming this is some
path related issue... any thoughts on how to fix this?

Thanks...

Bill-

On Thu, May 21, 2015 at 12:10:13AM -0400, Bill Speirs
wrote:
> I'm working on a python plugin for nbdkit that treats Google Cloud
> Storage as a block device. I have a library, gcs.py, that implements
> simple commands: list, get, put, delete; source is here:
> https://github.com/wspeirs/cldblkdev/blob/master/gcs.py
> 
> When I attempt to run nbdkit I get the following error:
I get a slightly different error:

Traceback (most recent call last):
  File "/tmp/gcs.py", line 1, in <module>
    import json
  File "/usr/lib64/python2.7/json/__init__.py", line 108, in
<module>
    from .decoder import JSONDecoder
  File "/usr/lib64/python2.7/json/decoder.py", line 5, in
<module>
    import struct
  File "/usr/lib64/python2.7/struct.py", line 1, in <module>
    from _struct import *
ImportError: /usr/lib64/python2.7/lib-dynload/_struct.so: undefined symbol:
PyFloat_Type
nbdkit: error: /tmp/gcs.py: error running this script

A smallest possible reproducer is to have a file that has a single line:

  import json

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

On Thu, May 21, 2015 at 09:32:48AM +0100, Richard W.M. Jones
wrote:
> On Thu, May 21, 2015 at 12:10:13AM -0400, Bill Speirs wrote:
> > I'm working on a python plugin for nbdkit that treats Google Cloud
> > Storage as a block device. I have a library, gcs.py, that implements
> > simple commands: list, get, put, delete; source is here:
> > https://github.com/wspeirs/cldblkdev/blob/master/gcs.py
> > 
> > When I attempt to run nbdkit I get the following error:
> 
> I get a slightly different error:
> 
> Traceback (most recent call last):
>   File "/tmp/gcs.py", line 1, in <module>
>     import json
>   File "/usr/lib64/python2.7/json/__init__.py", line 108, in
<module>
>     from .decoder import JSONDecoder
>   File "/usr/lib64/python2.7/json/decoder.py", line 5, in
<module>
>     import struct
>   File "/usr/lib64/python2.7/struct.py", line 1, in
<module>
>     from _struct import *
> ImportError: /usr/lib64/python2.7/lib-dynload/_struct.so: undefined symbol:
PyFloat_Type
> nbdkit: error: /tmp/gcs.py: error running this script
> 
> A smallest possible reproducer is to have a file that has a single line:
> 
>   import json
I can fix it (for my simple json case) by doing:

LD_PRELOAD=/usr/lib64/python2.7/lib-dynload/_json.so nbdkit python
script=/tmp/json.py

I wonder why Python doesn't dlopen the correct shared libraries?

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org