Xapian devel - Oct 2015 - Trying to get Search::Xapian perl module to compile on win32

The disclaimer automatically added and is not intended to limit legitimate
discourse, and any code I write for this is expected to be licensed under
whatever license the original work is using.

I haven't tried compiling latest code under visual studio yet (is this
possible?).  I wonder if there is a way to tell gcc to not mangle the names
during cross compile.  I was able to successfully cross compile libxml and a few
others without impacting the perl module compile but that is probably due to
them being just a C project.

--
Dez.


-----Original Message-----	
From: Olly Betts [mailto:olly at survex.com] 
Sent: Wednesday, October 28, 2015 5:35 PM
To: Desmond Daignault <desmond.daignault at oneil.com>
Cc: Xapian Development <xapian-devel at lists.xapian.org>
Subject: Re: [Xapian-devel] Trying to get Search::Xapian perl module to compile
on win32

On Wed, Oct 28, 2015 at 08:46:15PM +0000, Desmond Daignault
wrote:
> Come to find out, on windows, perl does indeed define open to be win32_open
and close to be win32_close.  This is done in perl/lib/CORE/win3iop.h.  So, to
get past this problem, I had to modify Xapian.xs to do an #undef of open and
close if WIN32 is defined.
> 
> Also, I had to modify the Makefile.PL to not use xapian_config to get the
values it needs.
> 
> Attached is a work in progress patch for both files.  (hope the attachments
come through)  I am now getting linking errors.  I am working with the latest
stable version of xapian-core, which I cross-compiled on Ubuntu for win32.  I
had to do a couple of manual steps to convert the libxapian-22.dll into a
libxapian-22.lib using info from https://support.microsoft.com/en-us/kb/131313. 
Specifically using dumpbin /exports to create a definition file and then feeding
the definition file to lib /def to create the .lib and .exp files.
> 
> The call to generate the Makefile on win32 is now something like: 
>     perl Makefile.PL CXX=cl.exe INC=-IC:\Install\include "
LIBS=-lxapian-22 -lzlib" XAPIAN_VERSION=1.2.21 "CXXFLAGS=-EHsc
-TP"
Presumably you're cross-compiling with GCC?
> Now when I run nmake, it gets past the compile portion and dies when
linking with errors like:
> Xapian.obj : error LNK2001: unresolved external symbol "public:
__thiscall Xapian::WritableDatabase:
> :WritableDatabase(class Xapian::WritableDatabase const &)"
(??0WritableDatabase at Xapian@@QAE at ABV01@@Z)
> 
> Any clues on how to resolve the linking errors?
I'd be surprised if you can mix C++ code compiled with GCC with C++ code
compiled with MSVC - you probably need to use either GCC or MSVC to
build both xapian-core and Search::Xapian.  C code should interwork
fine, so it shouldn't matter what perl was built with.

Before I can apply the patches, I need to check you're OK with the
licensing requirements:

http://trac.xapian.org/browser/trunk/xapian-core/HACKING#L1364

Actually for Search::Xapian it ought to be a "same licence as Perl" in
place of "GPL version 2 and all later versions".  (The dual licence
with
MIT/X is still useful there as it allows us to apply patches to the
SWIG-based Perl bindings on trunk).
> **********************************************************************
> Confidentiality Notice
> The information contained in this e-mail is confidential and intended for
use only by the person(s) or organization listed in the address. If you have
received this communication in error, please contact the sender at O'Neil
& Associates, Inc., immediately. Any copying, dissemination, or distribution
of this communication, other than by the intended recipient, is strictly
prohibited.
> **********************************************************************
Also, please explicitly confirm that this notice isn't intended to apply
to the patches.  If we apply them they'll definitely be used by people
you didn't address this email to...

Cheers,
    Olly

Actually, looks like I need to patch the configure script when cross compiling
to have gcc produce a .def file which I can then use to produce a .lib file for
linking against.  I'll send that patch along as soon as I get it tested out.

See: http://www.mingw.org/wiki/MSVC_and_MinGW_DLLs

--
Dez.


-----Original Message-----
From: Xapian-devel [mailto:xapian-devel-bounces at lists.xapian.org] On Behalf
Of Desmond Daignault
Sent: Thursday, October 29, 2015 9:51 AM
To: Xapian Development <xapian-devel at lists.xapian.org>
Subject: RE: [Xapian-devel] Trying to get Search::Xapian perl module to compile
on win32

The disclaimer automatically added and is not intended to limit legitimate
discourse, and any code I write for this is expected to be licensed under
whatever license the original work is using.

I haven't tried compiling latest code under visual studio yet (is this
possible?).  I wonder if there is a way to tell gcc to not mangle the names
during cross compile.  I was able to successfully cross compile libxml and a few
others without impacting the perl module compile but that is probably due to
them being just a C project.

--
Dez.


-----Original Message-----	
From: Olly Betts [mailto:olly at survex.com] 
Sent: Wednesday, October 28, 2015 5:35 PM
To: Desmond Daignault <desmond.daignault at oneil.com>
Cc: Xapian Development <xapian-devel at lists.xapian.org>
Subject: Re: [Xapian-devel] Trying to get Search::Xapian perl module to compile
on win32

On Wed, Oct 28, 2015 at 08:46:15PM +0000, Desmond Daignault
wrote:
> Come to find out, on windows, perl does indeed define open to be win32_open
and close to be win32_close.  This is done in perl/lib/CORE/win3iop.h.  So, to
get past this problem, I had to modify Xapian.xs to do an #undef of open and
close if WIN32 is defined.
> 
> Also, I had to modify the Makefile.PL to not use xapian_config to get the
values it needs.
> 
> Attached is a work in progress patch for both files.  (hope the attachments
come through)  I am now getting linking errors.  I am working with the latest
stable version of xapian-core, which I cross-compiled on Ubuntu for win32.  I
had to do a couple of manual steps to convert the libxapian-22.dll into a
libxapian-22.lib using info from https://support.microsoft.com/en-us/kb/131313. 
Specifically using dumpbin /exports to create a definition file and then feeding
the definition file to lib /def to create the .lib and .exp files.
> 
> The call to generate the Makefile on win32 is now something like: 
>     perl Makefile.PL CXX=cl.exe INC=-IC:\Install\include "
LIBS=-lxapian-22 -lzlib" XAPIAN_VERSION=1.2.21 "CXXFLAGS=-EHsc
-TP"
Presumably you're cross-compiling with GCC?
> Now when I run nmake, it gets past the compile portion and dies when
linking with errors like:
> Xapian.obj : error LNK2001: unresolved external symbol "public:
__thiscall Xapian::WritableDatabase:
> :WritableDatabase(class Xapian::WritableDatabase const &)"
(??0WritableDatabase at Xapian@@QAE at ABV01@@Z)
> 
> Any clues on how to resolve the linking errors?
I'd be surprised if you can mix C++ code compiled with GCC with C++ code
compiled with MSVC - you probably need to use either GCC or MSVC to
build both xapian-core and Search::Xapian.  C code should interwork
fine, so it shouldn't matter what perl was built with.

Before I can apply the patches, I need to check you're OK with the
licensing requirements:

http://trac.xapian.org/browser/trunk/xapian-core/HACKING#L1364

Actually for Search::Xapian it ought to be a "same licence as Perl" in
place of "GPL version 2 and all later versions".  (The dual licence
with
MIT/X is still useful there as it allows us to apply patches to the
SWIG-based Perl bindings on trunk).
> **********************************************************************
> Confidentiality Notice
> The information contained in this e-mail is confidential and intended for
use only by the person(s) or organization listed in the address. If you have
received this communication in error, please contact the sender at O'Neil
& Associates, Inc., immediately. Any copying, dissemination, or distribution
of this communication, other than by the intended recipient, is strictly
prohibited.
> **********************************************************************
Also, please explicitly confirm that this notice isn't intended to apply
to the patches.  If we apply them they'll definitely be used by people
you didn't address this email to...

Cheers,
    Olly

On Thu, Oct 29, 2015 at 01:51:07PM +0000, Desmond Daignault
wrote:
> any code I write for this is expected to be licensed under whatever
> license the original work is using.
That isn't what the patch licensing policy asks for though.  Please read
it carefully, as it explains both what we're asking for and why we're
asking for it.
> I haven't tried compiling latest code under visual studio yet (is this
> possible?).
It should be, though it's not regularly tested lately, so may need some
changes.
> I wonder if there is a way to tell gcc to not mangle the names during
> cross compile.  I was able to successfully cross compile libxml and a
> few others without impacting the perl module compile but that is
> probably due to them being just a C project.
It's not the name mangling that's the big issue - GCC and MSVC have
their own versions of the C++ standard library - I very much doubt
these would be ABI compatible.

For C code this isn't an issue, as mingw uses MSVC's CRT.

Cheers,
    Olly

On Thu, Oct 29, 2015 at 02:21:52PM +0000, Desmond Daignault
wrote:
> Actually, looks like I need to patch the configure script when cross
> compiling to have gcc produce a .def file which I can then use to
> produce a .lib file for linking against.  I'll send that patch along
> as soon as I get it tested out.
> 
> See: http://www.mingw.org/wiki/MSVC_and_MinGW_DLLs
GCC on mingw should be able to link against a DLL without needing an
import library, so I don't think you do need to patch configure as you
describe.

As I've said, building Xapian with GCC and then trying to use that from
MSVC is not likely to work (it would for a C library, which is what that
URL you point to is discussing - note that the source file in their
example is 'testdll.c').

Cheers,
    Olly

Olly Betts writes:
 > On Thu, Oct 29, 2015 at 01:51:07PM +0000, Desmond Daignault wrote:
 > > any code I write for this is expected to be licensed under whatever
 > > license the original work is using.
 > 
 > That isn't what the patch licensing policy asks for though.  Please
read
 > it carefully, as it explains both what we're asking for and why
we're
 > asking for it.
 > 
 > > I haven't tried compiling latest code under visual studio yet (is
this
 > > possible?).
 > 
 > It should be, though it's not regularly tested lately, so may need
some
 > changes.

If it is of any interest, I have compiled xapian-core-1.2.21 with Visual
Studio 2015 recently. I think I started from the files available for 1.2.8.
The project files are at the following URL (look at the README).

https://bitbucket.org/medoc/recoll-windows-deps/src/6debe5f31ec6369c204f3c3cbf41e47cbfcdc0fc/xapian/?at=default

The Xapian source needed no changes (there were small issues in 1.2.8, but
they were all fixed in 1.2.21).

Cheers,

jf

 > > I wonder if there is a way to tell gcc to not mangle the names during
 > > cross compile.  I was able to successfully cross compile libxml and a
 > > few others without impacting the perl module compile but that is
 > > probably due to them being just a C project.
 > 
 > It's not the name mangling that's the big issue - GCC and MSVC
have
 > their own versions of the C++ standard library - I very much doubt
 > these would be ABI compatible.
 > 
 > For C code this isn't an issue, as mingw uses MSVC's CRT.
 > 
 > Cheers,
 >     Olly
 > 
 >