Ferret talk - Aug 2006 - Windows build with Visual Studio 2005 - some success

Hi all,

first up, big thanks to Dave for doing the hard work of porting
Lucene. I have come to love Lucene through my Java work and was
extremely pleased to find the Ferret project for Ruby.

Now, I am tinkering with building the C extension using Visual Studio 2005.

So far, I have had some success in getting something built and working
in my Rails app (diffs attached). I am also encountering a very
strange problem - described at the bottom of this post.

So far, I have come up with the following:

1. checked out fresh ferret from svn
2. ran rake ext:
... copying stuff...
helper.c(3) : error C2061: syntax error : identifier ''inline''
... some more stuff

Aha. VS2005 does not like the "inline" keyword, that appears to only
apply to C++ code.

3. Changed the Makefile to the following magical incantation:
CFLAGS   =  -MD -Zi -O2b2x /Dinline=__inline /Ob /LTCG /DWIN32
Also added
mt.exe -outputresource:$(DLLIB);2 -manifest $(DLLIB).manifest
to the $(DLLIB): target (this is required for VS2005 to generate
compatible dlls):
$(DLLIB): $(DEFFILE) $(OBJS)
       @-$(RM) $@
       $(LDSHARED) -Fe$(@) $(OBJS) $(LIBS) $(LOCAL_LIBS) $(DLDFLAGS)
       mt.exe -outputresource:$(DLLIB);2 -manifest $(DLLIB).manifest


4. ran nmake in ext dir:
index_rw.c(192) : error C2275: ''DocField'' : illegal use of
this type
as an expression

VC also does not like statements before declarations. Changed index.rw
(lines 190-192) to:
DocField **fields = doc->df_arr, *field;
text_buf[MAX_WORD_SIZE - 1] = ''\0'';

5. nmake again. Lots of variations on the theme:
C:\Program Files\Microsoft Visual Studio
8\VC\PlatformSDK\include\windef.h(154): error C2059: syntax error :
''constant''

This is because q_parser.c redefines "WORD", which is also defined in
one of the windows include files. Changed "WORD" to "YYWORD"
in
q_parser.c on lines 59, 70, 440, and 1774.

6. nmake now complains about missing symbols:
str_hash
exists
bv_scan_next
bv_scan_next_from
tk_set
is_skip_vints
is_read_chars

Fixed by adding "extern inline" or "inline" to various
function
declarations in header files.

Compiles and links!

Running the tests, however, reveals a segfault:

C:\work\ruby\ferret2\ferret\test>ruby test_all.rb
Loaded suite test_all
Started
........F.....E................./unit/../unit/analysis/../../unit/document/../../unit/index/tc_index
_reader.rb:539: [BUG] Segmentation fault
ruby 1.8.4 (2006-04-14) [i386-mswin32]

I have managed to trace this to the following line in fs_store.c:201:
if (fclose((FILE *)os->file))

For some reason, fclose() segfaults.
It also segfaults when closing the file when it was just opened, it
segfaults when I add some test code:

FILE * f2;
f2 = fopen("c:\\test.txt", "wb");
fprintf(f2, "testing...");
printf("closing\n");
fclose(f2);
printf("closed\n");

before the original fclose() call.

I am stumped. Commenting out the two lines makes it work, but this is
obviously not a solution.

With the two lines commented out, I get 8 test failures and 10 errors,
which does not seem too bad compared with 5 failures and 17 errors
when run against the pure ruby version.

If anybody could shed some light on the fclose() problem, I''d be
extremely grateful.

Cheers,
Max
-------------- next part --------------
--- ../../ferret/ferret/ext/Makefile	2006-08-15 15:29:01.277950100 +1000
+++ ext/Makefile	2006-08-15 15:10:28.020936700 +1000
@@ -36,7 +36,7 @@
 LIBRUBYARG_SHARED = $(LIBRUBY)
 LIBRUBYARG_STATIC = $(LIBRUBY_A)
 
-CFLAGS   =  -MD -Zi -O2b2xg- -G6 
+CFLAGS   =  -MD -Zi -O2b2x /Dinline=__inline /Ob /LTCG /DWIN32
 CPPFLAGS = -I. -I$(topdir) -I$(hdrdir) -I$(srcdir)  
 CXXFLAGS = $(CFLAGS) 
 DLDFLAGS =  -link -incremental:no -debug -opt:ref -opt:icf -dll $(LIBPATH)
-def:$(DEFFILE) -implib:$(*F:.so=)-$(arch).lib -pdb:$(*F:.so=)-$(arch).pdb
@@ -169,6 +169,7 @@
 $(DLLIB): $(DEFFILE) $(OBJS)
 	@-$(RM) $@
 	$(LDSHARED) -Fe$(@) $(OBJS) $(LIBS) $(LOCAL_LIBS) $(DLDFLAGS)
+	mt.exe -outputresource:$(DLLIB);2 -manifest $(DLLIB).manifest
 
 
 

--- ../../ferret/ferret/ext/analysis.h	2006-08-15 15:25:38.081297400 +1000
+++ ext/analysis.h	2006-08-15 14:54:01.574393500 +1000
@@ -19,7 +19,7 @@
 
 Token *tk_create();
 void tk_destroy(void *p);
-Token *tk_set(Token *tk, char *text, int tlen, int start, int end, int
pos_inc);
+inline Token *tk_set(Token *tk, char *text, int tlen, int start, int end, int
pos_inc);
 Token *tk_set_no_len(Token *tk, char *text, int start, int end, int pos_inc);
 int tk_eq(Token *tk1, Token *tk2);
 int tk_cmp(Token *tk1, Token *tk2);



--- ../../ferret/ferret/ext/bitvector.h	2006-08-15 15:25:38.096914500 +1000
+++ ext/bitvector.h	2006-08-15 14:49:59.479916700 +1000
@@ -23,7 +23,7 @@
 void bv_write(BitVector *bv, Store *store, char *name);
 BitVector *bv_read(Store *store, char *name);
 void bv_scan_reset(BitVector *bv);
-int bv_scan_next(BitVector *bv);
-int bv_scan_next_from(BitVector *bv, register const int from);
+extern inline int bv_scan_next(BitVector *bv);
+extern inline int bv_scan_next_from(BitVector *bv, register const int from);
 
 #endif


--- ../../ferret/ferret/ext/hash.h	2006-08-15 15:25:38.112531600 +1000
+++ ext/hash.h	2006-08-15 14:42:51.406612100 +1000
@@ -72,7 +72,7 @@
 int h_set(HshTable *ht, const void *key, void *value);
 int h_set_safe(HshTable *ht, const void *key, void *value);
 int h_has_key(HshTable *ht, const void *key);
-unsigned int str_hash(const char *const str);
+inline unsigned int str_hash(const char *const str);
 
 void h_each(HshTable *ht,
     void (*each_kv)(void *key, void *value, void *arg),


--- ../../ferret/ferret/ext/index_rw.c	2006-08-15 15:25:38.159382900 +1000
+++ ext/index_rw.c	2006-08-15 14:37:57.369990000 +1000
@@ -187,9 +187,9 @@
   Token *token;
   FieldInfo *fi;
   char text_buf[MAX_WORD_SIZE];
+  DocField **fields = doc->df_arr, *field;
   text_buf[MAX_WORD_SIZE - 1] = ''\0'';
 
-  DocField **fields = doc->df_arr, *field;
   for (i = 0; i < dfcnt; i++) {
     field = fields[i];
     field_name = field->name;


--- ../../ferret/ferret/ext/q_parser.c	2006-08-15 15:25:40.704970200 +1000
+++ ext/q_parser.c	2006-08-15 14:40:03.153451700 +1000
@@ -56,7 +56,7 @@
    /* Put the tokens into the symbol table, so that GDB and other debuggers
       know about them.  */
    enum yytokentype {
-     WORD = 258,
+     YYWORD = 258,
      WILD_STR = 259,
      LOW = 260,
      OR = 261,
@@ -67,7 +67,7 @@
    };
 #endif
 /* Tokens.  */
-#define WORD 258
+#define YYWORD 258
 #define WILD_STR 259
 #define LOW 260
 #define OR 261
@@ -437,7 +437,7 @@
    First, the terminals, then, starting at YYNTOKENS, nonterminals. */
 static const char *const yytname[]  {
-  "$end", "error", "$undefined",
"WORD", "WILD_STR", "LOW", "OR",
"AND",
+  "$end", "error", "$undefined",
"YYWORD", "WILD_STR", "LOW", "OR",
"AND",
   "NOT", "REQ", "'':''",
"HIGH", "''^''",
"''(''", "'')''",
"''~''", "''*''",
"''|''",
   "''\"''",
"''<''", "''>''",
"''[''", "'']''",
"''}''", "''{''",
"''=''", "$accept",
   "bool_q", "bool_clss", "bool_cls",
"boosted_q", "q", "term_q", "wild_q",
@@ -1771,7 +1771,7 @@
   /* found a word so return it. */
   lvalp->str = buf;
   if (is_wild) return WILD_STR;
-  return WORD;
+  return YYWORD;
 }
 
 int yylex(YYSTYPE *lvalp, QParser *qp)


--- ../../ferret/ferret/ext/store.h	2006-08-15 15:25:38.143765800 +1000
+++ ext/store.h	2006-08-15 14:54:55.157804300 +1000
@@ -121,8 +121,8 @@
 unsigned int is_read_uint(InStream *is);
 ullong is_read_ulong(InStream *is);
 ullong is_read_vint(InStream *is);
-void is_skip_vints(InStream *is, register int cnt);
-void is_read_chars(InStream *is, char* buffer, int off, int len) ;
+extern inline void is_skip_vints(InStream *is, register int cnt);
+extern inline void is_read_chars(InStream *is, char* buffer, int off, int len)
;
 char *is_read_string(InStream *is);
 void os_write_int(OutStream *os, int l);
 void os_write_long(OutStream *os, llong l);

Replying to myself again... ah well.

It seems that Ruby''s win32.h redefines fclose() to rb_w32_fclose().
This leads to all kinds of unforeseen trouble, as the calling
conventions between VC6 and VC2005 have changed.

I managed to get Ferret compiling and running nicely by adding

#undef fclose
#include <stdio.h>

to fs_store.c underneath the rest of the include declarations. That
fixes most of the test errors, although there are still some failures
I suspect are to do with handling of slightly exotic characters.

If somebody wanst to volunteer for some windows testing, let me know.

Max

Hi Max,

I''ve been working on a new version of Ferret in a different repository;

    svn co svn://www.davebalmain.com/exp

I got the C code compiling on VC6. I''ve still got to do some work on
the bindings but it should run a lot better on Windows. I''ll be
releasing a windows gem in the next week or so.

Cheers,
Dave

On 8/15/06, Max Muermann <ruby at muermann.org>
wrote:
> Replying to myself again... ah well.
>
> It seems that Ruby''s win32.h redefines fclose() to
rb_w32_fclose().
> This leads to all kinds of unforeseen trouble, as the calling
> conventions between VC6 and VC2005 have changed.
>
> I managed to get Ferret compiling and running nicely by adding
>
> #undef fclose
> #include <stdio.h>
>
> to fs_store.c underneath the rest of the include declarations. That
> fixes most of the test errors, although there are still some failures
> I suspect are to do with handling of slightly exotic characters.
>
> If somebody wanst to volunteer for some windows testing, let me know.
>
> Max
> _______________________________________________
> Ferret-talk mailing list
> Ferret-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/ferret-talk
>