Michael Felt
2016-Jan-03 15:59 UTC
[Rd] R, AIX 64-bit builds - trying to understand root cause for message: "Error: Line starting 'Package: tools ...' is malformed!"
On 2016-01-01 23:48, peter dalgaard wrote:> Nice catch you two!!! > > Happy New Year > -pdI am much happier with this great start! Simon - which compiler)s) did you use: xlc and xlfortran, or gcc/gfortran? I have made some changes to configure(.ac) so maybe my problems are self-inflicted. But would be good to know what environment you are using. Thanks for looking - and finding!!! Michael>> On 01 Jan 2016, at 22:06 , Simon Urbanek<simon.urbanek at r-project.org> wrote: >> >> Ok, found the problem - on platforms that support it TRE uses wint_t (from wchar.h) as its type for characters (tre_cint_t) which on AIX is *signed* int. TRE uses liberally conversions between int and tre_cint_t apparently assuming that the latter is unsigned so conversions back to int are suitable for comparisons etc. On other platforms wint_t is unsigned so it works. Manually defining tre_cint_t to unsigned int fixes the issue. >> >> Cheers, >> Simon >> >> >> On Jan 1, 2016, at 12:20 PM, Simon Urbanek<simon.urbanek at r-project.org> wrote: >> >>> Michael, >>> >>> thanks, I'll have a look once my PDP VMs are up again (later today). This may be a signedness issue although it's unclear why other platforms wouldn't be affected. >>> >>> Cheers, >>> Simon >>> >>> >>> On Dec 31, 2015, at 10:14 AM, Michael Felt<aixtools at gmail.com> wrote: >>> >>>> On 2015-12-30 09:58, Michael Felt wrote: >>>>> On 2015-12-29 11:02, Michael Felt wrote: >>>>>> This seems to be a problem that goes back a long time - and I hope someone who understands what tre is suppossed to be doing will look at this. >>>>>> >>>>>> A short history of other people who have reported on this on different versions of AIX. I shall only add that I get the same results on AIX 5.3 TL7, AIX 6.1 TL9 and AIX 7.1 TL3. >>>>>> >>>>>> Basically, with settings that work for AIX and 32-bit - the only changes being >>>>>> -maix32 becomes -maix64 >>>>>> and >>>>>> export OBJECT_MODE=32 becomes export OBJECT_MODE=64 >>>>>> >>>>>> Then to shorten the 'make' bla bla, first run just make, then >>>>>> >>>>>> cd src/library/tools >>>>>> make -s sysdata >>>>>> >>>>>> http://article.gmane.org/gmane.comp.lang.r.devel/38817/match=package+tools+malformed >>>>>> http://article.gmane.org/gmane.comp.lang.r.devel/36886/match=package+tools+malformed >>>>>> http://article.gmane.org/gmane.comp.lang.r.devel/23372/match=package+tools+malformed Date: 2010-01-25 06:55:41 GMT (5 years, 48 weeks, 1 day, 20 hours and 30 minutes ago) >>>>>> >>>>>> To that, to get debug data, I have >>>>>> >>>>>> * added -DTRE_DUGUG to src/extra/tre/Makefile # ALL_CFLAGS = $(ALL_CFLAGS_LO) -DTRE_DEBUG >>>>>> * rm src/extra/tre/tre-match-parallel.o >>>>>> * find . -name \*.so -exec rm {} \; >>>>>> * make >>>>>> * cd src/library/tools >>>>>> * make -s sysdata >>>>>> >>>>>> Attached are the two script files of the screen output. The 32-bit one is more verbose - and contains magically lines such as: >>>>>> found match 3037fd14 (while "found" does not occur in the 64-bit output) >>>>>> >>>>>> root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]wc /tmp/sysdata.??.* >>>>>> 4730 14123 139916 /tmp/sysdata.32.text >>>>>> 1312 3688 40528 /tmp/sysdata.64.text >>>>>> 6042 17811 180444 total >>>>>> >>>>>> root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]grep -c found /tmp/sysdata.??.* >>>>>> /tmp/sysdata.32.text:19 >>>>>> /tmp/sysdata.64.text:0 >>>>>> >>>>>> >>>>>> Hope this brings us (or me), closer to a resolution to an old concern. >>>>>> >>>>>> And, best wishes for the new year! >>>>>> >>>>>> Michael >>>>>> >>>>>> >>>>> Still hoping for someones curiosity/willingness. >>>>> >>>>> The differences show up in the first comparision that is made (of the string "3.2.3" it seems) - 32-bit is on the left, 64-bit on the right. >>>>> >>>>> Script command is started on Tue Dec 29 08:39:16 UTC 2015. | Script command is started on Tue Dec 29 08:39:56 UTC 2015. >>>>> root at x069:[/data/prj/cran/32/R-aix-3.2.3/src/library/tools]make -s sysdata | root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]make -s sysdata >>>>> installing 'sysdata.rda' | installing 'sysdata.rda' >>>>> tre_tnfa_run_parallel, input type 1 | tre_tnfa_run_parallel, input type 1 >>>>> length: -1 | length: -1 >>>>> pos:chr/code | states and tags | pos:chr/code | states and tags >>>>> -------------+------------------------------------------------ | -------------+------------------------------------------------ >>>>> init> 30380200 3038014c 30380098 | init> 110cc3040 110cc2f28 110cc2e10 >>>>> match end offset = -1 | match end offset = -1 >>>>> tre_tnfa_run_parallel, input type 1 | tre_tnfa_run_parallel, input type 1 >>>>> length: -1 | length: -1 >>>>> pos:chr/code | states and tags | pos:chr/code | states and tags >>>>> -------------+------------------------------------------------ | -------------+------------------------------------------------ >>>>> init> 3037fb88 | init> 110cc3310 >>>>> 0: 3/00051 | 3037fb88/0:0 | 0: 3/00051 | 110cc3310/0:0 >>>>> 1: ./00046 | 3037fb88/0:0 | 1: ./00046 | 110cc3310/0:0 >>>>> init> 3037fb88 | init> 110cc3310 >>>>> 1: ./00046 | 3037fb88/0:1 | 1: ./00046 | 110cc3310/0:1 >>>>> 2: 2/00050 | 3037fb88/0:1 | 2: 2/00050 | 110cc3310/0:1 >>>>> assertion failed | assertion failed >>>>> init> 3037fb88 | init> 110cc3310 >>>>> 2: 2/00050 | 3037fc18/0:1 3037fb88/0:2 | 2: 2/00050 | 110cc33f0/0:1 110cc3310/0:2 >>>>> 3: ./00046 | 3037fc18/0:1 3037fb88/0:2 | 3: ./00046 | 110cc33f0/0:1 110cc3310/0:2 >>>>> assertion failed *** DIFFERENCE *** | init> 110cc3310 >>>>> init> 3037fb88 | 3: ./00046 | 110cc3310/0:3 >>>>> 3: ./00046 | 3037fc18/0:1 3037fb88/0:3 | 4: 3/00051 | 110cc3310/0:3 >>>>> 4: 3/00051 | 3037fc18/0:1 3037fb88/0:3 | assertion failed >>>>> assertion failed | init> 110cc3310 >>>>> init> 3037fb88 | 4: 3/00051 | 110cc33f0/0:3 110cc3310/0:4 >>>>> 4: 3/00051 | 3037fc18/0:3 3037fb88/0:4 | 5: /00000 | 110cc33f0/0:3 110cc3310/0:4 >>>>> 5: /00000 | 3037fc18/0:3 3037fb88/0:4 | init> 110cc3310 >>>>> found match 3037fd14 *** DIFFERENCE *** | match end offset = -1 >>>>> match end offset = 5 *** DIFFERENCE *** | tre_tnfa_run_parallel, input type 1 >>>>> tre_tnfa_run_parallel, input type 1 | length: -1 >>>>> length: -1 | pos:chr/code | states and tags >>>>> pos:chr/code | states and tags | -------------+------------------------------------------------ >>>>> -------------+------------------------------------------------ | init> 110cc4780 110cc4668 110cc4550 >>>>> init> 303811c0 3038110c 30381058 | match end offset = -1 >>>>> match end offset = -1 | tre_tnfa_run_parallel, input type 1 >>>>> tre_tnfa_run_parallel, input type 1 | length: -1 >>>>> length: -1 | pos:chr/code | states and tags >>>>> pos:chr/code | states and tags | -------------+------------------------------------------------ >>>>> -------------+------------------------------------------------ | init> 110cc5700 110cc55e8 110cc54d0 >>>>> >>>> One day further - looks like tre_compile (or just before, after all). >>>> >>>> With TRE_DEBUG switched on in tre-compile.c and tre-ast.c I see (snip) >>>> >>>> --- /tmp/x.32 2015-12-31 15:09:44.000000000 +0000 >>>> +++ /tmp/x.64 2015-12-31 15:09:30.000000000 +0000 >>>> @@ -1,5 +1,5 @@ >>>> - Script command is started on Thu Dec 31 15:04:39 2015. >>>> - root at x069:[/data/prj/cran/32/R-aix-3.2.3/src/library/tools]make sysdata >>>> + Script command is started on Thu Dec 31 15:08:43 2015. >>>> + root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]make sysdata >>>> installing 'sysdata.rda' >>>> echo "tools:::sysdata2LazyLoadDB(\"/data/prj/cran/R-3.2.3/src/library/tools/R/sysdata.rda\",\"../../../library/tools/R\")" | \ >>>> R_DEFAULT_PACKAGES=NULL LC_ALL=C ../../../bin/R --vanilla --slave >>>> @@ -167,7 +167,7 @@ >>>> initial: 1/1,0, assert 0 >>>> initial: 0/0, assert 0 >>>> initial: 0/0, assert 0 >>>> - final state 30370718 >>>> + final state 110cba530 >>>> tre_compile: parsing '(^|[^%])(%%)*%V' >>>> AST: >>>> catenation, sub 0, 0 tags >>>> @@ -177,7 +177,7 @@ >>>> assertions: bol >>>> union, sub -1, 0 tags >>>> literal (, $) (0, 36), pos 0, sub -1, 0 tags >>>> - literal (&, M-^?) (38, 65535), pos 0, sub -1, 0 tags >>>> + literal (&, M-^?) (38, -1), pos 0, sub -1, 0 tags >>>> iteration {0, -1}, sub -1, 0 tags, greedy >>>> catenation, sub 2, 0 tags >>>> literal (%, %) (37, 37), pos 1, sub -1, 0 tags >>>> @@ -197,7 +197,7 @@ >>>> Union >>>> Literal 0-36 >>>> After union left >>>> - Literal 38-65535 >>>> + Literal 38--1 >>>> After union right >>>> After union right >>>> num_tags += 2 >>>> @@ -231,7 +231,7 @@ >>>> assertions: bol >>>> union, sub -1, 0 tags >>>> literal (, $) (0, 36), pos 0, sub -1, 0 tags >>>> - literal (&, M-^?) (38, 65535), pos 0, sub -1, 0 tags >>>> + literal (&, M-^?) (38, -1), pos 0, sub -1, 0 tags >>>> iteration {0, -1}, sub -1, 2 tags, greedy >>>> catenation, sub 2, 1 tags >>>> literal (%, %) (37, 37), pos 1, sub -1, 1 tags >>>> @@ -255,7 +255,7 @@ >>>> Union >>>> Literal 0-36 >>>> After union left >>>> - Literal 38-65535 >>>> + Literal 38--1 >>>> After union right >>>> After union right >>>> tre_add_tag_right: tag 3 >>>> @@ -342,7 +342,7 @@ >>>> catenation, sub -1, 0 tags >>>> union, sub -1, 0 tags >>>> literal (, $) (0, 36), pos 0, sub -1, 0 tags >>>> - literal (&, M-^?) (38, 65535), pos 0, sub -1, 0 tags >>>> + literal (&, M-^?) (38, -1), pos 0, sub -1, 0 tags >>>> tag 4 >>>> >>>> It seems in 32-bit mode -1 is unsigned (65535) but -1 == -1 in 64-bit mode. >>>> >>>> I suspect I will "find it" - but a proposed change is appreciated. >>>> >>>> Happy New Year, >>>> Michael >>>> >>>> ______________________________________________ >>>> R-devel at r-project.org mailing list >>>> https://stat.ethz.ch/mailman/listinfo/r-devel >>>> >>> ______________________________________________ >>> R-devel at r-project.org mailing list >>> https://stat.ethz.ch/mailman/listinfo/r-devel >>> >> ______________________________________________ >> R-devel at r-project.org mailing list >> https://stat.ethz.ch/mailman/listinfo/r-devel
Michael Felt
2016-Jan-03 16:28 UTC
[Rd] R, AIX 64-bit builds - trying to understand root cause for message: "Error: Line starting 'Package: tools ...' is malformed!"
On 2016-01-03 16:59, Michael Felt wrote:> On 2016-01-01 23:48, peter dalgaard wrote: >> Nice catch you two!!! >> >> Happy New Year >> -pd > I am much happier with this great start! > > Simon - which compiler)s) did you use: xlc and xlfortran, or > gcc/gfortran? > > I have made some changes to configure(.ac) so maybe my problems are > self-inflicted. But would be good to know what environment you are using. >Using the dist, plus the following change: diff -ru R-3.2.3.dist/src/extra/tre/tre-internal.h R-3.2.3.1/src/extra/tre/tre-internal.h --- R-3.2.3.dist/src/extra/tre/tre-internal.h 2014-06-13 22:15:07.000000000 +0000 +++ R-3.2.3.1/src/extra/tre/tre-internal.h 2016-01-03 16:08:31.000000000 +0000 @@ -47,7 +47,7 @@ #ifdef TRE_WCHAR /* Wide characters. */ - typedef wint_t tre_cint_t; + typedef uint16_t tre_cint_t; #define TRE_CHAR_MAX WCHAR_MAX #ifdef TRE_MULTIBYTE @@ -78,7 +78,7 @@ #else /* !TRE_WCHAR */ /* 8 bit characters. */ - typedef short tre_cint_t; + typedef uint16_t tre_cint_t; #define TRE_CHAR_MAX 255 #define TRE_MB_CUR_MAX 1 @@ -118,7 +118,7 @@ #define tre_ctype(s) wctype(s) #else /* !TRE_USE_SYSTEM_WCTYPE */ /* Define our own versions of iswctype() and wctype(). */ - typedef int (*tre_ctype_t)(tre_cint_t); + typedef uint16_t (*tre_ctype_t)(tre_cint_t); #define tre_isctype(c, type) ( (type)(c) ) tre_ctype_t tre_ctype(const char *name); #endif /* !TRE_USE_SYSTEM_WCTYPE */ I get the same result as with my modified configure(.ac), namely: make[4]: Entering directory '/data/prj/cran/64/R-aix-3.2.3/src/library/grDevices' byte-compiling package 'grDevices' Warning in solve.default(rgb) : unable to load shared object '/data/prj/cran/64/R-aix-3.2.3/modules//lapack.so': rtld: 0712-001 Symbol _gfortran_pow_i4_i4 was referenced from module /data/prj/cran/64/R-aix-3.2.3/lib/libRlapack.so(), but a runtime definition of the symbol was not found. rtld: 0712-001 Symbol _gfortran_compare_string/data/prj/cran/64/R-aix-3.2.3/lib/libRlapack.so was referenced from module /data/prj/cran/64/R-aix-3.2.3/lib/libRlapack.so(), but a runtime definition of the symbol was not found. rtld: 0712-001 Symbol _gfortran_concat_string was referenced from module /data/prj/cran/64/R-aix-3.2.3/lib/libRlapack.so(), but a runtime definition of the symbol was not found. rtld: 0712-002 fatal error: exiting. Error in solve.default(rgb) : LAPACK routines cannot be loaded Error: unable to load R code in package 'grDevices' Execution halted My expectation is that this has to do with some incorrect flags being passed to the linker (not getting the 64-bit libraries of the compiler). I have had several long discussions with an AIX gcc specialist and he has given me several recommendations for changes to the flags used for creating shared libraries. I shall continue looking into this. I am happy it is not my configure.ac changes - and I have a few ideas on how to resolve in configure.ac and/or in m4 files. MF> Thanks for looking - and finding!!! > > Michael >>> On 01 Jan 2016, at 22:06 , Simon >>> Urbanek<simon.urbanek at r-project.org> wrote: >>> >>> Ok, found the problem - on platforms that support it TRE uses wint_t >>> (from wchar.h) as its type for characters (tre_cint_t) which on AIX >>> is *signed* int. TRE uses liberally conversions between int and >>> tre_cint_t apparently assuming that the latter is unsigned so >>> conversions back to int are suitable for comparisons etc. On other >>> platforms wint_t is unsigned so it works. Manually defining >>> tre_cint_t to unsigned int fixes the issue. >>> >>> Cheers, >>> Simon >>> >>> >>> On Jan 1, 2016, at 12:20 PM, Simon >>> Urbanek<simon.urbanek at r-project.org> wrote: >>> >>>> Michael, >>>> >>>> thanks, I'll have a look once my PDP VMs are up again (later >>>> today). This may be a signedness issue although it's unclear why >>>> other platforms wouldn't be affected. >>>> >>>> Cheers, >>>> Simon >>>> >>>> >>>> On Dec 31, 2015, at 10:14 AM, Michael Felt<aixtools at gmail.com> wrote: >>>> >>>>> On 2015-12-30 09:58, Michael Felt wrote: >>>>>> On 2015-12-29 11:02, Michael Felt wrote: >>>>>>> This seems to be a problem that goes back a long time - and I >>>>>>> hope someone who understands what tre is suppossed to be doing >>>>>>> will look at this. >>>>>>> >>>>>>> A short history of other people who have reported on this on >>>>>>> different versions of AIX. I shall only add that I get the same >>>>>>> results on AIX 5.3 TL7, AIX 6.1 TL9 and AIX 7.1 TL3. >>>>>>> >>>>>>> Basically, with settings that work for AIX and 32-bit - the only >>>>>>> changes being >>>>>>> -maix32 becomes -maix64 >>>>>>> and >>>>>>> export OBJECT_MODE=32 becomes export OBJECT_MODE=64 >>>>>>> >>>>>>> Then to shorten the 'make' bla bla, first run just make, then >>>>>>> >>>>>>> cd src/library/tools >>>>>>> make -s sysdata >>>>>>> >>>>>>> http://article.gmane.org/gmane.comp.lang.r.devel/38817/match=package+tools+malformed >>>>>>> >>>>>>> http://article.gmane.org/gmane.comp.lang.r.devel/36886/match=package+tools+malformed >>>>>>> >>>>>>> http://article.gmane.org/gmane.comp.lang.r.devel/23372/match=package+tools+malformed >>>>>>> Date: 2010-01-25 06:55:41 GMT (5 years, 48 weeks, 1 day, 20 >>>>>>> hours and 30 minutes ago) >>>>>>> >>>>>>> To that, to get debug data, I have >>>>>>> >>>>>>> * added -DTRE_DUGUG to src/extra/tre/Makefile # ALL_CFLAGS = >>>>>>> $(ALL_CFLAGS_LO) -DTRE_DEBUG >>>>>>> * rm src/extra/tre/tre-match-parallel.o >>>>>>> * find . -name \*.so -exec rm {} \; >>>>>>> * make >>>>>>> * cd src/library/tools >>>>>>> * make -s sysdata >>>>>>> >>>>>>> Attached are the two script files of the screen output. The >>>>>>> 32-bit one is more verbose - and contains magically lines such as: >>>>>>> found match 3037fd14 (while "found" does not occur in the 64-bit >>>>>>> output) >>>>>>> >>>>>>> root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]wc >>>>>>> /tmp/sysdata.??.* >>>>>>> 4730 14123 139916 /tmp/sysdata.32.text >>>>>>> 1312 3688 40528 /tmp/sysdata.64.text >>>>>>> 6042 17811 180444 total >>>>>>> >>>>>>> root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]grep >>>>>>> -c found /tmp/sysdata.??.* >>>>>>> /tmp/sysdata.32.text:19 >>>>>>> /tmp/sysdata.64.text:0 >>>>>>> >>>>>>> >>>>>>> Hope this brings us (or me), closer to a resolution to an old >>>>>>> concern. >>>>>>> >>>>>>> And, best wishes for the new year! >>>>>>> >>>>>>> Michael >>>>>>> >>>>>>> >>>>>> Still hoping for someones curiosity/willingness. >>>>>> >>>>>> The differences show up in the first comparision that is made (of >>>>>> the string "3.2.3" it seems) - 32-bit is on the left, 64-bit on >>>>>> the right. >>>>>> >>>>>> Script command is started on Tue Dec 29 08:39:16 UTC >>>>>> 2015. | Script command is started on Tue Dec >>>>>> 29 08:39:56 UTC 2015. >>>>>> root at x069:[/data/prj/cran/32/R-aix-3.2.3/src/library/tools]make >>>>>> -s sysdata | >>>>>> root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]make >>>>>> -s sysdata >>>>>> installing >>>>>> 'sysdata.rda' >>>>>> | installing 'sysdata.rda' >>>>>> tre_tnfa_run_parallel, input type >>>>>> 1 | >>>>>> tre_tnfa_run_parallel, input type 1 >>>>>> length: >>>>>> -1 >>>>>> | length: -1 >>>>>> pos:chr/code | states and >>>>>> tags | >>>>>> pos:chr/code | states and tags >>>>>> -------------+------------------------------------------------ >>>>>> | -------------+------------------------------------------------ >>>>>> init> 30380200 3038014c >>>>>> 30380098 | init> >>>>>> 110cc3040 110cc2f28 110cc2e10 >>>>>> match end offset = >>>>>> -1 | >>>>>> match end offset = -1 >>>>>> tre_tnfa_run_parallel, input type >>>>>> 1 | >>>>>> tre_tnfa_run_parallel, input type 1 >>>>>> length: >>>>>> -1 >>>>>> | length: -1 >>>>>> pos:chr/code | states and >>>>>> tags | >>>>>> pos:chr/code | states and tags >>>>>> -------------+------------------------------------------------ >>>>>> | -------------+------------------------------------------------ >>>>>> init> >>>>>> 3037fb88 >>>>>> | init> 110cc3310 >>>>>> 0: 3/00051 | >>>>>> 3037fb88/0:0 >>>>>> | 0: 3/00051 | 110cc3310/0:0 >>>>>> 1: ./00046 | >>>>>> 3037fb88/0:0 >>>>>> | 1: ./00046 | 110cc3310/0:0 >>>>>> init> >>>>>> 3037fb88 >>>>>> | init> 110cc3310 >>>>>> 1: ./00046 | >>>>>> 3037fb88/0:1 >>>>>> | 1: ./00046 | 110cc3310/0:1 >>>>>> 2: 2/00050 | >>>>>> 3037fb88/0:1 >>>>>> | 2: 2/00050 | 110cc3310/0:1 >>>>>> assertion >>>>>> failed >>>>>> | assertion failed >>>>>> init> >>>>>> 3037fb88 >>>>>> | init> 110cc3310 >>>>>> 2: 2/00050 | 3037fc18/0:1 >>>>>> 3037fb88/0:2 | 2: >>>>>> 2/00050 | 110cc33f0/0:1 110cc3310/0:2 >>>>>> 3: ./00046 | 3037fc18/0:1 >>>>>> 3037fb88/0:2 | 3: >>>>>> ./00046 | 110cc33f0/0:1 110cc3310/0:2 >>>>>> assertion failed *** DIFFERENCE >>>>>> *** | init> 110cc3310 >>>>>> init> >>>>>> 3037fb88 >>>>>> | 3: ./00046 | 110cc3310/0:3 >>>>>> 3: ./00046 | 3037fc18/0:1 >>>>>> 3037fb88/0:3 | 4: >>>>>> 3/00051 | 110cc3310/0:3 >>>>>> 4: 3/00051 | 3037fc18/0:1 >>>>>> 3037fb88/0:3 | assertion >>>>>> failed >>>>>> assertion >>>>>> failed >>>>>> | init> 110cc3310 >>>>>> init> >>>>>> 3037fb88 >>>>>> | 4: 3/00051 | 110cc33f0/0:3 110cc3310/0:4 >>>>>> 4: 3/00051 | 3037fc18/0:3 >>>>>> 3037fb88/0:4 | 5: /00000 >>>>>> | 110cc33f0/0:3 110cc3310/0:4 >>>>>> 5: /00000 | 3037fc18/0:3 3037fb88/0:4 | init> 110cc3310 >>>>>> found match 3037fd14 *** DIFFERENCE *** | >>>>>> match end offset = -1 >>>>>> match end offset = 5 *** DIFFERENCE *** | >>>>>> tre_tnfa_run_parallel, input type 1 >>>>>> tre_tnfa_run_parallel, input type >>>>>> 1 | length: -1 >>>>>> length: >>>>>> -1 >>>>>> | pos:chr/code | states and tags >>>>>> pos:chr/code | states and >>>>>> tags | >>>>>> -------------+------------------------------------------------ >>>>>> -------------+------------------------------------------------ >>>>>> | init> 110cc4780 110cc4668 110cc4550 >>>>>> init> 303811c0 3038110c >>>>>> 30381058 | match end >>>>>> offset = -1 >>>>>> match end offset = >>>>>> -1 | >>>>>> tre_tnfa_run_parallel, input type 1 >>>>>> tre_tnfa_run_parallel, input type >>>>>> 1 | length: -1 >>>>>> length: >>>>>> -1 >>>>>> | pos:chr/code | states and tags >>>>>> pos:chr/code | states and >>>>>> tags | >>>>>> -------------+------------------------------------------------ >>>>>> -------------+------------------------------------------------ >>>>>> | init> 110cc5700 110cc55e8 110cc54d0 >>>>>> >>>>> One day further - looks like tre_compile (or just before, after all). >>>>> >>>>> With TRE_DEBUG switched on in tre-compile.c and tre-ast.c I see >>>>> (snip) >>>>> >>>>> --- /tmp/x.32 2015-12-31 15:09:44.000000000 +0000 >>>>> +++ /tmp/x.64 2015-12-31 15:09:30.000000000 +0000 >>>>> @@ -1,5 +1,5 @@ >>>>> - Script command is started on Thu Dec 31 15:04:39 2015. >>>>> - root at x069:[/data/prj/cran/32/R-aix-3.2.3/src/library/tools]make >>>>> sysdata >>>>> + Script command is started on Thu Dec 31 15:08:43 2015. >>>>> + root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]make >>>>> sysdata >>>>> installing 'sysdata.rda' >>>>> echo >>>>> "tools:::sysdata2LazyLoadDB(\"/data/prj/cran/R-3.2.3/src/library/tools/R/sysdata.rda\",\"../../../library/tools/R\")" >>>>> | \ >>>>> R_DEFAULT_PACKAGES=NULL LC_ALL=C ../../../bin/R --vanilla --slave >>>>> @@ -167,7 +167,7 @@ >>>>> initial: 1/1,0, assert 0 >>>>> initial: 0/0, assert 0 >>>>> initial: 0/0, assert 0 >>>>> - final state 30370718 >>>>> + final state 110cba530 >>>>> tre_compile: parsing '(^|[^%])(%%)*%V' >>>>> AST: >>>>> catenation, sub 0, 0 tags >>>>> @@ -177,7 +177,7 @@ >>>>> assertions: bol >>>>> union, sub -1, 0 tags >>>>> literal (, $) (0, 36), pos 0, sub -1, 0 tags >>>>> - literal (&, M-^?) (38, 65535), pos 0, sub -1, 0 tags >>>>> + literal (&, M-^?) (38, -1), pos 0, sub -1, 0 tags >>>>> iteration {0, -1}, sub -1, 0 tags, greedy >>>>> catenation, sub 2, 0 tags >>>>> literal (%, %) (37, 37), pos 1, sub -1, 0 tags >>>>> @@ -197,7 +197,7 @@ >>>>> Union >>>>> Literal 0-36 >>>>> After union left >>>>> - Literal 38-65535 >>>>> + Literal 38--1 >>>>> After union right >>>>> After union right >>>>> num_tags += 2 >>>>> @@ -231,7 +231,7 @@ >>>>> assertions: bol >>>>> union, sub -1, 0 tags >>>>> literal (, $) (0, 36), pos 0, sub -1, 0 tags >>>>> - literal (&, M-^?) (38, 65535), pos 0, sub -1, 0 tags >>>>> + literal (&, M-^?) (38, -1), pos 0, sub -1, 0 tags >>>>> iteration {0, -1}, sub -1, 2 tags, greedy >>>>> catenation, sub 2, 1 tags >>>>> literal (%, %) (37, 37), pos 1, sub -1, 1 tags >>>>> @@ -255,7 +255,7 @@ >>>>> Union >>>>> Literal 0-36 >>>>> After union left >>>>> - Literal 38-65535 >>>>> + Literal 38--1 >>>>> After union right >>>>> After union right >>>>> tre_add_tag_right: tag 3 >>>>> @@ -342,7 +342,7 @@ >>>>> catenation, sub -1, 0 tags >>>>> union, sub -1, 0 tags >>>>> literal (, $) (0, 36), pos 0, sub -1, 0 tags >>>>> - literal (&, M-^?) (38, 65535), pos 0, sub -1, 0 tags >>>>> + literal (&, M-^?) (38, -1), pos 0, sub -1, 0 tags >>>>> tag 4 >>>>> >>>>> It seems in 32-bit mode -1 is unsigned (65535) but -1 == -1 in >>>>> 64-bit mode. >>>>> >>>>> I suspect I will "find it" - but a proposed change is appreciated. >>>>> >>>>> Happy New Year, >>>>> Michael >>>>> >>>>> ______________________________________________ >>>>> R-devel at r-project.org mailing list >>>>> https://stat.ethz.ch/mailman/listinfo/r-devel >>>>> >>>> ______________________________________________ >>>> R-devel at r-project.org mailing list >>>> https://stat.ethz.ch/mailman/listinfo/r-devel >>>> >>> ______________________________________________ >>> R-devel at r-project.org mailing list >>> https://stat.ethz.ch/mailman/listinfo/r-devel >
Michael Felt
2016-Jan-03 17:00 UTC
[Rd] R, AIX 64-bit builds - trying to understand root cause for message: "Error: Line starting 'Package: tools ...' is malformed!"
On 2016-01-03 17:28, Michael Felt wrote:> On 2016-01-03 16:59, Michael Felt wrote: >> On 2016-01-01 23:48, peter dalgaard wrote: >>> Nice catch you two!!! >>> >>> Happy New Year >>> -pd >> I am much happier with this great start! >> >> Simon - which compiler)s) did you use: xlc and xlfortran, or >> gcc/gfortran? >> >> I have made some changes to configure(.ac) so maybe my problems are >> self-inflicted. But would be good to know what environment you are >> using. >> > Using the dist, plus the following change: > > diff -ru R-3.2.3.dist/src/extra/tre/tre-internal.h > R-3.2.3.1/src/extra/tre/tre-internal.h > --- R-3.2.3.dist/src/extra/tre/tre-internal.h 2014-06-13 > 22:15:07.000000000 +0000 > +++ R-3.2.3.1/src/extra/tre/tre-internal.h 2016-01-03 > 16:08:31.000000000 +0000 > @@ -47,7 +47,7 @@ > #ifdef TRE_WCHAR > > /* Wide characters. */ > - typedef wint_t tre_cint_t; > + typedef uint16_t tre_cint_t; > #define TRE_CHAR_MAX WCHAR_MAX > > #ifdef TRE_MULTIBYTE > @@ -78,7 +78,7 @@ > #else /* !TRE_WCHAR */ > > /* 8 bit characters. */ > - typedef short tre_cint_t; > + typedef uint16_t tre_cint_t; > #define TRE_CHAR_MAX 255 > #define TRE_MB_CUR_MAX 1 > > @@ -118,7 +118,7 @@ > #define tre_ctype(s) wctype(s) > #else /* !TRE_USE_SYSTEM_WCTYPE */ > /* Define our own versions of iswctype() and wctype(). */ > - typedef int (*tre_ctype_t)(tre_cint_t); > + typedef uint16_t (*tre_ctype_t)(tre_cint_t); > #define tre_isctype(c, type) ( (type)(c) ) > tre_ctype_t tre_ctype(const char *name); > #endif /* !TRE_USE_SYSTEM_WCTYPE */ > > I get the same result as with my modified configure(.ac), namely: > > make[4]: Entering directory > '/data/prj/cran/64/R-aix-3.2.3/src/library/grDevices' > byte-compiling package 'grDevices' > Warning in solve.default(rgb) : > unable to load shared object > '/data/prj/cran/64/R-aix-3.2.3/modules//lapack.so': > rtld: 0712-001 Symbol _gfortran_pow_i4_i4 was referenced > from module /data/prj/cran/64/R-aix-3.2.3/lib/libRlapack.so(), > but a runtime definition > of the symbol was not found. > rtld: 0712-001 Symbol > _gfortran_compare_string/data/prj/cran/64/R-aix-3.2.3/lib/libRlapack.so was > referenced > from module /data/prj/cran/64/R-aix-3.2.3/lib/libRlapack.so(), > but a runtime definition > of the symbol was not found. > rtld: 0712-001 Symbol _gfortran_concat_string was referenced > from module /data/prj/cran/64/R-aix-3.2.3/lib/libRlapack.so(), > but a runtime definition > of the symbol was not found. > rtld: 0712-002 fatal error: exiting. > Error in solve.default(rgb) : LAPACK routines cannot be loaded > Error: unable to load R code in package 'grDevices' > Execution halted > > My expectation is that this has to do with some incorrect flags being > passed to the linker (not getting the 64-bit libraries of the > compiler). I have had several long discussions with an AIX gcc > specialist and he has given me several recommendations for changes to > the flags used for creating shared libraries. I shall continue looking > into this. > > I am happy it is not my configure.ac changes - and I have a few ideas > on how to resolve in configure.ac and/or in m4 files. > > MFFYI: Still running into problems in grDevices using /data/prj/cran/${RELEASE}/configure --disable-lto --prefix=/opt \ --disable-R-shlib --enable-R-static-lib \ --with-cairo=no --with-libpng=no --with-jpeglib=no --with-libtiff=no \ --with-readline=no --with-x=no So, Simon - very interested in your environment and whether make completes successfully - to the end. p.s. - I am a bit surprised about the messages below (line 4) because I am assuming that shared libraries are not being used (with --disable-R-shlib --enable-R-static-lib ). fyi - from memory - but my expectation is that it has to do with the -bexpall and an incorrect library search path being inserted. The symbol is in /opt/lib/ppc64 archives: root at x069:[/opt/lib/ppc64]nm -X64 -Ae *.a | grep __register_frame_info_table libgcc_s.a[shr.o]: .__register_frame_info_table T 268466492 116 libgcc_s.a[shr.o]: .__register_frame_info_table_bases T 268466364 120 libgcc_s.a[shr.o]: __register_frame_info_table D 536877472 libgcc_s.a[shr.o]: __register_frame_info_table d 536877472 24 libgcc_s.a[shr.o]: __register_frame_info_table_bases D 536880832 libgcc_s.a[shr.o]: __register_frame_info_table_bases d 536880832 24 libstdc++.a[libstdc++.so.6]: .__register_frame_info_table T 269107692 libstdc++.a[libstdc++.so.6]: .__register_frame_info_table t 269107692 40 libstdc++.a[libstdc++.so.6]: __register_frame_info_table U - libstdc++.a[libstdc++.so.6]: __register_frame_info_table d 537156720 8 But these are not being found, rather specified. Somehow the default behavior of gcc (i.e., the flags it passes to ld are being overruled). It will take time, but I shall find them. make[4]: Entering directory '/data/prj/cran/64/R-aix-3.2.3/src/library/grDevices' byte-compiling package 'grDevices' *** caught segfault *** address 6c20776173206e6f, cause 'memory not mapped' Traceback: 1: `$<-`(`*tmp*`, "name", value = "muffleWarning") 2: makeRestartList(...) 3: withRestarts({ .Internal(.signalCondition(simpleWarning(msg, call), msg, call)) .Internal(.dfltWarn(msg, call))}, muffleWarning = function() NULL) 4: .signalSimpleWarning("unable to load shared object '/data/prj/cran/64/R-aix-3.2.3/modules//lapack.so':\n rtld: 0712-001 Symbol __register_frame_info_table was referenced\n from module /data/prj/cran/64/R-aix-3.2.3/modules/ /lapack.so(), but a runtime definition\n\t of the symbol was not found.\nrtld: 0712-001 Symbol __deregister_frame _info was referenced\n from module /data/prj/cran/64/R-aix-3.2.3/modules//lapack.so(), but a runtime definition \n\t of the symbol was not found.\nrtld: 0712-001 Symbol __register_frame_info_table was referenced\n from module /data/prj/cran/64/R-aix-3.2.3/lib/libRlapack.so(), but a runtime definition\n\t of the symbol was not found.\nrtld: 0712-001 Symbol __deregister_frame_info was referenced\n from module /data/prj/cran/64/R-aix-3.2.3/lib/libRlapack.so(), but a runtime definition\n\t of the symbol was not found.\nrtld: 0712-001 Symbol _gfortran_pow_i4_i4 was referenced\n from module /data/prj/cran/64/R-aix-3.2.3/lib/libRlapack.so(), but a runtime definition", quote(solve.default(rgb))) 5: solve.default(rgb) 6: solve(rgb) 7: drop(whitexyz %*% solve(rgb)) 8: make.rgb(red = c(0.625, 0.34), green = c(0.28, 0.595), blue = c(0.155, 0.07), gamma = 1.8, white = "D65", name = "Apple RGB") 9: eval(expr, envir, enclos) 10: eval(exprs[i], envir) 11: sys.source(codeFile, env, keep.source = keep.source) 12: doTryCatch(return(expr), name, parentenv, handler) 13: tryCatchOne(expr, names, parentenv, handlers[[1L]]) 14: tryCatchList(expr, classes, parentenv, handlers) 15: tryCatch(expr, error = function(e) { call <- conditionCall(e) if (!is.null(call)) { if (identical(call[[1L]], quote(doTryCatch))) call <- sys.call(-4L) dcall <- deparse(call)[1L] prefix <- paste("Error in", dcall, ": ") LONG <- 75L msg <- conditionMessage(e) sm <- strsplit(msg, "\n") [[1L]] w <- 14L + nchar(dcall, type = "w") + nchar(sm[1L], type = "w") if (is.na(w)) w <-14L + nchar(dcall, type = "b") + nchar(sm[1L], type = "b") if (w > LONG) prefix <- paste0(prefix, "\n ") } else prefix <- "Error : " msg <- paste0(prefix, conditionMessage(e), "\n") .Internal(seterrmessage(msg[1L])) if (!silent && identical(getOption("show.error.messages"), TRUE)) { cat(msg, file = stderr()) .Internal(printDeferredWarnings()) } invisible(structure(msg, class = "try-error", condition = e))}) 16: try(sys.source(codeFile, env, keep.source = keep.source)) 17: loadNamespace(package, lib.loc, keep.source, partial = TRUE) 18: withCallingHandlers(expr, packageStartupMessage = function(c) invokeRestart("muffleMessage")) 19: suppressPackageStartupMessages(loadNamespace(package, lib.loc, keep.source, partial = TRUE)) 20: code2LazyLoadDB(package, lib.loc = lib.loc, keep.source = keep.source, compress = compress) 21: tools:::makeLazyLoading("grDevices") aborting ... /data/prj/cran/R-3.2.3.1/share/make/lazycomp.mk:7: recipe for target '../../../library/grDevices/R/grDevices.rdb' failed make[4]: *** [../../../library/grDevices/R/grDevices.rdb] Segmentation fault (core dumped) make[4]: Leaving directory '/data/prj/cran/64/R-aix-3.2.3/src/library/grDevices' Makefile:24: recipe for target 'all' failed make[3]: *** [all] Error 2 make[3]: Leaving directory '/data/prj/cran/64/R-aix-3.2.3/src/library/grDevices' Makefile:36: recipe for target 'R' failed make[2]: *** [R] Error 1 make[2]: Leaving directory '/data/prj/cran/64/R-aix-3.2.3/src/library' Makefile:28: recipe for target 'R' failed make[1]: *** [R] Error 1 make[1]: Leaving directory '/data/prj/cran/64/R-aix-3.2.3/src' Makefile:60: recipe for target 'R' failed make: *** [R] Error 1 root at x065:[/data/prj/cran/64/R-aix-3.2.3]>> Thanks for looking - and finding!!! >> >> Michael >>>> On 01 Jan 2016, at 22:06 , Simon >>>> Urbanek<simon.urbanek at r-project.org> wrote: >>>> >>>> Ok, found the problem - on platforms that support it TRE uses >>>> wint_t (from wchar.h) as its type for characters (tre_cint_t) which >>>> on AIX is *signed* int. TRE uses liberally conversions between int >>>> and tre_cint_t apparently assuming that the latter is unsigned so >>>> conversions back to int are suitable for comparisons etc. On other >>>> platforms wint_t is unsigned so it works. Manually defining >>>> tre_cint_t to unsigned int fixes the issue. >>>> >>>> Cheers, >>>> Simon >>>> >>>> >>>> On Jan 1, 2016, at 12:20 PM, Simon >>>> Urbanek<simon.urbanek at r-project.org> wrote: >>>> >>>>> Michael, >>>>> >>>>> thanks, I'll have a look once my PDP VMs are up again (later >>>>> today). This may be a signedness issue although it's unclear why >>>>> other platforms wouldn't be affected. >>>>> >>>>> Cheers, >>>>> Simon >>>>> >>>>> >>>>> On Dec 31, 2015, at 10:14 AM, Michael Felt<aixtools at gmail.com> >>>>> wrote: >>>>> >>>>>> On 2015-12-30 09:58, Michael Felt wrote: >>>>>>> On 2015-12-29 11:02, Michael Felt wrote: >>>>>>>> This seems to be a problem that goes back a long time - and I >>>>>>>> hope someone who understands what tre is suppossed to be doing >>>>>>>> will look at this. >>>>>>>> >>>>>>>> A short history of other people who have reported on this on >>>>>>>> different versions of AIX. I shall only add that I get the same >>>>>>>> results on AIX 5.3 TL7, AIX 6.1 TL9 and AIX 7.1 TL3. >>>>>>>> >>>>>>>> Basically, with settings that work for AIX and 32-bit - the >>>>>>>> only changes being >>>>>>>> -maix32 becomes -maix64 >>>>>>>> and >>>>>>>> export OBJECT_MODE=32 becomes export OBJECT_MODE=64 >>>>>>>> >>>>>>>> Then to shorten the 'make' bla bla, first run just make, then >>>>>>>> >>>>>>>> cd src/library/tools >>>>>>>> make -s sysdata >>>>>>>> >>>>>>>> http://article.gmane.org/gmane.comp.lang.r.devel/38817/match=package+tools+malformed >>>>>>>> >>>>>>>> http://article.gmane.org/gmane.comp.lang.r.devel/36886/match=package+tools+malformed >>>>>>>> >>>>>>>> http://article.gmane.org/gmane.comp.lang.r.devel/23372/match=package+tools+malformed >>>>>>>> Date: 2010-01-25 06:55:41 GMT (5 years, 48 weeks, 1 day, 20 >>>>>>>> hours and 30 minutes ago) >>>>>>>> >>>>>>>> To that, to get debug data, I have >>>>>>>> >>>>>>>> * added -DTRE_DUGUG to src/extra/tre/Makefile # ALL_CFLAGS = >>>>>>>> $(ALL_CFLAGS_LO) -DTRE_DEBUG >>>>>>>> * rm src/extra/tre/tre-match-parallel.o >>>>>>>> * find . -name \*.so -exec rm {} \; >>>>>>>> * make >>>>>>>> * cd src/library/tools >>>>>>>> * make -s sysdata >>>>>>>> >>>>>>>> Attached are the two script files of the screen output. The >>>>>>>> 32-bit one is more verbose - and contains magically lines such as: >>>>>>>> found match 3037fd14 (while "found" does not occur in the >>>>>>>> 64-bit output) >>>>>>>> >>>>>>>> root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]wc >>>>>>>> /tmp/sysdata.??.* >>>>>>>> 4730 14123 139916 /tmp/sysdata.32.text >>>>>>>> 1312 3688 40528 /tmp/sysdata.64.text >>>>>>>> 6042 17811 180444 total >>>>>>>> >>>>>>>> root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]grep >>>>>>>> -c found /tmp/sysdata.??.* >>>>>>>> /tmp/sysdata.32.text:19 >>>>>>>> /tmp/sysdata.64.text:0 >>>>>>>> >>>>>>>> >>>>>>>> Hope this brings us (or me), closer to a resolution to an old >>>>>>>> concern. >>>>>>>> >>>>>>>> And, best wishes for the new year! >>>>>>>> >>>>>>>> Michael >>>>>>>> >>>>>>>> >>>>>>> Still hoping for someones curiosity/willingness. >>>>>>> >>>>>>> The differences show up in the first comparision that is made >>>>>>> (of the string "3.2.3" it seems) - 32-bit is on the left, 64-bit >>>>>>> on the right. >>>>>>> >>>>>>> Script command is started on Tue Dec 29 08:39:16 UTC >>>>>>> 2015. | Script command is started on Tue >>>>>>> Dec 29 08:39:56 UTC 2015. >>>>>>> root at x069:[/data/prj/cran/32/R-aix-3.2.3/src/library/tools]make >>>>>>> -s sysdata | >>>>>>> root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]make >>>>>>> -s sysdata >>>>>>> installing >>>>>>> 'sysdata.rda' >>>>>>> | installing 'sysdata.rda' >>>>>>> tre_tnfa_run_parallel, input type >>>>>>> 1 | >>>>>>> tre_tnfa_run_parallel, input type 1 >>>>>>> length: >>>>>>> -1 >>>>>>> | length: -1 >>>>>>> pos:chr/code | states and >>>>>>> tags | >>>>>>> pos:chr/code | states and tags >>>>>>> -------------+------------------------------------------------ >>>>>>> | -------------+------------------------------------------------ >>>>>>> init> 30380200 3038014c >>>>>>> 30380098 | init> >>>>>>> 110cc3040 110cc2f28 110cc2e10 >>>>>>> match end offset = >>>>>>> -1 | >>>>>>> match end offset = -1 >>>>>>> tre_tnfa_run_parallel, input type >>>>>>> 1 | >>>>>>> tre_tnfa_run_parallel, input type 1 >>>>>>> length: >>>>>>> -1 >>>>>>> | length: -1 >>>>>>> pos:chr/code | states and >>>>>>> tags | >>>>>>> pos:chr/code | states and tags >>>>>>> -------------+------------------------------------------------ >>>>>>> | -------------+------------------------------------------------ >>>>>>> init> >>>>>>> 3037fb88 >>>>>>> | init> 110cc3310 >>>>>>> 0: 3/00051 | >>>>>>> 3037fb88/0:0 >>>>>>> | 0: 3/00051 | 110cc3310/0:0 >>>>>>> 1: ./00046 | >>>>>>> 3037fb88/0:0 >>>>>>> | 1: ./00046 | 110cc3310/0:0 >>>>>>> init> >>>>>>> 3037fb88 >>>>>>> | init> 110cc3310 >>>>>>> 1: ./00046 | >>>>>>> 3037fb88/0:1 >>>>>>> | 1: ./00046 | 110cc3310/0:1 >>>>>>> 2: 2/00050 | >>>>>>> 3037fb88/0:1 >>>>>>> | 2: 2/00050 | 110cc3310/0:1 >>>>>>> assertion >>>>>>> failed >>>>>>> | assertion failed >>>>>>> init> >>>>>>> 3037fb88 >>>>>>> | init> 110cc3310 >>>>>>> 2: 2/00050 | 3037fc18/0:1 >>>>>>> 3037fb88/0:2 | 2: >>>>>>> 2/00050 | 110cc33f0/0:1 110cc3310/0:2 >>>>>>> 3: ./00046 | 3037fc18/0:1 >>>>>>> 3037fb88/0:2 | 3: >>>>>>> ./00046 | 110cc33f0/0:1 110cc3310/0:2 >>>>>>> assertion failed *** DIFFERENCE >>>>>>> *** | init> 110cc3310 >>>>>>> init> >>>>>>> 3037fb88 >>>>>>> | 3: ./00046 | 110cc3310/0:3 >>>>>>> 3: ./00046 | 3037fc18/0:1 >>>>>>> 3037fb88/0:3 | 4: >>>>>>> 3/00051 | 110cc3310/0:3 >>>>>>> 4: 3/00051 | 3037fc18/0:1 >>>>>>> 3037fb88/0:3 | assertion >>>>>>> failed >>>>>>> assertion >>>>>>> failed >>>>>>> | init> 110cc3310 >>>>>>> init> >>>>>>> 3037fb88 >>>>>>> | 4: 3/00051 | 110cc33f0/0:3 110cc3310/0:4 >>>>>>> 4: 3/00051 | 3037fc18/0:3 >>>>>>> 3037fb88/0:4 | 5: >>>>>>> /00000 | 110cc33f0/0:3 110cc3310/0:4 >>>>>>> 5: /00000 | 3037fc18/0:3 3037fb88/0:4 | init> 110cc3310 >>>>>>> found match 3037fd14 *** DIFFERENCE *** | >>>>>>> match end offset = -1 >>>>>>> match end offset = 5 *** DIFFERENCE *** | >>>>>>> tre_tnfa_run_parallel, input type 1 >>>>>>> tre_tnfa_run_parallel, input type >>>>>>> 1 | length: -1 >>>>>>> length: >>>>>>> -1 >>>>>>> | pos:chr/code | states and tags >>>>>>> pos:chr/code | states and >>>>>>> tags | >>>>>>> -------------+------------------------------------------------ >>>>>>> -------------+------------------------------------------------ >>>>>>> | init> 110cc4780 110cc4668 110cc4550 >>>>>>> init> 303811c0 3038110c >>>>>>> 30381058 | match >>>>>>> end offset = -1 >>>>>>> match end offset = >>>>>>> -1 | >>>>>>> tre_tnfa_run_parallel, input type 1 >>>>>>> tre_tnfa_run_parallel, input type >>>>>>> 1 | length: -1 >>>>>>> length: >>>>>>> -1 >>>>>>> | pos:chr/code | states and tags >>>>>>> pos:chr/code | states and >>>>>>> tags | >>>>>>> -------------+------------------------------------------------ >>>>>>> -------------+------------------------------------------------ >>>>>>> | init> 110cc5700 110cc55e8 110cc54d0 >>>>>>> >>>>>> One day further - looks like tre_compile (or just before, after >>>>>> all). >>>>>> >>>>>> With TRE_DEBUG switched on in tre-compile.c and tre-ast.c I see >>>>>> (snip) >>>>>> >>>>>> --- /tmp/x.32 2015-12-31 15:09:44.000000000 +0000 >>>>>> +++ /tmp/x.64 2015-12-31 15:09:30.000000000 +0000 >>>>>> @@ -1,5 +1,5 @@ >>>>>> - Script command is started on Thu Dec 31 15:04:39 2015. >>>>>> - root at x069:[/data/prj/cran/32/R-aix-3.2.3/src/library/tools]make >>>>>> sysdata >>>>>> + Script command is started on Thu Dec 31 15:08:43 2015. >>>>>> + root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]make >>>>>> sysdata >>>>>> installing 'sysdata.rda' >>>>>> echo >>>>>> "tools:::sysdata2LazyLoadDB(\"/data/prj/cran/R-3.2.3/src/library/tools/R/sysdata.rda\",\"../../../library/tools/R\")" >>>>>> | \ >>>>>> R_DEFAULT_PACKAGES=NULL LC_ALL=C ../../../bin/R --vanilla --slave >>>>>> @@ -167,7 +167,7 @@ >>>>>> initial: 1/1,0, assert 0 >>>>>> initial: 0/0, assert 0 >>>>>> initial: 0/0, assert 0 >>>>>> - final state 30370718 >>>>>> + final state 110cba530 >>>>>> tre_compile: parsing '(^|[^%])(%%)*%V' >>>>>> AST: >>>>>> catenation, sub 0, 0 tags >>>>>> @@ -177,7 +177,7 @@ >>>>>> assertions: bol >>>>>> union, sub -1, 0 tags >>>>>> literal (, $) (0, 36), pos 0, sub -1, 0 tags >>>>>> - literal (&, M-^?) (38, 65535), pos 0, sub -1, 0 tags >>>>>> + literal (&, M-^?) (38, -1), pos 0, sub -1, 0 tags >>>>>> iteration {0, -1}, sub -1, 0 tags, greedy >>>>>> catenation, sub 2, 0 tags >>>>>> literal (%, %) (37, 37), pos 1, sub -1, 0 tags >>>>>> @@ -197,7 +197,7 @@ >>>>>> Union >>>>>> Literal 0-36 >>>>>> After union left >>>>>> - Literal 38-65535 >>>>>> + Literal 38--1 >>>>>> After union right >>>>>> After union right >>>>>> num_tags += 2 >>>>>> @@ -231,7 +231,7 @@ >>>>>> assertions: bol >>>>>> union, sub -1, 0 tags >>>>>> literal (, $) (0, 36), pos 0, sub -1, 0 tags >>>>>> - literal (&, M-^?) (38, 65535), pos 0, sub -1, 0 tags >>>>>> + literal (&, M-^?) (38, -1), pos 0, sub -1, 0 tags >>>>>> iteration {0, -1}, sub -1, 2 tags, greedy >>>>>> catenation, sub 2, 1 tags >>>>>> literal (%, %) (37, 37), pos 1, sub -1, 1 tags >>>>>> @@ -255,7 +255,7 @@ >>>>>> Union >>>>>> Literal 0-36 >>>>>> After union left >>>>>> - Literal 38-65535 >>>>>> + Literal 38--1 >>>>>> After union right >>>>>> After union right >>>>>> tre_add_tag_right: tag 3 >>>>>> @@ -342,7 +342,7 @@ >>>>>> catenation, sub -1, 0 tags >>>>>> union, sub -1, 0 tags >>>>>> literal (, $) (0, 36), pos 0, sub -1, 0 tags >>>>>> - literal (&, M-^?) (38, 65535), pos 0, sub -1, 0 tags >>>>>> + literal (&, M-^?) (38, -1), pos 0, sub -1, 0 tags >>>>>> tag 4 >>>>>> >>>>>> It seems in 32-bit mode -1 is unsigned (65535) but -1 == -1 in >>>>>> 64-bit mode. >>>>>> >>>>>> I suspect I will "find it" - but a proposed change is appreciated. >>>>>> >>>>>> Happy New Year, >>>>>> Michael >>>>>> >>>>>> ______________________________________________ >>>>>> R-devel at r-project.org mailing list >>>>>> https://stat.ethz.ch/mailman/listinfo/r-devel >>>>>> >>>>> ______________________________________________ >>>>> R-devel at r-project.org mailing list >>>>> https://stat.ethz.ch/mailman/listinfo/r-devel >>>>> >>>> ______________________________________________ >>>> R-devel at r-project.org mailing list >>>> https://stat.ethz.ch/mailman/listinfo/r-devel >> >
Simon Urbanek
2016-Jan-04 00:26 UTC
[Rd] R, AIX 64-bit builds - trying to understand root cause for message: "Error: Line starting 'Package: tools ...' is malformed!"
Michael, I'm using xlc + xlf - the exact flags are configure CC=xlc_r CXX=xlc++_r F77=xlf_r FC=xlf95_r LIBS='-L/opt/freeware/lib /opt/freeware/lib/libiconv.a -lpthread' --prefix=/opt/freeware CPPFLAGS=-I/opt/freeware/include with export OBJECT_MODE=64 in the env (and libiconv from perzl.org RPMs - iconv is a serious pain). I have changed the TRE typedef from "wint_t" to "unsigned int" since that is what the other platforms use anyway. On my AIX7 VM I'm running out of memory in xz when lazy-loading is enabled which his odd - the VM has 4Gb which one would think should be enough. The symptom is that installing MASS fails when creating lazy-load rds or the same happens when running make check as memCompress() test for xz fails. I wish there was a way to disable xz altogether.. I'd like to get IBM compilers to work first since those are the official ones. I may try gcc later, but I'm really interested in the IBM ones because that gives us a good non-GNU test (e.g., TRE fails in JIT due some alleged GNUisms so you can't use the strict compiler mode). Cheers, Simon On Jan 3, 2016, at 10:59 AM, Michael Felt <aixtools at gmail.com> wrote:> On 2016-01-01 23:48, peter dalgaard wrote: >> Nice catch you two!!! >> >> Happy New Year >> -pd > I am much happier with this great start! > > Simon - which compiler)s) did you use: xlc and xlfortran, or gcc/gfortran? > > I have made some changes to configure(.ac) so maybe my problems are self-inflicted. But would be good to know what environment you are using. > > Thanks for looking - and finding!!! > > Michael >>> On 01 Jan 2016, at 22:06 , Simon Urbanek<simon.urbanek at r-project.org> wrote: >>> >>> Ok, found the problem - on platforms that support it TRE uses wint_t (from wchar.h) as its type for characters (tre_cint_t) which on AIX is *signed* int. TRE uses liberally conversions between int and tre_cint_t apparently assuming that the latter is unsigned so conversions back to int are suitable for comparisons etc. On other platforms wint_t is unsigned so it works. Manually defining tre_cint_t to unsigned int fixes the issue. >>> >>> Cheers, >>> Simon >>> >>> >>> On Jan 1, 2016, at 12:20 PM, Simon Urbanek<simon.urbanek at r-project.org> wrote: >>> >>>> Michael, >>>> >>>> thanks, I'll have a look once my PDP VMs are up again (later today). This may be a signedness issue although it's unclear why other platforms wouldn't be affected. >>>> >>>> Cheers, >>>> Simon >>>> >>>> >>>> On Dec 31, 2015, at 10:14 AM, Michael Felt<aixtools at gmail.com> wrote: >>>> >>>>> On 2015-12-30 09:58, Michael Felt wrote: >>>>>> On 2015-12-29 11:02, Michael Felt wrote: >>>>>>> This seems to be a problem that goes back a long time - and I hope someone who understands what tre is suppossed to be doing will look at this. >>>>>>> >>>>>>> A short history of other people who have reported on this on different versions of AIX. I shall only add that I get the same results on AIX 5.3 TL7, AIX 6.1 TL9 and AIX 7.1 TL3. >>>>>>> >>>>>>> Basically, with settings that work for AIX and 32-bit - the only changes being >>>>>>> -maix32 becomes -maix64 >>>>>>> and >>>>>>> export OBJECT_MODE=32 becomes export OBJECT_MODE=64 >>>>>>> >>>>>>> Then to shorten the 'make' bla bla, first run just make, then >>>>>>> >>>>>>> cd src/library/tools >>>>>>> make -s sysdata >>>>>>> >>>>>>> http://article.gmane.org/gmane.comp.lang.r.devel/38817/match=package+tools+malformed >>>>>>> http://article.gmane.org/gmane.comp.lang.r.devel/36886/match=package+tools+malformed >>>>>>> http://article.gmane.org/gmane.comp.lang.r.devel/23372/match=package+tools+malformed Date: 2010-01-25 06:55:41 GMT (5 years, 48 weeks, 1 day, 20 hours and 30 minutes ago) >>>>>>> >>>>>>> To that, to get debug data, I have >>>>>>> >>>>>>> * added -DTRE_DUGUG to src/extra/tre/Makefile # ALL_CFLAGS = $(ALL_CFLAGS_LO) -DTRE_DEBUG >>>>>>> * rm src/extra/tre/tre-match-parallel.o >>>>>>> * find . -name \*.so -exec rm {} \; >>>>>>> * make >>>>>>> * cd src/library/tools >>>>>>> * make -s sysdata >>>>>>> >>>>>>> Attached are the two script files of the screen output. The 32-bit one is more verbose - and contains magically lines such as: >>>>>>> found match 3037fd14 (while "found" does not occur in the 64-bit output) >>>>>>> >>>>>>> root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]wc /tmp/sysdata.??.* >>>>>>> 4730 14123 139916 /tmp/sysdata.32.text >>>>>>> 1312 3688 40528 /tmp/sysdata.64.text >>>>>>> 6042 17811 180444 total >>>>>>> >>>>>>> root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]grep -c found /tmp/sysdata.??.* >>>>>>> /tmp/sysdata.32.text:19 >>>>>>> /tmp/sysdata.64.text:0 >>>>>>> >>>>>>> >>>>>>> Hope this brings us (or me), closer to a resolution to an old concern. >>>>>>> >>>>>>> And, best wishes for the new year! >>>>>>> >>>>>>> Michael >>>>>>> >>>>>>> >>>>>> Still hoping for someones curiosity/willingness. >>>>>> >>>>>> The differences show up in the first comparision that is made (of the string "3.2.3" it seems) - 32-bit is on the left, 64-bit on the right. >>>>>> >>>>>> Script command is started on Tue Dec 29 08:39:16 UTC 2015. | Script command is started on Tue Dec 29 08:39:56 UTC 2015. >>>>>> root at x069:[/data/prj/cran/32/R-aix-3.2.3/src/library/tools]make -s sysdata | root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]make -s sysdata >>>>>> installing 'sysdata.rda' | installing 'sysdata.rda' >>>>>> tre_tnfa_run_parallel, input type 1 | tre_tnfa_run_parallel, input type 1 >>>>>> length: -1 | length: -1 >>>>>> pos:chr/code | states and tags | pos:chr/code | states and tags >>>>>> -------------+------------------------------------------------ | -------------+------------------------------------------------ >>>>>> init> 30380200 3038014c 30380098 | init> 110cc3040 110cc2f28 110cc2e10 >>>>>> match end offset = -1 | match end offset = -1 >>>>>> tre_tnfa_run_parallel, input type 1 | tre_tnfa_run_parallel, input type 1 >>>>>> length: -1 | length: -1 >>>>>> pos:chr/code | states and tags | pos:chr/code | states and tags >>>>>> -------------+------------------------------------------------ | -------------+------------------------------------------------ >>>>>> init> 3037fb88 | init> 110cc3310 >>>>>> 0: 3/00051 | 3037fb88/0:0 | 0: 3/00051 | 110cc3310/0:0 >>>>>> 1: ./00046 | 3037fb88/0:0 | 1: ./00046 | 110cc3310/0:0 >>>>>> init> 3037fb88 | init> 110cc3310 >>>>>> 1: ./00046 | 3037fb88/0:1 | 1: ./00046 | 110cc3310/0:1 >>>>>> 2: 2/00050 | 3037fb88/0:1 | 2: 2/00050 | 110cc3310/0:1 >>>>>> assertion failed | assertion failed >>>>>> init> 3037fb88 | init> 110cc3310 >>>>>> 2: 2/00050 | 3037fc18/0:1 3037fb88/0:2 | 2: 2/00050 | 110cc33f0/0:1 110cc3310/0:2 >>>>>> 3: ./00046 | 3037fc18/0:1 3037fb88/0:2 | 3: ./00046 | 110cc33f0/0:1 110cc3310/0:2 >>>>>> assertion failed *** DIFFERENCE *** | init> 110cc3310 >>>>>> init> 3037fb88 | 3: ./00046 | 110cc3310/0:3 >>>>>> 3: ./00046 | 3037fc18/0:1 3037fb88/0:3 | 4: 3/00051 | 110cc3310/0:3 >>>>>> 4: 3/00051 | 3037fc18/0:1 3037fb88/0:3 | assertion failed >>>>>> assertion failed | init> 110cc3310 >>>>>> init> 3037fb88 | 4: 3/00051 | 110cc33f0/0:3 110cc3310/0:4 >>>>>> 4: 3/00051 | 3037fc18/0:3 3037fb88/0:4 | 5: /00000 | 110cc33f0/0:3 110cc3310/0:4 >>>>>> 5: /00000 | 3037fc18/0:3 3037fb88/0:4 | init> 110cc3310 >>>>>> found match 3037fd14 *** DIFFERENCE *** | match end offset = -1 >>>>>> match end offset = 5 *** DIFFERENCE *** | tre_tnfa_run_parallel, input type 1 >>>>>> tre_tnfa_run_parallel, input type 1 | length: -1 >>>>>> length: -1 | pos:chr/code | states and tags >>>>>> pos:chr/code | states and tags | -------------+------------------------------------------------ >>>>>> -------------+------------------------------------------------ | init> 110cc4780 110cc4668 110cc4550 >>>>>> init> 303811c0 3038110c 30381058 | match end offset = -1 >>>>>> match end offset = -1 | tre_tnfa_run_parallel, input type 1 >>>>>> tre_tnfa_run_parallel, input type 1 | length: -1 >>>>>> length: -1 | pos:chr/code | states and tags >>>>>> pos:chr/code | states and tags | -------------+------------------------------------------------ >>>>>> -------------+------------------------------------------------ | init> 110cc5700 110cc55e8 110cc54d0 >>>>>> >>>>> One day further - looks like tre_compile (or just before, after all). >>>>> >>>>> With TRE_DEBUG switched on in tre-compile.c and tre-ast.c I see (snip) >>>>> >>>>> --- /tmp/x.32 2015-12-31 15:09:44.000000000 +0000 >>>>> +++ /tmp/x.64 2015-12-31 15:09:30.000000000 +0000 >>>>> @@ -1,5 +1,5 @@ >>>>> - Script command is started on Thu Dec 31 15:04:39 2015. >>>>> - root at x069:[/data/prj/cran/32/R-aix-3.2.3/src/library/tools]make sysdata >>>>> + Script command is started on Thu Dec 31 15:08:43 2015. >>>>> + root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]make sysdata >>>>> installing 'sysdata.rda' >>>>> echo "tools:::sysdata2LazyLoadDB(\"/data/prj/cran/R-3.2.3/src/library/tools/R/sysdata.rda\",\"../../../library/tools/R\")" | \ >>>>> R_DEFAULT_PACKAGES=NULL LC_ALL=C ../../../bin/R --vanilla --slave >>>>> @@ -167,7 +167,7 @@ >>>>> initial: 1/1,0, assert 0 >>>>> initial: 0/0, assert 0 >>>>> initial: 0/0, assert 0 >>>>> - final state 30370718 >>>>> + final state 110cba530 >>>>> tre_compile: parsing '(^|[^%])(%%)*%V' >>>>> AST: >>>>> catenation, sub 0, 0 tags >>>>> @@ -177,7 +177,7 @@ >>>>> assertions: bol >>>>> union, sub -1, 0 tags >>>>> literal (, $) (0, 36), pos 0, sub -1, 0 tags >>>>> - literal (&, M-^?) (38, 65535), pos 0, sub -1, 0 tags >>>>> + literal (&, M-^?) (38, -1), pos 0, sub -1, 0 tags >>>>> iteration {0, -1}, sub -1, 0 tags, greedy >>>>> catenation, sub 2, 0 tags >>>>> literal (%, %) (37, 37), pos 1, sub -1, 0 tags >>>>> @@ -197,7 +197,7 @@ >>>>> Union >>>>> Literal 0-36 >>>>> After union left >>>>> - Literal 38-65535 >>>>> + Literal 38--1 >>>>> After union right >>>>> After union right >>>>> num_tags += 2 >>>>> @@ -231,7 +231,7 @@ >>>>> assertions: bol >>>>> union, sub -1, 0 tags >>>>> literal (, $) (0, 36), pos 0, sub -1, 0 tags >>>>> - literal (&, M-^?) (38, 65535), pos 0, sub -1, 0 tags >>>>> + literal (&, M-^?) (38, -1), pos 0, sub -1, 0 tags >>>>> iteration {0, -1}, sub -1, 2 tags, greedy >>>>> catenation, sub 2, 1 tags >>>>> literal (%, %) (37, 37), pos 1, sub -1, 1 tags >>>>> @@ -255,7 +255,7 @@ >>>>> Union >>>>> Literal 0-36 >>>>> After union left >>>>> - Literal 38-65535 >>>>> + Literal 38--1 >>>>> After union right >>>>> After union right >>>>> tre_add_tag_right: tag 3 >>>>> @@ -342,7 +342,7 @@ >>>>> catenation, sub -1, 0 tags >>>>> union, sub -1, 0 tags >>>>> literal (, $) (0, 36), pos 0, sub -1, 0 tags >>>>> - literal (&, M-^?) (38, 65535), pos 0, sub -1, 0 tags >>>>> + literal (&, M-^?) (38, -1), pos 0, sub -1, 0 tags >>>>> tag 4 >>>>> >>>>> It seems in 32-bit mode -1 is unsigned (65535) but -1 == -1 in 64-bit mode. >>>>> >>>>> I suspect I will "find it" - but a proposed change is appreciated. >>>>> >>>>> Happy New Year, >>>>> Michael >>>>> >>>>> ______________________________________________ >>>>> R-devel at r-project.org mailing list >>>>> https://stat.ethz.ch/mailman/listinfo/r-devel >>>>> >>>> ______________________________________________ >>>> R-devel at r-project.org mailing list >>>> https://stat.ethz.ch/mailman/listinfo/r-devel >>>> >>> ______________________________________________ >>> R-devel at r-project.org mailing list >>> https://stat.ethz.ch/mailman/listinfo/r-devel >
Michael Felt
2016-Jan-04 06:34 UTC
[Rd] R, AIX 64-bit builds - trying to understand root cause for message: "Error: Line starting 'Package: tools ...' is malformed!"
I would be "pleased" if you would try packages - i.e., none of "Toolbox" or Perzl rpm's. The iconv I supply might not be working as is (I will need to install a new system to check). However, this is why I have been testing with R-3.2.3 - to side-step the system library dependencies. re: xz - I have "installp" version - and any of my packages should work side-by-side with the Toolbox/.rpm files - my PATH prefix is /opt (so, /opt/bin and /opt/sbin rather than /usr/bin, /usr/local/bin (etc.). I am testing on a Power6, usually with 2 to 4G RAM active. When I get the compile to finish I will up to 8G to test some data processing that fails with the 32-bit limitations. Another question: are you also getting lots of duplicate symbol warnings? Curious me. If you are I would like to send my modifications to configure(.ac) for you to test (I put .ac in () because I am actually editting configure atm but shall put them into configure.ac when I finish) FYI: I shall be downloading the "try and buy" xlc and xlfortran - and I think you will certainly prefer my packaging then as they work without the libgc dependencies that many of the rpm packages need. And, at your option - we can take this discussion over tools - "out of forums". Or at least start a new thread. re: unsigned short - I have adopted the convention to use the *NN_t types after running into a problem with unsigned longlong (from a very old program). It goes back many years - and maybe they have finalized short to mean 16-bits - but I remember when short was meant to help when moving from 16-bit to 32-bit and the "word" size changed - i.e., int became 32-bit same as long. nd for a long (no pun intended) long was 32-bit and long long was 64-bit. Those definitions are extinct. imho - the standard for wint_t is wrong as well - based on an assumption about how "short" is defined. And, I consider it poor practice that there are som many cases of type cast switches between ushort and int. Michael On 04-Jan-16 01:26, Simon Urbanek wrote:> Michael, > > I'm using xlc + xlf - the exact flags are > > configure CC=xlc_r CXX=xlc++_r F77=xlf_r FC=xlf95_r LIBS='-L/opt/freeware/lib /opt/freeware/lib/libiconv.a -lpthread' --prefix=/opt/freeware CPPFLAGS=-I/opt/freeware/include > > with export OBJECT_MODE=64 in the env (and libiconv from perzl.org RPMs - iconv is a serious pain). > > I have changed the TRE typedef from "wint_t" to "unsigned int" since that is what the other platforms use anyway. > > On my AIX7 VM I'm running out of memory in xz when lazy-loading is enabled which his odd - the VM has 4Gb which one would think should be enough. The symptom is that installing MASS fails when creating lazy-load rds or the same happens when running make check as memCompress() test for xz fails. I wish there was a way to disable xz altogether.. > > I'd like to get IBM compilers to work first since those are the official ones. I may try gcc later, but I'm really interested in the IBM ones because that gives us a good non-GNU test (e.g., TRE fails in JIT due some alleged GNUisms so you can't use the strict compiler mode). > > Cheers, > Simon > > > On Jan 3, 2016, at 10:59 AM, Michael Felt <aixtools at gmail.com> wrote: > >> On 2016-01-01 23:48, peter dalgaard wrote: >>> Nice catch you two!!! >>> >>> Happy New Year >>> -pd >> I am much happier with this great start! >> >> Simon - which compiler)s) did you use: xlc and xlfortran, or gcc/gfortran? >> >> I have made some changes to configure(.ac) so maybe my problems are self-inflicted. But would be good to know what environment you are using. >> >> Thanks for looking - and finding!!! >> >> Michael >>>> On 01 Jan 2016, at 22:06 , Simon Urbanek<simon.urbanek at r-project.org> wrote: >>>> >>>> Ok, found the problem - on platforms that support it TRE uses wint_t (from wchar.h) as its type for characters (tre_cint_t) which on AIX is *signed* int. TRE uses liberally conversions between int and tre_cint_t apparently assuming that the latter is unsigned so conversions back to int are suitable for comparisons etc. On other platforms wint_t is unsigned so it works. Manually defining tre_cint_t to unsigned int fixes the issue. >>>> >>>> Cheers, >>>> Simon >>>> >>>> >>>> On Jan 1, 2016, at 12:20 PM, Simon Urbanek<simon.urbanek at r-project.org> wrote: >>>> >>>>> Michael, >>>>> >>>>> thanks, I'll have a look once my PDP VMs are up again (later today). This may be a signedness issue although it's unclear why other platforms wouldn't be affected. >>>>> >>>>> Cheers, >>>>> Simon >>>>> >>>>> >>>>> On Dec 31, 2015, at 10:14 AM, Michael Felt<aixtools at gmail.com> wrote: >>>>> >>>>>> On 2015-12-30 09:58, Michael Felt wrote: >>>>>>> On 2015-12-29 11:02, Michael Felt wrote: >>>>>>>> This seems to be a problem that goes back a long time - and I hope someone who understands what tre is suppossed to be doing will look at this. >>>>>>>> >>>>>>>> A short history of other people who have reported on this on different versions of AIX. I shall only add that I get the same results on AIX 5.3 TL7, AIX 6.1 TL9 and AIX 7.1 TL3. >>>>>>>> >>>>>>>> Basically, with settings that work for AIX and 32-bit - the only changes being >>>>>>>> -maix32 becomes -maix64 >>>>>>>> and >>>>>>>> export OBJECT_MODE=32 becomes export OBJECT_MODE=64 >>>>>>>> >>>>>>>> Then to shorten the 'make' bla bla, first run just make, then >>>>>>>> >>>>>>>> cd src/library/tools >>>>>>>> make -s sysdata >>>>>>>> >>>>>>>> http://article.gmane.org/gmane.comp.lang.r.devel/38817/match=package+tools+malformed >>>>>>>> http://article.gmane.org/gmane.comp.lang.r.devel/36886/match=package+tools+malformed >>>>>>>> http://article.gmane.org/gmane.comp.lang.r.devel/23372/match=package+tools+malformed Date: 2010-01-25 06:55:41 GMT (5 years, 48 weeks, 1 day, 20 hours and 30 minutes ago) >>>>>>>> >>>>>>>> To that, to get debug data, I have >>>>>>>> >>>>>>>> * added -DTRE_DUGUG to src/extra/tre/Makefile # ALL_CFLAGS = $(ALL_CFLAGS_LO) -DTRE_DEBUG >>>>>>>> * rm src/extra/tre/tre-match-parallel.o >>>>>>>> * find . -name \*.so -exec rm {} \; >>>>>>>> * make >>>>>>>> * cd src/library/tools >>>>>>>> * make -s sysdata >>>>>>>> >>>>>>>> Attached are the two script files of the screen output. The 32-bit one is more verbose - and contains magically lines such as: >>>>>>>> found match 3037fd14 (while "found" does not occur in the 64-bit output) >>>>>>>> >>>>>>>> root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]wc /tmp/sysdata.??.* >>>>>>>> 4730 14123 139916 /tmp/sysdata.32.text >>>>>>>> 1312 3688 40528 /tmp/sysdata.64.text >>>>>>>> 6042 17811 180444 total >>>>>>>> >>>>>>>> root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]grep -c found /tmp/sysdata.??.* >>>>>>>> /tmp/sysdata.32.text:19 >>>>>>>> /tmp/sysdata.64.text:0 >>>>>>>> >>>>>>>> >>>>>>>> Hope this brings us (or me), closer to a resolution to an old concern. >>>>>>>> >>>>>>>> And, best wishes for the new year! >>>>>>>> >>>>>>>> Michael >>>>>>>> >>>>>>>> >>>>>>> Still hoping for someones curiosity/willingness. >>>>>>> >>>>>>> The differences show up in the first comparision that is made (of the string "3.2.3" it seems) - 32-bit is on the left, 64-bit on the right. >>>>>>> >>>>>>> Script command is started on Tue Dec 29 08:39:16 UTC 2015. | Script command is started on Tue Dec 29 08:39:56 UTC 2015. >>>>>>> root at x069:[/data/prj/cran/32/R-aix-3.2.3/src/library/tools]make -s sysdata | root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]make -s sysdata >>>>>>> installing 'sysdata.rda' | installing 'sysdata.rda' >>>>>>> tre_tnfa_run_parallel, input type 1 | tre_tnfa_run_parallel, input type 1 >>>>>>> length: -1 | length: -1 >>>>>>> pos:chr/code | states and tags | pos:chr/code | states and tags >>>>>>> -------------+------------------------------------------------ | -------------+------------------------------------------------ >>>>>>> init> 30380200 3038014c 30380098 | init> 110cc3040 110cc2f28 110cc2e10 >>>>>>> match end offset = -1 | match end offset = -1 >>>>>>> tre_tnfa_run_parallel, input type 1 | tre_tnfa_run_parallel, input type 1 >>>>>>> length: -1 | length: -1 >>>>>>> pos:chr/code | states and tags | pos:chr/code | states and tags >>>>>>> -------------+------------------------------------------------ | -------------+------------------------------------------------ >>>>>>> init> 3037fb88 | init> 110cc3310 >>>>>>> 0: 3/00051 | 3037fb88/0:0 | 0: 3/00051 | 110cc3310/0:0 >>>>>>> 1: ./00046 | 3037fb88/0:0 | 1: ./00046 | 110cc3310/0:0 >>>>>>> init> 3037fb88 | init> 110cc3310 >>>>>>> 1: ./00046 | 3037fb88/0:1 | 1: ./00046 | 110cc3310/0:1 >>>>>>> 2: 2/00050 | 3037fb88/0:1 | 2: 2/00050 | 110cc3310/0:1 >>>>>>> assertion failed | assertion failed >>>>>>> init> 3037fb88 | init> 110cc3310 >>>>>>> 2: 2/00050 | 3037fc18/0:1 3037fb88/0:2 | 2: 2/00050 | 110cc33f0/0:1 110cc3310/0:2 >>>>>>> 3: ./00046 | 3037fc18/0:1 3037fb88/0:2 | 3: ./00046 | 110cc33f0/0:1 110cc3310/0:2 >>>>>>> assertion failed *** DIFFERENCE *** | init> 110cc3310 >>>>>>> init> 3037fb88 | 3: ./00046 | 110cc3310/0:3 >>>>>>> 3: ./00046 | 3037fc18/0:1 3037fb88/0:3 | 4: 3/00051 | 110cc3310/0:3 >>>>>>> 4: 3/00051 | 3037fc18/0:1 3037fb88/0:3 | assertion failed >>>>>>> assertion failed | init> 110cc3310 >>>>>>> init> 3037fb88 | 4: 3/00051 | 110cc33f0/0:3 110cc3310/0:4 >>>>>>> 4: 3/00051 | 3037fc18/0:3 3037fb88/0:4 | 5: /00000 | 110cc33f0/0:3 110cc3310/0:4 >>>>>>> 5: /00000 | 3037fc18/0:3 3037fb88/0:4 | init> 110cc3310 >>>>>>> found match 3037fd14 *** DIFFERENCE *** | match end offset = -1 >>>>>>> match end offset = 5 *** DIFFERENCE *** | tre_tnfa_run_parallel, input type 1 >>>>>>> tre_tnfa_run_parallel, input type 1 | length: -1 >>>>>>> length: -1 | pos:chr/code | states and tags >>>>>>> pos:chr/code | states and tags | -------------+------------------------------------------------ >>>>>>> -------------+------------------------------------------------ | init> 110cc4780 110cc4668 110cc4550 >>>>>>> init> 303811c0 3038110c 30381058 | match end offset = -1 >>>>>>> match end offset = -1 | tre_tnfa_run_parallel, input type 1 >>>>>>> tre_tnfa_run_parallel, input type 1 | length: -1 >>>>>>> length: -1 | pos:chr/code | states and tags >>>>>>> pos:chr/code | states and tags | -------------+------------------------------------------------ >>>>>>> -------------+------------------------------------------------ | init> 110cc5700 110cc55e8 110cc54d0 >>>>>>> >>>>>> One day further - looks like tre_compile (or just before, after all). >>>>>> >>>>>> With TRE_DEBUG switched on in tre-compile.c and tre-ast.c I see (snip) >>>>>> >>>>>> --- /tmp/x.32 2015-12-31 15:09:44.000000000 +0000 >>>>>> +++ /tmp/x.64 2015-12-31 15:09:30.000000000 +0000 >>>>>> @@ -1,5 +1,5 @@ >>>>>> - Script command is started on Thu Dec 31 15:04:39 2015. >>>>>> - root at x069:[/data/prj/cran/32/R-aix-3.2.3/src/library/tools]make sysdata >>>>>> + Script command is started on Thu Dec 31 15:08:43 2015. >>>>>> + root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]make sysdata >>>>>> installing 'sysdata.rda' >>>>>> echo "tools:::sysdata2LazyLoadDB(\"/data/prj/cran/R-3.2.3/src/library/tools/R/sysdata.rda\",\"../../../library/tools/R\")" | \ >>>>>> R_DEFAULT_PACKAGES=NULL LC_ALL=C ../../../bin/R --vanilla --slave >>>>>> @@ -167,7 +167,7 @@ >>>>>> initial: 1/1,0, assert 0 >>>>>> initial: 0/0, assert 0 >>>>>> initial: 0/0, assert 0 >>>>>> - final state 30370718 >>>>>> + final state 110cba530 >>>>>> tre_compile: parsing '(^|[^%])(%%)*%V' >>>>>> AST: >>>>>> catenation, sub 0, 0 tags >>>>>> @@ -177,7 +177,7 @@ >>>>>> assertions: bol >>>>>> union, sub -1, 0 tags >>>>>> literal (, $) (0, 36), pos 0, sub -1, 0 tags >>>>>> - literal (&, M-^?) (38, 65535), pos 0, sub -1, 0 tags >>>>>> + literal (&, M-^?) (38, -1), pos 0, sub -1, 0 tags >>>>>> iteration {0, -1}, sub -1, 0 tags, greedy >>>>>> catenation, sub 2, 0 tags >>>>>> literal (%, %) (37, 37), pos 1, sub -1, 0 tags >>>>>> @@ -197,7 +197,7 @@ >>>>>> Union >>>>>> Literal 0-36 >>>>>> After union left >>>>>> - Literal 38-65535 >>>>>> + Literal 38--1 >>>>>> After union right >>>>>> After union right >>>>>> num_tags += 2 >>>>>> @@ -231,7 +231,7 @@ >>>>>> assertions: bol >>>>>> union, sub -1, 0 tags >>>>>> literal (, $) (0, 36), pos 0, sub -1, 0 tags >>>>>> - literal (&, M-^?) (38, 65535), pos 0, sub -1, 0 tags >>>>>> + literal (&, M-^?) (38, -1), pos 0, sub -1, 0 tags >>>>>> iteration {0, -1}, sub -1, 2 tags, greedy >>>>>> catenation, sub 2, 1 tags >>>>>> literal (%, %) (37, 37), pos 1, sub -1, 1 tags >>>>>> @@ -255,7 +255,7 @@ >>>>>> Union >>>>>> Literal 0-36 >>>>>> After union left >>>>>> - Literal 38-65535 >>>>>> + Literal 38--1 >>>>>> After union right >>>>>> After union right >>>>>> tre_add_tag_right: tag 3 >>>>>> @@ -342,7 +342,7 @@ >>>>>> catenation, sub -1, 0 tags >>>>>> union, sub -1, 0 tags >>>>>> literal (, $) (0, 36), pos 0, sub -1, 0 tags >>>>>> - literal (&, M-^?) (38, 65535), pos 0, sub -1, 0 tags >>>>>> + literal (&, M-^?) (38, -1), pos 0, sub -1, 0 tags >>>>>> tag 4 >>>>>> >>>>>> It seems in 32-bit mode -1 is unsigned (65535) but -1 == -1 in 64-bit mode. >>>>>> >>>>>> I suspect I will "find it" - but a proposed change is appreciated. >>>>>> >>>>>> Happy New Year, >>>>>> Michael >>>>>> >>>>>> ______________________________________________ >>>>>> R-devel at r-project.org mailing list >>>>>> https://stat.ethz.ch/mailman/listinfo/r-devel >>>>>> >>>>> ______________________________________________ >>>>> R-devel at r-project.org mailing list >>>>> https://stat.ethz.ch/mailman/listinfo/r-devel >>>>> >>>> ______________________________________________ >>>> R-devel at r-project.org mailing list >>>> https://stat.ethz.ch/mailman/listinfo/r-devel
Maybe Matching Threads
- R, AIX 64-bit builds - trying to understand root cause for message: "Error: Line starting 'Package: tools ...' is malformed!"
- R, AIX 64-bit builds - trying to understand root cause for message: "Error: Line starting 'Package: tools ...' is malformed!"
- R, AIX 64-bit builds - trying to understand root cause for message: "Error: Line starting 'Package: tools ...' is malformed!"
- R, AIX 64-bit builds - trying to understand root cause for message: "Error: Line starting 'Package: tools ...' is malformed!"
- R, AIX 64-bit builds - trying to understand root cause for message: "Error: Line starting 'Package: tools ...' is malformed!"