Michael Felt
2016-Jan-04 06:34 UTC
[Rd] R, AIX 64-bit builds - trying to understand root cause for message: "Error: Line starting 'Package: tools ...' is malformed!"
I would be "pleased" if you would try packages - i.e., none of "Toolbox" or Perzl rpm's. The iconv I supply might not be working as is (I will need to install a new system to check). However, this is why I have been testing with R-3.2.3 - to side-step the system library dependencies. re: xz - I have "installp" version - and any of my packages should work side-by-side with the Toolbox/.rpm files - my PATH prefix is /opt (so, /opt/bin and /opt/sbin rather than /usr/bin, /usr/local/bin (etc.). I am testing on a Power6, usually with 2 to 4G RAM active. When I get the compile to finish I will up to 8G to test some data processing that fails with the 32-bit limitations. Another question: are you also getting lots of duplicate symbol warnings? Curious me. If you are I would like to send my modifications to configure(.ac) for you to test (I put .ac in () because I am actually editting configure atm but shall put them into configure.ac when I finish) FYI: I shall be downloading the "try and buy" xlc and xlfortran - and I think you will certainly prefer my packaging then as they work without the libgc dependencies that many of the rpm packages need. And, at your option - we can take this discussion over tools - "out of forums". Or at least start a new thread. re: unsigned short - I have adopted the convention to use the *NN_t types after running into a problem with unsigned longlong (from a very old program). It goes back many years - and maybe they have finalized short to mean 16-bits - but I remember when short was meant to help when moving from 16-bit to 32-bit and the "word" size changed - i.e., int became 32-bit same as long. nd for a long (no pun intended) long was 32-bit and long long was 64-bit. Those definitions are extinct. imho - the standard for wint_t is wrong as well - based on an assumption about how "short" is defined. And, I consider it poor practice that there are som many cases of type cast switches between ushort and int. Michael On 04-Jan-16 01:26, Simon Urbanek wrote:> Michael, > > I'm using xlc + xlf - the exact flags are > > configure CC=xlc_r CXX=xlc++_r F77=xlf_r FC=xlf95_r LIBS='-L/opt/freeware/lib /opt/freeware/lib/libiconv.a -lpthread' --prefix=/opt/freeware CPPFLAGS=-I/opt/freeware/include > > with export OBJECT_MODE=64 in the env (and libiconv from perzl.org RPMs - iconv is a serious pain). > > I have changed the TRE typedef from "wint_t" to "unsigned int" since that is what the other platforms use anyway. > > On my AIX7 VM I'm running out of memory in xz when lazy-loading is enabled which his odd - the VM has 4Gb which one would think should be enough. The symptom is that installing MASS fails when creating lazy-load rds or the same happens when running make check as memCompress() test for xz fails. I wish there was a way to disable xz altogether.. > > I'd like to get IBM compilers to work first since those are the official ones. I may try gcc later, but I'm really interested in the IBM ones because that gives us a good non-GNU test (e.g., TRE fails in JIT due some alleged GNUisms so you can't use the strict compiler mode). > > Cheers, > Simon > > > On Jan 3, 2016, at 10:59 AM, Michael Felt <aixtools at gmail.com> wrote: > >> On 2016-01-01 23:48, peter dalgaard wrote: >>> Nice catch you two!!! >>> >>> Happy New Year >>> -pd >> I am much happier with this great start! >> >> Simon - which compiler)s) did you use: xlc and xlfortran, or gcc/gfortran? >> >> I have made some changes to configure(.ac) so maybe my problems are self-inflicted. But would be good to know what environment you are using. >> >> Thanks for looking - and finding!!! >> >> Michael >>>> On 01 Jan 2016, at 22:06 , Simon Urbanek<simon.urbanek at r-project.org> wrote: >>>> >>>> Ok, found the problem - on platforms that support it TRE uses wint_t (from wchar.h) as its type for characters (tre_cint_t) which on AIX is *signed* int. TRE uses liberally conversions between int and tre_cint_t apparently assuming that the latter is unsigned so conversions back to int are suitable for comparisons etc. On other platforms wint_t is unsigned so it works. Manually defining tre_cint_t to unsigned int fixes the issue. >>>> >>>> Cheers, >>>> Simon >>>> >>>> >>>> On Jan 1, 2016, at 12:20 PM, Simon Urbanek<simon.urbanek at r-project.org> wrote: >>>> >>>>> Michael, >>>>> >>>>> thanks, I'll have a look once my PDP VMs are up again (later today). This may be a signedness issue although it's unclear why other platforms wouldn't be affected. >>>>> >>>>> Cheers, >>>>> Simon >>>>> >>>>> >>>>> On Dec 31, 2015, at 10:14 AM, Michael Felt<aixtools at gmail.com> wrote: >>>>> >>>>>> On 2015-12-30 09:58, Michael Felt wrote: >>>>>>> On 2015-12-29 11:02, Michael Felt wrote: >>>>>>>> This seems to be a problem that goes back a long time - and I hope someone who understands what tre is suppossed to be doing will look at this. >>>>>>>> >>>>>>>> A short history of other people who have reported on this on different versions of AIX. I shall only add that I get the same results on AIX 5.3 TL7, AIX 6.1 TL9 and AIX 7.1 TL3. >>>>>>>> >>>>>>>> Basically, with settings that work for AIX and 32-bit - the only changes being >>>>>>>> -maix32 becomes -maix64 >>>>>>>> and >>>>>>>> export OBJECT_MODE=32 becomes export OBJECT_MODE=64 >>>>>>>> >>>>>>>> Then to shorten the 'make' bla bla, first run just make, then >>>>>>>> >>>>>>>> cd src/library/tools >>>>>>>> make -s sysdata >>>>>>>> >>>>>>>> http://article.gmane.org/gmane.comp.lang.r.devel/38817/match=package+tools+malformed >>>>>>>> http://article.gmane.org/gmane.comp.lang.r.devel/36886/match=package+tools+malformed >>>>>>>> http://article.gmane.org/gmane.comp.lang.r.devel/23372/match=package+tools+malformed Date: 2010-01-25 06:55:41 GMT (5 years, 48 weeks, 1 day, 20 hours and 30 minutes ago) >>>>>>>> >>>>>>>> To that, to get debug data, I have >>>>>>>> >>>>>>>> * added -DTRE_DUGUG to src/extra/tre/Makefile # ALL_CFLAGS = $(ALL_CFLAGS_LO) -DTRE_DEBUG >>>>>>>> * rm src/extra/tre/tre-match-parallel.o >>>>>>>> * find . -name \*.so -exec rm {} \; >>>>>>>> * make >>>>>>>> * cd src/library/tools >>>>>>>> * make -s sysdata >>>>>>>> >>>>>>>> Attached are the two script files of the screen output. The 32-bit one is more verbose - and contains magically lines such as: >>>>>>>> found match 3037fd14 (while "found" does not occur in the 64-bit output) >>>>>>>> >>>>>>>> root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]wc /tmp/sysdata.??.* >>>>>>>> 4730 14123 139916 /tmp/sysdata.32.text >>>>>>>> 1312 3688 40528 /tmp/sysdata.64.text >>>>>>>> 6042 17811 180444 total >>>>>>>> >>>>>>>> root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]grep -c found /tmp/sysdata.??.* >>>>>>>> /tmp/sysdata.32.text:19 >>>>>>>> /tmp/sysdata.64.text:0 >>>>>>>> >>>>>>>> >>>>>>>> Hope this brings us (or me), closer to a resolution to an old concern. >>>>>>>> >>>>>>>> And, best wishes for the new year! >>>>>>>> >>>>>>>> Michael >>>>>>>> >>>>>>>> >>>>>>> Still hoping for someones curiosity/willingness. >>>>>>> >>>>>>> The differences show up in the first comparision that is made (of the string "3.2.3" it seems) - 32-bit is on the left, 64-bit on the right. >>>>>>> >>>>>>> Script command is started on Tue Dec 29 08:39:16 UTC 2015. | Script command is started on Tue Dec 29 08:39:56 UTC 2015. >>>>>>> root at x069:[/data/prj/cran/32/R-aix-3.2.3/src/library/tools]make -s sysdata | root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]make -s sysdata >>>>>>> installing 'sysdata.rda' | installing 'sysdata.rda' >>>>>>> tre_tnfa_run_parallel, input type 1 | tre_tnfa_run_parallel, input type 1 >>>>>>> length: -1 | length: -1 >>>>>>> pos:chr/code | states and tags | pos:chr/code | states and tags >>>>>>> -------------+------------------------------------------------ | -------------+------------------------------------------------ >>>>>>> init> 30380200 3038014c 30380098 | init> 110cc3040 110cc2f28 110cc2e10 >>>>>>> match end offset = -1 | match end offset = -1 >>>>>>> tre_tnfa_run_parallel, input type 1 | tre_tnfa_run_parallel, input type 1 >>>>>>> length: -1 | length: -1 >>>>>>> pos:chr/code | states and tags | pos:chr/code | states and tags >>>>>>> -------------+------------------------------------------------ | -------------+------------------------------------------------ >>>>>>> init> 3037fb88 | init> 110cc3310 >>>>>>> 0: 3/00051 | 3037fb88/0:0 | 0: 3/00051 | 110cc3310/0:0 >>>>>>> 1: ./00046 | 3037fb88/0:0 | 1: ./00046 | 110cc3310/0:0 >>>>>>> init> 3037fb88 | init> 110cc3310 >>>>>>> 1: ./00046 | 3037fb88/0:1 | 1: ./00046 | 110cc3310/0:1 >>>>>>> 2: 2/00050 | 3037fb88/0:1 | 2: 2/00050 | 110cc3310/0:1 >>>>>>> assertion failed | assertion failed >>>>>>> init> 3037fb88 | init> 110cc3310 >>>>>>> 2: 2/00050 | 3037fc18/0:1 3037fb88/0:2 | 2: 2/00050 | 110cc33f0/0:1 110cc3310/0:2 >>>>>>> 3: ./00046 | 3037fc18/0:1 3037fb88/0:2 | 3: ./00046 | 110cc33f0/0:1 110cc3310/0:2 >>>>>>> assertion failed *** DIFFERENCE *** | init> 110cc3310 >>>>>>> init> 3037fb88 | 3: ./00046 | 110cc3310/0:3 >>>>>>> 3: ./00046 | 3037fc18/0:1 3037fb88/0:3 | 4: 3/00051 | 110cc3310/0:3 >>>>>>> 4: 3/00051 | 3037fc18/0:1 3037fb88/0:3 | assertion failed >>>>>>> assertion failed | init> 110cc3310 >>>>>>> init> 3037fb88 | 4: 3/00051 | 110cc33f0/0:3 110cc3310/0:4 >>>>>>> 4: 3/00051 | 3037fc18/0:3 3037fb88/0:4 | 5: /00000 | 110cc33f0/0:3 110cc3310/0:4 >>>>>>> 5: /00000 | 3037fc18/0:3 3037fb88/0:4 | init> 110cc3310 >>>>>>> found match 3037fd14 *** DIFFERENCE *** | match end offset = -1 >>>>>>> match end offset = 5 *** DIFFERENCE *** | tre_tnfa_run_parallel, input type 1 >>>>>>> tre_tnfa_run_parallel, input type 1 | length: -1 >>>>>>> length: -1 | pos:chr/code | states and tags >>>>>>> pos:chr/code | states and tags | -------------+------------------------------------------------ >>>>>>> -------------+------------------------------------------------ | init> 110cc4780 110cc4668 110cc4550 >>>>>>> init> 303811c0 3038110c 30381058 | match end offset = -1 >>>>>>> match end offset = -1 | tre_tnfa_run_parallel, input type 1 >>>>>>> tre_tnfa_run_parallel, input type 1 | length: -1 >>>>>>> length: -1 | pos:chr/code | states and tags >>>>>>> pos:chr/code | states and tags | -------------+------------------------------------------------ >>>>>>> -------------+------------------------------------------------ | init> 110cc5700 110cc55e8 110cc54d0 >>>>>>> >>>>>> One day further - looks like tre_compile (or just before, after all). >>>>>> >>>>>> With TRE_DEBUG switched on in tre-compile.c and tre-ast.c I see (snip) >>>>>> >>>>>> --- /tmp/x.32 2015-12-31 15:09:44.000000000 +0000 >>>>>> +++ /tmp/x.64 2015-12-31 15:09:30.000000000 +0000 >>>>>> @@ -1,5 +1,5 @@ >>>>>> - Script command is started on Thu Dec 31 15:04:39 2015. >>>>>> - root at x069:[/data/prj/cran/32/R-aix-3.2.3/src/library/tools]make sysdata >>>>>> + Script command is started on Thu Dec 31 15:08:43 2015. >>>>>> + root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]make sysdata >>>>>> installing 'sysdata.rda' >>>>>> echo "tools:::sysdata2LazyLoadDB(\"/data/prj/cran/R-3.2.3/src/library/tools/R/sysdata.rda\",\"../../../library/tools/R\")" | \ >>>>>> R_DEFAULT_PACKAGES=NULL LC_ALL=C ../../../bin/R --vanilla --slave >>>>>> @@ -167,7 +167,7 @@ >>>>>> initial: 1/1,0, assert 0 >>>>>> initial: 0/0, assert 0 >>>>>> initial: 0/0, assert 0 >>>>>> - final state 30370718 >>>>>> + final state 110cba530 >>>>>> tre_compile: parsing '(^|[^%])(%%)*%V' >>>>>> AST: >>>>>> catenation, sub 0, 0 tags >>>>>> @@ -177,7 +177,7 @@ >>>>>> assertions: bol >>>>>> union, sub -1, 0 tags >>>>>> literal (, $) (0, 36), pos 0, sub -1, 0 tags >>>>>> - literal (&, M-^?) (38, 65535), pos 0, sub -1, 0 tags >>>>>> + literal (&, M-^?) (38, -1), pos 0, sub -1, 0 tags >>>>>> iteration {0, -1}, sub -1, 0 tags, greedy >>>>>> catenation, sub 2, 0 tags >>>>>> literal (%, %) (37, 37), pos 1, sub -1, 0 tags >>>>>> @@ -197,7 +197,7 @@ >>>>>> Union >>>>>> Literal 0-36 >>>>>> After union left >>>>>> - Literal 38-65535 >>>>>> + Literal 38--1 >>>>>> After union right >>>>>> After union right >>>>>> num_tags += 2 >>>>>> @@ -231,7 +231,7 @@ >>>>>> assertions: bol >>>>>> union, sub -1, 0 tags >>>>>> literal (, $) (0, 36), pos 0, sub -1, 0 tags >>>>>> - literal (&, M-^?) (38, 65535), pos 0, sub -1, 0 tags >>>>>> + literal (&, M-^?) (38, -1), pos 0, sub -1, 0 tags >>>>>> iteration {0, -1}, sub -1, 2 tags, greedy >>>>>> catenation, sub 2, 1 tags >>>>>> literal (%, %) (37, 37), pos 1, sub -1, 1 tags >>>>>> @@ -255,7 +255,7 @@ >>>>>> Union >>>>>> Literal 0-36 >>>>>> After union left >>>>>> - Literal 38-65535 >>>>>> + Literal 38--1 >>>>>> After union right >>>>>> After union right >>>>>> tre_add_tag_right: tag 3 >>>>>> @@ -342,7 +342,7 @@ >>>>>> catenation, sub -1, 0 tags >>>>>> union, sub -1, 0 tags >>>>>> literal (, $) (0, 36), pos 0, sub -1, 0 tags >>>>>> - literal (&, M-^?) (38, 65535), pos 0, sub -1, 0 tags >>>>>> + literal (&, M-^?) (38, -1), pos 0, sub -1, 0 tags >>>>>> tag 4 >>>>>> >>>>>> It seems in 32-bit mode -1 is unsigned (65535) but -1 == -1 in 64-bit mode. >>>>>> >>>>>> I suspect I will "find it" - but a proposed change is appreciated. >>>>>> >>>>>> Happy New Year, >>>>>> Michael >>>>>> >>>>>> ______________________________________________ >>>>>> R-devel at r-project.org mailing list >>>>>> https://stat.ethz.ch/mailman/listinfo/r-devel >>>>>> >>>>> ______________________________________________ >>>>> R-devel at r-project.org mailing list >>>>> https://stat.ethz.ch/mailman/listinfo/r-devel >>>>> >>>> ______________________________________________ >>>> R-devel at r-project.org mailing list >>>> https://stat.ethz.ch/mailman/listinfo/r-devel
Simon Urbanek
2016-Jan-04 14:52 UTC
[Rd] R, AIX 64-bit builds - trying to understand root cause for message: "Error: Line starting 'Package: tools ...' is malformed!"
On Jan 4, 2016, at 1:34 AM, Michael Felt <aixtools at gmail.com> wrote:> I would be "pleased" if you would try packages - i.e., none of "Toolbox" or Perzl rpm's. >R requires iconv, so you have to get that somehow. I tried to bypass the check and build it anyway against the system one, but that fails as there is a iconv conversion further down the line that is not supported. Building iconv is a PITA because it conflicts with the system one so other tools break when you modify the load path. Perzl's iconv fixes that by incorporating both the system objects and the GNU ones into the same binary and that seems to work reliably so far. That is the only dependency I was using (his iconv doesn't have any dependencies itself).> The iconv I supply might not be working as is (I will need to install a new system to check). However, this is why I have been testing with R-3.2.3 - to side-step the system library dependencies. > > re: xz - I have "installp" version - and any of my packages should work side-by-side with the Toolbox/.rpm files - my PATH prefix is /opt (so, /opt/bin and /opt/sbin rather than /usr/bin, /usr/local/bin (etc.). >It fails even with the built-in one in R so it has nothing to do with the xz used itself.> I am testing on a Power6, usually with 2 to 4G RAM active. When I get the compile to finish I will up to 8G to test some data processing that fails with the 32-bit limitations. > > Another question: are you also getting lots of duplicate symbol warnings? Curious me. If you are I would like to send my modifications to configure(.ac) for you to test (I put .ac in () because I am actually editting configure atm but shall put them into configure.ac when I finish) >No, no duplicate warnings.> FYI: I shall be downloading the "try and buy" xlc and xlfortran - and I think you will certainly prefer my packaging then as they work without the libgc dependencies that many of the rpm packages need. > > And, at your option - we can take this discussion over tools - "out of forums". Or at least start a new thread. >We could leave the list out and create a Wiki or something with the results of our tests.> re: unsigned short - I have adopted the convention to use the *NN_t types after running into a problem with unsigned longlong (from a very old program). It goes back many years - and maybe they have finalized short to mean 16-bits - but I remember when short was meant to help when moving from 16-bit to 32-bit and the "word" size changed - i.e., int became 32-bit same as long. nd for a long (no pun intended) long was 32-bit and long long was 64-bit. Those definitions are extinct. >I'm not sure what you refer here - the issue with TRE has nothing to do with short - it can take any int type and like I said most platforms use unsigned int which is big enough on all platforms.> imho - the standard for wint_t is wrong as well - based on an assumption about how "short" is defined. And, I consider it poor practice that there are som many cases of type cast switches between ushort and int. >Not really - it doesn't care about short at all - note that the short typedef is never actually used on AIX since it has wchar support so TRE is only using int. Cheers, Simon> Michael > > On 04-Jan-16 01:26, Simon Urbanek wrote: >> Michael, >> >> I'm using xlc + xlf - the exact flags are >> >> configure CC=xlc_r CXX=xlc++_r F77=xlf_r FC=xlf95_r LIBS='-L/opt/freeware/lib /opt/freeware/lib/libiconv.a -lpthread' --prefix=/opt/freeware CPPFLAGS=-I/opt/freeware/include >> >> with export OBJECT_MODE=64 in the env (and libiconv from perzl.org RPMs - iconv is a serious pain). >> >> I have changed the TRE typedef from "wint_t" to "unsigned int" since that is what the other platforms use anyway. >> >> On my AIX7 VM I'm running out of memory in xz when lazy-loading is enabled which his odd - the VM has 4Gb which one would think should be enough. The symptom is that installing MASS fails when creating lazy-load rds or the same happens when running make check as memCompress() test for xz fails. I wish there was a way to disable xz altogether.. >> >> I'd like to get IBM compilers to work first since those are the official ones. I may try gcc later, but I'm really interested in the IBM ones because that gives us a good non-GNU test (e.g., TRE fails in JIT due some alleged GNUisms so you can't use the strict compiler mode). >> >> Cheers, >> Simon >> >> >> On Jan 3, 2016, at 10:59 AM, Michael Felt <aixtools at gmail.com> wrote: >> >>> On 2016-01-01 23:48, peter dalgaard wrote: >>>> Nice catch you two!!! >>>> >>>> Happy New Year >>>> -pd >>> I am much happier with this great start! >>> >>> Simon - which compiler)s) did you use: xlc and xlfortran, or gcc/gfortran? >>> >>> I have made some changes to configure(.ac) so maybe my problems are self-inflicted. But would be good to know what environment you are using. >>> >>> Thanks for looking - and finding!!! >>> >>> Michael >>>>> On 01 Jan 2016, at 22:06 , Simon Urbanek<simon.urbanek at r-project.org> wrote: >>>>> >>>>> Ok, found the problem - on platforms that support it TRE uses wint_t (from wchar.h) as its type for characters (tre_cint_t) which on AIX is *signed* int. TRE uses liberally conversions between int and tre_cint_t apparently assuming that the latter is unsigned so conversions back to int are suitable for comparisons etc. On other platforms wint_t is unsigned so it works. Manually defining tre_cint_t to unsigned int fixes the issue. >>>>> >>>>> Cheers, >>>>> Simon >>>>> >>>>> >>>>> On Jan 1, 2016, at 12:20 PM, Simon Urbanek<simon.urbanek at r-project.org> wrote: >>>>> >>>>>> Michael, >>>>>> >>>>>> thanks, I'll have a look once my PDP VMs are up again (later today). This may be a signedness issue although it's unclear why other platforms wouldn't be affected. >>>>>> >>>>>> Cheers, >>>>>> Simon >>>>>> >>>>>> >>>>>> On Dec 31, 2015, at 10:14 AM, Michael Felt<aixtools at gmail.com> wrote: >>>>>> >>>>>>> On 2015-12-30 09:58, Michael Felt wrote: >>>>>>>> On 2015-12-29 11:02, Michael Felt wrote: >>>>>>>>> This seems to be a problem that goes back a long time - and I hope someone who understands what tre is suppossed to be doing will look at this. >>>>>>>>> >>>>>>>>> A short history of other people who have reported on this on different versions of AIX. I shall only add that I get the same results on AIX 5.3 TL7, AIX 6.1 TL9 and AIX 7.1 TL3. >>>>>>>>> >>>>>>>>> Basically, with settings that work for AIX and 32-bit - the only changes being >>>>>>>>> -maix32 becomes -maix64 >>>>>>>>> and >>>>>>>>> export OBJECT_MODE=32 becomes export OBJECT_MODE=64 >>>>>>>>> >>>>>>>>> Then to shorten the 'make' bla bla, first run just make, then >>>>>>>>> >>>>>>>>> cd src/library/tools >>>>>>>>> make -s sysdata >>>>>>>>> >>>>>>>>> http://article.gmane.org/gmane.comp.lang.r.devel/38817/match=package+tools+malformed >>>>>>>>> http://article.gmane.org/gmane.comp.lang.r.devel/36886/match=package+tools+malformed >>>>>>>>> http://article.gmane.org/gmane.comp.lang.r.devel/23372/match=package+tools+malformed Date: 2010-01-25 06:55:41 GMT (5 years, 48 weeks, 1 day, 20 hours and 30 minutes ago) >>>>>>>>> >>>>>>>>> To that, to get debug data, I have >>>>>>>>> >>>>>>>>> * added -DTRE_DUGUG to src/extra/tre/Makefile # ALL_CFLAGS = $(ALL_CFLAGS_LO) -DTRE_DEBUG >>>>>>>>> * rm src/extra/tre/tre-match-parallel.o >>>>>>>>> * find . -name \*.so -exec rm {} \; >>>>>>>>> * make >>>>>>>>> * cd src/library/tools >>>>>>>>> * make -s sysdata >>>>>>>>> >>>>>>>>> Attached are the two script files of the screen output. The 32-bit one is more verbose - and contains magically lines such as: >>>>>>>>> found match 3037fd14 (while "found" does not occur in the 64-bit output) >>>>>>>>> >>>>>>>>> root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]wc /tmp/sysdata.??.* >>>>>>>>> 4730 14123 139916 /tmp/sysdata.32.text >>>>>>>>> 1312 3688 40528 /tmp/sysdata.64.text >>>>>>>>> 6042 17811 180444 total >>>>>>>>> >>>>>>>>> root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]grep -c found /tmp/sysdata.??.* >>>>>>>>> /tmp/sysdata.32.text:19 >>>>>>>>> /tmp/sysdata.64.text:0 >>>>>>>>> >>>>>>>>> >>>>>>>>> Hope this brings us (or me), closer to a resolution to an old concern. >>>>>>>>> >>>>>>>>> And, best wishes for the new year! >>>>>>>>> >>>>>>>>> Michael >>>>>>>>> >>>>>>>>> >>>>>>>> Still hoping for someones curiosity/willingness. >>>>>>>> >>>>>>>> The differences show up in the first comparision that is made (of the string "3.2.3" it seems) - 32-bit is on the left, 64-bit on the right. >>>>>>>> >>>>>>>> Script command is started on Tue Dec 29 08:39:16 UTC 2015. | Script command is started on Tue Dec 29 08:39:56 UTC 2015. >>>>>>>> root at x069:[/data/prj/cran/32/R-aix-3.2.3/src/library/tools]make -s sysdata | root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]make -s sysdata >>>>>>>> installing 'sysdata.rda' | installing 'sysdata.rda' >>>>>>>> tre_tnfa_run_parallel, input type 1 | tre_tnfa_run_parallel, input type 1 >>>>>>>> length: -1 | length: -1 >>>>>>>> pos:chr/code | states and tags | pos:chr/code | states and tags >>>>>>>> -------------+------------------------------------------------ | -------------+------------------------------------------------ >>>>>>>> init> 30380200 3038014c 30380098 | init> 110cc3040 110cc2f28 110cc2e10 >>>>>>>> match end offset = -1 | match end offset = -1 >>>>>>>> tre_tnfa_run_parallel, input type 1 | tre_tnfa_run_parallel, input type 1 >>>>>>>> length: -1 | length: -1 >>>>>>>> pos:chr/code | states and tags | pos:chr/code | states and tags >>>>>>>> -------------+------------------------------------------------ | -------------+------------------------------------------------ >>>>>>>> init> 3037fb88 | init> 110cc3310 >>>>>>>> 0: 3/00051 | 3037fb88/0:0 | 0: 3/00051 | 110cc3310/0:0 >>>>>>>> 1: ./00046 | 3037fb88/0:0 | 1: ./00046 | 110cc3310/0:0 >>>>>>>> init> 3037fb88 | init> 110cc3310 >>>>>>>> 1: ./00046 | 3037fb88/0:1 | 1: ./00046 | 110cc3310/0:1 >>>>>>>> 2: 2/00050 | 3037fb88/0:1 | 2: 2/00050 | 110cc3310/0:1 >>>>>>>> assertion failed | assertion failed >>>>>>>> init> 3037fb88 | init> 110cc3310 >>>>>>>> 2: 2/00050 | 3037fc18/0:1 3037fb88/0:2 | 2: 2/00050 | 110cc33f0/0:1 110cc3310/0:2 >>>>>>>> 3: ./00046 | 3037fc18/0:1 3037fb88/0:2 | 3: ./00046 | 110cc33f0/0:1 110cc3310/0:2 >>>>>>>> assertion failed *** DIFFERENCE *** | init> 110cc3310 >>>>>>>> init> 3037fb88 | 3: ./00046 | 110cc3310/0:3 >>>>>>>> 3: ./00046 | 3037fc18/0:1 3037fb88/0:3 | 4: 3/00051 | 110cc3310/0:3 >>>>>>>> 4: 3/00051 | 3037fc18/0:1 3037fb88/0:3 | assertion failed >>>>>>>> assertion failed | init> 110cc3310 >>>>>>>> init> 3037fb88 | 4: 3/00051 | 110cc33f0/0:3 110cc3310/0:4 >>>>>>>> 4: 3/00051 | 3037fc18/0:3 3037fb88/0:4 | 5: /00000 | 110cc33f0/0:3 110cc3310/0:4 >>>>>>>> 5: /00000 | 3037fc18/0:3 3037fb88/0:4 | init> 110cc3310 >>>>>>>> found match 3037fd14 *** DIFFERENCE *** | match end offset = -1 >>>>>>>> match end offset = 5 *** DIFFERENCE *** | tre_tnfa_run_parallel, input type 1 >>>>>>>> tre_tnfa_run_parallel, input type 1 | length: -1 >>>>>>>> length: -1 | pos:chr/code | states and tags >>>>>>>> pos:chr/code | states and tags | -------------+------------------------------------------------ >>>>>>>> -------------+------------------------------------------------ | init> 110cc4780 110cc4668 110cc4550 >>>>>>>> init> 303811c0 3038110c 30381058 | match end offset = -1 >>>>>>>> match end offset = -1 | tre_tnfa_run_parallel, input type 1 >>>>>>>> tre_tnfa_run_parallel, input type 1 | length: -1 >>>>>>>> length: -1 | pos:chr/code | states and tags >>>>>>>> pos:chr/code | states and tags | -------------+------------------------------------------------ >>>>>>>> -------------+------------------------------------------------ | init> 110cc5700 110cc55e8 110cc54d0 >>>>>>>> >>>>>>> One day further - looks like tre_compile (or just before, after all). >>>>>>> >>>>>>> With TRE_DEBUG switched on in tre-compile.c and tre-ast.c I see (snip) >>>>>>> >>>>>>> --- /tmp/x.32 2015-12-31 15:09:44.000000000 +0000 >>>>>>> +++ /tmp/x.64 2015-12-31 15:09:30.000000000 +0000 >>>>>>> @@ -1,5 +1,5 @@ >>>>>>> - Script command is started on Thu Dec 31 15:04:39 2015. >>>>>>> - root at x069:[/data/prj/cran/32/R-aix-3.2.3/src/library/tools]make sysdata >>>>>>> + Script command is started on Thu Dec 31 15:08:43 2015. >>>>>>> + root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]make sysdata >>>>>>> installing 'sysdata.rda' >>>>>>> echo "tools:::sysdata2LazyLoadDB(\"/data/prj/cran/R-3.2.3/src/library/tools/R/sysdata.rda\",\"../../../library/tools/R\")" | \ >>>>>>> R_DEFAULT_PACKAGES=NULL LC_ALL=C ../../../bin/R --vanilla --slave >>>>>>> @@ -167,7 +167,7 @@ >>>>>>> initial: 1/1,0, assert 0 >>>>>>> initial: 0/0, assert 0 >>>>>>> initial: 0/0, assert 0 >>>>>>> - final state 30370718 >>>>>>> + final state 110cba530 >>>>>>> tre_compile: parsing '(^|[^%])(%%)*%V' >>>>>>> AST: >>>>>>> catenation, sub 0, 0 tags >>>>>>> @@ -177,7 +177,7 @@ >>>>>>> assertions: bol >>>>>>> union, sub -1, 0 tags >>>>>>> literal (, $) (0, 36), pos 0, sub -1, 0 tags >>>>>>> - literal (&, M-^?) (38, 65535), pos 0, sub -1, 0 tags >>>>>>> + literal (&, M-^?) (38, -1), pos 0, sub -1, 0 tags >>>>>>> iteration {0, -1}, sub -1, 0 tags, greedy >>>>>>> catenation, sub 2, 0 tags >>>>>>> literal (%, %) (37, 37), pos 1, sub -1, 0 tags >>>>>>> @@ -197,7 +197,7 @@ >>>>>>> Union >>>>>>> Literal 0-36 >>>>>>> After union left >>>>>>> - Literal 38-65535 >>>>>>> + Literal 38--1 >>>>>>> After union right >>>>>>> After union right >>>>>>> num_tags += 2 >>>>>>> @@ -231,7 +231,7 @@ >>>>>>> assertions: bol >>>>>>> union, sub -1, 0 tags >>>>>>> literal (, $) (0, 36), pos 0, sub -1, 0 tags >>>>>>> - literal (&, M-^?) (38, 65535), pos 0, sub -1, 0 tags >>>>>>> + literal (&, M-^?) (38, -1), pos 0, sub -1, 0 tags >>>>>>> iteration {0, -1}, sub -1, 2 tags, greedy >>>>>>> catenation, sub 2, 1 tags >>>>>>> literal (%, %) (37, 37), pos 1, sub -1, 1 tags >>>>>>> @@ -255,7 +255,7 @@ >>>>>>> Union >>>>>>> Literal 0-36 >>>>>>> After union left >>>>>>> - Literal 38-65535 >>>>>>> + Literal 38--1 >>>>>>> After union right >>>>>>> After union right >>>>>>> tre_add_tag_right: tag 3 >>>>>>> @@ -342,7 +342,7 @@ >>>>>>> catenation, sub -1, 0 tags >>>>>>> union, sub -1, 0 tags >>>>>>> literal (, $) (0, 36), pos 0, sub -1, 0 tags >>>>>>> - literal (&, M-^?) (38, 65535), pos 0, sub -1, 0 tags >>>>>>> + literal (&, M-^?) (38, -1), pos 0, sub -1, 0 tags >>>>>>> tag 4 >>>>>>> >>>>>>> It seems in 32-bit mode -1 is unsigned (65535) but -1 == -1 in 64-bit mode. >>>>>>> >>>>>>> I suspect I will "find it" - but a proposed change is appreciated. >>>>>>> >>>>>>> Happy New Year, >>>>>>> Michael >>>>>>> >>>>>>> ______________________________________________ >>>>>>> R-devel at r-project.org mailing list >>>>>>> https://stat.ethz.ch/mailman/listinfo/r-devel >>>>>>> >>>>>> ______________________________________________ >>>>>> R-devel at r-project.org mailing list >>>>>> https://stat.ethz.ch/mailman/listinfo/r-devel >>>>>> >>>>> ______________________________________________ >>>>> R-devel at r-project.org mailing list >>>>> https://stat.ethz.ch/mailman/listinfo/r-devel >
Michael Felt
2016-Jan-04 20:47 UTC
[Rd] R, AIX 64-bit builds - trying to understand root cause for message: "Error: Line starting 'Package: tools ...' is malformed!"
On 04-Jan-16 15:52, Simon Urbanek wrote:> On Jan 4, 2016, at 1:34 AM, Michael Felt <aixtools at gmail.com> wrote: > >> I would be "pleased" if you would try packages - i.e., none of "Toolbox" or Perzl rpm's. >> > R requires iconv, so you have to get that somehow. I tried to bypass the check and build it anyway against the system one, but that fails as there is a iconv conversion further down the line that is not supported. Building iconv is a PITA because it conflicts with the system one so other tools break when you modify the load path. Perzl's iconv fixes that by incorporating both the system objects and the GNU ones into the same binary and that seems to work reliably so far. That is the only dependency I was using (his iconv doesn't have any dependencies itself).I would have to install another system and install perzl iconv - however, I assume that it also sits in /usr/lib (and/or has a symbolic link to it). I have mine installed in /opt/lib - so the default shared libraries look like: cran at x065:[/home/cran/64/R]ldd /home/cran/64/R/bin/exec/R /home/cran/64/R/bin/exec/R needs: /usr/lib/libc.a(shr_64.o) /usr/lib/libpthread.a(shr_xpg5_64.o) /opt/lib/libintl.a(libintl.so.8) /opt/lib/libiconv.a(libiconv.so.2) /unix /usr/lib/libcrypt.a(shr_64.o) /usr/lib/libpthread.a(shr_xpg5.o) /usr/lib/libc.a(shr.o) /usr/lib/libpthreads.a(shr_comm.o) /usr/lib/libcrypt.a(shr.o) I have tested with adding my (gnu iconv) to the /usr/lib archive, as the search does seem to get corrupted. For testing purposes I have added the 64-bit member to my 32-bit packaging - rather than a seperate archive in /opt/lib64 cran at x065:[/home/cran/64/R]ar -X32_64 -tv /opt/lib/libiconv.a rwxr-xr-x 0/0 1010404 Nov 27 17:28 2014 libiconv.so.2 rwxr-xr-x 0/0 111901 Nov 27 17:28 2014 shr4.o rwxr-xr-x 0/0 112151 Nov 27 17:28 2014 shr.o rwxr-xr-x 0/0 1035470 Dec 3 15:11 2015 libiconv.so.2 I did something similiar to this for all the dependencies earlier when I was working with R-devel branch. When the build is finishing I will workout how to install both into a single archive (the hard part is a clean uninstall afterwards, and harder is managing an update. So, ideally, I will go to having one package with both 32 and 64 bit members.)> >> The iconv I supply might not be working as is (I will need to install a new system to check). However, this is why I have been testing with R-3.2.3 - to side-step the system library dependencies. >> >> re: xz - I have "installp" version - and any of my packages should work side-by-side with the Toolbox/.rpm files - my PATH prefix is /opt (so, /opt/bin and /opt/sbin rather than /usr/bin, /usr/local/bin (etc.). >> > It fails even with the built-in one in R so it has nothing to do with the xz used itself.What is the built-in level - do not know off the top of my head. Perzl is 5.0.7 and I have been using 5.0.8 (when working with R-devel).> > >> I am testing on a Power6, usually with 2 to 4G RAM active. When I get the compile to finish I will up to 8G to test some data processing that fails with the 32-bit limitations. >> >> Another question: are you also getting lots of duplicate symbol warnings? Curious me. If you are I would like to send my modifications to configure(.ac) for you to test (I put .ac in () because I am actually editting configure atm but shall put them into configure.ac when I finish) >> > No, no duplicate warnings.Then I suspect the duplicate warnings have to do with how the m4 macros are processing the gcc -v flags (actually, the gfortran -v flags). Adding full path to libgcc.a and including -bexpall causes symbols to be imported and reexported - hence duplicate. Another reason for me to get xlfortran installed and "test" against that as well.> > >> FYI: I shall be downloading the "try and buy" xlc and xlfortran - and I think you will certainly prefer my packaging then as they work without the libgc dependencies that many of the rpm packages need. >> >> And, at your option - we can take this discussion over tools - "out of forums". Or at least start a new thread. >> > We could leave the list out and create a Wiki or something with the results of our tests.I would be happy to sponsor that - either on my forums (http://forums.rootvg.net/aixtools/r-for-aix/) or I can look into how I create a userid for you on my aixtools wiki. I started something in November http://www.aixtools.net/index.php/Projects#%22Work%20in%20Progress%22, but basically, am waiting for better results before dedicating a page to R. In short, I would prefer the forums - as anyone could join the discussion without any intervention from me.> > >> re: unsigned short - I have adopted the convention to use the *NN_t types after running into a problem with unsigned longlong (from a very old program). It goes back many years - and maybe they have finalized short to mean 16-bits - but I remember when short was meant to help when moving from 16-bit to 32-bit and the "word" size changed - i.e., int became 32-bit same as long. nd for a long (no pun intended) long was 32-bit and long long was 64-bit. Those definitions are extinct. >> > I'm not sure what you refer here - the issue with TRE has nothing to do with short - it can take any int type and like I said most platforms use unsigned int which is big enough on all platforms.I claim no expertise. My observation is that UTF-XX seems to be 16-bit. I say seems, because there may be a multi-byte requirement that does not fit in 16-bits and then a larger size is appropriate. From my perspective (as a porter) it is easier to identify inconsistency when typedef definitions are not directly related to short, int or long. In binary it may not make any difference as far as sizeof() is concerned. I just hope for improved legibility. I am not "r-project" - and my assumption is that "r-project" does the commits. My contribution is merely my opinion. What you decide is what counts - not my opinion.> >> imho - the standard for wint_t is wrong as well - based on an assumption about how "short" is defined. And, I consider it poor practice that there are som many cases of type cast switches between ushort and int. >> > Not really - it doesn't care about short at all - note that the short typedef is never actually used on AIX since it has wchar support so TRE is only using int.You lost me here. I am aware that AIX has a very different set of include files that tre is not using. My perspective is that R is ignoring/overrulling any IBM/AIX wchar processing. See my "not an expert" defense above. Mainly - I am happy that "make sysdata" is working. p.s. I shall make a new entry on my forums re: where I am at now re: Warning in solve.default(rgb) : unable to load shared object '/home/cran/64/R/modules//lapack.so': rtld: 0712-001 Symbol __powidf2 was referenced from module /home/cran/64/R/lib/libRlapack.so(), but a runtime definition of the symbol was not found. rtld: 0712-001 Symbol _gfortran_pow_i4_i4 was referenced from module /home/cran/64/R/lib/libRlapack.so(), but a runtime definition of the symbol was not found. rtld: 0712-001 Symbol _gfortran_compare_string/home/cran/64/R/lib/libRlapack.so was referenced from module /home/cran/64/R/lib/libRlapack.so(), but a runtime definition of the symbol was not found. rtld: 0712-001 Symbol _gfortran_concat_string was referenced from module /home/cran/64/R/lib/libRlapack.so(), but a runtime definition of the symbol was not found. rtld: 0712-002 fatal error: exiting. Error in solve.default(rgb) : LAPACK routines cannot be loaded Error: unable to load R code in package 'grDevices'> > Cheers, > Simon > > >> Michael >> >> On 04-Jan-16 01:26, Simon Urbanek wrote: >>> Michael, >>> >>> I'm using xlc + xlf - the exact flags are >>> >>> configure CC=xlc_r CXX=xlc++_r F77=xlf_r FC=xlf95_r LIBS='-L/opt/freeware/lib /opt/freeware/lib/libiconv.a -lpthread' --prefix=/opt/freeware CPPFLAGS=-I/opt/freeware/include >>> >>> with export OBJECT_MODE=64 in the env (and libiconv from perzl.org RPMs - iconv is a serious pain). >>> >>> I have changed the TRE typedef from "wint_t" to "unsigned int" since that is what the other platforms use anyway. >>> >>> On my AIX7 VM I'm running out of memory in xz when lazy-loading is enabled which his odd - the VM has 4Gb which one would think should be enough. The symptom is that installing MASS fails when creating lazy-load rds or the same happens when running make check as memCompress() test for xz fails. I wish there was a way to disable xz altogether.. >>> >>> I'd like to get IBM compilers to work first since those are the official ones. I may try gcc later, but I'm really interested in the IBM ones because that gives us a good non-GNU test (e.g., TRE fails in JIT due some alleged GNUisms so you can't use the strict compiler mode). >>> >>> Cheers, >>> Simon >>> >>> >>> On Jan 3, 2016, at 10:59 AM, Michael Felt <aixtools at gmail.com> wrote: >>> >>>> On 2016-01-01 23:48, peter dalgaard wrote: >>>>> Nice catch you two!!! >>>>> >>>>> Happy New Year >>>>> -pd >>>> I am much happier with this great start! >>>> >>>> Simon - which compiler)s) did you use: xlc and xlfortran, or gcc/gfortran? >>>> >>>> I have made some changes to configure(.ac) so maybe my problems are self-inflicted. But would be good to know what environment you are using. >>>> >>>> Thanks for looking - and finding!!! >>>> >>>> Michael >>>>>> On 01 Jan 2016, at 22:06 , Simon Urbanek<simon.urbanek at r-project.org> wrote: >>>>>> >>>>>> Ok, found the problem - on platforms that support it TRE uses wint_t (from wchar.h) as its type for characters (tre_cint_t) which on AIX is *signed* int. TRE uses liberally conversions between int and tre_cint_t apparently assuming that the latter is unsigned so conversions back to int are suitable for comparisons etc. On other platforms wint_t is unsigned so it works. Manually defining tre_cint_t to unsigned int fixes the issue. >>>>>> >>>>>> Cheers, >>>>>> Simon >>>>>> >>>>>> >>>>>> On Jan 1, 2016, at 12:20 PM, Simon Urbanek<simon.urbanek at r-project.org> wrote: >>>>>> >>>>>>> Michael, >>>>>>> >>>>>>> thanks, I'll have a look once my PDP VMs are up again (later today). This may be a signedness issue although it's unclear why other platforms wouldn't be affected. >>>>>>> >>>>>>> Cheers, >>>>>>> Simon >>>>>>> >>>>>>> >>>>>>> On Dec 31, 2015, at 10:14 AM, Michael Felt<aixtools at gmail.com> wrote: >>>>>>> >>>>>>>> On 2015-12-30 09:58, Michael Felt wrote: >>>>>>>>> On 2015-12-29 11:02, Michael Felt wrote: >>>>>>>>>> This seems to be a problem that goes back a long time - and I hope someone who understands what tre is suppossed to be doing will look at this. >>>>>>>>>> >>>>>>>>>> A short history of other people who have reported on this on different versions of AIX. I shall only add that I get the same results on AIX 5.3 TL7, AIX 6.1 TL9 and AIX 7.1 TL3. >>>>>>>>>> >>>>>>>>>> Basically, with settings that work for AIX and 32-bit - the only changes being >>>>>>>>>> -maix32 becomes -maix64 >>>>>>>>>> and >>>>>>>>>> export OBJECT_MODE=32 becomes export OBJECT_MODE=64 >>>>>>>>>> >>>>>>>>>> Then to shorten the 'make' bla bla, first run just make, then >>>>>>>>>> >>>>>>>>>> cd src/library/tools >>>>>>>>>> make -s sysdata >>>>>>>>>> >>>>>>>>>> http://article.gmane.org/gmane.comp.lang.r.devel/38817/match=package+tools+malformed >>>>>>>>>> http://article.gmane.org/gmane.comp.lang.r.devel/36886/match=package+tools+malformed >>>>>>>>>> http://article.gmane.org/gmane.comp.lang.r.devel/23372/match=package+tools+malformed Date: 2010-01-25 06:55:41 GMT (5 years, 48 weeks, 1 day, 20 hours and 30 minutes ago) >>>>>>>>>> >>>>>>>>>> To that, to get debug data, I have >>>>>>>>>> >>>>>>>>>> * added -DTRE_DUGUG to src/extra/tre/Makefile # ALL_CFLAGS = $(ALL_CFLAGS_LO) -DTRE_DEBUG >>>>>>>>>> * rm src/extra/tre/tre-match-parallel.o >>>>>>>>>> * find . -name \*.so -exec rm {} \; >>>>>>>>>> * make >>>>>>>>>> * cd src/library/tools >>>>>>>>>> * make -s sysdata >>>>>>>>>> >>>>>>>>>> Attached are the two script files of the screen output. The 32-bit one is more verbose - and contains magically lines such as: >>>>>>>>>> found match 3037fd14 (while "found" does not occur in the 64-bit output) >>>>>>>>>> >>>>>>>>>> root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]wc /tmp/sysdata.??.* >>>>>>>>>> 4730 14123 139916 /tmp/sysdata.32.text >>>>>>>>>> 1312 3688 40528 /tmp/sysdata.64.text >>>>>>>>>> 6042 17811 180444 total >>>>>>>>>> >>>>>>>>>> root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]grep -c found /tmp/sysdata.??.* >>>>>>>>>> /tmp/sysdata.32.text:19 >>>>>>>>>> /tmp/sysdata.64.text:0 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Hope this brings us (or me), closer to a resolution to an old concern. >>>>>>>>>> >>>>>>>>>> And, best wishes for the new year! >>>>>>>>>> >>>>>>>>>> Michael >>>>>>>>>> >>>>>>>>>> >>>>>>>>> Still hoping for someones curiosity/willingness. >>>>>>>>> >>>>>>>>> The differences show up in the first comparision that is made (of the string "3.2.3" it seems) - 32-bit is on the left, 64-bit on the right. >>>>>>>>> >>>>>>>>> Script command is started on Tue Dec 29 08:39:16 UTC 2015. | Script command is started on Tue Dec 29 08:39:56 UTC 2015. >>>>>>>>> root at x069:[/data/prj/cran/32/R-aix-3.2.3/src/library/tools]make -s sysdata | root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]make -s sysdata >>>>>>>>> installing 'sysdata.rda' | installing 'sysdata.rda' >>>>>>>>> tre_tnfa_run_parallel, input type 1 | tre_tnfa_run_parallel, input type 1 >>>>>>>>> length: -1 | length: -1 >>>>>>>>> pos:chr/code | states and tags | pos:chr/code | states and tags >>>>>>>>> -------------+------------------------------------------------ | -------------+------------------------------------------------ >>>>>>>>> init> 30380200 3038014c 30380098 | init> 110cc3040 110cc2f28 110cc2e10 >>>>>>>>> match end offset = -1 | match end offset = -1 >>>>>>>>> tre_tnfa_run_parallel, input type 1 | tre_tnfa_run_parallel, input type 1 >>>>>>>>> length: -1 | length: -1 >>>>>>>>> pos:chr/code | states and tags | pos:chr/code | states and tags >>>>>>>>> -------------+------------------------------------------------ | -------------+------------------------------------------------ >>>>>>>>> init> 3037fb88 | init> 110cc3310 >>>>>>>>> 0: 3/00051 | 3037fb88/0:0 | 0: 3/00051 | 110cc3310/0:0 >>>>>>>>> 1: ./00046 | 3037fb88/0:0 | 1: ./00046 | 110cc3310/0:0 >>>>>>>>> init> 3037fb88 | init> 110cc3310 >>>>>>>>> 1: ./00046 | 3037fb88/0:1 | 1: ./00046 | 110cc3310/0:1 >>>>>>>>> 2: 2/00050 | 3037fb88/0:1 | 2: 2/00050 | 110cc3310/0:1 >>>>>>>>> assertion failed | assertion failed >>>>>>>>> init> 3037fb88 | init> 110cc3310 >>>>>>>>> 2: 2/00050 | 3037fc18/0:1 3037fb88/0:2 | 2: 2/00050 | 110cc33f0/0:1 110cc3310/0:2 >>>>>>>>> 3: ./00046 | 3037fc18/0:1 3037fb88/0:2 | 3: ./00046 | 110cc33f0/0:1 110cc3310/0:2 >>>>>>>>> assertion failed *** DIFFERENCE *** | init> 110cc3310 >>>>>>>>> init> 3037fb88 | 3: ./00046 | 110cc3310/0:3 >>>>>>>>> 3: ./00046 | 3037fc18/0:1 3037fb88/0:3 | 4: 3/00051 | 110cc3310/0:3 >>>>>>>>> 4: 3/00051 | 3037fc18/0:1 3037fb88/0:3 | assertion failed >>>>>>>>> assertion failed | init> 110cc3310 >>>>>>>>> init> 3037fb88 | 4: 3/00051 | 110cc33f0/0:3 110cc3310/0:4 >>>>>>>>> 4: 3/00051 | 3037fc18/0:3 3037fb88/0:4 | 5: /00000 | 110cc33f0/0:3 110cc3310/0:4 >>>>>>>>> 5: /00000 | 3037fc18/0:3 3037fb88/0:4 | init> 110cc3310 >>>>>>>>> found match 3037fd14 *** DIFFERENCE *** | match end offset = -1 >>>>>>>>> match end offset = 5 *** DIFFERENCE *** | tre_tnfa_run_parallel, input type 1 >>>>>>>>> tre_tnfa_run_parallel, input type 1 | length: -1 >>>>>>>>> length: -1 | pos:chr/code | states and tags >>>>>>>>> pos:chr/code | states and tags | -------------+------------------------------------------------ >>>>>>>>> -------------+------------------------------------------------ | init> 110cc4780 110cc4668 110cc4550 >>>>>>>>> init> 303811c0 3038110c 30381058 | match end offset = -1 >>>>>>>>> match end offset = -1 | tre_tnfa_run_parallel, input type 1 >>>>>>>>> tre_tnfa_run_parallel, input type 1 | length: -1 >>>>>>>>> length: -1 | pos:chr/code | states and tags >>>>>>>>> pos:chr/code | states and tags | -------------+------------------------------------------------ >>>>>>>>> -------------+------------------------------------------------ | init> 110cc5700 110cc55e8 110cc54d0 >>>>>>>>> >>>>>>>> One day further - looks like tre_compile (or just before, after all). >>>>>>>> >>>>>>>> With TRE_DEBUG switched on in tre-compile.c and tre-ast.c I see (snip) >>>>>>>> >>>>>>>> --- /tmp/x.32 2015-12-31 15:09:44.000000000 +0000 >>>>>>>> +++ /tmp/x.64 2015-12-31 15:09:30.000000000 +0000 >>>>>>>> @@ -1,5 +1,5 @@ >>>>>>>> - Script command is started on Thu Dec 31 15:04:39 2015. >>>>>>>> - root at x069:[/data/prj/cran/32/R-aix-3.2.3/src/library/tools]make sysdata >>>>>>>> + Script command is started on Thu Dec 31 15:08:43 2015. >>>>>>>> + root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]make sysdata >>>>>>>> installing 'sysdata.rda' >>>>>>>> echo "tools:::sysdata2LazyLoadDB(\"/data/prj/cran/R-3.2.3/src/library/tools/R/sysdata.rda\",\"../../../library/tools/R\")" | \ >>>>>>>> R_DEFAULT_PACKAGES=NULL LC_ALL=C ../../../bin/R --vanilla --slave >>>>>>>> @@ -167,7 +167,7 @@ >>>>>>>> initial: 1/1,0, assert 0 >>>>>>>> initial: 0/0, assert 0 >>>>>>>> initial: 0/0, assert 0 >>>>>>>> - final state 30370718 >>>>>>>> + final state 110cba530 >>>>>>>> tre_compile: parsing '(^|[^%])(%%)*%V' >>>>>>>> AST: >>>>>>>> catenation, sub 0, 0 tags >>>>>>>> @@ -177,7 +177,7 @@ >>>>>>>> assertions: bol >>>>>>>> union, sub -1, 0 tags >>>>>>>> literal (, $) (0, 36), pos 0, sub -1, 0 tags >>>>>>>> - literal (&, M-^?) (38, 65535), pos 0, sub -1, 0 tags >>>>>>>> + literal (&, M-^?) (38, -1), pos 0, sub -1, 0 tags >>>>>>>> iteration {0, -1}, sub -1, 0 tags, greedy >>>>>>>> catenation, sub 2, 0 tags >>>>>>>> literal (%, %) (37, 37), pos 1, sub -1, 0 tags >>>>>>>> @@ -197,7 +197,7 @@ >>>>>>>> Union >>>>>>>> Literal 0-36 >>>>>>>> After union left >>>>>>>> - Literal 38-65535 >>>>>>>> + Literal 38--1 >>>>>>>> After union right >>>>>>>> After union right >>>>>>>> num_tags += 2 >>>>>>>> @@ -231,7 +231,7 @@ >>>>>>>> assertions: bol >>>>>>>> union, sub -1, 0 tags >>>>>>>> literal (, $) (0, 36), pos 0, sub -1, 0 tags >>>>>>>> - literal (&, M-^?) (38, 65535), pos 0, sub -1, 0 tags >>>>>>>> + literal (&, M-^?) (38, -1), pos 0, sub -1, 0 tags >>>>>>>> iteration {0, -1}, sub -1, 2 tags, greedy >>>>>>>> catenation, sub 2, 1 tags >>>>>>>> literal (%, %) (37, 37), pos 1, sub -1, 1 tags >>>>>>>> @@ -255,7 +255,7 @@ >>>>>>>> Union >>>>>>>> Literal 0-36 >>>>>>>> After union left >>>>>>>> - Literal 38-65535 >>>>>>>> + Literal 38--1 >>>>>>>> After union right >>>>>>>> After union right >>>>>>>> tre_add_tag_right: tag 3 >>>>>>>> @@ -342,7 +342,7 @@ >>>>>>>> catenation, sub -1, 0 tags >>>>>>>> union, sub -1, 0 tags >>>>>>>> literal (, $) (0, 36), pos 0, sub -1, 0 tags >>>>>>>> - literal (&, M-^?) (38, 65535), pos 0, sub -1, 0 tags >>>>>>>> + literal (&, M-^?) (38, -1), pos 0, sub -1, 0 tags >>>>>>>> tag 4 >>>>>>>> >>>>>>>> It seems in 32-bit mode -1 is unsigned (65535) but -1 == -1 in 64-bit mode. >>>>>>>> >>>>>>>> I suspect I will "find it" - but a proposed change is appreciated. >>>>>>>> >>>>>>>> Happy New Year, >>>>>>>> Michael >>>>>>>> >>>>>>>> ______________________________________________ >>>>>>>> R-devel at r-project.org mailing list >>>>>>>> https://stat.ethz.ch/mailman/listinfo/r-devel >>>>>>>> >>>>>>> ______________________________________________ >>>>>>> R-devel at r-project.org mailing list >>>>>>> https://stat.ethz.ch/mailman/listinfo/r-devel >>>>>>> >>>>>> ______________________________________________ >>>>>> R-devel at r-project.org mailing list >>>>>> https://stat.ethz.ch/mailman/listinfo/r-devel
Michael Felt
2016-Jan-04 22:24 UTC
[Rd] R, AIX 64-bit builds - trying to understand root cause for message: "Error: Line starting 'Package: tools ...' is malformed!"
The bulk is on my forums - the final post for today is:
Results to date:
A. It looks like I am going to need a newer compiler for C - xlc/xlC V11
apparently does not understand this code:
"/data/prj/cran/R-3.2.3/src/main/memory.c", line 2149.31: 1506-046 (S)
Syntax error.
I will have to check if R-devel has different code before asking for
assistence.
+2139 #ifdef HAVE_STDALIGN_H
+2140 # include <stdalign.h>
+2141 #endif
+2142
+2143 #include <stdint.h>
+2144
+2145 long double *R_allocLD(size_t nelem)
+2146 {
+2147 #if __alignof_is_defined
+2148 // This is C11: picky compilers may warn.
+2149 size_t ld_align = alignof(long double);
+2150 #elif __GNUC__
+2151 // This is C99, but do not rely on it.
+2152 size_t ld_align = offsetof(struct { char __a; long double
__b; }, __b);
+2153 #else
+2154 size_t ld_align = 0x0F; // value of x86_64, known others
are 4 or 8
+2155 #endif
+2156 if (ld_align > 8) {
+2157 uintptr_t tmp = (uintptr_t) R_alloc(nelem + 1,
sizeof(long double));
+2158 tmp = (tmp + ld_align - 1) & ~ld_align;
+2159 return (long double *) tmp;
+2160 } else {
+2161 return (long double *) R_alloc(nelem, sizeof(long double));
+2162 }
+2163 }
If someone has a suggestion for how to test/fix so that I can proceed
with an older xlc compiler, that would be great!
If not, I shall download the try and buy C compiler to test.
On 04-Jan-16 15:52, Simon Urbanek wrote:> No, no duplicate warnings.
>
B. There is a very big difference in the way libraries are made when
gcc/gfortran are not used.
When gcc is being used "everything" is being turned into a shared
library. The only library I seem to be able
to affect via configure is libR.so (yes/no). The snip here shows that
all members of the .a archives are
"non-shared" objects rather that a combined group of .o files into a
single .so shared object.
So, it is not surprising that there are no duplicate symbols.
cran at x068:[/home/cran/64/R]dump -H src/*/*.a | head
src/appl/libappl.a[integrate.o]:
Loader section is not available
src/appl/libappl.a[interv.o]:
Loader section is not available
src/appl/libappl.a[maxcol.o]:>> >FYI: I shall be downloading the "try and buy" xlc and
xlfortran - and I think you will certainly prefer my packaging then as they work
without the libgc dependencies that many of the rpm packages need.
>> >
>> >And, at your option - we can take this discussion over tools -
"out of forums". Or at least start a new thread.
>> >
> We could leave the list out and create a Wiki or something with the results
of our tests.
>
>
>> >re: unsigned short - I have adopted the convention to use the *NN_t
types after running into a problem with unsigned longlong (from a very old
program). It goes back many years - and maybe they have finalized short to mean
16-bits - but I remember when short was meant to help when moving from 16-bit to
32-bit and the "word" size changed - i.e., int became 32-bit same as
long. nd for a long (no pun intended) long was 32-bit and long long was 64-bit.
Those definitions are extinct.
>> >
> I'm not sure what you refer here - the issue with TRE has nothing to do
with short - it can take any int type and like I said most platforms use
unsigned int which is big enough on all platforms.
>
>
>> >imho - the standard for wint_t is wrong as well - based on an
assumption about how "short" is defined. And, I consider it poor
practice that there are som many cases of type cast switches between ushort and
int.
>> >
> Not really - it doesn't care about short at all - note that the short
typedef is never actually used on AIX since it has wchar support so TRE is only
using int.
>
> Cheers,
> Simon
>
>
Michael Felt
2016-Feb-19 10:06 UTC
[Rd] R, AIX 64-bit builds - trying to understand root cause for message: "Error: Line starting 'Package: tools ...' is malformed!"
Hi Simon, I have been busy with work, but I finally got around to repackaging libiconv for AIX - in a way that both adds GNU libiconv support (libiconv.so.2 member) and is both 32 and 64 bit without breaking support for IBM iconv applications. I concur that iconv is a pain as IBM and GNU made different decesions about an 'implementor' choice to call a translation to an error or not - when the output map(?) does not include a character. IBM choose to output ASCII 32 (space) and GNU calls it an error. The standard states it is an error when a char (or code) is used in the input, but is not in the input table. Enough said imho - IBM and GNU differ in their approach and autotools test for this difference. So, I have a way to have both approaches available via /usr/lib. For a bit more detail please see: http://www.aixtools.net/index.php/libiconv - I do hope this 'fixes' the libiconv issues! for OSS. Using this library you should be able to replace: (what you sent in an earlier e-mail as your configure statement) On 2016-01-04 15:52, Simon Urbanek wrote:>> > I would be "pleased" if you would try packages - i.e., none of "Toolbox" or Perzl rpm's. >> > > R requires iconv, so you have to get that somehow. I tried to bypass the check and build it anyway against the system one, but that fails as there is a iconv conversion further down the line that is not supported. Building iconv is a PITA because it conflicts with the system one so other tools break when you modify the load path. Perzl's iconv fixes that by incorporating both the system objects and the GNU ones into the same binary and that seems to work reliably so far. That is the only dependency I was using (his iconv doesn't have any dependencies itself). > >configure CC=xlc_r CXX=xlc++_r F77=xlf_r FC=xlf95_r LIBS='-L/opt/freeware/lib /opt/freeware/lib/libiconv.a -lpthread' --prefix=/opt/freeware CPPFLAGS=-I/opt/freeware/include with export OBJECT_MODE=64 in the env (and libiconv from perzl.org RPMs - iconv is a serious pain). with export OBJECT_MODE=64 configure CC=xlc_r CXX=xlc++_r F77=xlf_r FC=xlf95_r LIBS='-lpthread' --prefix=/opt/freeware You may even be able to replace --prefix=/opt/freeware to --prefix=/opt (I use the latter to not step on anything in /opt/freeware)
Apparently Analagous Threads
- R for AIX - guidance re: error messages requested
- Pacaging/build issues with AIX and vac (dovecot-2.2.25)
- Construction of a large sparse matrix
- R, AIX 64-bit builds - trying to understand root cause for message: "Error: Line starting 'Package: tools ...' is malformed!"
- Error when using cobs library