thr3ads.net - R devel - [Rd] R, AIX 64-bit builds - trying to understand root cause for message: "Error: Line starting 'Package: tools ...' is malformed!" [Jan 2016]

If this information is useful, please help other people find it:
Share via:

Michael Felt

2016-Jan-04 06:34 UTC

[Rd] R, AIX 64-bit builds - trying to understand root cause for message: "Error: Line starting 'Package: tools ...' is malformed!"

I would be "pleased" if you would try packages - i.e., none of
"Toolbox"
or Perzl rpm's.

The iconv I supply might not be working as is (I will need to install a 
new system to check). However, this is why I have been testing with 
R-3.2.3 - to side-step the system library dependencies.

re: xz - I have "installp" version - and any of my packages should
work
side-by-side with the Toolbox/.rpm files - my PATH prefix is /opt (so, 
/opt/bin and /opt/sbin rather than /usr/bin, /usr/local/bin (etc.).

I am testing on a Power6, usually with 2 to 4G RAM active. When I get 
the compile to finish I will up to 8G to test some data processing that 
fails with the 32-bit limitations.

Another question: are you also getting lots of duplicate symbol 
warnings? Curious me. If you are I would like to send my modifications 
to configure(.ac) for you to test (I put .ac in () because I am actually 
editting configure atm but shall put them into configure.ac when I finish)

FYI: I shall be downloading the "try and buy" xlc and xlfortran - and
I
think you will certainly prefer my packaging then as they work without 
the libgc dependencies that many of the rpm packages need.

And, at your option - we can take this discussion over tools - "out of 
forums". Or at least start a new thread.

re: unsigned short - I have adopted the convention to use the *NN_t 
types after running into a problem with unsigned longlong (from a very 
old program). It goes back many years - and maybe they have finalized 
short to mean 16-bits - but I remember when short was meant to help when 
moving from 16-bit to 32-bit and the "word" size changed - i.e., int 
became 32-bit same as long. nd for a long (no pun intended) long was 
32-bit and long long was 64-bit. Those definitions are extinct.

imho - the standard for wint_t is wrong as well - based on an assumption 
about how "short" is defined. And, I consider it poor practice that 
there are som many cases of type cast switches between ushort and int.

Michael

On 04-Jan-16 01:26, Simon Urbanek wrote:> Michael,
>
> I'm using xlc + xlf - the exact flags are
>
> configure CC=xlc_r CXX=xlc++_r F77=xlf_r FC=xlf95_r
LIBS='-L/opt/freeware/lib /opt/freeware/lib/libiconv.a -lpthread'
--prefix=/opt/freeware CPPFLAGS=-I/opt/freeware/include
>
> with export OBJECT_MODE=64 in the env (and libiconv from perzl.org RPMs -
iconv is a serious pain).
>
> I have changed the TRE typedef from "wint_t" to "unsigned
int" since that is what the other platforms use anyway.
>
> On my AIX7 VM I'm running out of memory in xz when lazy-loading is
enabled which his odd - the VM has 4Gb which one would think should be enough.
The symptom is that installing MASS fails when creating lazy-load rds or the
same happens when running make check as memCompress() test for xz fails. I wish
there was a way to disable xz altogether..
>
> I'd like to get IBM compilers to work first since those are the
official ones. I may try gcc later, but I'm really interested in the IBM
ones because that gives us a good non-GNU test (e.g., TRE fails in JIT due some
alleged GNUisms so you can't use the strict compiler mode).
>
> Cheers,
> Simon
>
>
> On Jan 3, 2016, at 10:59 AM, Michael Felt <aixtools at gmail.com>
wrote:
>
>> On 2016-01-01 23:48, peter dalgaard wrote:
>>> Nice catch you two!!!
>>>
>>> Happy New Year
>>> -pd
>> I am much happier with this great start!
>>
>> Simon - which compiler)s) did you use: xlc and xlfortran, or
gcc/gfortran?
>>
>> I have made some changes to configure(.ac) so maybe my problems are
self-inflicted. But would be good to know what environment you are using.
>>
>> Thanks for looking - and finding!!!
>>
>> Michael
>>>> On 01 Jan 2016, at 22:06 , Simon Urbanek<simon.urbanek at
r-project.org>  wrote:
>>>>
>>>> Ok, found the problem - on platforms that support it TRE uses
wint_t (from wchar.h) as its type for characters (tre_cint_t) which on AIX is
*signed* int. TRE uses liberally conversions between int and tre_cint_t
apparently assuming that the latter is unsigned so conversions back to int are
suitable for comparisons etc. On other platforms wint_t is unsigned so it works.
Manually defining tre_cint_t to unsigned int fixes the issue.
>>>>
>>>> Cheers,
>>>> Simon
>>>>
>>>>
>>>> On Jan 1, 2016, at 12:20 PM, Simon Urbanek<simon.urbanek at
r-project.org>  wrote:
>>>>
>>>>> Michael,
>>>>>
>>>>> thanks, I'll have a look once my PDP VMs are up again
(later today). This may be a signedness issue although it's unclear why
other platforms wouldn't be affected.
>>>>>
>>>>> Cheers,
>>>>> Simon
>>>>>
>>>>>
>>>>> On Dec 31, 2015, at 10:14 AM, Michael Felt<aixtools at
gmail.com>  wrote:
>>>>>
>>>>>> On 2015-12-30 09:58, Michael Felt wrote:
>>>>>>> On 2015-12-29 11:02, Michael Felt wrote:
>>>>>>>> This seems to be a problem that goes back a
long time - and I hope someone who understands what tre is suppossed to be doing
will look at this.
>>>>>>>>
>>>>>>>> A short history of other people who have
reported on this on different versions of AIX. I shall only add that I get the
same results on AIX 5.3 TL7, AIX 6.1 TL9 and AIX 7.1 TL3.
>>>>>>>>
>>>>>>>> Basically, with settings that work for AIX and
32-bit - the only changes being
>>>>>>>> -maix32 becomes -maix64
>>>>>>>> and
>>>>>>>> export OBJECT_MODE=32 becomes export
OBJECT_MODE=64
>>>>>>>>
>>>>>>>> Then to shorten the 'make' bla bla,
first run just make, then
>>>>>>>>
>>>>>>>> cd src/library/tools
>>>>>>>> make -s sysdata
>>>>>>>>
>>>>>>>>
http://article.gmane.org/gmane.comp.lang.r.devel/38817/match=package+tools+malformed
>>>>>>>>
http://article.gmane.org/gmane.comp.lang.r.devel/36886/match=package+tools+malformed
>>>>>>>>
http://article.gmane.org/gmane.comp.lang.r.devel/23372/match=package+tools+malformed
Date: 2010-01-25 06:55:41 GMT (5 years, 48 weeks, 1 day, 20 hours and 30 minutes
ago)
>>>>>>>>
>>>>>>>> To that, to get debug data, I have
>>>>>>>>
>>>>>>>> * added -DTRE_DUGUG to src/extra/tre/Makefile #
ALL_CFLAGS = $(ALL_CFLAGS_LO) -DTRE_DEBUG
>>>>>>>> * rm src/extra/tre/tre-match-parallel.o
>>>>>>>> * find . -name \*.so -exec rm {} \;
>>>>>>>> * make
>>>>>>>> * cd src/library/tools
>>>>>>>> * make -s sysdata
>>>>>>>>
>>>>>>>> Attached are the two script files of the screen
output. The 32-bit one is more verbose - and contains magically lines such as:
>>>>>>>> found match 3037fd14 (while "found"
does not occur in the 64-bit output)
>>>>>>>>
>>>>>>>> root at
x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]wc /tmp/sysdata.??.*
>>>>>>>>    4730   14123  139916 /tmp/sysdata.32.text
>>>>>>>>    1312    3688   40528 /tmp/sysdata.64.text
>>>>>>>>    6042   17811  180444 total
>>>>>>>>
>>>>>>>> root at
x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]grep -c found
/tmp/sysdata.??.*
>>>>>>>> /tmp/sysdata.32.text:19
>>>>>>>> /tmp/sysdata.64.text:0
>>>>>>>>
>>>>>>>>
>>>>>>>> Hope this brings us (or me), closer to a
resolution to an old concern.
>>>>>>>>
>>>>>>>> And, best wishes for the new year!
>>>>>>>>
>>>>>>>> Michael
>>>>>>>>
>>>>>>>>
>>>>>>> Still hoping for someones curiosity/willingness.
>>>>>>>
>>>>>>> The differences show up in the first comparision
that is made (of the string "3.2.3" it seems) - 32-bit is on the left,
64-bit on the right.
>>>>>>>
>>>>>>> Script command is started on Tue Dec 29 08:39:16
UTC 2015.                     |  Script command is started on Tue Dec 29
08:39:56 UTC 2015.
>>>>>>> root at
x069:[/data/prj/cran/32/R-aix-3.2.3/src/library/tools]make -s sysdata     | 
root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]make -s sysdata
>>>>>>> installing 'sysdata.rda'                   
|  installing 'sysdata.rda'
>>>>>>> tre_tnfa_run_parallel, input type 1                
|  tre_tnfa_run_parallel, input type 1
>>>>>>> length: -1                                         
|  length: -1
>>>>>>> pos:chr/code | states and tags                     
|  pos:chr/code | states and tags
>>>>>>>
-------------+------------------------------------------------                 |
-------------+------------------------------------------------
>>>>>>> init>  30380200 3038014c 30380098               
|   init>  110cc3040 110cc2f28 110cc2e10
>>>>>>> match end offset = -1                              
|  match end offset = -1
>>>>>>> tre_tnfa_run_parallel, input type 1                
|  tre_tnfa_run_parallel, input type 1
>>>>>>> length: -1                                         
|  length: -1
>>>>>>> pos:chr/code | states and tags                     
|  pos:chr/code | states and tags
>>>>>>>
-------------+------------------------------------------------                 |
-------------+------------------------------------------------
>>>>>>> init>  3037fb88                                 
|   init>  110cc3310
>>>>>>> 0: 3/00051 | 3037fb88/0:0                          
|    0: 3/00051 | 110cc3310/0:0
>>>>>>> 1: ./00046 | 3037fb88/0:0                          
|    1: ./00046 | 110cc3310/0:0
>>>>>>> init>  3037fb88                                 
|   init>  110cc3310
>>>>>>> 1: ./00046 | 3037fb88/0:1                          
|    1: ./00046 | 110cc3310/0:1
>>>>>>> 2: 2/00050 | 3037fb88/0:1                          
|    2: 2/00050 | 110cc3310/0:1
>>>>>>> assertion failed                                   
|  assertion failed
>>>>>>> init>  3037fb88                                 
|   init>  110cc3310
>>>>>>> 2: 2/00050 | 3037fc18/0:1 3037fb88/0:2             
|    2: 2/00050 | 110cc33f0/0:1 110cc3310/0:2
>>>>>>> 3: ./00046 | 3037fc18/0:1 3037fb88/0:2             
|    3: ./00046 | 110cc33f0/0:1 110cc3310/0:2
>>>>>>> assertion failed                           ***
DIFFERENCE ***                  |   init>  110cc3310
>>>>>>> init>  3037fb88                                 
|    3: ./00046 | 110cc3310/0:3
>>>>>>> 3: ./00046 | 3037fc18/0:1 3037fb88/0:3             
|    4: 3/00051 | 110cc3310/0:3
>>>>>>> 4: 3/00051 | 3037fc18/0:1 3037fb88/0:3             
|  assertion failed
>>>>>>> assertion failed                                   
|   init>  110cc3310
>>>>>>> init>  3037fb88                                 
|    4: 3/00051 | 110cc33f0/0:3 110cc3310/0:4
>>>>>>> 4: 3/00051 | 3037fc18/0:3 3037fb88/0:4             
|    5: /00000 | 110cc33f0/0:3 110cc3310/0:4
>>>>>>> 5: /00000 | 3037fc18/0:3 3037fb88/0:4 |   init> 
110cc3310
>>>>>>> found match 3037fd14 *** DIFFERENCE ***            
|  match end offset = -1
>>>>>>> match end offset = 5 *** DIFFERENCE ***            
|  tre_tnfa_run_parallel, input type 1
>>>>>>> tre_tnfa_run_parallel, input type 1                
|  length: -1
>>>>>>> length: -1                                         
|  pos:chr/code | states and tags
>>>>>>> pos:chr/code | states and tags                     
|  -------------+------------------------------------------------
>>>>>>>
-------------+------------------------------------------------                 |
init>  110cc4780 110cc4668 110cc4550
>>>>>>> init>  303811c0 3038110c 30381058               
|  match end offset = -1
>>>>>>> match end offset = -1                              
|  tre_tnfa_run_parallel, input type 1
>>>>>>> tre_tnfa_run_parallel, input type 1                
|  length: -1
>>>>>>> length: -1                                         
|  pos:chr/code | states and tags
>>>>>>> pos:chr/code | states and tags                     
|  -------------+------------------------------------------------
>>>>>>>
-------------+------------------------------------------------                 |
init>  110cc5700 110cc55e8 110cc54d0
>>>>>>>
>>>>>> One day further - looks like tre_compile (or just
before, after all).
>>>>>>
>>>>>> With TRE_DEBUG switched on in tre-compile.c and
tre-ast.c I see (snip)
>>>>>>
>>>>>> --- /tmp/x.32   2015-12-31 15:09:44.000000000 +0000
>>>>>> +++ /tmp/x.64   2015-12-31 15:09:30.000000000 +0000
>>>>>> @@ -1,5 +1,5 @@
>>>>>> - Script command is started on Thu Dec 31 15:04:39
2015.
>>>>>> - root at
x069:[/data/prj/cran/32/R-aix-3.2.3/src/library/tools]make sysdata
>>>>>> + Script command is started on Thu Dec 31 15:08:43
2015.
>>>>>> + root at
x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]make sysdata
>>>>>> installing 'sysdata.rda'
>>>>>> echo
"tools:::sysdata2LazyLoadDB(\"/data/prj/cran/R-3.2.3/src/library/tools/R/sysdata.rda\",\"../../../library/tools/R\")"
| \
>>>>>>   R_DEFAULT_PACKAGES=NULL LC_ALL=C ../../../bin/R
--vanilla --slave
>>>>>> @@ -167,7 +167,7 @@
>>>>>> initial: 1/1,0, assert 0
>>>>>> initial: 0/0, assert 0
>>>>>> initial: 0/0, assert 0
>>>>>> - final state 30370718
>>>>>> + final state 110cba530
>>>>>> tre_compile: parsing '(^|[^%])(%%)*%V'
>>>>>> AST:
>>>>>> catenation, sub 0, 0 tags
>>>>>> @@ -177,7 +177,7 @@
>>>>>>         assertions: bol
>>>>>>         union, sub -1, 0 tags
>>>>>>           literal (, $) (0, 36), pos 0, sub -1, 0 tags
>>>>>> -           literal (&, M-^?) (38, 65535), pos 0,
sub -1, 0 tags
>>>>>> +           literal (&, M-^?) (38, -1), pos 0, sub
-1, 0 tags
>>>>>>       iteration {0, -1}, sub -1, 0 tags, greedy
>>>>>>         catenation, sub 2, 0 tags
>>>>>>           literal (%, %) (37, 37), pos 1, sub -1, 0
tags
>>>>>> @@ -197,7 +197,7 @@
>>>>>> Union
>>>>>> Literal 0-36
>>>>>> After union left
>>>>>> - Literal 38-65535
>>>>>> + Literal 38--1
>>>>>> After union right
>>>>>> After union right
>>>>>>   num_tags += 2
>>>>>> @@ -231,7 +231,7 @@
>>>>>>         assertions: bol
>>>>>>         union, sub -1, 0 tags
>>>>>>           literal (, $) (0, 36), pos 0, sub -1, 0 tags
>>>>>> -           literal (&, M-^?) (38, 65535), pos 0,
sub -1, 0 tags
>>>>>> +           literal (&, M-^?) (38, -1), pos 0, sub
-1, 0 tags
>>>>>>       iteration {0, -1}, sub -1, 2 tags, greedy
>>>>>>         catenation, sub 2, 1 tags
>>>>>>           literal (%, %) (37, 37), pos 1, sub -1, 1
tags
>>>>>> @@ -255,7 +255,7 @@
>>>>>> Union
>>>>>> Literal 0-36
>>>>>> After union left
>>>>>> - Literal 38-65535
>>>>>> + Literal 38--1
>>>>>> After union right
>>>>>> After union right
>>>>>> tre_add_tag_right: tag 3
>>>>>> @@ -342,7 +342,7 @@
>>>>>>           catenation, sub -1, 0 tags
>>>>>>             union, sub -1, 0 tags
>>>>>>               literal (, $) (0, 36), pos 0, sub -1, 0
tags
>>>>>> -               literal (&, M-^?) (38, 65535), pos
0, sub -1, 0 tags
>>>>>> +               literal (&, M-^?) (38, -1), pos 0,
sub -1, 0 tags
>>>>>>             tag 4
>>>>>>
>>>>>> It seems in 32-bit mode -1 is unsigned (65535) but -1
== -1 in 64-bit mode.
>>>>>>
>>>>>> I suspect I will "find it" - but a proposed
change is appreciated.
>>>>>>
>>>>>> Happy New Year,
>>>>>> Michael
>>>>>>
>>>>>> ______________________________________________
>>>>>> R-devel at r-project.org mailing list
>>>>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>>>>>
>>>>> ______________________________________________
>>>>> R-devel at r-project.org mailing list
>>>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>>>>
>>>> ______________________________________________
>>>> R-devel at r-project.org mailing list
>>>> https://stat.ethz.ch/mailman/listinfo/r-devel

Simon Urbanek

2016-Jan-04 14:52 UTC

head link

[Rd] R, AIX 64-bit builds - trying to understand root cause for message: "Error: Line starting 'Package: tools ...' is malformed!"

On Jan 4, 2016, at 1:34 AM, Michael Felt <aixtools at gmail.com> wrote:
> I would be "pleased" if you would try packages - i.e., none of
"Toolbox" or Perzl rpm's.
> 
R requires iconv, so you have to get that somehow. I tried to bypass the check
and build it anyway against the system one, but that fails as there is a iconv
conversion further down the line that is not supported. Building iconv is a PITA
because it conflicts with the system one so other tools break when you modify
the load path. Perzl's iconv fixes that by incorporating both the system
objects and the GNU ones into the same binary and that seems to work reliably so
far. That is the only dependency I was using (his iconv doesn't have any
dependencies itself).

> The iconv I supply might not be working as is (I will need to install a new
system to check). However, this is why I have been testing with R-3.2.3 - to
side-step the system library dependencies.
> 
> re: xz - I have "installp" version - and any of my packages
should work side-by-side with the Toolbox/.rpm files - my PATH prefix is /opt
(so, /opt/bin and /opt/sbin rather than /usr/bin, /usr/local/bin (etc.).
> 
It fails even with the built-in one in R so it has nothing to do with the xz
used itself.

> I am testing on a Power6, usually with 2 to 4G RAM active. When I get the
compile to finish I will up to 8G to test some data processing that fails with
the 32-bit limitations.
> 
> Another question: are you also getting lots of duplicate symbol warnings?
Curious me. If you are I would like to send my modifications to configure(.ac)
for you to test (I put .ac in () because I am actually editting configure atm
but shall put them into configure.ac when I finish)
> 
No, no duplicate warnings.

> FYI: I shall be downloading the "try and buy" xlc and xlfortran -
and I think you will certainly prefer my packaging then as they work without the
libgc dependencies that many of the rpm packages need.
> 
> And, at your option - we can take this discussion over tools - "out of
forums". Or at least start a new thread.
> 
We could leave the list out and create a Wiki or something with the results of
our tests.

> re: unsigned short - I have adopted the convention to use the *NN_t types
after running into a problem with unsigned longlong (from a very old program).
It goes back many years - and maybe they have finalized short to mean 16-bits -
but I remember when short was meant to help when moving from 16-bit to 32-bit
and the "word" size changed - i.e., int became 32-bit same as long. nd
for a long (no pun intended) long was 32-bit and long long was 64-bit. Those
definitions are extinct.
> 
I'm not sure what you refer here - the issue with TRE has nothing to do with
short - it can take any int type and like I said most platforms use unsigned int
which is big enough on all platforms.

> imho - the standard for wint_t is wrong as well - based on an assumption
about how "short" is defined. And, I consider it poor practice that
there are som many cases of type cast switches between ushort and int.
> 
Not really - it doesn't care about short at all - note that the short
typedef is never actually used on AIX since it has wchar support so TRE is only
using int.

Cheers,
Simon

> Michael
> 
> On 04-Jan-16 01:26, Simon Urbanek wrote:
>> Michael,
>> 
>> I'm using xlc + xlf - the exact flags are
>> 
>> configure CC=xlc_r CXX=xlc++_r F77=xlf_r FC=xlf95_r
LIBS='-L/opt/freeware/lib /opt/freeware/lib/libiconv.a -lpthread'
--prefix=/opt/freeware CPPFLAGS=-I/opt/freeware/include
>> 
>> with export OBJECT_MODE=64 in the env (and libiconv from perzl.org RPMs
- iconv is a serious pain).
>> 
>> I have changed the TRE typedef from "wint_t" to
"unsigned int" since that is what the other platforms use anyway.
>> 
>> On my AIX7 VM I'm running out of memory in xz when lazy-loading is
enabled which his odd - the VM has 4Gb which one would think should be enough.
The symptom is that installing MASS fails when creating lazy-load rds or the
same happens when running make check as memCompress() test for xz fails. I wish
there was a way to disable xz altogether..
>> 
>> I'd like to get IBM compilers to work first since those are the
official ones. I may try gcc later, but I'm really interested in the IBM
ones because that gives us a good non-GNU test (e.g., TRE fails in JIT due some
alleged GNUisms so you can't use the strict compiler mode).
>> 
>> Cheers,
>> Simon
>> 
>> 
>> On Jan 3, 2016, at 10:59 AM, Michael Felt <aixtools at gmail.com>
wrote:
>> 
>>> On 2016-01-01 23:48, peter dalgaard wrote:
>>>> Nice catch you two!!!
>>>> 
>>>> Happy New Year
>>>> -pd
>>> I am much happier with this great start!
>>> 
>>> Simon - which compiler)s) did you use: xlc and xlfortran, or
gcc/gfortran?
>>> 
>>> I have made some changes to configure(.ac) so maybe my problems are
self-inflicted. But would be good to know what environment you are using.
>>> 
>>> Thanks for looking - and finding!!!
>>> 
>>> Michael
>>>>> On 01 Jan 2016, at 22:06 , Simon Urbanek<simon.urbanek
at r-project.org>  wrote:
>>>>> 
>>>>> Ok, found the problem - on platforms that support it TRE
uses wint_t (from wchar.h) as its type for characters (tre_cint_t) which on AIX
is *signed* int. TRE uses liberally conversions between int and tre_cint_t
apparently assuming that the latter is unsigned so conversions back to int are
suitable for comparisons etc. On other platforms wint_t is unsigned so it works.
Manually defining tre_cint_t to unsigned int fixes the issue.
>>>>> 
>>>>> Cheers,
>>>>> Simon
>>>>> 
>>>>> 
>>>>> On Jan 1, 2016, at 12:20 PM, Simon Urbanek<simon.urbanek
at r-project.org>  wrote:
>>>>> 
>>>>>> Michael,
>>>>>> 
>>>>>> thanks, I'll have a look once my PDP VMs are up
again (later today). This may be a signedness issue although it's unclear
why other platforms wouldn't be affected.
>>>>>> 
>>>>>> Cheers,
>>>>>> Simon
>>>>>> 
>>>>>> 
>>>>>> On Dec 31, 2015, at 10:14 AM, Michael Felt<aixtools
at gmail.com>  wrote:
>>>>>> 
>>>>>>> On 2015-12-30 09:58, Michael Felt wrote:
>>>>>>>> On 2015-12-29 11:02, Michael Felt wrote:
>>>>>>>>> This seems to be a problem that goes back a
long time - and I hope someone who understands what tre is suppossed to be doing
will look at this.
>>>>>>>>> 
>>>>>>>>> A short history of other people who have
reported on this on different versions of AIX. I shall only add that I get the
same results on AIX 5.3 TL7, AIX 6.1 TL9 and AIX 7.1 TL3.
>>>>>>>>> 
>>>>>>>>> Basically, with settings that work for AIX
and 32-bit - the only changes being
>>>>>>>>> -maix32 becomes -maix64
>>>>>>>>> and
>>>>>>>>> export OBJECT_MODE=32 becomes export
OBJECT_MODE=64
>>>>>>>>> 
>>>>>>>>> Then to shorten the 'make' bla bla,
first run just make, then
>>>>>>>>> 
>>>>>>>>> cd src/library/tools
>>>>>>>>> make -s sysdata
>>>>>>>>> 
>>>>>>>>>
http://article.gmane.org/gmane.comp.lang.r.devel/38817/match=package+tools+malformed
>>>>>>>>>
http://article.gmane.org/gmane.comp.lang.r.devel/36886/match=package+tools+malformed
>>>>>>>>>
http://article.gmane.org/gmane.comp.lang.r.devel/23372/match=package+tools+malformed
Date: 2010-01-25 06:55:41 GMT (5 years, 48 weeks, 1 day, 20 hours and 30 minutes
ago)
>>>>>>>>> 
>>>>>>>>> To that, to get debug data, I have
>>>>>>>>> 
>>>>>>>>> * added -DTRE_DUGUG to
src/extra/tre/Makefile # ALL_CFLAGS = $(ALL_CFLAGS_LO) -DTRE_DEBUG
>>>>>>>>> * rm src/extra/tre/tre-match-parallel.o
>>>>>>>>> * find . -name \*.so -exec rm {} \;
>>>>>>>>> * make
>>>>>>>>> * cd src/library/tools
>>>>>>>>> * make -s sysdata
>>>>>>>>> 
>>>>>>>>> Attached are the two script files of the
screen output. The 32-bit one is more verbose - and contains magically lines
such as:
>>>>>>>>> found match 3037fd14 (while
"found" does not occur in the 64-bit output)
>>>>>>>>> 
>>>>>>>>> root at
x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]wc /tmp/sysdata.??.*
>>>>>>>>>   4730   14123  139916 /tmp/sysdata.32.text
>>>>>>>>>   1312    3688   40528 /tmp/sysdata.64.text
>>>>>>>>>   6042   17811  180444 total
>>>>>>>>> 
>>>>>>>>> root at
x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]grep -c found
/tmp/sysdata.??.*
>>>>>>>>> /tmp/sysdata.32.text:19
>>>>>>>>> /tmp/sysdata.64.text:0
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Hope this brings us (or me), closer to a
resolution to an old concern.
>>>>>>>>> 
>>>>>>>>> And, best wishes for the new year!
>>>>>>>>> 
>>>>>>>>> Michael
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> Still hoping for someones
curiosity/willingness.
>>>>>>>> 
>>>>>>>> The differences show up in the first
comparision that is made (of the string "3.2.3" it seems) - 32-bit is
on the left, 64-bit on the right.
>>>>>>>> 
>>>>>>>> Script command is started on Tue Dec 29
08:39:16 UTC 2015.                     |  Script command is started on Tue Dec
29 08:39:56 UTC 2015.
>>>>>>>> root at
x069:[/data/prj/cran/32/R-aix-3.2.3/src/library/tools]make -s sysdata     | 
root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]make -s sysdata
>>>>>>>> installing 'sysdata.rda'               
|  installing 'sysdata.rda'
>>>>>>>> tre_tnfa_run_parallel, input type 1            
|  tre_tnfa_run_parallel, input type 1
>>>>>>>> length: -1                                     
|  length: -1
>>>>>>>> pos:chr/code | states and tags                 
|  pos:chr/code | states and tags
>>>>>>>>
-------------+------------------------------------------------                 |
-------------+------------------------------------------------
>>>>>>>> init>  30380200 3038014c 30380098           
|   init>  110cc3040 110cc2f28 110cc2e10
>>>>>>>> match end offset = -1                          
|  match end offset = -1
>>>>>>>> tre_tnfa_run_parallel, input type 1            
|  tre_tnfa_run_parallel, input type 1
>>>>>>>> length: -1                                     
|  length: -1
>>>>>>>> pos:chr/code | states and tags                 
|  pos:chr/code | states and tags
>>>>>>>>
-------------+------------------------------------------------                 |
-------------+------------------------------------------------
>>>>>>>> init>  3037fb88                             
|   init>  110cc3310
>>>>>>>> 0: 3/00051 | 3037fb88/0:0                      
|    0: 3/00051 | 110cc3310/0:0
>>>>>>>> 1: ./00046 | 3037fb88/0:0                      
|    1: ./00046 | 110cc3310/0:0
>>>>>>>> init>  3037fb88                             
|   init>  110cc3310
>>>>>>>> 1: ./00046 | 3037fb88/0:1                      
|    1: ./00046 | 110cc3310/0:1
>>>>>>>> 2: 2/00050 | 3037fb88/0:1                      
|    2: 2/00050 | 110cc3310/0:1
>>>>>>>> assertion failed                               
|  assertion failed
>>>>>>>> init>  3037fb88                             
|   init>  110cc3310
>>>>>>>> 2: 2/00050 | 3037fc18/0:1 3037fb88/0:2         
|    2: 2/00050 | 110cc33f0/0:1 110cc3310/0:2
>>>>>>>> 3: ./00046 | 3037fc18/0:1 3037fb88/0:2         
|    3: ./00046 | 110cc33f0/0:1 110cc3310/0:2
>>>>>>>> assertion failed                           ***
DIFFERENCE ***                  |   init>  110cc3310
>>>>>>>> init>  3037fb88                             
|    3: ./00046 | 110cc3310/0:3
>>>>>>>> 3: ./00046 | 3037fc18/0:1 3037fb88/0:3         
|    4: 3/00051 | 110cc3310/0:3
>>>>>>>> 4: 3/00051 | 3037fc18/0:1 3037fb88/0:3         
|  assertion failed
>>>>>>>> assertion failed                               
|   init>  110cc3310
>>>>>>>> init>  3037fb88                             
|    4: 3/00051 | 110cc33f0/0:3 110cc3310/0:4
>>>>>>>> 4: 3/00051 | 3037fc18/0:3 3037fb88/0:4         
|    5: /00000 | 110cc33f0/0:3 110cc3310/0:4
>>>>>>>> 5: /00000 | 3037fc18/0:3 3037fb88/0:4 |  
init>  110cc3310
>>>>>>>> found match 3037fd14 *** DIFFERENCE ***        
|  match end offset = -1
>>>>>>>> match end offset = 5 *** DIFFERENCE ***        
|  tre_tnfa_run_parallel, input type 1
>>>>>>>> tre_tnfa_run_parallel, input type 1            
|  length: -1
>>>>>>>> length: -1                                     
|  pos:chr/code | states and tags
>>>>>>>> pos:chr/code | states and tags                 
|  -------------+------------------------------------------------
>>>>>>>>
-------------+------------------------------------------------                 |
init>  110cc4780 110cc4668 110cc4550
>>>>>>>> init>  303811c0 3038110c 30381058           
|  match end offset = -1
>>>>>>>> match end offset = -1                          
|  tre_tnfa_run_parallel, input type 1
>>>>>>>> tre_tnfa_run_parallel, input type 1            
|  length: -1
>>>>>>>> length: -1                                     
|  pos:chr/code | states and tags
>>>>>>>> pos:chr/code | states and tags                 
|  -------------+------------------------------------------------
>>>>>>>>
-------------+------------------------------------------------                 |
init>  110cc5700 110cc55e8 110cc54d0
>>>>>>>> 
>>>>>>> One day further - looks like tre_compile (or just
before, after all).
>>>>>>> 
>>>>>>> With TRE_DEBUG switched on in tre-compile.c and
tre-ast.c I see (snip)
>>>>>>> 
>>>>>>> --- /tmp/x.32   2015-12-31 15:09:44.000000000 +0000
>>>>>>> +++ /tmp/x.64   2015-12-31 15:09:30.000000000 +0000
>>>>>>> @@ -1,5 +1,5 @@
>>>>>>> - Script command is started on Thu Dec 31 15:04:39
2015.
>>>>>>> - root at
x069:[/data/prj/cran/32/R-aix-3.2.3/src/library/tools]make sysdata
>>>>>>> + Script command is started on Thu Dec 31 15:08:43
2015.
>>>>>>> + root at
x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]make sysdata
>>>>>>> installing 'sysdata.rda'
>>>>>>> echo
"tools:::sysdata2LazyLoadDB(\"/data/prj/cran/R-3.2.3/src/library/tools/R/sysdata.rda\",\"../../../library/tools/R\")"
| \
>>>>>>>  R_DEFAULT_PACKAGES=NULL LC_ALL=C ../../../bin/R
--vanilla --slave
>>>>>>> @@ -167,7 +167,7 @@
>>>>>>> initial: 1/1,0, assert 0
>>>>>>> initial: 0/0, assert 0
>>>>>>> initial: 0/0, assert 0
>>>>>>> - final state 30370718
>>>>>>> + final state 110cba530
>>>>>>> tre_compile: parsing '(^|[^%])(%%)*%V'
>>>>>>> AST:
>>>>>>> catenation, sub 0, 0 tags
>>>>>>> @@ -177,7 +177,7 @@
>>>>>>>        assertions: bol
>>>>>>>        union, sub -1, 0 tags
>>>>>>>          literal (, $) (0, 36), pos 0, sub -1, 0
tags
>>>>>>> -           literal (&, M-^?) (38, 65535), pos
0, sub -1, 0 tags
>>>>>>> +           literal (&, M-^?) (38, -1), pos 0,
sub -1, 0 tags
>>>>>>>      iteration {0, -1}, sub -1, 0 tags, greedy
>>>>>>>        catenation, sub 2, 0 tags
>>>>>>>          literal (%, %) (37, 37), pos 1, sub -1, 0
tags
>>>>>>> @@ -197,7 +197,7 @@
>>>>>>> Union
>>>>>>> Literal 0-36
>>>>>>> After union left
>>>>>>> - Literal 38-65535
>>>>>>> + Literal 38--1
>>>>>>> After union right
>>>>>>> After union right
>>>>>>>  num_tags += 2
>>>>>>> @@ -231,7 +231,7 @@
>>>>>>>        assertions: bol
>>>>>>>        union, sub -1, 0 tags
>>>>>>>          literal (, $) (0, 36), pos 0, sub -1, 0
tags
>>>>>>> -           literal (&, M-^?) (38, 65535), pos
0, sub -1, 0 tags
>>>>>>> +           literal (&, M-^?) (38, -1), pos 0,
sub -1, 0 tags
>>>>>>>      iteration {0, -1}, sub -1, 2 tags, greedy
>>>>>>>        catenation, sub 2, 1 tags
>>>>>>>          literal (%, %) (37, 37), pos 1, sub -1, 1
tags
>>>>>>> @@ -255,7 +255,7 @@
>>>>>>> Union
>>>>>>> Literal 0-36
>>>>>>> After union left
>>>>>>> - Literal 38-65535
>>>>>>> + Literal 38--1
>>>>>>> After union right
>>>>>>> After union right
>>>>>>> tre_add_tag_right: tag 3
>>>>>>> @@ -342,7 +342,7 @@
>>>>>>>          catenation, sub -1, 0 tags
>>>>>>>            union, sub -1, 0 tags
>>>>>>>              literal (, $) (0, 36), pos 0, sub -1,
0 tags
>>>>>>> -               literal (&, M-^?) (38, 65535),
pos 0, sub -1, 0 tags
>>>>>>> +               literal (&, M-^?) (38, -1), pos
0, sub -1, 0 tags
>>>>>>>            tag 4
>>>>>>> 
>>>>>>> It seems in 32-bit mode -1 is unsigned (65535) but
-1 == -1 in 64-bit mode.
>>>>>>> 
>>>>>>> I suspect I will "find it" - but a
proposed change is appreciated.
>>>>>>> 
>>>>>>> Happy New Year,
>>>>>>> Michael
>>>>>>> 
>>>>>>> ______________________________________________
>>>>>>> R-devel at r-project.org mailing list
>>>>>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>>>>>> 
>>>>>> ______________________________________________
>>>>>> R-devel at r-project.org mailing list
>>>>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>>>>> 
>>>>> ______________________________________________
>>>>> R-devel at r-project.org mailing list
>>>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>

Michael Felt

2016-Jan-04 20:47 UTC

head link

[Rd] R, AIX 64-bit builds - trying to understand root cause for message: "Error: Line starting 'Package: tools ...' is malformed!"

On 04-Jan-16 15:52, Simon Urbanek wrote:> On Jan 4, 2016, at 1:34 AM, Michael Felt <aixtools at gmail.com>
wrote:
>
>> I would be "pleased" if you would try packages - i.e., none
of "Toolbox" or Perzl rpm's.
>>
> R requires iconv, so you have to get that somehow. I tried to bypass the
check and build it anyway against the system one, but that fails as there is a
iconv conversion further down the line that is not supported. Building iconv is
a PITA because it conflicts with the system one so other tools break when you
modify the load path. Perzl's iconv fixes that by incorporating both the
system objects and the GNU ones into the same binary and that seems to work
reliably so far. That is the only dependency I was using (his iconv doesn't
have any dependencies itself).I would have to install another system and install perzl iconv - 
however, I assume that it also sits in /usr/lib (and/or has a symbolic 
link to it). I have mine installed in /opt/lib - so the default shared 
libraries look like:

cran at x065:[/home/cran/64/R]ldd /home/cran/64/R/bin/exec/R
/home/cran/64/R/bin/exec/R needs:
          /usr/lib/libc.a(shr_64.o)
          /usr/lib/libpthread.a(shr_xpg5_64.o)
          /opt/lib/libintl.a(libintl.so.8)
          /opt/lib/libiconv.a(libiconv.so.2)
          /unix
          /usr/lib/libcrypt.a(shr_64.o)
          /usr/lib/libpthread.a(shr_xpg5.o)
          /usr/lib/libc.a(shr.o)
          /usr/lib/libpthreads.a(shr_comm.o)
          /usr/lib/libcrypt.a(shr.o)

I have tested with adding my (gnu iconv) to the /usr/lib archive, as the 
search does seem to get corrupted.
For testing purposes I have added the 64-bit member to my 32-bit 
packaging - rather than a seperate archive in /opt/lib64

cran at x065:[/home/cran/64/R]ar -X32_64 -tv /opt/lib/libiconv.a
rwxr-xr-x     0/0     1010404 Nov 27 17:28 2014 libiconv.so.2
rwxr-xr-x     0/0     111901 Nov 27 17:28 2014 shr4.o
rwxr-xr-x     0/0     112151 Nov 27 17:28 2014 shr.o
rwxr-xr-x     0/0     1035470 Dec  3 15:11 2015 libiconv.so.2

I did something similiar to this for all the dependencies earlier when I 
was working with R-devel branch. When the build is finishing I will 
workout how to install both into a single archive (the hard part is a 
clean uninstall afterwards, and harder is managing an update. So, 
ideally, I will go to having one package with both 32 and 64 bit members.)
>
>> The iconv I supply might not be working as is (I will need to install a
new system to check). However, this is why I have been testing with R-3.2.3 - to
side-step the system library dependencies.
>>
>> re: xz - I have "installp" version - and any of my packages
should work side-by-side with the Toolbox/.rpm files - my PATH prefix is /opt
(so, /opt/bin and /opt/sbin rather than /usr/bin, /usr/local/bin (etc.).
>>
> It fails even with the built-in one in R so it has nothing to do with the
xz used itself.What is the built-in level - do not know off the top of my head. Perzl 
is 5.0.7 and I have been using 5.0.8 (when working with R-devel).
>
>
>> I am testing on a Power6, usually with 2 to 4G RAM active. When I get
the compile to finish I will up to 8G to test some data processing that fails
with the 32-bit limitations.
>>
>> Another question: are you also getting lots of duplicate symbol
warnings? Curious me. If you are I would like to send my modifications to
configure(.ac) for you to test (I put .ac in () because I am actually editting
configure atm but shall put them into configure.ac when I finish)
>>
> No, no duplicate warnings.Then I suspect the duplicate warnings have to do with how the m4 macros 
are processing the gcc -v flags (actually, the gfortran -v flags). 
Adding full path to libgcc.a and including -bexpall causes symbols to be 
imported and reexported - hence duplicate.

Another reason for me to get xlfortran installed and "test" against
that
as well.>
>
>> FYI: I shall be downloading the "try and buy" xlc and
xlfortran - and I think you will certainly prefer my packaging then as they work
without the libgc dependencies that many of the rpm packages need.
>>
>> And, at your option - we can take this discussion over tools -
"out of forums". Or at least start a new thread.
>>
> We could leave the list out and create a Wiki or something with the results
of our tests.I would be happy to sponsor that - either on my forums 
(http://forums.rootvg.net/aixtools/r-for-aix/) or I can look into how I 
create a userid for you on my aixtools wiki. I started something in 
November 
http://www.aixtools.net/index.php/Projects#%22Work%20in%20Progress%22, 
but basically, am waiting for better results before dedicating a page to 
R. In short, I would prefer the forums - as anyone could join the 
discussion without any intervention from me.>
>
>> re: unsigned short - I have adopted the convention to use the *NN_t
types after running into a problem with unsigned longlong (from a very old
program). It goes back many years - and maybe they have finalized short to mean
16-bits - but I remember when short was meant to help when moving from 16-bit to
32-bit and the "word" size changed - i.e., int became 32-bit same as
long. nd for a long (no pun intended) long was 32-bit and long long was 64-bit.
Those definitions are extinct.
>>
> I'm not sure what you refer here - the issue with TRE has nothing to do
with short - it can take any int type and like I said most platforms use
unsigned int which is big enough on all platforms.I claim no expertise. My observation is that UTF-XX seems to be 16-bit. 
I say seems, because there may be a multi-byte requirement that does not 
fit in 16-bits and then a larger size is appropriate. From my 
perspective (as a porter) it is easier to identify inconsistency when 
typedef definitions are not directly related to short, int or long. In 
binary it may not make any difference as far as sizeof() is concerned. I 
just hope for improved legibility.

I am not "r-project" - and my assumption is that "r-project"
does the
commits. My contribution is merely my opinion. What you decide is what 
counts - not my opinion.>
>> imho - the standard for wint_t is wrong as well - based on an
assumption about how "short" is defined. And, I consider it poor
practice that there are som many cases of type cast switches between ushort and
int.
>>
> Not really - it doesn't care about short at all - note that the short
typedef is never actually used on AIX since it has wchar support so TRE is only
using int.You lost me here. I am aware that AIX has a very different set of 
include files that tre is not using. My perspective is that R is 
ignoring/overrulling any IBM/AIX wchar processing. See my "not an 
expert" defense above.

Mainly - I am happy that "make sysdata" is working.

p.s. I shall make a new entry on my forums re: where I am at now re:
Warning in solve.default(rgb) :
   unable to load shared object '/home/cran/64/R/modules//lapack.so':
   rtld: 0712-001 Symbol __powidf2 was referenced
       from module /home/cran/64/R/lib/libRlapack.so(), but a runtime 
definition
             of the symbol was not found.
rtld: 0712-001 Symbol _gfortran_pow_i4_i4 was referenced
       from module /home/cran/64/R/lib/libRlapack.so(), but a runtime 
definition
             of the symbol was not found.
rtld: 0712-001 Symbol 
_gfortran_compare_string/home/cran/64/R/lib/libRlapack.so was referenced
       from module /home/cran/64/R/lib/libRlapack.so(), but a runtime 
definition
             of the symbol was not found.
rtld: 0712-001 Symbol _gfortran_concat_string was referenced
       from module /home/cran/64/R/lib/libRlapack.so(), but a runtime 
definition
             of the symbol was not found.
rtld: 0712-002 fatal error: exiting.
Error in solve.default(rgb) : LAPACK routines cannot be loaded
Error: unable to load R code in package 'grDevices'
>
> Cheers,
> Simon
>
>
>> Michael
>>
>> On 04-Jan-16 01:26, Simon Urbanek wrote:
>>> Michael,
>>>
>>> I'm using xlc + xlf - the exact flags are
>>>
>>> configure CC=xlc_r CXX=xlc++_r F77=xlf_r FC=xlf95_r
LIBS='-L/opt/freeware/lib /opt/freeware/lib/libiconv.a -lpthread'
--prefix=/opt/freeware CPPFLAGS=-I/opt/freeware/include
>>>
>>> with export OBJECT_MODE=64 in the env (and libiconv from perzl.org
RPMs - iconv is a serious pain).
>>>
>>> I have changed the TRE typedef from "wint_t" to
"unsigned int" since that is what the other platforms use anyway.
>>>
>>> On my AIX7 VM I'm running out of memory in xz when lazy-loading
is enabled which his odd - the VM has 4Gb which one would think should be
enough. The symptom is that installing MASS fails when creating lazy-load rds or
the same happens when running make check as memCompress() test for xz fails. I
wish there was a way to disable xz altogether..
>>>
>>> I'd like to get IBM compilers to work first since those are the
official ones. I may try gcc later, but I'm really interested in the IBM
ones because that gives us a good non-GNU test (e.g., TRE fails in JIT due some
alleged GNUisms so you can't use the strict compiler mode).
>>>
>>> Cheers,
>>> Simon
>>>
>>>
>>> On Jan 3, 2016, at 10:59 AM, Michael Felt <aixtools at
gmail.com> wrote:
>>>
>>>> On 2016-01-01 23:48, peter dalgaard wrote:
>>>>> Nice catch you two!!!
>>>>>
>>>>> Happy New Year
>>>>> -pd
>>>> I am much happier with this great start!
>>>>
>>>> Simon - which compiler)s) did you use: xlc and xlfortran, or
gcc/gfortran?
>>>>
>>>> I have made some changes to configure(.ac) so maybe my problems
are self-inflicted. But would be good to know what environment you are using.
>>>>
>>>> Thanks for looking - and finding!!!
>>>>
>>>> Michael
>>>>>> On 01 Jan 2016, at 22:06 , Simon
Urbanek<simon.urbanek at r-project.org>  wrote:
>>>>>>
>>>>>> Ok, found the problem - on platforms that support it
TRE uses wint_t (from wchar.h) as its type for characters (tre_cint_t) which on
AIX is *signed* int. TRE uses liberally conversions between int and tre_cint_t
apparently assuming that the latter is unsigned so conversions back to int are
suitable for comparisons etc. On other platforms wint_t is unsigned so it works.
Manually defining tre_cint_t to unsigned int fixes the issue.
>>>>>>
>>>>>> Cheers,
>>>>>> Simon
>>>>>>
>>>>>>
>>>>>> On Jan 1, 2016, at 12:20 PM, Simon
Urbanek<simon.urbanek at r-project.org>  wrote:
>>>>>>
>>>>>>> Michael,
>>>>>>>
>>>>>>> thanks, I'll have a look once my PDP VMs are up
again (later today). This may be a signedness issue although it's unclear
why other platforms wouldn't be affected.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Simon
>>>>>>>
>>>>>>>
>>>>>>> On Dec 31, 2015, at 10:14 AM, Michael
Felt<aixtools at gmail.com>  wrote:
>>>>>>>
>>>>>>>> On 2015-12-30 09:58, Michael Felt wrote:
>>>>>>>>> On 2015-12-29 11:02, Michael Felt wrote:
>>>>>>>>>> This seems to be a problem that goes
back a long time - and I hope someone who understands what tre is suppossed to
be doing will look at this.
>>>>>>>>>>
>>>>>>>>>> A short history of other people who
have reported on this on different versions of AIX. I shall only add that I get
the same results on AIX 5.3 TL7, AIX 6.1 TL9 and AIX 7.1 TL3.
>>>>>>>>>>
>>>>>>>>>> Basically, with settings that work for
AIX and 32-bit - the only changes being
>>>>>>>>>> -maix32 becomes -maix64
>>>>>>>>>> and
>>>>>>>>>> export OBJECT_MODE=32 becomes export
OBJECT_MODE=64
>>>>>>>>>>
>>>>>>>>>> Then to shorten the 'make' bla
bla, first run just make, then
>>>>>>>>>>
>>>>>>>>>> cd src/library/tools
>>>>>>>>>> make -s sysdata
>>>>>>>>>>
>>>>>>>>>>
http://article.gmane.org/gmane.comp.lang.r.devel/38817/match=package+tools+malformed
>>>>>>>>>>
http://article.gmane.org/gmane.comp.lang.r.devel/36886/match=package+tools+malformed
>>>>>>>>>>
http://article.gmane.org/gmane.comp.lang.r.devel/23372/match=package+tools+malformed
Date: 2010-01-25 06:55:41 GMT (5 years, 48 weeks, 1 day, 20 hours and 30 minutes
ago)
>>>>>>>>>>
>>>>>>>>>> To that, to get debug data, I have
>>>>>>>>>>
>>>>>>>>>> * added -DTRE_DUGUG to
src/extra/tre/Makefile # ALL_CFLAGS = $(ALL_CFLAGS_LO) -DTRE_DEBUG
>>>>>>>>>> * rm src/extra/tre/tre-match-parallel.o
>>>>>>>>>> * find . -name \*.so -exec rm {} \;
>>>>>>>>>> * make
>>>>>>>>>> * cd src/library/tools
>>>>>>>>>> * make -s sysdata
>>>>>>>>>>
>>>>>>>>>> Attached are the two script files of
the screen output. The 32-bit one is more verbose - and contains magically lines
such as:
>>>>>>>>>> found match 3037fd14 (while
"found" does not occur in the 64-bit output)
>>>>>>>>>>
>>>>>>>>>> root at
x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]wc /tmp/sysdata.??.*
>>>>>>>>>>    4730   14123  139916
/tmp/sysdata.32.text
>>>>>>>>>>    1312    3688   40528
/tmp/sysdata.64.text
>>>>>>>>>>    6042   17811  180444 total
>>>>>>>>>>
>>>>>>>>>> root at
x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]grep -c found
/tmp/sysdata.??.*
>>>>>>>>>> /tmp/sysdata.32.text:19
>>>>>>>>>> /tmp/sysdata.64.text:0
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hope this brings us (or me), closer to
a resolution to an old concern.
>>>>>>>>>>
>>>>>>>>>> And, best wishes for the new year!
>>>>>>>>>>
>>>>>>>>>> Michael
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> Still hoping for someones
curiosity/willingness.
>>>>>>>>>
>>>>>>>>> The differences show up in the first
comparision that is made (of the string "3.2.3" it seems) - 32-bit is
on the left, 64-bit on the right.
>>>>>>>>>
>>>>>>>>> Script command is started on Tue Dec 29
08:39:16 UTC 2015.                     |  Script command is started on Tue Dec
29 08:39:56 UTC 2015.
>>>>>>>>> root at
x069:[/data/prj/cran/32/R-aix-3.2.3/src/library/tools]make -s sysdata     | 
root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]make -s sysdata
>>>>>>>>> installing 'sysdata.rda'           
|  installing 'sysdata.rda'
>>>>>>>>> tre_tnfa_run_parallel, input type 1        
|  tre_tnfa_run_parallel, input type 1
>>>>>>>>> length: -1                                 
|  length: -1
>>>>>>>>> pos:chr/code | states and tags             
|  pos:chr/code | states and tags
>>>>>>>>>
-------------+------------------------------------------------                 |
-------------+------------------------------------------------
>>>>>>>>> init>  30380200 3038014c 30380098       
|   init>  110cc3040 110cc2f28 110cc2e10
>>>>>>>>> match end offset = -1                      
|  match end offset = -1
>>>>>>>>> tre_tnfa_run_parallel, input type 1        
|  tre_tnfa_run_parallel, input type 1
>>>>>>>>> length: -1                                 
|  length: -1
>>>>>>>>> pos:chr/code | states and tags             
|  pos:chr/code | states and tags
>>>>>>>>>
-------------+------------------------------------------------                 |
-------------+------------------------------------------------
>>>>>>>>> init>  3037fb88                         
|   init>  110cc3310
>>>>>>>>> 0: 3/00051 | 3037fb88/0:0                  
|    0: 3/00051 | 110cc3310/0:0
>>>>>>>>> 1: ./00046 | 3037fb88/0:0                  
|    1: ./00046 | 110cc3310/0:0
>>>>>>>>> init>  3037fb88                         
|   init>  110cc3310
>>>>>>>>> 1: ./00046 | 3037fb88/0:1                  
|    1: ./00046 | 110cc3310/0:1
>>>>>>>>> 2: 2/00050 | 3037fb88/0:1                  
|    2: 2/00050 | 110cc3310/0:1
>>>>>>>>> assertion failed                           
|  assertion failed
>>>>>>>>> init>  3037fb88                         
|   init>  110cc3310
>>>>>>>>> 2: 2/00050 | 3037fc18/0:1 3037fb88/0:2     
|    2: 2/00050 | 110cc33f0/0:1 110cc3310/0:2
>>>>>>>>> 3: ./00046 | 3037fc18/0:1 3037fb88/0:2     
|    3: ./00046 | 110cc33f0/0:1 110cc3310/0:2
>>>>>>>>> assertion failed                          
*** DIFFERENCE ***                  |   init>  110cc3310
>>>>>>>>> init>  3037fb88                         
|    3: ./00046 | 110cc3310/0:3
>>>>>>>>> 3: ./00046 | 3037fc18/0:1 3037fb88/0:3     
|    4: 3/00051 | 110cc3310/0:3
>>>>>>>>> 4: 3/00051 | 3037fc18/0:1 3037fb88/0:3     
|  assertion failed
>>>>>>>>> assertion failed                           
|   init>  110cc3310
>>>>>>>>> init>  3037fb88                         
|    4: 3/00051 | 110cc33f0/0:3 110cc3310/0:4
>>>>>>>>> 4: 3/00051 | 3037fc18/0:3 3037fb88/0:4     
|    5: /00000 | 110cc33f0/0:3 110cc3310/0:4
>>>>>>>>> 5: /00000 | 3037fc18/0:3 3037fb88/0:4 |  
init>  110cc3310
>>>>>>>>> found match 3037fd14 *** DIFFERENCE ***    
|  match end offset = -1
>>>>>>>>> match end offset = 5 *** DIFFERENCE ***    
|  tre_tnfa_run_parallel, input type 1
>>>>>>>>> tre_tnfa_run_parallel, input type 1        
|  length: -1
>>>>>>>>> length: -1                                 
|  pos:chr/code | states and tags
>>>>>>>>> pos:chr/code | states and tags             
|  -------------+------------------------------------------------
>>>>>>>>>
-------------+------------------------------------------------                 |
init>  110cc4780 110cc4668 110cc4550
>>>>>>>>> init>  303811c0 3038110c 30381058       
|  match end offset = -1
>>>>>>>>> match end offset = -1                      
|  tre_tnfa_run_parallel, input type 1
>>>>>>>>> tre_tnfa_run_parallel, input type 1        
|  length: -1
>>>>>>>>> length: -1                                 
|  pos:chr/code | states and tags
>>>>>>>>> pos:chr/code | states and tags             
|  -------------+------------------------------------------------
>>>>>>>>>
-------------+------------------------------------------------                 |
init>  110cc5700 110cc55e8 110cc54d0
>>>>>>>>>
>>>>>>>> One day further - looks like tre_compile (or
just before, after all).
>>>>>>>>
>>>>>>>> With TRE_DEBUG switched on in tre-compile.c and
tre-ast.c I see (snip)
>>>>>>>>
>>>>>>>> --- /tmp/x.32   2015-12-31 15:09:44.000000000
+0000
>>>>>>>> +++ /tmp/x.64   2015-12-31 15:09:30.000000000
+0000
>>>>>>>> @@ -1,5 +1,5 @@
>>>>>>>> - Script command is started on Thu Dec 31
15:04:39 2015.
>>>>>>>> - root at
x069:[/data/prj/cran/32/R-aix-3.2.3/src/library/tools]make sysdata
>>>>>>>> + Script command is started on Thu Dec 31
15:08:43 2015.
>>>>>>>> + root at
x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]make sysdata
>>>>>>>> installing 'sysdata.rda'
>>>>>>>> echo
"tools:::sysdata2LazyLoadDB(\"/data/prj/cran/R-3.2.3/src/library/tools/R/sysdata.rda\",\"../../../library/tools/R\")"
| \
>>>>>>>>   R_DEFAULT_PACKAGES=NULL LC_ALL=C
../../../bin/R --vanilla --slave
>>>>>>>> @@ -167,7 +167,7 @@
>>>>>>>> initial: 1/1,0, assert 0
>>>>>>>> initial: 0/0, assert 0
>>>>>>>> initial: 0/0, assert 0
>>>>>>>> - final state 30370718
>>>>>>>> + final state 110cba530
>>>>>>>> tre_compile: parsing '(^|[^%])(%%)*%V'
>>>>>>>> AST:
>>>>>>>> catenation, sub 0, 0 tags
>>>>>>>> @@ -177,7 +177,7 @@
>>>>>>>>         assertions: bol
>>>>>>>>         union, sub -1, 0 tags
>>>>>>>>           literal (, $) (0, 36), pos 0, sub -1,
0 tags
>>>>>>>> -           literal (&, M-^?) (38, 65535),
pos 0, sub -1, 0 tags
>>>>>>>> +           literal (&, M-^?) (38, -1), pos
0, sub -1, 0 tags
>>>>>>>>       iteration {0, -1}, sub -1, 0 tags, greedy
>>>>>>>>         catenation, sub 2, 0 tags
>>>>>>>>           literal (%, %) (37, 37), pos 1, sub
-1, 0 tags
>>>>>>>> @@ -197,7 +197,7 @@
>>>>>>>> Union
>>>>>>>> Literal 0-36
>>>>>>>> After union left
>>>>>>>> - Literal 38-65535
>>>>>>>> + Literal 38--1
>>>>>>>> After union right
>>>>>>>> After union right
>>>>>>>>   num_tags += 2
>>>>>>>> @@ -231,7 +231,7 @@
>>>>>>>>         assertions: bol
>>>>>>>>         union, sub -1, 0 tags
>>>>>>>>           literal (, $) (0, 36), pos 0, sub -1,
0 tags
>>>>>>>> -           literal (&, M-^?) (38, 65535),
pos 0, sub -1, 0 tags
>>>>>>>> +           literal (&, M-^?) (38, -1), pos
0, sub -1, 0 tags
>>>>>>>>       iteration {0, -1}, sub -1, 2 tags, greedy
>>>>>>>>         catenation, sub 2, 1 tags
>>>>>>>>           literal (%, %) (37, 37), pos 1, sub
-1, 1 tags
>>>>>>>> @@ -255,7 +255,7 @@
>>>>>>>> Union
>>>>>>>> Literal 0-36
>>>>>>>> After union left
>>>>>>>> - Literal 38-65535
>>>>>>>> + Literal 38--1
>>>>>>>> After union right
>>>>>>>> After union right
>>>>>>>> tre_add_tag_right: tag 3
>>>>>>>> @@ -342,7 +342,7 @@
>>>>>>>>           catenation, sub -1, 0 tags
>>>>>>>>             union, sub -1, 0 tags
>>>>>>>>               literal (, $) (0, 36), pos 0, sub
-1, 0 tags
>>>>>>>> -               literal (&, M-^?) (38,
65535), pos 0, sub -1, 0 tags
>>>>>>>> +               literal (&, M-^?) (38, -1),
pos 0, sub -1, 0 tags
>>>>>>>>             tag 4
>>>>>>>>
>>>>>>>> It seems in 32-bit mode -1 is unsigned (65535)
but -1 == -1 in 64-bit mode.
>>>>>>>>
>>>>>>>> I suspect I will "find it" - but a
proposed change is appreciated.
>>>>>>>>
>>>>>>>> Happy New Year,
>>>>>>>> Michael
>>>>>>>>
>>>>>>>> ______________________________________________
>>>>>>>> R-devel at r-project.org mailing list
>>>>>>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>>>>>>>
>>>>>>> ______________________________________________
>>>>>>> R-devel at r-project.org mailing list
>>>>>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>>>>>>
>>>>>> ______________________________________________
>>>>>> R-devel at r-project.org mailing list
>>>>>> https://stat.ethz.ch/mailman/listinfo/r-devel

Michael Felt

2016-Jan-04 22:24 UTC

head link

[Rd] R, AIX 64-bit builds - trying to understand root cause for message: "Error: Line starting 'Package: tools ...' is malformed!"

The bulk is on my forums - the final post for today is:

Results to date:

A. It looks like I am going to need a newer compiler for C - xlc/xlC V11 
apparently does not understand this code:

"/data/prj/cran/R-3.2.3/src/main/memory.c", line 2149.31: 1506-046 (S)
Syntax error.

I will have to check if R-devel has different code before asking for 
assistence.

  +2139  #ifdef HAVE_STDALIGN_H
  +2140  # include <stdalign.h>
  +2141  #endif
  +2142
  +2143  #include <stdint.h>
  +2144
  +2145  long double *R_allocLD(size_t nelem)
  +2146  {
  +2147  #if __alignof_is_defined
  +2148      // This is C11: picky compilers may warn.
  +2149      size_t ld_align = alignof(long double);
  +2150  #elif __GNUC__
  +2151      // This is C99, but do not rely on it.
  +2152      size_t ld_align = offsetof(struct { char __a; long double 
__b; }, __b);
  +2153  #else
  +2154      size_t ld_align = 0x0F; // value of x86_64, known others 
are 4 or 8
  +2155  #endif
  +2156      if (ld_align > 8) {
  +2157          uintptr_t tmp = (uintptr_t) R_alloc(nelem + 1, 
sizeof(long double));
  +2158          tmp = (tmp + ld_align - 1) & ~ld_align;
  +2159          return (long double *) tmp;
  +2160      } else {
  +2161          return (long double *) R_alloc(nelem, sizeof(long double));
  +2162      }
  +2163  }

If someone has a suggestion for how to test/fix so that I can proceed 
with an older xlc compiler, that would be great!
If not, I shall download the try and buy C compiler to test.

On 04-Jan-16 15:52, Simon Urbanek wrote:> No, no duplicate warnings.
>B. There is a very big difference in the way libraries are made when 
gcc/gfortran are not used.

When gcc is being used "everything" is being turned into a shared 
library. The only library I seem to be able
to affect via configure is libR.so (yes/no). The snip here shows that 
all members of the .a archives are
"non-shared" objects rather that a combined group of .o files into a 
single .so shared object.

So, it is not surprising that there are no duplicate symbols.

cran at x068:[/home/cran/64/R]dump -H src/*/*.a | head

src/appl/libappl.a[integrate.o]:

Loader section is not available

src/appl/libappl.a[interv.o]:

Loader section is not available

src/appl/libappl.a[maxcol.o]:>> >FYI: I shall be downloading the "try and buy" xlc and
xlfortran - and I think you will certainly prefer my packaging then as they work
without the libgc dependencies that many of the rpm packages need.
>> >
>> >And, at your option - we can take this discussion over tools -
"out of forums". Or at least start a new thread.
>> >
> We could leave the list out and create a Wiki or something with the results
of our tests.
>
>
>> >re: unsigned short - I have adopted the convention to use the *NN_t
types after running into a problem with unsigned longlong (from a very old
program). It goes back many years - and maybe they have finalized short to mean
16-bits - but I remember when short was meant to help when moving from 16-bit to
32-bit and the "word" size changed - i.e., int became 32-bit same as
long. nd for a long (no pun intended) long was 32-bit and long long was 64-bit.
Those definitions are extinct.
>> >
> I'm not sure what you refer here - the issue with TRE has nothing to do
with short - it can take any int type and like I said most platforms use
unsigned int which is big enough on all platforms.
>
>
>> >imho - the standard for wint_t is wrong as well - based on an
assumption about how "short" is defined. And, I consider it poor
practice that there are som many cases of type cast switches between ushort and
int.
>> >
> Not really - it doesn't care about short at all - note that the short
typedef is never actually used on AIX since it has wchar support so TRE is only
using int.
>
> Cheers,
> Simon
>
>

Michael Felt

2016-Feb-19 10:06 UTC

head link

[Rd] R, AIX 64-bit builds - trying to understand root cause for message: "Error: Line starting 'Package: tools ...' is malformed!"

Hi Simon,

I have been busy with work, but I finally got around to repackaging 
libiconv for AIX - in a way that both adds GNU libiconv support 
(libiconv.so.2 member) and is both 32 and 64 bit without breaking 
support for IBM iconv applications.

I concur that iconv is a pain as IBM and GNU made different decesions 
about an 'implementor' choice to call a translation to an error or not -
when the output map(?) does not include a character. IBM choose to 
output ASCII 32 (space) and GNU calls it an error. The standard states 
it is an error when a char (or code) is used in the input, but is not in 
the input table. Enough said imho - IBM and GNU differ in their approach 
and autotools test for this difference.

So, I have a way to have both approaches available via /usr/lib. For a 
bit more detail please see: http://www.aixtools.net/index.php/libiconv - 
I do hope this 'fixes' the libiconv issues! for OSS.

Using this library you should be able to replace: (what you sent in an 
earlier e-mail as your configure statement)

On 2016-01-04 15:52, Simon Urbanek wrote:>> >  I would be "pleased" if you would try packages - i.e.,
none of "Toolbox" or Perzl rpm's.
>> >  
> R requires iconv, so you have to get that somehow. I tried to bypass the
check and build it anyway against the system one, but that fails as there is a
iconv conversion further down the line that is not supported. Building iconv is
a PITA because it conflicts with the system one so other tools break when you
modify the load path. Perzl's iconv fixes that by incorporating both the
system objects and the GNU ones into the same binary and that seems to work
reliably so far. That is the only dependency I was using (his iconv doesn't
have any dependencies itself).
>
>configure CC=xlc_r CXX=xlc++_r F77=xlf_r FC=xlf95_r
LIBS='-L/opt/freeware/lib /opt/freeware/lib/libiconv.a -lpthread'
--prefix=/opt/freeware CPPFLAGS=-I/opt/freeware/include

with export OBJECT_MODE=64 in the env (and libiconv from perzl.org RPMs - iconv
is a serious pain).

with

export OBJECT_MODE=64
configure CC=xlc_r CXX=xlc++_r F77=xlf_r FC=xlf95_r LIBS='-lpthread'
--prefix=/opt/freeware

You may even be able to replace --prefix=/opt/freeware to --prefix=/opt (I use
the latter to not step on anything in /opt/freeware)

Maybe Matching Threads

Search for more possibly parallel threads

R devel - Jan 2016 - R, AIX 64-bit builds - trying to understand root cause for message: "Error: Line starting 'Package: tools ...' is malformed!"

[Rd] R, AIX 64-bit builds - trying to understand root cause for message: "Error: Line starting 'Package: tools ...' is malformed!"

[Rd] R, AIX 64-bit builds - trying to understand root cause for message: "Error: Line starting 'Package: tools ...' is malformed!"

[Rd] R, AIX 64-bit builds - trying to understand root cause for message: "Error: Line starting 'Package: tools ...' is malformed!"

[Rd] R, AIX 64-bit builds - trying to understand root cause for message: "Error: Line starting 'Package: tools ...' is malformed!"

[Rd] R, AIX 64-bit builds - trying to understand root cause for message: "Error: Line starting 'Package: tools ...' is malformed!"

Maybe Matching Threads