dovecot - Sep 2018 - v2.3.3 rc1 - Error: sieve: !!BUG!!: Binary compiled from dovecot.sieve is still corrupt

On 25/09/2018 22:39, Stephan Bosch wrote:
> Something mightily weird is going on at your end. It doesn't fail here
> (see below). First of all, what is your configuration (output from
> `dovecot -n`)?
You have dovecot.conf but...

$ dovecot -c dovecot.conf -n
# 2.3.3.rc1 (14e4920d8): dovecot.conf
# Pigeonhole version 0.5.2 (7704de5e)
# OS: SunOS 5.11 i86pc
# Hostname: mailhost
doveconf: Warning: please set ssl_dh=</etc/opt/xxx/dovecot/dh.pem
doveconf: Warning: You can generate it with: dd 
if=/var/opt/xxx/lib/dovecot/ssl-parameters.dat bs=1 skip=88 | openssl 
dhparam -inform der > /etc/opt/xxx/dovecot/dh.pem
mail_debug = yes
mail_gid = staff
mail_location = maildir:/path/to/%d/%n/Maildir
mail_uid = james
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope 
encoded-character vacation subaddress comparator-i;ascii-numeric 
relational regex imap4flags copy include variables body enotify 
environment mailbox date index ihave duplicate mime foreverypart extracttext
postmaster_address = postmaster at domain.tld
ssl_dh = # hidden, use -P to show it


> Also, can you make a hex dump of the binary (using `sieve-dump -h
> <your-script-binary>.svbin`).
As said the svbin is identical to the one create by the previous version.

Comparing the dump:
Block 0 differs because it has the source file name.
Block 1 is identical
Block 2 is identical. It is this block that is declared corrupt.


> Finally, can you try to explicitly delete the binary (preferably after
> preserving it elsewhere) so that it is guaranteed to be created fresh?
I did each time.


> ## Success at my end:
Spot the difference...
> Address   Line  Code
> 00000000:       DEBUG BLOCK: 3
> 00000001:       EXTENSIONS [1]:
> 00000002:         vacation
> 00000004:    3: VACATION
> 00000007:    5:   seconds: NUM 86400
Address   Line  Code
00000000:       DEBUG BLOCK: 3
00000001:       EXTENSIONS [1]:
00000002:         vacation
00000004:    2: VACATION
00000007:    4:   seconds: NUM 5
00000009:         Binary is corrupt.

The line numbers differs and 86400 is read as 5.  It is like it has 
forgotten the size of an integer or is confused about endianness.  There 
is something strange, like an #if that guesses wrong.  At least I have 
somewhere to start looking.

Thank you for checking at your end, I was worried the RC had introduced 
an error and your result suggests not.  RCs are for testing and I am.

Op 27-9-2018 om 11:42 schreef James:
> On 25/09/2018 22:39, Stephan Bosch wrote:
>
>> Something mightily weird is going on at your end. It doesn't fail
here
>> (see below). First of all, what is your configuration (output from
>> `dovecot -n`)?
>
> You have dovecot.conf but...
>
> $ dovecot -c dovecot.conf -n
> # 2.3.3.rc1 (14e4920d8): dovecot.conf
> # Pigeonhole version 0.5.2 (7704de5e)
> # OS: SunOS 5.11 i86pc
> # Hostname: mailhost
> doveconf: Warning: please set ssl_dh=</etc/opt/xxx/dovecot/dh.pem
> doveconf: Warning: You can generate it with: dd 
> if=/var/opt/xxx/lib/dovecot/ssl-parameters.dat bs=1 skip=88 | openssl 
> dhparam -inform der > /etc/opt/xxx/dovecot/dh.pem
> mail_debug = yes
> mail_gid = staff
> mail_location = maildir:/path/to/%d/%n/Maildir
> mail_uid = james
> managesieve_notify_capability = mailto
> managesieve_sieve_capability = fileinto reject envelope 
> encoded-character vacation subaddress comparator-i;ascii-numeric 
> relational regex imap4flags copy include variables body enotify 
> environment mailbox date index ihave duplicate mime foreverypart 
> extracttext
> postmaster_address = postmaster at domain.tld
> ssl_dh = # hidden, use -P to show it
>
>
>
>> Also, can you make a hex dump of the binary (using `sieve-dump -h
>> <your-script-binary>.svbin`).
>
> As said the svbin is identical to the one create by the previous version.
>
> Comparing the dump:
> Block 0 differs because it has the source file name.
> Block 1 is identical
> Block 2 is identical. It is this block that is declared corrupt.
>
>
>
>> Finally, can you try to explicitly delete the binary (preferably after
>> preserving it elsewhere) so that it is guaranteed to be created fresh?
>
> I did each time.
>
>
>
>> ## Success at my end:
>
> Spot the difference...
>
>> Address?? Line? Code
>> 00000000:?????? DEBUG BLOCK: 3
>> 00000001:?????? EXTENSIONS [1]:
>> 00000002:???????? vacation
>> 00000004:??? 3: VACATION
>> 00000007:??? 5:?? seconds: NUM 86400
>
> Address?? Line? Code
> 00000000:?????? DEBUG BLOCK: 3
> 00000001:?????? EXTENSIONS [1]:
> 00000002:???????? vacation
> 00000004:??? 2: VACATION
> 00000007:??? 4:?? seconds: NUM 5
> 00000009:???????? Binary is corrupt.
>
> The line numbers differs and 86400 is read as 5.? It is like it has 
> forgotten the size of an integer or is confused about endianness.? 
> There is something strange, like an #if that guesses wrong.? At least 
> I have somewhere to start looking.
>
> Thank you for checking at your end, I was worried the RC had 
> introduced an error and your result suggests not.? RCs are for testing 
> and I am.
The number is stored as a chain of bytes of which the most significant 
bit indicates whether the next byte still belongs to the number. If this 
bit is somehow interpreted wrong, the first byte of this number would 
read as 5, thereby returning '5' as the result and ignoring subsequent 
bytes (causing corruption at the next item to read).

Since you're using SunOS, your compiler may be doing something funky. 
Which compiler is used anyway? Perhaps different versions for the 
Dovecot releases that do and don't work?

Does it help when you change the "> 0" at the following code
position to
"!= 0" ?

https://github.com/dovecot/pigeonhole/blob/master/src/lib-sieve/sieve-binary-code.c#L300

Regards,

Stephan.

On 27/09/2018 13:40, Stephan Bosch wrote:
>> Address   Line  Code
>> 00000000:       DEBUG BLOCK: 3
>> 00000001:       EXTENSIONS [1]:
>> 00000002:         vacation
>> 00000004:    2: VACATION
>> 00000007:    4:   seconds: NUM 5
>> 00000009:         Binary is corrupt.
>>
>> The line numbers differs and 86400 is read as 5.  It is like it has
>> forgotten the size of an integer or is confused about endianness.
>> There is something strange, like an #if that guesses wrong.  At least
>> I have somewhere to start looking.
>>
>> Thank you for checking at your end, I was worried the RC had
>> introduced an error and your result suggests not.  RCs are for testing
>> and I am.
>
> The number is stored as a chain of bytes of which the most significant
> bit indicates whether the next byte still belongs to the number. If this
> bit is somehow interpreted wrong, the first byte of this number would
> read as 5, thereby returning '5' as the result and ignoring
subsequent
> bytes (causing corruption at the next item to read).
>
> Since you're using SunOS, your compiler may be doing something funky.
> Which compiler is used anyway? Perhaps different versions for the
> Dovecot releases that do and don't work?
It was studio cc.  gcc doesn't make it through configure and I didn't 
ask why.  I have some other things to do but will look at this again 
later.  Thank you for the byte code explanations.  The coding at this 
point is hard to follow with the pointers-to-functions and #defines.



James.

On Thu, Sep 27, 2018 at 10:42:16 +0100, James wrote:
> On 25/09/2018 22:39, Stephan Bosch wrote:
> 
> > Something mightily weird is going on at your end. It doesn't fail
here
> > (see below). First of all, what is your configuration (output from
> > `dovecot -n`)?
> 
> You have dovecot.conf but...
> 
> $ dovecot -c dovecot.conf -n
> # 2.3.3.rc1 (14e4920d8): dovecot.conf
> # Pigeonhole version 0.5.2 (7704de5e)
> # OS: SunOS 5.11 i86pc
Out of curiosity, is this a Solaris system or an illumos system?

Jeff.

-- 
I'm somewhere between geek and normal.
		- Linus Torvalds

On 27/09/2018 14:55, Josef 'Jeff' Sipek wrote:
> On Thu, Sep 27, 2018 at 10:42:16 +0100, James wrote:
>> $ dovecot -c dovecot.conf -n
>> # 2.3.3.rc1 (14e4920d8): dovecot.conf
>> # Pigeonhole version 0.5.2 (7704de5e)
>> # OS: SunOS 5.11 i86pc
>
> Out of curiosity, is this a Solaris system or an illumos system?
Not Illumos.
Test builds on 10 and 11.3 - RC exercise, I'll add Sparc when my machine 
is powered on.
Tested run on Solaris 10 and 11.3  - RC exercise.
Reports from 11.3, results so far the same either way.
Packages built on Solaris 10 as LCD unless there is good reason to have 
a package per rev.