On 16 May 2015, at 09:39, Xin Li <delphij at delphij.net> wrote:> > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > Hi, > > On 5/15/15 10:58, Xin Li wrote: >> Hi, >> >> I have seen the following assertion fails on my own mail server >> (indexer-worker): >> >> Panic: file charset-iconv.c: line 132 (charset_to_utf8): assertion >> failed: (*src_size - pos <= CHARSET_MAX_PENDING_BUF_SIZE) > > I've managed to narrow down the issue to one mail, and can reliably > trigger the assertion with 'doveadm index -u user INBOX' with only that > email.Can you send me some mail that triggers the crash? I'm having trouble reproducing.> Maybe we should reset the iconv descriptor after each full conversion? > i.e.: > > diff -r 91bbd6753689 src/lib-mail/message-decoder.c > - --- a/src/lib-mail/message-decoder.c Thu May 14 23:49:12 2015 +0300 > +++ b/src/lib-mail/message-decoder.c Fri May 15 22:52:41 2015 -0700 > @@ -347,6 +347,8 @@ > sizeof(ctx->translation_buf)); > memcpy(ctx->translation_buf, data + pos, > ctx->translation_size); > + } else { > + charset_to_utf8_reset(ctx->charset_trans); > }I think the fix probably belongs to charset-iconv.c
I've been seeing this same error as well now:
Fatal: master: service(indexer-worker): child 20979 killed with signal 6
(core not dumped - set service indexer-worker { drop_priv_before_exec=yes })
Panic: file charset-iconv.c: line 132 (charset_to_utf8): assertion failed:
(*src_size - pos <= CHARSET_MAX_PENDING_BUF_SIZE)
It's only shown up on one particular user, and after tracking down the
message(s) that caused it I found out that there were two identical
messages with the same exact MD5/SHA hashes from our storage server in 2012
or so. Removing one of the duplicates fixed it for me. Not sure how that
happened, but wonder if it's worth checking for?
Thanks,
-c
On Sat, May 16, 2015 at 3:32 AM, Timo Sirainen <tss at iki.fi> wrote:
> On 16 May 2015, at 09:39, Xin Li <delphij at delphij.net> wrote:
> >
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA512
> >
> > Hi,
> >
> > On 5/15/15 10:58, Xin Li wrote:
> >> Hi,
> >>
> >> I have seen the following assertion fails on my own mail server
> >> (indexer-worker):
> >>
> >> Panic: file charset-iconv.c: line 132 (charset_to_utf8): assertion
> >> failed: (*src_size - pos <= CHARSET_MAX_PENDING_BUF_SIZE)
> >
> > I've managed to narrow down the issue to one mail, and can
reliably
> > trigger the assertion with 'doveadm index -u user INBOX' with
only that
> > email.
>
> Can you send me some mail that triggers the crash? I'm having trouble
> reproducing.
>
> > Maybe we should reset the iconv descriptor after each full conversion?
> > i.e.:
> >
> > diff -r 91bbd6753689 src/lib-mail/message-decoder.c
> > - --- a/src/lib-mail/message-decoder.c Thu May 14 23:49:12 2015
+0300
> > +++ b/src/lib-mail/message-decoder.c Fri May 15 22:52:41 2015 -0700
> > @@ -347,6 +347,8 @@
> > sizeof(ctx->translation_buf));
> > memcpy(ctx->translation_buf, data + pos,
> > ctx->translation_size);
> > + } else {
> > + charset_to_utf8_reset(ctx->charset_trans);
> > }
>
> I think the fix probably belongs to charset-iconv.c
>
On 21 May 2015, at 18:47, Cassidy Larson <alandaluz at gmail.com> wrote:> > I've been seeing this same error as well now: > > Fatal: master: service(indexer-worker): child 20979 killed with signal 6 (core not dumped - set service indexer-worker { drop_priv_before_exec=yes }) > Panic: file charset-iconv.c: line 132 (charset_to_utf8): assertion failed: (*src_size - pos <= CHARSET_MAX_PENDING_BUF_SIZE) > > It's only shown up on one particular user, and after tracking down the message(s) that caused it I found out that there were two identical messages with the same exact MD5/SHA hashes from our storage server in 2012 or so. Removing one of the duplicates fixed it for me. Not sure how that happened, but wonder if it's worth checking for?What OS or Linux distro are you using?> > Thanks, > > -c > > On Sat, May 16, 2015 at 3:32 AM, Timo Sirainen <tss at iki.fi <mailto:tss at iki.fi>> wrote: > On 16 May 2015, at 09:39, Xin Li <delphij at delphij.net <mailto:delphij at delphij.net>> wrote: > > > > -----BEGIN PGP SIGNED MESSAGE----- > > Hash: SHA512 > > > > Hi, > > > > On 5/15/15 10:58, Xin Li wrote: > >> Hi, > >> > >> I have seen the following assertion fails on my own mail server > >> (indexer-worker): > >> > >> Panic: file charset-iconv.c: line 132 (charset_to_utf8): assertion > >> failed: (*src_size - pos <= CHARSET_MAX_PENDING_BUF_SIZE) > > > > I've managed to narrow down the issue to one mail, and can reliably > > trigger the assertion with 'doveadm index -u user INBOX' with only that > > email. > > Can you send me some mail that triggers the crash? I'm having trouble reproducing. > > > Maybe we should reset the iconv descriptor after each full conversion? > > i.e.: > > > > diff -r 91bbd6753689 src/lib-mail/message-decoder.c > > - --- a/src/lib-mail/message-decoder.c Thu May 14 23:49:12 2015 +0300 > > +++ b/src/lib-mail/message-decoder.c Fri May 15 22:52:41 2015 -0700 > > @@ -347,6 +347,8 @@ > > sizeof(ctx->translation_buf)); > > memcpy(ctx->translation_buf, data + pos, > > ctx->translation_size); > > + } else { > > + charset_to_utf8_reset(ctx->charset_trans); > > } > > I think the fix probably belongs to charset-iconv.c >
We're using FreeBSD 10.1 Thanks On Thu, May 21, 2015 at 4:56 PM, Xin Li <delphij at delphij.net> wrote:> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > On 05/21/15 15:47, Cassidy Larson wrote: > > I've been seeing this same error as well now: > > > > Fatal: master: service(indexer-worker): child 20979 killed with > > signal 6 (core not dumped - set service indexer-worker { > > drop_priv_before_exec=yes }) Panic: file charset-iconv.c: line 132 > > (charset_to_utf8): assertion failed: (*src_size - pos <> > CHARSET_MAX_PENDING_BUF_SIZE) > > Quick question -- Are you using FreeBSD (10.0+)? > > Cheers, > - -- > Xin LI <delphij at delphij.net> https://www.delphij.net/ > FreeBSD - The Power to Serve! Live free or die > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v2.1.4 (FreeBSD) > > iQIcBAEBCgAGBQJVXmKfAAoJEJW2GBstM+nspmoP/0CKS0g8ka/4uPF+jaq7+DD6 > 1PxCufdz/p528rclbXvEK3XE++sULihNVtO24TgXMOIJZB6jCfbfo+O17N9ZS7g9 > Vaz0r0URU28joAeDX/NR1yzYpHdo277RhXiPlKuyep+fEgi6Szm/mNaR+aCtYrge > 8Qm2U2eaHFQk/2qpyvC4ZlFzXSVIR+AIMSh87YqJSWSVd3PjNDLxZLDeUaQyygG5 > 1lW3lsy0yDOTISI58rXavOi7qa4qKo0ySDOMLct3GrdXzL5S8N/Pc3RnRTP5p9dY > 22cGoH3XloijWAef93qHSuPFuC9Yiko4g/FZAtfuQg6mCETvhPDfhXBgRf3IFYAr > vuMOyBX+3TrQQI+zz3aolou10WH+E+PPbJFFZcNq4sRGQYtSXBA4RmDDKU4NgwoB > gIzS43GjPidJTxdzEC+eedmS6Rt5wvyFaDK1h5+x/mgnfEie6E9Hfk8IBqRHZMUW > HixCmBnC3p1GN+R8rbPxjuycP06bB7YXLJ28o06gM2agrGv5vu83j+WqO2g6wJIP > hRKnyCVVPUeAnkLpS+7nXZwW5R/j8M5y7V8rbiYCX4q+zWCJZl/ac1G12zwva5k1 > MwyjciDP951soMM2RYsQQSE5RxRKMC03kqGt+qnBx3jBR7afzhOmcO9gcKQJEY5R > qjwnZ0voZtV7u2Ptcv1Q > =hWt9 > -----END PGP SIGNATURE----- >
On 05/22/15 13:18, Cassidy Larson wrote:> We're using FreeBSD 10.1I see. Yes that's the same problem I have seen. There is a behavioral difference (I think it's a FreeBSD bug) between FreeBSD's iconv(3) and GNU implementation, and there is arguably a bug with Dovecot that iconv(3) state should be reset for each multipart part. The two together would trigger the problem when the previous part was ended with UTF-7 shift sequence. Timo have applied a fix (hg: 18721:9809f68aaa36) for Dovecot which should mitigates this issue at Dovecot side a few days ago. I have created a reproduction program and discussed the issue with our iconv(3) developer, Tijl Coosemans <tijl@>, who have created a patch to correct this FreeBSD iconv(3) bug (not yet committed against -HEAD but I'd expect it happening soon). Cheers, -- Xin LI <delphij at delphij.net> https://www.delphij.net/ FreeBSD - The Power to Serve! Live free or die -------------- next part -------------- A non-text attachment was scrubbed... Name: iconv-utf7.patch Type: text/x-patch Size: 2887 bytes Desc: not available URL: <http://dovecot.org/pipermail/dovecot/attachments/20150522/78061d7c/attachment.bin> -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: <http://dovecot.org/pipermail/dovecot/attachments/20150522/78061d7c/attachment.sig>