dovecot - Jan 2018 - Panic: file mail-index-util.c: line 37 (mail_index_uint32_to_offset): assertion failed: (offset < 0x40000000)

Hello,

i have a problem with a very huge mail account.

The INBOX of this account has ~1.1 - ~1.7 million mails inside. And
from time to time the index cache make problems and i got the
following in the logs:

--- Snip ---
Jan 10 20:43:04 XXXXX dovecot: imap(XXXXX): Panic: file
mail-index-util.c: line 37 (mail_index_uint32_to_offset): assertion
failed: (offset < 0x40000000)
--- Snap ---

I skip the raw backtrace to keep the post "small", but can hand it in
later if needed. Got this for UID FETCH, UID SEARCH, UID SORT, ...

The size of the file dovecot.index.cache of INBOX was in the last seen
problem situations between 1.1GB to 2.3GB.

To solve the problem the last times was to reduce the amount of mails
to under 1.7 million (seems to be the limit to trigger the problem),
delete the index cache and let him rebuild with a controled request.
But this is a odd situation (because it confused 3rd party tooling,
which access this mailbox) and i cannot find any helpful informations
to solve this problem.

Sharding it into several folders to keep the amount of mails in INBOX
small is not an option currently.

If further informations are needed, just let me know. Thanks for any help!


Kind regards
Stefan

Sorry, i forgot to mention, that the storage format is mdbox and the
version is 2.2.27-3~bpo8+1 from jessie backports.

Kind regards
Stefan

2018-01-10 21:00 GMT+01:00 Stefan Neben <stefan.neben at
gmail.com>:
> Hello,
>
> i have a problem with a very huge mail account.
>
> The INBOX of this account has ~1.1 - ~1.7 million mails inside. And
> from time to time the index cache make problems and i got the
> following in the logs:
>
> --- Snip ---
> Jan 10 20:43:04 XXXXX dovecot: imap(XXXXX): Panic: file
> mail-index-util.c: line 37 (mail_index_uint32_to_offset): assertion
> failed: (offset < 0x40000000)
> --- Snap ---
>
> I skip the raw backtrace to keep the post "small", but can hand
it in
> later if needed. Got this for UID FETCH, UID SEARCH, UID SORT, ...
>
> The size of the file dovecot.index.cache of INBOX was in the last seen
> problem situations between 1.1GB to 2.3GB.
>
> To solve the problem the last times was to reduce the amount of mails
> to under 1.7 million (seems to be the limit to trigger the problem),
> delete the index cache and let him rebuild with a controled request.
> But this is a odd situation (because it confused 3rd party tooling,
> which access this mailbox) and i cannot find any helpful informations
> to solve this problem.
>
> Sharding it into several folders to keep the amount of mails in INBOX
> small is not an option currently.
>
> If further informations are needed, just let me know. Thanks for any help!
>
>
> Kind regards
> Stefan

You are storing 1.7 million e-mails in a single mbox file. I would rather store
the archive using one file per e-mail.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20180110/18899b94/attachment.html>

On 10.01.2018 22:00, Stefan Neben wrote:
> Hello,
>
> i have a problem with a very huge mail account.
>
> The INBOX of this account has ~1.1 - ~1.7 million mails inside. And
> from time to time the index cache make problems and i got the
> following in the logs:
>
> --- Snip ---
> Jan 10 20:43:04 XXXXX dovecot: imap(XXXXX): Panic: file
> mail-index-util.c: line 37 (mail_index_uint32_to_offset): assertion
> failed: (offset < 0x40000000)
> --- Snap ---
>
> I skip the raw backtrace to keep the post "small", but can hand
it in
> later if needed. Got this for UID FETCH, UID SEARCH, UID SORT, ...
>
> The size of the file dovecot.index.cache of INBOX was in the last seen
> problem situations between 1.1GB to 2.3GB.
>
> To solve the problem the last times was to reduce the amount of mails
> to under 1.7 million (seems to be the limit to trigger the problem),
> delete the index cache and let him rebuild with a controled request.
> But this is a odd situation (because it confused 3rd party tooling,
> which access this mailbox) and i cannot find any helpful informations
> to solve this problem.
>
> Sharding it into several folders to keep the amount of mails in INBOX
> small is not an option currently.
>
> If further informations are needed, just let me know. Thanks for any help!
>
>
> Kind regards
> Stefan
Hi!

Unfortunately dovecot has an in-built limit of 2GB for the index cache
file. Your only solution is to move some of the mails to some other folder.

Aki

Hi,
> Unfortunately dovecot has an in-built limit of
> 2GB for the index cache file
okay. I assumed there is a limit but not that low, thought about 4GB
because of this:

https://www.dovecot.org/list/dovecot/2015-September/101863.html

And the strange thing is, i had this problem yesterday twice. The
first time the index cache was 1.2GB and the second time it was 1.1GB.
Is there a second factor, which could limit this?
> Your only solution is to move some of the mails to some other folder.
Yes, this was my fallback solution, but this means a lot of work to
adapt the 3rd party tooling around it :-(

2018-01-11 12:33 GMT+01:00 Aki Tuomi <aki.tuomi at
dovecot.fi>:
>
>
> On 10.01.2018 22:00, Stefan Neben wrote:
>> Hello,
>>
>> i have a problem with a very huge mail account.
>>
>> The INBOX of this account has ~1.1 - ~1.7 million mails inside. And
>> from time to time the index cache make problems and i got the
>> following in the logs:
>>
>> --- Snip ---
>> Jan 10 20:43:04 XXXXX dovecot: imap(XXXXX): Panic: file
>> mail-index-util.c: line 37 (mail_index_uint32_to_offset): assertion
>> failed: (offset < 0x40000000)
>> --- Snap ---
>>
>> I skip the raw backtrace to keep the post "small", but can
hand it in
>> later if needed. Got this for UID FETCH, UID SEARCH, UID SORT, ...
>>
>> The size of the file dovecot.index.cache of INBOX was in the last seen
>> problem situations between 1.1GB to 2.3GB.
>>
>> To solve the problem the last times was to reduce the amount of mails
>> to under 1.7 million (seems to be the limit to trigger the problem),
>> delete the index cache and let him rebuild with a controled request.
>> But this is a odd situation (because it confused 3rd party tooling,
>> which access this mailbox) and i cannot find any helpful informations
>> to solve this problem.
>>
>> Sharding it into several folders to keep the amount of mails in INBOX
>> small is not an option currently.
>>
>> If further informations are needed, just let me know. Thanks for any
help!
>>
>>
>> Kind regards
>> Stefan
> Hi!
>
> Unfortunately dovecot has an in-built limit of 2GB for the index cache
> file. Your only solution is to move some of the mails to some other folder.
>
> Aki