dovecot - Jun 2016 - fts_solr crashs

Hi,

I?ve set up in dovecot 2.2.24-1~auto+49 (from dovecot repo) fts_solr and 
fts_tika - jetti8 (from Debian Jessie) and latest tika-server running on 
a seperate machine. But if I want to rescan all messages for reindexing 
for instance all attachments with "doveadm -v index -u user at domain.tld 
INBOX" with 3137 mail in the INBOX  it counts and then by 2900 mails the 
doveadm crashes with the following error message:

doveadm(user at domain.tld): Info: INBOX: Caching mails seq=1..3137
2900/3137doveadm(user at domain.tld): Error: fts_solr: Indexing failed: 500 
Java heap space  java.lang.OutOfMemoryError: Java heap space
doveadm(user at domain.tld): Panic: file http-client-request.c: line 769 
(http_client_request_send_payload): assertion failed: (ret == 0)
doveadm(user at domain.tld): Error: Raw backtrace: 
/usr/lib/dovecot/libdovecot.so.0(+0x8dc1e) [0x7f269375ec1e] -> 
/usr/lib/dovecot/libdovecot.so.0(+0x8dc98) [0x7f269375ec98] -> 
/usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f26936fd02e] -> 
/usr/lib/dovecot/libdovecot.so.0(http_client_request_send_payload+0xc8) 
[0x7f2693716b88] -> 
/usr/lib/dovecot/modules/lib21_fts_solr_plugin.so(solr_connection_post_more+0x49)
[0x7f2691c538b9] -> 
/usr/lib/dovecot/modules/lib21_fts_solr_plugin.so(+0x361e) 
[0x7f2691c4f61e] -> 
/usr/lib/dovecot/modules/lib20_fts_plugin.so(+0x9a17) [0x7f2692ad5a17] 
-> /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0x9e20) 
[0x7f2692ad5e20] -> 
/usr/lib/dovecot/modules/lib20_fts_plugin.so(fts_build_mail+0x999) 
[0x7f2692ad6809] -> 
/usr/lib/dovecot/modules/lib20_fts_plugin.so(+0x1056e) [0x7f2692adc56e] 
-> /usr/lib/dovecot/libdovecot-storage.so.0(mail_precache+0x29) 
[0x7f2693a1f7b9] -> doveadm(+0x2e947) [0x7f26943be947] -> 
doveadm(+0x28daf) [0x7f26943b8daf] -> doveadm(+0x2989d) [0x7f26943b989d] 
-> doveadm(doveadm_cmd_ver2_to_mail_cmd_wrapper+0x278) [0x7f26943ba788] 
-> doveadm(doveadm_cmd_run_ver2+0x560) [0x7f26943c8c70] -> 
doveadm(doveadm_cmd_try_run_ver2+0x37) [0x7f26943c8cc7] -> 
doveadm(main+0x1df) [0x7f26943aa33f] -> 
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f2693347b45] 
-> doveadm(+0x1a716) [0x7f26943aa716]

I don?t see any shortages on the seperate solr machine. What could I do? 
How do I get more details to locate the problem?!

Thanks a lot and regards,

Michael

On 27.06.2016 12:11, M. Koehler wrote:
> Hi,
>
> I?ve set up in dovecot 2.2.24-1~auto+49 (from dovecot repo) fts_solr
> and fts_tika - jetti8 (from Debian Jessie) and latest tika-server
> running on a seperate machine. But if I want to rescan all messages
> for reindexing for instance all attachments with "doveadm -v index -u
> user at domain.tld INBOX" with 3137 mail in the INBOX  it counts and
then
> by 2900 mails the doveadm crashes with the following error message:
>
> doveadm(user at domain.tld): Info: INBOX: Caching mails seq=1..3137
> 2900/3137doveadm(user at domain.tld): Error: fts_solr: Indexing failed:
> 500 Java heap space  java.lang.OutOfMemoryError: Java heap space
> doveadm(user at domain.tld): Panic: file http-client-request.c: line 769
> (http_client_request_send_payload): assertion failed: (ret == 0)
> doveadm(user at domain.tld): Error: Raw backtrace:
> /usr/lib/dovecot/libdovecot.so.0(+0x8dc1e) [0x7f269375ec1e] ->
> /usr/lib/dovecot/libdovecot.so.0(+0x8dc98) [0x7f269375ec98] ->
> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f26936fd02e] ->
> /usr/lib/dovecot/libdovecot.so.0(http_client_request_send_payload+0xc8)
[0x7f2693716b88]
> ->
>
/usr/lib/dovecot/modules/lib21_fts_solr_plugin.so(solr_connection_post_more+0x49)
> [0x7f2691c538b9] ->
> /usr/lib/dovecot/modules/lib21_fts_solr_plugin.so(+0x361e)
> [0x7f2691c4f61e] ->
> /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0x9a17) [0x7f2692ad5a17]
> -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0x9e20)
> [0x7f2692ad5e20] ->
> /usr/lib/dovecot/modules/lib20_fts_plugin.so(fts_build_mail+0x999)
> [0x7f2692ad6809] ->
> /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0x1056e)
> [0x7f2692adc56e] ->
> /usr/lib/dovecot/libdovecot-storage.so.0(mail_precache+0x29)
> [0x7f2693a1f7b9] -> doveadm(+0x2e947) [0x7f26943be947] ->
> doveadm(+0x28daf) [0x7f26943b8daf] -> doveadm(+0x2989d)
> [0x7f26943b989d] ->
> doveadm(doveadm_cmd_ver2_to_mail_cmd_wrapper+0x278) [0x7f26943ba788]
> -> doveadm(doveadm_cmd_run_ver2+0x560) [0x7f26943c8c70] ->
> doveadm(doveadm_cmd_try_run_ver2+0x37) [0x7f26943c8cc7] ->
> doveadm(main+0x1df) [0x7f26943aa33f] ->
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)
> [0x7f2693347b45] -> doveadm(+0x1a716) [0x7f26943aa716]
>
> I don?t see any shortages on the seperate solr machine. What could I
> do? How do I get more details to locate the problem?!
>
> Thanks a lot and regards,
>
> Michael
Well. Your server reports:

2900/3137doveadm(user at domain.tld): Error: fts_solr: Indexing failed: 500
Java heap space  java.lang.OutOfMemoryError: Java heap space

So I guess this is your problem?

The dovecot crash is due to assertion failure, which happens because the
request failed.

Aki

Hi Aki,

Am 27.06.2016 um 11:14 schrieb Aki Tuomi:
> On 27.06.2016 12:11, M. Koehler wrote:
>> I?ve set up in dovecot 2.2.24-1~auto+49 (from dovecot repo) fts_solr
>> and fts_tika - jetti8 (from Debian Jessie) and latest tika-server
>> running on a seperate machine. But if I want to rescan all messages
>> for reindexing for instance all attachments with "doveadm -v index
-u
>> user at domain.tld INBOX" with 3137 mail in the INBOX  it counts
and then
>> by 2900 mails the doveadm crashes with the following error message:
>>
>> doveadm(user at domain.tld): Info: INBOX: Caching mails seq=1..3137
>> 2900/3137doveadm(user at domain.tld): Error: fts_solr: Indexing failed:
>> 500 Java heap space  java.lang.OutOfMemoryError: Java heap space
>> doveadm(user at domain.tld): Panic: file http-client-request.c: line
769
>> (http_client_request_send_payload): assertion failed: (ret == 0)
>> doveadm(user at domain.tld): Error: Raw backtrace:
>> /usr/lib/dovecot/libdovecot.so.0(+0x8dc1e) [0x7f269375ec1e] ->
>> /usr/lib/dovecot/libdovecot.so.0(+0x8dc98) [0x7f269375ec98] ->
>> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f26936fd02e] ->
>> /usr/lib/dovecot/libdovecot.so.0(http_client_request_send_payload+0xc8)
[0x7f2693716b88]
>> ->
>>
/usr/lib/dovecot/modules/lib21_fts_solr_plugin.so(solr_connection_post_more+0x49)
>> [0x7f2691c538b9] ->
>> /usr/lib/dovecot/modules/lib21_fts_solr_plugin.so(+0x361e)
>> [0x7f2691c4f61e] ->
>> /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0x9a17) [0x7f2692ad5a17]
>> -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0x9e20)
>> [0x7f2692ad5e20] ->
>> /usr/lib/dovecot/modules/lib20_fts_plugin.so(fts_build_mail+0x999)
>> [0x7f2692ad6809] ->
>> /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0x1056e)
>> [0x7f2692adc56e] ->
>> /usr/lib/dovecot/libdovecot-storage.so.0(mail_precache+0x29)
>> [0x7f2693a1f7b9] -> doveadm(+0x2e947) [0x7f26943be947] ->
>> doveadm(+0x28daf) [0x7f26943b8daf] -> doveadm(+0x2989d)
>> [0x7f26943b989d] ->
>> doveadm(doveadm_cmd_ver2_to_mail_cmd_wrapper+0x278) [0x7f26943ba788]
>> -> doveadm(doveadm_cmd_run_ver2+0x560) [0x7f26943c8c70] ->
>> doveadm(doveadm_cmd_try_run_ver2+0x37) [0x7f26943c8cc7] ->
>> doveadm(main+0x1df) [0x7f26943aa33f] ->
>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)
>> [0x7f2693347b45] -> doveadm(+0x1a716) [0x7f26943aa716]
>>
>> I don?t see any shortages on the seperate solr machine. What could I
>> do? How do I get more details to locate the problem?!
> Well. Your server reports:
>
> 2900/3137doveadm(user at domain.tld): Error: fts_solr: Indexing failed: 500
> Java heap space  java.lang.OutOfMemoryError: Java heap space
>
> So I guess this is your problem?
>
> The dovecot crash is due to assertion failure, which happens because the
> request failed.
Well, that?s right - I?ve checked it with the solr server but I can?t 
imagine what the problem is. I increase the java heap memory from 256MB 
to 2GB but it crashes already by 2900 messages. Will Dovecot forward 
message by message to solr server  or "as a block"? And how could I
find
out which message maybe caused the problem? I?ve checked the INBOX and 
the biggest mail got 48MB as an attachment.

Best regards,
Michael