On 02/05/18 22:17, Andr? Rodier wrote:> On 02/05/18 11:45, Andr? Rodier wrote: >> On 2018-05-01 21:29, Andr? Rodier wrote: >>> On 2018-05-01 07:22, Andr? Rodier wrote: >>>> Hello, >>>> >>>> I am trying to use Doevecot fts, with solr the script provided. >>>> >>>> To rebuild the index, I use the command: >>>> doveadm -D index -u mirina 'inbox' >>>> >>>> To rescan, I use: doveadm -D fts rescan -u mirina >>>> >>>> But when I do a search, with doveadm, the program hangs: >>>> >>>> doveadm -D search -u mirina text Madagascar >>>> >>>>> Debug: Loading modules from directory: /usr/lib/dovecot/modules >>>>> Debug: Module loaded: /usr/lib/dovecot/modules/lib10_quota_plugin.so >>>>> Debug: Module loaded: /usr/lib/dovecot/modules/lib20_fts_plugin.so >>>>> Debug: Module loaded: >>>>> /usr/lib/dovecot/modules/lib21_fts_solr_plugin.so >>>>> Debug: Loading modules from directory: >>>>> /usr/lib/dovecot/modules/doveadm >>>>> Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: >>>>> /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: >>>>> undefined symbol: acl_lookup_dict_iterate_visible_next (this is >>>>> usually intentional, so just ignore this message) >>>>> Debug: Skipping module doveadm_expire_plugin, because dlopen() >>>>> failed: >>>>> /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so: >>>>> undefined symbol: expire_set_deinit (this is usually intentional, >>>>> so just ignore this message) >>>>> Debug: Module loaded: >>>>> /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so >>>>> Debug: Module loaded: >>>>> /usr/lib/dovecot/modules/doveadm/lib10_doveadm_sieve_plugin.so >>>>> Debug: Skipping module doveadm_fts_lucene_plugin, because dlopen() >>>>> failed: >>>>> /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_lucene_plugin.so: undefined >>>>> symbol: lucene_index_iter_deinit (this is usually intentional, so >>>>> just ignore this m$ >>>>> ssage) >>>>> Debug: Module loaded: >>>>> /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so >>>>> Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() >>>>> failed: >>>>> /usr/lib/dovecot/modules/doveadm/libdoveadm_mail_crypt_plugin.so: >>>>> undefined symbol: mail_crypt_box_get_pvt_digests (this is usually >>>>> intentional, so just ignore thi$ message) >>>>> doveadm(mirina): Debug: auth USER input: mirina >>>>> home=/home/users/mirina uid=1002 gid=1001 mail=mirina at homebox.space >>>>> doveadm(mirina): Debug: Added userdb setting: >>>>> mail=mirina at homebox.space doveadm(mirina): Debug: Effective >>>>> uid=1002, gid=1001, home=/home/users/mirina >>>>> doveadm(mirina): Debug: Quota root: name=User quota backend=maildir >>>>> args>>>>> doveadm(mirina): Debug: Quota rule: root=User quota mailbox=* >>>>> bytes=2147483648 messages=0 >>>>> doveadm(mirina): Debug: Quota grace: root=User quota >>>>> bytes=214748364 (10%) >>>>> doveadm(mirina): Debug: Namespace inbox: type=private, prefix=, >>>>> sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes >>>>> location=maildir:~/mails/maildir:INDEX=~/mails/indexes/ >>>>> doveadm(mirina): Debug: maildir++: >>>>> root=/home/users/mirina/mails/maildir, >>>>> index=/home/users/mirina/mails/indexes, indexpvt=, control=, >>>>> inbox=/home/users/mirina/mails/maildir, alt>>>>> doveadm(mirina): Debug: quota: quota_over_flag check: STORAGE ret=1 >>>>> value=134 limit=2097152 >>>>> doveadm(mirina): Debug: quota: quota_over_flag check: MESSAGE ret=0 >>>>> value=3 limit=0 >>>>> doveadm(mirina): Debug: quota: quota_over_flag=0((null)) vs >>>>> currently overquota=0 >>>>> doveadm(mirina): Debug: Namespace : Using permissions from >>>>> /home/users/mirina/mails/maildir: mode=0700 gid=default >>>>> doveadm(mirina): Debug: http-client: host localhost: Host created >>>>> doveadm(mirina): Debug: http-client: host localhost: DNS lookup >>>>> successful; got 2 IPs >>>>> doveadm(mirina): Debug: http-client: peer [::1]:8080: Peer created >>>>> doveadm(mirina): Debug: http-client: queue http://localhost:8080: >>>>> Setting up connection to [::1]:8080 (1 requests pending) >>>>> doveadm(mirina): Debug: http-client: peer [::1]:8080: Linked queue >>>>> http://localhost:8080 (1 queues linked) >>>>> doveadm(mirina): Debug: http-client: queue http://localhost:8080: >>>>> Started new connection to [::1]:8080 >>>>> doveadm(mirina): Debug: http-client: request [Req1: GET >>>>> http://localhost:8080/solr/select?fl=uid,score&rows=1&sort=uid+asc&q=%7b!lucene+q.op%3dAND%7d(hdr:Madagascar+OR+body:Madagascar)&fq=%2Bbox:838a1701af1ae65a3e12000032c62018+%2Buser:mi$ >>>>> >>>>> ina]: Submitted >>>>> doveadm(mirina): Debug: http-client: Waiting for 1 requests to finish >>>>> doveadm(mirina): Debug: http-client: peer [::1]:8080: Creating 1 >>>>> new connections to handle requests (already 0 usable, connecting to >>>>> 0, closing 0) >>>>> doveadm(mirina): Debug: http-client: peer [::1]:8080: Making new >>>>> connection 1 of 1 >>>>> doveadm(mirina): Debug: http-client: conn [::1]:8080 [0]: HTTP >>>>> connection created (1 parallel connections exist) >>>>> doveadm(mirina): Debug: http-client: conn [::1]:8080 [0]: Connected >>>>> doveadm(mirina): Debug: http-client: conn [::1]:8080 [0]: Ready for >>>>> requests >>>>> doveadm(mirina): Debug: http-client: peer [::1]:8080: Successfully >>>>> connected (connections=1) >>>>> doveadm(mirina): Debug: http-client: peer [::1]:8080: Using 1 idle >>>>> connections to handle 1 requests (1 total connections ready) >>>>> doveadm(mirina): Debug: http-client: queue http://localhost:8080: >>>>> Connection to peer [::1]:8080 claimed request [Req1: GET >>>>> http://localhost:8080/solr/select?fl=uid,score&rows=1&sort=uid+asc&q=%7b!lucene+q.op%3dAND%7d(hdr:Madagascar+OR+bod$ >>>>> >>>>> :Madagascar)&fq=%2Bbox:838a1701af1ae65a3e12000032c62018+%2Buser:mirina] >>>>> >>>>> doveadm(mirina): Debug: http-client: conn [::1]:8080 [0]: Claimed >>>>> request [Req1: GET >>>>> http://localhost:8080/solr/select?fl=uid,score&rows=1&sort=uid+asc&q=%7b!lucene+q.op%3dAND%7d(hdr:Madagascar+OR+body:Madagascar)&fq=%2Bbox:838a1701af1ae6$ >>>>> >>>>> a3e12000032c62018+%2Buser:mirina] >>>>> [...] >>>>> doveadm(mirina): Debug: http-client: request [Req5: GET >>>>> http://localhost:8080/solr/select?fl=uid,score&rows=1&sort=uid+asc&q=%7b!lucene+q.op%3dAND%7d(hdr:Madagascar+OR+body:Madagascar)&fq=%2Bbox:6dd14d3aae1ae65a3e12000032c62018+%2Buser:mirina]: >>>>> Sent header >>>>> doveadm(mirina): Debug: http-client: peer [::1]:8080: No more >>>>> requests to service for this peer (1 connections exist) >>>>> doveadm(mirina): Debug: http-client: conn [::1]:8080 [0]: Got 200 >>>>> response for request [Req5: GET >>>>> http://localhost:8080/solr/select?fl=uid,score&rows=1&sort=uid+asc&q=%7b!lucene+q.op%3dAND%7d(hdr:Madagascar+OR+body:Madagascar)&fq=%2Bbox:6dd14d3aae1ae65a3e12000032c62018+%2Buser:mirina] >>>>> (took 2 ms + 0 ms in queue) >>>>> doveadm(mirina): Debug: http-client: conn [::1]:8080 [0]: Response >>>>> payload stream destroyed (0 ms after initial response) >>>>> doveadm(mirina): Debug: http-client: request [Req5: GET >>>>> http://localhost:8080/solr/select?fl=uid,score&rows=1&sort=uid+asc&q=%7b!lucene+q.op%3dAND%7d(hdr:Madagascar+OR+body:Madagascar)&fq=%2Bbox:6dd14d3aae1ae65a3e12000032c62018+%2Buser:mirina]: >>>>> Finished >>>>> doveadm(mirina): Debug: http-client: queue http://localhost:8080: >>>>> Dropping request [Req5: GET >>>>> http://localhost:8080/solr/select?fl=uid,score&rows=1&sort=uid+asc&q=%7b!lucene+q.op%3dAND%7d(hdr:Madagascar+OR+body:Madagascar)&fq=%2Bbox:6dd14d3aae1ae65a3e12000032c62018+%2Buser:mirina] >>>>> >>>>> doveadm(mirina): Debug: http-client: host localhost: Host is idle >>>>> (timeout = 1799974 msecs) >>>>> doveadm(mirina): Debug: http-client: request [Req5: GET >>>>> http://localhost:8080/solr/select?fl=uid,score&rows=1&sort=uid+asc&q=%7b!lucene+q.op%3dAND%7d(hdr:Madagascar+OR+body:Madagascar)&fq=%2Bbox:6dd14d3aae1ae65a3e12000032c62018+%2Buser:mirina]: >>>>> Free (requests left=1) >>>>> doveadm(mirina): Debug: http-client: All requests finished >>>> >>>> When I do the search with an imap client, I have this error after a >>>> long timeout: >>>> >>>>> May 01 07:08:58 homebox dovecot[5683]: imap(mirina): Panic: file >>>>> imap-client.c: line 854 (client_check_command_hangs): assertion >>>>> failed: (client->io != NULL || (client->output_cmd_lock != NULL && >>>>> client->output_cmd_lock != client->input_lock)) >>>>> May 01 07:08:58 homebox dovecot[5683]: imap(mirina): Error: Raw >>>>> backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x95e92) >>>>> [0x7f216b6f6e92] -> /usr/lib/dovecot/libdovecot.so.0(+0x95f8d) >>>>> [0x7f216b6f6f8d] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) >>>>> [0x7f216b68ca91] -> >>>>> dovecot/imap(client_continue_pending_input+0x1fc) [0x55f44a7eb15c] >>>>> -> dovecot/imap(client_input+0x120) [0x55f44a7eb460] -> >>>>> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52) >>>>> [0x7f216b70b9f2] -> >>>>> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x109) [0x7f216b70d029] >>>>> -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c) >>>>> [0x7f216b70ba8c] -> >>>>> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f216b70bc38] >>>>> -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) >>>>> [0x7f216b692fd3] -> dovecot/imap(main+0x328) [0x55f44a7dde68] -> >>>>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) >>>>> [0x7f216b2e22e1] -> dovecot/imap(_start+0x2a) [0x55f44a7ddfea] >>>>> May 01 07:08:58 homebox dovecot[5683]: imap(mirina): Fatal: master: >>>>> service(imap): child 6149 killed with signal 6 (core dumps disabled) >>>>> May 01 07:08:58 homebox in.imapproxyd[20305]: Raw_Proxy(): IMAP >>>>> server unexpectedly closed the connection on sd 11 >>>> >>>> My 10-master.conf: >>>> >>>> service decode2text { >>>> ? executable = script /usr/local/bin/decode2text.sh >>>> ? user = dovecot >>>> ? unix_listener decode2text { >>>> ??? mode = 0666 >>>> ? } >>>> } >>>> >>>> And my 90-plugins.cof: >>>> ? # Add dovecot fts config >>>> ? fts = solr >>>> ? fts_autoindex = no >>>> ? fts_solr = url=http://localhost:8080/solr/ debug >>>> ? fts_decoder = decode2text >>>> >>>> Am I doing anything wrong? >>>> >>>> Thanks for your help. >>>> >>>> Andr? >>> >>> hello again, >>> >>> The issue is not solved, but I have been able to see an error message: >>> indexer-worker(andre): Error: parser script didn't send any data >>> >>> This is weird, because 1) this error is intermittent, 2) I explicitly >>> modified the script to echo a specific word in al the cases. >>> >>> Any idea, someone? >>> >>> Thanks >> >> Hello again, >> >> I am still having this issue. >> >> The inbox I am testing has only three emails, with three attachments. >> The same document in doc, pdf and odt... >> >> Thanks for your help and advice, if you have any. >> >> Kind regards, >> Andr? > > Hello again, I really hope to find some Dovecot support on this list. > > On Debian stretch, I have tried versions from stable (2.2.27-3+deb9u2) > and from the backports (2.2.34-2~bpo9+1) > > None of the version works. The operation doveadm search, or by imap is > frozen...Even if I use the exact script provided by Dovecot decode2txt.sh. > > Any idea from the Dovecot core team would be nice. > > Thanks. > Andr? Rodier.Hello - again, I can only find a few explanations for the lack of answers to my questions.. If nobody knows how the dovecot fts code works anymore, it's OK, just just tell, so I stop searching what I am doing wrong... If Dovecot team is planing to push another fts engine, it's OK as well, but I need to know... I am loosing a lot of time on this, and I start to be upset. To summarise: - I have a custom script that extract text from attachments, for instance doc or pdf files. - I successfully tested my script on the command line, and it returns the right text every time, in a few ms. I use a one page doc or pdf, with just a paragraph. - When using the dovecot search function with the same documents attached to an email, I have a timeout, and sometimes an error message. My dovecot plugin configuration:> fts = solr > fts_solr = url=http://localhost:8080/solr/ > fts_decoder = decode2text > fts_index_timeout = 10The script is a non modified version of decode2text provided by the Dovecot installation. Please, just point me in some direction. Andr? Rodier.
Hello Could you plese 1. send the full output of doveconf -n 2. Check the SOLR logs for any errors 3. Describe your dovecot architecture, ie. if you are running a single backend or a more complex configuration. 4. Provide a backtrace of the core dump using the instructions on https://dovecot.org/bugreport.html br, Teemu On 04.05.2018 01:13, Andr? Rodier wrote:> On 02/05/18 22:17, Andr? Rodier wrote: >> On 02/05/18 11:45, Andr? Rodier wrote: >>> On 2018-05-01 21:29, Andr? Rodier wrote: >>>> On 2018-05-01 07:22, Andr? Rodier wrote: >>>>> Hello, >>>>> >>>>> I am trying to use Doevecot fts, with solr the script provided. >>>>> >>>>> To rebuild the index, I use the command: >>>>> doveadm -D index -u mirina 'inbox' >>>>> >>>>> To rescan, I use: doveadm -D fts rescan -u mirina >>>>> >>>>> But when I do a search, with doveadm, the program hangs: >>>>> >>>>> doveadm -D search -u mirina text Madagascar >>>>> >>>>>> Debug: Loading modules from directory: /usr/lib/dovecot/modules >>>>>> Debug: Module loaded: /usr/lib/dovecot/modules/lib10_quota_plugin.so >>>>>> Debug: Module loaded: /usr/lib/dovecot/modules/lib20_fts_plugin.so >>>>>> Debug: Module loaded: /usr/lib/dovecot/modules/lib21_fts_solr_plugin.so >>>>>> Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm >>>>>> Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_lookup_dict_iterate_visible_next (this is usually intentional, so just ignore this message) >>>>>> Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_deinit (this is usually intentional, so just ignore this message) >>>>>> Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so >>>>>> Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_sieve_plugin.so >>>>>> Debug: Skipping module doveadm_fts_lucene_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_lucene_plugin.so: undefined symbol: lucene_index_iter_deinit (this is usually intentional, so just ignore this m$ >>>>>> ssage) >>>>>> Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so >>>>>> Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/libdoveadm_mail_crypt_plugin.so: undefined symbol: mail_crypt_box_get_pvt_digests (this is usually intentional, so just ignore thi$ message) >>>>>> doveadm(mirina): Debug: auth USER input: mirina home=/home/users/mirina uid=1002 gid=1001 mail=mirina at homebox.space >>>>>> doveadm(mirina): Debug: Added userdb setting: mail=mirina at homebox.space doveadm(mirina): Debug: Effective uid=1002, gid=1001, home=/home/users/mirina >>>>>> doveadm(mirina): Debug: Quota root: name=User quota backend=maildir args>>>>>> doveadm(mirina): Debug: Quota rule: root=User quota mailbox=* bytes=2147483648 messages=0 >>>>>> doveadm(mirina): Debug: Quota grace: root=User quota bytes=214748364 (10%) >>>>>> doveadm(mirina): Debug: Namespace inbox: type=private, prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir:~/mails/maildir:INDEX=~/mails/indexes/ >>>>>> doveadm(mirina): Debug: maildir++: root=/home/users/mirina/mails/maildir, index=/home/users/mirina/mails/indexes, indexpvt=, control=, inbox=/home/users/mirina/mails/maildir, alt>>>>>> doveadm(mirina): Debug: quota: quota_over_flag check: STORAGE ret=1 value=134 limit=2097152 >>>>>> doveadm(mirina): Debug: quota: quota_over_flag check: MESSAGE ret=0 value=3 limit=0 >>>>>> doveadm(mirina): Debug: quota: quota_over_flag=0((null)) vs currently overquota=0 >>>>>> doveadm(mirina): Debug: Namespace : Using permissions from /home/users/mirina/mails/maildir: mode=0700 gid=default >>>>>> doveadm(mirina): Debug: http-client: host localhost: Host created >>>>>> doveadm(mirina): Debug: http-client: host localhost: DNS lookup successful; got 2 IPs >>>>>> doveadm(mirina): Debug: http-client: peer [::1]:8080: Peer created >>>>>> doveadm(mirina): Debug: http-client: queue http://localhost:8080: Setting up connection to [::1]:8080 (1 requests pending) >>>>>> doveadm(mirina): Debug: http-client: peer [::1]:8080: Linked queue http://localhost:8080 (1 queues linked) >>>>>> doveadm(mirina): Debug: http-client: queue http://localhost:8080: Started new connection to [::1]:8080 >>>>>> doveadm(mirina): Debug: http-client: request [Req1: GET http://localhost:8080/solr/select?fl=uid,score&rows=1&sort=uid+asc&q=%7b!lucene+q.op%3dAND%7d(hdr:Madagascar+OR+body:Madagascar)&fq=%2Bbox:838a1701af1ae65a3e12000032c62018+%2Buser:mi$ >>>>>> ina]: Submitted >>>>>> doveadm(mirina): Debug: http-client: Waiting for 1 requests to finish >>>>>> doveadm(mirina): Debug: http-client: peer [::1]:8080: Creating 1 new connections to handle requests (already 0 usable, connecting to 0, closing 0) >>>>>> doveadm(mirina): Debug: http-client: peer [::1]:8080: Making new connection 1 of 1 >>>>>> doveadm(mirina): Debug: http-client: conn [::1]:8080 [0]: HTTP connection created (1 parallel connections exist) >>>>>> doveadm(mirina): Debug: http-client: conn [::1]:8080 [0]: Connected >>>>>> doveadm(mirina): Debug: http-client: conn [::1]:8080 [0]: Ready for requests >>>>>> doveadm(mirina): Debug: http-client: peer [::1]:8080: Successfully connected (connections=1) >>>>>> doveadm(mirina): Debug: http-client: peer [::1]:8080: Using 1 idle connections to handle 1 requests (1 total connections ready) >>>>>> doveadm(mirina): Debug: http-client: queue http://localhost:8080: Connection to peer [::1]:8080 claimed request [Req1: GET http://localhost:8080/solr/select?fl=uid,score&rows=1&sort=uid+asc&q=%7b!lucene+q.op%3dAND%7d(hdr:Madagascar+OR+bod$ >>>>>> :Madagascar)&fq=%2Bbox:838a1701af1ae65a3e12000032c62018+%2Buser:mirina] >>>>>> doveadm(mirina): Debug: http-client: conn [::1]:8080 [0]: Claimed request [Req1: GET http://localhost:8080/solr/select?fl=uid,score&rows=1&sort=uid+asc&q=%7b!lucene+q.op%3dAND%7d(hdr:Madagascar+OR+body:Madagascar)&fq=%2Bbox:838a1701af1ae6$ >>>>>> a3e12000032c62018+%2Buser:mirina] >>>>>> [...] >>>>>> doveadm(mirina): Debug: http-client: request [Req5: GET http://localhost:8080/solr/select?fl=uid,score&rows=1&sort=uid+asc&q=%7b!lucene+q.op%3dAND%7d(hdr:Madagascar+OR+body:Madagascar)&fq=%2Bbox:6dd14d3aae1ae65a3e12000032c62018+%2Buser:mirina]: Sent header >>>>>> doveadm(mirina): Debug: http-client: peer [::1]:8080: No more requests to service for this peer (1 connections exist) >>>>>> doveadm(mirina): Debug: http-client: conn [::1]:8080 [0]: Got 200 response for request [Req5: GET http://localhost:8080/solr/select?fl=uid,score&rows=1&sort=uid+asc&q=%7b!lucene+q.op%3dAND%7d(hdr:Madagascar+OR+body:Madagascar)&fq=%2Bbox:6dd14d3aae1ae65a3e12000032c62018+%2Buser:mirina] (took 2 ms + 0 ms in queue) >>>>>> doveadm(mirina): Debug: http-client: conn [::1]:8080 [0]: Response payload stream destroyed (0 ms after initial response) >>>>>> doveadm(mirina): Debug: http-client: request [Req5: GET http://localhost:8080/solr/select?fl=uid,score&rows=1&sort=uid+asc&q=%7b!lucene+q.op%3dAND%7d(hdr:Madagascar+OR+body:Madagascar)&fq=%2Bbox:6dd14d3aae1ae65a3e12000032c62018+%2Buser:mirina]: Finished >>>>>> doveadm(mirina): Debug: http-client: queue http://localhost:8080: Dropping request [Req5: GET http://localhost:8080/solr/select?fl=uid,score&rows=1&sort=uid+asc&q=%7b!lucene+q.op%3dAND%7d(hdr:Madagascar+OR+body:Madagascar)&fq=%2Bbox:6dd14d3aae1ae65a3e12000032c62018+%2Buser:mirina] >>>>>> doveadm(mirina): Debug: http-client: host localhost: Host is idle (timeout = 1799974 msecs) >>>>>> doveadm(mirina): Debug: http-client: request [Req5: GET http://localhost:8080/solr/select?fl=uid,score&rows=1&sort=uid+asc&q=%7b!lucene+q.op%3dAND%7d(hdr:Madagascar+OR+body:Madagascar)&fq=%2Bbox:6dd14d3aae1ae65a3e12000032c62018+%2Buser:mirina]: Free (requests left=1) >>>>>> doveadm(mirina): Debug: http-client: All requests finished >>>>> >>>>> When I do the search with an imap client, I have this error after a >>>>> long timeout: >>>>> >>>>>> May 01 07:08:58 homebox dovecot[5683]: imap(mirina): Panic: file imap-client.c: line 854 (client_check_command_hangs): assertion failed: (client->io != NULL || (client->output_cmd_lock != NULL && client->output_cmd_lock != client->input_lock)) >>>>>> May 01 07:08:58 homebox dovecot[5683]: imap(mirina): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x95e92) [0x7f216b6f6e92] -> /usr/lib/dovecot/libdovecot.so.0(+0x95f8d) [0x7f216b6f6f8d] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f216b68ca91] -> dovecot/imap(client_continue_pending_input+0x1fc) [0x55f44a7eb15c] -> dovecot/imap(client_input+0x120) [0x55f44a7eb460] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52) [0x7f216b70b9f2] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x109) [0x7f216b70d029] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c) [0x7f216b70ba8c] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f216b70bc38] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f216b692fd3] -> dovecot/imap(main+0x328) [0x55f44a7dde68] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) [0x7f216b2e22e1] -> dovecot/imap(_start+0x2a) [0x55f44a7ddfea] >>>>>> May 01 07:08:58 homebox dovecot[5683]: imap(mirina): Fatal: master: service(imap): child 6149 killed with signal 6 (core dumps disabled) >>>>>> May 01 07:08:58 homebox in.imapproxyd[20305]: Raw_Proxy(): IMAP server unexpectedly closed the connection on sd 11 >>>>> >>>>> My 10-master.conf: >>>>> >>>>> service decode2text { >>>>> ? executable = script /usr/local/bin/decode2text.sh >>>>> ? user = dovecot >>>>> ? unix_listener decode2text { >>>>> ??? mode = 0666 >>>>> ? } >>>>> } >>>>> >>>>> And my 90-plugins.cof: >>>>> ? # Add dovecot fts config >>>>> ? fts = solr >>>>> ? fts_autoindex = no >>>>> ? fts_solr = url=http://localhost:8080/solr/ debug >>>>> ? fts_decoder = decode2text >>>>> >>>>> Am I doing anything wrong? >>>>> >>>>> Thanks for your help. >>>>> >>>>> Andr? >>>> >>>> hello again, >>>> >>>> The issue is not solved, but I have been able to see an error message: >>>> indexer-worker(andre): Error: parser script didn't send any data >>>> >>>> This is weird, because 1) this error is intermittent, 2) I explicitly >>>> modified the script to echo a specific word in al the cases. >>>> >>>> Any idea, someone? >>>> >>>> Thanks >>> >>> Hello again, >>> >>> I am still having this issue. >>> >>> The inbox I am testing has only three emails, with three attachments. The same document in doc, pdf and odt... >>> >>> Thanks for your help and advice, if you have any. >>> >>> Kind regards, >>> Andr? >> >> Hello again, I really hope to find some Dovecot support on this list. >> >> On Debian stretch, I have tried versions from stable (2.2.27-3+deb9u2) and from the backports (2.2.34-2~bpo9+1) >> >> None of the version works. The operation doveadm search, or by imap is frozen...Even if I use the exact script provided by Dovecot decode2txt.sh. >> >> Any idea from the Dovecot core team would be nice. >> >> Thanks. >> Andr? Rodier. > > Hello - again, > > I can only find a few explanations for the lack of answers to my questions.. > > If nobody knows how the dovecot fts code works anymore, it's OK, just just tell, so I stop searching what I am doing wrong... > > If Dovecot team is planing to push another fts engine, it's OK as well, but I need to know... > > I am loosing a lot of time on this, and I start to be upset. > > To summarise: > > - I have a custom script that extract text from attachments, for instance doc or pdf files. > - I successfully tested my script on the command line, and it returns the right text every time, in a few ms. I use a one page doc or pdf, with just a paragraph. > - When using the dovecot search function with the same documents attached to an email, I have a timeout, and sometimes an error message. > > My dovecot plugin configuration: > >> ? fts = solr >> ? fts_solr = url=http://localhost:8080/solr/ >> ? fts_decoder = decode2text >> ? fts_index_timeout = 10 > > The script is a non modified version of decode2text provided by the Dovecot installation. > > Please, just point me in some direction. > > Andr? Rodier.
Hello Teemu, I am working on this today. The errors I have might be an artefact of the virtual machine I am using (KVM), especially the multiple snapshots. I will send a confirmation when I am sure. However, I had some difficulties understanding the principle of the original script, and how it runs. Perhaps the documentation can be improved a little bit. Kind regards, Andr? On 04/05/18 07:11, Teemu Huovila wrote:> Hello > > Could you plese > 1. send the full output of doveconf -n > 2. Check the SOLR logs for any errors > 3. Describe your dovecot architecture, ie. if you are running a single backend or a more complex configuration. > 4. Provide a backtrace of the core dump using the instructions on https://dovecot.org/bugreport.html > > br, > Teemu > > On 04.05.2018 01:13, Andr? Rodier wrote: >> On 02/05/18 22:17, Andr? Rodier wrote: >>> On 02/05/18 11:45, Andr? Rodier wrote: >>>> On 2018-05-01 21:29, Andr? Rodier wrote: >>>>> On 2018-05-01 07:22, Andr? Rodier wrote: >>>>>> Hello, >>>>>> >>>>>> I am trying to use Doevecot fts, with solr the script provided. >>>>>> >>>>>> To rebuild the index, I use the command: >>>>>> doveadm -D index -u mirina 'inbox' >>>>>> >>>>>> To rescan, I use: doveadm -D fts rescan -u mirina >>>>>> >>>>>> But when I do a search, with doveadm, the program hangs: >>>>>> >>>>>> doveadm -D search -u mirina text Madagascar >>>>>> >>>>>>> Debug: Loading modules from directory: /usr/lib/dovecot/modules >>>>>>> Debug: Module loaded: /usr/lib/dovecot/modules/lib10_quota_plugin.so >>>>>>> Debug: Module loaded: /usr/lib/dovecot/modules/lib20_fts_plugin.so >>>>>>> Debug: Module loaded: /usr/lib/dovecot/modules/lib21_fts_solr_plugin.so >>>>>>> Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm >>>>>>> Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_lookup_dict_iterate_visible_next (this is usually intentional, so just ignore this message) >>>>>>> Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_deinit (this is usually intentional, so just ignore this message) >>>>>>> Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so >>>>>>> Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_sieve_plugin.so >>>>>>> Debug: Skipping module doveadm_fts_lucene_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_lucene_plugin.so: undefined symbol: lucene_index_iter_deinit (this is usually intentional, so just ignore this m$ >>>>>>> ssage) >>>>>>> Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so >>>>>>> Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/libdoveadm_mail_crypt_plugin.so: undefined symbol: mail_crypt_box_get_pvt_digests (this is usually intentional, so just ignore thi$ message) >>>>>>> doveadm(mirina): Debug: auth USER input: mirina home=/home/users/mirina uid=1002 gid=1001 mail=mirina at homebox.space >>>>>>> doveadm(mirina): Debug: Added userdb setting: mail=mirina at homebox.space doveadm(mirina): Debug: Effective uid=1002, gid=1001, home=/home/users/mirina >>>>>>> doveadm(mirina): Debug: Quota root: name=User quota backend=maildir args>>>>>>> doveadm(mirina): Debug: Quota rule: root=User quota mailbox=* bytes=2147483648 messages=0 >>>>>>> doveadm(mirina): Debug: Quota grace: root=User quota bytes=214748364 (10%) >>>>>>> doveadm(mirina): Debug: Namespace inbox: type=private, prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir:~/mails/maildir:INDEX=~/mails/indexes/ >>>>>>> doveadm(mirina): Debug: maildir++: root=/home/users/mirina/mails/maildir, index=/home/users/mirina/mails/indexes, indexpvt=, control=, inbox=/home/users/mirina/mails/maildir, alt>>>>>>> doveadm(mirina): Debug: quota: quota_over_flag check: STORAGE ret=1 value=134 limit=2097152 >>>>>>> doveadm(mirina): Debug: quota: quota_over_flag check: MESSAGE ret=0 value=3 limit=0 >>>>>>> doveadm(mirina): Debug: quota: quota_over_flag=0((null)) vs currently overquota=0 >>>>>>> doveadm(mirina): Debug: Namespace : Using permissions from /home/users/mirina/mails/maildir: mode=0700 gid=default >>>>>>> doveadm(mirina): Debug: http-client: host localhost: Host created >>>>>>> doveadm(mirina): Debug: http-client: host localhost: DNS lookup successful; got 2 IPs >>>>>>> doveadm(mirina): Debug: http-client: peer [::1]:8080: Peer created >>>>>>> doveadm(mirina): Debug: http-client: queue http://localhost:8080: Setting up connection to [::1]:8080 (1 requests pending) >>>>>>> doveadm(mirina): Debug: http-client: peer [::1]:8080: Linked queue http://localhost:8080 (1 queues linked) >>>>>>> doveadm(mirina): Debug: http-client: queue http://localhost:8080: Started new connection to [::1]:8080 >>>>>>> doveadm(mirina): Debug: http-client: request [Req1: GET http://localhost:8080/solr/select?fl=uid,score&rows=1&sort=uid+asc&q=%7b!lucene+q.op%3dAND%7d(hdr:Madagascar+OR+body:Madagascar)&fq=%2Bbox:838a1701af1ae65a3e12000032c62018+%2Buser:mi$ >>>>>>> ina]: Submitted >>>>>>> doveadm(mirina): Debug: http-client: Waiting for 1 requests to finish >>>>>>> doveadm(mirina): Debug: http-client: peer [::1]:8080: Creating 1 new connections to handle requests (already 0 usable, connecting to 0, closing 0) >>>>>>> doveadm(mirina): Debug: http-client: peer [::1]:8080: Making new connection 1 of 1 >>>>>>> doveadm(mirina): Debug: http-client: conn [::1]:8080 [0]: HTTP connection created (1 parallel connections exist) >>>>>>> doveadm(mirina): Debug: http-client: conn [::1]:8080 [0]: Connected >>>>>>> doveadm(mirina): Debug: http-client: conn [::1]:8080 [0]: Ready for requests >>>>>>> doveadm(mirina): Debug: http-client: peer [::1]:8080: Successfully connected (connections=1) >>>>>>> doveadm(mirina): Debug: http-client: peer [::1]:8080: Using 1 idle connections to handle 1 requests (1 total connections ready) >>>>>>> doveadm(mirina): Debug: http-client: queue http://localhost:8080: Connection to peer [::1]:8080 claimed request [Req1: GET http://localhost:8080/solr/select?fl=uid,score&rows=1&sort=uid+asc&q=%7b!lucene+q.op%3dAND%7d(hdr:Madagascar+OR+bod$ >>>>>>> :Madagascar)&fq=%2Bbox:838a1701af1ae65a3e12000032c62018+%2Buser:mirina] >>>>>>> doveadm(mirina): Debug: http-client: conn [::1]:8080 [0]: Claimed request [Req1: GET http://localhost:8080/solr/select?fl=uid,score&rows=1&sort=uid+asc&q=%7b!lucene+q.op%3dAND%7d(hdr:Madagascar+OR+body:Madagascar)&fq=%2Bbox:838a1701af1ae6$ >>>>>>> a3e12000032c62018+%2Buser:mirina] >>>>>>> [...] >>>>>>> doveadm(mirina): Debug: http-client: request [Req5: GET http://localhost:8080/solr/select?fl=uid,score&rows=1&sort=uid+asc&q=%7b!lucene+q.op%3dAND%7d(hdr:Madagascar+OR+body:Madagascar)&fq=%2Bbox:6dd14d3aae1ae65a3e12000032c62018+%2Buser:mirina]: Sent header >>>>>>> doveadm(mirina): Debug: http-client: peer [::1]:8080: No more requests to service for this peer (1 connections exist) >>>>>>> doveadm(mirina): Debug: http-client: conn [::1]:8080 [0]: Got 200 response for request [Req5: GET http://localhost:8080/solr/select?fl=uid,score&rows=1&sort=uid+asc&q=%7b!lucene+q.op%3dAND%7d(hdr:Madagascar+OR+body:Madagascar)&fq=%2Bbox:6dd14d3aae1ae65a3e12000032c62018+%2Buser:mirina] (took 2 ms + 0 ms in queue) >>>>>>> doveadm(mirina): Debug: http-client: conn [::1]:8080 [0]: Response payload stream destroyed (0 ms after initial response) >>>>>>> doveadm(mirina): Debug: http-client: request [Req5: GET http://localhost:8080/solr/select?fl=uid,score&rows=1&sort=uid+asc&q=%7b!lucene+q.op%3dAND%7d(hdr:Madagascar+OR+body:Madagascar)&fq=%2Bbox:6dd14d3aae1ae65a3e12000032c62018+%2Buser:mirina]: Finished >>>>>>> doveadm(mirina): Debug: http-client: queue http://localhost:8080: Dropping request [Req5: GET http://localhost:8080/solr/select?fl=uid,score&rows=1&sort=uid+asc&q=%7b!lucene+q.op%3dAND%7d(hdr:Madagascar+OR+body:Madagascar)&fq=%2Bbox:6dd14d3aae1ae65a3e12000032c62018+%2Buser:mirina] >>>>>>> doveadm(mirina): Debug: http-client: host localhost: Host is idle (timeout = 1799974 msecs) >>>>>>> doveadm(mirina): Debug: http-client: request [Req5: GET http://localhost:8080/solr/select?fl=uid,score&rows=1&sort=uid+asc&q=%7b!lucene+q.op%3dAND%7d(hdr:Madagascar+OR+body:Madagascar)&fq=%2Bbox:6dd14d3aae1ae65a3e12000032c62018+%2Buser:mirina]: Free (requests left=1) >>>>>>> doveadm(mirina): Debug: http-client: All requests finished >>>>>> >>>>>> When I do the search with an imap client, I have this error after a >>>>>> long timeout: >>>>>> >>>>>>> May 01 07:08:58 homebox dovecot[5683]: imap(mirina): Panic: file imap-client.c: line 854 (client_check_command_hangs): assertion failed: (client->io != NULL || (client->output_cmd_lock != NULL && client->output_cmd_lock != client->input_lock)) >>>>>>> May 01 07:08:58 homebox dovecot[5683]: imap(mirina): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x95e92) [0x7f216b6f6e92] -> /usr/lib/dovecot/libdovecot.so.0(+0x95f8d) [0x7f216b6f6f8d] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f216b68ca91] -> dovecot/imap(client_continue_pending_input+0x1fc) [0x55f44a7eb15c] -> dovecot/imap(client_input+0x120) [0x55f44a7eb460] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52) [0x7f216b70b9f2] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x109) [0x7f216b70d029] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c) [0x7f216b70ba8c] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f216b70bc38] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f216b692fd3] -> dovecot/imap(main+0x328) [0x55f44a7dde68] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) [0x7f216b2e22e1] -> dovecot/imap(_start+0x2a) [0x55f44a7ddfea] >>>>>>> May 01 07:08:58 homebox dovecot[5683]: imap(mirina): Fatal: master: service(imap): child 6149 killed with signal 6 (core dumps disabled) >>>>>>> May 01 07:08:58 homebox in.imapproxyd[20305]: Raw_Proxy(): IMAP server unexpectedly closed the connection on sd 11 >>>>>> >>>>>> My 10-master.conf: >>>>>> >>>>>> service decode2text { >>>>>> ? executable = script /usr/local/bin/decode2text.sh >>>>>> ? user = dovecot >>>>>> ? unix_listener decode2text { >>>>>> ??? mode = 0666 >>>>>> ? } >>>>>> } >>>>>> >>>>>> And my 90-plugins.cof: >>>>>> ? # Add dovecot fts config >>>>>> ? fts = solr >>>>>> ? fts_autoindex = no >>>>>> ? fts_solr = url=http://localhost:8080/solr/ debug >>>>>> ? fts_decoder = decode2text >>>>>> >>>>>> Am I doing anything wrong? >>>>>> >>>>>> Thanks for your help. >>>>>> >>>>>> Andr? >>>>> >>>>> hello again, >>>>> >>>>> The issue is not solved, but I have been able to see an error message: >>>>> indexer-worker(andre): Error: parser script didn't send any data >>>>> >>>>> This is weird, because 1) this error is intermittent, 2) I explicitly >>>>> modified the script to echo a specific word in al the cases. >>>>> >>>>> Any idea, someone? >>>>> >>>>> Thanks >>>> >>>> Hello again, >>>> >>>> I am still having this issue. >>>> >>>> The inbox I am testing has only three emails, with three attachments. The same document in doc, pdf and odt... >>>> >>>> Thanks for your help and advice, if you have any. >>>> >>>> Kind regards, >>>> Andr? >>> >>> Hello again, I really hope to find some Dovecot support on this list. >>> >>> On Debian stretch, I have tried versions from stable (2.2.27-3+deb9u2) and from the backports (2.2.34-2~bpo9+1) >>> >>> None of the version works. The operation doveadm search, or by imap is frozen...Even if I use the exact script provided by Dovecot decode2txt.sh. >>> >>> Any idea from the Dovecot core team would be nice. >>> >>> Thanks. >>> Andr? Rodier. >> >> Hello - again, >> >> I can only find a few explanations for the lack of answers to my questions.. >> >> If nobody knows how the dovecot fts code works anymore, it's OK, just just tell, so I stop searching what I am doing wrong... >> >> If Dovecot team is planing to push another fts engine, it's OK as well, but I need to know... >> >> I am loosing a lot of time on this, and I start to be upset. >> >> To summarise: >> >> - I have a custom script that extract text from attachments, for instance doc or pdf files. >> - I successfully tested my script on the command line, and it returns the right text every time, in a few ms. I use a one page doc or pdf, with just a paragraph. >> - When using the dovecot search function with the same documents attached to an email, I have a timeout, and sometimes an error message. >> >> My dovecot plugin configuration: >> >>> ? fts = solr >>> ? fts_solr = url=http://localhost:8080/solr/ >>> ? fts_decoder = decode2text >>> ? fts_index_timeout = 10 >> >> The script is a non modified version of decode2text provided by the Dovecot installation. >> >> Please, just point me in some direction. >> >> Andr? Rodier.