Hi, I installed and tested your version, but the indexer process crashes reproducible with the following command after about 2000 messages were indexed: doveadm index -u paul at iwascoding.com -q \* Jan 14 09:26:15 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Mailbox sent: UID 2038: Opened mail because: fts indexing Jan 14 09:26:15 mail dovecot: indexer-worker: Error: terminate called after throwing an instance of 'std::bad_alloc' Jan 14 09:26:15 mail dovecot: indexer-worker: Error: what(): std::bad_alloc Jan 14 09:26:15 mail dovecot: indexer: Error: Indexer worker disconnected, discarding 48 requests for paul at iwascoding.com Jan 14 09:26:15 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Fatal: master: service(indexer-worker): child 16777 killed with signal 6 (core dumps disabled - https://dovecot.org/bugreport.html#coredumps) I tried to delete the message, but this does not help (crashes e.g. after message 2029 or 2044). Other folders with fewer messages were successfully indexed before. Sorry, could not convince dovecot to create core dumps (read the docs, changed /proc/sys/kernel/core_pattern, added LimitCORE=unlimited/infinity, even created /etc/systemd/system/dovecot.service.d/coredump.conf to no avail). Custom Dovecot 2.3.4 on Debian Stretch. Thanks, Paul> On 14. Jan 2019, at 07:42, Joan Moreau via dovecot <dovecot at dovecot.org> wrote: > > Thank you Stephan. > > The version here shall be up and running : https://github.com/grosjo/fts-xapian > > > > > > > On 2019-01-14 00:07, Stephan Bosch wrote: > >> >> >> Op 13/01/2019 om 21:25 schreef Joan Moreau via dovecot: >>> >>> I tried to combined it, the "autoreconf" errors are solved >>> >>> Now, when I type "make install", the lib is not pushed into dovecot folder, but somewhere in /usr/local/... >>> >>> How to adjust this to have it arriving in the proper folder ? >>> >> >> Depends on your system. It mostly a matter of setting a proper --prefix directory for configure, but other paths are configurable as well. I usually check what the official distribution package for Dovecot is doing and use that as a basis. >> >> For Debian I use the following configure command: >> >> ./configure --with-ldap=plugin --with-ssl=openssl --with-sql=plugin --with-lua=plugin --with-pgsql --with-mysql --with-sqlite \ >> --with-gssapi=plugin --with-solr --with-ioloop=best --enable-maintainer-mode \ >> --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib --localstatedir=/var --mandir=/usr/share/man \ >> --infodir=/usr/share/info --with-moduledir=/usr/lib/dovecot/modules --disable-rpath --disable-static >> >> Regards, >> >> Stephan >> >>> >>> On 2019-01-13 21:01, Tuomi, Aki wrote: >>> >>>> You copied your Makefile.am there. Stephan made you a working version, can you try that? >>>> (sorry for dup) >>>> Aki >>>> -------- Original message -------- >>>> From: Joan Moreau <jom at grosjo.net> >>>> Date: 13/01/2019 21:39 (GMT+02:00) >>>> To: Stephan Bosch <stephan at rename-it.nl> >>>> Cc: Aki Tuomi <aki.tuomi at open-xchange.com> >>>> Subject: Re: [FTS Xapian] Beta release >>>> >>>> I used the skeleton from Aki : https://github.com/grosjo/fts-xapian >>>> >>>> However, when I try to act as a visitor, I reach teh follwoing error: >>>> >>>> # autoreconf -vi >>>> autoreconf: Entering directory `.' >>>> autoreconf: configure.ac: not using Gettext >>>> autoreconf: running: aclocal -I m4 >>>> autoreconf: configure.ac: tracing >>>> autoreconf: running: libtoolize --copy >>>> libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, '.'. >>>> libtoolize: copying file './ltmain.sh' >>>> libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'. >>>> libtoolize: copying file 'm4/libtool.m4' >>>> libtoolize: copying file 'm4/ltoptions.m4' >>>> libtoolize: copying file 'm4/ltsugar.m4' >>>> libtoolize: copying file 'm4/ltversion.m4' >>>> libtoolize: copying file 'm4/lt~obsolete.m4' >>>> autoreconf: running: /usr/bin/autoconf >>>> autoreconf: running: /usr/bin/autoheader >>>> autoreconf: running: automake --add-missing --copy --no-force >>>> configure.ac:9: installing './compile' >>>> configure.ac:11: installing './config.guess' >>>> configure.ac:11: installing './config.sub' >>>> configure.ac:7: installing './install-sh' >>>> configure.ac:7: installing './missing' >>>> src/Makefile.am: installing './depcomp' >>>> /usr/share/automake-1.16/am/depend2.am: error: am__fastdepCXX does not appear in AM_CONDITIONAL >>>> /usr/share/automake-1.16/am/depend2.am: The usual way to define 'am__fastdepCXX' is to add 'AC_PROG_CXX' >>>> /usr/share/automake-1.16/am/depend2.am: to 'configure.ac' and run 'aclocal' and 'autoconf' again >>>> src/Makefile.am: error: C++ source seen but 'CXX' is undefined >>>> src/Makefile.am: The usual way to define 'CXX' is to add 'AC_PROG_CXX' >>>> src/Makefile.am: to 'configure.ac' and run 'autoconf' again. >>>> src/Makefile.am:11: warning: variable 'NOPLUGIN_LDFLAGS' is defined but no program or >>>> src/Makefile.am:11: library has 'NOPLUGIN' as canonical name (possible typo) >>>> autoreconf: automake failed with exit status: 1 >>>> >>>> >>>> >>>> On 2019-01-13 20:24, Stephan Bosch wrote: >>>> >>>> Oh, right, a distribution tarball doesn't include some of the >>>> necessary files for your repository like autogen.sh and >>>> .gitignore. The attached tarball includes all those and is ready >>>> for `git init`. The previous tarball was made with `make >>>> distcheck` from this one. >>>> >>>> Regards, >>>> >>>> Stephan. >>>> >>>> Op 13/01/2019 om 20:14 schreef Stephan Bosch: >>>> >>>> Hi Joan, >>>> >>>> Op 13/01/2019 om 19:03 schreef Aki Tuomi: >>>> >>>> Yes, from compiling point of view it is done. >>>> >>>> Unfortunately what is not done is all the other work >>>> involved, such as fixing all the inevitable bugs it has >>>> and maintaining it. We do not want, at this moment, take >>>> up maintaining and developing yet another FTS plugin as >>>> we have plenty of things to do already. >>>> >>>> I invite you to setup your own repository and provide >>>> this plugin from there, being the maintainer of this >>>> plugin. We can add a link to your plugin on our FTS page >>>> so people can also find it. >>>> >>>> There are other plugins like this, e.g. >>>> https://github.com/st3fan/dovecot-xaps-plugin >>>> >>>> >>>> I turned the code you provided into a separate plugin >>>> package. The distribution tarball is attached. >>>> >>>> Notable changes: >>>> >>>> - Added example copyright headers and COPYING and AUTHORS >>>> files. You should modify those to your preference. >>>> - Added README and INSTALL files (in markdown using Pandoc). >>>> Those need to be amended with details. >>>> - Amended the plugin code to display a debug message with the >>>> plugin name and version upon plugin load. >>>> >>>> I advise you to turn this into a git repository and continue from there. >>>> >>>> I do not recommend releasing this plugin with the >>>> -fpermissive flag and the resulting warning as it is now. But >>>> I'm assuming this is still a work in progress, so that is OK. >>>> >>>> Regards, >>>> >>>> Stephan. >>>> >>>> >>>> On 13 January 2019 at 19:52 Joan Moreau <jom at grosjo.net >>>> <mailto:jom at grosjo.net>> wrote: >>>> >>>> >>>> The only point here of this fts-xapian is to get rid >>>> of solr (because it >>>> is just a nightmare to setup) and squat (because it is considere >>>> obsolete). >>>> >>>> I already sent the changed in configure.ac, makefile.am, etc.. in order >>>> to include it in the dovecot, and it compiles properly >>>> >>>> The only remaining point is to push it in hte git (yes, everything is >>>> already done) >>>> >>>> On 2019-01-13 18:45, Aki Tuomi wrote: >>>> >>>> On 13 January 2019 at 17:05 Joan Moreau via >>>> dovecot <dovecot at dovecot.org >>>> <mailto:dovecot at dovecot.org>> wrote: >>>> >>>> Hi >>>> >>>> Please find attached the beta release of FTS >>>> Xapian, with the objective >>>> to replace fts_squat that is being deprecated. >>>> >>>> Configuration is exactly the same as for fts_squat: >>>> >>>> plugin { >>>> >>>> plugin = fts fts_xapian (...) >>>> fts = xapian >>>> fts_autoindex = yes >>>> fts_enforced = yes >>>> fts_xapian = partial=2 full=20 >>>> >>>> This is installed on my production server >>>> (>120Gb of mailboxes), and I >>>> will observe it during the coming days. >>>> >>>> I will definitely appreciate that this is added in the core git of >>>> docevot, in order to have a versionning of >>>> it, to remove squat and let >>>> basic users able to avoid Solr alternative as much as possible. >>>> >>>> Thanks >>>> >>>> JM >>>> >>>> Hi! >>>> >>>> I still recommend you setup a, say, github >>>> repository for your plugin. We are not able to >>>> currently include your work in dovecot core as it >>>> is more work than just pushing the code into the >>>> repo. Maybe it can be included in the future. >>>> >>>> If you want, I can help you in setting up the >>>> required configuration scripts and such to make >>>> it possible to compile it as plugin. >>>> >>>> Then anyone can download it and install it for >>>> their dovecot, even if dovecot itself has been >>>> installed from packages, and also makes it >>>> possible for package maintainers to consider >>>> including it in distributions. >>>> >>>> Aki >>>> >> >>Paul Hecker paul at iwascoding.com http://www.iwascoding.com iwascoding GmbH Weserstr 18, 10247 Berlin HRB 97269 B Amtsgericht Berlin-Charlottenburg Gesch?ftsf?hrer: Paul Hecker, Ilja Iwas -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4127 bytes Desc: not available URL: <https://dovecot.org/pipermail/dovecot/attachments/20190114/d26c219c/attachment.p7s>
Can you send the log part that includes the "init" of the plugins (something similar as below) ? WHich version of Xapian are you on ? Jan 14 09:10:04 gjserver dovecot[31082]: indexer-worker(admin at grosjo.net)<14725><apZfHexRPFy9QAAA0thIag:UL+TNOxRPFyFOQAA0thIag>: FTS Xapian: Partial=2, Full=20 DB_PATH=/data/mail/grosjo.net/admin/xapian-indexes Jan 14 09:10:04 gjserver dovecot[31082]: indexer-worker(admin at grosjo.net)<14725><apZfHexRPFy9QAAA0thIag:UL+TNOxRPFyFOQAA0thIag>: FTS Xapian : Mailbox Mail : Last UID=815055 Jan 14 09:10:04 gjserver dovecot[31082]: indexer-worker(admin at grosjo.net)<14725><apZfHexRPFy9QAAA0thIag:UL+TNOxRPFyFOQAA0thIag>: FTS Xapian : Mailbox Mail : Last UID=815055 Jan 14 09:10:04 gjserver dovecot[31082]: indexer-worker(admin at grosjo.net)<14725><apZfHexRPFy9QAAA0thIag:UL+TNOxRPFyFOQAA0thIag>: Opening RW /data/mail/grosjo.net/admin/xapian-indexes/db_5c935034609bc14c0e550000d6a3092d On 2019-01-14 10:08, Paul Hecker via dovecot wrote:> Hi, > > I installed and tested your version, but the indexer process crashes reproducible with the following command after about 2000 messages were indexed: > > doveadm index -u paul at iwascoding.com -q \* > > Jan 14 09:26:15 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Mailbox sent: UID 2038: Opened mail because: fts indexing > Jan 14 09:26:15 mail dovecot: indexer-worker: Error: terminate called after throwing an instance of 'std::bad_alloc' > Jan 14 09:26:15 mail dovecot: indexer-worker: Error: what(): std::bad_alloc > Jan 14 09:26:15 mail dovecot: indexer: Error: Indexer worker disconnected, discarding 48 requests for paul at iwascoding.com > Jan 14 09:26:15 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Fatal: master: service(indexer-worker): child 16777 killed with signal 6 (core dumps disabled - https://dovecot.org/bugreport.html#coredumps) > > I tried to delete the message, but this does not help (crashes e.g. after message 2029 or 2044). Other folders with fewer messages were successfully indexed before. > > Sorry, could not convince dovecot to create core dumps (read the docs, changed /proc/sys/kernel/core_pattern, added LimitCORE=unlimited/infinity, even created /etc/systemd/system/dovecot.service.d/coredump.conf to no avail). Custom Dovecot 2.3.4 on Debian Stretch. > > Thanks, > Paul > > On 14. Jan 2019, at 07:42, Joan Moreau via dovecot <dovecot at dovecot.org> wrote: > > Thank you Stephan. > > The version here shall be up and running : https://github.com/grosjo/fts-xapian > > On 2019-01-14 00:07, Stephan Bosch wrote: > > Op 13/01/2019 om 21:25 schreef Joan Moreau via dovecot: > I tried to combined it, the "autoreconf" errors are solved > > Now, when I type "make install", the lib is not pushed into dovecot folder, but somewhere in /usr/local/... > > How to adjust this to have it arriving in the proper folder ? > > Depends on your system. It mostly a matter of setting a proper --prefix directory for configure, but other paths are configurable as well. I usually check what the official distribution package for Dovecot is doing and use that as a basis. > > For Debian I use the following configure command: > > ./configure --with-ldap=plugin --with-ssl=openssl --with-sql=plugin --with-lua=plugin --with-pgsql --with-mysql --with-sqlite \ > --with-gssapi=plugin --with-solr --with-ioloop=best --enable-maintainer-mode \ > --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib --localstatedir=/var --mandir=/usr/share/man \ > --infodir=/usr/share/info --with-moduledir=/usr/lib/dovecot/modules --disable-rpath --disable-static > > Regards, > > Stephan > > On 2019-01-13 21:01, Tuomi, Aki wrote: > > You copied your Makefile.am there. Stephan made you a working version, can you try that? > (sorry for dup) > Aki > -------- Original message -------- > From: Joan Moreau <jom at grosjo.net> > Date: 13/01/2019 21:39 (GMT+02:00) > To: Stephan Bosch <stephan at rename-it.nl> > Cc: Aki Tuomi <aki.tuomi at open-xchange.com> > Subject: Re: [FTS Xapian] Beta release > > I used the skeleton from Aki : https://github.com/grosjo/fts-xapian > > However, when I try to act as a visitor, I reach teh follwoing error: > > # autoreconf -vi > autoreconf: Entering directory `.' > autoreconf: configure.ac: not using Gettext > autoreconf: running: aclocal -I m4 > autoreconf: configure.ac: tracing > autoreconf: running: libtoolize --copy > libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, '.'. > libtoolize: copying file './ltmain.sh' > libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'. > libtoolize: copying file 'm4/libtool.m4' > libtoolize: copying file 'm4/ltoptions.m4' > libtoolize: copying file 'm4/ltsugar.m4' > libtoolize: copying file 'm4/ltversion.m4' > libtoolize: copying file 'm4/lt~obsolete.m4' > autoreconf: running: /usr/bin/autoconf > autoreconf: running: /usr/bin/autoheader > autoreconf: running: automake --add-missing --copy --no-force > configure.ac:9: installing './compile' > configure.ac:11: installing './config.guess' > configure.ac:11: installing './config.sub' > configure.ac:7: installing './install-sh' > configure.ac:7: installing './missing' > src/Makefile.am: installing './depcomp' > /usr/share/automake-1.16/am/depend2.am: error: am__fastdepCXX does not appear in AM_CONDITIONAL > /usr/share/automake-1.16/am/depend2.am: The usual way to define 'am__fastdepCXX' is to add 'AC_PROG_CXX' > /usr/share/automake-1.16/am/depend2.am: to 'configure.ac' and run 'aclocal' and 'autoconf' again > src/Makefile.am: error: C++ source seen but 'CXX' is undefined > src/Makefile.am: The usual way to define 'CXX' is to add 'AC_PROG_CXX' > src/Makefile.am: to 'configure.ac' and run 'autoconf' again. > src/Makefile.am:11: warning: variable 'NOPLUGIN_LDFLAGS' is defined but no program or > src/Makefile.am:11: library has 'NOPLUGIN' as canonical name (possible typo) > autoreconf: automake failed with exit status: 1 > > On 2019-01-13 20:24, Stephan Bosch wrote: > > Oh, right, a distribution tarball doesn't include some of the > necessary files for your repository like autogen.sh and > .gitignore. The attached tarball includes all those and is ready > for `git init`. The previous tarball was made with `make > distcheck` from this one. > > Regards, > > Stephan. > > Op 13/01/2019 om 20:14 schreef Stephan Bosch: > > Hi Joan, > > Op 13/01/2019 om 19:03 schreef Aki Tuomi: > > Yes, from compiling point of view it is done. > > Unfortunately what is not done is all the other work > involved, such as fixing all the inevitable bugs it has > and maintaining it. We do not want, at this moment, take > up maintaining and developing yet another FTS plugin as > we have plenty of things to do already. > > I invite you to setup your own repository and provide > this plugin from there, being the maintainer of this > plugin. We can add a link to your plugin on our FTS page > so people can also find it. > > There are other plugins like this, e.g. > https://github.com/st3fan/dovecot-xaps-plugin > > I turned the code you provided into a separate plugin > package. The distribution tarball is attached. > > Notable changes: > > - Added example copyright headers and COPYING and AUTHORS > files. You should modify those to your preference. > - Added README and INSTALL files (in markdown using Pandoc). > Those need to be amended with details. > - Amended the plugin code to display a debug message with the > plugin name and version upon plugin load. > > I advise you to turn this into a git repository and continue from there. > > I do not recommend releasing this plugin with the > -fpermissive flag and the resulting warning as it is now. But > I'm assuming this is still a work in progress, so that is OK. > > Regards, > > Stephan. > > On 13 January 2019 at 19:52 Joan Moreau <jom at grosjo.net > <mailto:jom at grosjo.net>> wrote: > > The only point here of this fts-xapian is to get rid > of solr (because it > is just a nightmare to setup) and squat (because it is considere > obsolete). > > I already sent the changed in configure.ac, makefile.am, etc.. in order > to include it in the dovecot, and it compiles properly > > The only remaining point is to push it in hte git (yes, everything is > already done) > > On 2019-01-13 18:45, Aki Tuomi wrote: > > On 13 January 2019 at 17:05 Joan Moreau via > dovecot <dovecot at dovecot.org > <mailto:dovecot at dovecot.org>> wrote: > > Hi > > Please find attached the beta release of FTS > Xapian, with the objective > to replace fts_squat that is being deprecated. > > Configuration is exactly the same as for fts_squat: > > plugin { > > plugin = fts fts_xapian (...) > fts = xapian > fts_autoindex = yes > fts_enforced = yes > fts_xapian = partial=2 full=20 > > This is installed on my production server > (>120Gb of mailboxes), and I > will observe it during the coming days. > > I will definitely appreciate that this is added in the core git of > docevot, in order to have a versionning of > it, to remove squat and let > basic users able to avoid Solr alternative as much as possible. > > Thanks > > JM > > Hi! > > I still recommend you setup a, say, github > repository for your plugin. We are not able to > currently include your work in dovecot core as it > is more work than just pushing the code into the > repo. Maybe it can be included in the future. > > If you want, I can help you in setting up the > required configuration scripts and such to make > it possible to compile it as plugin. > > Then anyone can download it and install it for > their dovecot, even if dovecot itself has been > installed from packages, and also makes it > possible for package maintainers to consider > including it in distributions. > > AkiPaul Hecker paul at iwascoding.com http://www.iwascoding.com iwascoding GmbH Weserstr 18, 10247 Berlin HRB 97269 B Amtsgericht Berlin-Charlottenburg Gesch?ftsf?hrer: Paul Hecker, Ilja Iwas -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://dovecot.org/pipermail/dovecot/attachments/20190114/8c8ed457/attachment-0001.html>
Here it is: Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Effective uid=8, gid=8, home=/var/spool/mail/iwascoding/paul Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Quota root: name=User quota backend=dict args=:file:/var/spool/mail/iwascoding/paul/dovecot-quota Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Quota rule: root=User quota mailbox=* bytes=2147483648 messages=0 Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Quota rule: root=User quota mailbox=* bytes=2147483648 messages=60000 Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Quota grace: root=User quota bytes=214748364 (10%) Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: dict quota: user=paul at iwascoding.com, uri=file:/var/spool/mail/iwascoding/paul/dovecot-quota, noenforcing=0 Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Namespace inbox: type=private, prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes location=mdbox:~/mdbox Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: fs: root=/var/spool/mail/iwascoding/paul/mdbox, index=, indexpvt=, control=, inbox=, altJan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: FTS Xapian: Partial=2, Full=20 DB_PATH=/var/spool/mail/iwascoding/paul/mdbox/xapian-indexes Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: quota: quota_over_flag check: quota_over_script unset - skipping Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Mailbox sent: Mailbox opened because: indexing Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: FTS Xapian : Mailbox sent : Last UID=0 Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: FTS Xapian : Mailbox sent : Last UID=0 Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Namespace : Using permissions from /var/spool/mail/iwascoding/paul/mdbox: mode=0700 gid=default Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Mailbox sent: UID 1: Opened mail because: fts indexing Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Opening RW /var/spool/mail/iwascoding/paul/mdbox/xapian-indexes/db_9ddfe10d8a8a8a568c120000654d370e Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Mailbox sent: UID 2: Opened mail because: fts indexing Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Mailbox sent: UID 3: Opened mail because: fts indexing Thank you!> On 14. Jan 2019, at 10:11, Joan Moreau via dovecot <dovecot at dovecot.org> wrote: > > Can you send the log part that includes the "init" of the plugins (something similar as below) ? > > WHich version of Xapian are you on ? > > Jan 14 09:10:04 gjserver dovecot[31082]: indexer-worker(admin at grosjo.net)<14725><apZfHexRPFy9QAAA0thIag:UL+TNOxRPFyFOQAA0thIag>: FTS Xapian: Partial=2, Full=20 DB_PATH=/data/mail/grosjo.net/admin/xapian-indexes > Jan 14 09:10:04 gjserver dovecot[31082]: indexer-worker(admin at grosjo.net)<14725><apZfHexRPFy9QAAA0thIag:UL+TNOxRPFyFOQAA0thIag>: FTS Xapian : Mailbox Mail : Last UID=815055 > Jan 14 09:10:04 gjserver dovecot[31082]: indexer-worker(admin at grosjo.net)<14725><apZfHexRPFy9QAAA0thIag:UL+TNOxRPFyFOQAA0thIag>: FTS Xapian : Mailbox Mail : Last UID=815055 > Jan 14 09:10:04 gjserver dovecot[31082]: indexer-worker(admin at grosjo.net)<14725><apZfHexRPFy9QAAA0thIag:UL+TNOxRPFyFOQAA0thIag>: Opening RW /data/mail/grosjo.net/admin/xapian-indexes/db_5c935034609bc14c0e550000d6a3092d > > > > > > > On 2019-01-14 10:08, Paul Hecker via dovecot wrote: > >> Hi, >> >> I installed and tested your version, but the indexer process crashes reproducible with the following command after about 2000 messages were indexed: >> >> doveadm index -u paul at iwascoding.com <mailto:paul at iwascoding.com> -q \* >> >> Jan 14 09:26:15 mail dovecot: indexer-worker(paul at iwascoding.com <mailto:paul at iwascoding.com>)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Mailbox sent: UID 2038: Opened mail because: fts indexing >> Jan 14 09:26:15 mail dovecot: indexer-worker: Error: terminate called after throwing an instance of 'std::bad_alloc' >> Jan 14 09:26:15 mail dovecot: indexer-worker: Error: what(): std::bad_alloc >> Jan 14 09:26:15 mail dovecot: indexer: Error: Indexer worker disconnected, discarding 48 requests for paul at iwascoding.com <mailto:paul at iwascoding.com> >> Jan 14 09:26:15 mail dovecot: indexer-worker(paul at iwascoding.com <mailto:paul at iwascoding.com>)<16777><IKpfOqBHPFyJQQAADYqDFA>: Fatal: master: service(indexer-worker): child 16777 killed with signal 6 (core dumps disabled - https://dovecot.org/bugreport.html#coredumps <https://dovecot.org/bugreport.html#coredumps>) >> >> I tried to delete the message, but this does not help (crashes e.g. after message 2029 or 2044). Other folders with fewer messages were successfully indexed before. >> >> Sorry, could not convince dovecot to create core dumps (read the docs, changed /proc/sys/kernel/core_pattern, added LimitCORE=unlimited/infinity, even created /etc/systemd/system/dovecot.service.d/coredump.conf to no avail). Custom Dovecot 2.3.4 on Debian Stretch. >> >> Thanks, >> Paul >> >> >>> On 14. Jan 2019, at 07:42, Joan Moreau via dovecot <dovecot at dovecot.org <mailto:dovecot at dovecot.org>> wrote: >>> >>> Thank you Stephan. >>> >>> The version here shall be up and running : https://github.com/grosjo/fts-xapian <https://github.com/grosjo/fts-xapian> >>> >>> >>> >>> >>> >>> >>> On 2019-01-14 00:07, Stephan Bosch wrote: >>> >>>> >>>> >>>> Op 13/01/2019 om 21:25 schreef Joan Moreau via dovecot: >>>>> >>>>> >>>>> I tried to combined it, the "autoreconf" errors are solved >>>>> >>>>> Now, when I type "make install", the lib is not pushed into dovecot folder, but somewhere in /usr/local/... >>>>> >>>>> How to adjust this to have it arriving in the proper folder ? >>>>> >>>> >>>> Depends on your system. It mostly a matter of setting a proper --prefix directory for configure, but other paths are configurable as well. I usually check what the official distribution package for Dovecot is doing and use that as a basis. >>>> >>>> For Debian I use the following configure command: >>>> >>>> ./configure --with-ldap=plugin --with-ssl=openssl --with-sql=plugin --with-lua=plugin --with-pgsql --with-mysql --with-sqlite \ >>>> --with-gssapi=plugin --with-solr --with-ioloop=best --enable-maintainer-mode \ >>>> --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib --localstatedir=/var --mandir=/usr/share/man \ >>>> --infodir=/usr/share/info --with-moduledir=/usr/lib/dovecot/modules --disable-rpath --disable-static >>>> >>>> Regards, >>>> >>>> Stephan >>>> >>>>> >>>>> On 2019-01-13 21:01, Tuomi, Aki wrote: >>>>> >>>>>> You copied your Makefile.am there. Stephan made you a working version, can you try that? >>>>>> (sorry for dup) >>>>>> Aki >>>>>> -------- Original message -------- >>>>>> From: Joan Moreau <jom at grosjo.net <mailto:jom at grosjo.net>> >>>>>> Date: 13/01/2019 21:39 (GMT+02:00) >>>>>> To: Stephan Bosch <stephan at rename-it.nl <mailto:stephan at rename-it.nl>> >>>>>> Cc: Aki Tuomi <aki.tuomi at open-xchange.com <mailto:aki.tuomi at open-xchange.com>> >>>>>> Subject: Re: [FTS Xapian] Beta release >>>>>> >>>>>> I used the skeleton from Aki : https://github.com/grosjo/fts-xapian <https://github.com/grosjo/fts-xapian> >>>>>> >>>>>> However, when I try to act as a visitor, I reach teh follwoing error: >>>>>> >>>>>> # autoreconf -vi >>>>>> autoreconf: Entering directory `.' >>>>>> autoreconf: configure.ac: not using Gettext >>>>>> autoreconf: running: aclocal -I m4 >>>>>> autoreconf: configure.ac: tracing >>>>>> autoreconf: running: libtoolize --copy >>>>>> libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, '.'. >>>>>> libtoolize: copying file './ltmain.sh' >>>>>> libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'. >>>>>> libtoolize: copying file 'm4/libtool.m4' >>>>>> libtoolize: copying file 'm4/ltoptions.m4' >>>>>> libtoolize: copying file 'm4/ltsugar.m4' >>>>>> libtoolize: copying file 'm4/ltversion.m4' >>>>>> libtoolize: copying file 'm4/lt~obsolete.m4' >>>>>> autoreconf: running: /usr/bin/autoconf >>>>>> autoreconf: running: /usr/bin/autoheader >>>>>> autoreconf: running: automake --add-missing --copy --no-force >>>>>> configure.ac:9: installing './compile' >>>>>> configure.ac:11: installing './config.guess' >>>>>> configure.ac:11: installing './config.sub' >>>>>> configure.ac:7: installing './install-sh' >>>>>> configure.ac:7: installing './missing' >>>>>> src/Makefile.am: installing './depcomp' >>>>>> /usr/share/automake-1.16/am/depend2.am: error: am__fastdepCXX does not appear in AM_CONDITIONAL >>>>>> /usr/share/automake-1.16/am/depend2.am: The usual way to define 'am__fastdepCXX' is to add 'AC_PROG_CXX' >>>>>> /usr/share/automake-1.16/am/depend2.am: to 'configure.ac' and run 'aclocal' and 'autoconf' again >>>>>> src/Makefile.am: error: C++ source seen but 'CXX' is undefined >>>>>> src/Makefile.am: The usual way to define 'CXX' is to add 'AC_PROG_CXX' >>>>>> src/Makefile.am: to 'configure.ac' and run 'autoconf' again. >>>>>> src/Makefile.am:11: warning: variable 'NOPLUGIN_LDFLAGS' is defined but no program or >>>>>> src/Makefile.am:11: library has 'NOPLUGIN' as canonical name (possible typo) >>>>>> autoreconf: automake failed with exit status: 1 >>>>>> >>>>>> >>>>>> >>>>>> On 2019-01-13 20:24, Stephan Bosch wrote: >>>>>> >>>>>> Oh, right, a distribution tarball doesn't include some of the >>>>>> necessary files for your repository like autogen.sh and >>>>>> .gitignore. The attached tarball includes all those and is ready >>>>>> for `git init`. The previous tarball was made with `make >>>>>> distcheck` from this one. >>>>>> >>>>>> Regards, >>>>>> >>>>>> Stephan. >>>>>> >>>>>> Op 13/01/2019 om 20:14 schreef Stephan Bosch: >>>>>> >>>>>> Hi Joan, >>>>>> >>>>>> Op 13/01/2019 om 19:03 schreef Aki Tuomi: >>>>>> >>>>>> Yes, from compiling point of view it is done. >>>>>> >>>>>> Unfortunately what is not done is all the other work >>>>>> involved, such as fixing all the inevitable bugs it has >>>>>> and maintaining it. We do not want, at this moment, take >>>>>> up maintaining and developing yet another FTS plugin as >>>>>> we have plenty of things to do already. >>>>>> >>>>>> I invite you to setup your own repository and provide >>>>>> this plugin from there, being the maintainer of this >>>>>> plugin. We can add a link to your plugin on our FTS page >>>>>> so people can also find it. >>>>>> >>>>>> There are other plugins like this, e.g. >>>>>> https://github.com/st3fan/dovecot-xaps-plugin <https://github.com/st3fan/dovecot-xaps-plugin> >>>>>> >>>>>> >>>>>> I turned the code you provided into a separate plugin >>>>>> package. The distribution tarball is attached. >>>>>> >>>>>> Notable changes: >>>>>> >>>>>> - Added example copyright headers and COPYING and AUTHORS >>>>>> files. You should modify those to your preference. >>>>>> - Added README and INSTALL files (in markdown using Pandoc). >>>>>> Those need to be amended with details. >>>>>> - Amended the plugin code to display a debug message with the >>>>>> plugin name and version upon plugin load. >>>>>> >>>>>> I advise you to turn this into a git repository and continue from there. >>>>>> >>>>>> I do not recommend releasing this plugin with the >>>>>> -fpermissive flag and the resulting warning as it is now. But >>>>>> I'm assuming this is still a work in progress, so that is OK. >>>>>> >>>>>> Regards, >>>>>> >>>>>> Stephan. >>>>>> >>>>>> >>>>>> On 13 January 2019 at 19:52 Joan Moreau <jom at grosjo.net <mailto:jom at grosjo.net> >>>>>> <mailto:jom at grosjo.net <mailto:jom at grosjo.net>>> wrote: >>>>>> >>>>>> >>>>>> The only point here of this fts-xapian is to get rid >>>>>> of solr (because it >>>>>> is just a nightmare to setup) and squat (because it is considere >>>>>> obsolete). >>>>>> >>>>>> I already sent the changed in configure.ac, makefile.am, etc.. in order >>>>>> to include it in the dovecot, and it compiles properly >>>>>> >>>>>> The only remaining point is to push it in hte git (yes, everything is >>>>>> already done) >>>>>> >>>>>> On 2019-01-13 18:45, Aki Tuomi wrote: >>>>>> >>>>>> On 13 January 2019 at 17:05 Joan Moreau via >>>>>> dovecot <dovecot at dovecot.org <mailto:dovecot at dovecot.org> >>>>>> <mailto:dovecot at dovecot.org <mailto:dovecot at dovecot.org>>> wrote: >>>>>> >>>>>> Hi >>>>>> >>>>>> Please find attached the beta release of FTS >>>>>> Xapian, with the objective >>>>>> to replace fts_squat that is being deprecated. >>>>>> >>>>>> Configuration is exactly the same as for fts_squat: >>>>>> >>>>>> plugin { >>>>>> >>>>>> plugin = fts fts_xapian (...) >>>>>> fts = xapian >>>>>> fts_autoindex = yes >>>>>> fts_enforced = yes >>>>>> fts_xapian = partial=2 full=20 >>>>>> >>>>>> This is installed on my production server >>>>>> (>120Gb of mailboxes), and I >>>>>> will observe it during the coming days. >>>>>> >>>>>> I will definitely appreciate that this is added in the core git of >>>>>> docevot, in order to have a versionning of >>>>>> it, to remove squat and let >>>>>> basic users able to avoid Solr alternative as much as possible. >>>>>> >>>>>> Thanks >>>>>> >>>>>> JM >>>>>> >>>>>> Hi! >>>>>> >>>>>> I still recommend you setup a, say, github >>>>>> repository for your plugin. We are not able to >>>>>> currently include your work in dovecot core as it >>>>>> is more work than just pushing the code into the >>>>>> repo. Maybe it can be included in the future. >>>>>> >>>>>> If you want, I can help you in setting up the >>>>>> required configuration scripts and such to make >>>>>> it possible to compile it as plugin. >>>>>> >>>>>> Then anyone can download it and install it for >>>>>> their dovecot, even if dovecot itself has been >>>>>> installed from packages, and also makes it >>>>>> possible for package maintainers to consider >>>>>> including it in distributions. >>>>>> >>>>>> Aki >>>>>> >>>> >>>> >> >> >> Paul Hecker >> paul at iwascoding.com <mailto:paul at iwascoding.com> >> http://www.iwascoding.com <http://www.iwascoding.com/> >> >> iwascoding GmbH >> Weserstr 18, 10247 Berlin >> HRB 97269 B Amtsgericht Berlin-Charlottenburg >> Gesch?ftsf?hrer: Paul Hecker, Ilja Iwas >> >> >> >> >> >> >> >>Paul Hecker paul at iwascoding.com <mailto:paul at iwascoding.com> http://www.iwascoding.com <http://www.iwascoding.com/> iwascoding GmbH Weserstr 18, 10247 Berlin HRB 97269 B Amtsgericht Berlin-Charlottenburg Gesch?ftsf?hrer: Paul Hecker, Ilja Iwas -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://dovecot.org/pipermail/dovecot/attachments/20190114/70ee63d5/attachment-0001.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4127 bytes Desc: not available URL: <https://dovecot.org/pipermail/dovecot/attachments/20190114/70ee63d5/attachment-0001.p7s>
Here it is: Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com <mailto:paul at iwascoding.com>)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Effective uid=8, gid=8, home=/var/spool/mail/iwascoding/paul Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com <mailto:paul at iwascoding.com>)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Quota root: name=User quota backend=dict args=:file:/var/spool/mail/iwascoding/paul/dovecot-quota Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com <mailto:paul at iwascoding.com>)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Quota rule: root=User quota mailbox=* bytes=2147483648 messages=0 Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com <mailto:paul at iwascoding.com>)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Quota rule: root=User quota mailbox=* bytes=2147483648 messages=60000 Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com <mailto:paul at iwascoding.com>)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Quota grace: root=User quota bytes=214748364 (10%) Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com <mailto:paul at iwascoding.com>)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: dict quota: user=paul at iwascoding.com <mailto:user=paul at iwascoding.com>, uri=file:/var/spool/mail/iwascoding/paul/dovecot-quota, noenforcing=0 Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com <mailto:paul at iwascoding.com>)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Namespace inbox: type=private, prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes location=mdbox:~/mdbox Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com <mailto:paul at iwascoding.com>)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: fs: root=/var/spool/mail/iwascoding/paul/mdbox, index=, indexpvt=, control=, inbox=, altJan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com <mailto:paul at iwascoding.com>)<16777><IKpfOqBHPFyJQQAADYqDFA>: FTS Xapian: Partial=2, Full=20 DB_PATH=/var/spool/mail/iwascoding/paul/mdbox/xapian-indexes Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com <mailto:paul at iwascoding.com>)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: quota: quota_over_flag check: quota_over_script unset - skipping Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com <mailto:paul at iwascoding.com>)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Mailbox sent: Mailbox opened because: indexing Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com <mailto:paul at iwascoding.com>)<16777><IKpfOqBHPFyJQQAADYqDFA>: FTS Xapian : Mailbox sent : Last UID=0 Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com <mailto:paul at iwascoding.com>)<16777><IKpfOqBHPFyJQQAADYqDFA>: FTS Xapian : Mailbox sent : Last UID=0 Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com <mailto:paul at iwascoding.com>)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Namespace : Using permissions from /var/spool/mail/iwascoding/paul/mdbox: mode=0700 gid=default Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com <mailto:paul at iwascoding.com>)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Mailbox sent: UID 1: Opened mail because: fts indexing Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com <mailto:paul at iwascoding.com>)<16777><IKpfOqBHPFyJQQAADYqDFA>: Opening RW /var/spool/mail/iwascoding/paul/mdbox/xapian-indexes/db_9ddfe10d8a8a8a568c120000654d370e Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com <mailto:paul at iwascoding.com>)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Mailbox sent: UID 2: Opened mail because: fts indexing Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com <mailto:paul at iwascoding.com>)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Mailbox sent: UID 3: Opened mail because: fts indexing Thank you!> On 14. Jan 2019, at 10:11, Joan Moreau via dovecot <dovecot at dovecot.org <mailto:dovecot at dovecot.org>> wrote: > > Can you send the log part that includes the "init" of the plugins (something similar as below) ? > > WHich version of Xapian are you on ? > > Jan 14 09:10:04 gjserver dovecot[31082]: indexer-worker(admin at grosjo.net <mailto:admin at grosjo.net>)<14725><apZfHexRPFy9QAAA0thIag:UL+TNOxRPFyFOQAA0thIag>: FTS Xapian: Partial=2, Full=20 DB_PATH=/data/mail/grosjo.net/admin/xapian-indexes <http://grosjo.net/admin/xapian-indexes> > Jan 14 09:10:04 gjserver dovecot[31082]: indexer-worker(admin at grosjo.net <mailto:admin at grosjo.net>)<14725><apZfHexRPFy9QAAA0thIag:UL+TNOxRPFyFOQAA0thIag>: FTS Xapian : Mailbox Mail : Last UID=815055 > Jan 14 09:10:04 gjserver dovecot[31082]: indexer-worker(admin at grosjo.net <mailto:admin at grosjo.net>)<14725><apZfHexRPFy9QAAA0thIag:UL+TNOxRPFyFOQAA0thIag>: FTS Xapian : Mailbox Mail : Last UID=815055 > Jan 14 09:10:04 gjserver dovecot[31082]: indexer-worker(admin at grosjo.net <mailto:admin at grosjo.net>)<14725><apZfHexRPFy9QAAA0thIag:UL+TNOxRPFyFOQAA0thIag>: Opening RW /data/mail/grosjo.net/admin/xapian-indexes/db_5c935034609bc14c0e550000d6a3092d <http://grosjo.net/admin/xapian-indexes/db_5c935034609bc14c0e550000d6a3092d> > > > > > On 2019-01-14 10:08, Paul Hecker via dovecot wrote: > >> Hi, >> >> I installed and tested your version, but the indexer process crashes reproducible with the following command after about 2000 messages were indexed: >> >> doveadm index -u paul at iwascoding.com <mailto:paul at iwascoding.com> -q \* >> >> Jan 14 09:26:15 mail dovecot: indexer-worker(paul at iwascoding.com <mailto:paul at iwascoding.com>)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Mailbox sent: UID 2038: Opened mail because: fts indexing >> Jan 14 09:26:15 mail dovecot: indexer-worker: Error: terminate called after throwing an instance of 'std::bad_alloc' >> Jan 14 09:26:15 mail dovecot: indexer-worker: Error: what(): std::bad_alloc >> Jan 14 09:26:15 mail dovecot: indexer: Error: Indexer worker disconnected, discarding 48 requests for paul at iwascoding.com <mailto:paul at iwascoding.com> >> Jan 14 09:26:15 mail dovecot: indexer-worker(paul at iwascoding.com <mailto:paul at iwascoding.com>)<16777><IKpfOqBHPFyJQQAADYqDFA>: Fatal: master: service(indexer-worker): child 16777 killed with signal 6 (core dumps disabled - https://dovecot.org/bugreport.html#coredumps <https://dovecot.org/bugreport.html#coredumps>) >> >> I tried to delete the message, but this does not help (crashes e.g. after message 2029 or 2044). Other folders with fewer messages were successfully indexed before. >> >> Sorry, could not convince dovecot to create core dumps (read the docs, changed /proc/sys/kernel/core_pattern, added LimitCORE=unlimited/infinity, even created /etc/systemd/system/dovecot.service.d/coredump.conf to no avail). Custom Dovecot 2.3.4 on Debian Stretch. >> >> Thanks, >> Paul >>-------------- next part -------------- An HTML attachment was scrubbed... URL: <https://dovecot.org/pipermail/dovecot/attachments/20190114/ed78b16b/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4127 bytes Desc: not available URL: <https://dovecot.org/pipermail/dovecot/attachments/20190114/ed78b16b/attachment.p7s>