search for: fts_backend_update_build_mor

FTS indexing of binary attachments is broken in dovecot-2.1.15: the binary data which fts_build_mail_real() sends to fts_build_body_block() (which sends it to fts_backend_update_build_more()) is garbled. This patch ungarbles it but I?m not positive it?s the best fix. --- dovecot-2.1.15/src/plugins/fts/fts-build-mail.c 2012-11-27 02:07:27.000000000 -0600 +++ dovecot/src/plugins/fts/fts-build-mail.c 2013-02-21 19:48:15.000000000 -0600 @@ -178,7 +178,8 @@ *binary_body_r = TRUE;...

...contain only full valid UTF-8 characters, ?? but it doesn't need to be NUL-terminated. size contains the data size in ?? bytes, not characters. This function may be called many times and the data ?? block sizes may be small. Backend returns 0 if ok, -1 if build should be ?? aborted. */ int fts_backend_update_build_more(struct fts_backend_update_context *ctx, ??? ??? ??? ??? ? const unsigned char *data, size_t size); You should look at the sources of a few backends like squat and solr to get a feel of what exactly this is doing. > what is refresh versus rescan ? From fts-api.h: /* Refresh index to make...

...pci 0000:00:1c.0: restoring config space at offset 0x1 (was 0x100000, writing "..., size=8188) > at fts-backend-lucene.c:415 > _data_stack_cur_id = 6 > ctx = 0x8b3c578 > backend = 0x8b23850 > ret = <optimized out> > #5 0xb7586cda in fts_backend_update_build_more (ctx=0x8b3c578, > data=0x8b38f08 "1: kernel: i7core_edac 0000:00:14.0: restoring config space at offset 0x3 (was 0x800000, writing 0x800010)\n 1: kernel: pci 0000:00:1c.0: restoring config space at offset 0x1 (was 0x100000, writing "..., size=8188) > at fts-api.c:193 &g...

...only full valid UTF-8 characters, > but it doesn't need to be NUL-terminated. size contains the data size in > bytes, not characters. This function may be called many times and the data > block sizes may be small. Backend returns 0 if ok, -1 if build should be > aborted. */ > int fts_backend_update_build_more(struct fts_backend_update_context *ctx, > const unsigned char *data, size_t size); > > You should look at the sources of a few backends like squat and solr to get a feel of what exactly this is doing. > > what is refresh versus rescan ? > From fts-api.h: > > /* Refresh...

...only full valid UTF-8 characters, > but it doesn't need to be NUL-terminated. size contains the data size in > bytes, not characters. This function may be called many times and the data > block sizes may be small. Backend returns 0 if ok, -1 if build should be > aborted. */ > int fts_backend_update_build_more(struct fts_backend_update_context *ctx, > const unsigned char *data, size_t size); > > You should look at the sources of a few backends like squat and solr to get a feel of what exactly this is doing. > > what is refresh versus rescan ? > From fts-api.h: > > /* Refresh...

Hi Are data inputs of fts_backend_update_build_more(struct fts_backend_update_context *_ctx, const unsigned char *data, size_t size) already converted to UTF8 ? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://dovecot.org/pipermail/dovecot/attachments/20190122/e86c8fd3/attachment.html>

...[0x14bac7] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x1211ae] -> /usr/local/lib/dovecot/lib21_fts_solr_plugin.so(solr_connection_post_more+0x293) [0xb1a7d3] -> /usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x2599) [0xb16599] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(fts_backend_update_build_more+0x31) [0x2273f1] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(fts_build_mail+0x3b2) [0x228272] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(+0xa74e) [0x22d74e] -> /usr/local/lib/dovecot/libdovecot-storage.so.0(mail_precache+0xf) [0x937a6f] -> doveadm() [0x80520b6] -> doveadm() [0x...

...r_t const*, wchar_t const*, int, bool) () from /usr/lib/i386-linux-gnu/libclucene-core.so.1 #3 0xb7b5cd48 in lucene_index_build_more () from /usr/lib/dovecot/modules/lib21_fts_lucene_plugin.so #4 0xb7b5b7b9 in ?? () from /usr/lib/dovecot/modules/lib21_fts_lucene_plugin.so #5 0xb7b6ac2a in fts_backend_update_build_more () from /usr/lib/dovecot/modules/lib20_fts_plugin.so #6 0xb7b6b674 in ?? () from /usr/lib/dovecot/modules/lib20_fts_plugin.so #7 0xb7b6bbf4 in fts_build_mail () from /usr/lib/dovecot/modules/lib20_fts_plugin.so #8 0xb7b713cc in ?? () from /usr/lib/dovecot/modules/lib20_fts_plugin.so #9 0xb...

...only full valid UTF-8 characters, > but it doesn't need to be NUL-terminated. size contains the data size in > bytes, not characters. This function may be called many times and the data > block sizes may be small. Backend returns 0 if ok, -1 if build should be > aborted. */ > int fts_backend_update_build_more(struct fts_backend_update_context *ctx, > const unsigned char *data, size_t size); > > You should look at the sources of a few backends like squat and solr to get a feel of what exactly this is doing. > >> what is refresh versus rescan ? > > From fts-api.h: > > /*...

...only full valid UTF-8 characters, > but it doesn't need to be NUL-terminated. size contains the data size in > bytes, not characters. This function may be called many times and the data > block sizes may be small. Backend returns 0 if ok, -1 if build should be > aborted. */ > int fts_backend_update_build_more(struct fts_backend_update_context *ctx, > const unsigned char *data, size_t size); > > You should look at the sources of a few backends like squat and solr to get a feel of what exactly this is doing. > > what is refresh versus rescan ? > From fts-api.h: > > /* Refresh...

...only full valid UTF-8 characters, > but it doesn't need to be NUL-terminated. size contains the data size in > bytes, not characters. This function may be called many times and the data > block sizes may be small. Backend returns 0 if ok, -1 if build should be > aborted. */ > int fts_backend_update_build_more(struct fts_backend_update_context *ctx, > const unsigned char *data, size_t size); > > You should look at the sources of a few backends like squat and solr to get a feel of what exactly this is doing. > > what is refresh versus rescan ? > From fts-api.h: > > /* Refresh...

...only full valid UTF-8 characters, > but it doesn't need to be NUL-terminated. size contains the data size in > bytes, not characters. This function may be called many times and the data > block sizes may be small. Backend returns 0 if ok, -1 if build should be > aborted. */ > int fts_backend_update_build_more(struct fts_backend_update_context *ctx, > const unsigned char *data, size_t size); > > You should look at the sources of a few backends like squat and solr to get a feel of what exactly this is doing. > > what is refresh versus rescan ? > From fts-api.h: > > /* Refresh...

...only full valid UTF-8 characters, > but it doesn't need to be NUL-terminated. size contains the data size in > bytes, not characters. This function may be called many times and the data > block sizes may be small. Backend returns 0 if ok, -1 if build should be > aborted. */ > int fts_backend_update_build_more(struct fts_backend_update_context *ctx, > const unsigned char *data, size_t size); > > You should look at the sources of a few backends like squat and solr to get a feel of what exactly this is doing. > > what is refresh versus rescan ? > From fts-api.h: > > /* Refresh...

Anyone willing to explain those functions ? On January 5, 2019 14:23:10 Joan Moreau via dovecot <dovecot at dovecot.org> wrote: > Thank Stephan > I basically need to know the role/description of each of the functions of > the fts_backend: > > struct fts_backend fts_backend_xapian = { > .name = "xapian", > .flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT, -> what

...pdate_build_more ( _ctx=0x801c21240, data=0x806041000 "<m.ray.mullins+caf_=mrm=lerctr.org at gmail.com>", size=45) at fts-backend-lucene.c:432 _data_stack_cur_id = 6 ctx = 0x801c21240 backend = 0x801c3a200 ret = 8 #12 0x000000080220e035 in fts_backend_update_build_more (ctx=0x801c21240, data=0x806041000 "<m.ray.mullins+caf_=mrm=lerctr.org at gmail.com>", size=45) at fts-api.c:193 No locals. #13 0x000000080221015b in fts_build_full_words (ctx=0x7fffffffdc98, data=0x806041000 "<m.ray.mullins+caf_=mrm=lerctr.org at gmail.com>&...

Hi! We found some problems with those patches, and ended up doing slightly different fix: https://github.com/dovecot/core/compare/3e41b3d%5E...cca98b.patch Aki On 24.10.2016 10:17, Aki Tuomi wrote: > Hi! > > Can you try these two patches? > > Aki > > > On 24.10.2016 08:48, Aki Tuomi wrote: >> Ok so that timeval makes no sense. We'll look into it. >>

...ib/dovecot/libdovecot.so.0(i_fatal+0) [0x7fe8a908b274] -> /usr/mailsuite/lib/dovecot/lib21_fts_solr_plugin.so(solr_connection_post_more+0x2d2) [0x7fe8a75fe973] -> /usr/mailsuite/lib/dovecot/lib21_fts_solr_plugin.so(+0x4d03) [0x7fe8a75f9d03] -> /usr/mailsuite/lib/dovecot/lib20_fts_plugin.so(fts_backend_update_build_more+0x77) [0x7fe8a7c1d401] -> /usr/mailsuite/lib/dovecot/lib20_fts_plugin.so(+0x7fe2) [0x7fe8a7c1dfe2] -> /usr/mailsuite/lib/dovecot/lib20_fts_plugin.so(+0x80d5) [0x7fe8a7c1e0d5] -> /usr/mailsuite/lib/dovecot/lib20_fts_plugin.so(+0x89e4) [0x7fe8a7c1e9e4] -> /usr/mailsuite/lib/dovecot/lib20...