dovecot - Jul 2019 - index worker 2.3.7 undefined symbol errors

Suddenly I am getting undefined symbol errors having updated to 2.3.7.

Jul 18 01:02:49 localhost dovecot: indexer-worker: Error: User <someone> 
lookup failed: Couldn't load required plugin 
/usr/lib/dovecot/modules/lib95_imap_sieve_plugin.so: dlopen() failed: 
/usr/lib/dovecot/modules/lib95_imap_sieve_plugin.so: undefined symbol: 
command_hook_register

(I think I fixed this one by a config change)

But I am left with this:

Jul 19 14:09:52 localhost dovecot: indexer-worker: Error: User <someone> 
lookup failed: Couldn't load required plugin 
/usr/lib/dovecot/modules/lib90_sieve_plugin.so: dlopen() faile: 
/usr/lib/dovecot/modules/lib90_sieve_plugin.so: undefined symbol: 
mail_deliver_ctx_get_log_var_expand_table

My (redacted) dovecot -n:

# 2.3.7 (494d20bdc): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.7 (5a4e63b5)
# OS: Linux 4.15.0-54-generic x86_64 Ubuntu 18.04.2 LTS
# Hostname: localhost
auth_verbose = yes
first_valid_uid = 2000
imapc_features = rfc822.size fetch-headers
imapc_host = mailman.tobit.co.uk
imapc_port = 993
imapc_ssl = imaps
imapc_ssl_verify = no
imapc_user = %n
lda_mailbox_autocreate = yes
mail_attachment_dir = %h/attachments
mail_attachment_min_size = 64 k
mail_location = mdbox:%h:INBOX=%h/mail
mail_plugins = mail_log notify replication fts fts_lucene sieve
mail_prefetch_count = 20
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope 
encoded-character vacation subaddress comparator-i;ascii-numeric 
relational regex imap4flags copy include variables body enotify 
environment mailbox date index ihave duplicate mime foreverypart 
extracttext imapsieve vnd.dovecot.imapsieve
namespace inbox {
 ? inbox = yes
 ? location  ? mailbox Drafts {
 ??? special_use = \Drafts
 ? }
 ? mailbox Junk {
 ??? special_use = \Junk
 ? }
 ? mailbox Sent {
 ??? special_use = \Sent
 ? }
 ? mailbox "Sent Messages" {
 ??? special_use = \Sent
 ? }
 ? mailbox Trash {
 ??? special_use = \Trash
 ? }
 ? prefix }
passdb {
 ? args = # hidden
 ? driver = sql
}
plugin {
 ? fts = lucene
 ? fts_autoindex = yes
 ? fts_autoindex_exclude = \Junk
 ? fts_autoindex_exclude2 = \Trash
 ? fts_autoindex_exclude3 = \Deleted
 ? fts_lucene = whitespace_chars=@.
 ? imapsieve_mailbox1_before = file:/etc/dovecot/sieve/report-spam.sieve
 ? imapsieve_mailbox1_causes = COPY
 ? imapsieve_mailbox1_name = Junk
 ? imapsieve_mailbox2_before = file:/etc/dovecot/sieve/report-ham.sieve
 ? imapsieve_mailbox2_causes = COPY
 ? imapsieve_mailbox2_from = Junk
 ? imapsieve_mailbox2_name = *
 ? mail_log_events = delete undelete expunge copy save mailbox_delete 
mailbox_rename mailbox_create
 ? mail_log_fields = uid box msgid size
 ? sieve = file:~/sieve;active=~/.dovecot.sieve
 ? sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.environment
 ? sieve_pipe_bin_dir = /etc/dovecot/sieve
 ? sieve_plugins = sieve_imapsieve sieve_extprograms
 ? sieve_trace_dir = /tmp
 ? sieve_trace_level = matching
}
postmaster_address = postmaster at tobit.co.uk
protocols = imap lmtp sieve
service auth {
 ? unix_listener exim-auth {
 ??? group = Debian-exim
 ??? mode = 0660
 ??? user = Debian-exim
 ? }
}
service imap-login {
 ? inet_listener imap {
 ??? port = 143
 ? }
 ? inet_listener imaps {
 ??? port = 993
 ??? ssl = yes
 ? }
}
service lmtp {
 ? inet_listener lmtp {
 ??? address = 127.0.0.1
 ??? port = # hidden
 ? }
 ? unix_listener lmtp {
 ??? mode = 0666
 ? }
 ? user = vmail
}
service managesieve-login {
 ? inet_listener sieve {
 ??? port = # hidden
 ? }
 ? process_min_avail = 0
 ? service_count = 1
 ? vsz_limit = 64 M
}
service managesieve {
 ? process_limit = 100
}
ssl_cert = # hidden
ssl_client_ca_dir = /etc/ssl/certs
ssl_dh = # hidden, use -P to show it
ssl_key = # hidden, use -P to show it
userdb {
 ? args = # hidden
 ? driver = sql
}
protocol imap {
 ? mail_plugins = mail_log notify replication fts fts_lucene imap_sieve
}
protocol lmtp {
 ? mail_plugins = fts fts_lucene sieve
}

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20190719/c4fdb7d3/attachment.html>

Some supplemental information:

This is happening on every email delivered into Dovecot via LMTP. The 
curious things are that the message is a) successfully delivered and b) 
sieved into the correct directory.

Another observation is that:

mail_deliver_ctx_get_log_var_expand_table

is defined globally in core/src/lib-lda/mail-deliver.c (and used there) 
but the ONLY external call in the entire dovecot tree is from 
pigeonhole/src/plugins/lda-sieve/lda-sieve-log.c.

I am not using lda but it seems to be part of core. So, as I am only 
using lmtp, why is pigeonhole using lda-sieve at all?

Can I work around the error message by some config magic (as I did by 
calling the correct plugin for imap_sieve) or is this an actual bug? 
Could this be fixed simply by including mail-deliver.h in lda-sieve-log.c?

Dirk

On 19/07/2019 13:20, Dirk Koopman via dovecot wrote:
> Suddenly I am getting undefined symbol errors having updated to 2.3.7.
>
> Jul 18 01:02:49 localhost dovecot: indexer-worker: Error: User 
> <someone> lookup failed: Couldn't load required plugin 
> /usr/lib/dovecot/modules/lib95_imap_sieve_plugin.so: dlopen() failed: 
> /usr/lib/dovecot/modules/lib95_imap_sieve_plugin.so: undefined symbol: 
> command_hook_register
>
> (I think I fixed this one by a config change)
>
> But I am left with this:
>
> Jul 19 14:09:52 localhost dovecot: indexer-worker: Error: User 
> <someone> lookup failed: Couldn't load required plugin 
> /usr/lib/dovecot/modules/lib90_sieve_plugin.so: dlopen() faile: 
> /usr/lib/dovecot/modules/lib90_sieve_plugin.so: undefined symbol: 
> mail_deliver_ctx_get_log_var_expand_table
>
> My (redacted) dovecot -n:
>
> # 2.3.7 (494d20bdc): /etc/dovecot/dovecot.conf
> # Pigeonhole version 0.5.7 (5a4e63b5)
> # OS: Linux 4.15.0-54-generic x86_64 Ubuntu 18.04.2 LTS
> # Hostname: localhost
> auth_verbose = yes
> first_valid_uid = 2000
> imapc_features = rfc822.size fetch-headers
> imapc_host = mailman.tobit.co.uk
> imapc_port = 993
> imapc_ssl = imaps
> imapc_ssl_verify = no
> imapc_user = %n
> lda_mailbox_autocreate = yes
> mail_attachment_dir = %h/attachments
> mail_attachment_min_size = 64 k
> mail_location = mdbox:%h:INBOX=%h/mail
> mail_plugins = mail_log notify replication fts fts_lucene sieve
> mail_prefetch_count = 20
> managesieve_notify_capability = mailto
> managesieve_sieve_capability = fileinto reject envelope 
> encoded-character vacation subaddress comparator-i;ascii-numeric 
> relational regex imap4flags copy include variables body enotify 
> environment mailbox date index ihave duplicate mime foreverypart 
> extracttext imapsieve vnd.dovecot.imapsieve
> namespace inbox {
> ? inbox = yes
> ? location > ? mailbox Drafts {
> ??? special_use = \Drafts
> ? }
> ? mailbox Junk {
> ??? special_use = \Junk
> ? }
> ? mailbox Sent {
> ??? special_use = \Sent
> ? }
> ? mailbox "Sent Messages" {
> ??? special_use = \Sent
> ? }
> ? mailbox Trash {
> ??? special_use = \Trash
> ? }
> ? prefix > }
> passdb {
> ? args = # hidden
> ? driver = sql
> }
> plugin {
> ? fts = lucene
> ? fts_autoindex = yes
> ? fts_autoindex_exclude = \Junk
> ? fts_autoindex_exclude2 = \Trash
> ? fts_autoindex_exclude3 = \Deleted
> ? fts_lucene = whitespace_chars=@.
> ? imapsieve_mailbox1_before = file:/etc/dovecot/sieve/report-spam.sieve
> ? imapsieve_mailbox1_causes = COPY
> ? imapsieve_mailbox1_name = Junk
> ? imapsieve_mailbox2_before = file:/etc/dovecot/sieve/report-ham.sieve
> ? imapsieve_mailbox2_causes = COPY
> ? imapsieve_mailbox2_from = Junk
> ? imapsieve_mailbox2_name = *
> ? mail_log_events = delete undelete expunge copy save mailbox_delete 
> mailbox_rename mailbox_create
> ? mail_log_fields = uid box msgid size
> ? sieve = file:~/sieve;active=~/.dovecot.sieve
> ? sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.environment
> ? sieve_pipe_bin_dir = /etc/dovecot/sieve
> ? sieve_plugins = sieve_imapsieve sieve_extprograms
> ? sieve_trace_dir = /tmp
> ? sieve_trace_level = matching
> }
> postmaster_address = postmaster at tobit.co.uk
> protocols = imap lmtp sieve
> service auth {
> ? unix_listener exim-auth {
> ??? group = Debian-exim
> ??? mode = 0660
> ??? user = Debian-exim
> ? }
> }
> service imap-login {
> ? inet_listener imap {
> ??? port = 143
> ? }
> ? inet_listener imaps {
> ??? port = 993
> ??? ssl = yes
> ? }
> }
> service lmtp {
> ? inet_listener lmtp {
> ??? address = 127.0.0.1
> ??? port = # hidden
> ? }
> ? unix_listener lmtp {
> ??? mode = 0666
> ? }
> ? user = vmail
> }
> service managesieve-login {
> ? inet_listener sieve {
> ??? port = # hidden
> ? }
> ? process_min_avail = 0
> ? service_count = 1
> ? vsz_limit = 64 M
> }
> service managesieve {
> ? process_limit = 100
> }
> ssl_cert = # hidden
> ssl_client_ca_dir = /etc/ssl/certs
> ssl_dh = # hidden, use -P to show it
> ssl_key = # hidden, use -P to show it
> userdb {
> ? args = # hidden
> ? driver = sql
> }
> protocol imap {
> ? mail_plugins = mail_log notify replication fts fts_lucene imap_sieve
> }
> protocol lmtp {
> ? mail_plugins = fts fts_lucene sieve
> }
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20190721/6505237d/attachment-0001.html>

On 21 Jul 2019, at 23.14, Dirk Koopman via dovecot <dovecot at
dovecot.org> wrote:
> 
> Some supplemental information:
> 
> This is happening on every email delivered into Dovecot via LMTP. The
curious things are that the message is a) successfully delivered and b) sieved
into the correct directory.
> 
> Another observation is that:
> 
> mail_deliver_ctx_get_log_var_expand_table
> 
> is defined globally in core/src/lib-lda/mail-deliver.c (and used there) but
the ONLY external call in the entire dovecot tree is from
pigeonhole/src/plugins/lda-sieve/lda-sieve-log.c.
> 
> I am not using lda but it seems to be part of core. So, as I am only using
lmtp, why is pigeonhole using lda-sieve at all?
It's part of lib-lda, which is also used by lmtp.
> Can I work around the error message by some config magic (as I did by
calling the correct plugin for imap_sieve) or is this an actual bug? Could this
be fixed simply by including mail-deliver.h in lda-sieve-log.c?
I think you're not linking lmtp binary correctly somehow. That symbol should
be part of it:

% nm /usr/libexec/dovecot/lmtp | grep mail_deliver_ctx_get_log_var_expand_table
0000000000061960 T mail_deliver_ctx_get_log_var_expand_table


-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20190722/5bd200d1/attachment.html>

On 19 Jul 2019, at 13.20, Dirk Koopman via dovecot <dovecot at
dovecot.org> wrote:
> But I am left with this:
> 
> Jul 19 14:09:52 localhost dovecot: indexer-worker: Error: User
<someone> lookup failed: Couldn't load required plugin
/usr/lib/dovecot/modules/lib90_sieve_plugin.so: dlopen() faile:
/usr/lib/dovecot/modules/lib90_sieve_plugin.so: undefined symbol:
mail_deliver_ctx_get_log_var_expand_table
Oh.. it's logged by indexer-worker.
> mail_plugins = mail_log notify replication fts fts_lucene sieve
You can't load sieve globally. It needs to be inside protocol lmtp {}

On 22/07/2019 16:24, Timo Sirainen wrote:
> On 19 Jul 2019, at 13.20, Dirk Koopman via dovecot <dovecot at
dovecot.org> wrote:
>> But I am left with this:
>>
>> Jul 19 14:09:52 localhost dovecot: indexer-worker: Error: User
<someone> lookup failed: Couldn't load required plugin
/usr/lib/dovecot/modules/lib90_sieve_plugin.so: dlopen() faile:
/usr/lib/dovecot/modules/lib90_sieve_plugin.so: undefined symbol:
mail_deliver_ctx_get_log_var_expand_table
> Oh.. it's logged by indexer-worker.
>
>> mail_plugins = mail_log notify replication fts fts_lucene sieve
> You can't load sieve globally. It needs to be inside protocol lmtp {}
>
That has fixed it, thank you.

It's not obvious from the documentation which are global plugins and 
which are protocol based ones. Would it be possible to put some words on 
plugin pages to indicate which is what?

Thanks for the help

Dirk
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20190722/50c97033/attachment.html>