Hi, I am using dovecot (see dovecot -n output below) with metadata plugin in my own plugin. I want to insert and delete some large metadata (>4KiB). It seems, that the delete in a second call of my plugin smashes the file dovecot.dict in users dir: 1st call: *dovecot.dict is empty -delete metadata 1 -delete metadata 2 -insert metadata 1 ~8KiB -insert metadata 2 <1KiB *dovecot.dict contains two entries 2nd call: -delete metadata 1 -delete metadata 2 -insert metadata 1 ~8KiB -insert metadata 2 <1KiB *dovecot.dict contains the two entries + some waste If all eight steps are called together (in the same plugin context) the file dovecot.dict seems to be ok. I have created a test plugin to reproduce this behaviour: <http://temp-share.com/show/dPf3myu5W>. It needs the notify and (patched) metadata plugin to run. To get it work i copy an email from one IMAP folder to another (to trigger the copy notifier). It seems, that there is an issue on writing metadata/dict, can anybody reproduce it or knows what is wrong? TIA -- Andre
Andre Gröbe
2012-Aug-08 12:33 UTC
[Dovecot] [bulk]: Deleting metadata smashes file dovecot.dict
missing patch and info: metadata patch ============= --- a/src/metadata-backend.c +++ b/src/metadata-backend.c @@ -130,7 +130,8 @@ metadata_set_entry(struct metadata_entry *entry, struct mail_user *user) { if (!metadata_entry_is_valid(entry)) return -METADATA_ERROR_INVALID; - if (strlen(metadata_entry_get_value(entry)) > muser->set->maxsize) + if (metadata_entry_get_value(entry) != NULL + && strlen(metadata_entry_get_value(entry)) > muser->set->maxsize) return -METADATA_ERROR_TOOLARGE; if (count_entries(muser) > muser->set->maxentries) return -METADATA_ERROR_TOOMANY; --- a/Makefile.am +++ b/Makefile.am @@ -1,3 +1,6 @@ ACLOCAL_AMFLAGS = -I m4 SUBDIRS = src + +pkginc_libdir=$(dovecot_pkgincludedir)/metadata +nodist_pkginc_lib_HEADERS = metadata-config.h --- a/src/Makefile.am +++ b/src/Makefile.am @@ -42,7 +42,7 @@ lib90_imap_annotatemore_plugin_la_LIBADD = \ lib80_metadata_plugin.la endif -noinst_HEADERS = \ +headers = \ metadata-backend.h \ metadata-entry.h \ metadata-entry-private.h \ @@ -55,3 +55,6 @@ noinst_HEADERS = \ imap-arg-ext.h \ mailbox-ext.h \ str-ext.h + +pkginc_libdir=$(dovecot_pkgincludedir)/metadata +pkginc_lib_HEADERS = $(headers) dovecot -n ========= # 2.1.7: /etc/dovecot/dovecot.conf # OS: Linux 2.6.35-1exp2-intern-k8-kvm x86_64 Debian 6.0.5 ext3 debug_log_path = /var/opt/dovecot/log/dovecot.log default_vsz_limit = 512 M disable_plaintext_auth = no first_valid_uid = 0 info_log_path = /var/opt/dovecot/log/dovecot.log lmtp_check_rcpt = yes lmtp_rcpt_to_parameters { * = i: ignorequota = b:plugin/quota_ignore xtolain = s:plugin/lain xtomsgtype = s:plugin/mailtype_xtomsgtype xtospam = b:plugin/spamflag_xtospam xtovacation = B:plugin/sieve_xtovacation } log_path = /var/opt/dovecot/log/dovecot.log log_timestamp = "%y%m%H%M%S " mail_gid = mail mail_home = /var/spool/dovecot/%2.256Hu/%3.4096Hu/%u mail_location = mdbox:/var/spool/dovecot/%2.256Hu/%3.4096Hu/%u mail_plugin_dir = /opt/app/dovecot/lib/dovecot mail_plugins = notify metadata test mail_uid = mail namespace inbox { inbox = yes location prefix = INBOX. separator = . type = private } passdb { args = nocache=y /var/spool/dovecot/deny deny = yes driver = passwd-file } passdb { args = nopassword=y driver = static } plugin { acl = vfile autocreate = INBOX.Drafts autocreate2 = INBOX.Sent autocreate3 = INBOX.Trash metadata_dict = file:/var/spool/dovecot/%2.256Hu/%3.4096Hu/%u/dovecot.dict metadata_maxsize = 10240 quota = dict:User quota::file:/var/spool/dovecot/%2.256Hu/%3.4096Hu/%u/dovecot.dict quota_rule = *:bytes=999999 quota_rule2 = *:messages=999 quota_rule3 = *:mailboxes=99 quota_rule4 = *:levels=9 quota_rule5 = INBOX.Voicebox:ignore } service anvil { unix_listener anvil { group = dovecot mode = 0660 } } service auth-worker { user = $default_internal_user } service auth { unix_listener auth-userdb { group = dovecot mode = 0660 } } service config { unix_listener config { group = dovecot mode = 0660 } } service dict { unix_listener dict { mode = 0600 user = mail } } service lmtp { inet_listener lmtp { port = 2003 } } ssl = no userdb { args = /opt/app/dovecot/etc/dovecot/dovecot-tdbfcall.conf.ext driver = tdbfcall } protocol imap { mail_plugins = notify metadata test autocreate imap_metadata imap_annotatemore } protocol pop3 { pop3_uidl_format = %v.%u } -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4910 bytes Desc: S/MIME Cryptographic Signature URL: <http://dovecot.org/pipermail/dovecot/attachments/20120808/0d589074/attachment-0004.bin>
Hi it seems to me, that it depends on how dovecot handles line reading. Dovecot presumes that a line is of max length of IO_BLOCK_SIZE (=8192) Bytes. Is there a fix so dovecot reads a line until EOL or EOF (if this is the real reason)? Regards Andre -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4910 bytes Desc: S/MIME Cryptographic Signature URL: <http://dovecot.org/pipermail/dovecot/attachments/20120820/78b38a2d/attachment-0004.bin>
On 20.8.2012, at 16.35, Andre Gr?be wrote:> it seems to me, that it depends on how dovecot handles line reading. Dovecot presumes that a line is of max length of IO_BLOCK_SIZE (=8192) Bytes. > > Is there a fix so dovecot reads a line until EOL or EOF (if this is the real reason)?Without having actually looked at the code, I'd guess you can just replace IO_BLOCK_SIZE with (size_t)-1.
Ewald Dieterich
2012-Aug-28 14:42 UTC
[Dovecot] Deleting metadata smashes file dovecot.dict
On 08/08/12 14:23, Andre Gr?be wrote:> I am using dovecot (see dovecot -n output below) with metadata plugin in > my own plugin. I want to insert and delete some large metadata (>4KiB). > It seems, that the delete in a second call of my plugin smashes the file > dovecot.dict in users dir:That's an error in dict-file.c, file_dict_refresh(): while ((key = i_stream_read_next_line(input)) != NULL && (value = i_stream_read_next_line(input)) != NULL) { If reading the value requires reading from the stream (because the value doesn't fit into the input buffer), the key is overwritten. The attached patch duplicates the key before reading the value. -------------- next part -------------- A non-text attachment was scrubbed... Name: dict-file.c.patch Type: text/x-diff Size: 665 bytes Desc: not available URL: <http://dovecot.org/pipermail/dovecot/attachments/20120828/848f73a2/attachment-0004.bin>
On 28.8.2012, at 17.42, Ewald Dieterich wrote:> On 08/08/12 14:23, Andre Gr?be wrote: >> I am using dovecot (see dovecot -n output below) with metadata plugin in >> my own plugin. I want to insert and delete some large metadata (>4KiB). >> It seems, that the delete in a second call of my plugin smashes the file >> dovecot.dict in users dir: > > That's an error in dict-file.c, file_dict_refresh(): > > while ((key = i_stream_read_next_line(input)) != NULL && > (value = i_stream_read_next_line(input)) != NULL) { > > If reading the value requires reading from the stream (because the value doesn't fit into the input buffer), the key is overwritten. The attached patch duplicates the key before reading the value.Committed: http://hg.dovecot.org/dovecot-2.1/rev/6e53209030f6
Seemingly Similar Threads
- Crash: setannotation Trash "/vendor/cmu/cyrus-imapd/expire" ("value.shared" NIL)
- Converting UW MBX mail folder
- backup mails from alternate storage
- Coredump: Panic: file smtp-address.c: line 530 (smtp_address_write): assertion failed: (smtp_char_is_qpair(*p))
- Passing info from mail process to mail_filter plugin script?