dovecot - Aug 2012 - Deleting metadata smashes file dovecot.dict

Hi,

I am using dovecot (see dovecot -n output below) with metadata plugin in 
my own plugin. I want to insert and delete some large metadata (>4KiB). 
It seems, that the delete in a second call of my plugin smashes the file 
dovecot.dict in users dir:

1st call:
*dovecot.dict is empty
-delete metadata 1
-delete metadata 2
-insert metadata 1 ~8KiB
-insert metadata 2 <1KiB
*dovecot.dict contains two entries

2nd call:
-delete metadata 1
-delete metadata 2
-insert metadata 1 ~8KiB
-insert metadata 2 <1KiB
*dovecot.dict contains the two entries + some waste

If all eight steps are called together (in the same plugin context) the 
file dovecot.dict seems to be ok.

I have created a test plugin to reproduce this behaviour: 
<http://temp-share.com/show/dPf3myu5W>. It needs the notify and 
(patched) metadata plugin to run. To get it work i copy an email from 
one IMAP folder to another (to trigger the copy notifier).

It seems, that there is an issue on writing metadata/dict, can anybody 
reproduce it or knows what is wrong?

TIA
-- 
Andre

missing patch and info:

metadata patch
=============
--- a/src/metadata-backend.c
+++ b/src/metadata-backend.c
@@ -130,7 +130,8 @@ metadata_set_entry(struct metadata_entry *entry, 
struct mail_user *user) {

  	if (!metadata_entry_is_valid(entry))
  		return -METADATA_ERROR_INVALID;
-	if (strlen(metadata_entry_get_value(entry)) > muser->set->maxsize)
+	if (metadata_entry_get_value(entry) != NULL
+	    && strlen(metadata_entry_get_value(entry)) >
muser->set->maxsize)
  		return -METADATA_ERROR_TOOLARGE;
  	if (count_entries(muser) > muser->set->maxentries)
  		return -METADATA_ERROR_TOOMANY;

--- a/Makefile.am
+++ b/Makefile.am
@@ -1,3 +1,6 @@
  ACLOCAL_AMFLAGS = -I m4

  SUBDIRS = src
+
+pkginc_libdir=$(dovecot_pkgincludedir)/metadata
+nodist_pkginc_lib_HEADERS = metadata-config.h

--- a/src/Makefile.am
+++ b/src/Makefile.am
@@ -42,7 +42,7 @@ lib90_imap_annotatemore_plugin_la_LIBADD = \
  	lib80_metadata_plugin.la
  endif

-noinst_HEADERS = \
+headers = \
  	metadata-backend.h \
  	metadata-entry.h \
  	metadata-entry-private.h \
@@ -55,3 +55,6 @@ noinst_HEADERS = \
  	imap-arg-ext.h \
  	mailbox-ext.h \
  	str-ext.h
+
+pkginc_libdir=$(dovecot_pkgincludedir)/metadata
+pkginc_lib_HEADERS = $(headers)


dovecot -n
=========
# 2.1.7: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.35-1exp2-intern-k8-kvm x86_64 Debian 6.0.5 ext3
debug_log_path = /var/opt/dovecot/log/dovecot.log
default_vsz_limit = 512 M
disable_plaintext_auth = no
first_valid_uid = 0
info_log_path = /var/opt/dovecot/log/dovecot.log
lmtp_check_rcpt = yes
lmtp_rcpt_to_parameters {
   * = i:
   ignorequota = b:plugin/quota_ignore
   xtolain = s:plugin/lain
   xtomsgtype = s:plugin/mailtype_xtomsgtype
   xtospam = b:plugin/spamflag_xtospam
   xtovacation = B:plugin/sieve_xtovacation
}
log_path = /var/opt/dovecot/log/dovecot.log
log_timestamp = "%y%m%H%M%S "
mail_gid = mail
mail_home = /var/spool/dovecot/%2.256Hu/%3.4096Hu/%u
mail_location = mdbox:/var/spool/dovecot/%2.256Hu/%3.4096Hu/%u
mail_plugin_dir = /opt/app/dovecot/lib/dovecot
mail_plugins = notify metadata test
mail_uid = mail
namespace inbox {
   inbox = yes
   location    prefix = INBOX.
   separator = .
   type = private
}
passdb {
   args = nocache=y /var/spool/dovecot/deny
   deny = yes
   driver = passwd-file
}
passdb {
   args = nopassword=y
   driver = static
}
plugin {
   acl = vfile
   autocreate = INBOX.Drafts
   autocreate2 = INBOX.Sent
   autocreate3 = INBOX.Trash
   metadata_dict = 
file:/var/spool/dovecot/%2.256Hu/%3.4096Hu/%u/dovecot.dict
   metadata_maxsize = 10240
   quota = dict:User 
quota::file:/var/spool/dovecot/%2.256Hu/%3.4096Hu/%u/dovecot.dict
   quota_rule = *:bytes=999999
   quota_rule2 = *:messages=999
   quota_rule3 = *:mailboxes=99
   quota_rule4 = *:levels=9
   quota_rule5 = INBOX.Voicebox:ignore
}
service anvil {
   unix_listener anvil {
     group = dovecot
     mode = 0660
   }
}
service auth-worker {
   user = $default_internal_user
}
service auth {
   unix_listener auth-userdb {
     group = dovecot
     mode = 0660
   }
}
service config {
   unix_listener config {
     group = dovecot
     mode = 0660
   }
}
service dict {
   unix_listener dict {
     mode = 0600
     user = mail
   }
}
service lmtp {
   inet_listener lmtp {
     port = 2003
   }
}
ssl = no
userdb {
   args = /opt/app/dovecot/etc/dovecot/dovecot-tdbfcall.conf.ext
   driver = tdbfcall
}
protocol imap {
   mail_plugins = notify metadata test autocreate imap_metadata 
imap_annotatemore
}
protocol pop3 {
   pop3_uidl_format = %v.%u
}

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4910 bytes
Desc: S/MIME Cryptographic Signature
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20120808/0d589074/attachment-0004.bin>

Hi

it seems to me, that it depends on how dovecot handles line reading. 
Dovecot presumes that a line is of max length of IO_BLOCK_SIZE (=8192) 
Bytes.

Is there a fix so dovecot reads a line until EOL or EOF (if this is the 
real reason)?

Regards
Andre

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4910 bytes
Desc: S/MIME Cryptographic Signature
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20120820/78b38a2d/attachment-0004.bin>

On 20.8.2012, at 16.35, Andre Gr?be wrote:
> it seems to me, that it depends on how dovecot handles line reading.
Dovecot presumes that a line is of max length of IO_BLOCK_SIZE (=8192) Bytes.
> 
> Is there a fix so dovecot reads a line until EOL or EOF (if this is the
real reason)?
Without having actually looked at the code, I'd guess you can just replace
IO_BLOCK_SIZE with (size_t)-1.

On 08/08/12 14:23, Andre Gr?be wrote:
> I am using dovecot (see dovecot -n output below) with metadata plugin in
> my own plugin. I want to insert and delete some large metadata (>4KiB).
> It seems, that the delete in a second call of my plugin smashes the file
> dovecot.dict in users dir:
That's an error in dict-file.c, file_dict_refresh():

   while ((key = i_stream_read_next_line(input)) != NULL &&
          (value = i_stream_read_next_line(input)) != NULL) {

If reading the value requires reading from the stream (because the value 
doesn't fit into the input buffer), the key is overwritten. The attached 
patch duplicates the key before reading the value.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: dict-file.c.patch
Type: text/x-diff
Size: 665 bytes
Desc: not available
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20120828/848f73a2/attachment-0004.bin>

On 28.8.2012, at 17.42, Ewald Dieterich wrote:
> On 08/08/12 14:23, Andre Gr?be wrote:
>> I am using dovecot (see dovecot -n output below) with metadata plugin
in
>> my own plugin. I want to insert and delete some large metadata
(>4KiB).
>> It seems, that the delete in a second call of my plugin smashes the
file
>> dovecot.dict in users dir:
> 
> That's an error in dict-file.c, file_dict_refresh():
> 
>  while ((key = i_stream_read_next_line(input)) != NULL &&
>         (value = i_stream_read_next_line(input)) != NULL) {
> 
> If reading the value requires reading from the stream (because the value
doesn't fit into the input buffer), the key is overwritten. The attached
patch duplicates the key before reading the value.
Committed: http://hg.dovecot.org/dovecot-2.1/rev/6e53209030f6