Greetings.
I use Dovecot 2.2.29.1 as my IMAP server. Owing to a bug in my mail
client [1], several unique messages (mostly in my Sent folder) have
duplicate Message-ID headers. Dovecot itself doesn't seem to be
bothered by this, though my mail client is confused by the false
duplicates. (It screws up the threading display, and results in data
loss when I run the client's deduplication filter.)
I would like to change the Message-ID headers in the "duplicate"
message
files stored in my IMAP server so that they are unique. I realize that
this has disadvantages of its own, but I can't think of a better
alternative.
So this leads me to two questions:
1) Will Dovecot get confused if I simply open the message files in a
text editor and manually change their Message-ID headers? If so, how
should I go about changing the Message-ID headers?
2) Does there exist any tool that will scan a maildir folder to find
messages with duplicate Message-IDs, and automatically rewrite the
Message-ID headers so that they are (probably) unique? I could
probably kludge together a quick shell script, but if someone's already
gone to the trouble of writing such a tool and making it relatively
robust, I'd rather use that instead.
Regards,
Tristan
[1]
http://www.thewildbeast.co.uk/claws-mail/bugzilla/show_bug.cgi?id=3828
--
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Tristan Miller
Free Software developer, ferret herder, logologist
https://logological.org/
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20170610/b3dbc893/attachment.sig>
> On June 10, 2017 at 10:30 PM Tristan Miller <psychonaut at nothingisreal.com> wrote: > > > Greetings. > > I use Dovecot 2.2.29.1 as my IMAP server. Owing to a bug in my mail > client [1], several unique messages (mostly in my Sent folder) have > duplicate Message-ID headers. Dovecot itself doesn't seem to be > bothered by this, though my mail client is confused by the false > duplicates. (It screws up the threading display, and results in data > loss when I run the client's deduplication filter.) > > I would like to change the Message-ID headers in the "duplicate" message > files stored in my IMAP server so that they are unique. I realize that > this has disadvantages of its own, but I can't think of a better > alternative. > > So this leads me to two questions: > > 1) Will Dovecot get confused if I simply open the message files in a > text editor and manually change their Message-ID headers? If so, how > should I go about changing the Message-ID headers? >Expunge the mails, then edit message-id's and import them back.> 2) Does there exist any tool that will scan a maildir folder to find > messages with duplicate Message-IDs, and automatically rewrite the > Message-ID headers so that they are (probably) unique? I could > probably kludge together a quick shell script, but if someone's already > gone to the trouble of writing such a tool and making it relatively > robust, I'd rather use that instead. >https://wiki2.dovecot.org/Tools/Doveadm/Deduplicate should let you expunge duplicate messages.> Regards, > Tristan > > [1] > http://www.thewildbeast.co.uk/claws-mail/bugzilla/show_bug.cgi?id=3828 > > -- > =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- > Tristan Miller > Free Software developer, ferret herder, logologist > https://logological.org/ > =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Greetings. On Sun, 11 Jun 2017 08:25:33 +0300 (EEST), Aki Tuomi <aki.tuomi at dovecot.fi> wrote:> > 2) Does there exist any tool that will scan a maildir folder to find > > messages with duplicate Message-IDs, and automatically rewrite the > > Message-ID headers so that they are (probably) unique? I could > > probably kludge together a quick shell script, but if someone's > > already gone to the trouble of writing such a tool and making it > > relatively robust, I'd rather use that instead. > > > > https://wiki2.dovecot.org/Tools/Doveadm/Deduplicate should let you > expunge duplicate messages.Yeah, simply removing duplicates is pretty trivial, and any decent mail client will also have a nice way of doing this. The tricky part in my case is not removing the duplicates, but preserving them and giving them unique Message-IDs. If there's no existing way of automating this then I'll do as you suggest. Regards, Tristan -- =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Tristan Miller Free Software developer, ferret herder, logologist https://logological.org/ =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: <http://dovecot.org/pipermail/dovecot/attachments/20170611/d55c5497/attachment.sig>
Greetings. On Sun, 11 Jun 2017 08:25:33 +0300 (EEST), Aki Tuomi <aki.tuomi at dovecot.fi> wrote:> > On June 10, 2017 at 10:30 PM Tristan Miller > > <psychonaut at nothingisreal.com> wrote: > > > > I use Dovecot 2.2.29.1 as my IMAP server. Owing to a bug in my mail > > client [1], several unique messages (mostly in my Sent folder) have > > duplicate Message-ID headers. Dovecot itself doesn't seem to be > > bothered by this, though my mail client is confused by the false > > duplicates. (It screws up the threading display, and results in > > data loss when I run the client's deduplication filter.) > > > > I would like to change the Message-ID headers in the "duplicate" > > message files stored in my IMAP server so that they are unique. I > > realize that this has disadvantages of its own, but I can't think > > of a better alternative. > > > > So this leads me to two questions: > > > > 1) Will Dovecot get confused if I simply open the message files in a > > text editor and manually change their Message-ID headers? If so, > > how should I go about changing the Message-ID headers? > > Expunge the mails, then edit message-id's and import them back.I finally got around to this and thought I'd report on my experiences in case anyone else wants to do the same thing. From a Bash shell, I made a backup copy of the entire IMAP folder containing the messages with duplicate Message-ID headers. I then used "doveadm deduplicate" with the -m option to deduplicate the original folder. This didn't produce any logging output (even with -v), so from both the original and backup folders, I ran ls and redirected the output to a file, then used comm to find out which files had been deleted. I copied those files to a temporary directory and then rewrote the Message-ID headers as follows: for f in *; do sed -i -e 's/^Message-ID: </Message-ID: <dedup.'$(pwgen 3)'./' "$f" done So far, so good. The biggest problem was re-importing the revised messages. The usual "doveadm import" refuses to process the revised messages, throwing a bunch of "Cached message size larger than expected" errors. I eventually realized that this is because the size of the message is encoded in the filename, and dovecot actually reads this size and compares it against the actual size of the message. Unless the Message-ID headers are replaced with ones of identical length, the actual size of the new message will be different from what's encoded in the filename. The Dovecot website provides a Perl script for recomputing the message size and renaming the files accordingly: <https://www.dovecot.org/tools/maildir-size-fix.pl> Running this script threw an error about trying to pass "kill" a non-numeric argument, but it seemed to work anyway, and I was finally able to run "doveadm import". Regards, Tristan -- =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Tristan Miller Free Software developer, ferret herder, logologist https://logological.org/ =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-