dovecot - Jul 2016 - IMAP flags and dovecot-keywords not working as expected

I've converted most of our users from Outlook to Thunderbird.  One important
feature of
Thunderbird is that it pays attention to the IMAP non-standard message flags via
the 'tags'
feature (see http://wiki2.dovecot.org/MailboxFormat/Maildir).  This is important
because 2
users make extensive use of Outlook categories (Tags, in Thunderbird).  I found
that when I set
a message to the 1st Tbird tag 0 (Important), the corresponding IMAP file got an
'a' suffix.
Likewise, if I manually added an 'a' suffix to an IMAP file the
corresponding tag was set
displayed in Thunderbird.  This made it easy for me to exports Outlook
Categories and set IMAP
messages flags accordingly.  This worked perfectly on the user I was
experimenting with (me).

Recently, I converted one of these Outlook 'categories' users to
Thunderbird.  I added the
appropriate flag letters to their IMAP messages, changed their tag names in
Thunderbird, and
created the following dovecot-keywords file as:

0 Board_and_Committee
1 Completed
2 Health_Care_meetings
3 Notifications
4 OSHP-DAS
5 personal_or_To_Do
6 Retirement_exits
7 $label5
8 Junk
9 $Forwarded
10 $MDNSent
11 $label2

However, none of the tags show up correctly in Thunderbird.  If I manually set a
message to
have a tag of 0, the corresponding IMAP file gets a flag of 'm', not
'a' and the following is
added to the dovecot-keywords files:

12 $label1

How can I fix this? Where is "$label1" text defined? Why did
Thunderbird not snag the text for
'0' from the dovecot-keywords file and give the IMAP file a tag of
'a'?

My current theory is that the "Default" Thunderbird Tags corresponding
to IMAP flags are not
changeable and if new tags are create in Tbird, they get new flag letters.  That
would, of
course, mean that if a user changes Thunderbird tag name, they would lose all
tag settings on
existing message.  That doesn't seem right and I hope my theory is wrong. 

Any insight would be appreciated.

--Mark

Am 2016-07-30 um 08:00 schrieb Mark Foley:
?
>
> However, none of the tags show up correctly in Thunderbird.  If I manually
set a message to
> have a tag of 0, the corresponding IMAP file gets a flag of 'm',
not 'a' and the following is
> added to the dovecot-keywords files:
>
> 12 $label1
>
> How can I fix this? Where is "$label1" text defined? Why did
Thunderbird not snag the text for
> '0' from the dovecot-keywords file and give the IMAP file a tag of
'a'?
Thunderbird flags are stored in the users prefs.js, eg:
- user_pref("mailnews.tags.$label1.tag", "Important");
- user_pref("mailnews.tags.$label1.color", "#FF0000");

A kind of key->value assignment. The "$label[1-9]" keys are
special,
where the number magically corresponds to the keyboard shorcut to tag 
messages, 0 meaning clear all tags.

There can be more than nine tags, but they wont have a shortcut then:
- user_pref("mailnews.tags.ten.tag", "ten");

1) The server will only ever see the key. The user will only ever see 
the value.

2) If you rename a label in TB, then only the value will change and the 
server will still see the same key as before.

3) If you rename a key in dovecot, TB will not create a label for it and 
the affected messages will no longer appear tagged, if TB does not know 
about the key.

4) Dovecot adds to the keywords as it receives requests from clients: 
Very likely there is a limit of 26 (letters of the alphabet) per 
account; a-d=0-3 are already taken for internal use, so 22 remain.

> My current theory is that the "Default" Thunderbird Tags
corresponding to IMAP flags are not
> changeable and if new tags are create in Tbird, they get new flag letters. 
That would, of
> course, mean that if a user changes Thunderbird tag name, they would lose
all tag settings on
> existing message.  That doesn't seem right and I hope my theory is
wrong.
I think you are mostly wrong: as long as you only use TB to work and as 
long as you do not exceed the limit you should be save.

Notice that tags are a scarce resource: any key you ever created counts 
toward the limit - reusing old tags requires you to text-edit both 
dovecot-keywords and TB prefs.js.

-- 
peter

I think I've partially sorted out my issues.  First off, as the wiki:
http://wiki2.dovecot.org/MailboxFormat/Maildir points out: "The file
[dovecot-keywords] must
not be directly modified ...", which is what I tried.  So the
dovecot-keywords file I manually
created (entries 0-6):

0 Board_and_Committee
1 Completed
2 Health_Care_meetings
3 Notifications
4 OSHP-DAS
5 personal_or_To_Do
6 Retirement_exits
7 $label5
8 Junk
9 $Forwarded
10 $MDNSent
11 $label2

Was not used by Thunderbird. So, I removed that file, went into Thunderbird, and
manually set a
random message to tag 1 which I had renamed as "Board and Committee".
And behold! a new
dovecot-keyword file was created with the single entry:

0 $label1

As you (Peter) point out, Thunderbird retrieves the text for these from
prefs.js.  The reason,
therefore, I have e.g.  "11 $label2" in my original hand-edited file
is that Thunderbird caused
what I had designated as "Health Care Meeting" to be added to #11
since the previous slots were
already used by other things which it did not recognize.  Hence, they ended up
with a IMAP flag
of 'l'. 

I repeated this for a total of 4 more tags, resulting in dovecot-keyword
entries:

0 $label1
1 $label2
2 $label3
3 $label4
4 $label5

Now, setting the tag in Tbird results in the correct text and color being shown
for the message
and the resulting IMAP tag being set correspondingly to 'a' thru
'e'. Flush with victory, I
then foolish violated the "don't modify directly principle" by
adding:

5 $label6
6 $label7

Which I thought would correspond to my remaining TB tags. Of course, wrong. When
I then set a
message to the 6th tag (keyword entry 5, $label6), it did not do it. Instead,
for my last 2
tags it created 2 new entries:

7 personal_or_to_do
8 retirement_exits

So, IMAP flags 'f' and 'g' are skipped and "Personal or To
Do" tags get an 'h' flag and
"Retirement Exits" get an 'i' flag. What was I thinking! Not
worth starting over as this user
had no Outlook categories set to these values, so so-what if that user's
files get flagged with
'h', 'i'.

Also interesting to note that the actual text of these last 2 tags is stored in
the
dovecot-keywords file. I supposed this means that Thunderbird has a limit of 5
(or 7?) $label's
and any added beyond that get stored differently.

But, I'm not finished.  Of course, Thunderbird expects to set tags/IMAPflas
on virgin messages
in the user's mail folders.  However, my need is to import Outlook
categories and have
Thunderbird interpret the IMAP flags correctly having never set them directly. 

So, I repeated the same procedure in other folders with messages having Outlook
categories.
The problem here is that setting a message to "Board and Committee"
(Tbird tag 1) did not
necessarily resulting in an IMAP flag of 'a'.  In some folders it was
'b', some 'c' and some
'd'.  I think my problem here was that I had manually set the IMAP flags
on these message
beforehand.  So (I think) when Thunderbird, coordinating with Dovecot, went to
set Tag 1, it
saw that flag 'a' was already used and it picked the next free flag. 
Some folders had IMAP
files with 2 and 3 flags set, hence Tag 1 getting set to 'c' or
'd' in those folders.  I
believe if I had not initially pre-set the IMAP flags on these files I would
have seen the
correct correspondance: Tag 1 = flag 'a', Tag 2 = flag 'b', etc.

What I then did in these cases was first, find out what IMAP flag a Tbird tagged
file would be
set to (e.g.  'c'), then renamed all my *a files to e.g. *c. That
worked.  Suddenly, folders
with 100 flags thusly set showed up with correct tags and colors in Thunderbird.

To summarize, and this what I will do next time:

1. DO NOT pre-add IMAP flags to mail files.

2. For each folder in which there are Outlook categorized message, set an
arbitrary message in
that folder to each of the Thunderbird flags corresponding to Outlook categories
to determine
the correct flag letter -- which also causes Thunderbird and dovecot to save
their respective
settings. 

3. Then, rename each mail file to have the correct IMAP flag(s). That can be
quicky done using a
script and the list of message having Outlook categories.

I believe this procedure will work correctly and I will confirm that when I
process my 2nd
Outlook user (who is, btw, the organization Director and who is a super user of
Outlook
categories! Failure is not an option!). 

For anyone needing to do the same thing (migrate Outlook categories), I've
included below a VB
script to run in Outlook which will output a list of all messages having
categories.  The IMAP
mail file's Message ID can be used to locate the categorized message file in
the IMAP folder
hierarchy and the category name (following the "~") indicates which
Thunderbird tag to map it
to.  I'll not include the bash script to mass-append IMAP flags to these
files as that script
will need some revising based on my recent experimentation, but should be a
rather simple bash
exercise in any case. 

Note that the Outlook messages are also the same MAPI files, only the client
used (Outlook
versus Thunderbird) are different. Outlook does not set IMAP flags to designate
categories.
Categories are apparently stored in the user's .pst file.

------------CUT----------
Public Sub ListOutlookFolders()

    Dim olApp As Outlook.Application
    Dim olNamespace As Outlook.Namespace
    Dim olFolder As Outlook.MAPIFolder

    Set olApp = New Outlook.Application
    Set olNamespace = olApp.GetNamespace("MAPI")

    For Each olFolder In olNamespace.Folders
        Debug.Print olFolder.Name; ":", olFolder.Description
        ListFolders olFolder, 1
    Next

    Set olFolder = Nothing
    Set olNamespace = Nothing
    Set olApp = Nothing

End Sub

Sub ListFolders(myFolder As Outlook.MAPIFolder, Level As Integer)
    Dim olFolder As Outlook.MAPIFolder
'  go through each email
    scanFolder myFolder

'  Now we'll check for subfolders
    For Each olFolder In myFolder.Folders
'        Debug.Print ":"; String(Level * 2, "-");
olFolder.Name

'        go through each email
        scanFolder olFolder

        If olFolder.Folders.Count > 0 Then
            ListFolders olFolder, Level + 1
        End If
    Next
End Sub

Sub scanFolder(sFolder As Outlook.MAPIFolder)
Dim src As Folder
Dim oItem As Object
Dim propertyAccessor As Outlook.propertyAccessor
Set src = sFolder

Dim strHeader As String

For Each oItem In src.Items
    If TypeOf oItem Is Outlook.MailItem And oItem.Categories <>
"" Then
'        Debug.Print "Cat: " + oItem.Categories
        Set propertyAccessor = oItem.propertyAccessor
        header
propertyAccessor.GetProperty("http://schemas.microsoft.com/mapi/proptag/0x007D001E")
        Dim headerLines() As String
        headerLines() = Split(header, vbCrLf)

        Dim thisHeader As Variant

        For Each thisHeader In headerLines
            If InStr(thisHeader, "Message-ID:") > 0 Then
               Debug.Print thisHeader + "~" + oItem.Categories
               Exit For
           End If
        Next
    End If
Next
End Sub
----------CUT---------

Example of resulting output:

Message-ID: <201109011105.p81B5666028910 at webserver.ohprs.org>~Red
Category

Hopefully someone finds this useful.

THX --Mark

-----Original Message-----
> Subject: Re: IMAP flags and dovecot-keywords not working as expected
> To: dovecot at dovecot.org
> From: Peter Chiochetti <pch at myzel.net>
> Date: Sat, 30 Jul 2016 11:26:09 +0200
>
> Am 2016-07-30 um 08:00 schrieb Mark Foley:
> ?
> >
> > However, none of the tags show up correctly in Thunderbird.  If I
manually set a message to
> > have a tag of 0, the corresponding IMAP file gets a flag of
'm', not 'a' and the following is
> > added to the dovecot-keywords files:
> >
> > 12 $label1
> >
> > How can I fix this? Where is "$label1" text defined? Why did
Thunderbird not snag the text for
> > '0' from the dovecot-keywords file and give the IMAP file a
tag of 'a'?
>
> Thunderbird flags are stored in the users prefs.js, eg:
> - user_pref("mailnews.tags.$label1.tag", "Important");
> - user_pref("mailnews.tags.$label1.color", "#FF0000");
>
> A kind of key->value assignment. The "$label[1-9]" keys are
special,
> where the number magically corresponds to the keyboard shorcut to tag 
> messages, 0 meaning clear all tags.
>
> There can be more than nine tags, but they wont have a shortcut then:
> - user_pref("mailnews.tags.ten.tag", "ten");
>
> 1) The server will only ever see the key. The user will only ever see 
> the value.
>
> 2) If you rename a label in TB, then only the value will change and the 
> server will still see the same key as before.
>
> 3) If you rename a key in dovecot, TB will not create a label for it and 
> the affected messages will no longer appear tagged, if TB does not know 
> about the key.
>
> 4) Dovecot adds to the keywords as it receives requests from clients: 
> Very likely there is a limit of 26 (letters of the alphabet) per 
> account; a-d=0-3 are already taken for internal use, so 22 remain.
>
>
> > My current theory is that the "Default" Thunderbird Tags
corresponding to IMAP flags are not
> > changeable and if new tags are create in Tbird, they get new flag
letters.  That would, of
> > course, mean that if a user changes Thunderbird tag name, they would
lose all tag settings on
> > existing message.  That doesn't seem right and I hope my theory is
wrong.
>
> I think you are mostly wrong: as long as you only use TB to work and as 
> long as you do not exceed the limit you should be save.
>
> Notice that tags are a scarce resource: any key you ever created counts 
> toward the limit - reusing old tags requires you to text-edit both 
> dovecot-keywords and TB prefs.js.
>
> -- 
> peter
>