Arnold Opio Oree
2019-Jul-04 13:52 UTC
Applying Dovecot for a large / deep folder-hierarchy archive.
Hi all, The guidance provided so far has been really helpful, and has helped a great deal to bringing down wasted energy on finding and executing a viable path. I am now at the final due action to complete our Dovecot application to our use-case, but am stuck on an issue that I cannot find any easily accessible documentation on. Generally this is what has been done: 1. Uploaded the enterprise data PST to the target groupware server. 2. Prepared the server by changing the mailbox format to sdbox and the the Dovecot mail location to mail_location=/var/vmail/domain/user/mail/ 3. Converted the pst (on-server) to a recursive mbox hierarchy using readpst 4. Executed doveadm-sync to convert mbox hierarchy data into sdbox and to copy it into the enterprise archive user's mailboxes 4.i. The biggest issue I faced at this point was doveadm-sync saying that the source and destination pointed to the same location, whereas they clearly did not. 4.i.a. I resolved this by removing the location= setting from the target namespace, and allowing it to default to mail_location setting, and then using a completely different DIRNAME for the import doveadm-sync execution (which was the desired final DIRNAME); I then once the sync had been successful, changed the mail_location DIRNAME so that it pointed to the imported mail DIRNAME; and hence the imported email data was in the live mailboxes 4.i.b. doveadm-import failed several times, and was throwing quite inexplicable errors, so I moved onto doveadm-sync 4.i.c. I also had to make sure that the source and destination folder names matched, otherwise doveadm-syc threw very many errors and only partially imported the data 4.i.d. An issue which I decided just to live with is that an mbox DIRNAME was added to each mailbox as well as the DIRNAME specified so the path to mail is mbox/dbox-Mails. My thought is that with the data live on an IMAP server it will be possible to do a dysync through TCP to correct this problem. The final issue that I am facing now, is that when readpst finds empty folders in the source pst hierarchy, it does not create an mbox file in the mbox hierarchy folder space. This causes doveadm-sync to not create the target data required for its mailbox structure i.e. DIRNAME sub- folder and index file (with our configuration). At this point either doveadm-sync or the dovecot process makes these empty folders not selectable. The question now is how would I go about making all of these folders selectable, e.g. with an internal or external command line tool to change flags / create necessary sdbox mailbox constituent data? -----Original Message----- From: Arnold Opio Oree via dovecot <dovecot at dovecot.org> Reply-To: arnoldoree at parallaxict.com, Arnold Opio Oree < arnold.oree at parallaxict.com> To: Robert Schetterer <rs at sys4.de>, dovecot at dovecot.org Subject: Re: Applying Dovecot for a large / deep folder-hierarchy archive. Date: Thu, 27 Jun 2019 12:05:35 +0100> Also you may run into client limits.... i ve seen this with outlook, > apple mail, thunderbird via imap in the pastThanks for this note Robert, it was not really an aspect that I wasconsidering. We are operating our groupware services user access through bothEvolution Groupware and KDE Kontact / KMail on Debian Linuxworkstations. Hopefully if there is a client issue it should be localto only one groupware client. I will be sure to study / investigate in this - client - area should any issues that are not traceable to the server-side arise. Many thanks, Arnold Opio OreeChief Executive OfficerParallax Digital Technologies arnoldoree at parallaxdt.com http://www.parallaxdt.com tel : +44 (0) 333 577 8587fax : +44 (0) 20 8711 2477 Parallax Digital Technologies is a trading name of Parallax GlobalLimited. U.K. Co. No. 08836288 The contents of this e-mail are confidential. If you are not theintended recipient you are to delete this e-mail immediately, disregardits contents and disclose them to no other persons. -----Original Message-----From: Robert Schetterer via dovecot < dovecot at dovecot.org>Reply-To: Robert Schetterer <rs at sys4.de>To: dovecot at dovecot.org Subject: Re: Applying Dovecot for a large / deep folder- hierarchyarchive.Date: Thu, 27 Jun 2019 12:53:49 +0200 Am 27.06.2019 um 07:35 schrieb Aki Tuomi via dovecot:> On 26.6.2019 22.12, Arnold Opio Oree via dovecot wrote: > > Hello to you all, > > I'd like to ask about my intended application of Dovecot to createa > > folder-hierarchy for storing our enterprise emails, which > > aretreated as live data rather than archives for compliance > > oroccasional / reactive retrieval. > > The data is presently not that large (a few gigabytes), but it > > isexpected to grow rapidly. Up to this stage the data has > > beencontainedin a Microsoft Exchange mailbox (2013), and then in an > > offline PST.The move to the offline PST was by necessity, as the > > large numberoffolders, and depth of hierarchy to my best > > understanding caused theexchange server / outlook / evolution mail > > clients to begin tomalfunction. To cope with this the archive was > > broken up and thebulk stored in the offline PST and the most active > > componentsstored inonline Exchange mailboxes. > > I have some understanding of the fs mbox format, and also > > themitigations to be made for certain characters / strings. My > > mainconcern is whether Dovecot is likely to be able to cope well > > with alarge number of folders / depth of hierarcy. > > I will really appreciate any help / advice you can give. > > Best regards, > > Arnold Opio Oree > > Hi! > Dovecot 2.2.34/2.3 supports unlimited folder depth, the only > limitingfactor is that the total name may not exceed 4096 bytes. > Alsoindividualfolder names may not exceed 255 bytes. > Prior to that the limit is 255 per folder up to 16 levels. > I can't recommend using 'mbox' storage format, please consider > usingmaildir or sdbox instead. > AkiMany thanks, Arnold Opio Oree Chief Executive Officer Parallax Digital Technologies arnoldoree at parallaxdt.com http://www.parallaxdt.com tel : +44 (0) 333 577 8587 fax : +44 (0) 20 8711 2477 Parallax Digital Technologies is a trading name of Parallax Global Limited. U.K. Co. No. 08836288 The contents of this e-mail are confidential. If you are not the intended recipient you are to delete this e-mail immediately, disregard its contents and disclose them to no other persons. -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://dovecot.org/pipermail/dovecot/attachments/20190704/d600dbe2/attachment.html>
Arnold Opio Oree
2019-Jul-07 15:12 UTC
Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS!
Dovecot Team, I'd like to report a number of bugs, that are to my view all critical. System: Replicated on multiple Debian 10 (Buster) systems Dovecot Version(s): 2.3.4.1 doveadm-sync -1/general 1) If DIRNAMEs are not different between command line and mail_location doveadm sync will fail, saying that the source and destination directories are the same 2) The -n / -N flags do not work, and a sync will fail strangely if location is specified in the namespace definition 3) Adds mbox to path name under mailbox directory (where syncing from an mbox source) 4) Not having the mailboxes at source named the same as those at destination causes errors and partial sync 5) Not having the target mailboxes formatted to receive the sync (/<mailboxroot>/DIRNAME/) will cause sync errors. doveadm-sync 1) With large synchronizations UIDs are corrupted where multiple syncs are executed and the program can no longer synchronize dovecot 1) Panics and fails to expand ~ to user home: observed cases are where multiple namespaces are being used Please let me know if you need me to elaborate or to provide any further information that you may need to replicate the bugs, or if I can help in any other way. With regards to the last error that I requested help on i.e. \Noselect. This has been resolved more-or-less by the workarounds that I have implemented for the bugs reported above. I have seen a number of threads whilst researching the \Noselect issue where people have been very confused. My finding was that \Noselect is a function of the IMAP specification server-side implementation RFC3501 (https://tools.ietf.org/html/rfc3501#section-6.3.6). And for me the server was returning directories with \Noselect because the mailboxes were malformed on account of dovadm-sync errors. In order to fix this I formed a bash command to transverse the mailbox hierarchy and create the missing folders critical to the sdbox format, namely DIRNAME. Kind regards, Arnold Opio Oree -----Original Message----- From: Arnold Opio Oree via dovecot <dovecot at dovecot.org> Reply-To: arnoldoree at parallaxict.com, Arnold Opio Oree < arnold.oree at parallaxict.com> To: dovecot at dovecot.org Cc: rs at sys4.de, aki.tuomi at open-xchange.com Subject: Re: Applying Dovecot for a large / deep folder-hierarchy archive. Date: Thu, 04 Jul 2019 14:52:28 +0100 Hi all, The guidance provided so far has been really helpful, and has helped a great deal to bringing down wasted energy on finding and executing a viable path. I am now at the final due action to complete our Dovecot application to our use-case, but am stuck on an issue that I cannot find any easily accessible documentation on. Generally this is what has been done: 1. Uploaded the enterprise data PST to the target groupware server. 2. Prepared the server by changing the mailbox format to sdbox and the the Dovecot mail location to mail_location=/var/vmail/domain/user/mail/ 3. Converted the pst (on-server) to a recursive mbox hierarchy using readpst 4. Executed doveadm-sync to convert mbox hierarchy data into sdbox and to copy it into the enterprise archive user's mailboxes 4.i. The biggest issue I faced at this point was doveadm-sync saying that the source and destination pointed to the same location, whereas they clearly did not. 4.i.a. I resolved this by removing the location= setting from the target namespace, and allowing it to default to mail_location setting, and then using a completely different DIRNAME for the import doveadm-sync execution (which was the desired final DIRNAME); I then once the sync had been successful, changed the mail_location DIRNAME so that it pointed to the imported mail DIRNAME; and hence the imported email data was in the live mailboxes 4.i.b. doveadm-import failed several times, and was throwing quite inexplicable errors, so I moved onto doveadm-sync 4.i.c. I also had to make sure that the source and destination folder names matched, otherwise doveadm-syc threw very many errors and only partially imported the data 4.i.d. An issue which I decided just to live with is that an mbox DIRNAME was added to each mailbox as well as the DIRNAME specified so the path to mail is mbox/dbox-Mails. My thought is that with the data live on an IMAP server it will be possible to do a dysync through TCP to correct this problem. The final issue that I am facing now, is that when readpst finds empty folders in the source pst hierarchy, it does not create an mbox file in the mbox hierarchy folder space. This causes doveadm-sync to not create the target data required for its mailbox structure i.e. DIRNAME sub- folder and index file (with our configuration). At this point either doveadm-sync or the dovecot process makes these empty folders not selectable. The question now is how would I go about making all of these folders selectable, e.g. with an internal or external command line tool to change flags / create necessary sdbox mailbox constituent data? Many thanks, Arnold Opio Oree Chief Executive Officer Parallax Digital Technologies arnoldoree at parallaxdt.com http://www.parallaxdt.com tel : +44 (0) 333 577 8587 fax : +44 (0) 20 8711 2477 Parallax Digital Technologies is a trading name of Parallax Global Limited. U.K. Co. No. 08836288 The contents of this e-mail are confidential. If you are not the intended recipient you are to delete this e-mail immediately, disregard its contents and disclose them to no other persons. -----Original Message----- From: Arnold Opio Oree via dovecot <dovecot at dovecot.org> Reply-To: arnoldoree at parallaxict.com, Arnold Opio Oree < arnold.oree at parallaxict.com> To: Robert Schetterer <rs at sys4.de>, dovecot at dovecot.org Subject: Re: Applying Dovecot for a large / deep folder-hierarchy archive. Date: Thu, 27 Jun 2019 12:05:35 +0100> Also you may run into client limits.... i ve seen this with outlook, > apple mail, thunderbird via imap in the pastThanks for this note Robert, it was not really an aspect that I was considering. We are operating our groupware services user access through both Evolution Groupware and KDE Kontact / KMail on Debian Linux workstations. Hopefully if there is a client issue it should be local to only one groupware client. I will be sure to study / investigate in this - client - area should any issues that are not traceable to the server-side arise. Many thanks, Arnold Opio Oree Chief Executive Officer Parallax Digital Technologies arnoldoree at parallaxdt.com http://www.parallaxdt.com tel : +44 (0) 333 577 8587 fax : +44 (0) 20 8711 2477 Parallax Digital Technologies is a trading name of Parallax Global Limited. U.K. Co. No. 08836288 The contents of this e-mail are confidential. If you are not the intended recipient you are to delete this e-mail immediately, disregard its contents and disclose them to no other persons. -----Original Message----- From: Robert Schetterer via dovecot < dovecot at dovecot.org>Reply-To: Robert Schetterer < rs at sys4.de>To: dovecot at dovecot.org Subject: Re: Applying Dovecot for a large / deep folder-hierarchy archive. Date: Thu, 27 Jun 2019 12:53:49 +0200 Am 27.06.2019 um 07:35 schrieb Aki Tuomi via dovecot:> On 26.6.2019 22.12, Arnold Opio Oree via dovecot wrote: > > Hello to you all, > > > > I'd like to ask about my intended application of Dovecot to create > > a folder-hierarchy for storing our enterprise emails, which are > > treated as live data rather than archives for compliance or > > occasional / reactive retrieval. > > > > The data is presently not that large (a few gigabytes), but it is > > expected to grow rapidly. Up to this stage the data has been > > contained > > in a Microsoft Exchange mailbox (2013), and then in an offline PST. > > The move to the offline PST was by necessity, as the large number > > of > > folders, and depth of hierarchy to my best understanding caused the > > exchange server / outlook / evolution mail clients to begin to > > malfunction. To cope with this the archive was broken up and the > > bulk stored in the offline PST and the most active components > > stored in > > online Exchange mailboxes. > > > > I have some understanding of the fs mbox format, and also the > > mitigations to be made for certain characters / strings. My main > > concern is whether Dovecot is likely to be able to cope well with a > > large number of folders / depth of hierarcy. > > > > I will really appreciate any help / advice you can give. > > > > Best regards, > > > > Arnold Opio Oree > > > > Hi! > > Dovecot 2.2.34/2.3 supports unlimited folder depth, the only limiting > factor is that the total name may not exceed 4096 bytes. Also > individual > folder names may not exceed 255 bytes. > > Prior to that the limit is 255 per folder up to 16 levels. > > I can't recommend using 'mbox' storage format, please consider using > maildir or sdbox instead. > > Aki >-- Dear [Recipient], [Content] Regards, Arnold Opio Oree Chief Executive Officer Parallax Digital Technologies arnoldoree at parallaxdt.com http://www.parallaxdt.com tel : +44 (0) 333 577 8587 fax : +44 (0) 20 8711 2477 Parallax Digital Technologies 78A Hatfeild Mead London SM4 5PF Parallax Digital Technologies is a trading name of Parallax Global Limited. U.K. Co. No. 08836288 The contents of this e-mail are confidential. If you are not the intended recipient you are to delete this e-mail immediately, disregard its contents and disclose them to no other persons. Chief Executive Officer Parallax Digital Technologies arnoldoree at parallaxdt.com http://www.parallaxdt.com tel : +44 (0) 333 577 8587 fax : +44 (0) 20 8711 2477 Parallax Digital Technologies is a trading name of Parallax Global Limited. U.K. Co. No. 08836288 The contents of this e-mail are confidential. If you are not the intended recipient you are to delete this e-mail immediately, disregard its contents and disclose them to no other persons. -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://dovecot.org/pipermail/dovecot/attachments/20190707/6dc7d717/attachment-0001.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: divide.jpg Type: image/jpeg Size: 4590 bytes Desc: not available URL: <https://dovecot.org/pipermail/dovecot/attachments/20190707/6dc7d717/attachment-0002.jpg> -------------- next part -------------- A non-text attachment was scrubbed... Name: pdtsignaturelogo.jpg Type: image/jpeg Size: 35023 bytes Desc: not available URL: <https://dovecot.org/pipermail/dovecot/attachments/20190707/6dc7d717/attachment-0003.jpg>
Aki Tuomi
2019-Jul-08 05:48 UTC
Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS!
<!doctype html>
<html>
<head>
<meta charset="UTF-8">
</head>
<body>
<div>
Hi!
</div>
<div>
<br>
</div>
<div>
Thank you for reporting these. We will look into them. In the mean time, can
you see if any of these are fixed in 2.3.6?
</div>
<div>
<br>
</div>
<div>
Aki
</div>
<blockquote type="cite">
<div>
On 07/07/2019 18:12 Arnold Opio Oree via dovecot <dovecot@dovecot.org>
wrote:
</div>
<div>
<br>
</div>
<div>
<br>
</div>
<div>
Dovecot Team,
</div>
<div>
<br>
</div>
<div>
I'd like to report a number of bugs, that are to my view all critical.
</div>
<div>
<br>
</div>
<div>
System: Replicated on multiple Debian 10 (Buster) systems
</div>
<div>
Dovecot Version(s): 2.3.4.1
</div>
<div>
<br>
</div>
<div>
doveadm-sync -1/general
</div>
<div>
<br>
</div>
<div>
1) If DIRNAMEs are not different between command line and mail_location
doveadm sync will fail, saying that the source and destination directories are
the same
</div>
<div>
<br>
</div>
<div>
2) The -n / -N flags do not work, and a sync will fail strangely if location
is specified in the namespace definition
</div>
<div>
<br>
</div>
<div>
3) Adds mbox to path name under mailbox directory (where syncing from an
mbox source)
</div>
<div>
<br>
</div>
<div>
4) Not having the mailboxes at source named the same as those at destination
causes errors and partial sync
</div>
<div>
<br>
</div>
<div>
5) Not having the target mailboxes formatted to receive the sync
(/<mailboxroot>/DIRNAME/) will cause sync errors.
</div>
<div>
<br>
</div>
<div>
doveadm-sync
</div>
<div>
<br>
</div>
<div>
1) With large synchronizations UIDs are corrupted where multiple syncs are
executed and the program can no longer synchronize
</div>
<div>
<br>
</div>
<div>
dovecot
</div>
<div>
<br>
</div>
<div>
1) Panics and fails to expand ~ to user home: observed cases are where
multiple namespaces are being used
</div>
<div>
<br>
</div>
<div>
Please let me know if you need me to elaborate or to provide any further
information that you may need to replicate the bugs, or if I can help in any
other way.
</div>
<div>
<br>
</div>
<div>
With regards to the last error that I requested help on i.e. \Noselect. This
has been resolved more-or-less by the workarounds that I have implemented for
the bugs reported above.
</div>
<div>
<br>
</div>
<div>
I have seen a number of threads whilst researching the \Noselect issue where
people have been very confused. My finding was that \Noselect is a function of
the IMAP specification server-side implementation RFC3501 (
<a
href="http://h5lm.mjt.lu/lnk/AMwAAEPRQ68AAAAAAAAAAKAFynQAASKA_AUAAAAAAAf-tABdIgvzBSx7xYU6SQW2MDXORHSy0wAHm5I/2/cturGvUAbhIkMPkkWYpZGw/aHR0cHM6Ly90b29scy5pZXRmLm9yZy9odG1sL3JmYzM1MDEjc2VjdGlvbi02LjMu#section-6.3.">https://tools.ietf.org/html/rfc3501#section-6.3.</a>6).
And for me the server was returning directories with \Noselect because the
mailboxes were malformed on account of dovadm-sync errors. In order to fix this
I formed a bash command to transverse the mailbox hierarchy and create the
missing folders critical to the sdbox format, namely DIRNAME.
</div>
<div>
<br>
</div>
<div>
Kind regards,
</div>
<div>
<br>
</div>
<div>
Arnold Opio Oree
</div>
<div>
<div>
Chief Executive Officer
</div>
<div>
Parallax Digital Technologies
</div>
<div>
<br>
</div>
<div>
<a
href="mailto:arnoldoree@parallaxdt.com"><strong><span
style="color:
#850404;">arnoldoree@parallaxdt.com</span></strong></a>
</div>
<div>
<br>
</div>
<div>
<a
href="http://h5lm.mjt.lu/lnk/AMwAAEPRQ68AAAAAAAAAAKAFynQAASKA_AUAAAAAAAf-tABdIgvzBSx7xYU6SQW2MDXORHSy0wAHm5I/3/oKwvqwVipSqwvpqdaumlnA/aHR0cDovL3d3dy5wYXJhbGxheGR0LmNvbQ"><strong><span
style="color:
#850404;">http://www.parallaxdt.com</span></strong></a>
</div>
<div>
<br>
</div>
<div>
<strong>tel :</strong> +44 (0) 333 577 8587
</div>
<div>
<strong>fax :</strong> +44 (0) 20 8711 2477
</div>
<div>
<br>
</div>
<div>
<span style="font-size: small;">Parallax Digital
Technologies is a trading name of Parallax Global Limited. U.K. Co. No. <span
style="color: #850404;">08836288</span></span>
</div>
<div>
<br>
</div>
<div>
<span style="color: #c0c0c0; font-size: xx-small;">The
contents of this e-mail are confidential. If you are not the intended recipient
you are to delete this e-mail immediately, disregard its contents and disclose
them to no other </span>persons.
</div>
</div>
<div>
<br>
</div>
<div>
-----Original Message-----
</div>
<div>
<strong>From</strong>: Arnold Opio Oree via dovecot <
<a
href="mailto:Arnold%20Opio%20Oree%20via%20dovecot%20%3cdovecot@dovecot.org%3e">dovecot@dovecot.org</a>>
</div>
<div>
<strong>Reply-To</strong>:
<a
href="mailto:arnoldoree@parallaxict.com">arnoldoree@parallaxict.com</a>,
Arnold Opio Oree <
<a
href="mailto:Arnold%20Opio%20Oree%20%3carnold.oree@parallaxict.com%3e">arnold.oree@parallaxict.com</a>>
</div>
<div>
<strong>To</strong>:
<a
href="mailto:dovecot@dovecot.org">dovecot@dovecot.org</a>
</div>
<div>
<strong>Cc</strong>:
<a href="mailto:rs@sys4.de">rs@sys4.de</a>,
<a
href="mailto:aki.tuomi@open-xchange.com">aki.tuomi@open-xchange.com</a>
</div>
<div>
<strong>Subject</strong>: Re: Applying Dovecot for a large /
deep folder-hierarchy archive.
</div>
<div>
<strong>Date</strong>: Thu, 04 Jul 2019 14:52:28 +0100
</div>
<div>
<br>
</div>
<div>
<br>
</div>
<div>
Hi all,
</div>
<div>
<br>
</div>
<div>
The guidance provided so far has been really helpful, and has helped a great
deal to bringing down wasted energy on finding and executing a viable path. I am
now at the final due action to complete our Dovecot application to our use-case,
but am stuck on an issue that I cannot find any easily accessible documentation
on.
</div>
<div>
<br>
</div>
<div>
Generally this is what has been done:
</div>
<div>
<br>
</div>
<div>
1. Uploaded the enterprise data PST to the target groupware server.
</div>
<div>
2. Prepared the server by changing the mailbox format to sdbox and the the
Dovecot mail location to mail_location=/var/vmail/domain/user/mail/
</div>
<div>
3. Converted the pst (on-server) to a recursive mbox hierarchy using readpst
</div>
<div>
4. Executed doveadm-sync to convert mbox hierarchy data into sdbox and to
copy it into the enterprise archive user's mailboxes
</div>
<div>
4.i. The biggest issue I faced at this point was doveadm-sync saying that
the source and destination pointed to the same location, whereas they clearly
did not.
</div>
<div>
4.i.a. I resolved this by removing the location= setting from the target
namespace, and allowing it to default to mail_location = setting, and then using
a completely different DIRNAME for the import doveadm-sync execution (which was
the desired final DIRNAME); I then once the sync had been successful, changed
the mail_location DIRNAME so that it pointed to the imported mail DIRNAME; and
hence the imported email data was in the live mailboxes
</div>
<div>
4.i.b. doveadm-import failed several times, and was throwing quite
inexplicable errors, so I moved onto doveadm-sync
</div>
<div>
4.i.c. I also had to make sure that the source and destination folder names
matched, otherwise doveadm-syc threw very many errors and only partially
imported the data
</div>
<div>
4.i.d. An issue which I decided just to live with is that an mbox DIRNAME
was added to each mailbox as well as the DIRNAME specified so the path to mail
is mbox/dbox-Mails. My thought is that with the data live on an IMAP server it
will be possible to do a dysync through TCP to correct this problem.
</div>
<div>
<br>
</div>
<div>
The final issue that I am facing now, is that when readpst finds empty
folders in the source pst hierarchy, it does not create an mbox file in the mbox
hierarchy folder space. This causes doveadm-sync to not create the target data
required for its mailbox structure i.e. DIRNAME sub-folder and index file (with
our configuration). At this point either doveadm-sync or the dovecot process
makes these empty folders not selectable.
</div>
<div>
<br>
</div>
<div>
The question now is how would I go about making all of these folders
selectable, e.g. with an internal or external command line tool to change flags
/ create necessary sdbox mailbox constituent data?
</div>
<div>
<br>
</div>
<div>
<div>
Many thanks,
</div>
<div>
<br>
</div>
<div>
Arnold Opio Oree
</div>
<div>
Chief Executive Officer
</div>
<div>
Parallax Digital Technologies
</div>
<div>
<br>
</div>
<div>
<a
href="mailto:arnoldoree@parallaxdt.com"><strong><span
style="color:
#850404;">arnoldoree@parallaxdt.com</span></strong></a>
</div>
<div>
<br>
</div>
<div>
<a
href="http://h5lm.mjt.lu/lnk/AMwAAEPRQ68AAAAAAAAAAKAFynQAASKA_AUAAAAAAAf-tABdIgvzBSx7xYU6SQW2MDXORHSy0wAHm5I/4/UcYJWOlfpjTcpmsYQnA3HA/aHR0cDovL2g1bG0ubWp0Lmx1L2xuay9BTUlBQUVMcVhrb0FBQUFBQUFBQUFLQUZ5blFBQVNLQV9BVUFBQUFBQUFmLXRBQmRIZ1NkTnNtVUpISzRRdVctVVp0WndhSmdYQUFIbTVJLzEvVGtYNTI4RXZyd3p3S04yYUtHN1dGUS9hSFIwY0RvdkwzZDNkeTV3WVhKaGJHeGhlR1IwTG1OdmJR"><strong><span
style="color:
#850404;">http://www.parallaxdt.com</span></strong></a>
</div>
<div>
<br>
</div>
<div>
<strong>tel :</strong> +44 (0) 333 577 8587
</div>
<div>
<strong>fax :</strong> +44 (0) 20 8711 2477
</div>
<div>
<br>
</div>
<div>
<span style="font-size: small;">Parallax Digital
Technologies is a trading name of Parallax Global Limited. U.K. Co. No. <span
style="color: #850404;">08836288</span></span>
</div>
<div>
<br>
</div>
<div>
<span style="color: #c0c0c0; font-size: xx-small;">The
contents of this e-mail are confidential. If you are not the intended recipient
you are to delete this e-mail immediately, disregard its contents and disclose
them to no other </span>persons.
</div>
</div>
<div>
<br>
</div>
<div>
-----Original Message-----
</div>
<div>
<strong>From</strong>: Arnold Opio Oree via dovecot <
<a
href="mailto:Arnold%20Opio%20Oree%20via%20dovecot%20%3cdovecot@dovecot.org%3e">dovecot@dovecot.org</a>>
</div>
<div>
<strong>Reply-To</strong>:
<a
href="mailto:arnoldoree@parallaxict.com">arnoldoree@parallaxict.com</a>,
Arnold Opio Oree <
<a
href="mailto:Arnold%20Opio%20Oree%20%3carnold.oree@parallaxict.com%3e">arnold.oree@parallaxict.com</a>>
</div>
<div>
<strong>To</strong>: Robert Schetterer <
<a
href="mailto:Robert%20Schetterer%20%3crs@sys4.de%3e">rs@sys4.de</a>>,
<a
href="mailto:dovecot@dovecot.org">dovecot@dovecot.org</a>
</div>
<div>
<strong>Subject</strong>: Re: Applying Dovecot for a large /
deep folder-hierarchy archive.
</div>
<div>
<strong>Date</strong>: Thu, 27 Jun 2019 12:05:35 +0100
</div>
<div>
<br>
</div>
<blockquote type="cite">
<div>
Also you may run into client limits.... i ve seen this with outlook,
</div>
<div>
apple mail, thunderbird via imap in the past
</div>
</blockquote>
<div>
<br>
</div>
<div>
Thanks for this note Robert, it was not really an aspect that I was
</div>
<div>
considering.
</div>
<div>
<br>
</div>
<div>
We are operating our groupware services user access through both
</div>
<div>
Evolution Groupware and KDE Kontact / KMail on Debian Linux
</div>
<div>
workstations. Hopefully if there is a client issue it should be local
</div>
<div>
to only one groupware client.
</div>
<div>
<br>
</div>
<div>
I will be sure to study / investigate in this - client - area should any
issues that are not traceable to the server-side arise.
</div>
<div>
<br>
</div>
<div>
Many thanks,
</div>
<div>
<br>
</div>
<div>
Arnold Opio Oree
</div>
<div>
Chief Executive Officer
</div>
<div>
Parallax Digital Technologies
</div>
<div>
<br>
</div>
<div>
<pre>arnoldoree@parallaxdt.com</pre>
</div>
<div>
<br>
</div>
<div>
<br>
</div>
<div>
<pre>http://www.parallaxdt.com</pre>
</div>
<div>
<br>
</div>
<div>
<br>
</div>
<div>
tel : +44 (0) 333 577 8587
</div>
<div>
fax : +44 (0) 20 8711 2477
</div>
<div>
<br>
</div>
<div>
Parallax Digital Technologies is a trading name of Parallax Global
</div>
<div>
Limited. U.K. Co. No. 08836288
</div>
<div>
<br>
</div>
<div>
The contents of this e-mail are confidential. If you are not the
</div>
<div>
intended recipient you are to delete this e-mail immediately, disregard
</div>
<div>
its contents and disclose them to no other persons.
</div>
<div>
<br>
</div>
<div>
<br>
</div>
<div>
-----Original Message-----
</div>
<div>
From: Robert Schetterer via dovecot <
</div>
<div>
<pre>dovecot@dovecot.org</pre>
</div>
<div>
>
</div>
<div>
Reply-To: Robert Schetterer <
</div>
<div>
<pre>rs@sys4.de</pre>
</div>
<div>
>
</div>
<div>
To:
</div>
<div>
<pre>dovecot@dovecot.org</pre>
</div>
<div>
<br>
</div>
<div>
Subject: Re: Applying Dovecot for a large / deep folder-hierarchy
</div>
<div>
archive.
</div>
<div>
Date: Thu, 27 Jun 2019 12:53:49 +0200
</div>
<div>
<br>
</div>
<div>
Am 27.06.2019 um 07:35 schrieb Aki Tuomi via dovecot:
</div>
<blockquote type="cite">
<div>
On 26.6.2019 22.12, Arnold Opio Oree via dovecot wrote:
</div>
<blockquote type="cite">
<div>
Hello to you all,
</div>
<div>
<br>
</div>
<div>
I'd like to ask about my intended application of Dovecot to create
</div>
<div>
a folder-hierarchy for storing our enterprise emails, which are
</div>
<div>
treated as live data rather than archives for compliance or
</div>
<div>
occasional / reactive retrieval.
</div>
<div>
<br>
</div>
<div>
The data is presently not that large (a few gigabytes), but it is
</div>
<div>
expected to grow rapidly. Up to this stage the data has been
</div>
<div>
contained
</div>
<div>
in a Microsoft Exchange mailbox (2013), and then in an offline PST.
</div>
<div>
The move to the offline PST was by necessity, as the large number
</div>
<div>
of
</div>
<div>
folders, and depth of hierarchy to my best understanding caused the
</div>
<div>
exchange server / outlook / evolution mail clients to begin to
</div>
<div>
malfunction. To cope with this the archive was broken up and the
</div>
<div>
bulk stored in the offline PST and the most active components
</div>
<div>
stored in
</div>
<div>
online Exchange mailboxes.
</div>
<div>
<br>
</div>
<div>
I have some understanding of the fs mbox format, and also the
</div>
<div>
mitigations to be made for certain characters / strings. My main
</div>
<div>
concern is whether Dovecot is likely to be able to cope well with a
</div>
<div>
large number of folders / depth of hierarcy.
</div>
<div>
<br>
</div>
<div>
I will really appreciate any help / advice you can give.
</div>
<div>
<br>
</div>
<div>
Best regards,
</div>
<div>
<br>
</div>
<div>
Arnold Opio Oree
</div>
<div>
<br>
</div>
</blockquote>
<div>
<br>
</div>
<div>
Hi!
</div>
<div>
<br>
</div>
<div>
Dovecot 2.2.34/2.3 supports unlimited folder depth, the only limiting
</div>
<div>
factor is that the total name may not exceed 4096 bytes. Also
</div>
<div>
individual
</div>
<div>
folder names may not exceed 255 bytes.
</div>
<div>
<br>
</div>
<div>
Prior to that the limit is 255 per folder up to 16 levels.
</div>
<div>
<br>
</div>
<div>
I can't recommend using 'mbox' storage format, please consider
using
</div>
<div>
maildir or sdbox instead.
</div>
<div>
<br>
</div>
<div>
Aki
</div>
<div>
<br>
</div>
</blockquote>
<div>
<br>
</div>
<div>
<br>
</div>
<div>
<br>
</div>
<div>
<br>
</div>
<div>
<br>
</div>
<div>
<br>
</div>
<br>
<img style="height: 1px; width: 1px; border: 0;"
border="0" alt="" width="1" height="1"
src="http://h5lm.mjt.lu/oo/AMwAAEPRQ68AAAAAAAAAAKAFynQAASKA_AUAAAAAAAf-tABdIgvzBSx7xYU6SQW2MDXORHSy0wAHm5I/4f4e16d8/e.gif">
</blockquote>
<div>
<br>
</div>
<div class="io-ox-signature">
<pre>---
Aki Tuomi</pre>
</div>
</body>
</html>
Timo Sirainen
2019-Jul-10 08:09 UTC
Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS!
On 7 Jul 2019, at 18.12, Arnold Opio Oree via dovecot <dovecot at dovecot.org> wrote:> > Dovecot Team, > > I'd like to report a number of bugs, that are to my view all critical.It would help to get your doveconf -n, example command lines causing the problems and the error messages it outputs or what the wrong behavior looks like in filesystem. It's now rather difficult to guess what exactly you tried and what happened. Also what kind of output does readpst make? I'm not sure why you're using DIRNAMEs here.> doveadm-sync -1/general > > 1) If DIRNAMEs are not different between command line and mail_location doveadm sync will fail, saying that the source and destination directories are the sameThis sounds very strange. I'm not sure what exactly you did, and I couldn't reproduce with a small test.> 2) The -n / -N flags do not work, and a sync will fail strangely if location is specified in the namespace definitionAgain, sounds strange.> 3) Adds mbox to path name under mailbox directory (where syncing from an mbox source)Probably with different parameters you could avoid it.> 4) Not having the mailboxes at source named the same as those at destination causes errors and partial sync > > 5) Not having the target mailboxes formatted to receive the sync (/<mailboxroot>/DIRNAME/) will cause sync errors.I don't understand these. Target mailboxes are supposed to be empty initially, and after the initial sync they should be in the expected format. Why would they be different?> doveadm-sync > > 1) With large synchronizations UIDs are corrupted where multiple syncs are executed and the program can no longer synchronizeWhat exactly is the error message?> dovecot > > 1) Panics and fails to expand ~ to user home: observed cases are where multiple namespaces are being usedPanic message and more details would also help.> With regards to the last error that I requested help on i.e. \Noselect. This has been resolved more-or-less by the workarounds that I have implemented for the bugs reported above. > > I have seen a number of threads whilst researching the \Noselect issue where people have been very confused. My finding was that \Noselect is a function of the IMAP specification server-side implementation RFC3501 ( <http://h5lm.mjt.lu/lnk/AMwAAEPRQ68AAAAAAAAAAKAFynQAASKA_AUAAAAAAAf-tABdIgvzBSx7xYU6SQW2MDXORHSy0wAHm5I/1/nJ1qwIFgSyW4nfuZp-VMgA/aHR0cHM6Ly90b29scy5pZXRmLm9yZy9odG1sL3JmYzM1MDEjc2VjdGlvbi02LjMuNg#section-6.3.6>https://tools.ietf.org/html/rfc3501#section-6.3. <http://h5lm.mjt.lu/lnk/AMwAAEPRQ68AAAAAAAAAAKAFynQAASKA_AUAAAAAAAf-tABdIgvzBSx7xYU6SQW2MDXORHSy0wAHm5I/2/cturGvUAbhIkMPkkWYpZGw/aHR0cHM6Ly90b29scy5pZXRmLm9yZy9odG1sL3JmYzM1MDEjc2VjdGlvbi02LjMu#section-6.3.>6). And for me the server was returning directories with \Noselect because the mailboxes were malformed on account of dovadm-sync errors. In order to fix this I formed a bash command to transverse the mailbox hierarchy and create the missing folders critical to the sdbox format, namely DIRNAME.Nowadays we have also an option to disable creation of \Noselect folders, because they confuse people. Using mail_location = ...:NO-NOSELECT - It won't affect existing folders immediately though. -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://dovecot.org/pipermail/dovecot/attachments/20190710/bd71ec6f/attachment-0001.html>
Arnold Opio Oree
2019-Jul-10 11:01 UTC
Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS!
Hello Timo, I have scheduled some time to carry through the test requested by Aki. I will add responding to you as part of that process. Not sure if I have said or done anything to offend you, as your tone appears quite abrasive, even where you do not have an actual response to an observation. It is my view that it is a privilege to provide an open source platform, and therefore a responsibility wherever feasible not only to properly maintain and support it, but also to properly document it; both of which from my experience and findings to date, Dovecot have failed to do. Kind regards, Arnold Opio Oree Chief Executive Officer Parallax Digital Technologies arnoldoree at parallaxdt.com http://www.parallaxdt.com tel : +44 (0) 333 577 8587 fax : +44 (0) 20 8711 2477 Parallax Digital Technologies is a trading name of Parallax Global Limited. U.K. Co. No. 08836288 The contents of this e-mail are confidential. If you are not the intended recipient you are to delete this e-mail immediately, disregard its contents and disclose them to no other persons. -----Original Message----- From: Timo Sirainen via dovecot <dovecot at dovecot.org> Reply-To: Timo Sirainen <timo at sirainen.com> To: Arnold Opio Oree <arnold.oree at parallaxict.com> Cc: Dovecot Mailing List <dovecot at dovecot.org> Subject: Re: Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS! Date: Wed, 10 Jul 2019 11:09:57 +0300 On 7 Jul 2019, at 18.12, Arnold Opio Oree via dovecot < dovecot at dovecot.org> wrote:> > Dovecot Team, > > > > I'd like to report a number of bugs, that are to my view all > > critical.It would help to get your doveconf -n, example command lines causing the problems and the error messages it outputs or what the wrong behavior looks like in filesystem. It's now rather difficult to guess what exactly you tried and what happened. Also what kind of output does readpst make? I'm not sure why you're using DIRNAMEs here.> > doveadm-sync -1/general > > > > 1) If DIRNAMEs are not different between command line and > > mail_location doveadm sync will fail, saying that the source and > > destination directories are the sameThis sounds very strange. I'm not sure what exactly you did, and I couldn't reproduce with a small test.> > 2) The -n / -N flags do not work, and a sync will fail strangely if > > location is specified in the namespace definitionAgain, sounds strange.> > 3) Adds mbox to path name under mailbox directory (where syncing > from > > an mbox source)Probably with different parameters you could avoid it.> > 4) Not having the mailboxes at source named the same as those at > > destination causes errors and partial sync > > > > 5) Not having the target mailboxes formatted to receive the sync > > (/<mailboxroot>/DIRNAME/) will cause sync errors.I don't understand these. Target mailboxes are supposed to be empty initially, and after the initial sync they should be in the expected format. Why would they be different?> > doveadm-sync > > > > 1) With large synchronizations UIDs are corrupted where multiple > > syncs are executed and the program can no longer synchronizeWhat exactly is the error message?> > dovecot > > > > 1) Panics and fails to expand ~ to user home: observed cases are > > where multiple namespaces are being usedPanic message and more details would also help.> > With regards to the last error that I requested help on i.e. > > \Noselect. This has been resolved more-or-less by the workarounds > > that I have implemented for the bugs reported above. > > > > I have seen a number of threads whilst researching the \Noselect > > issue where people have been very confused. My finding was that > > \Noselect is a function of the IMAP specification server-side > > implementation RFC3501 ( > > https://tools.ietf.org/html/rfc3501#section-6.3.6). And for me the > > server was returning directories with \Noselect because the > mailboxes > > were malformed on account of dovadm-sync errors. In order to fix > this > > I formed a bash command to transverse the mailbox hierarchy and > > create the missing folders critical to the sdbox format, namely > > DIRNAME.Nowadays we have also an option to disable creation of \Noselect folders, because they confuse people. Using mail_location = ...:NO- NOSELECT - It won't affect existing folders immediately though. -----Original Message----- From: Arnold Opio Oree <arnoldoree at parallaxict.com> Reply-To: arnoldoree at parallaxict.com To: Aki Tuomi <aki.tuomi at open-xchange.com>, Dovecot Mailing List < dovecot at dovecot.org> Cc: debian-release at lists.debian.org Subject: Re: Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS! Date: Mon, 08 Jul 2019 13:21:51 +0100 Hello Aki, Thanks for looking into these. I will as requested attempt the relevant procedures under Dovecot 2.3.6. To make the test fair, I will need to fork the relevant production groupware stack (which is now stable and in operation, with our enterprise (email) data successfully migrated from Microsoft Exchange) to a new staging server; given that the current staging server is now of a materially different configuration to the production server (where the most controlled observations of these bugs were made). Kindly give me some time, as I have an urgent internal openstack deployment project to kick-off, that has been delayed by an overrun of this internal groupware stack deployment project. Best regards, Arnold Opio Oree Chief Executive Officer Parallax Digital Technologies arnoldoree at parallaxdt.com http://www.parallaxdt.com tel : +44 (0) 333 577 8587 fax : +44 (0) 20 8711 2477 Parallax Digital Technologies is a trading name of Parallax Global Limited. U.K. Co. No. 08836288 The contents of this e-mail are confidential. If you are not the intended recipient you are to delete this e-mail immediately, disregard its contents and disclose them to no other persons. -----Original Message----- From: Aki Tuomi via dovecot < dovecot at dovecot.org>Reply-To: Aki Tuomi < aki.tuomi at open-xchange.com>To: arnoldoree at parallaxict.com , Arnold Opio Oree < arnold.oree at parallaxict.com>, Arnold Opio Oree via dovecot <dovecot at dovecot.org>Cc: debian-release at lists.debian.org Subject: Re: Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS! Date: Mon, 8 Jul 2019 08:48:08 +0300 (EEST) Hi! Thank you for reporting these. We will look into them. In the mean time, can you see if any of these are fixed in 2.3.6? Aki> On 07/07/2019 18:12 Arnold Opio Oree via dovecot < > dovecot at dovecot.org > > > wrote: > > > Dovecot Team, > > I'd like to report a number of bugs, that are to my view all > critical. > > System: Replicated on multiple Debian 10 (Buster) systems > Dovecot Version(s): 2.3.4.1 > > doveadm-sync -1/general > > 1) If DIRNAMEs are not different between command line and > mail_location doveadm sync will fail, saying that the source and > destination directories are the same > > 2) The -n / -N flags do not work, and a sync will fail strangely if > location is specified in the namespace definition > > 3) Adds mbox to path name under mailbox directory (where syncing from > an mbox source) > > 4) Not having the mailboxes at source named the same as those at > destination causes errors and partial sync > > 5) Not having the target mailboxes formatted to receive the sync > (/<mailboxroot>/DIRNAME/) will cause sync errors. > > doveadm-sync > > 1) With large synchronizations UIDs are corrupted where multiple > syncs are executed and the program can no longer synchronize > > dovecot > > 1) Panics and fails to expand ~ to user home: observed cases are > where multiple namespaces are being used > > Please let me know if you need me to elaborate or to provide any > further information that you may need to replicate the bugs, or if I > can help in any other way. > > With regards to the last error that I requested help on i.e. > \Noselect. This has been resolved more-or-less by the workarounds > that I have implemented for the bugs reported above. > > I have seen a number of threads whilst researching the \Noselect > issue where people have been very confused. My finding was that > \Noselect is a function of the IMAP specification server-side > implementation RFC3501 ( > https://tools.ietf.org/html/rfc3501#section-6.3.6 > ). And for me the > server was returning directories with \Noselect because the mailboxes > were malformed on account of dovadm-sync errors. In order to fix this > I formed a bash command to transverse the mailbox hierarchy and > create the missing folders critical to the sdbox format, namely > DIRNAME. > > Kind regards, > > Arnold Opio Oree > Chief Executive Officer > Parallax Digital Technologies > > arnoldoree at parallaxdt.com > > > http://www.parallaxdt.com > > > tel : +44 (0) 333 577 8587 > fax : +44 (0) 20 8711 2477 > > Parallax Digital Technologies is a trading name of Parallax Global > Limited. U.K. Co. No. 08836288 > > The contents of this e-mail are confidential. If you are not the > intended recipient you are to delete this e-mail immediately, > disregard its contents and disclose them to no other persons. > > -----Original Message----- > From: Arnold Opio Oree via dovecot < > dovecot at dovecot.org > > > Reply-To: > arnoldoree at parallaxict.com > , Arnold Opio Oree < > arnold.oree at parallaxict.com > > > To: > dovecot at dovecot.org > > Cc: > rs at sys4.de > , > aki.tuomi at open-xchange.com > > Subject: Re: Applying Dovecot for a large / deep folder-hierarchy > archive. > Date: Thu, 04 Jul 2019 14:52:28 +0100 > > > Hi all, > > The guidance provided so far has been really helpful, and has helped > a great deal to bringing down wasted energy on finding and executing > a viable path. I am now at the final due action to complete our > Dovecot application to our use-case, but am stuck on an issue that I > cannot find any easily accessible documentation on. > > Generally this is what has been done: > > 1. Uploaded the enterprise data PST to the target groupware server. > 2. Prepared the server by changing the mailbox format to sdbox and > the the Dovecot mail location to > mail_location=/var/vmail/domain/user/mail/ > 3. Converted the pst (on-server) to a recursive mbox hierarchy using > readpst > 4. Executed doveadm-sync to convert mbox hierarchy data into sdbox > and to copy it into the enterprise archive user's mailboxes > 4.i. The biggest issue I faced at this point was doveadm-sync saying > that the source and destination pointed to the same location, whereas > they clearly did not. > 4.i.a. I resolved this by removing the location= setting from the > target namespace, and allowing it to default to mail_location > setting, and then using a completely different DIRNAME for the import > doveadm-sync execution (which was the desired final DIRNAME); I then > once the sync had been successful, changed the mail_location DIRNAME > so that it pointed to the imported mail DIRNAME; and hence the > imported email data was in the live mailboxes > 4.i.b. doveadm-import failed several times, and was throwing quite > inexplicable errors, so I moved onto doveadm-sync > 4.i.c. I also had to make sure that the source and destination folder > names matched, otherwise doveadm-syc threw very many errors and only > partially imported the data > 4.i.d. An issue which I decided just to live with is that an mbox > DIRNAME was added to each mailbox as well as the DIRNAME specified so > the path to mail is mbox/dbox-Mails. My thought is that with the data > live on an IMAP server it will be possible to do a dysync through TCP > to correct this problem. > > The final issue that I am facing now, is that when readpst finds > empty folders in the source pst hierarchy, it does not create an mbox > file in the mbox hierarchy folder space. This causes doveadm-sync to > not create the target data required for its mailbox structure i.e. > DIRNAME sub-folder and index file (with our configuration). At this > point either doveadm-sync or the dovecot process makes these empty > folders not selectable. > > The question now is how would I go about making all of these folders > selectable, e.g. with an internal or external command line tool to > change flags / create necessary sdbox mailbox constituent data? > > Many thanks, > > Arnold Opio Oree > Chief Executive Officer > Parallax Digital Technologies > > arnoldoree at parallaxdt.com > > > http://www.parallaxdt.com > > > tel : +44 (0) 333 577 8587 > fax : +44 (0) 20 8711 2477 > > Parallax Digital Technologies is a trading name of Parallax Global > Limited. U.K. Co. No. 08836288 > > The contents of this e-mail are confidential. If you are not the > intended recipient you are to delete this e-mail immediately, > disregard its contents and disclose them to no other persons. > > -----Original Message----- > From: Arnold Opio Oree via dovecot < > dovecot at dovecot.org > > > Reply-To: > arnoldoree at parallaxict.com > , Arnold Opio Oree < > arnold.oree at parallaxict.com > > > To: Robert Schetterer < > rs at sys4.de > >, > dovecot at dovecot.org > > Subject: Re: Applying Dovecot for a large / deep folder-hierarchy > archive. > Date: Thu, 27 Jun 2019 12:05:35 +0100 > > > Also you may run into client limits.... i ve seen this with > > outlook, > > apple mail, thunderbird via imap in the past > > Thanks for this note Robert, it was not really an aspect that I was > considering. > > We are operating our groupware services user access through both > Evolution Groupware and KDE Kontact / KMail on Debian Linux > workstations. Hopefully if there is a client issue it should be local > to only one groupware client. > > I will be sure to study / investigate in this - client - area should > any issues that are not traceable to the server-side arise. > > Many thanks, > > Arnold Opio Oree > Chief Executive Officer > Parallax Digital Technologies > > arnoldoree at parallaxdt.com > > > > http://www.parallaxdt.com > > > > tel : +44 (0) 333 577 8587 > fax : +44 (0) 20 8711 2477 > > Parallax Digital Technologies is a trading name of Parallax Global > Limited. U.K. Co. No. 08836288 > > The contents of this e-mail are confidential. If you are not the > intended recipient you are to delete this e-mail immediately, > disregard > its contents and disclose them to no other persons. > > > -----Original Message----- > From: Robert Schetterer via dovecot < > dovecot at dovecot.org > > > Reply-To: Robert Schetterer < > rs at sys4.de > > > To: > dovecot at dovecot.org > > > Subject: Re: Applying Dovecot for a large / deep folder-hierarchy > archive. > Date: Thu, 27 Jun 2019 12:53:49 +0200 > > Am 27.06.2019 um 07:35 schrieb Aki Tuomi via dovecot: > > On 26.6.2019 22.12, Arnold Opio Oree via dovecot wrote: > > > Hello to you all, > > > > > > I'd like to ask about my intended application of Dovecot to > > > create > > > a folder-hierarchy for storing our enterprise emails, which are > > > treated as live data rather than archives for compliance or > > > occasional / reactive retrieval. > > > > > > The data is presently not that large (a few gigabytes), but it is > > > expected to grow rapidly. Up to this stage the data has been > > > contained > > > in a Microsoft Exchange mailbox (2013), and then in an offline > > > PST. > > > The move to the offline PST was by necessity, as the large number > > > of > > > folders, and depth of hierarchy to my best understanding caused > > > the > > > exchange server / outlook / evolution mail clients to begin to > > > malfunction. To cope with this the archive was broken up and the > > > bulk stored in the offline PST and the most active components > > > stored in > > > online Exchange mailboxes. > > > > > > I have some understanding of the fs mbox format, and also the > > > mitigations to be made for certain characters / strings. My main > > > concern is whether Dovecot is likely to be able to cope well with > > > a > > > large number of folders / depth of hierarcy. > > > > > > I will really appreciate any help / advice you can give. > > > > > > Best regards, > > > > > > Arnold Opio Oree > > > > > > > Hi! > > > > Dovecot 2.2.34/2.3 supports unlimited folder depth, the only > > limiting > > factor is that the total name may not exceed 4096 bytes. Also > > individual > > folder names may not exceed 255 bytes. > > > > Prior to that the limit is 255 per folder up to 16 levels. > > > > I can't recommend using 'mbox' storage format, please consider > > using > > maildir or sdbox instead. > > > > Aki > > > > > > > > > >--- Aki Tuomi
Aki Tuomi
2019-Jul-10 11:13 UTC
Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS!
Your report is most appreciated, though it lacked quite a lot of detail that could've expedited the investigation why it happened. With the further information we can hopefully figure out why you are experiencing the problems you reported. There is no offense intended. Aki On 10.7.2019 14.01, Arnold Opio Oree via dovecot wrote:> Hello Timo, > > I have scheduled some time to carry through the test requested by Aki. > > I will add responding to you as part of that process. > > Not sure if I have said or done anything to offend you, as your tone > appears quite abrasive, even where you do not have an actual response > to an observation. > > It is my view that it is a privilege to provide an open source > platform, and therefore a responsibility wherever feasible not only to properly maintain and support it, but also to properly document it; both of which from my experience and findings to date, Dovecot have failed to do. > > Kind regards, > > Arnold Opio Oree > Chief Executive Officer > Parallax Digital Technologies > > arnoldoree at parallaxdt.com > > http://www.parallaxdt.com > > tel : +44 (0) 333 577 8587 > fax : +44 (0) 20 8711 2477 > > Parallax Digital Technologies is a trading name of Parallax Global > Limited. U.K. Co. No. 08836288 > > The contents of this e-mail are confidential. If you are not the > intended recipient you are to delete this e-mail immediately, disregard > its contents and disclose them to no other persons. > > > -----Original Message----- > From: Timo Sirainen via dovecot <dovecot at dovecot.org> > Reply-To: Timo Sirainen <timo at sirainen.com> > To: Arnold Opio Oree <arnold.oree at parallaxict.com> > Cc: Dovecot Mailing List <dovecot at dovecot.org> > Subject: Re: Applying Dovecot for a large / deep folder-hierarchy > archive - BUG REPORTS! > Date: Wed, 10 Jul 2019 11:09:57 +0300 > > On 7 Jul 2019, at 18.12, Arnold Opio Oree via dovecot < > dovecot at dovecot.org> wrote: >>> Dovecot Team, >>> >>> I'd like to report a number of bugs, that are to my view all >>> critical. > It would help to get your doveconf -n, example command lines causing > the problems and the error messages it outputs or what the wrong > behavior looks like in filesystem. It's now rather difficult to guess > what exactly you tried and what happened. > > Also what kind of output does readpst make? I'm not sure why you're > using DIRNAMEs here. > >>> doveadm-sync -1/general >>> >>> 1) If DIRNAMEs are not different between command line and >>> mail_location doveadm sync will fail, saying that the source and >>> destination directories are the same > This sounds very strange. I'm not sure what exactly you did, and I > couldn't reproduce with a small test. > >>> 2) The -n / -N flags do not work, and a sync will fail strangely if >>> location is specified in the namespace definition > Again, sounds strange. > >>> 3) Adds mbox to path name under mailbox directory (where syncing >> from >>> an mbox source) > Probably with different parameters you could avoid it. > >>> 4) Not having the mailboxes at source named the same as those at >>> destination causes errors and partial sync >>> >>> 5) Not having the target mailboxes formatted to receive the sync >>> (/<mailboxroot>/DIRNAME/) will cause sync errors. > I don't understand these. Target mailboxes are supposed to be empty > initially, and after the initial sync they should be in the expected > format. Why would they be different? > >>> doveadm-sync >>> >>> 1) With large synchronizations UIDs are corrupted where multiple >>> syncs are executed and the program can no longer synchronize > What exactly is the error message? > >>> dovecot >>> >>> 1) Panics and fails to expand ~ to user home: observed cases are >>> where multiple namespaces are being used > Panic message and more details would also help. > >>> With regards to the last error that I requested help on i.e. >>> \Noselect. This has been resolved more-or-less by the workarounds >>> that I have implemented for the bugs reported above. >>> >>> I have seen a number of threads whilst researching the \Noselect >>> issue where people have been very confused. My finding was that >>> \Noselect is a function of the IMAP specification server-side >>> implementation RFC3501 ( >>> https://tools.ietf.org/html/rfc3501#section-6.3.6). And for me the >>> server was returning directories with \Noselect because the >> mailboxes >>> were malformed on account of dovadm-sync errors. In order to fix >> this >>> I formed a bash command to transverse the mailbox hierarchy and >>> create the missing folders critical to the sdbox format, namely >>> DIRNAME. > Nowadays we have also an option to disable creation of \Noselect > folders, because they confuse people. Using mail_location = ...:NO- > NOSELECT - It won't affect existing folders immediately though. > > -----Original Message----- > From: Arnold Opio Oree <arnoldoree at parallaxict.com> > Reply-To: arnoldoree at parallaxict.com > To: Aki Tuomi <aki.tuomi at open-xchange.com>, Dovecot Mailing List < > dovecot at dovecot.org> > Cc: debian-release at lists.debian.org > Subject: Re: Applying Dovecot for a large / deep folder-hierarchy > archive - BUG REPORTS! > Date: Mon, 08 Jul 2019 13:21:51 +0100 > > Hello Aki, > > Thanks for looking into these. > > I will as requested attempt the relevant procedures under Dovecot > 2.3.6. > > To make the test fair, I will need to fork the relevant production > groupware stack (which is now stable and in operation, with our > enterprise (email) data successfully migrated from Microsoft Exchange) > to a new staging server; given that the current staging server is now > of a materially different configuration to the production server (where > the most controlled observations of these bugs were made). > > Kindly give me some time, as I have an urgent internal openstack > deployment project to kick-off, that has been delayed by an overrun of > this internal groupware stack deployment project. > > Best regards, > > Arnold Opio Oree > Chief Executive Officer > Parallax Digital Technologies > > arnoldoree at parallaxdt.com > > > http://www.parallaxdt.com > > > tel : +44 (0) 333 577 8587 > fax : +44 (0) 20 8711 2477 > > Parallax Digital Technologies is a trading name of Parallax Global > Limited. U.K. Co. No. 08836288 > > The contents of this e-mail are confidential. If you are not the > intended recipient you are to delete this e-mail immediately, disregard > its contents and disclose them to no other persons. > > > -----Original Message----- > From: Aki Tuomi via dovecot < > dovecot at dovecot.org > Reply-To: Aki Tuomi < > aki.tuomi at open-xchange.com > To: > arnoldoree at parallaxict.com > , Arnold Opio Oree < > arnold.oree at parallaxict.com >> , Arnold Opio Oree via dovecot < > dovecot at dovecot.org > Cc: > debian-release at lists.debian.org > > Subject: Re: Applying Dovecot for a large / deep folder-hierarchy > archive - BUG REPORTS! > Date: Mon, 8 Jul 2019 08:48:08 +0300 (EEST) > > Hi! > > Thank you for reporting these. We will look into them. In the mean > time, can you see if any of these are fixed in 2.3.6? > > Aki >> On 07/07/2019 18:12 Arnold Opio Oree via dovecot < >> dovecot at dovecot.org >> >>> wrote: >> >> Dovecot Team, >> >> I'd like to report a number of bugs, that are to my view all >> critical. >> >> System: Replicated on multiple Debian 10 (Buster) systems >> Dovecot Version(s): 2.3.4.1 >> >> doveadm-sync -1/general >> >> 1) If DIRNAMEs are not different between command line and >> mail_location doveadm sync will fail, saying that the source and >> destination directories are the same >> >> 2) The -n / -N flags do not work, and a sync will fail strangely if >> location is specified in the namespace definition >> >> 3) Adds mbox to path name under mailbox directory (where syncing from >> an mbox source) >> >> 4) Not having the mailboxes at source named the same as those at >> destination causes errors and partial sync >> >> 5) Not having the target mailboxes formatted to receive the sync >> (/<mailboxroot>/DIRNAME/) will cause sync errors. >> >> doveadm-sync >> >> 1) With large synchronizations UIDs are corrupted where multiple >> syncs are executed and the program can no longer synchronize >> >> dovecot >> >> 1) Panics and fails to expand ~ to user home: observed cases are >> where multiple namespaces are being used >> >> Please let me know if you need me to elaborate or to provide any >> further information that you may need to replicate the bugs, or if I >> can help in any other way. >> >> With regards to the last error that I requested help on i.e. >> \Noselect. This has been resolved more-or-less by the workarounds >> that I have implemented for the bugs reported above. >> >> I have seen a number of threads whilst researching the \Noselect >> issue where people have been very confused. My finding was that >> \Noselect is a function of the IMAP specification server-side >> implementation RFC3501 ( >> https://tools.ietf.org/html/rfc3501#section-6.3.6 >> ). And for me the >> server was returning directories with \Noselect because the mailboxes >> were malformed on account of dovadm-sync errors. In order to fix this >> I formed a bash command to transverse the mailbox hierarchy and >> create the missing folders critical to the sdbox format, namely >> DIRNAME. >> >> Kind regards, >> >> Arnold Opio Oree >> Chief Executive Officer >> Parallax Digital Technologies >> >> arnoldoree at parallaxdt.com >> >> >> http://www.parallaxdt.com >> >> >> tel : +44 (0) 333 577 8587 >> fax : +44 (0) 20 8711 2477 >> >> Parallax Digital Technologies is a trading name of Parallax Global >> Limited. U.K. Co. No. 08836288 >> >> The contents of this e-mail are confidential. If you are not the >> intended recipient you are to delete this e-mail immediately, >> disregard its contents and disclose them to no other persons. >> >> -----Original Message----- >> From: Arnold Opio Oree via dovecot < >> dovecot at dovecot.org >> Reply-To: >> arnoldoree at parallaxict.com >> , Arnold Opio Oree < >> arnold.oree at parallaxict.com >> To: >> dovecot at dovecot.org >> >> Cc: >> rs at sys4.de >> , >> aki.tuomi at open-xchange.com >> >> Subject: Re: Applying Dovecot for a large / deep folder-hierarchy >> archive. >> Date: Thu, 04 Jul 2019 14:52:28 +0100 >> >> >> Hi all, >> >> The guidance provided so far has been really helpful, and has helped >> a great deal to bringing down wasted energy on finding and executing >> a viable path. I am now at the final due action to complete our >> Dovecot application to our use-case, but am stuck on an issue that I >> cannot find any easily accessible documentation on. >> >> Generally this is what has been done: >> >> 1. Uploaded the enterprise data PST to the target groupware server. >> 2. Prepared the server by changing the mailbox format to sdbox and >> the the Dovecot mail location to >> mail_location=/var/vmail/domain/user/mail/ >> 3. Converted the pst (on-server) to a recursive mbox hierarchy using >> readpst >> 4. Executed doveadm-sync to convert mbox hierarchy data into sdbox >> and to copy it into the enterprise archive user's mailboxes >> 4.i. The biggest issue I faced at this point was doveadm-sync saying >> that the source and destination pointed to the same location, whereas >> they clearly did not. >> 4.i.a. I resolved this by removing the location= setting from the >> target namespace, and allowing it to default to mail_location >> setting, and then using a completely different DIRNAME for the import >> doveadm-sync execution (which was the desired final DIRNAME); I then >> once the sync had been successful, changed the mail_location DIRNAME >> so that it pointed to the imported mail DIRNAME; and hence the >> imported email data was in the live mailboxes >> 4.i.b. doveadm-import failed several times, and was throwing quite >> inexplicable errors, so I moved onto doveadm-sync >> 4.i.c. I also had to make sure that the source and destination folder >> names matched, otherwise doveadm-syc threw very many errors and only >> partially imported the data >> 4.i.d. An issue which I decided just to live with is that an mbox >> DIRNAME was added to each mailbox as well as the DIRNAME specified so >> the path to mail is mbox/dbox-Mails. My thought is that with the data >> live on an IMAP server it will be possible to do a dysync through TCP >> to correct this problem. >> >> The final issue that I am facing now, is that when readpst finds >> empty folders in the source pst hierarchy, it does not create an mbox >> file in the mbox hierarchy folder space. This causes doveadm-sync to >> not create the target data required for its mailbox structure i.e. >> DIRNAME sub-folder and index file (with our configuration). At this >> point either doveadm-sync or the dovecot process makes these empty >> folders not selectable. >> >> The question now is how would I go about making all of these folders >> selectable, e.g. with an internal or external command line tool to >> change flags / create necessary sdbox mailbox constituent data? >> >> Many thanks, >> >> Arnold Opio Oree >> Chief Executive Officer >> Parallax Digital Technologies >> >> arnoldoree at parallaxdt.com >> >> >> http://www.parallaxdt.com >> >> >> tel : +44 (0) 333 577 8587 >> fax : +44 (0) 20 8711 2477 >> >> Parallax Digital Technologies is a trading name of Parallax Global >> Limited. U.K. Co. No. 08836288 >> >> The contents of this e-mail are confidential. If you are not the >> intended recipient you are to delete this e-mail immediately, >> disregard its contents and disclose them to no other persons. >> >> -----Original Message----- >> From: Arnold Opio Oree via dovecot < >> dovecot at dovecot.org >> Reply-To: >> arnoldoree at parallaxict.com >> , Arnold Opio Oree < >> arnold.oree at parallaxict.com >> To: Robert Schetterer < >> rs at sys4.de >>> , >> dovecot at dovecot.org >> >> Subject: Re: Applying Dovecot for a large / deep folder-hierarchy >> archive. >> Date: Thu, 27 Jun 2019 12:05:35 +0100 >> >>> Also you may run into client limits.... i ve seen this with >>> outlook, >>> apple mail, thunderbird via imap in the past >> Thanks for this note Robert, it was not really an aspect that I was >> considering. >> >> We are operating our groupware services user access through both >> Evolution Groupware and KDE Kontact / KMail on Debian Linux >> workstations. Hopefully if there is a client issue it should be local >> to only one groupware client. >> >> I will be sure to study / investigate in this - client - area should >> any issues that are not traceable to the server-side arise. >> >> Many thanks, >> >> Arnold Opio Oree >> Chief Executive Officer >> Parallax Digital Technologies >> >> arnoldoree at parallaxdt.com >> >> >> >> http://www.parallaxdt.com >> >> >> >> tel : +44 (0) 333 577 8587 >> fax : +44 (0) 20 8711 2477 >> >> Parallax Digital Technologies is a trading name of Parallax Global >> Limited. U.K. Co. No. 08836288 >> >> The contents of this e-mail are confidential. If you are not the >> intended recipient you are to delete this e-mail immediately, >> disregard >> its contents and disclose them to no other persons. >> >> >> -----Original Message----- >> From: Robert Schetterer via dovecot < >> dovecot at dovecot.org >> >> >> Reply-To: Robert Schetterer < >> rs at sys4.de >> >> >> To: >> dovecot at dovecot.org >> >> >> Subject: Re: Applying Dovecot for a large / deep folder-hierarchy >> archive. >> Date: Thu, 27 Jun 2019 12:53:49 +0200 >> >> Am 27.06.2019 um 07:35 schrieb Aki Tuomi via dovecot: >>> On 26.6.2019 22.12, Arnold Opio Oree via dovecot wrote: >>>> Hello to you all, >>>> >>>> I'd like to ask about my intended application of Dovecot to >>>> create >>>> a folder-hierarchy for storing our enterprise emails, which are >>>> treated as live data rather than archives for compliance or >>>> occasional / reactive retrieval. >>>> >>>> The data is presently not that large (a few gigabytes), but it is >>>> expected to grow rapidly. Up to this stage the data has been >>>> contained >>>> in a Microsoft Exchange mailbox (2013), and then in an offline >>>> PST. >>>> The move to the offline PST was by necessity, as the large number >>>> of >>>> folders, and depth of hierarchy to my best understanding caused >>>> the >>>> exchange server / outlook / evolution mail clients to begin to >>>> malfunction. To cope with this the archive was broken up and the >>>> bulk stored in the offline PST and the most active components >>>> stored in >>>> online Exchange mailboxes. >>>> >>>> I have some understanding of the fs mbox format, and also the >>>> mitigations to be made for certain characters / strings. My main >>>> concern is whether Dovecot is likely to be able to cope well with >>>> a >>>> large number of folders / depth of hierarcy. >>>> >>>> I will really appreciate any help / advice you can give. >>>> >>>> Best regards, >>>> >>>> Arnold Opio Oree >>>> >>> Hi! >>> >>> Dovecot 2.2.34/2.3 supports unlimited folder depth, the only >>> limiting >>> factor is that the total name may not exceed 4096 bytes. Also >>> individual >>> folder names may not exceed 255 bytes. >>> >>> Prior to that the limit is 255 per folder up to 16 levels. >>> >>> I can't recommend using 'mbox' storage format, please consider >>> using >>> maildir or sdbox instead. >>> >>> Aki >>> >> >> >> >> >> >> >> > --- > Aki Tuomi >
Adam D. Barratt
2019-Jul-10 11:24 UTC
Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS!
On 2019-07-10 12:01, Arnold Opio Oree wrote: Please stop CCing this thread to the Debian Release Management mailing list. If there are issues with the Dovecot packages in Debian then they should be reported as bugs against the packages. Thanks. Adam (Debian Release Team member)
Arnold Opio Oree
2019-Jul-10 11:26 UTC
Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS!
Understood Adam, My thinking is that this is a package released with Debian 10. And so has everything to do with the release. I am new to the community, and have not yet studied the guidelines. Regards, Arnold Opio Oree Chief Executive Officer Parallax Digital Technologies arnoldoree at parallaxdt.com http://www.parallaxdt.com tel : +44 (0) 333 577 8587 fax : +44 (0) 20 8711 2477 Parallax Digital Technologies is a trading name of Parallax Global Limited. U.K. Co. No. 08836288 The contents of this e-mail are confidential. If you are not the intended recipient you are to delete this e-mail immediately, disregard its contents and disclose them to no other persons. -----Original Message----- From: Adam D. Barratt <adam at adam-barratt.org.uk> To: arnoldoree at parallaxict.com Cc: timo at sirainen.com, Dovecot Mailing List <dovecot at dovecot.org>, debian-release at lists.debian.org Subject: Re: Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS! Date: Wed, 10 Jul 2019 12:24:44 +0100 On 2019-07-10 12:01, Arnold Opio Oree wrote: Please stop CCing this thread to the Debian Release Management mailing list. If there are issues with the Dovecot packages in Debian then they should be reported as bugs against the packages. Thanks. Adam(Debian Release Team member) -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://dovecot.org/pipermail/dovecot/attachments/20190710/2201f7af/attachment.html>
Adam D. Barratt
2019-Jul-12 12:43 UTC
Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS!
On Wed, 2019-07-10 at 12:26 +0100, Arnold Opio Oree wrote:> Understood Adam, > > My thinking is that this is a package released with Debian 10. And so > has everything to do with the release.For clarity here - debian-release is neither a support forum nor a discussion list, nor a means of reporting issues in software contained within a Debian release. Debian has support fora and a bug tracking system to handle such reports and issues, as per https://www.debian.org /support . Having a single list that discusses any issue in any piece of software shipped by Debian cannot possibly scale. Rather, this list is the contact point and team alias for the Release Team, who oversee and manage the release process. It is up to individual package maintainers to triage and deal with issues reported against their packages and then to liaise with us if required. Regards, Adam
Arnold Opio Oree
2019-Jul-13 10:15 UTC
Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS!
Sure Adam, this makes perfect sense. And having properly reviewed the "debian-release" mailing list description, I can see that this is made quite clear:> Coordinating Debian releases > Coordination of Debian releases issues such as testing migrations, > transitions and removals. This list should not be considered a > discussion list; discussions related to releases issues should be > held on more appropriate lists such as debian-devel, debian-legal or > debian-project.Best regards, Arnold Opio Oree Chief Executive Officer Parallax Digital Technologies arnoldoree at parallaxdt.com http://www.parallaxdt.com tel : +44 (0) 333 577 8587 fax : +44 (0) 20 8711 2477 Parallax Digital Technologies is a trading name of Parallax Global Limited. U.K. Co. No. 08836288 The contents of this e-mail are confidential. If you are not the intended recipient you are to delete this e-mail immediately, disregard its contents and disclose them to no other persons. -----Original Message----- From: Adam D. Barratt via dovecot <dovecot at dovecot.org> Reply-To: Adam D. Barratt <adam at adam-barratt.org.uk> To: arnoldoree at parallaxict.com Cc: timo at sirainen.com, debian-release at lists.debian.org, Dovecot Mailing List <dovecot at dovecot.org> Subject: Re: Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS! Date: Fri, 12 Jul 2019 13:43:34 +0100 On Wed, 2019-07-10 at 12:26 +0100, Arnold Opio Oree wrote:> Understood Adam, > > My thinking is that this is a package released with Debian 10. And so > has everything to do with the release.For clarity here - debian-release is neither a support forum nor a discussion list, nor a means of reporting issues in software contained within a Debian release. Debian has support fora and a bug tracking system to handle such reports and issues, as per https://www.debian.org /support . Having a single list that discusses any issue in any piece of software shipped by Debian cannot possibly scale. Rather, this list is the contact point and team alias for the Release Team, who oversee and manage the release process. It is up to individual package maintainers to triage and deal with issues reported against their packages and then to liaise with us if required. Regards, Adam --
Possibly Parallel Threads
- Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS!
- Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS!
- Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS!
- Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS!
- Applying Dovecot for a large / deep folder-hierarchy archive.