dovecot - Aug 2009 - Quick and dirty server optimized for IMAP upload speed?

Early next week I need to upload over 100,000 emails to an IMAP server 
as quickly as possible from an Outlook client. I am looking for any 
methods I can use to (temporarily?) speed up the rate at which dovecot 
can accept and store IMAP uploads, whether it be storing on local disk, 
ram disk, etc.  I can setup a temporary server on a laptop for example 
and once the upload has finished I can use standard file copying methods 
to transfer the mail to stable, permanent storage.  I haven't been able 
to see over about 7 msgs/sec upload speed from a local folder in any 
mail client to dovecot (only NFS or ZFS backend tested so far with 
Maildir).  Is there something horribly wrong with the speed I am seeing 
or are there just tricks I can try?  Any tips?  I'll be working on it 
all weekend until I find something satisfactory.  It seems like I can 
upload mails to an Exchange server quicker.  I'll setup just about 
anything that my experience allows me to, I can be very resourceful with 
adhoc hardware and software.

Alternatively I'll take a fast way of converting Exchange email to a 
tree of local mbox files which I can then run mb2md on.  I tried using 
Thunderbird to Import the mails from Outlook and while it was fast, it 
messed up the formatting of some of the mails so I don't think I can use 
that.  I tried readpst briefly from libpst but it took a long time to 
run, took alot of cpu, and was spewing lots of errors so I canceled it.

Thanks for any input!

On Aug 28, 2009, at 8:38 PM, Adam McDougall wrote:
> Early next week I need to upload over 100,000 emails to an IMAP  
> server as quickly as possible from an Outlook client. I am looking  
> for any methods I can use to (temporarily?) speed up the rate at  
> which dovecot can accept and store IMAP uploads, whether it be  
> storing on local disk, ram disk, etc.  I can setup a temporary  
> server on a laptop for example and once the upload has finished I  
> can use standard file copying methods to transfer the mail to  
> stable, permanent storage.  I haven't been able to see over about 7  
> msgs/sec upload speed from a local folder in any mail client to  
> dovecot (only NFS or ZFS backend tested so far with Maildir).  Is  
> there something horribly wrong with the speed I am seeing or are  
> there just tricks I can try?  Any tips?  I'll be working on it all  
> weekend until I find something satisfactory.  It seems like I can  
> upload mails to an Exchange server quicker.  I'll setup just about  
> anything that my experience allows me to, I can be very resourceful  
> with adhoc hardware and software.
 From Dovecot's side the only thing you can do is fsync_disable=yes.  
The main problem is probably network latency, because Outlook doesn't  
support MULTIAPPEND extension (and perhaps not even LITERAL+  
extension?) Did you already try running Dovecot on the same computer  
as Outlook (some virtual thingy or maybe it works in cygwin)?
> Alternatively I'll take a fast way of converting Exchange email to a  
> tree of local mbox files which I can then run mb2md on.
If the mails are in Exchange, can't you connect to it using IMAP?

On Sun, Aug 30, 2009 at 12:33:43PM -0400, Charles Marcus wrote:

  On 8/30/2009, Adam McDougall (mcdouga9 at egr.msu.edu) wrote:
  > The two biggest real bottlenecks:
  > - Thunderbird is just slow at uploading to IMAP.  With a bunch of small
msgs it only does a few per second and you can tell the server is waiting for
something to do.
  
  Did you try TBird 3.0b3? It has many, many IMAP improvements...
  
  -- 
  
  Best regards,
  
  Charles
  
Wow it sure is faster at uploading, thanks for mentioning it!