dovecot - May 2009 - How to run Dovecot as IMAP Proxy?

Hi All,

I have migrated from uw-imap to Dovecot for POP3 & IMAP service. I run
webmail using squirrelmail. When running uw-imap I used to run up.imapproxyd on
the webmail server for faster responses. After migrating to Dovecot, I find that
up.imapproxyd does not work well with dovecot.

I tried to use dovecot as IMAP PRoxy server as per the document 

http://wiki.dovecot.org/PasswordDatabase/ExtraFields/Proxy

I don't understand Databases and don't want to install mysql etc for a
simple thing like IMAP PRoxy.

My question is can Dovecot be run as a IMAP PRoxy without the hassle of setting
up mysql etc? I use /etc/passwd for auth on the actual dovecot server running
IMAP & POP3.

Thanks
V S Rao

On Thu, 2009-05-21 at 05:21 -0700, V S Rao wrote:
> Hi All,
> 
> I have migrated from uw-imap to Dovecot for POP3 & IMAP service. I run
> webmail using squirrelmail. When running uw-imap I used to run
> up.imapproxyd on the webmail server for faster responses. After
> migrating to Dovecot, I find that up.imapproxyd does not work well
> with dovecot. 
Why not?
> I don't understand Databases and don't want to install mysql etc
for a
> simple thing like IMAP PRoxy.
Do you just want it to proxy everyone to a single destination or to
different servers based on username? If you want everyone to the same
destination, why do you even want to use a proxy?
> My question is can Dovecot be run as a IMAP PRoxy without the hassle
> of setting up mysql etc? I use /etc/passwd for auth on the actual
> dovecot server running IMAP & POP3.
Using checkpassword or SQLite would probably be the easiest solution. Or
generation of a passwd-file, but you'd have to keep it updated if users
change.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20090521/b18228fe/attachment-0002.bin>

Hi Timo,

Thanks for the response. Apologize, but my responses are going to be a bit
lengthy.
> I have migrated from uw-imap to Dovecot for POP3 & IMAP service. I run
> webmail using squirrelmail. When running uw-imap I used to run
> up.imapproxyd on the webmail server for faster responses. After
> migrating to Dovecot, I find that up.imapproxyd does not work well
> with dovecot. 
> Why not?
Here are my observations. I have around 6000+ mailboxes and roughly the same
number of users. Earlier the mail server (running sendmail + uw-imap for POP3
& IMAP) would have around 80 pop3 connections (peak) and around 300 IMAP
connections, concurrent.
(ps -aef | grep imap | wc -l or ps -aef | grep pop3 | wc -l). There have been
cases where I have observed upto 500 concurrent IMAP sessions.

The IMAP connections are from a webmail server running Apache with Squirrelmail.
The observation was that response was slow & so based on the suggestion on
Squirrelmail for performance improvement we have installed up-imapproxyd on the
squirrelmail machine. There was a significant improvement in the response times
for the users, because of caching.

After that I encountered some strange problems of POP3 timing out for users
(earlier I did post that problem in this forum). I opened a ticket with Redhat
and naturally they refused to support me with uw-imap running. So switched to
Dovecot 0.99.x (I run the server on RHEL 4.0 and that is the max version
supported by Redhat for that version). Ever since I did that POP3 works fine but
now webmail is almost not available to the users. People usually get
"connection dropped by IMAP server". However the IMAP server seems to
work fine. Checked through manual "telnet <ip.address.of.mailserver>
143 & also through other client such as outlook & Thunderbird.

I noticed that while the number of established connections for the webmail to
mail server was shown as around 40 to 50 on the mail server, the same is shown
as around 700 on webmail server.
{ netstat -an | grep :143 | grep ESTABLISHED | wc -l }. 

imapproxyd is configured to listen on port 9143 on the localhost and
communicates to the mail server on std port 143. What was really bizzare is that
when I check for the number of connections on port 9143 it shows around 8000!

netstat -an | grep :9143 | grep ESTABLISHED | wc -l

So I disabled the imapproxyd on the webmail & now at least the 50
connections are consistent. However the response is extremely slow and so was
contemplating on the idea of trying out using another IMAP Proxy server and
hence the question.
> I don't understand Databases and don't want to install mysql etc
for a
> simple thing like IMAP PRoxy.
> Do you just want it to proxy everyone to a single destination or to
> different servers based on username? If you want everyone to the same
> destination, why do you even want to use a proxy?
I want to proxy all user to the same server. The idea being as explained above
benefit from caching.
> My question is can Dovecot be run as a IMAP PRoxy without the hassle
> of setting up mysql etc? I use /etc/passwd for auth on the actual
> dovecot server running IMAP & POP3.
> Using checkpassword or SQLite would probably be the easiest solution. Or
> generation of a passwd-file, but you'd have to keep it updated if users
> change.
If I used a passwd file on the webmail machine, running dovecot as a Proxy how
would I configure dovecot on the webmail server? It has to accept connections
from the squirrelmail and talk to the mailserver. I can do a rsync of the passwd
file from the mail server to the webmail server using a small script whenever
the file changes on the mail server.

Of course the fundamental question from your response, I guess is, without a
proxy why is the webmail slow?
If proxy is not my problem, can you pls throw some light on why I am facing
performance issues on my webmail alone?

Thanks and Regards
V S Rao

Thanks for the many responses and views.

I have taken RH support for my mail server only and so have to ensure they
support me. So am going with their recommended version which is 0.99.x.

Now on the webmail I have migrated to RHEL 5.3 and the dovecot being used there
is 1.0.7.x. So that should support proxying, right?

BTW, originally I did not go to Redhat. I moved from uw-imap to Dovecot 1.1.14
on the mail server for POP3. The problem of POP3 timeouts continued and I could
not find any reason why POP3 was timing out when 400+ concurrent IMAP sessions
were working fine. Moreover other network services such as SMTP, Telnet, SSH etc
were working fine. Had the box installed behind a IPS box to see if there were
any DoS attacks, but had to rule that out. As a last resort, paid RH for support
and they made me downgrade Dovecot from 1.1.14 to 0.99.x. Surprisingly the
moment I downgraded dovecot, the problem shifted to IMAP & remained ever
since. So naturally they told me uw-imap was the problem and made me move to
Dovecot 0.99.x for IMAP as well, but no improvement.

But the surprising thing, as I mentioned in my original post, IMAP works fine on
command line, with clients such as Outlook, THunderbird etc., but Webmail is
very slow that its almost not available.

BTW webmail was also running on RHEL 4.0. Seeing that the problem existed in
webmail alone I upgraded webmail to RHEL 5.3 with all the latest RH supported
packages on a new hardware.

Any ideas to help me debug this issue would be of great help as I am stuck on
this issue for almost a month now.

Oh, btw regarding stats, I don't have a measure, but when webmail was
working the login was well within 5 seconds. Now it takes forever (more than a
min) or timeouts most of the time. Even after login, any request such as opening
a mail or changing to a mail folder was well within 5 seconds. Now again it
takes forever or timeouts most of the time.

Thanks and Regards
V S Rao
----- Original Message -----
From: "Timo Sirainen" <tss at iki.fi>
To: "Dovecot Mailing List" <dovecot at dovecot.org>
Sent: Friday, May 22, 2009 12:32:24 AM GMT +05:30 Chennai, Kolkata, Mumbai, New
Delhi
Subject: Re: [Dovecot] How to run Dovecot as IMAP Proxy?

On Thu, 2009-05-21 at 11:47 -0700, Seth Mattinen wrote:
> Yes, that too. My point was not to judge Dovecot's performance based on
> a 0.99 version. ;) I've successfully used several versions of dovecot
> 1.1.x with imapproxyd and IMAP-only webmail clients.
Do you have any numbers about how much faster it works with imapproxyd
vs. without?

So I guess I will take the problem to imapproxy group and find out if I can get
a clue to the issue I am facing.

BTW, just to confirm once again, are there any known issues with Dovecot &
imapproxyd? Because I find it strange that there should be around 8000
established connections between Squirrelmail and imapproxyd & around 400
connections between imapproxyd and Dovecot while on the mail server I see only a
couple of connections (typically around 40). Or is it some configuration in
Dovecot which is causing this kind of behavior, as I did not notice this while
running uw-imap.

Regards
V S Rao

On 5/22/2009, V S Rao (viriyala at yahoo.com) wrote:
> BTW, just to confirm once again, are there any known issues with Dovecot
& imapproxyd?
You will not get any support for 0.99 here, unless Timo can provide it.
It is *that* old, and your decision to use it - regardless of your
claimed reason - is baffling.

The only experience I had with it - what, 5 years ago? - was its broken
mbox problem... we had to move to courier because of it.

-- 

Best regards,

Charles

* Charles Marcus <dovecot at dovecot.org>:
> On 5/22/2009, V S Rao (viriyala at yahoo.com) wrote:
> > BTW, just to confirm once again, are there any known issues with
Dovecot & imapproxyd?
> 
> You will not get any support for 0.99 here, unless Timo can provide it.
> It is *that* old, and your decision to use it - regardless of your
> claimed reason - is baffling.
We use recent Dovecot from atrpms.net with RHEL 5.3. Userbase is 25.000.
Dovecot runs on three machines - as master (proxy) and with two slaves.

The system is stable. We hardly need to touch it.

p at rick


-- 
state of mind
Digitale Kommunikation

http://www.state-of-mind.de

Franziskanerstra?e 15	   Telefon +49 89 3090 4664
81669 M?nchen              Telefax +49 89 3090 4666

Amtsgericht M?nchen        Partnerschaftsregister PR 563

> 
> You will not get any support for 0.99 here, unless Timo can provide it.
> It is *that* old, and your decision to use it - regardless of your
> claimed reason - is baffling.
I admit I am running a pretty old version and frankly speaking I was not hoping
for any support. Let RH earn their money we paid as support. As you would have
noticed I was looking for help in running Dovecot as a IMAP Proxy without the
need to setup SQL etc. I only went into details to give clarity.

I really appreciate all the responses and suggestions.
>> We use recent Dovecot from atrpms.net with RHEL 5.3. Userbase is
25.000.
>> Dovecot runs on three machines - as master (proxy) and with two slaves.
>> The system is stable. We hardly need to touch it.
I would be very much interested in knowing the details of your setup as I am
suffering here with only 6000+ users. And that too with a config, which I
frankly think is an overkill for our requirement. However maybe I should start a
separate thread for that?

Thanks and Regards
V S Rao.