dovecot - Feb 2012 - Multiple locations, 2 servers - planning questions...

Hello all/Timo,

Up until now, my main Clients office has consisted of a single location, 
and I have never had to deal with the situation of multiple locations 
for a single company.

They have just told me that they are acquiring an additional floor at a 
building that is about 4 minutes away - but obviously far enough away 
that I now have to deal with supporting users in the same domain but at 
two disparate physical locations.

These two locations will be connected via a private Gb ethernet 
connection, and each location will have its own internet connection (I 
think - still waiting on some numbers to present to the owner to see 
what he wants to do in that regard, but that will be my recommendation), 
so bandwidth for replication won't be an issue.

I have a couple of months to get this done, and I am already planning on 
hiring Timo's new commercial support company to help with the final and 
actual design and implementation, but obviously first I need to know 
what my actual options are.

Just a rough idea of what I'd like to do is:

1. Set up one dovecot server at each location (these will be VMs), so
    users at each location are accessing the local server for that office

2. Full replication between the two for the mail/indexes, and configure
    them such that each can act as a failover for the other in case one
    goes down for whatever reason

This is my first/main question...

I recall that 'dsync based replication' is actually on the map for 2.1, 
but, since apparently dsync can't do this now, Timo, do you have even a 
rough idea how much work this would be to get it working for only 2 
locations (assuming it *may* be easier to get the initial support for 
only 2 locations, my client may be willing to pay for it if it isn't a 
huge amount - feel free to reply privately to this question), then you 
could revisit it later to make it more scalable? Or, if it is going to 
take more work than my client is willing to pay for (I'm hoping not, 
since you said it was on the map for 2.1, not 2.2+), maybe the notify 
plugin could be leveraged in some way to provide something 'close 
enough' until it is fully implemented in dsync?

On that note (something 'close enough' until dsync fully supports this 
natively), would setting up a dsync cron job, say, every 5 or 10 
minutes, be asking for trouble? Our mail server is not all that busy, 
really, so in 5 or 10 minutes, there wouldn't be many changes at all.

If that is not recommended, although I want to avoid the hassles of NFS 
if at all possible, maybe there is another shared filesystem that will 
work ok - or... since I will be forcing users to a single server always 
anyway, maybe NFS or some other shared filesystem is really the best 
option here, and just let it take care of the syncing?

and

3. Configure things such that each offices users are directed to the 
local server for that office, but connections will fail-over to the 
remote server in the case of one of them going down for whatever reason?

I'm fairly sure that some combination of Dovecot Proxy/Director will 
accomplish this, but one concern is - for internal users, my 
understanding is it will redirect them via the private IP, but that 
would result in lots of traffic across the Gb connection between  the 
two locations, and I'd like to eliminate that if possible - so how will 
this work when they are accessing it from outside the office, where each 
office has its own public IP? I'd prefer to not rely on users using the 
correct hostname (currently, we just use 'mail.example.com', and I know 
I could set up two new ones - office1.example.com and 
office2.example.com - but then I'd be relying on the users to get it 
right, and I'd prefer to avoid that can of worms). I guess a worst case 
scenario (if there is no better way) would be to do it that way, then 
watch the logs for users who get it wrong and are using the inter-office 
connection, and deal with them on a case by case basis.

Thanks to any/all for reading this far and for any thoughts, suggestions 
and/or ideas...

-- 

Best regards,

Charles

On 27.02.2012 17:54, Charles Marcus wrote:
> These two locations will be connected via a private Gb ethernet 
> connection, and each location will have its own internet connection (I 
> think - still waiting on some numbers to present to the owner to see 
> what he wants to do in that regard, but that will be my 
> recommendation), so bandwidth for replication won't be an issue.
[cut]

I do have a basic question... How many users will be in this new, remote 
location? Will the traffic be so vast, that 1GbE link will not be 
enough, or are you using two servers for reliability?

The simpler the configuration, it is almost always the better. Maybe you 
can stay with one server in yours primary location?

-- 
Adam Szpakowski

On 2012-02-27 1:34 PM, Sven Hartge <sven at svenhartge.de>
wrote:
> Charles Marcus<CMarcus at media-brokers.com>  wrote:
>> Each location is an entire floor of a 6 story building. The remote
>> location has the capacity for about 60 users, the new location about
>> 100. We only allow IMAP access to email, so if everyone is using email
>> at the same time, that would be a lot of traffic over a single Gb link
>> I think...
> Naa, most clients download mails only once and then keep them cached
> locally (at least Thunderbird and Outlook do).
>
> Looking at the used bandwidth of the mailserver of my small university
> (10.000 users, about 1000 concurrently active during the daytime)
> shows a steady amount of roughly 5MBit/s with peaks to 10MBit/s in and
> out.
Interesting - thanks for the numbers...

But, again, my main reason for 2 servers is not performance, it is for 
redundancy...

-- 

Best regards,

Charles

On 27.2.2012, at 18.54, Charles Marcus wrote:
> I recall that 'dsync based replication' is actually on the map for
2.1, but, since apparently dsync can't do this now, Timo, do you have even a
rough idea how much work this would be to get it working for only 2 locations
(assuming it *may* be easier to get the initial support for only 2 locations, my
client may be willing to pay for it if it isn't a huge amount - feel free to
reply privately to this question), then you could revisit it later to make it
more scalable?
I'll initially build it for only 2 locations, but I think it will be pretty
simple to scale to more than 2.
> If that is not recommended, although I want to avoid the hassles of NFS if
at all possible, maybe there is another shared filesystem that will work ok -
or... since I will be forcing users to a single server always anyway, maybe NFS
or some other shared filesystem is really the best option here, and just let it
take care of the syncing?
Synchronous drbd replication for a master/slave setup should also work, since
the latency between your servers is probably quite low. I wouldn't use
asynchronous replication since it can lose some of the last changes when failure
happens.

Then there are of course all the cluster filesystems, but I don't have much
experience with them other than what I've read in this list. I think GPFS is
the only one I haven't read any complains of (but it could be also that so
few people have actually used it).
> 3. Configure things such that each offices users are directed to the local
server for that office, but connections will fail-over to the remote server in
the case of one of them going down for whatever reason?
With a clusterfs setup you could do this. With dsync-replicated setup you could
assign a primary location for the user, and proxy the connection there if user
got connected to wrong server, except when the primary server is down.
> I'm fairly sure that some combination of Dovecot Proxy/Director will
accomplish this, but one concern is - for internal users, my understanding is it
will redirect them via the private IP, but that would result in lots of traffic
across the Gb connection between  the two locations, and I'd like to
eliminate that if possible - so how will this work when they are accessing it
from outside the office, where each office has its own public IP? I'd prefer
to not rely on users using the correct hostname (currently, we just use
'mail.example.com', and I know I could set up two new ones -
office1.example.com and office2.example.com - but then I'd be relying on the
users to get it right, and I'd prefer to avoid that can of worms). I guess a
worst case scenario (if there is no better way) would be to do it that way, then
watch the logs for users who get it wrong and are using the inter-office
connection, and deal with them on a case by case basis.
Like other mentioned, I don't think the cross-office traffic will be that
much of a problem, especially for external connections from outside office. For
internal connections you should be able to mostly avoid it.

On 2/27/2012 10:54 AM, Charles Marcus wrote:
> These two locations will be connected via a private Gb ethernet
> connection, and each location will have its own internet connection (I
> think - still waiting on some numbers to present to the owner to see
> what he wants to do in that regard, but that will be my recommendation),
> so bandwidth for replication won't be an issue.
Say you're a boutique mail services provider or some such.  In your own
datacenter you have a Dovecot server w/64 processors, 512GB RAM, and 4
dual port 8Gb fiber channel cards.  It's connected via 8 redundant fiber
channel links to 4 SAN array units, each housing 120 x15k SAS drives,
480 drives total, ~140,000 random IOPs.  This gear eats 36U of a 40U
rack, and about $400,000 USD out of your wallet.  In the remaining 4U at
the top of the rack you have a router, with two GbE links connected to
the server, and an OC-12 SONET fiber link (~$15k-20k USD/month) to a
national ISP backbone.  Not many years ago OC-12s comprised the backbone
links of the net.  OC-48s handle that today.  Today OC-12s are most
often used to link midsized ISPs to national ISPs, act as the internal
backbone of midsized ISPs, and link large ISPs' remote facilities to the
backbone.

Q:  How many concurrent IMAP clients could you serve with this setup
    before hitting a bottleneck at any point in the architecture?  What
    is the first bottleneck you'd run into?

The correct answer to this question, and the subsequent discussion that
will surely take place, may open your eyes a bit, and prompt you to
rethink some of your assumptions that went into the architectural
decisions you've presented here.

-- 
Stan