dovecot - Jan 2012 - Providing shared folders with multiple backend servers

Hi *,

I am currently in the planning stage for a "new and improved" mail
system at my university.

Right now, everything is on one big backend server but this is causing
me increasing amounts of pain, beginning with the time a full backup
takes.

So naturally, I want to split this big server into smaller ones.

To keep things simple, I want to pin a user to a server so I can avoid
things like NFS or cluster aware filesystems. The mapping for each
account is then inserted into the LDAP object for each user and the
frontend proxy (perdition at the moment) then uses this information to
route each access to the correct backend storage server running dovecot.

So far this has been working nice with my test setup.

But: I also have to provide shared folders for users. Thankfully users
don't have the right to share their own folders, which makes things
easier (I hope).

Right now, the setup works like this, using Courier:

 - complete virtual mail setup
 - global shared folders configured in /etc/courier/shared/index
 - inside /home/shared-folder-name/Maildir/courierimapacl specific user
   get access to a folder
 - each folder a user has access is mapped to the namespace #shared
   like #shared.shared-folder-name

Now, if I split my backend storage server into multiple ones and user-A
is on server-1 and user-B is on server-2, but both need to access the
same shared folder, I have a problem.

I could of course move all users needing access to a shared folder to
the same server, but in the end, this will be a nightmare for me,
because I forsee having to move users around on a daily basis.

Right now, I am pondering with using an additional server with just the
shared folders on it and using NFS (or a cluster FS) to mount the shared
folder filesystem to each backend storage server, so each user has
potential access to a shared folders data.



Ideas? Suggestions? Nudges in the right direction?


Gr??e,
Sven.

-- 
Sigmentation fault. Core dumped.

On 1/7/2012 4:20 PM, Sven Hartge wrote:
> Hi *,
> 
> I am currently in the planning stage for a "new and improved"
mail
> system at my university.
> 
> Right now, everything is on one big backend server but this is causing
> me increasing amounts of pain, beginning with the time a full backup
> takes.
You failed to mention your analysis and diagnosis identifying the source
of the slow backup, and other issues your eluded to but didn't mention
specifically.  You also didn't mention how you're doing this full backup
(tar, IMAP; D2D or tape), where the backup bottleneck is, what mailbox
storage format you're using, total mailbox count and filesystem space
occupied.  What is your disk storage configuration?  Direct attach?
Hardware or software RAID?  What RAID level?  How many disks?  SAS or SATA?

It's highly likely your problems can be solved without the drastic
architecture change, and new problems it will introduce, that you
describe below.
> So naturally, I want to split this big server into smaller ones.
Naturally?  Many OPs spend significant x/y/z resources trying to avoid
the "shared nothing" storage backend setup below.
> To keep things simple, I want to pin a user to a server so I can avoid
> things like NFS or cluster aware filesystems. The mapping for each
> account is then inserted into the LDAP object for each user and the
> frontend proxy (perdition at the moment) then uses this information to
> route each access to the correct backend storage server running dovecot.
Splitting the IMAP workload like this isn't keeping things simple, but
increases complexity, on many levels.  And there's nothing wrong with
NFS and cluster filesystems if they are used correctly.
> So far this has been working nice with my test setup.
> 
> But: I also have to provide shared folders for users. Thankfully users
> don't have the right to share their own folders, which makes things
> easier (I hope).
> 
> Right now, the setup works like this, using Courier:
> 
>  - complete virtual mail setup
>  - global shared folders configured in /etc/courier/shared/index
>  - inside /home/shared-folder-name/Maildir/courierimapacl specific user
>    get access to a folder
>  - each folder a user has access is mapped to the namespace #shared
>    like #shared.shared-folder-name
> 
> Now, if I split my backend storage server into multiple ones and user-A
> is on server-1 and user-B is on server-2, but both need to access the
> same shared folder, I have a problem.
Yes, you do.
> I could of course move all users needing access to a shared folder to
> the same server, but in the end, this will be a nightmare for me,
> because I forsee having to move users around on a daily basis.
See my comments above.
> Right now, I am pondering with using an additional server with just the
> shared folders on it and using NFS (or a cluster FS) to mount the shared
> folder filesystem to each backend storage server, so each user has
> potential access to a shared folders data.
So you're going to implement a special case of what you're desperately
trying to avoid?  This makes no sense.
> Ideas? Suggestions? Nudges in the right direction?
Yes.  We need more real information.  Please provide:

1.  Mailbox count, total maildir file count and size
2.  Average/peak concurrent user connections
3.  CPU type/speed/total core count, total RAM, free RAM (incl buffers)
4.  Storage configuration--total spindles, RAID level, hard or soft RAID
5.  Filesystem type
6.  Backup software/method
7.  Operating system

Instead of telling us what you think the solution to your unidentified
bottleneck is and then asking "yeah or nay", tell us what the problem
is
and allow us to recommend solutions.  This way you'll get some education
and multiple solutions that may very well be a better fit, will perform
better, and possibly cost less in capital outlay and administration
time/effort.

-- 
Stan

Too much text in the rest of this thread so I haven't read it, but:

On 8.1.2012, at 0.20, Sven Hartge wrote:
> Right now, I am pondering with using an additional server with just the
> shared folders on it and using NFS (or a cluster FS) to mount the shared
> folder filesystem to each backend storage server, so each user has
> potential access to a shared folders data.
With NFS you'll run into problems with caching
(http://wiki2.dovecot.org/NFS). Some cluster fs might work better.

The "proper" solution for this that I've been thinking about would
be to use v2.1's imapc backend with master users. So that when user A wants
to access user B's shared folder, Dovecot connects to B's IMAP server
using master user login, and accesses the mailbox via IMAP. Probably
wouldn't be a big job to implement, mainly I'd need to figure out how
this should be configured..

Sven Hartge <sven at svenhartge.de> wrote:
> I am currently in the planning stage for a "new and improved"
mail
> system at my university.
OK, executive summary of the design ideas so far:

- deployment of X (starting with 4, but easily scalable) virtual servers
  on VMware ESX

- storage will be backed by a RDM on our iSCSI SAN.
  + main mailbox storage will be on 15k SAS6 600GB disks
  + backup rsnapshot storage will be on 7.2k SAS6 2TB disks

- XFS filesystem on LVM, allowing easy local snapshots for rsnapshot

- sharing folders from one user to another is not needed

- central public shared folders reside on its own storage server and are
  accessed through the imapc-backend configured for the
"#shared."-namespace
  (needs dovecot 2.1~rc3 or higher)

- mdbox with compression (23h lifetime, 50MB max size)

- quota in MySQL, allowing my MXes to check the quota for a user
  _before_ accepting any mail for him. This is a much needed feature,
  currently not possible and thus leading to backscatter right now.

- + Backup with bacula for file level backup every 24 hours (120 days
    retention)
  + rsnapshot to node local backup space for easier access (14 days
    retention)
  + possibly SAN-based remote snapshots to different storage tier.


Because sharing a RDM (or VMDK) with multiple VMs pins the VM to an ESX
server and prohibits HA and DRS in the ESX cluster and because of my bad
experience with cluster FS I want to avoid one and use only local
storage for the personal mailboxes of the users.

Each user is fixed to one server, routing/redirecting of IMAP/POP3
connections happens via perdition (popmap feature via LDAP lookup) in a
frontend server (this component is already working since some 3-ish
years).

So each node is isolated from the other nodes, knows only its users and
does not care about users on other nodes. This prevents usage of the
dovecot director, which only works if all nodes are able to access all
mailboxes (correct?)

I am aware this creates a SPoF for an 1/X portion of my users in the
case of a VM failure, but this is deemed acceptable, since the use of
VMs will allow me to quickly deploy a new one and reattach the RDM.
(And if my whole iSCSI storage or ESX cluster fails, I have other,
bigger problems than a non-functional mail system.)

Comments?

Gr??e,
Sven.

-- 
Sigmentation fault. Core dumped.