Sven Hartge
2012-Jan-07 22:20 UTC
[Dovecot] Providing shared folders with multiple backend servers
Hi *, I am currently in the planning stage for a "new and improved" mail system at my university. Right now, everything is on one big backend server but this is causing me increasing amounts of pain, beginning with the time a full backup takes. So naturally, I want to split this big server into smaller ones. To keep things simple, I want to pin a user to a server so I can avoid things like NFS or cluster aware filesystems. The mapping for each account is then inserted into the LDAP object for each user and the frontend proxy (perdition at the moment) then uses this information to route each access to the correct backend storage server running dovecot. So far this has been working nice with my test setup. But: I also have to provide shared folders for users. Thankfully users don't have the right to share their own folders, which makes things easier (I hope). Right now, the setup works like this, using Courier: - complete virtual mail setup - global shared folders configured in /etc/courier/shared/index - inside /home/shared-folder-name/Maildir/courierimapacl specific user get access to a folder - each folder a user has access is mapped to the namespace #shared like #shared.shared-folder-name Now, if I split my backend storage server into multiple ones and user-A is on server-1 and user-B is on server-2, but both need to access the same shared folder, I have a problem. I could of course move all users needing access to a shared folder to the same server, but in the end, this will be a nightmare for me, because I forsee having to move users around on a daily basis. Right now, I am pondering with using an additional server with just the shared folders on it and using NFS (or a cluster FS) to mount the shared folder filesystem to each backend storage server, so each user has potential access to a shared folders data. Ideas? Suggestions? Nudges in the right direction? Gr??e, Sven. -- Sigmentation fault. Core dumped.
Stan Hoeppner
2012-Jan-08 00:35 UTC
[Dovecot] Providing shared folders with multiple backend servers
On 1/7/2012 4:20 PM, Sven Hartge wrote:> Hi *, > > I am currently in the planning stage for a "new and improved" mail > system at my university. > > Right now, everything is on one big backend server but this is causing > me increasing amounts of pain, beginning with the time a full backup > takes.You failed to mention your analysis and diagnosis identifying the source of the slow backup, and other issues your eluded to but didn't mention specifically. You also didn't mention how you're doing this full backup (tar, IMAP; D2D or tape), where the backup bottleneck is, what mailbox storage format you're using, total mailbox count and filesystem space occupied. What is your disk storage configuration? Direct attach? Hardware or software RAID? What RAID level? How many disks? SAS or SATA? It's highly likely your problems can be solved without the drastic architecture change, and new problems it will introduce, that you describe below.> So naturally, I want to split this big server into smaller ones.Naturally? Many OPs spend significant x/y/z resources trying to avoid the "shared nothing" storage backend setup below.> To keep things simple, I want to pin a user to a server so I can avoid > things like NFS or cluster aware filesystems. The mapping for each > account is then inserted into the LDAP object for each user and the > frontend proxy (perdition at the moment) then uses this information to > route each access to the correct backend storage server running dovecot.Splitting the IMAP workload like this isn't keeping things simple, but increases complexity, on many levels. And there's nothing wrong with NFS and cluster filesystems if they are used correctly.> So far this has been working nice with my test setup. > > But: I also have to provide shared folders for users. Thankfully users > don't have the right to share their own folders, which makes things > easier (I hope). > > Right now, the setup works like this, using Courier: > > - complete virtual mail setup > - global shared folders configured in /etc/courier/shared/index > - inside /home/shared-folder-name/Maildir/courierimapacl specific user > get access to a folder > - each folder a user has access is mapped to the namespace #shared > like #shared.shared-folder-name > > Now, if I split my backend storage server into multiple ones and user-A > is on server-1 and user-B is on server-2, but both need to access the > same shared folder, I have a problem.Yes, you do.> I could of course move all users needing access to a shared folder to > the same server, but in the end, this will be a nightmare for me, > because I forsee having to move users around on a daily basis.See my comments above.> Right now, I am pondering with using an additional server with just the > shared folders on it and using NFS (or a cluster FS) to mount the shared > folder filesystem to each backend storage server, so each user has > potential access to a shared folders data.So you're going to implement a special case of what you're desperately trying to avoid? This makes no sense.> Ideas? Suggestions? Nudges in the right direction?Yes. We need more real information. Please provide: 1. Mailbox count, total maildir file count and size 2. Average/peak concurrent user connections 3. CPU type/speed/total core count, total RAM, free RAM (incl buffers) 4. Storage configuration--total spindles, RAID level, hard or soft RAID 5. Filesystem type 6. Backup software/method 7. Operating system Instead of telling us what you think the solution to your unidentified bottleneck is and then asking "yeah or nay", tell us what the problem is and allow us to recommend solutions. This way you'll get some education and multiple solutions that may very well be a better fit, will perform better, and possibly cost less in capital outlay and administration time/effort. -- Stan
Timo Sirainen
2012-Jan-09 14:51 UTC
[Dovecot] Providing shared folders with multiple backend servers
Too much text in the rest of this thread so I haven't read it, but: On 8.1.2012, at 0.20, Sven Hartge wrote:> Right now, I am pondering with using an additional server with just the > shared folders on it and using NFS (or a cluster FS) to mount the shared > folder filesystem to each backend storage server, so each user has > potential access to a shared folders data.With NFS you'll run into problems with caching (http://wiki2.dovecot.org/NFS). Some cluster fs might work better. The "proper" solution for this that I've been thinking about would be to use v2.1's imapc backend with master users. So that when user A wants to access user B's shared folder, Dovecot connects to B's IMAP server using master user login, and accesses the mailbox via IMAP. Probably wouldn't be a big job to implement, mainly I'd need to figure out how this should be configured..
Sven Hartge
2012-Jan-11 12:50 UTC
[Dovecot] Providing shared folders with multiple backend servers
Sven Hartge <sven at svenhartge.de> wrote:> I am currently in the planning stage for a "new and improved" mail > system at my university.OK, executive summary of the design ideas so far: - deployment of X (starting with 4, but easily scalable) virtual servers on VMware ESX - storage will be backed by a RDM on our iSCSI SAN. + main mailbox storage will be on 15k SAS6 600GB disks + backup rsnapshot storage will be on 7.2k SAS6 2TB disks - XFS filesystem on LVM, allowing easy local snapshots for rsnapshot - sharing folders from one user to another is not needed - central public shared folders reside on its own storage server and are accessed through the imapc-backend configured for the "#shared."-namespace (needs dovecot 2.1~rc3 or higher) - mdbox with compression (23h lifetime, 50MB max size) - quota in MySQL, allowing my MXes to check the quota for a user _before_ accepting any mail for him. This is a much needed feature, currently not possible and thus leading to backscatter right now. - + Backup with bacula for file level backup every 24 hours (120 days retention) + rsnapshot to node local backup space for easier access (14 days retention) + possibly SAN-based remote snapshots to different storage tier. Because sharing a RDM (or VMDK) with multiple VMs pins the VM to an ESX server and prohibits HA and DRS in the ESX cluster and because of my bad experience with cluster FS I want to avoid one and use only local storage for the personal mailboxes of the users. Each user is fixed to one server, routing/redirecting of IMAP/POP3 connections happens via perdition (popmap feature via LDAP lookup) in a frontend server (this component is already working since some 3-ish years). So each node is isolated from the other nodes, knows only its users and does not care about users on other nodes. This prevents usage of the dovecot director, which only works if all nodes are able to access all mailboxes (correct?) I am aware this creates a SPoF for an 1/X portion of my users in the case of a VM failure, but this is deemed acceptable, since the use of VMs will allow me to quickly deploy a new one and reattach the RDM. (And if my whole iSCSI storage or ESX cluster fails, I have other, bigger problems than a non-functional mail system.) Comments? Gr??e, Sven. -- Sigmentation fault. Core dumped.