Hello, I found a posting on the mailing list from 2008 "Adding new storage backend to Dovecot" while searching for POP3 and IMAP servers, that allow to use a NoSQL backend rather than a traditional file system. The problem with traditional filesystems is, that they are very limited. You cannot easily extend it, even with RAID 10 all messages may be lost due to a RAID controller failure and just-in-time backups of a heavy loaded system with consistently changing messages folders are virtually impossible. Modern object stores like Scality, CouchDB or MongoDB can be very handy, as they easily allow to keep an arbitrary amount of copies of each message on different servers, you can simply add a storage node and new storage is immediately available and mailboxes aren't bound to a certain server any longer, but messages are evenly distributed across all storage nodes, thus distributing load as well. Since the original posting was dated 2008 I was in good hope, that someone worked on that already, but at least in dovecot-2.0.9/src/lib-storage I couldn't find anything relating to NoSQL message stores. Isn't this option of big interest for every bigger mailbox provider? How hard would it be to develop? Would it be possible to get this developed on a paid basis by the dovecot core team? Kind regards Marten
Am 08.02.2011 13:22, schrieb Marten Lehmann:> Hello, > > I found a posting on the mailing list from 2008 "Adding new storage > backend to Dovecot" while searching for POP3 and IMAP servers, that > allow to use a NoSQL backend rather than a traditional file system.There has been a recent mail from Timo regarding the future development plans for 2.1 where he mentioned a key/value storage backend. http://dovecot.markmail.org/message/juecu7wm2nwkluji?q=plans+2.1> > Isn't this option of big interest for every bigger mailbox provider? > How hard would it be to develop? Would it be possible to get this > developed on a paid basis by the dovecot core team?I am also very interested on seeing this implemented, maybe also on a paid basis.... I think the core team is Timo :-)>Regards, Oliver
On 8.2.2011, at 14.22, Marten Lehmann wrote:> I found a posting on the mailing list from 2008 "Adding new storage backend to Dovecot" while searching for POP3 and IMAP servers, that allow to use a NoSQL backend rather than a traditional file system.I don't remember any mails about NoSQL backend from 2008, but at the end of 2009 it became something I started planning on doing. It's listed in http://wiki2.dovecot.org/Roadmap> Isn't this option of big interest for every bigger mailbox provider? How hard would it be to develop? Would it be possible to get this developed on a paid basis by the dovecot core team?It should be pretty easy to develop if you don't care about performance, but since you likely do, it gets more difficult. The latencies for db lookups are higher than local reads, so it would be good to be able to do multiple lookups in parallel. I've recently started imapc backend, which has similar problems and I was planning on solving those problems with it, so it will be easier then for nosql backend. Also it would be good to design a storage backend for nosql that didn't require any kind of locking. One guy recently started looking into developing something for this, but I haven't heard from him for some weeks. I could ask if he's gotten anywhere.. As for me, I'm not entirely sure about my plans for this spring. There's another company wanting to pay me, but I don't know how much of my time they'll really need.
On Tue, Feb 8, 2011 at 1:22 PM, Marten Lehmann <lehmann at cnm.de> wrote:> Modern object stores like Scality, CouchDB or MongoDB can be very handy, as > they easily allow to keep an arbitrary amount of copies of each message on > different servers, you can simply add a storage node and new storage is > immediately available and mailboxes aren't bound to a certain server any > longer, but messages are evenly distributed across all storage nodes, thus > distributing load as well.The Scality webpages mentions they?ve developed storage connector for dovecot: http://www.scality.com/storage-solutions/ whatever that means.. -jf
On Tue, 8 Feb 2011, Marten Lehmann wrote:> The problem with traditional filesystems is, that they are very limited. You > cannot easily extend it, even with RAID 10 all messages may be lost due to a > RAID controller failure and just-in-time backups of a heavy loaded system > with consistently changing messages folders are virtually impossible.Hi Marten. There are many options that allow you to snapshop an active filesystem and guarantee a consistent backup. Eg LVM on Linux or many other OSes. WAFL on a NetApp, etc. In my experience a lot of high end mail systems store mail on the filesystem in Maildir format.> Modern object stores like Scality, CouchDB or MongoDB can be very handy, as > they easily allow to keep an arbitrary amount of copies of each message onThis is certainly an interesting idea.> different servers, you can simply add a storage node and new storage is > immediately available and mailboxes aren't bound to a certain server any > longer, but messages are evenly distributed across all storage nodes, thus > distributing load as well.Being able to easily rebalance load would be useful. I've known of several well known companies with a lot of mail users where mail was stored on filesystems across many systems and significant imbalances developed over time. There was no easy way to rebalance them without causing an outage for at least some users.> Isn't this option of big interest for every bigger mailbox provider? How hard > would it be to develop? Would it be possible to get this developed on a paid > basis by the dovecot core team?In my experience OSS developers are quite open to being paid to add features :) Cheers, Rob -- Email: robert at timetraveller.org Linux counter ID #16440 IRC: Solver (OFTC & Freenode) Web: http://www.practicalsysadmin.com Contributing member of Software in the Public Interest (http://spi-inc.org/) Open Source: The revolution that silently changed the world