I've been doing some thinking on a new better mailbox format that'd be
faster than mbox and maildir for most common uses. Comments welcome.
It won't come to Dovecot v1.0, but I'll probably create another branch
to Dovecot CVS where I'll start implementing it and some other possibly
destabilizing changes.
Generic
-------
The point is to have a mailbox format where the mailbox can consist of one
or more files. Grouping multiple files in a single file makes it faster to
read, but it's slower to expunge mails from the beginning of the file. So
this format would allow sysadmin to specify rules on how large the files
would be allowed to grow.
This format is mostly designed to work nicely with separate index files
such as Dovecot has. The flags are stored in the mailbox files mostly as a
backup in case the index gets lost or corrupted.
Index File
----------
Named "index".
Index file gives a mapping between UIDs <-> filenames:
Header: <uidvalidity> <last-used-uid>
Record: <uid-first> <uid-last> <filename>
Record: ..
eg.:
1127468129 5
1 3 msg.1
4 4 msg.4
All of the UIDs don't have to actually exist in the file if they have been
expunged. The record should be removed once all of the UIDs have been removed
from it (and the file itself been deleted).
The same file may appear multiple times in the index. For example:
1 2 msg.1
3 3 msg.3
4 5 msg.1
This means that UIDs 1, 2, 4 and 5 are in file msg.1, but UID 3 is in file
msg.3.
The filename doesn't matter as long as it's unique, but probably
wouldn't
hurt to include its first UID in the filename so humans can find mails more
easily. The filename however must not end with ".lock".
The file is modified by creating first exclusively owned "index.lock"
file,
updating it and then rename()ing it over the index file. This lock file
shouldn't be held for a long time, so if it exists, and it doesn't get
modified at all within 30 seconds, it can be overwritten. Before
rename()ing over the index file, you should check with stat() that the
index.lock file is still the same as you expect it to be (in case your
process was temporarily stopped for over 30 secs).
The index file's mtime timestamp must grow after each modification. This
makes it easy to check if new mail has been added to the mailbox.
Mail File
---------
File header:
Bytes Format Description
8 hex Header size
16 hex Append offset
4 hex Mail header size
4 hex Mail header padding
4 hex Keyword count in mail headers
Mail header:
Bytes Format Description
8 hex UID
16 hex Mail size
1 1/0 Answered flag
1 1/0 Flagged flag
1 1/0 Deleted flag
1 1/0 Seen flag
1 1/0 Draft flag
1 1/0 Expunged flag
n 1/0 Keywords
The file layout then goes:
<File header>
- possible padding due to Header size
<Mail header #1>
- possible padding due to Mail header size
<Mail data #1>
- possible padding due to Mail header padding (ie. next Mail header's
offset is divisible with Mail header padding)
<Mail header #2>
etc..
The UIDs must be ascending within the file (but as said above, some UIDs
between them may exist in other files).
When the file is opened, it must be shared locked with flock(). Expunging
mails requires exclusive flock(), so expunges can't happen while someone is
still reading the file. Mails can however be marked with expunged-flag, and
such mails should be treated as if they are already expunged. Expunged flag
must not be removed. If the mail for some reason is wanted to be
un-expunged, it must be copied as a new mail with new UID to the mailbox.
Modifying flags doesn't require any locks. You'll simply write over
those
flags you wish to change. That's why each flag takes one byte.
Safest way to expunge is to copy all non-expunged mails after the first
expunged one to new file. Then update the index file with the new location
for the UIDs, and at last ftruncate() at the position of first expunged
mail in the original file.
Alternative way to expunge is to just move the data over the expunged mails
and fruncate() at the end. No index file updates necessary. This is more
filesystem quota friendly, but if the expunging gets aborted in the middle
it will corrupt the mails.
Appending mails can be done by either locking existing file with a dotlock
and appending to it, or creating a new file. If appending to existing file,
also flock() it so expunging won't happen in the middle of append. If
appending to new file, create it with .lock suffix and then check that the
file without .lock wasn't already just created.
When appending to existing file, check "Append offset" from file's
header.
If that offset is at the end of file, just append there. Otherwise read the
mail's header from there. If it has a non-zero UID and index file says that
the same UID exists in this file, jump over the mail and do the same checks
for all the rest of the mails in the file. Once the check doesn't match or
you reach EOF, update the append offset in the header to that offset and
write your mail there.
Initially write the UID in mail's header as zeroes ('0', not NUL).
Once the
append is completely finished, you'll have to update it with the new UID.
You'll do this by locking index file, reading the latest last-used-uid and
using the next value for your UID. Once you've written it, rewrite the
index file with updated last-used-uid field and your UID added to it. After
this, update the append offset in the mail file and unlock it.
Keywords file
-------------
Named "index.keywords". This contains mappings between keyword numbers
and
names. TODO: may be problematic with lockless flag writes..
Recent file
-----------
"index.recent.<uid>" file tells what the last non-recent UID
was. The file
doesn't have any data inside it.
When opening the mailbox you should find and rename() the existing
index.recent.<uid> file to the new index.recent.<last-uid>. A
successful
rename() means the mails between <uid>+1 .. <last-uid> are recent
for you.
A failed rename() means that someone else managed to do it. You should look
for the file again and if it was renamed to less than <last-uid>, you
should try again.
Nightly cleanups
----------------
Because expunged mails may be left lying around for a while, there probably
should be some nightly check that cleans them up. Other cleanups that may be
necessary after crashes are:
- .lock files
- expunged data at the end of files (can be checked with same rules as how
appending determines the offset where to append)
One problem with nightly expunging is how to expunge mails if the file is
still locked by some process. Forced shutdowns may be annoying to users and
can be difficult to implement..
Alternative locking strategy
----------------------------
The locking strategy as described above is optimized for high concurrency:
Only if mail file is in the middle of being expunged, we're waiting on
locks. With appending we may be waiting for existing appends to finish if
we don't wish to create a new file for some reason. Other than those,
reading and flag updating are completely lockless.
The two downsides to this are:
1. flock() isn't available everywhere. It could however be replaced with
fcntl() if that's available. NFS may not have either.
2. Expunges can't happen while multiple processes are accessing the mailbox
at a same time. This may be a problem if the mailbox never has less than
two processes accessing it.
The alternative to flock()ing would be to simply handle unexpected
end-of-file reads as a sign of expunge, which would require re-reading
index file to see where the mails went (or if they were really expunged).
This has two problems however. One is that flag writes couldn't catch
expunges without locking, and another is that if you've already sent half a
mail to IMAP client, you have no other choice than to disconnect the client
if you can't send the rest of the mail.
It might be worth it optionally. Especially for NFS people. Then you'd
basically have the dotlock file that locks appends, expunges and flag
updates. Reading would still be lockless.
Also for the best performance the mail flags could be stored only in
Dovecot's index files. In that case there's no need to update flags and
the
only downside with this locking strategy would be the potential
disconnections, which probably aren't all that bad.
Compatibility
-------------
If needed, it would be possible to create new/ and tmp/ directories under
the mailbox to allow mail deliveries to be in maildir format. The maildir
files would be then either just moved or maybe be appended to existing mail
files.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20050923/136ff9b4/attachment.bin>
Timo Sirainen wrote:> The point is to have a mailbox format where the mailbox can consist of one > or more files. Grouping multiple files in a single file makes it faster to > read, but it's slower to expunge mails from the beginning of the file. So > this format would allow sysadmin to specify rules on how large the files > would be allowed to grow.This seems like a lot of complexity for an unknown amount of performance. Sure, it is going to be loads faster than multi-megabyte mbox mailboxes, but you can color me unconvinced that this will be a significant win over maildir. The primary advantage to maildir is the utter simplicity of all operations; at no time do you need to completely rewrite any files and all operations are 100% atomic. The index format under maildir is also very simple, since you only need to keep track of the filename (and flags) rather than filename and offset and flags. And with modern filesystems, disk access is intelligently cached. If you are trying to tune for where there are significant numbers of very small (< 2k) files (well smaller than the typical block size in the underlying filesystem), you may be aiming too small. It looks like the median file size in my maildir folders is about 3100 bytes. What sizes were you thinking the typical admin would set as the limit? Personally, I think your time would be better spent integrating a database message store and let the database engine deal with storage and indexing issues. YMMV. ;-) John -- John Peacock Director of Information Research and Technology Rowman & Littlefield Publishing Group 4501 Forbes Boulevard Suite H Lanham, MD 20706 301-459-3366 x.5010 fax 301-429-5748
Timo Sirainen wrote:> I've been doing some thinking on a new better mailbox format that'd be > faster than mbox and maildir for most common uses. Comments welcome.Didn't read the details, but sounds good. Only thing is: One reason I like dovecot is the support for an already good format that's support by a lot of software: maildir. The main reason I switched from Courier-IMAP was speed. Dovecot offers a good compromise between speed and standard-compliance (if you'd call maildir a standard). If speed was the most important thing for me, I'd use Cyrus. Other people may like things like dbmail. As long as Dovecot continues supporting the current formats, there's nothing bad with introducing YAMF (TM) (yet another mailbox format), of course. Only that the old ones will probably get less attention. And I'm not sure if a new format will solve all maildir's problems, especially since there are not much in practice (for me: none).
On Fri, 23 Sep 2005, Timo Sirainen wrote:> The point is to have a mailbox format where the mailbox can consist of one > or more files. Grouping multiple files in a single file makes it faster to > read, but it's slower to expunge mails from the beginning of the file. So > this format would allow sysadmin to specify rules on how large the files > would be allowed to grow.for about a decade now i've set up all my inbound mail to deliver to two mboxes -- one is an "inbox", the other is an "archive". the inbox is what i look at with my mta, and i delete things from it as soon as i'm done reading or dealing with them. the archive is there so i never have to think about whether i want to save something (and the disk space is totally manageable)... similar to how gmail works. my "archive" is a collection of mbox files. one named "current" which is where new deliveries occur, and the others named YYYYMMDD.bz2, which are compressed/read-only archived mboxes. the rotation/compression occurs in a cronjob depending on the size of the current file. it's a bit of a kludge, because the file boundaries are very obvious if you need to find a thread that's spread across a few of them. but it's all just mbox so it's easy to grep and concat a few files into a temporary mbox and extract a thread with any MUA. i've wanted to turn this into a "real" format supported by dovecot for a while but i never seem to get to it... it sounds like you're headed in a similar direction.> This format is mostly designed to work nicely with separate index files > such as Dovecot has. The flags are stored in the mailbox files mostly as a > backup in case the index gets lost or corrupted.here's one point where my thinking has differed -- i'd treat the mailbox files as read-only (plus one file which is append-only) and include an append-only modification log for recovery purposes... read-only mailbox files permit compression, and don't have the nightly cleanup lockout problems you mention later. the log is affordable in terms of disk space. the mailbox files plus the log are sufficient to recover the current state of the mailbox. in my case i'd probably grow the log forever... because i'd also never be doing expunges... but the process of doing an expunge can also update any info embedded into the (compressed|uncompressed) mailbox files. the cost of a log in terms of disk writes for updates is probably better than updates to the mailbox files themselves -- assuming a log entry is compact enough that dozens fit in a 4KiB filesystem block you can amortize several updates into one synchronous disk write rather than having dozens of synchronous writes to separate blocks.> When the file is opened, it must be shared locked with flock(). Expunging > mails requires exclusive flock(), so expunges can't happen while someone is > still reading the file.with read-only mailbox files there's no sharing restriction -- an expunge can create a new mbox file, update the index, and unlink the old one. (minor easy to solve race if a reader gets a filename from the index which is renamed before it's opened... just loop... expunges should be infrequent enough this has no livelock potential.) but i guess that's not very quota friendly... oops.> Compatibility > ------------- > > If needed, it would be possible to create new/ and tmp/ directories under > the mailbox to allow mail deliveries to be in maildir format. The maildir > files would be then either just moved or maybe be appended to existing mail > files.yeah i've definitely wanted delivery to require no changes -- in my case it just happens to the mbox file named "current". i think it's best not to deal with indices and other fancy things during delivery because there's no opportunity to amortize synchronized disk writes... and i know in the case of large ISP mail sites there tend to be a lot of users who never read their mail. you'll support more users on less hardware if the synchronous disk writes are at a minimum. -dean
> I've been doing some thinking on a new better mailbox format that'd be > faster than mbox and maildir for most common uses. Comments welcome. >[snip] Apologies for being rude, but I've not heard complaints about maildir being slow. Is it? It would be nice to see benchmarks. It seems like some kind of cache manager would solve the problem, if there was one, but then some might say we'd be doing the job of the kernel. Are there kernel parameters which can be tweaked to address the slowness?
Timo Sirainen wrote:> Index File > ----------[...]> The file is modified by creating first exclusively owned "index.lock" file, > updating it and then rename()ing it over the index file. This lock file > shouldn't be held for a long time, so if it exists, and it doesn't get > modified at all within 30 seconds, it can be overwritten. Before > rename()ing over the index file, you should check with stat() that the > index.lock file is still the same as you expect it to be (in case your > process was temporarily stopped for over 30 secs).The locking and handling of the index strikes me as a regression back to the mbox problems that Maildir tried to solve. Have you considered other approaches, such as having the index be under control of a daemon, and use IPC to communicate events to that daemon, which could then exclusively handle modifying the file? Or maybe a fixed size record structure, which would permit more of a random record access and modification with less chance of inter process conflicts? Any way you slice it, though, these are just approximations of a database server. Maybe embedding SQLite (just for indexes) is the answer? -Tom
Hi Timo, I think it should be a database like format. (keep reading) Since a couple of week ago, I was researching which solution was better, mbox or Maildir. And asking to myself, "is there a better way?" My ideal solution would be a format with an API so that it is easy to implement in other aplications like procmail, sendmail, etc, etc. As I see, - mbox is good because you don't get a fragmented "Folder", is easy to backup and does not eat inodes. But, it is a trouble to find, delete or insert and email in one big file. - Maildir is faster to access because it have and "index", in the filesystem. But it uses a lot of inodes and reading all emails in a big folder means reading too much files. So, an ideal solution should be: "an mbox like format, with an index and DB like allocation". The features should be: - mbox like format: just one big file or a folder separated in x bytes parts. For example, create a new file every 500MB. This will save inodes and will be easy to backup - with an index: it will make really fast to search, and access a particular email in that big file. No need to read 500MB just to get the last email. - DB like allocation: you don't to worry about deleting the first email in a 2GB mbox file. The space inside this big file will be administered like in a DB. You use constant size blocks inside this big file and your index help you get to your particular email easy. No need for a cronological storage of email in a big file. Final notes, this is an ideal setup. You could reinvent (a better) the wheel or use tools available like mysql, sleepycat, pgsql, etc. The success of this new mail format will be the easy implementation of the API. It must have a portable library so that you can easilly support this new format in a transparent way in old software like sendmail, procmail, qmail, postfix, etc Is just an idea I was having, not final, but is more or less what I was thinking. HTH and thanks for reading, Oliver Timo Sirainen wrote:>I've been doing some thinking on a new better mailbox format that'd be >faster than mbox and maildir for most common uses. Comments welcome. > >It won't come to Dovecot v1.0, but I'll probably create another branch >to Dovecot CVS where I'll start implementing it and some other possibly >destabilizing changes. > >Generic >------- > >The point is to have a mailbox format where the mailbox can consist of one >or more files. Grouping multiple files in a single file makes it faster to >read, but it's slower to expunge mails from the beginning of the file. So >this format would allow sysadmin to specify rules on how large the files >would be allowed to grow. > >This format is mostly designed to work nicely with separate index files >such as Dovecot has. The flags are stored in the mailbox files mostly as a >backup in case the index gets lost or corrupted. > >Index File >---------- > >Named "index". > >Index file gives a mapping between UIDs <-> filenames: > >Header: <uidvalidity> <last-used-uid> >Record: <uid-first> <uid-last> <filename> >Record: .. > >eg.: > > 1127468129 5 > 1 3 msg.1 > 4 4 msg.4 > >All of the UIDs don't have to actually exist in the file if they have been >expunged. The record should be removed once all of the UIDs have been removed >from it (and the file itself been deleted). > >The same file may appear multiple times in the index. For example: > > 1 2 msg.1 > 3 3 msg.3 > 4 5 msg.1 > >This means that UIDs 1, 2, 4 and 5 are in file msg.1, but UID 3 is in file >msg.3. > >The filename doesn't matter as long as it's unique, but probably wouldn't >hurt to include its first UID in the filename so humans can find mails more >easily. The filename however must not end with ".lock". > >The file is modified by creating first exclusively owned "index.lock" file, >updating it and then rename()ing it over the index file. This lock file >shouldn't be held for a long time, so if it exists, and it doesn't get >modified at all within 30 seconds, it can be overwritten. Before >rename()ing over the index file, you should check with stat() that the >index.lock file is still the same as you expect it to be (in case your >process was temporarily stopped for over 30 secs). > >The index file's mtime timestamp must grow after each modification. This >makes it easy to check if new mail has been added to the mailbox. > >Mail File >--------- > >File header: > >Bytes Format Description >8 hex Header size >16 hex Append offset >4 hex Mail header size >4 hex Mail header padding >4 hex Keyword count in mail headers > >Mail header: > >Bytes Format Description >8 hex UID >16 hex Mail size >1 1/0 Answered flag >1 1/0 Flagged flag >1 1/0 Deleted flag >1 1/0 Seen flag >1 1/0 Draft flag >1 1/0 Expunged flag >n 1/0 Keywords > >The file layout then goes: > ><File header> > - possible padding due to Header size ><Mail header #1> > - possible padding due to Mail header size ><Mail data #1> > - possible padding due to Mail header padding (ie. next Mail header's > offset is divisible with Mail header padding) ><Mail header #2> >etc.. > >The UIDs must be ascending within the file (but as said above, some UIDs >between them may exist in other files). > >When the file is opened, it must be shared locked with flock(). Expunging >mails requires exclusive flock(), so expunges can't happen while someone is >still reading the file. Mails can however be marked with expunged-flag, and >such mails should be treated as if they are already expunged. Expunged flag >must not be removed. If the mail for some reason is wanted to be >un-expunged, it must be copied as a new mail with new UID to the mailbox. > >Modifying flags doesn't require any locks. You'll simply write over those >flags you wish to change. That's why each flag takes one byte. > >Safest way to expunge is to copy all non-expunged mails after the first >expunged one to new file. Then update the index file with the new location >for the UIDs, and at last ftruncate() at the position of first expunged >mail in the original file. > >Alternative way to expunge is to just move the data over the expunged mails >and fruncate() at the end. No index file updates necessary. This is more >filesystem quota friendly, but if the expunging gets aborted in the middle >it will corrupt the mails. > >Appending mails can be done by either locking existing file with a dotlock >and appending to it, or creating a new file. If appending to existing file, >also flock() it so expunging won't happen in the middle of append. If >appending to new file, create it with .lock suffix and then check that the >file without .lock wasn't already just created. > >When appending to existing file, check "Append offset" from file's header. >If that offset is at the end of file, just append there. Otherwise read the >mail's header from there. If it has a non-zero UID and index file says that >the same UID exists in this file, jump over the mail and do the same checks >for all the rest of the mails in the file. Once the check doesn't match or >you reach EOF, update the append offset in the header to that offset and >write your mail there. > >Initially write the UID in mail's header as zeroes ('0', not NUL). Once the >append is completely finished, you'll have to update it with the new UID. >You'll do this by locking index file, reading the latest last-used-uid and >using the next value for your UID. Once you've written it, rewrite the >index file with updated last-used-uid field and your UID added to it. After >this, update the append offset in the mail file and unlock it. > >Keywords file >------------- > >Named "index.keywords". This contains mappings between keyword numbers and >names. TODO: may be problematic with lockless flag writes.. > >Recent file >----------- > >"index.recent.<uid>" file tells what the last non-recent UID was. The file >doesn't have any data inside it. > >When opening the mailbox you should find and rename() the existing >index.recent.<uid> file to the new index.recent.<last-uid>. A successful >rename() means the mails between <uid>+1 .. <last-uid> are recent for you. >A failed rename() means that someone else managed to do it. You should look >for the file again and if it was renamed to less than <last-uid>, you >should try again. > >Nightly cleanups >---------------- > >Because expunged mails may be left lying around for a while, there probably >should be some nightly check that cleans them up. Other cleanups that may be >necessary after crashes are: > > - .lock files > - expunged data at the end of files (can be checked with same rules as how > appending determines the offset where to append) > >One problem with nightly expunging is how to expunge mails if the file is >still locked by some process. Forced shutdowns may be annoying to users and >can be difficult to implement.. > >Alternative locking strategy >---------------------------- > >The locking strategy as described above is optimized for high concurrency: >Only if mail file is in the middle of being expunged, we're waiting on >locks. With appending we may be waiting for existing appends to finish if >we don't wish to create a new file for some reason. Other than those, >reading and flag updating are completely lockless. > >The two downsides to this are: > >1. flock() isn't available everywhere. It could however be replaced with >fcntl() if that's available. NFS may not have either. > >2. Expunges can't happen while multiple processes are accessing the mailbox >at a same time. This may be a problem if the mailbox never has less than >two processes accessing it. > >The alternative to flock()ing would be to simply handle unexpected >end-of-file reads as a sign of expunge, which would require re-reading >index file to see where the mails went (or if they were really expunged). >This has two problems however. One is that flag writes couldn't catch >expunges without locking, and another is that if you've already sent half a >mail to IMAP client, you have no other choice than to disconnect the client >if you can't send the rest of the mail. > >It might be worth it optionally. Especially for NFS people. Then you'd >basically have the dotlock file that locks appends, expunges and flag >updates. Reading would still be lockless. > >Also for the best performance the mail flags could be stored only in >Dovecot's index files. In that case there's no need to update flags and the >only downside with this locking strategy would be the potential >disconnections, which probably aren't all that bad. > >Compatibility >------------- > >If needed, it would be possible to create new/ and tmp/ directories under >the mailbox to allow mail deliveries to be in maildir format. The maildir >files would be then either just moved or maybe be appended to existing mail >files. > > >-- Oliver Schulze L. <oliver at samera.com.py>
On Fri, 2005-09-23 at 21:48 -0400, Tom Metro wrote:> Timo Sirainen wrote: > > Index File > > ---------- > [...] > > The file is modified by creating first exclusively owned "index.lock" file, > > updating it and then rename()ing it over the index file. This lock file > > shouldn't be held for a long time, so if it exists, and it doesn't get > > modified at all within 30 seconds, it can be overwritten. Before > > rename()ing over the index file, you should check with stat() that the > > index.lock file is still the same as you expect it to be (in case your > > process was temporarily stopped for over 30 secs). > > The locking and handling of the index strikes me as a regression back to > the mbox problems that Maildir tried to solve.Replying a bit late, but anyway.. I don't think the above index.lock is a problem in any way. Actually Dovecot already uses similar method with Maildir's dovecot-uidlist file. Maildir is lockless only in theory. Unless the maildir is globally locked while checking its contents, files may get temporarily lost with all of the filesystems that I know of. I think the locking is only a problem if you're holding the lock for a long time. For example if you need to keep the mailbox locked as long as some IMAP client is reading/writing messages, that's bad. That's a problem with mbox, but not with maildir/dbox. dbox's index.lock file needs to exist only in two situations: 1. While new message UIDs need to be allocated (as the final step of saving new mail(s) to mailbox). A global lock is needed for this with any kind of mail storage with IMAP, since UIDs must exist and they must always grow. 2. Writing message flag changes / expunging mails. These changes should go pretty quickly as well.> Have you considered other approaches, such as having the index be under > control of a daemon, and use IPC to communicate events to that daemon, > which could then exclusively handle modifying the file?One of the biggest reasons for dbox's existance is that it needs to work well with clustered filesystems. And relying on a daemon running in only one computer kind of defeats cluster's purpose then.. Anyway I'm not sure how that would actually benefit anything. A single process could be a bottleneck if it handled all users' indexes, so it should be able to scale to multiple processes. And to avoid locking in those cases, each process should handle only a group of specific users. And if we're going there, it might as well be the imap process itself that does it all. Redirecting all imap connections for one user to same imap process wouldn't be too difficult to implement (and it's been in my mind for a while already), but having pop3 and dovecot-lda also in the same process could get more tricky. But does it really matter if the locking is handled by serialization (by making everything go through a single process) or actual locking? If there's only a single writer, the locking succeeds easily always. If there are multiple writers, you'll need to wait in both cases. Although I suppose serialization provides more fair scheduling. And even if there was only a single process updating the index file, I'd probably still make it update the index using the exact same rename(index.lock, index) to make sure the file doesn't get corrupted in case of crashes :)> Any way you slice it, though, these are just approximations of a > database server. Maybe embedding SQLite (just for indexes) is the answer?Doesn't look like SQLite's locking (or writing in general) is in any way cheaper than what I'm currently doing: http://www.sqlite.org/lockingv3.html Actually it looks like it may even block on getting a shared lock if there are writes happening at the same time. I think many other databases don't block there but instead use rollback file to get the old data. Dovecot's index files and dbox's index files aren't needed to be read-locked at all, so they never block on reading. SQLite could be useful for storing messages' and mailboxes' metadata (ANNOTATE, ANNOTATEMORE extension) since those require almost a full database to work properly, but I don't think it's a good idea for what Dovecot/dbox currently uses index files for. SQL in general isn't very well suited for them. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: <http://dovecot.org/pipermail/dovecot/attachments/20051206/ba26b98d/attachment.bin>