Xapian discuss - Oct 2014 - How FastMail is using Xapian with Cyrus IMAPd

I've been meaning to make a blog post out of this for a while, but I
haven't done it yet, and since I'm pushing to get Xapian to include our
patches, I figured I should actually write something up to show how awesome
Xapian is for us, and how it's working on our systems.

Cyrus IMAPd is written in pure C - Xapian is the first piece of C++ that's
been included.  We knew we needed a search engine, and at first we were looking
at embedding Sphinx, but the single server model worried us.  We wanted to have
a search database per user for modularity.  That meant that loading the database
and writing a new message's fields into it had to be super-fast.

We also wanted snippet support, which Greg Banks (the programmer working on that
feature) added as a patch to Xapian.  I've attached an image to show the
cross folder search results in our interface using that feature.

We built a C API wrapper around the Xapian features we needed, and tried to run
in production.  Unfortunately, the IO hit from indexing new messages in real
time was too much - our server couldn't cope.

Luckily, Xapian supports searching from multiple databases at once, so we came
up with the idea of a tiered database structure.  We have 4 tiers at FastMail,
though we don't actually use the 'meta' one (SSD) at the moment:

* temp
* meta
* data
* archive

So new messages get indexed to tmpfs in a small database in the temp tier.  A
job runs every hour to see if tmpfs is getting too full (over 50% of the defined
size), in which case it compacts immediately, otherwise a job compacts the temp
and meta tiers down to the data tier every day.

Once a week, we compact all the tiers except archive back to the data tier.

The archive tier stays untouched unless we move a user, or as a manual
operation, where we compact everything into a single archive.

Both external locking (see my other post a minute ago about write vs read and
DatabaseModified errors) and the compaction logic are managed via a separate
file called xapianactive.  The xapianactive looks like this:

% cat /mnt/ssd30/sloti30t01/store23/conf/user/b/brong.xapianactive
temp:264 archive:2 data:37

The first item in the file is the writable index - all the others are read-only.

These map to paths on disk according to the config file:

% grep search /etc/cyrus/imapd-sloti30t01.conf 
search_engine: xapian
search_index_headers: no
search_batchsize: 8192
defaultsearchtier: temp
tempsearchpartition-default: /var/run/cyrus/search-sloti30t01
metasearchpartition-default: /mnt/ssd30/sloti30t01/store23/search
datasearchpartition-default: /mnt/i30search/sloti30t01/store23/search
archivesearchpartition-default: /mnt/i30search/sloti30t01/store23/search-archive

(the 'default tier' is to tell the system where to create a new search
item)

So based on these paths, we find.

% du -s /var/run/cyrus/search-sloti30t01/b/user/brong/*
/mnt/i30search/sloti30t01/store23/search/b/user/brong/*
/mnt/i30search/sloti30t01/store23/search-archive/b/user/brong/*
3328	/var/run/cyrus/search-sloti30t01/b/user/brong/xapian.264
1520432	/mnt/i30search/sloti30t01/store23/search/b/user/brong/xapian.37
3365336	/mnt/i30search/sloti30t01/store23/search-archive/b/user/brong/xapian.2


So I haven't compacted to archive for a while.  Let's watch one of
those.  I'm selecting all the tiers, and compressing to a single tier.  The
process is as follows:

1) take an exclusive lock on the xapianactive file
2) insert a new default tier database on the front and unlock xapianactive again
3) start compacting all the selected databases to a single database on the given
tier
4) take an exclusive lock on the xapianactive file again
5) if the xapianactive file has changed, discard all our work (we lock against
this, but it's a sanity check) and exit
6) replace all the source databases for the compact with a reference to the
destination database and unlock xapianactive again
7) delete all the now-unused databases.

Note that the xapianactive file is only locked for two VERY SHORT times.  All
the rest of the time, the compact runs at the side.

This allows us to only ever have a single thread compacting to disk, so our
search drives are mostly idle, and able to serve customer search requests very
quickly.

So here goes:

% time sudo -u cyrus /usr/cyrus/bin/squatter -C /etc/cyrus/imapd-sloti30t01.conf
-v -z archive -t temp,meta,data,archive -u brong
compressing temp:264,archive:2,data:37 to archive:3 for user.brong (active
temp:264,archive:2,data:37)
adding new initial search location temp:265
compacting databases
Compressing messages for brong
done /mnt/i30search/sloti30t01/store23/search-archive/b/user/brong/xapian.3.NEW
renaming tempdir into place
finished compact of user.brong (active temp:265,archive:3)

real	4m52.285s
user	2m29.348s
sys	0m13.948s

% du -s /var/run/cyrus/search-sloti30t01/b/user/brong/*
/mnt/i30search/sloti30t01/store23/search/b/user/brong/*
/mnt/i30search/sloti30t01/store23/search-archive/b/user/brong/*
368	/var/run/cyrus/search-sloti30t01/b/user/brong/xapian.265
du: cannot access `/mnt/i30search/sloti30t01/store23/search/b/user/brong/*':
No such file or directory
4614368	/mnt/i30search/sloti30t01/store23/search-archive/b/user/brong/xapian.3


....

And there we go :)

Bron.


-- 
  Bron Gondwana
  brong at fastmail.fm