Franz Fellner <alpine.art.de at gmail.com> writes:> I also suffer from bad performance of notmuch new. I used notmuch > some years ago and notmuch new always felt instantanious. Had to stop > using it because internet was too slow to sync my mails :/ Now (with > better internet and a completely new setup using mbsync) indexing one > mail takes at least 10 seconds, sometimes even more. It can go into > minutes when I get lots of mail (~30...). When I run it after a > reboot I can have breakfast while notmuch starts up... This is all on > spinning rust. I thought of getting an SSD but not in the near future.I do have at least one spinning rust configuration with about 300k messages, and notmuch new is still fast there.> What I observe during that time: notmuch doesn't really need much CPU. > iotop shows constant read and write with extremely low rates, under > 1MB/sec. So I think it might be an issue in xapian? >Just in case one of the xapian experts can suggest some kind of test for why you might be seeing this behaviour, I've included the xapian list in CC.> If there is anything I can do to help debug this please tell meWhat kind of filesystem do you have on your spinning rust? d
On Mon Apr 20 11:36:36 2020, David Bremner <david at tethera.net> wrote:> What kind of filesystem do you have on your spinning rust?It's ext4 on both HDDs.
On Mon, Apr 20, 2020 at 11:36:36AM -0300, David Bremner wrote:> Franz Fellner <alpine.art.de at gmail.com> writes: > > > I also suffer from bad performance of notmuch new. I used notmuch > > some years ago and notmuch new always felt instantanious. Had to stop > > using it because internet was too slow to sync my mails :/ Now (with > > better internet and a completely new setup using mbsync) indexing one > > mail takes at least 10 seconds, sometimes even more. It can go into > > minutes when I get lots of mail (~30...).First question: what version of Xapian are you using? And second thing to check, are you committing each message separately? The commit operation tries to ensure that the data has actually been written out to disk, so the time to index one message by itself isn't indicative as it'll often mostly just be waiting for fdatasync() or similar to return. If you index 30 messages but commit each separately (i.e. run "notmuch new" 30 times picking up one new message each time) that'll probably scale something like linearly, but indexing a batch of 30 messages should be much quicker per message.> > When I run it after a > > reboot I can have breakfast while notmuch starts up... This is all on > > spinning rust. I thought of getting an SSD but not in the near future.After reboot the disk cache won't have any of the database in, so the first operation will typically be slower, especially with a spinning drive where seeks are relatively slow.> > What I observe during that time: notmuch doesn't really need much CPU. > > iotop shows constant read and write with extremely low rates, under > > 1MB/sec. So I think it might be an issue in xapian? > > Just in case one of the xapian experts can suggest some kind of test for > why you might be seeing this behaviour, I've included the xapian list in > CC.It sounds like you're seek-limited in this "cold cache" phase. That is not necessarily related to the slow indexing, but it could be. I'd check the SMART diagnostics for the drive first (e.g. with smartctl). It's not the most likely cause, but it's quick to check and if the drive is starting to fail it's better to find out sooner rather than later. Then I'd try compacting the database (I think there's a "notmuch compact" subcommand to do this). If that doesn't help, profiling the I/O would probably be my next suggestion - there are some tools in the xapian git repo to help with this (in xapian-maintainer-tools/profiling). Under Linux I'd suggest the strace ones (there's also an LD_PRELOAD library but it may need tweaking for 32 vs 64 bit). Cheers, Olly
On Thu Apr 23 00:21:30 2020, Olly Betts <olly at survex.com> wrote:> First question: what version of Xapian are you using?On my laptop it's 1.4.15 (arch linux) and the desktop runs 1.4.14 (Gentoo linux)> And second thing to check, are you committing each message separately?No, I sync with mbsync which dosnloads a bunch of mails, then I run notmuch new which indexes all in one go.> After reboot the disk cache won't have any of the database in, so the > first operation will typically be slower, especially with a spinning > drive where seeks are relatively slow.Yes, I know that, I just wanted to mention the number, which IMO is insane. I want to setup notmuch for my dad on the desktop PC. 5 minutes to wait for his mail in the morning would have made notmuch a no-go.> It sounds like you're seek-limited in this "cold cache" phase. That is > not necessarily related to the slow indexing, but it could be. > > I'd check the SMART diagnostics for the drive first (e.g. with > smartctl). It's not the most likely cause, but it's quick to check and > if the drive is starting to fail it's better to find out sooner rather > than later.HDDs are healthy. I actually checked quite recently when converting the laptop from Gentoo to arch.> > Then I'd try compacting the database (I think there's a "notmuch > compact" subcommand to do this).And there we go. Cured the issues. Dropped the very first indexing from several minutes to 1.5 seconds on the desktop. ?!?! This is a really new setup and I suffered from bad performance from the very first notmuch new after the initial indexing. Is it really needed to run notmch compact directly after the initial notmuch new? Desktop currently has 38502 messages indexed, in case that matters. Regards Franz
David Bremner <david at tethera.net> wrote:> Franz Fellner <alpine.art.de at gmail.com> writes: > > mail takes at least 10 seconds, sometimes even more. It can go into > > minutes when I get lots of mail (~30...). When I run it after a > > reboot I can have breakfast while notmuch starts up... This is all on > > spinning rust. I thought of getting an SSD but not in the near future. > > I do have at least one spinning rust configuration with about 300k > messages, and notmuch new is still fast there.I've yet to figure out how spinning rust can work well with giant public-inboxes (git + Xapian + SQLite); but I have a fair bit of experience with SSDs + Xapian. But some of my recommendations below come from my experience with HDDs in the old days, before I used Xapian.> > What I observe during that time: notmuch doesn't really need much CPU. > > iotop shows constant read and write with extremely low rates, under > > 1MB/sec. So I think it might be an issue in xapian?Seek times, probably `iostat -x 1' can give you very useful information about I/O queue sizes and wait times for reads and writes (the `-x' is the good stuff :), `1' means it keeps outputting every second.> Just in case one of the xapian experts can suggest some kind of test for > why you might be seeing this behaviour, I've included the xapian list in > CC.Newer Xapian has a DB_NO_SYNC which notmuch could set as an option. Users of old Xapian (or on Perl XS bindings) also have libeatmydata LD_PRELOAD which I end up using all the time: https://www.flamingspork.com/projects/libeatmydata/ I run `sync' if I have anything important, but I usually don't ;) I do set the kernel do flush dirty data in the background fairly aggressively, though (more below) For public-inbox v2 hacking in 2018 (indexing LKML archives, ~3M messages), I found working on a freshly TRIM-ed SSD with plenty of free space made the SSD firmware happier. SSDs can get a LOT slower as they get fuller (so xapian-compact helps, there, too). SSD quality matters a lot; but even the low-end QLC stuff beats high-end HDDs in random I/O; but they will slow down more as they fill up more. For writes, I set /proc/sys/vm/dirty_background_bytes to 100M or something reasonably close to what the SSD can write quickly. Linux tended to hit I/O stalls with lots of dirty data, so making the kernel flush it sooner tends to help IME. Maybe newer kernels do better *shrug*; but it's basically the local storage version of the network "Bufferbloat" problem. Flushing dirty data more frequently also frees up more memory for the kernel to make better caching decisions about future/current data it needs to read. notmuch can probably run a background thread (or use liburing) to do POSIX_FADV_DONTNEED once its done with a message, too (and POSIX_FADV_WILLNEED for to-be-indexed messages). Uncompressed Maildir messages eat cache space real quick, which means less cache for Xapian. public-inbox indexes the v2 inbox format in parallel; but excessive parallelism still causes I/O contention with SSDs (at least upper-mid-range ones). So right now the default limit is 3 indexing processes regardless of CPU count. Reading from git is still synchronous atm, but will probably be async in a few months. git itself tends to generate decent I/O patterns with its pack format (but makes posix_fadvise hinting impractical). Anyways, indexing just under 3 million LKML messages took ~4 hours on 4-core system built in 2010 with a SATA SSD from 2014.