Xapian discuss - Apr 2020 - performance problems with notmuch new

On Thu Apr 23 00:21:30 2020, Olly Betts <olly at survex.com>
wrote:
> First question: what version of Xapian are you using?
On my laptop it's 1.4.15 (arch linux) and the desktop runs 1.4.14 (Gentoo
linux)
> And second thing to check, are you committing each message separately?
No, I sync with mbsync which dosnloads a bunch of mails,
then I run notmuch new which indexes all in one go.
> After reboot the disk cache won't have any of the database in, so the
> first operation will typically be slower, especially with a spinning
> drive where seeks are relatively slow.
Yes, I know that, I just wanted to mention the number, which IMO is insane.
I want to setup notmuch for my dad on the desktop PC.
5 minutes to wait for his mail in the morning would have made notmuch a no-go.
 
> It sounds like you're seek-limited in this "cold cache"
phase.  That is
> not necessarily related to the slow indexing, but it could be.
> 
> I'd check the SMART diagnostics for the drive first (e.g. with
> smartctl).  It's not the most likely cause, but it's quick to check
and
> if the drive is starting to fail it's better to find out sooner rather
> than later.
HDDs are healthy. I actually checked quite recently when converting 
the laptop from Gentoo to arch.
> 
> Then I'd try compacting the database (I think there's a
"notmuch
> compact" subcommand to do this).
And there we go. Cured the issues.
Dropped the very first indexing from several minutes to 1.5 seconds on the
desktop.
?!?!
This is a really new setup and I suffered from bad performance from the
very first notmuch new after the initial indexing.
Is it really needed to run notmch compact directly after the initial notmuch
new?
Desktop currently has 38502 messages indexed, in case that matters.

Regards
Franz

Franz Fellner <alpine.art.de at gmail.com> writes:
>
> On Thu Apr 23 00:21:30 2020, Olly Betts <olly at survex.com> wrote:
>> Then I'd try compacting the database (I think there's a
"notmuch
>> compact" subcommand to do this).
> And there we go. Cured the issues.  Dropped the very first indexing
> from several minutes to 1.5 seconds on the desktop.  ?!?!  This is a
> really new setup and I suffered from bad performance from the very
> first notmuch new after the initial indexing.  Is it really needed to
> run notmch compact directly after the initial notmuch new?  Desktop
> currently has 38502 messages indexed, in case that matters.
That's a fairly small number of messages by the usual notmuch
standards. It's possible there is something pathological about the
interaction of notmuch and your particular set of mail. Running the
performance test suite on your machine would be one way of
telling. Also, the size of your notmuch database on disk; really huge
databases are a sign that something is going wrong. I think most people
find the database is 1/3 to 1/2 the size of the mail it is indexing, but
we have had some odd cases where the database is 2 or 3 times the mail,
and that was an indexing bug.

d