search for: multidatabase

...orizon (we're > expecting tens of thousands of public inboxes). Was that "(*)" meant to have a matching footnote? > After bumping RLIMIT_NOFILE and running ->add_database a bunch, > the actual queries seem to be taking ~30s (not good :x). > > Now I'm thinking, MultiDatabase isn't the right way to go about > this... I'm not aware of anyone who's tried to use that many shards before, so it might be you're just hitting something easy to address. Anything to do with shards should be at worst O(n) in the number of shards (and it's often O(1)), but...

On Sun, Aug 23, 2020 at 11:12:39PM +0000, Eric Wong wrote: > Olly Betts <olly at survex.com> wrote: > > If most (or at least a significant amount) is CPU time then it would > > be useful to profile to see if there are any low-hanging fruit. I've > > been mostly using the profiler in gperftools lately if you want to > > try this and don't know what to use.

On Mon, Aug 24, 2020 at 05:58:02AM +0000, Eric Wong wrote: > Olly Betts <olly at survex.com> wrote: > > Can prof report time for a function including things it calls? > > callgraph? Attached is a profile the output of "perf report -g" > with callgraph info. I'm no perf expert, either, but slowly > learning more as I go along... The list seems to have

On Tue, Aug 25, 2020 at 10:15:42PM +0000, Eric Wong wrote: > So I managed to get current xapian.git (commit 61724d477edb) > built with CXXFLAGS=-ggdb3, and it's closer to 100%: > > https://80x24.org/spew/20200825215517.GA3936 at dcvr/2-perf-report-20200825-214820.gz > > The machine I'm working on is also significantly busier at the > moment trying to reproduce an

Olly Betts <olly at survex.com> wrote: > On Mon, Aug 24, 2020 at 05:58:02AM +0000, Eric Wong wrote: > > Olly Betts <olly at survex.com> wrote: > > > Can prof report time for a function including things it calls? > > > > callgraph? Attached is a profile the output of "perf report -g" > > with callgraph info. I'm no perf expert,

Olly Betts <olly at survex.com> wrote: > The report you show seems to be just the time take by each function > directly rather than including functions it calls. It looks like a lot > of the time is spent in cursor movement, as totalling up things that > seem like they'd be due to that I quickly get to 40+% but it's hard > to tell if that's about the actual total

I have been hearing a lot about workshops and/or groups of Ruby on Rails users meeting up in numerous cities, but I haven''t been too successful finding any such gatherings or events in Boston. On the wiki there is a link to a Boston group for Ruby, but the page refuses to load. Anyone from the Boston area know of anything? Cheers, Eric Czarny eczarny@stonehill.edu

...ction than another, this could correspond to it being more significant when it occurs in the collection in which it is less frequent.) At some point it would be nice to add the ability to have a mapping from "human-readable field name" to "prefix code" inside xapian, so the multidatabase stuff could be aware of this issue and generate the prefixes correctly for each database. However, that's not urgent, and not what I'm thinking about right now. It would also be nice to have a "virtual" posting list, which effectively returned a list of all the document IDs i...

...shards for 130 public inboxes I want to search against(*). There's more on the horizon (we're expecting tens of thousands of public inboxes). After bumping RLIMIT_NOFILE and running ->add_database a bunch, the actual queries seem to be taking ~30s (not good :x). Now I'm thinking, MultiDatabase isn't the right way to go about this... Perhaps creating a new, all-encompassing Xapian index with a reasonable shard count would be wise, at least for the normal WWW frontend? Managing removals of entire inboxes from an all-encompassing Xapian DB would get much trickier. IMAP search would s...

...meant to have a matching footnote? Oops, yes, it's a mirror of all the inboxes on https://lore.kernel.org/ > > After bumping RLIMIT_NOFILE and running ->add_database a bunch, > > the actual queries seem to be taking ~30s (not good :x). > > > > Now I'm thinking, MultiDatabase isn't the right way to go about > > this... > > I'm not aware of anyone who's tried to use that many shards before, so > it might be you're just hitting something easy to address. Anything to > do with shards should be at worst O(n) in the number of shards (and i...