search for: add_database

Olly Betts <olly at survex.com> wrote: > On Sat, Feb 08, 2020 at 06:04:42PM +0000, Eric Wong wrote: > > Olly Betts <olly at survex.com> wrote: > > > On Fri, Feb 07, 2020 at 09:33:08PM +0000, Eric Wong wrote: > > > > Or would I fiddle with wdf_inc for all ->index_text and ->add_term > > > > calls on a per-DB basis? > > > >

Sorry if this is the wrong forum to discuss Search::Xapian issues -- this just seems like the best place.. Anyways, I've been testing out using $db->add_database() when searching, and it seems like the docids I'm getting out of it are incorrect, almost as though they're "double" what they should be (numerically)... the docids that exist should be around 950,000 and 1000000 not around 1900000, etc... $xapiandirbase . '-11' and...

...an::MSetIterator i; i.get_document().get_data(); but I was happened to discover that the dictionaries got by such two method were different: both methods use DB1, DB2 method 1: Xapian::Database db = Xapian::Database(the path of DB1); Xapian::Database db2 = Xapian::Database(the path of DB2); db.add_database(db2); I pre-load the docid and its document data into dictionary DT1; method 2: DB1 and DB2 are opened by Xapian-tcpsrv for remote access. Xapian::Database dbr = Xapian::Remote::open(host of DB1, port of DB1, 0, 0); Xapian::Database dbr2 = Xapian::Remote::open(host of DB2, port of DB2, 0, 0);...

[Replying to xapian-devel, as I think a wider audience would be useful] On Mon, Oct 21, 2013 at 11:24:51PM +0800, jiangwen jiang wrote: > yes, it's less efficient. Lucene database has multiple segments, each > segment can treat as a independent database. The same term may exists in >= > 1 segments. Sorry for taking a while to respond - I've been both busy and mulling this

Hey all, I've been using ->add_database for a few years to tie sharded DBs together and it works great. Now, I want to be able to search across several DBs which aren't sharded, say: linux-DB, glibc-DB, freebsd-DB. I want to search for something across all of them, but prioritize results to favor one or some of those DBs over other...

IRIX 6.5.22, compiled omega.0.8.1 with native MIPS compiler. omega.cgi isn't running from apache. From the log: [Wed Sep 1 10:04:36 2004] [error] [client 198.149.32.31] malformed header from script. Bad header=Caught unknown exception: /var/www/cgi- bin/omega.cgi This is with a url of http://<machine>/cgi-bin/omega.cgi?DB=/open/ When I run it from the command line with no

Hi, I'm newbie in xapian. I just use xapian for a few week ago and I would like to know: How I can search in many database at once time? Please send some answer to me. p.s. Sorry about my english. Regrad Mr.T _________________________________________________________________ Hotmail: ??????????????????????????????????????????????????????????????????????

...re out which sub-DB a retrieved > document or mset item belongs to? Yes: https://trac.xapian.org/wiki/FAQ/MultiDatabaseDocumentID 1.4.12 added a Database::size() method which reports the number of shards - for older versions you have to keep track of that yourself (which needs a little care as add_database() "flattens", so after db1.add_database(db2) db1.size() will have increased by db2.size() (which could be 0 if db2 has no shards). Cheers, Olly

...; > document or mset item belongs to? > > Yes: https://trac.xapian.org/wiki/FAQ/MultiDatabaseDocumentID > > 1.4.12 added a Database::size() method which reports the number of > shards - for older versions you have to keep track of that yourself > (which needs a little care as add_database() "flattens", so after > db1.add_database(db2) db1.size() will have increased by db2.size() > (which could be 0 if db2 has no shards). Thanks, I'll have to use the older ways... Btw, is the XS Perl binding going to be updated for Database::size()? Or is the SWIG binding the on...

On Fri, Feb 07, 2020 at 09:33:08PM +0000, Eric Wong wrote: > Hey all, I've been using ->add_database for a few years > to tie sharded DBs together and it works great. > > Now, I want to be able to search across several DBs > which aren't sharded, say: linux-DB, glibc-DB, freebsd-DB. > > I want to search for something across all of them, but > prioritize results to favor...

...0, I've got 390 Xapian shards for 130 public inboxes > I want to search against(*). There's more on the horizon (we're > expecting tens of thousands of public inboxes). Was that "(*)" meant to have a matching footnote? > After bumping RLIMIT_NOFILE and running ->add_database a bunch, > the actual queries seem to be taking ~30s (not good :x). > > Now I'm thinking, MultiDatabase isn't the right way to go about > this... I'm not aware of anyone who's tried to use that many shards before, so it might be you're just hitting something easy t...

...r(const char *dest, const char **sources, int (*cb)(const char *cyrusid, void *rock), void *rock) { int r = 0; int count = 0; try { /* set up a cursor to read from all the source databases */ Xapian::Database *srcdb = new Xapian::Database(); while (*sources) { srcdb->add_database(Xapian::Database(*sources++)); } Xapian::Enquire enquire(*srcdb); enquire.set_query(Xapian::Query::MatchAll); Xapian::MSet matches = enquire.get_mset(0, srcdb->get_doccount()); /* create a destination database */ Xapian::WritableDatabase *destdb = new Xapian::WritableDatabase(dest, Xapian...

...need to run a match just to be able to iterate all the > [...] > There's no need to use transactions to do this - outside of > [...] v2: try { /* set up a cursor to read from all the source databases */ Xapian::Database srcdb = Xapian::Database(); while (*sources) { srcdb.add_database(Xapian::Database(*sources++)); } /* create a destination database */ Xapian::WritableDatabase destdb = Xapian::WritableDatabase(dest, Xapian::DB_CREATE); /* copy all matching documents to the new DB */ Xapian::PostingIterator it; for (it = srcdb.postlist_begin(""); it != srcdb.pos...

Olly Betts <olly at survex.com> wrote: > On Fri, Feb 07, 2020 at 09:33:08PM +0000, Eric Wong wrote: > > Hey all, I've been using ->add_database for a few years > > to tie sharded DBs together and it works great. > > > > Now, I want to be able to search across several DBs > > which aren't sharded, say: linux-DB, glibc-DB, freebsd-DB. > > > > I want to search for something across all of them, but &gt...

...ttp://xapian.org/docs/overview.html#specifying-a-database (talks > about stub databases, which you'll want to use with the remote > backend) Also the API doc for database: http://xapian.org/docs/apidoc/html/classXapian_1_1Database.html#2fc5aa368a7097ca787a831211a2e3bc Just call add_database for each db you have, and then a search will work over all of them (I think) -- E: sym.roe at talusdesign.co.uk<mailto:sym.roe at talusdesign.co.uk> M: 07742079314 @symroe

...back to the "prioritizing aggregated DBs" thread from February 2020, I've got 390 Xapian shards for 130 public inboxes I want to search against(*). There's more on the horizon (we're expecting tens of thousands of public inboxes). After bumping RLIMIT_NOFILE and running ->add_database a bunch, the actual queries seem to be taking ~30s (not good :x). Now I'm thinking, MultiDatabase isn't the right way to go about this... Perhaps creating a new, all-encompassing Xapian index with a reasonable shard count would be wise, at least for the normal WWW frontend? Managing remo...

Hi, While searching for a project which matches my interest andskill level, I found this project named Matcher Optimization. This project is really challenging and excting from my view point and I would like to be a part of this project. Optimization techniques metioned in the reference links provided will take some time for me to have a good understanding about them. But I am trying to get my

...name,char* results, int msize) { long docid; char* op; char fullDB[1024]; string queryString; sprintf(fullDB,"/var/lib/fulltext/%s",dbname); queryString=qs; try { Database db; db.add_database(Database(fullDB)); Enquire enquire(db); try { Xapian::QueryParser qp; Xapian::Stem stemmer("english"); qp.set_database(db); qp.set_default_op(Query::OP_OR); qp.set_stemmer(stemmer);...

...index, tid, NUM_THREADS, documents[tid],&dbs[tid])); } for (auto& th : threads) { th.join(); } infs.close(); Xapian::Database final_db; for(const Xapian::WritableDatabase &d : dbs) { final_db.add_database(d); } Xapian::Enquire enquire(final_db); Xapian::QueryParser parser; end = high_resolution_clock::now(); auto diff1 = duration_cast<milliseconds>(end - start).count(); cout << "Indexing time: " << diff1 << "ms" << endl; // s...

I've some trouble with my indexer, which builds on simpleindex.cc. The problem is that indexing process becomes very slow after we indexed 2000k docs (though the indexer works quite well with first 2000k docs). It took almost three weeks to index 8 million docs. However, we need to index about 20 million docs. I have to stop the indexer due to its performance. I think my question is