Eric Parusel
2004-Dec-21 21:33 UTC
[Xapian-discuss] Search::Xapian add_database'd search results are odd?
Sorry if this is the wrong forum to discuss Search::Xapian issues --
this just seems like the best place..
Anyways, I've been testing out using $db->add_database() when searching,
and it seems like the docids I'm getting out of it are incorrect, almost
as though they're "double" what they should be (numerically)...
the docids that exist should be around 950,000 and 1000000 not around
1900000, etc...
$xapiandirbase . '-11'
and
$xapiandirbase . '-10' both exist.
quick example:
==================================================================my $db =
Search::Xapian::Database->new($xapiandirbase . '-11' ||
die("Error.\n"));
my $db->add_database(Search::Xapian::Database->new($ARGV[0] .
'-10' ||
die("Error.\n")));
my $query = Search::Xapian::Query->new(OP_AND, 'word',
'word2');
print "Query: " . $query . "\n";
my $enq = $db->enquire($query);
my @matches = $enq->matches(0,1000000);
foreach my $match ( @matches ) {
print $match->get_docid . ',';
}
==================================================================
If I don't use add_database, or add_database is wrapped in eval and
fails due to me pointing to a xapian db that doesn't exist, and either
query xapian db "-10" or "-11", I get docids returned in the
900000-1000000 range approximately.
If I use add_database() in either order (-10 then -11, or vice versa)
then I get ids seemingly doubled.
-10 and -11 have unique markers in them (M10 and M11 respectively), so
if I use one of the markers as the keywords, I get the same number of
results whether I add on the 2nd database or not.
I just get results that are seemingly approximately doubled?
I'm running Search::Xapian 0.8.4.0 and xapian 0.8.4:
# rpm -qa | grep xapian
xapian-core-0.8.4-1
xapian-core-libs-0.8.4-1
xapian-core-debuginfo-0.8.4-1
xapian-core-devel-0.8.4-1
Thanks for any help you can offer,
Eric
-------------- next part --------------
xapian-read-test.pl in original form, reading domain.com-2004-11, no
add_database:
# ./xapian-read-test.pl /data1/xapian/domain.com-2004 M11 warren shannon
Query: Xapian::Query((M11 AND warren AND shannon))
Parsing query 'Xapian::Query((M11 AND warren AND shannon))'
611 results found
{975786,975788,976495,976496,976603,976604,976605,976653,976654,976655,976656,977247,978602,978603,978604,982193,982194,982195,982196,982237,982883,983505,983521,983522,983538,983539,983540,984358,984366,984367,984768,984769,984782,984924,987743,988632,988633,988634,988637,988638,988639,990152,990864,990865,990866,993950,993959,996317,996318,996319,996429,996430,996431,996503,996504,996505,996594,996595,996596,996981,1000204,1000205,1000215,1000286,1000287,1000288,1000340,1000382,1000433,1000434,1000435,1000867,1000868,1000869,977244,977245,977246,990160,975793,982927,982928,982929,994283,994284,994285,983865,983866,983867,988662,988663,988664,983762,983763,983764,977138,994079,977164,977165,977166,977167,984994,984995,984996,1001216,1001217,1001218,996426,984305,984306,984307,984352,984353,984354,996397,1000197,982933,982934,982935,1000211,984361,984362,984363,978573,989961,989962,989963,989322,994093,997031,984341,984342,983550,983551,983552,983553,989326,989327,989328,978580,983565,984826,984827,984828,994825,994826,994827,984990,984991,984992,1001031,983883,983884,983885,997075,997076,997077,984372,984373,984374,978618,996955,996956,996957,975801,975932,978553,978554,978555,982512,984902,984903,1000267,1000268,1000269,976616,976617,983819,983820,983821,989511,989512,989513,996964,996965,996966,984315,984843,984844,984845,987788,987789,987790,989466,989467,989468,977965,1000377,978000,978001,978002,978715,978716,978717,978718,978719,978720,978721,978722,978723,984864,984865,984866,977925,982923,982924,982925,995436,995437,995438,988400,988401,988402,990791,990792,976607,976608,976609,983004,995442,996973,996974,996975,984927,984928,984931,991434,991435,991436,984926,989351,989352,989353,983104,984929,984919,977517,978703,978704,978705,988537,988538,988539,993738,993739,984870,984871,984872,988783,988784,988785,989340,989341,989342,982930,982931,982932,984950,984951,984471,996214,996215,996216,996223,996224,996225,996219,996220,996221,996279,996280,996281,993758,993759,993760,993955,993956,993957,977858,977859,977860,994206,990079,990080,990081,990082,1000912,984887,984888,984889,1000929,983298,984487,984488,984489,989313,989314,989315,977864,977865,977866,1001133,975809,975810,975811,984777,984778,984779,978812,996978,975830,977936,995465,975828,978003,978004,978005,978817,988677,978825,973971,988680,975779,993839,993840,993841,983027,983028,983029,983030,977872,993860,993862,978828,990616,975822,978031,978874,978875,978876,978837,978838,978839,987814,987815,987816,988682,977206,983609,988689,988690,988691,977526,1000880,1000881,1000882,978801,983617,983618,983619,983620,990166,990167,990168,978808,978809,978810,987874,987875,987876,990170,990171,990172,978822,978823,978824,983640,990656,975974,983592,983593,983594,983595,983683,983684,983685,1000233,1000234,1000235,995500,995501,995502,995503,988613,988614,988615,996038,978831,978832,978833,987725,987726,990173,990174,990175,984878,984879,984880,984881,999531,988081,988082,988083,975829,989589,989590,989591,975676,975677,975678,975790,975791,975792,990176,996688,976048,976049,976050,989443,989444,989445,978853,987745,987746,987747,1000221,1000222,1000223,982164,982165,982166,983622,983623,983624,984144,1000650,985013,985014,985015,990408,995177,998011,984632,985018,985019,985020,982167,975883,991023,979094,985021,985022,985023,982187,982188,982189,978305,988266,982213,982214,982215,978007,978008,978009,1001188,1001189,1001190,978527,978528,978529,994592,994593,994594,990562,985184,986102,978532,978533,978534,988549,978624,978625,978626,988563,982299,982300,982301,987888,987889,989400,989401,989402,994539,994540,994541,1000225,1000226,1000227,988672,988673,988674,991809,991810,991811,976096,976097,976098,976103,976104,976105,976107,976108,976109,984382,984383,984384,975910,975911,975912,1000900,975918,975919,975920,975938,975939,975940,977326,990031,990032,990033,1001446,987948,990083,990084,990085,990182,990183,990184,990189,990190,990191,990199,990200,990201,1001255,990207,990208,990209,990090,990091,990092,990194,990195,990196,990692,990216,983571,983572,983573,1000361,1000362,976759,976760,976761,993181,995821,995822,995824,975453,987530,981892,995888,995889,995890,996415,996416,996417,978841,978842,978843,996442,996443,996444,978863,978864,978865,995893,995894,995895,995903,995904,995905}
Elapsed time: 0.0667 seconds.
# vi xapian-read-test.pl
xapian-read-test.pl modified to add-database domain.com-2004-10:
# ./xapian-read-test.pl /data1/xapian/domain.com-2004 M11 warren shannon
Query: Xapian::Query((M11 AND warren AND shannon))
Parsing query 'Xapian::Query((M11 AND warren AND shannon))'
611 results found
{1951571,1951575,1951585,1952989,1952991,1953205,1953207,1953209,1953305,1953307,1953309,1953311,1954487,1954489,1954491,1954493,1957203,1957205,1957207,1964385,1964387,1964389,1964391,1964473,1965765,1965853,1965855,1965857,1967009,1967041,1967043,1967075,1967077,1967079,1968715,1968731,1968733,1969535,1969537,1969563,1969847,1975485,1977263,1977265,1977267,1977273,1977275,1977277,1980303,1980319,1981727,1981729,1981731,1987899,1987917,1992633,1992635,1992637,1992857,1992859,1992861,1993005,1993007,1993009,1993187,1993189,1993191,1993961,2000407,2000409,2000429,2000571,2000573,2000575,2000679,2000763,2000865,2000867,2000869,2001733,2001735,2001737,1988565,1988567,1988569,1967729,1967731,1967733,1977323,1977325,1977327,1967523,1967525,1967527,1954275,1988157,1954327,1954329,1954331,1954333,1969987,1969989,1969991,2002431,2002433,2002435,1992851,1968609,1968611,1968613,1968703,1968705,1968707,1992793,2000393,1965865,1965867,1965869,2000421,1968721,1968723,1968725,1957145,1979921,1979923,1979925,1978643,1988185,1994061,1968681,1968683,1967099,1967101,1967103,1967105,1978651,1978653,1978655,1957159,1967129,1969651,1969653,1969655,1989649,1989651,1989653,1969979,1969981,1969983,2002061,1967765,1967767,1967769,1994149,1994151,1994153,1968743,1968745,1968747,1957235,1993909,1993911,1993913,1951601,1951863,1957105,1957107,1957109,1965023,1969803,1969805,2000533,2000535,2000537,1953231,1953233,1967637,1967639,1967641,1979021,1979023,1979025,1993927,1993929,1993931,1968629,1969685,1969687,1969689,1975575,1975577,1975579,1978931,1978933,1978935,1955929,2000753,1955999,1956001,1956003,1957429,1957431,1957433,1957435,1957437,1957439,1957441,1957443,1957445,1969727,1969729,1969731,1955849,1965845,1965847,1965849,1990871,1990873,1990875,1976799,1976801,1976803,1981581,1981583,1953213,1953215,1953217,1966007,1990883,1993945,1993947,1993949,1969853,1969855,1969861,1982867,1982869,1982871,1969851,1978701,1978703,1978705,1966207,1969857,1969837,1955033,1957405,1957407,1957409,1977073,1977075,1977077,1987475,1987477,1969739,1969741,1969743,1977565,1977567,1977569,1978679,1978681,1978683,1965859,1965861,1965863,1969899,1969901,1968941,1992427,1992429,1992431,1992445,1992447,1992449,1992437,1992439,1992441,1992557,1992559,1992561,1987515,1987517,1987519,1987909,1987911,1987913,1955715,1955717,1955719,1988411,1980157,1980159,1980161,1980163,2001823,1969773,1969775,1969777,2001857,1966595,1968973,1968975,1968977,1978625,1978627,1978629,1955727,1955729,1955731,2002265,1951617,1951619,1951621,1969553,1969555,1969557,1957623,1993955,1951659,1955871,1990929,1951655,1956005,1956007,1956009,1957633,1977353,1957649,1947941,1977359,1951557,1987677,1987679,1987681,1966053,1966055,1966057,1966059,1955743,1987719,1987723,1957655,1981231,1951643,1956061,1957747,1957749,1957751,1957673,1957675,1957677,1975627,1975629,1975631,1977363,1954411,1967217,1977377,1977379,1977381,1955051,2001759,2001761,2001763,1957601,1967233,1967235,1967237,1967239,1980331,1980333,1980335,1957615,1957617,1957619,1975747,1975749,1975751,1980339,1980341,1980343,1957643,1957645,1957647,1967279,1981311,1951947,1967183,1967185,1967187,1967189,1967365,1967367,1967369,2000465,2000467,2000469,1990999,1991001,1991003,1991005,1977225,1977227,1977229,1992075,1957661,1957663,1957665,1975449,1975451,1980345,1980347,1980349,1969755,1969757,1969759,1969761,1999061,1976161,1976163,1976165,1951657,1979177,1979179,1979181,1951351,1951353,1951355,1951579,1951581,1951583,1980351,1993375,1952095,1952097,1952099,1978885,1978887,1978889,1957705,1975489,1975491,1975493,2000441,2000443,2000445,1964327,1964329,1964331,1967243,1967245,1967247,1968287,2001299,1970025,1970027,1970029,1980815,1990353,1996021,1969263,1970035,1970037,1970039,1964333,1951765,1982045,1958187,1970041,1970043,1970045,1964373,1964375,1964377,1956609,1976531,1964425,1964427,1964429,1956013,1956015,1956017,2002375,2002377,2002379,1957053,1957055,1957057,1989183,1989185,1989187,1981123,1970367,1972203,1957063,1957065,1957067,1977097,1957247,1957249,1957251,1977125,1964597,1964599,1964601,1975775,1975777,1978799,1978801,1978803,1989077,1989079,1989081,2000449,2000451,2000453,1977343,1977345,1977347,1983617,1983619,1983621,1952191,1952193,1952195,1952205,1952207,1952209,1952213,1952215,1952217,1968763,1968765,1968767,1951819,1951821,1951823,2001799,1951835,1951837,1951839,1951875,1951877,1951879,1954651,1980061,1980063,1980065,2002891,1975895,1980165,1980167,1980169,1980363,1980365,1980367,1980377,1980379,1980381,1980397,1980399,1980401,2002509,1980413,1980415,1980417,1980179,1980181,1980183,1980387,1980389,1980391,1981383,1980431,1967141,1967143,1967145,2000721,2000723,1953517,1953519,1953521,1986361,1991641,1991643,1991647,1950905,1975059,1963783,1991775,1991777,1991779,1992829,1992831,1992833,1957681,1957683,1957685,1992883,1992885,1992887,1957725,1957727,1957729,1991785,1991787,1991789,1991805,1991807,1991809}
Elapsed time: 0.0847 seconds.
Olly Betts
2004-Dec-21 21:56 UTC
[Xapian-discuss] Search::Xapian add_database'd search results are odd?
On Tue, Dec 21, 2004 at 01:32:13PM -0800, Eric Parusel wrote:> Anyways, I've been testing out using $db->add_database() when searching, > and it seems like the docids I'm getting out of it are incorrect, almost > as though they're "double" what they should be (numerically)... > > the docids that exist should be around 950,000 and 1000000 not around > 1900000, etc...If you search over more than one database, the docids in the underlying databases are mapped to avoid collisions. The mapping is (at least currently): did_merged = (did_raw - 1) * number_of_databases + offset where offset ranges from 1 to number_of_databases. You can generally just treat did_merged as an opaque value and use it with the combined database to retrieve the appropriate document. Cheers, Olly