Ferret talk - Jun 2007 - another issue with highlighting

Hi,

I''m encountering another highlighting issue.
(about the first one "range search and highlighting", i received no
response. I don''t even know if somebody tried to reproduce and/or if
it''s normal behavior)

about the new issue, an example will be easier for you to reproduce:
I''m filling an index with random data, i try to match for
"*1*" and
then highlight the matched tokens. If it''s matched and not highlighted
i put it in z

It works as expected when there''re 100 entries (replace 500.times by
100.times), in that case z contains empty arrays.
When having 500 entries it doesn''t highlight every matches !

This example has been tested with 0.11.4
(r770 has been tested with the application i first discovered this
issue with)

I would appreciate if you could test and tell me if I''m the only one
having this problem.
TIA


<example>
require ''ferret''
include Ferret

# filling
index=Index::Index.new(:path => ''/tmp/test'')
chars1 = chars2 = chars3 = chars4 = ("a".."z").to_a +
("0".."9").to_a
chars2.concat(["-", "_", " "])
chars3 << " "
chars4 << "-"
chars5 = chars6 = ("0".."9").to_a
chars6 << "."
500.times do
 z={}
 t=""
 1.upto(15+rand(10)) { |i| t << chars4[rand(chars4.size-1)] }
 z[:un] = t
 t=""
 1.upto(40+rand(40)) { |i| t << chars2[rand(chars2.size-1)] }
 z[:deux] = t
 t=""
 1.upto(30+rand(10)) { |i| t << chars4[rand(chars4.size-1)] }
 z[:trois] = t
 t=""
 1.upto(30+rand(10)) { |i| t << chars1[rand(chars1.size-1)] }
 z[:quatre] = t
 t=""
 1.upto(30+rand(10)) { |i| t << chars2[rand(chars2.size-1)] }
 z[:cinq] = t
 t=""
 1.upto(12) { |i| t << chars5[rand(chars5.size-1)] }
 z[:six] = t
 t=""
 1.upto(12) { |i| t << chars6[rand(chars6.size-1)] }
 z[:sept] = t
 t=""
 1.upto(12) { |i| t << chars6[rand(chars6.size-1)] }
 z[:huit] = t
 t=""
 1.upto(24+rand(24)) { |i| t << chars3[rand(chars3.size-1)] }
 z[:neuf] = t
 t=""
 1.upto(100+rand(100)) { |i| t << chars2[rand(chars2.size-1)] }
 z[:dix] = t
 index << z
end

#testing
q="*1*"
z={}
index.search_each(q,:limit => :all) do |id,score|
for b in [:un, :deux, :trois, :quatre, :cinq, :six, :sept, :huit, :neuf, :dix]
z[b]=[] if not z[b]
z[b] << id.to_s + " : " + index.highlight(q,id,:field => b,
:pre_tag => "<em>", :post_tag => "</em>",
:num_excerpts => :all, :excerpt_length => :all).join(" | ") if
index[id][b].match(/1/) and index.highlight(q,id,:field => b, :pre_tag =>
"<em>", :post_tag => "</em>", :num_excerpts
=> :all, :excerpt_length => :all) and not index.highlight(q,id,:field
=> b, :pre_tag => "<em>", :post_tag =>
"</em>", :num_excerpts => :all, :excerpt_length =>
:all).join(" | ").match(/<em>/)
end
end
z
index.search("*",:limit => :all).total_hits
</example>


--

syrius.ml at no-log.org writes:
> Hi,
>
> I''m encountering another highlighting issue.
> (about the first one "range search and highlighting", i received
no
> response. I don''t even know if somebody tried to reproduce and/or
if
> it''s normal behavior)
>
> about the new issue, an example will be easier for you to reproduce:
> I''m filling an index with random data, i try to match for
"*1*" and
> then highlight the matched tokens. If it''s matched and not
highlighted
> i put it in z
>
> It works as expected when there''re 100 entries (replace 500.times
by
> 100.times), in that case z contains empty arrays.
> When having 500 entries it doesn''t highlight every matches !
>
> This example has been tested with 0.11.4
> (r770 has been tested with the application i first discovered this
> issue with)
>
> I would appreciate if you could test and tell me if I''m the only
one
> having this problem.
> TIA
>
>
> <example>
> require ''ferret''
> include Ferret
>
> # filling
> index=Index::Index.new(:path => ''/tmp/test'')
> chars1 = chars2 = chars3 = chars4 = ("a".."z").to_a +
("0".."9").to_a
> chars2.concat(["-", "_", " "])
> chars3 << " "
> chars4 << "-"
> chars5 = chars6 = ("0".."9").to_a
> chars6 << "."
> 500.times do
>  z={}
>  t=""
>  1.upto(15+rand(10)) { |i| t << chars4[rand(chars4.size-1)] }
>  z[:un] = t
>  t=""
>  1.upto(40+rand(40)) { |i| t << chars2[rand(chars2.size-1)] }
>  z[:deux] = t
>  t=""
>  1.upto(30+rand(10)) { |i| t << chars4[rand(chars4.size-1)] }
>  z[:trois] = t
>  t=""
>  1.upto(30+rand(10)) { |i| t << chars1[rand(chars1.size-1)] }
>  z[:quatre] = t
>  t=""
>  1.upto(30+rand(10)) { |i| t << chars2[rand(chars2.size-1)] }
>  z[:cinq] = t
>  t=""
>  1.upto(12) { |i| t << chars5[rand(chars5.size-1)] }
>  z[:six] = t
>  t=""
>  1.upto(12) { |i| t << chars6[rand(chars6.size-1)] }
>  z[:sept] = t
>  t=""
>  1.upto(12) { |i| t << chars6[rand(chars6.size-1)] }
>  z[:huit] = t
>  t=""
>  1.upto(24+rand(24)) { |i| t << chars3[rand(chars3.size-1)] }
>  z[:neuf] = t
>  t=""
>  1.upto(100+rand(100)) { |i| t << chars2[rand(chars2.size-1)] }
>  z[:dix] = t
>  index << z
> end
>
> #testing
> q="*1*"
> z={}
> index.search_each(q,:limit => :all) do |id,score|
> for b in [:un, :deux, :trois, :quatre, :cinq, :six, :sept, :huit, :neuf,
:dix]
> z[b]=[] if not z[b]
> z[b] << id.to_s + " : " + index.highlight(q,id,:field =>
b, :pre_tag => "<em>", :post_tag =>
"</em>", :num_excerpts => :all, :excerpt_length =>
:all).join(" | ") if index[id][b].match(/1/) and
index.highlight(q,id,:field => b, :pre_tag => "<em>",
:post_tag => "</em>", :num_excerpts => :all,
:excerpt_length => :all) and not index.highlight(q,id,:field => b,
:pre_tag => "<em>", :post_tag => "</em>",
:num_excerpts => :all, :excerpt_length => :all).join(" |
").match(/<em>/)
> end
> end
> z
> index.search("*",:limit => :all).total_hits
> </example>

Hi, 

Could somebody mind trying to reproduce this please ?

Highlighting is a very important feature for me, I need to know if I''m
doing something wrong or if it''s a dirty bug.

Thanks in advance


--

On Wed, Jun 20, 2007 at 12:30:03AM +0200, syrius.ml at no-log.org
wrote:
> 
> Hi,
> 
> I''m encountering another highlighting issue.
> (about the first one "range search and highlighting", i received
no
> response. I don''t even know if somebody tried to reproduce and/or
if
> it''s normal behavior)
> 
> about the new issue, an example will be easier for you to reproduce:
> I''m filling an index with random data, i try to match for
"*1*" and
> then highlight the matched tokens. If it''s matched and not
highlighted
> i put it in z
> 
> It works as expected when there''re 100 entries (replace 500.times
by
> 100.times), in that case z contains empty arrays.
> When having 500 entries it doesn''t highlight every matches !
> 
> This example has been tested with 0.11.4
> (r770 has been tested with the application i first discovered this
> issue with)
> 
> I would appreciate if you could test and tell me if I''m the only
one
> having this problem.
Here z is not empty in the 500 case, too.  Strange behaviour indeed,
imho highlight should either return strings with some highlighted
content, or nothing at all when there''s nothing to highlight...

Jens

 

-- 
Jens Kr?mer
webit! Gesellschaft f?r neue Medien mbH
Schnorrstra?e 76 | 01069 Dresden
Telefon +49 351 46766-0 | Telefax +49 351 46766-66
kraemer at webit.de | www.webit.de
 
Amtsgericht Dresden | HRB 15422
GF Sven Haubold, Hagen Malessa

Jens Kraemer <kraemer at webit.de> writes:
> Here z is not empty in the 500 case, too.  Strange behaviour indeed,
> imho highlight should either return strings with some highlighted
> content, or nothing at all when there''s nothing to highlight...
Hi,

Thanks Jens for your answer.
hmm since content has to be stored the default behavior doesn''t
surprise me.
But the issue does ! :)
I''m looking forward to read Dave about this :)


--