Xapian discuss - Dec 2004 - term iterators

Hi!

I need to iterate over pairs of terms to compute the term correlation
matrix. My first attempt was:

for(term1 = db.allterms_begin(); term1 != db.allterms_end(); ++term1){
 for(term2 = term1 + 1; term2 != db.allterms_end(); ++term2){	
	...
	}
  }

this doesn't work because term1 + 1 is not defined, so I did

for(term1 = db.allterms_begin(); term1 != db.allterms_end(); ++term1){
term2 = term1;
++term2;
 for(; term2 != db.allterms_end(); ++term2){	
	...
	}
  }

and to my surprise, incrementing term2 incremented as well term1. Is
this what is really intended? Finally, I solved the problem with

for(term1 = db.allterms_begin(); term1 != db.allterms_end(); ++term1){
      term2 = db.allterms_begin();    
      term2.skip_to(*term1);
      if(term2 == db.allterms_end()){
	cerr << "term2 end of list while term1 is '" <<
*term1 << "'\n";
	exit(1);
      }
      else
	++term2;
      for(; term2 != db.allterms_end(); ++term2){	
	...
	}
  }

Note that if term2 is not set to db.allterms_begin(), the code crashes.

Is there a more elegant way to iterate over pairs of terms?

Regards, and thanks for this nice library and all the work invested.

Georges Dupret

On Tue, Dec 14, 2004 at 12:36:53PM -0300, Georges Dupret
wrote:
> I need to iterate over pairs of terms to compute the term correlation
> matrix. My first attempt was:
> 
> for(term1 = db.allterms_begin(); term1 != db.allterms_end(); ++term1){
>  for(term2 = term1 + 1; term2 != db.allterms_end(); ++term2){	
> 	...
> 	}
>   }
> 
> this doesn't work because term1 + 1 is not defined, so I did
> 
> for(term1 = db.allterms_begin(); term1 != db.allterms_end(); ++term1){
> term2 = term1;
> ++term2;
>  for(; term2 != db.allterms_end(); ++term2){	
> 	...
> 	}
>   }
> 
> and to my surprise, incrementing term2 incremented as well term1. Is
> this what is really intended?
Yes.  TermIterators have the semantics of STL input iterators.  If you
copy and increment, using the old iterator gives undefined behaviour
(at present I believe you'll always get both incremented, but that
might change in future).
>Finally, I solved the problem with
> 
> for(term1 = db.allterms_begin(); term1 != db.allterms_end(); ++term1){
>       term2 = db.allterms_begin();    
>       term2.skip_to(*term1);
>       if(term2 == db.allterms_end()){
> 	cerr << "term2 end of list while term1 is '" <<
*term1 << "'\n";
> 	exit(1);
>       }
>       else
> 	++term2;
>       for(; term2 != db.allterms_end(); ++term2){	
> 	...
> 	}
>   }
That looks about right.  Perhaps we should offer a "clone" method
which
creates a separate iterator using allterms_begin() and skip_to().
> Note that if term2 is not set to db.allterms_begin(), the code crashes.
> 
> Is there a more elegant way to iterate over pairs of terms?
If you're manipulating the terms a lot, you could pull them out into a
vector or something first, then use that.  For just iterating twice, I
suspect it's not worthwhile, and sucking everything into memory works
less well if the database is too big, whereas iterating from the disk
table probably degrades more gracefully.

Cheers,
    Olly