search for: file_to_token

Displaying 1 result from an estimated 1 matches for "file_to_token".

2006 Apr 12
1
testing tokenizers
Hi, is there a way to test tokenizers? I mean, I want to give input stream and see the output tokens. AND is there a way to see an indexed document''s index tokens? Which words in the document are used to index this document? Thanks in advance Onur