Xapian discuss - Apr 2011 - For more UI functionality: OmegaScript, C++ or other?

Hello :-)

We have Xapian and Omega working nicely.  Now we would like more
functionality at the browser UI, such as being able to specify date ranges
and being able to search the file name including looking only for, say, .odt
files.

Question is whether the best choice for this would be to develop a template
using OmegaScript or to switch to C++ or some other language.

Availability of skills both for development and long-term support are
relevant.  For development, I am the only resource; my web programming
skills are basic (some hand edited static HTML pages), my C is very rusty
(no C++), I have Ruby (not Ruby on Rails), I like the look of Python and
have never liked perl. For long-term support we have several PHP people in
the organisation (not on this project).

Given the paucity of skills available for development, the Omega template
option is attractive but I am concerned that it may not be a good choice
long term.  Extensive browsing of the mailing lists suggest that it is not a
popular choice.  This is further suggested by the lack of examples of Omega
templates both in the Xapian user-contributed documentation and generally on
the web?  If this impression of Omega templates unpopularity is correct, why
so?  Is it restrictive?
 
Best

Charles

On Sun, Apr 17, 2011 at 10:33:23PM +0530, xapian at catcons.co.uk
wrote:
> We have Xapian and Omega working nicely.  Now we would like more
> functionality at the browser UI, such as being able to specify date ranges
> and being able to search the file name including looking only for, say,
.odt
> files.
> 
> Question is whether the best choice for this would be to develop a template
> using OmegaScript or to switch to C++ or some other language.
Both the features your mention are already supported by Omega, so aren't
good reasons to switch.

For date ranges, you just need to set START and/or END CGI parameters.

For filtering by an extension, you need to use version 1.2.5, and then
you can pass CGI parameter B=Eodt to return only files with extension .odt
(or B=Ttext/html for HTML files regardless of the extension - mime-type
filtering works with older versions too).
> Given the paucity of skills available for development, the Omega template
> option is attractive but I am concerned that it may not be a good choice
> long term.  Extensive browsing of the mailing lists suggest that it is not
a
> popular choice.  This is further suggested by the lack of examples of Omega
> templates both in the Xapian user-contributed documentation and generally
on
> the web?  If this impression of Omega templates unpopularity is correct,
why
> so?  Is it restrictive?
How did you try to find Omega templates on the web?  There isn't a
standard extension for them, and most of the command names are likely
to return many unrelated hits.  Also, not everyone is going to make
their templates public - they're often pretty specific to a site.

I'm not sure how widely used they are though - it's hard to get good
evidence on such things.  They can be a bit fiddly to write, depending
what you're trying to do.  The main annoyance in my experience is the
deep nesting of curly brackets you can end up.

Cheers,
    Olly

Hello :-)

How to make Omega CGI remove duplicate documents from its query output?

Apparently scriptindex can be used to add numbered indexes via the
INDEX_SCRIPT as documented at http://xapian.org/docs/omega/scriptindex.html.


Using Omega to query an index built with omindex suggests there are some
default numbered indexes.  Setting &COLLAPSE=<index number> in the URL
(where <index number> was 1, 2 or 3) got a listing that seemed to have
duplicates suppressed, the same number of documents for each of the three
indexes.

Before rolling this out to the users it would be nice to know what these
default numbered indexes are and which, if any, can be safely used to
suppress duplicates.

Is there a way to interrogate the index database?  Are the default numbered
indexes described in the documentation?

Best

Charles

> -----Original Message-----
> Date: Tue, 26 Apr 2011 11:08:28 +0100
> From: James Aylett <james-xapian at tartarus.org>
> Subject: Re: [Xapian-discuss] For more UI functionality: OmegaScript,
> 	C++or other?
> To: Chris <chris at s-4-u.net>
> Cc: xapian-discuss at lists.xapian.org
> Message-ID: <8062E94E-7640-44CC-804A-BCC94F56E0F5 at tartarus.org>
> Content-Type: text/plain; charset=us-ascii
> 
> On 26 Apr 2011, at 10:15, Chris wrote:
> > Checkboxes support a value attribute, something like
> > 
> > <input type="checkbox" name="B"
value="Edoc" /> doc
> > <input type="checkbox" name="B"
value="Edocx" /> docx
> > 
> > allows searching for multiple types.
> 
> 
> I'd recommend using SELECT MULTIPLE and then if you don't 
> like the layout, use Javascript to turn it into something 
> else. Select is the appropriate form element here.
> 
> J
> 
> -- 
>  James Aylett
>  talktorex.co.uk - xapian.org - devfort.com
> 
 
Thanks Chris, James and Richard for suggestions and clarifying discussion
:-)

Sorry it has taken a while to respond; I decided to try James' suggestion
first and, following an off-list exchange (thanks James), have spent days
getting jQquery.js to load!
(http://forum.jquery.com/topic/failed-to-load-source-for-jquery-1-5-2-js).

Now jQquery.js is loading, I've got jQuery "Hello World" HTML that
works
when browsed directly as a static page but not when apparently the same HTML
is produced by Omega CGI.

Here's the working HTML (/var/lib/omega/templates/index.html.  jQuery is an
Apache Alias for /opt/jQuery.  :

<html>                                                                  
<head>                                                                  
<script type="text/javascript"
src="/jQuery/jquery.js"></script>
<script type="text/javascript">
  $(document).ready(function() {
    $("a").click(function() {
      alert("Hello world!");
    });
  });
</script>                                                               
</head>                                                                 
<body>                                                                  
<a href="">Link</a>
</body>                                                                 
</html>

Notes:
* In the direct case, the HTML is /var/lib/omega/templates/index.html and a
virtual host has /var/lib/omega/templates as DocumentRoot.
* In the Omega case, the same HTML is /var/lib/omega/templates/jqtry

The HTML works in the direct case. Running it as an Omega template
(&FMT=jqtry) displays "Link" as expected but clicking does not
display
"Hello World".

In both cases, Firefox' View -> Page Source displays the same HTML as
above
but Firebug's Script tab shows them differently.  In the direct case, it
shows the HTML; for the page produced by Omega: "No Javascript on this
page".

How could that be?!  :-)

I plan to explore checkboxes next.  

Best

Charles

Hello  :-)

Thanks to information given in this mailing list, here's an OmegaScript
solution to allow the user to restrict the search by filename extensions.
For illustrative purposes it only covers three extensions -- ods, pdf and
xls.  It uses the CGI parameter B=E* functionality introduced in Xapian
Omega 1.2.5.

Here are the changes to the as-installed Omega template "query" (sorry
about
the line wrapping):

root at CW8vDS:/var/lib/omega/templates# diff -u query
query_with_extension_checkboxes 
--- query	2011-04-04 20:48:48.000000000 +0530
+++ query_with_extension_checkboxes	2011-05-05 17:41:11.016731118 +0530
@@ -52,6 +52,8 @@
 <INPUT TYPE=SUBMIT VALUE="Search">
 <hr>
 $if{,$include{inc/anyalldropbox},$include{inc/anyallradio}}
+<br>
+$include{inc/file_ext_checkboxes}
 $if{$opt{topterms},
  <div title="Suggested terms to add to your query"
  style="text-align:left;background:#cfc;border:1px solid
green;padding:2px;font:11px verdana$. arial$. helvetica$. sans-serif;">
@@ -136,9 +138,6 @@
 <INPUT TYPE=hidden NAME=xFILTERS VALUE="$html{$filters}">
 $list{$relevants,<INPUT TYPE=hidden NAME=R VALUE=",.,">}
 $if{$cgi{THRESHOLD},<INPUT TYPE=hidden NAME=THRESHOLD
VALUE="$html{$cgi{THRESHOLD}}">}
-${Preserve any B filters used - this needs modifying if you modify this
template
-  to have B on radio buttons, SELECT, etc}
-$map{$cgilist{B},<INPUT TYPE=hidden NAME=B VALUE="$html{$_}">}
 </FORM>
 <hr><div align=right><i><small><a
href="http://xapian.org/">$html{$version}</a></small></i></div>
 </body>

Here is the "helper fragment" in the inc subdirectory of the templates
directory (sorry about the line wrapping):

root at CW8vDS:/var/lib/omega/templates# cat inc/file_ext_checkboxes
$if{$cgilist{B},$setmap{BE,ods,0,pdf,0,xls,0}$map{$cgilist{B},$setmap{BE,$su
bstr{$_,1},1}},$setmap{BE,ods,1,pdf,1,xls,1}}
Extensions: 
ods:<INPUT TYPE=checkbox NAME=B VALUE=Eods
$if{$eq{$opt{BE,ods},1},CHECKED}>
pdf:<INPUT TYPE=checkbox NAME=B VALUE=Epdf
$if{$eq{$opt{BE,pdf},1},CHECKED}>
xls:<INPUT TYPE=checkbox NAME=B VALUE=Exls
$if{$eq{$opt{BE,xls},1},CHECKED}>

Pseudocode to explain the algorithm of the $if{ ... } line:

if there are any B CGI parameters then
    set all members of array BE to 0
    for each B parameter that is set
        set the corresponding member of array BE to 1
else
    set all members of array BE to 1
endif

Best

Charles

> Date: Mon, 9 May 2011 23:08:52 +0100
> From: Olly Betts <olly at survex.com>
> Subject: Re: [Xapian-discuss] For more UI functionality: OmegaScript,
> 	C++	or other?
> To: xapian at catcons.co.uk
> Cc: xapian-discuss at lists.xapian.org
> Message-ID: <20110509220852.GQ4381 at survex.com>
> Content-Type: text/plain; charset=us-ascii
> 
> On Thu, May 05, 2011 at 06:18:58PM +0530, xapian at catcons.co.uk wrote:
[snip]
> 
> There's no real need to set the elements to zero - if they aren't
set,
> then $opt{} returns an empty string - so you could just use:
> 
> $if{$cgilist{B},$map{$cgilist{B},$setmap{BE,$substr{$_,1},1}},
> $setmap{BE,ods,1,pdf,1,xls,1}}
> Extensions: 
> ods:<INPUT TYPE=checkbox NAME=B VALUE=Eods $if{$opt{BE,ods},CHECKED}>
> pdf:<INPUT TYPE=checkbox NAME=B VALUE=Epdf $if{$opt{BE,pdf},CHECKED}>
> xls:<INPUT TYPE=checkbox NAME=B VALUE=Exls $if{$opt{BE,xls},CHECKED}>
> 
> Or factor out the list of extensions, which makes it much simpler to
> modify the list in the future:
> 
> $set{EXTS,$split{Eods Epdf Exls}}
> $map{$or{$cgilist{B},$opt{EXTS}},$setmap{BE,$_,1}}
> Extensions:
> $map{$opt{EXTS},
> $substr{$_,1}:<INPUT TYPE=checkbox NAME=B VALUE=$_ 
> $if{$eq{$opt{BE,$_},1},CHECKED}>}
> 
> Cheers,
>     Olly
Thanks Olly :-)

That's much neater -- the difference between simply making it work and doing
so elegantly.

Thanks also for your explanation about multiple &COLLAPSEs and the final
"-1" on the &xFILTERS value.

Best

Charles