Friends, I would like to hear from anyone who has experience deploying Xapian on Android. I'm new to Xapian, but I know it is used by a couple partners for offline projects on Linux and Windows. Our small nonprofit, WiderNet, provides off-line access to thousands of Web sites for people who lack Internet connectivity (www.widernet.org). Over 2,000 universities, schools, health care sites, and libraries in the developing world have adopted this entirely offline solution, providing a bounty of educational information to millions of users. (www.widernet.org) For our big library (12GB and 45 million documents) we use Lucene and SOLR. Now we're developing smaller libraries that will fit on smartphones, tablets, and laptops. For example, our 16GB Ebola Pocket Library that is in the hands of hundreds of health workers in the Democratic Republic of Congo. We seek an Android search interface that would interact with an index of ~128GB of resources. We can create the index on more powerful machines with any OS in our development lab, but need to provide the end users with tools to search that index on their devices. Thanks in advance for your advice and insights! -- Cliff
On 27.08.20 17:55, Cliff Missen wrote:> Now we're developing smaller libraries that will fit on smartphones, tablets, and laptops. For example, our 16GB Ebola Pocket Library that is in the hands of hundreds of health workers in the Democratic Republic of Congo. > > We seek an Android search interface that would interact with an index of ~128GB of resources. > > We can create the index on more powerful machines with any OS in our development lab, but need to provide the end users with tools to search that index on their devices.You mean something like Kiwix? Regards Emmanuel -- Kiwix - Wikipedia Offline & more * Web: https://kiwix.org/ * Twitter: https://twitter.com/KiwixOffline * Wiki: https://wiki.kiwix.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 195 bytes Desc: OpenPGP digital signature URL: <http://lists.xapian.org/pipermail/xapian-discuss/attachments/20200827/38580965/attachment.sig>
Yes. Something like Kiwix, but a different infrastructure and manifold content options. Sent from my WebTV ________________________________ From: Emmanuel Engelhart <kelson at kiwix.org> Sent: Thursday, August 27, 2020 3:13:28 PM To: Cliff Missen <missenc at widernet.org>; Xapian Discussion <xapian-discuss at lists.xapian.org> Subject: Re: Xapian on Android? On 27.08.20 17:55, Cliff Missen wrote:> Now we're developing smaller libraries that will fit on smartphones, tablets, and laptops. For example, our 16GB Ebola Pocket Library that is in the hands of hundreds of health workers in the Democratic Republic of Congo. > > We seek an Android search interface that would interact with an index of ~128GB of resources. > > We can create the index on more powerful machines with any OS in our development lab, but need to provide the end users with tools to search that index on their devices.You mean something like Kiwix? Regards Emmanuel -- Kiwix - Wikipedia Offline & more * Web: https://kiwix.org/ * Twitter: https://twitter.com/KiwixOffline * Wiki: https://wiki.kiwix.org/
On Thu, Aug 27, 2020 at 03:55:07PM +0000, Cliff Missen wrote:> We can create the index on more powerful machines with any OS in our > development lab, but need to provide the end users with tools to > search that index on their devices.Not specific to Android, but for this sort of use case you may find the "single-file database" feature useful (the feature was sponsored by https://endlessos.com/ who use it to allow shipping a single file containing both data and a pre-built Xapian index of it). The single-file database is generated by compacting a database, and can't be updated directly: https://xapian.org/docs/apidoc/html/namespaceXapian.html#af230d8321f07e93dc62d57a6fcfe89fa It can be embedded inside another file (you can open it by providing a file descriptor positioned at the start of the embedded single file database): https://xapian.org/docs/apidoc/html/classXapian_1_1Database.html#aa882e42d12defaf578e68ac9a4fe36f0 Also, if you don't use any features which need the termlist table at search time (the main one is being able to list matching terms) then you could delete termlist.glass from the database directory before compacting to a single file database to save space. See Xapian::DB_NO_TERMLIST for details: https://xapian.org/docs/apidoc/html/namespaceXapian.html#abeb301c332a66efdb695447666c53726 Cheers, Olly
Hi Olly On 30.08.20 22:57, Olly Betts wrote:> On Thu, Aug 27, 2020 at 03:55:07PM +0000, Cliff Missen wrote: >> We can create the index on more powerful machines with any OS in our >> development lab, but need to provide the end users with tools to >> search that index on their devices. > > Not specific to Android, but for this sort of use case you may find the > "single-file database" feature useful (the feature was sponsored by > https://endlessos.com/ who use it to allow shipping a single file > containing both data and a pre-built Xapian index of it). The > single-file database is generated by compacting a database, and can't be > updated directly: > > https://xapian.org/docs/apidoc/html/namespaceXapian.html#af230d8321f07e93dc62d57a6fcfe89fa > > It can be embedded inside another file (you > can open it by providing a file descriptor positioned at the start of > the embedded single file database): > > https://xapian.org/docs/apidoc/html/classXapian_1_1Database.html#aa882e42d12defaf578e68ac9a4fe36f0Both of these features which have been created have allowed to embed Xapian indexes in ZIM files. Therefore super useful for Kiwix. And now Endless uses ZIM and the Zimfarm to build content for the EndlessOS (apps)... soo the circle comes back around :)> Also, if you don't use any features which need the termlist table at > search time (the main one is being able to list matching terms) then you > could delete termlist.glass from the database directory before > compacting to a single file database to save space. > > See Xapian::DB_NO_TERMLIST for details: > > https://xapian.org/docs/apidoc/html/namespaceXapian.html#abeb301c332a66efdb695447666c53726Nice, I should have a look in detail to that. Regards Emmanuel -- Kiwix - Wikipedia Offline & more * Web: https://kiwix.org/ * Twitter: https://twitter.com/KiwixOffline * Wiki: https://wiki.kiwix.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 195 bytes Desc: OpenPGP digital signature URL: <http://lists.xapian.org/pipermail/xapian-discuss/attachments/20200831/51c07072/attachment.sig>