Xapian discuss - Jul 2004 - large Xapian index files

Hello Arjen van der Meijden,

on xapian-discuss you mentioned that your Xapian installation has got up to
15 GB database size. Can you tell me about the largest index filesize you got?

According to <http://xapian.org/docs/scalability.html>, it seems that the
quartz database filesize is limited only by the OS and file system. Can you
confirm from your experience that there is no 2GB limit?

(I have grepped for "size_t" and "off_t" in the xapian-core
code, but I do not
know how to interpret what I see.)

-- 
Robert Pollak
GPG Key ID: 748646AD

Hi Robert,

We have been using Xapian/Omega for over a year now and even when we 
started (version 0.6.4 orso, although it was still called 0.4.1-cvs or 
something like that), we didn't run into any such problems. Here is a 
`ls -1sh` of our current working-database:

default-worknew$ ls -1sh
total 13G
4.0k meta
6.2G position_DB
104k position_baseA
104k position_baseB
3.1G postlist_DB
  56k postlist_baseA
  56k postlist_baseB
310M record_DB
8.0k record_baseA
8.0k record_baseB
2.8G termlist_DB
  44k termlist_baseA
  44k termlist_baseB
  67M value_DB
4.0k value_baseA
4.0k value_baseB

As you can see there are three tables which have passed the 2G barrier. 
This is (and has always been) on a Linux system with a 2.4 kernel.

I guess this answers your question.

Best regards,

Arjen


On 19-7-2004 15:04, Robert Pollak wrote:
> Hello Arjen van der Meijden,
> 
> on xapian-discuss you mentioned that your Xapian installation has got up to
> 15 GB database size. Can you tell me about the largest index filesize 
> you got?
> 
> According to <http://xapian.org/docs/scalability.html>, it seems that
the
> quartz database filesize is limited only by the OS and file system. Can you
> confirm from your experience that there is no 2GB limit?
> 
> (I have grepped for "size_t" and "off_t" in the
xapian-core code, but I
> do not
> know how to interpret what I see.)
>

On Mon, Jul 19, 2004 at 03:04:49PM +0200, Robert Pollak
wrote:
> According to <http://xapian.org/docs/scalability.html>, it seems that
the
> quartz database filesize is limited only by the OS and file system. Can you
> confirm from your experience that there is no 2GB limit?
That's correct.
> (I have grepped for "size_t" and "off_t" in the
xapian-core code, but I do
> not know how to interpret what I see.)
We use the standard autoconf AC_SYS_LARGEFILE test to decide what (if
anything) needs to be done to enable support for files >= 2G on the
host system.  Having done this, off_t will be a 64 bit type (assuming
the host actually has large file support).

Cheers,
    Olly