matthias lay
2015-Apr-27 12:23 UTC
[Xapian-discuss] empty FD after reopen since version 1.2.16
Hi all,
after upgrading xapian I encountered the same problem as described in
ticket
#645 Read block errors after reopen()
in our setup its 100% reproducible after each reopen(). I downgraded
again and it seems the problem occurs in Version 1.2.16 and above.
in <=1.2.15 everything works fine without seeing this error once.
attaches strace shows read ends on FD.
strace starts at reopen() call.
FDs are shown on bottom
Greetz Matze
-------------- next part --------------
[pid 27922] write(2, "DatabaseModifiedError: The revis"...,
134DatabaseModifiedError: The revision being read has been discarded - you
should call Xapian::Database::reopen() and retry the operation) = 134
[pid 27922] write(2, "\n", 1
) = 1
[pid 27922] write(2, "type=", 5type=) = 5
[pid 27922] write(2, "DatabaseModifiedError", 21DatabaseModifiedError)
= 21
[pid 27922] write(2, "\n", 1
) = 1
[pid 27922] write(2, "reopening database", 18reopening database) = 18
[pid 27922] write(2, "\n", 1
) = 1
[pid 27922] close(10) = 0
[pid 27922] open("/var/data/vmail/private/pdf/xapian.db/record.DB",
O_RDONLY|O_LARGEFILE) = 10
[pid 27922] open("/var/data/vmail/private/pdf/xapian.db/record.baseA",
O_RDONLY|O_LARGEFILE) = 16
[pid 27922] read(16,
"\4\5\200@\22\1\5\255\3%\0\0\4\377\376\377\357?\4", 1024) = 19
[pid 27922] read(16, "", 1005) = 0
[pid 27922] close(16) = 0
[pid 27922] open("/var/data/vmail/private/pdf/xapian.db/record.baseB",
O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
[pid 27922] pread64(10,
"\0\0\0\4\1\0362\0362\0Q\37\371\37\356\37\343\37\330\37\315\37\302\37\267\37\254\37\241\37\226\37"...,
8192, 147456) = 8192
[pid 27922] open("/var/data/vmail/private/pdf/xapian.db/spelling.DB",
O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
[pid 27922] open("/var/data/vmail/private/pdf/xapian.db/synonym.DB",
O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
[pid 27922] close(11) = 0
[pid 27922] open("/var/data/vmail/private/pdf/xapian.db/termlist.DB",
O_RDONLY|O_LARGEFILE) = 11
[pid 27922]
open("/var/data/vmail/private/pdf/xapian.db/termlist.baseA",
O_RDONLY|O_LARGEFILE) = 16
[pid 27922] read(16,
"\4\5\200@&\1\n\332\6H\0\0\4\377\377\375\377\377\377\377\376\377\1\4",
1024) = 24
[pid 27922] read(16, "", 1000) = 0
[pid 27922] close(16) = 0
[pid 27922]
open("/var/data/vmail/private/pdf/xapian.db/termlist.baseB",
O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
[pid 27922] pread64(11,
"\0\0\0\4\1\34a\34a\0\227\37\371\37\355\37\341\37\326\37\313\37\300\37\265\37\236\37\252\37\223\37"...,
8192, 311296) = 8192
[pid 27922] close(12) = 0
[pid 27922] open("/var/data/vmail/private/pdf/xapian.db/position.DB",
O_RDONLY|O_LARGEFILE) = 12
[pid 27922]
open("/var/data/vmail/private/pdf/xapian.db/position.baseA",
O_RDONLY|O_LARGEFILE) = 16
[pid 27922] read(16,
"\4\5\200@\32\1\6\315\226\1/\0\0\4\377\367\377\377\357\377\4", 1024) =
21
[pid 27922] read(16, "", 1003) = 0
[pid 27922] close(16) = 0
[pid 27922]
open("/var/data/vmail/private/pdf/xapian.db/position.baseB",
O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
[pid 27922] pread64(12,
"\0\0\0\4\1\35[\35[\0e\37\371\37\355\37\341\37\325\37\311\37\274\37\257\37\242\37\224\37\210\37"...,
8192, 212992) = 8192
[pid 27922] close(13) = 0
[pid 27922] open("/var/data/vmail/private/pdf/xapian.db/postlist.DB",
O_RDONLY|O_LARGEFILE) = 13
[pid 27922]
open("/var/data/vmail/private/pdf/xapian.db/postlist.baseA",
O_RDONLY|O_LARGEFILE) = 16
[pid 27922] read(16,
"\4\5\200@?\1\"\237w\217\2\0\0\4\322\257\177\16\0\0\0\300\377\377\377\377\377\377\377\377\377\7"...,
1024) = 49
[pid 27922] read(16, "", 975) = 0
[pid 27922] close(16) = 0
[pid 27922]
open("/var/data/vmail/private/pdf/xapian.db/postlist.baseB",
O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
[pid 27922] pread64(13,
"\0\0\0\4\1\26\30\26\30\1O\37\371\34?\27\343\37\241\37k\34\267\34W\34\25\33\254\33\202\33"...,
8192, 516096) = 8192
[pid 27922] pread64(13,
"\0\0\0\4\0\1\\\10\252\0A\37\371\37\347\37\321\37\264\37\216\37\7\36\367\36\330\36\311\36\262\4"...,
8192, 507904) = 8192
[pid 27922] pread64(13,
"\0\0\0\4\0\4\230\0051\0017\37\343\37\303\37\256\37\231\37\211\37v\37b\37N\37>\37'\37"...,
8192, 950272) = 8192
[pid 27922] pread64(13,
"\0\0\0\4\0\1\\\10\252\0A\37\371\37\347\37\321\37\264\37\216\37\7\36\367\36\330\36\311\36\262\4"...,
8192, 507904) = 8192
[pid 27922] pread64(13,
"\0\0\0\4\0\17\300\17\300\0\23\30#\17\323\20(\17\355\17\203\7W\5*\1w\1\5\0\25\10"...,
8192, 696320) = 8192
[pid 27922] pread64(13,
"\0\0\0\1\0\17\304\17\304\0\21\30\5\27\271\17\325\n\311\0\35\3G\3G\1w\1w\3*\10"...,
8192, 73728) = 8192
[pid 27922] pread64(13,
"\0\0\0\2\0\17\376\17\376\0\17\30\10\20\r\17\342\0170\n\236\0\35\4\3\1w\1w\3\346\10"...,
8192, 565248) = 8192
[pid 27922] pread64(13,
"\0\0\0\3\0\17\371\17\371\0\21\37\325\27\360\20\n\10\17\7\345\0W\0W\1w\1w\3\346\10"...,
8192, 106496) = 8192
[pid 27922] pread64(13,
"\0\0\0\3\0\17\320\17\320\0\21\30\5\27\333\17\341\1\204\0b\0b\0W\1w\1w\3\346\10"...,
8192, 163840) = 8192
[pid 27922] pread64(13,
"\0\0\0\3\0\17\350\17\350\0\21\30\5\27\353\17\371\2\351\0)\0)\0W\1w\1w\3\346\10"...,
8192, 221184) = 8192
[pid 27922] pread64(13,
"\0\0\0\4\0\17\274\17\274\0\21\30\5\27\310\17\315\4\256\0G\0G\0W\1w\1w\3\346\10"...,
8192, 704512) = 8192
[pid 27922] brk(0x813c000) = 0x813c000
[pid 27922] pread64(13,
"\0\0\0\4\0\20%\20%\0\17\30\24\0204\0\27\10T\2O\0G\0028\10\0\330\31\1\220\0"...,
8192, 794624) = 8192
[pid 27922] pread64(13,
"\0\0\0\4\0\0\340\7\264\0\23\30\6\7\320\20&\0\363\2O\0G\0028\10\0\330\31\1\220\0"...,
8192, 884736) = 8192
[pid 27922] pread64(13,
"\0\0\0\4\0\20\33\20\33\0\17\30\25\20*\10/\7\375\7\375\37'\3\373\10\0\330\v\1\5\0"...,
8192, 532480) = 8192
[pid 27922] pread64(13, "", 8192, 22960911941632) = 0
[pid 27922] write(2, "xapian Error in: ", 17xapian Error in: ) = 17
[pid 27922] write(2, "search_document", 15search_document) = 15
[pid 27922] write(2, "DatabaseError: Error reading blo"...,
62DatabaseError: Error reading block 2802845696: got end of file) = 62
[pid 27922] write(2, "\n", 1
**********************************************
root at vmuma:/var/data/src/dms# ll /proc/27922/fd
total 0K
dr-x------ 2 vmail http 0 Apr 15 11:00 .
dr-xr-xr-x 8 vmail http 0 Apr 15 11:00 ..
lrwx------ 1 vmail http 64 Apr 15 11:00 0 -> /dev/pts/0
lrwx------ 1 vmail http 64 Apr 15 11:00 1 -> /dev/pts/0
lr-x------ 1 vmail http 64 Apr 15 11:00 10 ->
/var/data/vmail/private/pdf/xapian.db/record.DB
lr-x------ 1 vmail http 64 Apr 15 11:00 11 ->
/var/data/vmail/private/pdf/xapian.db/termlist.DB
lr-x------ 1 vmail http 64 Apr 15 11:00 12 ->
/var/data/vmail/private/pdf/xapian.db/position.DB
lr-x------ 1 vmail http 64 Apr 15 11:00 13 ->
/var/data/vmail/private/pdf/xapian.db/postlist.DB
lrwx------ 1 vmail http 64 Apr 15 11:00 15 -> /dev/zero
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.xapian.org/pipermail/xapian-discuss/attachments/20150427/11530ac0/attachment.sig>
Olly Betts
2015-Apr-28 05:40 UTC
[Xapian-discuss] empty FD after reopen since version 1.2.16
On Mon, Apr 27, 2015 at 02:23:25PM +0200, matthias lay wrote:> after upgrading xapian I encountered the same problem as described in > ticket > #645 Read block errors after reopen() > > in our setup its 100% reproducible after each reopen(). I downgraded > again and it seems the problem occurs in Version 1.2.16 and above. > in <=1.2.15 everything works fine without seeing this error once.I suspect this commit fixes your issue - you can probably just apply the chert parts to 1.2.x easily, though I've not tried to backport it yet: http://trac.xapian.org/changeset/1900eca6d325e9185b9f9f5c75a4a4a545616d7f/git Thanks for narrowing down the release which introduced this - the relevant change in 1.2.16 is presumably this one: | * Reuse a cursor for reading values from valuestreams rather than creating | a new one each time. This can dramatically reduce the number of blocks | redundantly reread when sorting by value. The rereads will generally get | served from VM cache, but there's still an overhead to that. Before that we just created a cursor each time, so the underlying cursor bug wouldn't be triggered in this case. Cheers, Olly
matthias lay
2015-Apr-28 11:16 UTC
[Xapian-discuss] empty FD after reopen since version 1.2.16
Hi Olly, On 04/28/2015 07:40 AM, Olly Betts wrote:> On Mon, Apr 27, 2015 at 02:23:25PM +0200, matthias lay wrote: >> after upgrading xapian I encountered the same problem as >> described in ticket #645 Read block errors after reopen() >> >> in our setup its 100% reproducible after each reopen(). I >> downgraded again and it seems the problem occurs in Version >> 1.2.16 and above. in <=1.2.15 everything works fine without >> seeing this error once. > > I suspect this commit fixes your issue - you can probably just > apply the chert parts to 1.2.x easily, though I've not tried to > backport it yet: > > http://trac.xapian.org/changeset/1900eca6d325e9185b9f9f5c75a4a4a545616d7f/git > > Thanks for narrowing down the release which introduced this - the > relevant change in 1.2.16 is presumably this one: > > | * Reuse a cursor for reading values from valuestreams rather than > creating | a new one each time. This can dramatically reduce the > number of blocks | redundantly reread when sorting by value. The > rereads will generally get | served from VM cache, but there's > still an overhead to that. > > Before that we just created a cursor each time, so the underlying > cursor bug wouldn't be triggered in this case. > > Cheers, Olly >just tested your patch with 1.2.20 and the exception hasnt occured so far. so seems to fix the problem. thx for the fast fix Greetz Matze -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: <http://lists.xapian.org/pipermail/xapian-discuss/attachments/20150428/fd9041bb/attachment.sig>