Hello Everyone,
I have a small base of users (30), but a lot of emails.
I have an error again when I am indexing a virtual folder with a large number of
folders.
I appreciate this is a special case, but I am using dovecot and solr as it is,
according to the documentation, the favourite way.
One user is using a large number of archives sub folders, by years, months and
subfolders, since 5 years. I am convincing this user to give up this
organisation, but in the mean time, I hope to solve this problem.
The virtual folder, called Archives, references all the sub folders in the
Archive folder. There is about 380 folders, and worst of all, 55k characters
with
their names concatenated.
On solr, I have increased the value of headerBufferSize to a huge value, but I
have still this error.
Anyway, this is the error I have when I try to index this virtual folder:
---8<---------------------------------------------------------------------------------------------------------------------------------------
root at lapetus:/mnt/emails/indiefield.co.uk/jsmith/Mails/Virtual# doveadm -v
index -u 'john smith' 'Search/Archives'
doveadm(john smith): Info: Search/Archives: Caching mails seq=109580..127362
1800/17783doveadm(john smith): Panic: file solr-connection.c: line 416
(solr_connection_select): assertion failed: (!conn->posting)
doveadm(john smith): Error: Raw backtrace:
/usr/lib/dovecot/libdovecot.so.0(+0x4414a) [0x7fcb7b5b014a] ->
/usr/lib/dovecot/libdovecot.so.0(default_fatal_handler+0x2a) [0x7fcb7b5b020a]
-> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7fcb7b58782a] ->
/usr/lib/dovecot/modules/lib21_fts_solr_plugin.so(+0x754a) [0x7fcb790fa54a]
-> /usr/lib/dovecot/modules/lib21_fts_solr_plugin.so(+0x4037)
[0x7fcb790f7037] ->
/usr/lib/dovecot/modules/lib20_fts_plugin.so(fts_backend_get_last_uid+0x5a)
[0x7fcb79b2f66a] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0xc1fd)
[0x7fcb79b341fd] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0xc689)
[0x7fcb79b34689] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0xc51e)
[0x7fcb79b3451e] -> doveadm(+0x16719) [0x7fcb7bf71719] ->
doveadm(+0x13346) [0x7fcb7bf6e346] -> doveadm(+0x13e21) [0x7fcb7bf6ee21]
->
doveadm(doveadm_mail_try_run+0x161) [0x7fcb7bf6f2e1] -> doveadm(main+0x3d1)
[0x7fcb7bf6def1] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd)
[0x7fcb7b200ead] -> doveadm(+0x13125) [0x7fcb7bf6e125]
Aborted
---8<---------------------------------------------------------------------------------------------------------------------------------------
On the solr side:
---8<---------------------------------------------------------------------------------------------------------------------------------------
28-Apr-2013 17:13:33 org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: {add=[1/70af772eafc17c51fb350000812a8a77/john smith at indiefield.co.uk,
2/70af772eafc17c51fb350000812a8a77/john smith at indiefield.co.uk,
3/70af772eafc17c51fb350000812a8a
77/john smith at indiefield.co.uk, 4/70af772eafc17c51fb350000812a8a77/john smith
at indiefield.co.uk, 5/70af772eafc17c51fb350000812a8a77/john smith at
indiefield.co.uk, 6/70af772ea
fc17c51fb350000812a8a77/john smith at indiefield.co.uk,
7/70af772eafc17c51fb350000812a8a77/john smith at indiefield.co.uk,
8/70af772eafc17c51fb350000812a8a77/john smith at indiefie
ld.co.uk, ... (8 added)]} 0 19141
28-Apr-2013 17:13:33 org.apache.solr.common.SolrException log
SEVERE: org.apache.solr.common.SolrException: ParseError at
[row,col]:[1188418,131]
Message: Premature end of file.
at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:72)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:54)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1317)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at
org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:943)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:843)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: javax.xml.stream.XMLStreamException: ParseError at
[row,col]:[1188418,131]
Message: Premature end of file.
at
com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl.next(XMLStreamReaderImpl.java:592)
at org.apache.solr.handler.XMLLoader.readDoc(XMLLoader.java:273)
at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:138)
at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69)
... 22 more
---8<---------------------------------------------------------------------------------------------------------------------------------------
Kind regards,
Andre Rodier
Sorry, I checked my dovecot version, and it has not been updated by Debian yet. I will check if I can find a method to update without breaking dependancies. Andr? On 28/04/13 17:42, Andre Rodier wrote:> Hello Everyone, > > I have a small base of users (30), but a lot of emails. > > I have an error again when I am indexing a virtual folder with a large number of folders. > I appreciate this is a special case, but I am using dovecot and solr as it is, according to the documentation, the favourite way. > > One user is using a large number of archives sub folders, by years, months and subfolders, since 5 years. I am convincing this user to give up this > organisation, but in the mean time, I hope to solve this problem. > > The virtual folder, called Archives, references all the sub folders in the Archive folder. There is about 380 folders, and worst of all, 55k characters with > their names concatenated. > > On solr, I have increased the value of headerBufferSize to a huge value, but I have still this error. > > Anyway, this is the error I have when I try to index this virtual folder: > > ---8<--------------------------------------------------------------------------------------------------------------------------------------- > root at lapetus:/mnt/emails/indiefield.co.uk/jsmith/Mails/Virtual# doveadm -v index -u 'john smith' 'Search/Archives' > doveadm(john smith): Info: Search/Archives: Caching mails seq=109580..127362 > 1800/17783doveadm(john smith): Panic: file solr-connection.c: line 416 (solr_connection_select): assertion failed: (!conn->posting) > doveadm(john smith): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x4414a) [0x7fcb7b5b014a] -> > /usr/lib/dovecot/libdovecot.so.0(default_fatal_handler+0x2a) [0x7fcb7b5b020a] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7fcb7b58782a] -> > /usr/lib/dovecot/modules/lib21_fts_solr_plugin.so(+0x754a) [0x7fcb790fa54a] -> /usr/lib/dovecot/modules/lib21_fts_solr_plugin.so(+0x4037) [0x7fcb790f7037] -> > /usr/lib/dovecot/modules/lib20_fts_plugin.so(fts_backend_get_last_uid+0x5a) [0x7fcb79b2f66a] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0xc1fd) > [0x7fcb79b341fd] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0xc689) [0x7fcb79b34689] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0xc51e) > [0x7fcb79b3451e] -> doveadm(+0x16719) [0x7fcb7bf71719] -> doveadm(+0x13346) [0x7fcb7bf6e346] -> doveadm(+0x13e21) [0x7fcb7bf6ee21] -> > doveadm(doveadm_mail_try_run+0x161) [0x7fcb7bf6f2e1] -> doveadm(main+0x3d1) [0x7fcb7bf6def1] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd) > [0x7fcb7b200ead] -> doveadm(+0x13125) [0x7fcb7bf6e125] > Aborted > ---8<--------------------------------------------------------------------------------------------------------------------------------------- > > On the solr side: > > ---8<--------------------------------------------------------------------------------------------------------------------------------------- > 28-Apr-2013 17:13:33 org.apache.solr.update.processor.LogUpdateProcessor finish > INFO: {add=[1/70af772eafc17c51fb350000812a8a77/john smith at indiefield.co.uk, 2/70af772eafc17c51fb350000812a8a77/john smith at indiefield.co.uk, 3/70af772eafc17c51fb350000812a8a > 77/john smith at indiefield.co.uk, 4/70af772eafc17c51fb350000812a8a77/john smith at indiefield.co.uk, 5/70af772eafc17c51fb350000812a8a77/john smith at indiefield.co.uk, 6/70af772ea > fc17c51fb350000812a8a77/john smith at indiefield.co.uk, 7/70af772eafc17c51fb350000812a8a77/john smith at indiefield.co.uk, 8/70af772eafc17c51fb350000812a8a77/john smith at indiefie > ld.co.uk, ... (8 added)]} 0 19141 > 28-Apr-2013 17:13:33 org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: ParseError at [row,col]:[1188418,131] > Message: Premature end of file. > at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:72) > at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:54) > at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1317) > at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338) > at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241) > at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) > at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388) > at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) > at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) > at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) > at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418) > at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) > at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114) > at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) > at org.mortbay.jetty.Server.handle(Server.java:326) > at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) > at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:943) > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:843) > at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218) > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) > at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) > at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) > Caused by: javax.xml.stream.XMLStreamException: ParseError at [row,col]:[1188418,131] > Message: Premature end of file. > at com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl.next(XMLStreamReaderImpl.java:592) > at org.apache.solr.handler.XMLLoader.readDoc(XMLLoader.java:273) > at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:138) > at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69) > ... 22 more > ---8<--------------------------------------------------------------------------------------------------------------------------------------- > > Kind regards, > Andre Rodier
Am 29.04.2013 11:08, schrieb Andre Rodier:> Sorry, > > I checked my dovecot version, and it has not been updated by Debian yet. > > I will check if I can find a method to update without breaking dependancies. > > Andr?perhaps look at http://wiki.dovecot.org/PrebuiltBinaries#Automatically_Built_Packages leads to rep http://xi.rename-it.nl/debian/ Best Regards MfG Robert Schetterer -- [*] sys4 AG http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstra?e 15, 81669 M?nchen Sitz der Gesellschaft: M?nchen, Amtsgericht M?nchen: HRB 199263 Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein
Thanks Robert, I was not aware of this distribution. ATB, Andr? On 29/04/13 10:21, Robert Schetterer wrote:> Am 29.04.2013 11:08, schrieb Andre Rodier: >> Sorry, >> >> I checked my dovecot version, and it has not been updated by Debian yet. >> >> I will check if I can find a method to update without breaking dependancies. >> >> Andr? > > perhaps look at > > http://wiki.dovecot.org/PrebuiltBinaries#Automatically_Built_Packages > > leads to rep > > http://xi.rename-it.nl/debian/ > > Best Regards > MfG Robert Schetterer >