On 15/04/2019 10:59, Larry Rosenman via dovecot wrote:> I'll run a full test when I'm back in front of areal computer vs. My > phone.(in a few hours) > > Get Outlook for Android <https://aka.ms/ghei36> > > ------------------------------------------------------------------------ > *From:* dovecot <dovecot-bounces at dovecot.org> on behalf of John > Fawcett via dovecot <dovecot at dovecot.org> > *Sent:* Monday, April 15, 2019 3:57:08 AM > *To:* Dovecot Mailing List > *Subject:* Re: SOLR/Index? > ? > On 15/04/2019 10:31, Larry Rosenman via dovecot wrote: >> It always shows the autoindex. And yes built from sources.? I'm the >> FreeBSD port maintainer for mail/docecot.? This has been happening >> for several releases. >> >> Get Outlook for Android <https://aka.ms/ghei36> >> >> ------------------------------------------------------------------------ >> *From:* dovecot <dovecot-bounces at dovecot.org> on behalf of John >> Fawcett via dovecot <dovecot at dovecot.org> >> *Sent:* Monday, April 15, 2019 2:06:55 AM >> *To:* dovecot at dovecot.org >> *Subject:* Re: SOLR/Index? >> ? >> On 15/04/2019 08:09, Larry Rosenman via dovecot wrote: >>> Note the hits after the fts rescan/index. >>> >>> Get Outlook for Android <https://aka.ms/ghei36> >>> >>> ------------------------------------------------------------------------ >>> *From:* Aki Tuomi <aki.tuomi at open-xchange.com> >>> *Sent:* Monday, April 15, 2019 12:55:07 AM >>> *To:* Larry Rosenman; John Fawcett >>> *Cc:* Dovecot Mailing List >>> *Subject:* Re: SOLR/Index? >>> ? >>> >>> >>> On 15.4.2019 3.33, Larry Rosenman via dovecot wrote: >>>> ?72% [ler at thebighonker.lerctr.org:~] $ doveadm search mailbox >>>> lists/freebsd/ports-commiters? body 'sysutils' >>>> [ler at thebighonker.lerctr.org:~] $ doveadm fts rescan >>>> [ler at thebighonker.lerctr.org:~] $ doveadm index -q >>>> lists/freebsd/ports-commiters >>>> ?64% [ler at thebighonker.lerctr.org:~] $ tail -f /var/log/maillog >>>> Apr 14 19:30:27 thebighonker dovecot[2507]: imap-login: >>>> Disconnected (auth failed, 1 attempts in 2 secs): user=<gwc>, >>>> method=PLAIN, rip=180.180.217.124, lip=192.147.25.65, TLS: >>>> Connection closed, session=<EAk5woaGV4S0tNl8> >>>> Apr 14 19:30:28 thebighonker dovecot[2507]: imap-login: Login: >>>> user=<ler>, method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, >>>> lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14813, TLS, >>>> session=<lr1mwoaGwYMgAQRwHw8DrQC73P/+UNkA> >>>> Apr 14 19:30:30 thebighonker dovecot[2507]: imap(ler/14813): Logged >>>> out in=12412 out=66691 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0 >>>> Apr 14 19:30:54 thebighonker exim[14846]: no host name found for IP >>>> address 23.100.68.192 >>>> Apr 14 19:30:55 thebighonker exim[14846]: >>>> H=(DaVinci-MWare.prophet21lab.com >>>> <http://DaVinci-MWare.prophet21lab.com>) [23.100.68.192]:52130 >>>> I=[192.147.25.65]:25 sender verify defer for <info at duke.org >>>> <mailto:info at duke.org>>: host lookup did not complete >>>> Apr 14 19:30:55 thebighonker exim[14846]: >>>> H=(DaVinci-MWare.prophet21lab.com >>>> <http://DaVinci-MWare.prophet21lab.com>) [23.100.68.192]:52130 >>>> I=[192.147.25.65]:25 F=<info at duke.org <mailto:info at duke.org>> >>>> temporarily rejected RCPT <jpotyh at why.net <mailto:jpotyh at why.net>>: >>>> Could not complete sender verify >>>> Apr 14 19:31:04 thebighonker dovecot[2507]: imap-login: Login: >>>> user=<ctr>, method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, >>>> lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14910, TLS, >>>> session=<bVWMxIaGJtogAQRwHw8DrQC73P/+UNkA> >>>> Apr 14 19:31:04 thebighonker dovecot[2507]: imap(ctr/14910): Logged >>>> out in=169 out=1711 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0 >>>> Apr 14 19:31:16 thebighonker exim[14911]: no host name found for IP >>>> address 80.253.235.35 >>>> Apr 14 19:31:19 thebighonker dovecot[2507]: >>>> indexer-worker(ler/14919): Indexed 1578 messages in >>>> lists/freebsd/ports-commiters (UIDs 21067..22644) >>>> ^C >>>> [ler at thebighonker.lerctr.org:~] 130 $ doveadm search mailbox >>>> lists/freebsd/ports-commiters? body 'sysutils/' >>> >>> >>> Just minor nit, but you are searching for 'sysutils' first, then >>> 'sysutils/'. FTS does not do substring searches by default. >>> >>> Aki >>> >>>> >>>> --? >>>> Larry Rosenman? ? ? ? ? ? ? ? ? ?? http://www.lerctr.org/~ler >>>> <http://www.lerctr.org/~ler> >>>> Phone: +1 214-642-9640 (c) ? ? E-Mail: larryrtx at gmail.com >>>> <mailto:larryrtx at gmail.com> >>>> US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106 >> >> Larry >> >> just to be sure: are you running a standard unmodified 2.3.5.1 >> version which you built from source code? >> >> I can see that first you search for sysutils, then do a rescan and >> reindex (which is shown in the log) and then you are able to find >> sysutils/. >> >> It is better when doing these tests to search for the same string >> before and after, just to eliminate too many different factors in the >> test. >> >> Nevertheless I did not see your logging for what happens when you >> receive a test message containing sysutils/. Dovecot should be >> outputing info about autoindexing given your setup. Does it do that >> or does it give some other message? Can you show those logs? >> >> John >> > Larry > > Did you notice any difference between the logging for auto indexing > and the logging for indexing that you triggered manually? Would you > mind posting the auto indexing logging for a message to that same user > (ler)? > > best regards > > John >ok, the hypothesis I'm trying to confirm is that auto indexing is taking place under the full username ler at somedomain whereas in the command line tests you did your are using user ler (as presumably is roundcube). John -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://dovecot.org/pipermail/dovecot/attachments/20190415/bcb98ce6/attachment-0001.html>
?63% [ler at thebighonker.lerctr.org:~] $ grep fts1970 mail/INBOX
?67% [ler at thebighonker.lerctr.org:~] 1 $ mail -s "test fts1970"
ler at lerctr.org
test fts1970
test fts1970
.
EOT
[ler at thebighonker.lerctr.org:~] $ mailq
[ler at thebighonker.lerctr.org:~] $ grep fts1970 mail/INBOX
Subject: test fts1970
test fts1970
test fts1970
Apr 15 04:29:03 thebighonker exim[49528]: 1hFxvD-000Csq-P6 <= ler at
lerctr.org
U=ler P=local S=388
Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(49364): Connect from local
Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(ler at lerctr.org/49364):
save: box=INBOX, uid=175402, msgid=<
E1hFxvD-000Csq-P6 at thebighonker.lerctr.org>, size=640, vsize=660,
from=Larry
Rosenman <ler at lerctr.org>, subject=test fts1970, flags=()
Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(ler at lerctr.org/49364):
sieve: msgid=<E1hFxvD-000Csq-P6 at thebighonker.lerctr.org>: stored mail
into
mailbox 'INBOX' (subject=test fts1970 from=ler at lerctr.org size=660)
Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(49364): Disconnect from
local: Client has quit the connection (state=READY)
Apr 15 04:29:03 thebighonker exim[49535]: 1hFxvD-000Csq-P6 => ler <
ler at lerctr.org> R=localuser T=dovecot_lmtp S=404 C="250 2.0.0 <
ler at lerctr.org> 6ACWMN9OtFzUwAAAu+mOrA Saved" QT=0s DT=0s
Apr 15 04:29:03 thebighonker exim[49535]: 1hFxvD-000Csq-P6 Completed QT=0s
Apr 15 04:29:03 thebighonker dovecot[2507]: indexer-worker(
ler at lerctr.org/49366): Indexed 1 messages in INBOX (UIDs 175402..175402)
?81% [ler at thebighonker.lerctr.org:~] $ doveadm search mailbox INBOX body
'fts1970'
?83% [ler at thebighonker.lerctr.org:~] $
?65% [ler at thebighonker.lerctr.org:~] 75 $ doveadm search -u ler at lerctr.org
mailbox INBOX body 'fts1970'
a53a143be44bda5bd4830000bbe98eac 175402
[ler at thebighonker.lerctr.org:~] $ doveadm index -q INBOX
[ler at thebighonker.lerctr.org:~] $ doveadm search mailbox INBOX body
'fts1970'
[ler at thebighonker.lerctr.org:~] $ doveadm fts rescan
[ler at thebighonker.lerctr.org:~] $ doveadm index -q INBOX
[ler at thebighonker.lerctr.org:~] $ doveadm search mailbox INBOX body
'fts1970'
a53a143be44bda5bd4830000bbe98eac 175402
[ler at thebighonker.lerctr.org:~] $ doveadm search -u ler at lerctr.org
mailbox
INBOX body 'fts1970'
a53a143be44bda5bd4830000bbe98eac 175402
[ler at thebighonker.lerctr.org:~] $
So, yes, your hypothesis is correct.
Question: How can I make it consistent?
I have a script that runs on the first of the month that does archiving,
and I have similar issues in that namespace:
?67% [ler at thebighonker.lerctr.org:~] $ cat bin/archive-mail
#!/bin/sh
PATH=$PATH:/usr/local/bin
#Expects to be run after midnight on the first of the month
# to archive all the previous months mail
#Date Run:
TODAY=`date "+%Y-%m-%d"`
#last month in YYYY/MM
YEAR_LAST_MONTH=`date -v-1d "+%Y/%m"`
#1st of last month as 01-Mon-YYYY
FIRST_LAST_MONTH=`date -v-1d "+01-%b-%Y"`
echo 'TODAY=' ${TODAY}
echo 'YEAR_LAST_MONTH=' ${YEAR_LAST_MONTH}
echo 'FIRST_LAST_MONTH=' ${FIRST_LAST_MONTH}
# get a list of all the mailboxes with at least one real message
doveadm -f tab mailbox status vsize \* 2>/dev/null |
sed -e 1d | sort -k 1,1 |
awk 'BEGIN {FS="\t"} {if ($2 > 0) print $1}' |
while read i
do
echo `date` start ${i}
doveadm mailbox create "ARCHIVE/${YEAR_LAST_MONTH}/${i}"
doveadm -f tab mailbox status messages "${i}"
doveadm move "ARCHIVE/${YEAR_LAST_MONTH}/${i}" mailbox \
"${i}" BEFORE ${TODAY} SINCE ${FIRST_LAST_MONTH}
doveadm -f tab mailbox status messages "${i}"
echo `date` done ${i}
done
?64% [ler at thebighonker.lerctr.org:~] $
The Exim config can be provided as well if necessary.
ler & ler at lerctr.org *ARE THE SAME MAILBOX*
On Mon, Apr 15, 2019 at 4:05 AM John Fawcett via dovecot <
dovecot at dovecot.org> wrote:
> On 15/04/2019 10:59, Larry Rosenman via dovecot wrote:
>
> I'll run a full test when I'm back in front of areal computer vs.
My
> phone.(in a few hours)
>
> Get Outlook for Android <https://aka.ms/ghei36>
>
> ------------------------------
> *From:* dovecot <dovecot-bounces at dovecot.org>
> <dovecot-bounces at dovecot.org> on behalf of John Fawcett via
dovecot
> <dovecot at dovecot.org> <dovecot at dovecot.org>
> *Sent:* Monday, April 15, 2019 3:57:08 AM
> *To:* Dovecot Mailing List
> *Subject:* Re: SOLR/Index?
>
> On 15/04/2019 10:31, Larry Rosenman via dovecot wrote:
>
> It always shows the autoindex. And yes built from sources. I'm the
> FreeBSD port maintainer for mail/docecot. This has been happening for
> several releases.
>
> Get Outlook for Android <https://aka.ms/ghei36>
>
> ------------------------------
> *From:* dovecot <dovecot-bounces at dovecot.org>
> <dovecot-bounces at dovecot.org> on behalf of John Fawcett via
dovecot
> <dovecot at dovecot.org> <dovecot at dovecot.org>
> *Sent:* Monday, April 15, 2019 2:06:55 AM
> *To:* dovecot at dovecot.org
> *Subject:* Re: SOLR/Index?
>
> On 15/04/2019 08:09, Larry Rosenman via dovecot wrote:
>
> Note the hits after the fts rescan/index.
>
> Get Outlook for Android <https://aka.ms/ghei36>
>
> ------------------------------
> *From:* Aki Tuomi <aki.tuomi at open-xchange.com>
> <aki.tuomi at open-xchange.com>
> *Sent:* Monday, April 15, 2019 12:55:07 AM
> *To:* Larry Rosenman; John Fawcett
> *Cc:* Dovecot Mailing List
> *Subject:* Re: SOLR/Index?
>
>
>
> On 15.4.2019 3.33, Larry Rosenman via dovecot wrote:
>
> ?72% [ler at thebighonker.lerctr.org:~] $ doveadm search mailbox
> lists/freebsd/ports-commiters body 'sysutils'
> [ler at thebighonker.lerctr.org:~] $ doveadm fts rescan
> [ler at thebighonker.lerctr.org:~] $ doveadm index -q
> lists/freebsd/ports-commiters
> ?64% [ler at thebighonker.lerctr.org:~] $ tail -f /var/log/maillog
> Apr 14 19:30:27 thebighonker dovecot[2507]: imap-login: Disconnected (auth
> failed, 1 attempts in 2 secs): user=<gwc>, method=PLAIN,
> rip=180.180.217.124, lip=192.147.25.65, TLS: Connection closed,
> session=<EAk5woaGV4S0tNl8>
> Apr 14 19:30:28 thebighonker dovecot[2507]: imap-login: Login:
user=<ler>,
> method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900,
> lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14813, TLS,
> session=<lr1mwoaGwYMgAQRwHw8DrQC73P/+UNkA>
> Apr 14 19:30:30 thebighonker dovecot[2507]: imap(ler/14813): Logged out
> in=12412 out=66691 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0
> Apr 14 19:30:54 thebighonker exim[14846]: no host name found for IP
> address 23.100.68.192
> Apr 14 19:30:55 thebighonker exim[14846]: H=(
> DaVinci-MWare.prophet21lab.com) [23.100.68.192]:52130
> I=[192.147.25.65]:25 sender verify defer for <info at duke.org>: host
lookup
> did not complete
> Apr 14 19:30:55 thebighonker exim[14846]: H=(
> DaVinci-MWare.prophet21lab.com) [23.100.68.192]:52130
> I=[192.147.25.65]:25 F=<info at duke.org> temporarily rejected RCPT
<
> jpotyh at why.net>: Could not complete sender verify
> Apr 14 19:31:04 thebighonker dovecot[2507]: imap-login: Login:
user=<ctr>,
> method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900,
> lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14910, TLS,
> session=<bVWMxIaGJtogAQRwHw8DrQC73P/+UNkA>
> Apr 14 19:31:04 thebighonker dovecot[2507]: imap(ctr/14910): Logged out
> in=169 out=1711 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0
> Apr 14 19:31:16 thebighonker exim[14911]: no host name found for IP
> address 80.253.235.35
> Apr 14 19:31:19 thebighonker dovecot[2507]: indexer-worker(ler/14919):
> Indexed 1578 messages in lists/freebsd/ports-commiters (UIDs 21067..22644)
> ^C
> [ler at thebighonker.lerctr.org:~] 130 $ doveadm search mailbox
> lists/freebsd/ports-commiters body 'sysutils/'
>
>
> Just minor nit, but you are searching for 'sysutils' first, then
> 'sysutils/'. FTS does not do substring searches by default.
>
> Aki
>
>
> --
> Larry Rosenman http://www.lerctr.org/~ler
> Phone: +1 214-642-9640 (c) E-Mail: larryrtx at gmail.com
> US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106
>
> Larry
>
> just to be sure: are you running a standard unmodified 2.3.5.1 version
> which you built from source code?
>
> I can see that first you search for sysutils, then do a rescan and reindex
> (which is shown in the log) and then you are able to find sysutils/.
>
> It is better when doing these tests to search for the same string before
> and after, just to eliminate too many different factors in the test.
>
> Nevertheless I did not see your logging for what happens when you receive
> a test message containing sysutils/. Dovecot should be outputing info about
> autoindexing given your setup. Does it do that or does it give some other
> message? Can you show those logs?
>
> John
>
> Larry
>
> Did you notice any difference between the logging for auto indexing and
> the logging for indexing that you triggered manually? Would you mind
> posting the auto indexing logging for a message to that same user (ler)?
>
> best regards
>
> John
>
> ok, the hypothesis I'm trying to confirm is that auto indexing is
taking
> place under the full username ler at somedomain whereas in the command line
> tests you did your are using user ler (as presumably is roundcube).
>
> John
>
--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 214-642-9640 (c) E-Mail: larryrtx at gmail.com
US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20190415/5ee606d0/attachment-0001.html>
On 15/04/2019 11:38, Larry Rosenman via dovecot wrote:> ?63% [ler at thebighonker.lerctr.org:~] $ grep fts1970 mail/INBOX > ?67% [ler at thebighonker.lerctr.org:~] 1 $ mail -s "test fts1970" > ler at lerctr.org <mailto:ler at lerctr.org> > test fts1970 > > test fts1970 > . > EOT > [ler at thebighonker.lerctr.org:~] $ mailq > [ler at thebighonker.lerctr.org:~] $ grep fts1970 mail/INBOX > Subject: test fts1970 > test fts1970 > test fts1970 > > > Apr 15 04:29:03 thebighonker exim[49528]: 1hFxvD-000Csq-P6 <> ler at lerctr.org <mailto:ler at lerctr.org> U=ler P=local S=388 > Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(49364): Connect from > local > Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(ler at lerctr.org/49364 > <http://ler at lerctr.org/49364>): save: box=INBOX, uid=175402, > msgid=<E1hFxvD-000Csq-P6 at thebighonker.lerctr.org > <mailto:E1hFxvD-000Csq-P6 at thebighonker.lerctr.org>>, size=640, > vsize=660, from=Larry Rosenman <ler at lerctr.org > <mailto:ler at lerctr.org>>, subject=test fts1970, flags=() > Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(ler at lerctr.org/49364 > <http://ler at lerctr.org/49364>): sieve: > msgid=<E1hFxvD-000Csq-P6 at thebighonker.lerctr.org > <mailto:E1hFxvD-000Csq-P6 at thebighonker.lerctr.org>>: stored mail into > mailbox 'INBOX' (subject=test fts1970 from=ler at lerctr.org > <mailto:ler at lerctr.org> size=660) > Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(49364): Disconnect > from local: Client has quit the connection (state=READY) > Apr 15 04:29:03 thebighonker exim[49535]: 1hFxvD-000Csq-P6 => ler > <ler at lerctr.org <mailto:ler at lerctr.org>> R=localuser T=dovecot_lmtp > S=404 C="250 2.0.0 <ler at lerctr.org <mailto:ler at lerctr.org>> > 6ACWMN9OtFzUwAAAu+mOrA Saved" QT=0s DT=0s > Apr 15 04:29:03 thebighonker exim[49535]: 1hFxvD-000Csq-P6 Completed QT=0s > Apr 15 04:29:03 thebighonker dovecot[2507]: > indexer-worker(ler at lerctr.org/49366 <http://ler at lerctr.org/49366>): > Indexed 1 messages in INBOX (UIDs 175402..175402) > > > ?81% [ler at thebighonker.lerctr.org:~] $ doveadm search mailbox INBOX? > body 'fts1970' > ?83% [ler at thebighonker.lerctr.org:~] $ > > > ?65% [ler at thebighonker.lerctr.org:~] 75 $ doveadm search -u > ler at lerctr.org <mailto:ler at lerctr.org>? mailbox INBOX body 'fts1970' > a53a143be44bda5bd4830000bbe98eac 175402 > [ler at thebighonker.lerctr.org:~] $ doveadm index -q INBOX > [ler at thebighonker.lerctr.org:~] $ doveadm search mailbox INBOX? body > 'fts1970' > [ler at thebighonker.lerctr.org:~] $ doveadm fts rescan > [ler at thebighonker.lerctr.org:~] $ doveadm index -q INBOX > [ler at thebighonker.lerctr.org:~] $ doveadm search mailbox INBOX? body > 'fts1970' > a53a143be44bda5bd4830000bbe98eac 175402 > [ler at thebighonker.lerctr.org:~] $ doveadm search -u ler at lerctr.org > <mailto:ler at lerctr.org>? mailbox INBOX body 'fts1970' > a53a143be44bda5bd4830000bbe98eac 175402 > [ler at thebighonker.lerctr.org:~] $ > > So, yes, your hypothesis is correct. > > Question: How can I make it consistent??? > > I have a script that runs on the first of the month that does > archiving, and I have similar issues in that namespace: > ?67% [ler at thebighonker.lerctr.org:~] $ cat bin/archive-mail > #!/bin/sh > PATH=$PATH:/usr/local/bin > #Expects to be run after midnight on the first of the month > #? to archive all the previous months mail > #Date Run: > TODAY=`date "+%Y-%m-%d"` > #last month in YYYY/MM > YEAR_LAST_MONTH=`date -v-1d "+%Y/%m"` > #1st of last month as 01-Mon-YYYY > FIRST_LAST_MONTH=`date -v-1d "+01-%b-%Y"` > echo 'TODAY=' ${TODAY} > echo 'YEAR_LAST_MONTH=' ${YEAR_LAST_MONTH} > echo 'FIRST_LAST_MONTH=' ${FIRST_LAST_MONTH} > # get a list of all the mailboxes with at least one real message > doveadm -f tab mailbox status vsize \* 2>/dev/null | > ? ? ? ? sed -e 1d | sort -k 1,1 | > ? ? ? ? awk? 'BEGIN {FS="\t"} {if ($2 > 0)? print $1}' | > while read i > do > ? ?echo `date` start ${i} > ? ?doveadm mailbox create "ARCHIVE/${YEAR_LAST_MONTH}/${i}" > ? ?doveadm -f tab mailbox status messages "${i}" > ? ?doveadm move "ARCHIVE/${YEAR_LAST_MONTH}/${i}" mailbox \ > ? ? ? ? ? ? "${i}" BEFORE ${TODAY} SINCE ${FIRST_LAST_MONTH} > ? ?doveadm -f tab mailbox status messages "${i}" > ? ?echo `date` done? ${i} > done > ?64% [ler at thebighonker.lerctr.org:~] $ > > > The Exim config can be provided as well if necessary. > > ler & ler at lerctr.org <mailto:ler at lerctr.org> *ARE THE SAME MAILBOX* >At the moment it looks as though you have two sets of emails indexed in solr. One is indexed under username (the one you are running mannually and apparently the one used by roundcube too, but that's to be verified) and another set being indexed by autoindex = yes option using the full email address. Once you've got it working as you require, then you may want to clean out solr and reindex with just one of them just to reduce volumes. Your setup seems to have a mix of users from mysql and from /etc/passwd. Not sure if your mysql users are all mapped to real users or they have their own mailboxes with domain included. Your solution will depend on what you really need and if the setup is working correctly you may not want to tweak it too much or other things may start breaking. The simplest thing that comes to mind (providing your manually indexed mails show up in roundcube searches) is just to turn off autoindex and schedule indexing from cron. As things stand I believe autoindex works well with virtual users username at domain. You'll probably need to start off with a rescan if you do this. If you can swith to all virtual users without local users then probably you can use autoindex, but I can understand that may not be possible. For sure I'd recommend doing it in a test environment first so you can be sure of the configuration and then look at migrating your existing mailboxes over to virtual users. John -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://dovecot.org/pipermail/dovecot/attachments/20190415/6cf94d0a/attachment.html>
forgot to reply all. ---------- Forwarded message --------- From: Larry Rosenman <larryrtx at gmail.com> Date: Mon, Apr 15, 2019 at 5:58 AM Subject: Re: SOLR/Index? To: John Fawcett <john at voipsupport.it> the local users (myself, my wife, a friend) can authenticate EITHER as <username> or <username>@lerctr.org. switching to all virtual users is NOT going to happen. If I login to roundcube with <user>@lerctr.org it finds the autoindexed mail. So, if I make everyone always authenticate as <user>@lerctr.org we should be fine. and change my script to do doveadm -u <user>@<domain> instead of depending on the local user running the cron job. question: Is there some way to have dovecot change what it sees to be <user>@lerctr.org when they login as <user>? On Mon, Apr 15, 2019 at 5:54 AM John Fawcett via dovecot < dovecot at dovecot.org> wrote:> On 15/04/2019 11:38, Larry Rosenman via dovecot wrote: > > ?63% [ler at thebighonker.lerctr.org:~] $ grep fts1970 mail/INBOX > ?67% [ler at thebighonker.lerctr.org:~] 1 $ mail -s "test fts1970" > ler at lerctr.org > test fts1970 > > test fts1970 > . > EOT > [ler at thebighonker.lerctr.org:~] $ mailq > [ler at thebighonker.lerctr.org:~] $ grep fts1970 mail/INBOX > Subject: test fts1970 > test fts1970 > test fts1970 > > > Apr 15 04:29:03 thebighonker exim[49528]: 1hFxvD-000Csq-P6 <> ler at lerctr.org U=ler P=local S=388 > Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(49364): Connect from local > Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(ler at lerctr.org/49364): > save: box=INBOX, uid=175402, msgid=< > E1hFxvD-000Csq-P6 at thebighonker.lerctr.org>, size=640, vsize=660, > from=Larry Rosenman <ler at lerctr.org>, subject=test fts1970, flags=() > Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(ler at lerctr.org/49364): > sieve: msgid=<E1hFxvD-000Csq-P6 at thebighonker.lerctr.org>: stored mail > into mailbox 'INBOX' (subject=test fts1970 from=ler at lerctr.org size=660) > Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(49364): Disconnect from > local: Client has quit the connection (state=READY) > Apr 15 04:29:03 thebighonker exim[49535]: 1hFxvD-000Csq-P6 => ler < > ler at lerctr.org> R=localuser T=dovecot_lmtp S=404 C="250 2.0.0 < > ler at lerctr.org> 6ACWMN9OtFzUwAAAu+mOrA Saved" QT=0s DT=0s > Apr 15 04:29:03 thebighonker exim[49535]: 1hFxvD-000Csq-P6 Completed QT=0s > Apr 15 04:29:03 thebighonker dovecot[2507]: indexer-worker( > ler at lerctr.org/49366): Indexed 1 messages in INBOX (UIDs 175402..175402) > > > ?81% [ler at thebighonker.lerctr.org:~] $ doveadm search mailbox INBOX body > 'fts1970' > ?83% [ler at thebighonker.lerctr.org:~] $ > > > ?65% [ler at thebighonker.lerctr.org:~] 75 $ doveadm search -u ler at lerctr.org > mailbox INBOX body 'fts1970' > a53a143be44bda5bd4830000bbe98eac 175402 > [ler at thebighonker.lerctr.org:~] $ doveadm index -q INBOX > [ler at thebighonker.lerctr.org:~] $ doveadm search mailbox INBOX body > 'fts1970' > [ler at thebighonker.lerctr.org:~] $ doveadm fts rescan > [ler at thebighonker.lerctr.org:~] $ doveadm index -q INBOX > [ler at thebighonker.lerctr.org:~] $ doveadm search mailbox INBOX body > 'fts1970' > a53a143be44bda5bd4830000bbe98eac 175402 > [ler at thebighonker.lerctr.org:~] $ doveadm search -u ler at lerctr.org > mailbox INBOX body 'fts1970' > a53a143be44bda5bd4830000bbe98eac 175402 > [ler at thebighonker.lerctr.org:~] $ > > So, yes, your hypothesis is correct. > > Question: How can I make it consistent? > > I have a script that runs on the first of the month that does archiving, > and I have similar issues in that namespace: > ?67% [ler at thebighonker.lerctr.org:~] $ cat bin/archive-mail > #!/bin/sh > PATH=$PATH:/usr/local/bin > #Expects to be run after midnight on the first of the month > # to archive all the previous months mail > #Date Run: > TODAY=`date "+%Y-%m-%d"` > #last month in YYYY/MM > YEAR_LAST_MONTH=`date -v-1d "+%Y/%m"` > #1st of last month as 01-Mon-YYYY > FIRST_LAST_MONTH=`date -v-1d "+01-%b-%Y"` > echo 'TODAY=' ${TODAY} > echo 'YEAR_LAST_MONTH=' ${YEAR_LAST_MONTH} > echo 'FIRST_LAST_MONTH=' ${FIRST_LAST_MONTH} > # get a list of all the mailboxes with at least one real message > doveadm -f tab mailbox status vsize \* 2>/dev/null | > sed -e 1d | sort -k 1,1 | > awk 'BEGIN {FS="\t"} {if ($2 > 0) print $1}' | > while read i > do > echo `date` start ${i} > doveadm mailbox create "ARCHIVE/${YEAR_LAST_MONTH}/${i}" > doveadm -f tab mailbox status messages "${i}" > doveadm move "ARCHIVE/${YEAR_LAST_MONTH}/${i}" mailbox \ > "${i}" BEFORE ${TODAY} SINCE ${FIRST_LAST_MONTH} > doveadm -f tab mailbox status messages "${i}" > echo `date` done ${i} > done > ?64% [ler at thebighonker.lerctr.org:~] $ > > > The Exim config can be provided as well if necessary. > > ler & ler at lerctr.org *ARE THE SAME MAILBOX* > > At the moment it looks as though you have two sets of emails indexed in > solr. One is indexed under username (the one you are running mannually and > apparently the one used by roundcube too, but that's to be verified) and > another set being indexed by autoindex = yes option using the full email > address. Once you've got it working as you require, then you may want to > clean out solr and reindex with just one of them just to reduce volumes. > > Your setup seems to have a mix of users from mysql and from /etc/passwd. > Not sure if your mysql users are all mapped to real users or they have > their own mailboxes with domain included. Your solution will depend on what > you really need and if the setup is working correctly you may not want to > tweak it too much or other things may start breaking. > > The simplest thing that comes to mind (providing your manually indexed > mails show up in roundcube searches) is just to turn off autoindex and > schedule indexing from cron. As things stand I believe autoindex works well > with virtual users username at domain. You'll probably need to start off > with a rescan if you do this. > > If you can swith to all virtual users without local users then probably > you can use autoindex, but I can understand that may not be possible. For > sure I'd recommend doing it in a test environment first so you can be sure > of the configuration and then look at migrating your existing mailboxes > over to virtual users. > > John > > > >-- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx at gmail.com US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106 -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx at gmail.com US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106 -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://dovecot.org/pipermail/dovecot/attachments/20190415/cbbe4dd4/attachment-0001.html>
On 2019/04/15 06:59, Larry Rosenman via dovecot wrote:> forgot to reply all. > > ---------- Forwarded message --------- > From: *Larry Rosenman* <larryrtx at gmail.com <mailto:larryrtx at gmail.com>> > Date: Mon, Apr 15, 2019 at 5:58 AM > Subject: Re: SOLR/Index? > To: John Fawcett <john at voipsupport.it <mailto:john at voipsupport.it>> > > > the local users (myself, my wife, a friend) can authenticate EITHER as <username> or <username>@lerctr.org <http://lerctr.org>. > > switching to all virtual users is NOT going to happen.? > > If I login to roundcube?with <user>@lerctr.org <http://lerctr.org> it finds the autoindexed?mail. > > So, if I make everyone always authenticate as <user>@lerctr.org <http://lerctr.org> we should be fine. > > and change my script to do doveadm?-u <user>@<domain> instead of depending on the local user running the cron job.? > > question: Is there some way to have dovecot change what it sees to be <user>@lerctr.org <http://lerctr.org> when they login as <user>?FWIW, when I switched all users to be virtual to let the handful of people who had been logging in as <username> not have to switch to <username>@example.com I created a second passdb entry which points to different sql.conf file which automatically appends this specific domain, thereby letting them type in <username> but getting logged in as <username>@example.com so my dovecot logs always show <username>@example.com. Not sure if this helps you get around your issue. .jh> On Mon, Apr 15, 2019 at 5:54 AM John Fawcett via dovecot <dovecot at dovecot.org <mailto:dovecot at dovecot.org>> wrote: > > On 15/04/2019 11:38, Larry Rosenman via dovecot wrote: >> ?63% [ler at thebighonker.lerctr.org:~ <mailto:ler at thebighonker.lerctr.org:~>] $ grep fts1970 mail/INBOX >> ?67% [ler at thebighonker.lerctr.org:~ <mailto:ler at thebighonker.lerctr.org:~>] 1 $ mail -s "test fts1970" ler at lerctr.org <mailto:ler at lerctr.org> >> test fts1970 >> >> test fts1970 >> . >> EOT >> [ler at thebighonker.lerctr.org:~ <mailto:ler at thebighonker.lerctr.org:~>] $ mailq >> [ler at thebighonker.lerctr.org:~ <mailto:ler at thebighonker.lerctr.org:~>] $ grep fts1970 mail/INBOX >> Subject: test fts1970 >> test fts1970 >> test fts1970 >> >> >> Apr 15 04:29:03 thebighonker exim[49528]: 1hFxvD-000Csq-P6 <= ler at lerctr.org <mailto:ler at lerctr.org> U=ler P=local S=388 >> Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(49364): Connect from local >> Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(ler at lerctr.org/49364 <http://ler at lerctr.org/49364>): save: box=INBOX, uid=175402, msgid=<E1hFxvD-000Csq-P6 at thebighonker.lerctr.org <mailto:E1hFxvD-000Csq-P6 at thebighonker.lerctr.org>>, size=640, vsize=660, from=Larry Rosenman <ler at lerctr.org <mailto:ler at lerctr.org>>, subject=test fts1970, flags=() >> Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(ler at lerctr.org/49364 <http://ler at lerctr.org/49364>): sieve: msgid=<E1hFxvD-000Csq-P6 at thebighonker.lerctr.org <mailto:E1hFxvD-000Csq-P6 at thebighonker.lerctr.org>>: stored mail into mailbox 'INBOX' (subject=test fts1970 from=ler at lerctr.org <mailto:ler at lerctr.org> size=660) >> Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(49364): Disconnect from local: Client has quit the connection (state=READY) >> Apr 15 04:29:03 thebighonker exim[49535]: 1hFxvD-000Csq-P6 => ler <ler at lerctr.org <mailto:ler at lerctr.org>> R=localuser T=dovecot_lmtp S=404 C="250 2.0.0 <ler at lerctr.org <mailto:ler at lerctr.org>> 6ACWMN9OtFzUwAAAu+mOrA Saved" QT=0s DT=0s >> Apr 15 04:29:03 thebighonker exim[49535]: 1hFxvD-000Csq-P6 Completed QT=0s >> Apr 15 04:29:03 thebighonker dovecot[2507]: indexer-worker(ler at lerctr.org/49366 <http://ler at lerctr.org/49366>): Indexed 1 messages in INBOX (UIDs 175402..175402) >> >> >> ?81% [ler at thebighonker.lerctr.org:~ <mailto:ler at thebighonker.lerctr.org:~>] $ doveadm search mailbox INBOX? body 'fts1970' >> ?83% [ler at thebighonker.lerctr.org:~ <mailto:ler at thebighonker.lerctr.org:~>] $ >> >> >> ?65% [ler at thebighonker.lerctr.org:~ <mailto:ler at thebighonker.lerctr.org:~>] 75 $ doveadm search -u ler at lerctr.org <mailto:ler at lerctr.org>? mailbox INBOX body 'fts1970' >> a53a143be44bda5bd4830000bbe98eac 175402 >> [ler at thebighonker.lerctr.org:~ <mailto:ler at thebighonker.lerctr.org:~>] $ doveadm index -q INBOX >> [ler at thebighonker.lerctr.org:~ <mailto:ler at thebighonker.lerctr.org:~>] $ doveadm search mailbox INBOX? body 'fts1970' >> [ler at thebighonker.lerctr.org:~ <mailto:ler at thebighonker.lerctr.org:~>] $ doveadm fts rescan >> [ler at thebighonker.lerctr.org:~ <mailto:ler at thebighonker.lerctr.org:~>] $ doveadm index -q INBOX >> [ler at thebighonker.lerctr.org:~ <mailto:ler at thebighonker.lerctr.org:~>] $ doveadm search mailbox INBOX? body 'fts1970' >> a53a143be44bda5bd4830000bbe98eac 175402 >> [ler at thebighonker.lerctr.org:~ <mailto:ler at thebighonker.lerctr.org:~>] $ doveadm search -u ler at lerctr.org <mailto:ler at lerctr.org>? mailbox INBOX body 'fts1970' >> a53a143be44bda5bd4830000bbe98eac 175402 >> [ler at thebighonker.lerctr.org:~ <mailto:ler at thebighonker.lerctr.org:~>] $ >> >> So, yes, your hypothesis is correct. >> >> Question: How can I make it consistent??? >> >> I have a script that runs on the first of the month that does archiving, and I have similar issues in that namespace: >> ?67% [ler at thebighonker.lerctr.org:~ <mailto:ler at thebighonker.lerctr.org:~>] $ cat bin/archive-mail >> #!/bin/sh >> PATH=$PATH:/usr/local/bin >> #Expects to be run after midnight on the first of the month >> #? to archive all the previous months mail >> #Date Run: >> TODAY=`date "+%Y-%m-%d"` >> #last month in YYYY/MM >> YEAR_LAST_MONTH=`date -v-1d "+%Y/%m"` >> #1st of last month as 01-Mon-YYYY >> FIRST_LAST_MONTH=`date -v-1d "+01-%b-%Y"` >> echo 'TODAY=' ${TODAY} >> echo 'YEAR_LAST_MONTH=' ${YEAR_LAST_MONTH} >> echo 'FIRST_LAST_MONTH=' ${FIRST_LAST_MONTH} >> # get a list of all the mailboxes with at least one real message >> doveadm -f tab mailbox status vsize \* 2>/dev/null | >> ? ? ? ? sed -e 1d | sort -k 1,1 | >> ? ? ? ? awk? 'BEGIN {FS="\t"} {if ($2 > 0)? print $1}' | >> while read i >> do >> ? ?echo `date` start ${i} >> ? ?doveadm mailbox create "ARCHIVE/${YEAR_LAST_MONTH}/${i}" >> ? ?doveadm -f tab mailbox status messages "${i}" >> ? ?doveadm move "ARCHIVE/${YEAR_LAST_MONTH}/${i}" mailbox \ >> ? ? ? ? ? ? "${i}" BEFORE ${TODAY} SINCE ${FIRST_LAST_MONTH} >> ? ?doveadm -f tab mailbox status messages "${i}" >> ? ?echo `date` done? ${i} >> done >> ?64% [ler at thebighonker.lerctr.org:~ <mailto:ler at thebighonker.lerctr.org:~>] $ >> >> >> The Exim config can be provided as well if necessary. >> >> ler & ler at lerctr.org <mailto:ler at lerctr.org> *ARE THE SAME MAILBOX* >> > At the moment it looks as though you have two sets of emails indexed in solr. One is indexed under username (the one you are running mannually and apparently the one used by roundcube too, but that's to be verified) and another set being indexed by autoindex = yes option using the full email address. Once you've got it working as you require, then you may want to clean out solr and reindex with just one of them just to reduce volumes. > > Your setup seems to have a mix of users from mysql and from /etc/passwd. Not sure if your mysql users are all mapped to real users or they have their own mailboxes with domain included. Your solution will depend on what you really need and if the setup is working correctly you may not want to tweak it too much or other things may start breaking. > > The simplest thing that comes to mind (providing your manually indexed mails show up in roundcube searches) is just to turn off autoindex and schedule indexing from cron. As things stand I believe autoindex works well with virtual users username at domain. You'll probably need to start off with a rescan if you do this. > > If you can swith to all virtual users without local users then probably you can use autoindex, but I can understand that may not be possible. For sure I'd recommend doing it in a test environment first so you can be sure of the configuration and then look at migrating your existing mailboxes over to virtual users. > > John > > > > > > -- > Larry Rosenman? ? ? ? ? ? ? ? ? ?? http://www.lerctr.org/~ler > Phone: +1 214-642-9640 (c) ? ? E-Mail: larryrtx at gmail.com <mailto:larryrtx at gmail.com> > US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106 > > > -- > Larry Rosenman? ? ? ? ? ? ? ? ? ?? http://www.lerctr.org/~ler > Phone: +1 214-642-9640 (c) ? ? E-Mail: larryrtx at gmail.com <mailto:larryrtx at gmail.com> > US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106
On 15/04/2019 12:59, Larry Rosenman via dovecot wrote:> forgot to reply all. > > ---------- Forwarded message --------- > From: *Larry Rosenman* <larryrtx at gmail.com <mailto:larryrtx at gmail.com>> > Date: Mon, Apr 15, 2019 at 5:58 AM > Subject: Re: SOLR/Index? > To: John Fawcett <john at voipsupport.it <mailto:john at voipsupport.it>> > > > the local users (myself, my wife, a friend) can authenticate EITHER as > <username> or <username>@lerctr.org <http://lerctr.org>. > > switching to all virtual users is NOT going to happen.? > > If I login to roundcube?with <user>@lerctr.org <http://lerctr.org> it > finds the autoindexed?mail. > > So, if I make everyone always authenticate as <user>@lerctr.org > <http://lerctr.org> we should be fine. > > and change my script to do doveadm?-u <user>@<domain> instead of > depending on the local user running the cron job.? > > question: Is there some way to have dovecot change what it sees to be > <user>@lerctr.org <http://lerctr.org> when they login as <user>? > >Dovecot is very configurable, but it can also take some time, effort and testing to get the configuration you want. Personally I don't mix user types since it takes out an element of complexity. For your case you might find it useful to look into auth_default_realm that can specify a domain name when one is not supplied. https://wiki2.dovecot.org/DomainLost Other things that might be useful: there is a method for returning a "user" field from the userdb query or passdb query which will change the username. Or there is another setting that can overriding values of fields returned by the userdb. https://wiki.dovecot.org/UserDatabase Hope it helps! John -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://dovecot.org/pipermail/dovecot/attachments/20190415/b8431068/attachment.html>
On 4/15/2019, 6:59:59 AM, Larry Rosenman via dovecot <dovecot at dovecot.org> wrote:> If I login to roundcube?with <user>@lerctr.org <http://lerctr.org> it > finds the autoindexed?mail. > > So, if I make everyone always authenticate as <user>@lerctr.org > <http://lerctr.org> we should be fine.You can configure roundcube to always use the fqdn...