Aki Tuomi <aki.tuomi at dovecot.fi> writes:>> On 24 February 2018 at 15:47 Neil Jerram <neil at ossau.homelinux.net> wrote:[...]>> > Feb 24 10:24:24 arudy dovecot[1712]: imap(neil): Warning: >> > Transaction log file >> > /home/neil/dovecot-mail/.imap/INBOX/dovecot.index.log was locked >> > for 98 seconds (Mailbox was synchronized)[...]> You are using mbox format. This is ... bit slow. =) > > When you move mails between mbox files, it has to rewrite the entire mbox file every time. You should probably start using maildir or sdbox instead.Ah, right, thanks. My INBOX file has 22990 messages. Is the slowness that I am seeing definitely expected for an mbox of that size? (It may also be relevant that the HDD it's stored on is pretty old now, and has been known to report SMART errors...) If so, I'll start looking at how to migrate, given that my system is Postfix + Dovecot. If you have any particular recommendations or migration pointers for a system like that, I'd appreciate them. Best wishes - Neil
> On 24 February 2018 at 16:04 Neil Jerram <neil at ossau.homelinux.net> wrote: > > > Aki Tuomi <aki.tuomi at dovecot.fi> writes: > > >> On 24 February 2018 at 15:47 Neil Jerram <neil at ossau.homelinux.net> wrote: > > [...] > >> > Feb 24 10:24:24 arudy dovecot[1712]: imap(neil): Warning: > >> > Transaction log file > >> > /home/neil/dovecot-mail/.imap/INBOX/dovecot.index.log was locked > >> > for 98 seconds (Mailbox was synchronized) > [...] > > > You are using mbox format. This is ... bit slow. =) > > > > When you move mails between mbox files, it has to rewrite the entire mbox file every time. You should probably start using maildir or sdbox instead. > > Ah, right, thanks. > > My INBOX file has 22990 messages. Is the slowness that I am seeing > definitely expected for an mbox of that size? (It may also be relevant > that the HDD it's stored on is pretty old now, and has been known to > report SMART errors...) > > If so, I'll start looking at how to migrate, given that my system is > Postfix + Dovecot. If you have any particular recommendations or > migration pointers for a system like that, I'd appreciate them. > > Best wishes - NeilYes. You deffo are looking at several reasons for slowness. I can only recommend moving into maildir or sdbox format, and probably new HDD too. https://wiki2.dovecot.org/Tools/Doveadm/Sync here is example of 'converting' between mailbox formats using dsync. You should also read https://wiki2.dovecot.org/Migration/MailFormat mbox format has been known to act up with dsync occasionally, so I recommend using doveadm backup maildir:~/Maildir if you want to give it a try, instead of doveadm sync. Backup does dsync too, but it only works one way. Aki
Den 24. feb. 2018 15:04, skrev Neil Jerram:> Aki Tuomi <aki.tuomi at dovecot.fi> writes: > >>> On 24 February 2018 at 15:47 Neil Jerram <neil at ossau.homelinux.net> wrote: > > [...] >>>> Feb 24 10:24:24 arudy dovecot[1712]: imap(neil): Warning: >>>> Transaction log file >>>> /home/neil/dovecot-mail/.imap/INBOX/dovecot.index.log was locked >>>> for 98 seconds (Mailbox was synchronized) > [...] > >> You are using mbox format. This is ... bit slow. =) >> >> When you move mails between mbox files, it has to rewrite the entire mbox file every time. You should probably start using maildir or sdbox instead. > > Ah, right, thanks. > > My INBOX file has 22990 messages. Is the slowness that I am seeing > definitely expected for an mbox of that size? (It may also be relevant > that the HDD it's stored on is pretty old now, and has been known to > report SMART errors...) > > If so, I'll start looking at how to migrate, given that my system is > Postfix + Dovecot. If you have any particular recommendations or > migration pointers for a system like that, I'd appreciate them. >I'd go with whatever tools you are familiar with. If you don't know where to start, formail(1) can read an mbox and do whatever for each mail contained therein. This, together with procmail, used to be the go-to tools in the days before IMAP. Theese days you'd probably want to involve your local delivery agent on the output from formail. The lda would invoke sieve instead of procmail if that is your thing. Whatever you do, try to set up so you can do some tests before you blast 22000 mails to somewhere you do not want them :-)
Aki Tuomi <aki.tuomi at dovecot.fi> writes:> Yes. You deffo are looking at several reasons for slowness. > > I can only recommend moving into maildir or sdbox format, and probably new HDD too. > > https://wiki2.dovecot.org/Tools/Doveadm/Sync here is example of 'converting' between mailbox formats using dsync. You should also read https://wiki2.dovecot.org/Migration/MailFormat > > mbox format has been known to act up with dsync occasionally, so I recommend using > > doveadm backup maildir:~/MaildirThanks, I've done that now, and things are looking much better. I rediscovered that I've configured postfix to deliver locally using dovecot-lmtp - which meant that I then only needed to change dovecot's mail_location setting, and nothing at all in the postfix config. Many thanks! - Neil
On 2018-02-24 (07:04 MST), Neil Jerram <neil at ossau.homelinux.net> wrote:> > My INBOX file has 22990 messages. Is the slowness that I am seeing > definitely expected for an mbox of that size? (It may also be relevant > that the HDD it's stored on is pretty old now, and has been known to > report SMART errors...)back int he dark ages I would send and alert to users if their inbox for over 1,000 messages because an mbox that large drove the server to its knees and made merely logging in to mail take an excruciating amount of time. If they didn't fix it I'd archive their inbox and start over. I am astonished your machine can process an mbox with over 22 times that many messages. https://wiki1.dovecot.org/Migration/MailFormat That page doesn't exist on the wiki for dovecot 2, but that script to convert mbox to milder should still work. Obviously, keep backups and such. -- Did they get you to trade your heroes for ghosts? Hot ashes for trees? Hot air for a cool breeze? Cold comfort for change?
On 2018-02-24 (07:14 MST), Aki Tuomi <aki.tuomi at dovecot.fi> wrote:> > https://wiki2.dovecot.org/Migration/MailFormatThat didn't show up when searching wiki2 for "Migration" :/ -- "...Life is not a journey to the grave with the intention of arriving safely in one pretty and well-preserved piece, but to slide across the finish line broadside, thoroughly used up, worn out, leaking oil, and shouting GERONIMO!!!" -- Bill McKenna
On Sat, 24 Feb 2018, Neil Jerram wrote:> My INBOX file has 22990 messages. Is the slowness that I am seeing > definitely expected for an mbox of that size? (It may also be relevant > that the HDD it's stored on is pretty old now, and has been known to > report SMART errors...)Yeah, a copy of that mailbox will be that slow, esp. if the messages have large attachments. Even a simple operation like deleting/expunging the the first message will cause data shuffling of the entire mailbox. Joseph Tam <jtam.home at gmail.com>