Hello, I'm looking for an application/script/etc. which will allow me to recreate dovecot indexes by hand from shell. I'm preparing myself for migration to dovecot 1.0.rc15 (debian stable) and would like to create indexes before users start to login into new server to ease the load. My other option is to write a script which will log as each user and create indexes, however this seems a bit of painful way to do it. Anybody can point me in right direction? -- Maciej Poszywak
Gabriel Millerd wrote:> On Dec 6, 2007 6:35 AM, Maciej Poszywak <jet at surreal.pl> wrote: > >> I'm looking for an application/script/etc. which will allow me to recreate dovecot indexes by hand from shell. I'm preparing myself for migration to dovecot 1.0.rc15 (debian stable) and would like to create indexes before users start to login into new server to ease the load. My other option is to write a script which will log as each user and create indexes, however this seems a bit of painful way to do it. Anybody can point me in right direction? >> >> > > I have a perl script that will rip through all the users using imap > and touch every folder (or every subscribed folder). if you dont have > the authentication for a proper login you can always setup a temporary > method that returns "username, 'static password', and homedir" before > going live. > > Dunno if this was what you were asking. > >That's what I'm looking for. It would be nice if you sent me that script. -- Maciej Poszywak
On Thu, 2007-12-06 at 13:35 +0100, Maciej Poszywak wrote:> I'm looking for an application/script/etc. which will allow me to > recreate dovecot indexes by hand from shell. I'm preparing myself for > migration to dovecot 1.0.rc15 (debian stable)Post-v1.0.0 releases would be better, there have been a lot of fixes since then. backports.org for example has newer releases.> and would like to create > indexes before users start to login into new server to ease the load. > My other option is to write a script which will log as each user and > create indexes, however this seems a bit of painful way to do it. > Anybody can point me in right direction?I guess you're using maildirs? And you're migrating existing users' mailboxes? Do your users use actual IMAP clients or webmail? I'm not sure if you should worry about creating the indexes. If your users use IMAP clients, just make sure you preserve UIDVALIDITY and UIDs for messages, otherwise most of the clients will download all mails again. If you use webmail, there's some point in placing fields it accesses to dovecot.index.cache files. The only way to do this is to execute IMAP FETCH commands that accesses those fields. If you were thinking about just generating dovecot-uidlist and the main dovecot.index file, don't bother. Their creation is practically free. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: <http://dovecot.org/pipermail/dovecot/attachments/20071206/15c7a2a6/attachment-0002.bin>
Timo Sirainen wrote: > I guess you're using maildirs? And you're migrating existing users > mailboxes? Do your users use actual IMAP clients or webmail? POP3 and webmail. > I'm not sure if you should worry about creating the indexes. If your > users use IMAP clients, just make sure you preserve UIDVALIDITY and UIDs > for messages, otherwise most of the clients will download all mails > again. UIDs are already taken care of and thoroughly tested. > If you were thinking about just generating dovecot-uidlist and the main > dovecot.index file, don't bother. Their creation is practically free. It's not for a mailbox which has 30k+ messages and inbox around 600MB in size. Takes a while to build the index, I have tested this already. First login and index creation takes around 4-6 minutes, then next one just few seconds. -- Maciej Poszywak
El Thursday 06 December 2007 13:35:26 Maciej Poszywak escribi?:> My other option is to write a script which will log as each user and create > indexes, however this seems a bit of painful way to do it.That's almost the way I did; using a master user is quite simple, and even if it's no optimal, I did going into production, so time was not an issue. I used the attached perl script, where $pop3_password is the password for the master user /home/lgstosej/tmp/passwd is a copy of /etc/passwd HTH -- Joseba Torre. CIDIR Bizkaia. -------------- next part -------------- A non-text attachment was scrubbed... Name: crea_indices.pl Type: application/x-perl Size: 780 bytes Desc: not available URL: <http://dovecot.org/pipermail/dovecot/attachments/20071210/fd9bb9c5/attachment-0004.bin> -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. URL: <http://dovecot.org/pipermail/dovecot/attachments/20071210/fd9bb9c5/attachment-0005.bin>
>> Timo Sirainen wrote: >> >>> I guess you mean logging in with POP3? There the slowdown comes from >>> getting all messages' virtual sizes. So your preindexer could probably >>> do this by running pop3 for all users. If you use only a single UID for >>> users, you could do something like: >>> >>> for user in `cat users`; do >>> export MAIL=maildir:/home/$user/Maildir >>> echo "quit" | /usr/local/libexec/dovecot/pop3 >>> done >>> >>> >>> >> vmu1 at test:~$ env | grep MAIL >> MAIL=maildir:/var/mail/testuser >> vmu1 at test:~$ echo "quit" | /usr/lib/dovecot/pop3 >> pop3(vmu1): Fatal: pop3_uidl_format setting is missing from config file >> >> I have pop3_uidl_format correctly defined in the config file. I've also >> tried to strace pop3, to find where it does look for it, but with no >> success. >> > > It gets it from environment: > > POP3_UIDL_FORMAT=... pop3 > > In general all settings are passed in environment using uppercased > names. > >Are you sure that all variables can be read from environment ? vmu1 at mail:~$ env | grep -E "(MMAP|LOCK)" MMAP_DISABLED=yes MAIL_READ_MMAPED=no LOCK_METHOD=dotlock POP3_LOCK_SESSION=yes vmu1 at mail:~$ echo "quit" | /usr/lib/dovecot/pop3 pop3(vmu1): Fatal: lock_method=dotlock and mmap_disable=no combination isn't supported. You don't _really_ want it anyway. vmu1 at mail:~$ /usr/sbin/dovecot --version 1.0.rc15 -- Maciej Poszywak