Hello, I have used mb2md for an individual user like this: su ~username cd ~ ./mb2md -m This converted /var/spool/mail/username mbox to /home/users/username/Maildir just as I expected. To convert 100s of users the same way would be a chore (su as each user for permissions, etc.). Is there a way to batch process all /var/spool/mail/$USER to /home/users/$USER/Maildir with the correct permissions/file owners for each user?
This weekend I migrated the university I work for to Dovecot and maildir from UW and mbox. I made some pretty significant changes to mb2md such that it converts keywords (aka, custom flags) and builds the dovecot-uidlist file based on X-IMAP/X-IMAPbase and X-UID headers. Both of these additions are optional though. I was also tasked with converting not just /var/mail/%u, but also /home/%u and all mbox folders for all of users...~30,000 people. The uidlist file it generates is for 0.99.14. I don't know if 1.0 has a different list syntax. I also added the UW uidl patch so Dovecot generates uid numbers the same way UW did. This prevented, most of, our users from noticing we even did anything (except for the 12 hour email downtime ;). In order to convert as quickly as possible we created a wrapper around mb2md that forks mb2md processes for a list of users. The list is in a Berkeley db file. You can specify the maximum number of threads you want to use so you don't overload your system. We ran our conversion on 4 machines, converting 2 file systems (we have 8) per machine, with 32 "threads" per master converter. The backend is an EMC Celerra storage array that we read mboxes from and wrote new maildirs to. We converted all /var/mail and user folders, ~13,000,000 (yes, million) files/messages in about 3.5 to 4 hours. If you would like the source for this system I would be happy to provide it. There is no documentation right now as it was a one time only thing, but I can try to explain. If others are interested just let me know. The whole thing is written in Perl and expects to be run as root. It chowns everything in the final steps. Regards, ------------------------------------------------------------ | Todd Piket | Email: todd at mtu.edu | | Programmer/Analyst | Phone: (906) 487-1720 | | Distributed Computing Services | | | Michigan Technological University | | ------------------------------------------------------------ Netlink Tech wrote:> Hello, > I have used mb2md for an individual user like this: > > su ~username > cd ~ > ./mb2md -m > > This converted /var/spool/mail/username mbox to > /home/users/username/Maildir just as I expected. > > To convert 100s of users the same way would be a chore (su as each user > for permissions, etc.). > > Is there a way to batch process all /var/spool/mail/$USER to > /home/users/$USER/Maildir with the correct permissions/file owners for > each user? >
Since you only have a few hundred user's to convert, why just just write a quick shell wrapper on the command line. Something like this (this is bourne or bash): # for user in `ls /var/spool/mail|egrep -v "root|daemon"`; do \ echo md2md -s /var/spool/mail/$user -d /home/users/$user; \ echo chown -R $user /home/users/$user/Maildir; \ echo chown -R $user /home/users/$user/Maildir; \ echo chmod -R 700 /home/users/$user/Maildir; done Logged in as 'root', that will loop through all files in /var/spool/mail (assuming the filenames are the same as the username) and convert them to Maildir in the userhome, then set the owner to themselves and make them 700 permissions. I haven't tested that, I just typed it out. In fact I've never even used 'md2md', but the a 'for' wrapper is simple enough for anything. So you may want to double check the md2md arguments. You can add user's you don't want to the 'egrep -v' list to ignore, OR, if you'd prefer, take the '-v' out and make it a list of user's to only do. The 'echos' are there to just print what it will do for testing. Remove them when you have it like you want it. Hope that helps. ~Adam On Mon, 3 Apr 2006, Netlink Tech wrote:> Hello, > I have used mb2md for an individual user like this: > > su ~username > cd ~ > ./mb2md -m > > This converted /var/spool/mail/username mbox to > /home/users/username/Maildir just as I expected. > > To convert 100s of users the same way would be a chore (su as each user > for permissions, etc.). > > Is there a way to batch process all /var/spool/mail/$USER to > /home/users/$USER/Maildir with the correct permissions/file owners for > each user? > > >
I did not use the the libc-client Perl bindings...sorry. I used a modified version of mb2md-3.20. It was pretty simple really. If you'd like to look at/dowload the source please visit: http://svn.dcs.it.mtu.edu/email.mtu.edu/convert_tools/tags/mb2md-release-4.0/ (I have my own release numbers for the whole system I wrapped around mb2md). Not a lot of documentation and there is a lot that is specific to my site, but it shouldn't be too hard to decipher and you can always ask me questions, though I'm going to take a well earned next 2 days off. If anyone is interested, the conversion we just performed has been in place for ~5 days now. Only a handful of the ~30,000 users we converted noticed we even did anything (except performance) is better (this was in large part due to the UIDL code I added to mb2md). We are now huge Dovecot fans! Regards, ------------------------------------------------------------ | Todd Piket | Email: todd at mtu.edu | | Programmer/Analyst | Phone: (906) 487-1720 | | Distributed Computing Services | | | Michigan Technological University | | ------------------------------------------------------------ pod wrote:>>>>>> "TP" == Todd Piket <todd at mtu.edu> writes: > > [...] > TP> If you would like the source for this system I would be happy to > TP> provide it. There is no documentation right now as it was a one > TP> time only thing, but I can try to explain. If others are > TP> interested just let me know. The whole thing is written in Perl > TP> and expects to be run as root. It chowns everything in the final > TP> steps. > > I'm definitely interested. What you've done sounds fabulous. I'm going > to be needing to convert our users here at Oxford soon from uw-imap mbx > (not Berkley mbox) to dovecot maildir on a similar scale. > > Did you use the libc-client perl bindings for reading the mbox? If you > did then adapting for mbx should be a piece of cake. If not then I'll > need to do some work; I'm more than happy to submit patches back.