Hey all, I''ve got a quick question about how sup handles Maildir. I''m going to be migrating to a new machine soon, and I''ve got a couple of questions about it. 1. My mail is spread out in hundreds of Maildir directories, most of them "legacy" directories that I want to merge as I migrate. Can I do this by just copying the mail files from the various "new" and "cur" directories to a central "cur" directory? I know that Maildir uses unique names, so this shouldn''t clobber any messages, but I haven''t seen any information on whether this could cause other kinds of problems. 2. A more sup-related question: what things do I need to copy to migrate my labels, read/archived/starred status information, etc? I definitely want to maintain as much of my mail''s metadata as I can. I guess this probably interacts with the previous question, too; in the end, I would much prefer to keep my metadata than to merge Maildir directories if they are mutually exclusive. 3. I''m assuming I won''t lose anything if I update sup. I think this topic deserves a wiki page, so I''ll write up a summary of any advice I get as well as how my experience goes for future posterity. Thanks! ~d
Hi Daniel, Sorry for the delay in replying. I''ve been moving across country and only have sporadic internet access right now. Reformatted excerpts from Daniel Wagner''s message of 2008-11-19:> 1. My mail is spread out in hundreds of Maildir directories, most of > them "legacy" directories that I want to merge as I migrate. Can I do > this by just copying the mail files from the various "new" and "cur" > directories to a central "cur" directory?I *think* that just moving files from cur/ to cur/ and new/ to new/ will work, but moving from new/ to cur/ I think typically changes the filename slightly. I don''t know if the rename is required or optional though. Sup certainly doesn''t care, but other tools might.> 2. A more sup-related question: what things do I need to copy to > migrate my labels, read/archived/starred status information, etc? I > definitely want to maintain as much of my mail''s metadata as I can.Since you''re doing lots of moving things around, combining sources, and upgrading Sup at the same time, I think the best option is going to be to rebuild your index from scratch. You can use sup-dump to dump out state information into a huge text file, and sup-sync --restored --restore should be able to make use of that information to keep your message state once your new sources have been created. But definitely, DEFINITELY test it out first. I haven''t tried that stuff for a while and it might have experienced bitrot.> 3. I''m assuming I won''t lose anything if I update sup.Nope. And I can''t believe you''re still using Sup 0.4!> I think this topic deserves a wiki page, so I''ll write up a summary of > any advice I get as well as how my experience goes for future > posterity.That sounds great. Let us know how it goes, and I''ll do my best to help you though the process if things break. Just be sure to keep a backup of everything. Message state is a precious asset. -- William <wmorgan-sup at masanjin.net>
Much thanks to Nicolas and William. I''m just entering final projects season in school, so I may put off this project about a week or two longer, but will keep these messages +inbox until then. =) ~d
Excerpts from William Morgan''s message of Wed Nov 26 19:05:13 -0500 2008:> I *think* that just moving files from cur/ to cur/ and new/ to new/ will > work, but moving from new/ to cur/ I think typically changes the > filename slightly. I don''t know if the rename is required or optional > though. Sup certainly doesn''t care, but other tools might.As you said, sup didn''t care a jot. Lovely! Since I''m not using any other tools for my mail, this is just what I will do.> to rebuild your index from scratch. You can use sup-dump to dump out > state information into a huge text file, and sup-sync --restored > --restore should be able to make use of that information to keep your > message state once your new sources have been created. > > But definitely, DEFINITELY test it out first. I haven''t tried that stuff > for a while and it might have experienced bitrot.Of *course* I''m going to test first. I don''t trust software any more since I''ve seen how it''s made. =)> That sounds great. Let us know how it goes, and I''ll do my best to help > you though the process if things break. Just be sure to keep a backup of > everything. Message state is a precious asset.I have a project due tomorrow, so naturally I figured I''d spend a few hours playing with this stuff tonight. For the most part, I''ve been very pleased -- things have gone relatively smoothly compared to my expectations! I sup-dumped, copied all the files in oldmachine/Maildir/**/cur and oldmachine/Maildir/**/new to newmachine/Maildir/cur, started up sup-sync --restored --restore, then went and made dinner. Now for some numbers: The Numbers According To sup-sync entries in the dump file 26462 entries scanned 24779 entries added 24373 warnings 27 fakes 75 The Numbers According To sup old machine new machine messages in the index 26459 24416 messages with +inbox 50 45 messages with +Starred 67 57 The warnings took the form warning: error (Iconv::InvalidEncoding) decoding message body from XXX: invalid encoding ("utf8//IGNORE", "XXX") where "XXX" was "unicode-1-1utf7" 17 times, "unknown-8bit" 9 times, and "X-UNKNOWN" 1 time. It also faked message-ids 68 times, from headers 3 times, and date headers 4 times. So, I''m not sure I understand how to reconcile these numbers. Things are mismatched in odd ways; for example, how could there be more entries in the dump than there are messages in the index? Or how come "entries added" + "warnings" != "messages in the index"? That''s okay, though. There are people for whom impermanence is a way of life. There are a few categories of messages for which I really, really want the impermanence to start later, though -- specifically +inbox and +Starred messages. Since there''s only 15 of these lost, is there some way I can grab/index those manually? ...or a way to help identify and fix the underlying problem? ~d P.S. Sorry for all the details. I realize this is probably much more interesting to me than to any of you, since it affects me more directly, but I just couldn''t help myself. =P
Excerpts from Daniel Wagner''s message of Mon Dec 08 00:14:39 -0500 2008:> The Numbers According To sup-sync > entries in the dump file 26462 > entries scanned 24779 > entries added 24373 > warnings 27 > fakes 75 > > The Numbers According To sup > old machine new machine > messages in the index 26459 24416Ah, I just realized: copying everything to one directory is fine only if the filenames have unique names. A quick investigation shows I probably nuked around 500-600 messages with this assumption, so that''s one easily fixable problem. But that still leaves ~1500 messages in the aether... Love! ~d
Reformatted excerpts from Daniel Wagner''s message of 2008-12-07:> entries in the dump file 26462What is this number from? Is it a wc -l on the dumpfile, or sup-sync''s "Restored state on X messages" message? If the latter, what is wc -l? (And if the former, what''s the other number?)> entries scanned 24779 > entries added 24373 > warnings 27 > fakes 75Warnings and fakes we can ignore. (They still count as being added.) Did sup-sync report a number of updates? It should be the case that num adds + num updates = num scanned, and num adds = index size (when starting from an empty index).> The Numbers According To sup > old machine new machine > messages in the index 26459 24416 > messages with +inbox 50 45 > messages with +Starred 67 57 > > So, I''m not sure I understand how to reconcile these numbers. Things > are mismatched in odd ways; for example, how could there be more > entries in the dump than there are messages in the index?Yes, that is weird. 26462 vs 26459 is what concerns me. How did three new messages appear at dump time?> Or how come "entries added" + "warnings" != "messages in the index"?Messages with warnings or that needed fake headers should still be added, so the question is really 24416 vs 24373. That''s 43 new messages that appeared in the index.> That''s okay, though. There are people for whom impermanence is a way > of life.Uh oh.> There are a few categories of messages for which I really, really > want the impermanence to start later, though -- specifically +inbox and > +Starred messages. Since there''s only 15 of these lost, is there some > way I can grab/index those manually?The most direct way is to query the index through devel/console.sh. You can find examples on the wiki or in the mailing list archives. Something like Index.ferret.search("label:inbox", :limit => :all).hits.map { |h| Index.ferret[h.doc] } will give you all the Ferret records with the index label, and you can then query them for all the fields listed in index.rb. (They''ll appear as empty hashes, {}, when irb prints them out, but in reality they''re just lazily-filled. That should give you the information you need to compare the two sets of documents on both indexes, and figure out what the differences are. -- William <wmorgan-sup at masanjin.net>