sup talk - Oct 2007 - Question regarding Maildir (and possible patch?)

Sup looks great!

Tonight, I tried to configure it for a pretty large Maildir setup (about 250,000
messages and nearly 1,500 mailboxes for one user). When I went to import (ie -
index) these messages, it failed. As I looked into it, I realized that sup was
only looking at the top-level Maildir cur and new folders for messages. In my
dovecot setup, each mailbox has it''s own subdirectory, containing its
own Maildir cur and new directories. Since it seems sup is only looking at the
top-level Maildir cur and new directories, none of my mailboxes are indexed.
Since I prefer not to manually add all 1,500 mailboxes as "sources"
for sup, I modified maildir.rb as follows to allow sup to dive into mailboxes.

I''ve not fully tested this (it''s still indexing messages!),
but I wanted to check to see if this approach is okay or ill advised.

Cheers,

Steve



==[ maildir.rb ]=
+ require ''find''

  def scan_mailbox
    return if @last_scan && (Time.now - @last_scan) < SCAN_INTERVAL

-   cdir = File.join(@dir, ''cur'')
-   ndir = File.join(@dir, ''new'')

-   raise FatalSourceError, "#{cdir} not a directory" unless
File.directory? cdir
-   raise FatalSourceError, "#{ndir} not a directory" unless
File.directory? ndir
+   raise FatalSourceError, "#{@dir} not a directory" unless
File.directory? @dir

    begin
      @ids, @ids_to_fns = @mutex.synchronize do
        ids, ids_to_fns = [], {}
-       (Dir[File.join(cdir, "*")] + Dir[File.join(ndir,
"*")]).map do |fn|
+       Find.find(@dir) do |fn|
+         next unless File.file?(fn)
+         next unless File.basename(File.dirname(fn)) =~ /^cur|new$/
          id = make_id fn
          ids << id
          ids_to_fns[id] = fn
        end
        [ids.sort, ids_to_fns]
      end
    rescue SystemCallError, IOError => e
      raise FatalSourceError, "Problem scanning Maildir directories:
#{e.message}."
    end

    @last_scan = Time.now
  end
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/sup-talk/attachments/20071006/eef0d815/attachment-0001.html

Excerpts from steve.shreeve''s message of Sat Oct 06 01:34:55 -0700
2007:
> Since I prefer not to manually add all 1,500 mailboxes as
"sources"
> for sup, I modified maildir.rb as follows to allow sup to dive into
> mailboxes.
This is a very nice solution, but after some thought I''m not going to
apply it. Maildir folders should be handled the same way that mbox
folders are: each as its own source. If you have lots of them, use
sup-add from the command line in conjunction with shell globbing or the
find command.

The reason is that per-folder auto-applied labels are still important
(at least until I get filtering working), and you lose that capability
with recursion. One could make recursion a configurable property of a
Maildir source, though it strikes me as possibly overengineering.
Depends how many people would actually use it.

When this issue has come up before, my proposed solution has been to
modify sup-config to do the recursion for you at setup time. This is not
a high priority change since my target audience is the Unix-savvy.

In your case, 1500 folders is a lot, but Sup should be able to handle
it. I recommend adding them as unpolled sources---unless, of course,
they do receive new mail---so as to avoid slowing down new message
polling. (In Sup terminology these are (regrettably) called "unpolled"
sources.)

Please don''t let this discourage you from sending more patches, of
course! :)

-- 
William <wmorgan-sup at masanjin.net>