sup talk - Jul 2009 - Handling big mailing lists

I''m searching alternatives for my Gnus email + news setup. I
don''t
expect Sup to be able to do everything Gnus can yet, but maybe in
future? :) Note that I have never used Gmail.

I tried to test Sup, but I wasn''t able to get any emails because Sup
failed to login to my IMAP server. This happened because the server
requires client certificates and it looks like Sup doesn''t support
those.

The bigger question is about handling big amount of mailing list mail. 
I''m reading many small mailing lists and some big ones. Biggest is
linux-kernel. I can split this to two: daily reading routine and
expiration.

How would my daily reading routine work with Sup? I want to read
things in priority order:

1. Personal email
2. Emails related to my programming hobby
3. Emails related to some associations like user groups
4. Mailing lists, also in priority order

The order is such that if I need to stop reading, I have read the most
important ones already. The order is not static, I want to be able to
change it.

I have understood that labels are way to get this kind of grouping. 
Can I get a view where the labels are sorted like this? And can I
continue reading the next label in that order after I have finished
the one before?

Then about the expiration. The linux-kernel mailing list gets so much
email that some kind of expiration is a must. Can Sup do such
automatic deleting of old emails? Can Sup handle some other process
doing such automatic deletion? (I would actually prefer some other
process do it.)

I''m mostly reading just only few authors from linux-kernel and skipping
rest, but I do like to see the rest in the thread view as sometimes some
subject is interesting and sometimes some thread gets big and is
interesting because of that. How would this work?

I would actually recommend reading linux-kernel to test Sup. :)

-- 
Ilpo Nyyss?nen # biny # /* :-) */

You can use hooks to add labels:

my "~/.sup/hooks/before-add-message.rb":

  def matchR(email)
    not message.recipients.find{ |to|  /^#{email}$/i =~ to.email}.nil?
  end

  def matchRAddLabel(email, label, inbox = 0)
    if matchR email then
      message.add_label label 
      message.remove_label :inbox unless inbox
    end
  end

  def importantFrom(email)
    message.add_label :Starred if message.from.email == email
  end

  importantFrom "info at webkos.de"


  matchRAddLabel("mod_python at modpython.org","mod-python",
1)
  matchRAddLabel("mod_python at modpython.org","mod-python",
0)

  if message.subj =~ /^Project Notification$/ && message.from.email ==
"info at guru.com" then
    message.add_label "GURU_PROJECT_NOTIFICATION"
    message.add_label "delete_after_one_month"
  end

  if message.subj =~ /^WEBKOS_/ then
    message.remove_label :inbox
    message.add_label "WEBKOS_"
  end


So adding labels onle if a contsraint is met is easy.
About deleting mails: There is sup-sync-back. So there must be a way to delete
"old" / whatever mails. I haven''t used it yet.

I''m new to sup myself.

Yours
Marc Weber

Hi,

Excellent set of questions. Handling large volumes of mail is one of my
main goals with Sup.

Reformatted excerpts from iny+dev''s message of
2009-07-28:
> I tried to test Sup, but I wasn''t able to get any emails because
Sup
> failed to login to my IMAP server. This happened because the server
> requires client certificates and it looks like Sup doesn''t support
> those.
As far as IMAP goes, Sup supports whatever authentication the Ruby IMAP
libraries supports, which is probably not much.

Any serious use of Sup with IMAP is best done via an intermediate like
offlineimap, which syncs an IMAP folder to a local Maildir folder. The
Ruby IMAP libraries, and possibly IMAP itself, is a little too slow for
how Sup likes to work.
> How would my daily reading routine work with Sup? I want to read
> things in priority order:
> 
> 1. Personal email
> 2. Emails related to my programming hobby
> 3. Emails related to some associations like user groups
> 4. Mailing lists, also in priority order
Sup gives you two tools: search and labels. Sup will present a list of
threads that match any search query. So, each of those steps is
possible, to the extent that you can compose a search query that
reflects it.

You can automatically add labels via arbitrary code, so you have a fair
amount of flexibility here.
> The order is such that if I need to stop reading, I have read the most
> important ones already. The order is not static, I want to be able to
> change it.  I have understood that labels are way to get this kind of
> grouping.  Can I get a view where the labels are sorted like this?
Sup currently only orders chronologically. It would not be difficult to
add a second level of user-defined sorting *on top* of the chronological
one, but it would be difficult, if not impossible, to order all email in
the index by an arbitrary function.

(To be pedantic, if you can come up with a single number for each email,
which never changes, and which is known at add time, you can order by
that, with some work. But it doesn''t sound like that''s what
you want.)
> And can I continue reading the next label in that order after I have
> finished the one before?
Certainly, but not automatically. You can view one label, and then pick
another label to view, etc.

I suppose if the above second-round of ordering were implemented, you
could view both labels at once and make an ordering function that placed
one above the other.
> Then about the expiration. The linux-kernel mailing list gets so much
> email that some kind of expiration is a must.
You don''t want to just buy a bigger hard drive?
> Can Sup do such automatic deleting of old emails? Can Sup handle some
> other process doing such automatic deletion? (I would actually prefer
> some other process do it.)
That''s best left to another process. You''ll then have to tell
Sup which
messages were deleted so that it can remove them from the index. Let me
know when you''re at this point and I can help you---it will require a
brief Ruby script.
> I would actually recommend reading linux-kernel to test Sup. :)
I read ruby-talk, which is probably not too far off.

One last comment: large threads are irritatingly slow to create in
thread-view-mode with the Ferret index, but there''s a new Xapian index
which is fast for this. It''s still slightly experimental.
-- 
William <wmorgan-sup at masanjin.net>

William Morgan <wmorgan-sup at masanjin.net> writes:
> Sup currently only orders chronologically. It would not be difficult to
> add a second level of user-defined sorting *on top* of the chronological
> one, but it would be difficult, if not impossible, to order all email in
> the index by an arbitrary function.
I''m not talking about ordering of emails, I''m talking about
ordering of
the labels.

When I start my daily routine, I want to see a list of labels with
numbers telling how many unread emails each has. Labels containing
nothing new must be hidden. And this list must be sorted to priority
order. Then I want to read those labels through in that order.

How many labels would I have? At least tens, probably over hundred.
>> And can I continue reading the next label in that order after I have
>> finished the one before?
>
> Certainly, but not automatically. You can view one label, and then pick
> another label to view, etc.
Picking one from alphabetically sorted list or writing it is not usable. 
I don''t remember the order, I just want to get the next one.
>> Then about the expiration. The linux-kernel mailing list gets so much
>> email that some kind of expiration is a must.
>
> You don''t want to just buy a bigger hard drive?
It''s not a disk space problem. The problem is that many things would
get
slow (or I would need to configure exceptions) and I just do not have
any need for them. They are all in searchable mailing list archives
anyway.

And another point is that I actually prefer not to order mailing lists. 
It is much easier to just get them with nntp from news.gmane.org. How? I
have a nntp to imap gateway and I use it to read also from other nntp
servers. The thing here is that nntp servers usually do expire messages
and so to be able to use that with Sup, it would need to tolerate the
expiration.
> That''s best left to another process. You''ll then have to
tell Sup which
> messages were deleted so that it can remove them from the index.
So, I don''t do the expiration. I guess it should be possible to find
out
the ones that went away when checking for new ones.
> Let me know when you''re at this point and I can help you---it will
> require a brief Ruby script.
Well, it is looking like in current state I won''t even start trying.
>> I would actually recommend reading linux-kernel to test Sup. :)
>
> I read ruby-talk, which is probably not too far off.
Hmm,

http://dir.gmane.org/gmane.comp.lang.ruby.general
http://dir.gmane.org/gmane.linux.kernel

I guess so. How long it takes to index few months of ruby-talk?

Could Sup use IMAP search features?

-- 
Ilpo Nyyss?nen # biny # /* :-) */

Reformatted excerpts from iny+dev''s message of
2009-07-29:
> I guess so. How long it takes to index few months of ruby-talk?
Offhand I see index speeds of 50-80 messages/s depending on index size,
backend, etc.
> Could Sup use IMAP search features?
No, that would require a very different architecture and philosophy.
-- 
William <wmorgan-sup at masanjin.net>