Sam Kuper
2022-Apr-20 06:20 UTC
Bad Signature - Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox
On Tue, Apr 19, 2022 at 07:26:10PM -0600, Shawn Heisey wrote:> I would bet that if you accessed a gmail folder with 5 million > messages in it using IMAP, you would have similar problems with it to > those that have been described here in this thread.? IMAP is a > beautiful protocol, but I don't think it was designed for handling > that many messages.This. Sadly, Mark Crispin (author of IMAP and IMAP2) is no longer with us to confirm. Even at just ~100B for each message's headers, your IMAP2 client would likely need at least ~500MB free RAM to load 5 million mails. By the time Mark stopped working on UW IMAP (the reference IMAP implementation, aka Panda IMAP), c.2010, even top-of-the-range smartphones typically had only ~512MB RAM total, and top-of-the-range ThinkPads had max ~4GiB (which was the upper limit of what 32-bit operating systems, still popular then, could handle). When IMAP2 was invented, c.1988-1990, RAM like that was basically supercomputer territory. Had Mark intended or expected IMAP2 users to have had supercomputers at their disposal, and to be searching such large volumes of mail over the protocol, I suspect he would have designed the protocol differently: for raw efficiency over human readability. My advice for anyone who wants to be able to keep and search very large monolithic mailstores would be: synchronise (e.g. via https://isync.sourceforge.io/ or fetchmail or getmail or rsync or Unison or whatever) those mailstores from the server onto your local filesystem; use maildir on your local filesystem; and use either Mutt's "limiting" functions, or notmuch's index/search functions, for searching/browsing. Good luck in your quest! Sam
Turritopsis Dohrnii Teo En Ming
2022-Apr-20 13:55 UTC
Bad Signature - Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox
> My advice for anyone who wants to be able to keep and search very large > monolithic mailstores would be: synchronise (e.g. via > https://isync.sourceforge.io/ or fetchmail or getmail or rsync or Unison > or whatever) those mailstores from the server onto your local > filesystem; use maildir on your local filesystem; and use either Mutt's > "limiting" functions, or notmuch's index/search functions, for > searching/browsing.Wouldn't it be very tedious and time consuming to sync mailboxes from the server onto our local filesystems? Regards, Mr. Turritopsis Dohrnii Teo En Ming Targeted Individual in Singapore 20 Apr 2022 Wednesday On Wed, 20 Apr 2022 at 14:21, Sam Kuper <sampablokuper at posteo.net> wrote:> > On Tue, Apr 19, 2022 at 07:26:10PM -0600, Shawn Heisey wrote: > > I would bet that if you accessed a gmail folder with 5 million > > messages in it using IMAP, you would have similar problems with it to > > those that have been described here in this thread. IMAP is a > > beautiful protocol, but I don't think it was designed for handling > > that many messages. > > This. Sadly, Mark Crispin (author of IMAP and IMAP2) is no longer with > us to confirm. > > Even at just ~100B for each message's headers, your IMAP2 client would > likely need at least ~500MB free RAM to load 5 million mails. > > By the time Mark stopped working on UW IMAP (the reference IMAP > implementation, aka Panda IMAP), c.2010, even top-of-the-range > smartphones typically had only ~512MB RAM total, and top-of-the-range > ThinkPads had max ~4GiB (which was the upper limit of what 32-bit > operating systems, still popular then, could handle). > > When IMAP2 was invented, c.1988-1990, RAM like that was basically > supercomputer territory. > > Had Mark intended or expected IMAP2 users to have had supercomputers at > their disposal, and to be searching such large volumes of mail over the > protocol, I suspect he would have designed the protocol differently: for > raw efficiency over human readability. > > My advice for anyone who wants to be able to keep and search very large > monolithic mailstores would be: synchronise (e.g. via > https://isync.sourceforge.io/ or fetchmail or getmail or rsync or Unison > or whatever) those mailstores from the server onto your local > filesystem; use maildir on your local filesystem; and use either Mutt's > "limiting" functions, or notmuch's index/search functions, for > searching/browsing. > > Good luck in your quest! > > Sam