Paul Kudla (SCOM.CA Internet)
2022-Apr-18 09:44 UTC
Bad Signature - Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox
this is not the answer you are looking for but comes from years and years of experience with mail servers in general (cyrus first then dovecot) First off imap clients can handle emails counts but vary on performance 1. outlook for example is completely useless above 1000 count per folder/sub folder, imap was kinda supported to outlook 2010 but after 2012 fell off largely due to microsoft pushing their sync & exchange servers. 2. thunderbird will and can handle 10000~30000 emails but anything above 10000 really should be split into a subfolder, i usually take the inbox and split into the year of the messages to keep into a subfolder like 2020,2021 etc 3. apple mail and iphone related seem to handle large emails boxes ok but will slow down above the 1000+ per folder limit 4. android you need to run bluemail or something like that but again limits are like the iphone for around 1000 per folder, generic imap client does not work well with android. As for the 200,000+ emails in the inbox no email system was ever designed for that - ever. It needs to be understood that pending how your server hdd's are formatted (nfs, ufs, zfs etc) will directly effect the speed of email retreivals. nfs being the slowest (dependant on network speeds, server speeds etc) ufs is the old style unix partitions, ufs works ok but you need to tweek certain settings for how many inodes can be in memory at any one time to i believe a max of 10,000 (freebsd, other systems will vary) zfs is the best as it will store dir inode etc into? memory in the system so large file numbers in directories have a faster chance to respond, again system dependant suggest lots of memory no system will support 200,000 + emails, even if the server can handle that and running imap where you only download the headers the email client would just spin trying to update the email box constantantly. forget about trying to index searches etc. internet speed (even a 100M fibre connection) is also a factor. this is a sample dir structure from my server 1 email = 1 file in a sub dir. the operating system is where you are choking on its ability to handle 200,000 files in one directory is just unsustainable. if you have to handle that many incoming emails i strongly suggest a sieve filter that automatically moves incoming emails into a folder based on the email to keep the counts under control. I find dovecot can handle email counts better then the old cyrus but when i ran cyrus i was using ufs file system as well, when i upgraded to dovecot i had also upgraded to zfs. # cd cur [05:39:14] mail18.scom.ca [root:0] /data/dovecot/users/scom.ca/paul at scom.ca/cur # ll total 77333 drwx------??? 2 vmail? vmail? uarch? 361B Apr 18 05:35 . drwx------? 947 vmail? vmail? uarch? 963B Apr 18 05:36 .. -rw-------??? 1 vmail? vmail? uarch? 6.4K Feb 20 08:05 1645362302.M163606P37919.mail18.scom.ca,S=6536,W=6687:2,FS -rw-------??? 1 vmail? vmail? uarch? 8.9K Feb 20 08:05 1645362302.M535115P37919.mail18.scom.ca,S=9074,W=9224:2,S -rw-------??? 1 vmail? vmail? uarch? 5.5K Feb 20 08:05 1645362303.M133625P37919.mail18.scom.ca,S=5632,W=5729:2,S -rw-------??? 1 vmail? vmail? uarch?? 11K Feb 20 08:05 1645362305.M954336P38787.mail18.scom.ca,S=11438,W=11642:2,S -rw-------??? 1 vmail? vmail? uarch? 1.9K Feb 20 08:05 1645362305.M954355P38787.mail18.scom.ca,S=1912,W=1965:2,S -rw-------??? 1 vmail? vmail? uarch? 2.3K Feb 20 08:05 1645362306.M417932P37919.mail18.scom.ca,S=2333,W=2413:2,S -rw-------??? 1 vmail? vmail? uarch? 2.2K Feb 20 08:05 1645362306.M569866P37919.mail18.scom.ca,S=2287,W=2366:2,S -rw-------??? 1 vmail? vmail? uarch? 2.3K Feb 20 08:05 1645362307.M184339P37919.mail18.scom.ca,S=2371,W=2452:2,S -rw-------??? 1 vmail? vmail? uarch?? 22K Feb 20 08:05 1645362307.M706237P37919.mail18.scom.ca,S=22281,W=22692:2,S -rw-------??? 1 vmail? vmail? uarch? 189K Feb 20 08:05 1645362313.M247319P37919.mail18.scom.ca,S=193354,W=196073:2,S -rw-------??? 1 vmail? vmail? uarch? 182K Feb 20 08:05 1645362313.M68190P39861.mail18.scom.ca,S=186263,W=188880:2,S fyi Happy Monday !!! Thanks - paul Paul Kudla 004-1009 Byron Street South Whitby, Ontario - Canada L1N 4S3 Toronto 416.642.7266 Main?1.866.411.7266 Fax?1.888.892.7266 On 4/17/2022 5:19 AM, dovecot-bounces at dovecot.org wrote:> > On 2022-04-17 05:32, Turritopsis Dohrnii Teo En Ming wrote: >> Subject: Both Roundcube and Squirrelmail webmail cannot search for >> anything + cannot open many emails because there are more than 200,000 >> emails in my Inbox >> >> Good day from Singapore, >> >> I am using Virtualmin/Webmin web hosting control panel in my Virtual >> Private Servers (VPS) in Germany. >> >> [1st issue] >> ==========>> >> Both Roundcube and Squirrelmail webmail cannot search for anything >> because there are more than 200,000 emails in my Inbox. >> >> Roundcube webmail doesn't show any error. Search results simply don't >> show up after a long time. >> >> However, I got the following error in Squirrelmail webmail (using >> Firefox web browser). >> >> ERROR: Connection dropped by IMAP server. >> >> Query: SEARCH CHARSET ISO-8859-1 ALL SUBJECT "keyword" >> >> [2nd issue] >> ===========>> >> Both Roundcube and Squirrelmail webmail cannot open many emails >> because there are more than 200,000 emails in my inbox. >> >> Roundcube webmail gives the following error: >> >> SERVER ERROR! >> Could not load message from server. >> >> Squirrelmail webmail gives the following error: >> >> ERROR: Connection dropped by IMAP server. >> Query: FETCH 1 BODY[HEADER] >> >> My IMAP server is Dovecot version 2.2.36. >> >> Virtualmin version is 6.17. >> Webmin version is 1.990. >> Usermin version is 1.834. >> Authentic theme version is 19.85.1. >> >> However, using Samsung Email app on my Samsung Galaxy A32 5G Android >> phone, I am able to open emails it seems. Only Roundcube and >> Squirrelmail webmail appears to have issues opening emails. Using >> Samsung Email app, I have deleted around 6,000 useless emails so far >> but still unable to search and open emails. Looks like I need to >> delete more and more useless emails. >> >> How do I solve the above problems? >> >> I am looking forward to your reply. >> >> Thank you very much. >> >> Mr. Turritopsis Dohrnii Teo En Ming, 44 years old as of 17 April 2022 >> Sunday, is a TARGETED INDIVIDUAL living in Singapore. He is an IT >> Consultant with a Systems Integrator (SI)/computer firm in Singapore. >> He is an IT enthusiast. >> >> >> >> >> >> -----BEGIN EMAIL SIGNATURE----- >> >> The Gospel for all Targeted Individuals (TIs): >> >> [The New York Times] Microwave Weapons Are Prime Suspect in Ills of >> U.S. Embassy Workers >> >> Link: >> https://www.nytimes.com/2018/09/01/science/sonic-attack-cuba-microwave.html >> >> >> ******************************************************************************************** >> >> >> Singaporean Targeted Individual Mr. Turritopsis Dohrnii Teo En Ming's >> Academic Qualifications as at 14 Feb 2019 and refugee seeking attempts >> at the United Nations Refugee Agency Bangkok (21 Mar 2017), in Taiwan >> (5 Aug 2019) and Australia (25 Dec 2019 to 9 Jan 2020): >> >> [1] https://tdtemcerts.wordpress.com/ >> >> [2] https://tdtemcerts.blogspot.sg/ >> >> [3] https://www.scribd.com/user/270125049/Teo-En-Ming >> >> -----END EMAIL SIGNATURE----- > > Hi there, > > Try to increase ulimit for Dovecot and Solr service as well as the > default OS one. > > Also, you might want to consider opening a bug report in Roundcube git > repository, if you weren't lucky to have response from here, > Aleksander is active over there. > > Good luck. > > Zakaria >-------------- next part -------------- An HTML attachment was scrubbed... URL: <https://dovecot.org/pipermail/dovecot/attachments/20220418/f1e894ba/attachment.htm>
Joseph Tam
2022-Apr-18 22:23 UTC
Bad Signature - Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox
On Mon, 18 Apr 2022, Paul Kudla (SCOM.CA Internet) wrote:> As for the 200,000+ emails in the inbox no email system was ever designed for > that - ever. >... > no system will support 200,000 + emails, even if the server can handle that > and running imap where you only download the headers the email client would > just spin trying to update the email box constantantly.I think Gmail does exaclty this -- their mail system really has one big mesage repository, and they simulate mailboxes by using labels. They seem to encourage piling the message high and using their search or auto-labelling features to find what you're looking for. Users of mine who previously used Gmail expect our mail system to behave similarly, and I have to break them of their habit to packrat all their mail into their INBOX. Joseph Tam <jtam.home at gmail.com>
Turritopsis Dohrnii Teo En Ming
2022-Apr-19 23:12 UTC
Bad Signature - Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox
It is not true that no email system was ever designed for 200,000 or more emails. If I did not remember wrongly, more than 10 years ago, I used to have a Gmail email account, with 4-5 MILLION email messages in the Inbox. There was no problem searching even though I had 4-5 MILLION email messages in the Inbox. Regards, Mr. Turritopsis Dohrnii Teo En Ming Targeted Individual in Singapore 20 Apr 2022 Wednesday On Mon, 18 Apr 2022 at 17:44, Paul Kudla (SCOM.CA Internet) <paul at scom.ca> wrote:> > this is not the answer you are looking for but comes from years and years of experience with mail servers in general (cyrus first then dovecot) > > First off imap clients can handle emails counts but vary on performance > > 1. outlook for example is completely useless above 1000 count per folder/sub folder, imap was kinda supported to outlook 2010 but after 2012 fell off largely due to microsoft pushing their sync & exchange servers. > > 2. thunderbird will and can handle 10000~30000 emails but anything above 10000 really should be split into a subfolder, i usually take the inbox and split into the year of the messages to keep into a subfolder like 2020,2021 etc > > 3. apple mail and iphone related seem to handle large emails boxes ok but will slow down above the 1000+ per folder limit > > 4. android you need to run bluemail or something like that but again limits are like the iphone for around 1000 per folder, generic imap client does not work well with android. > > As for the 200,000+ emails in the inbox no email system was ever designed for that - ever. > > It needs to be understood that pending how your server hdd's are formatted (nfs, ufs, zfs etc) will directly effect the speed of email retreivals. > > nfs being the slowest (dependant on network speeds, server speeds etc) > > ufs is the old style unix partitions, ufs works ok but you need to tweek certain settings for how many inodes can be in memory at any one time to i believe a max of 10,000 (freebsd, other systems will vary) > > zfs is the best as it will store dir inode etc into memory in the system so large file numbers in directories have a faster chance to respond, again system dependant suggest lots of memory > > no system will support 200,000 + emails, even if the server can handle that and running imap where you only download the headers the email client would just spin trying to update the email box constantantly. > > forget about trying to index searches etc. > > internet speed (even a 100M fibre connection) is also a factor. > > this is a sample dir structure from my server > > 1 email = 1 file in a sub dir. > > the operating system is where you are choking on its ability to handle 200,000 files in one directory is just unsustainable. > > if you have to handle that many incoming emails i strongly suggest a sieve filter that automatically moves incoming emails into a folder based on the email to keep the counts under control. > > I find dovecot can handle email counts better then the old cyrus but when i ran cyrus i was using ufs file system as well, when i upgraded to dovecot i had also upgraded to zfs. > > # cd cur > > [05:39:14] mail18.scom.ca [root:0] /data/dovecot/users/scom.ca/paul at scom.ca/cur > # ll > total 77333 > drwx------ 2 vmail vmail uarch 361B Apr 18 05:35 . > drwx------ 947 vmail vmail uarch 963B Apr 18 05:36 .. > -rw------- 1 vmail vmail uarch 6.4K Feb 20 08:05 1645362302.M163606P37919.mail18.scom.ca,S=6536,W=6687:2,FS > -rw------- 1 vmail vmail uarch 8.9K Feb 20 08:05 1645362302.M535115P37919.mail18.scom.ca,S=9074,W=9224:2,S > -rw------- 1 vmail vmail uarch 5.5K Feb 20 08:05 1645362303.M133625P37919.mail18.scom.ca,S=5632,W=5729:2,S > -rw------- 1 vmail vmail uarch 11K Feb 20 08:05 1645362305.M954336P38787.mail18.scom.ca,S=11438,W=11642:2,S > -rw------- 1 vmail vmail uarch 1.9K Feb 20 08:05 1645362305.M954355P38787.mail18.scom.ca,S=1912,W=1965:2,S > -rw------- 1 vmail vmail uarch 2.3K Feb 20 08:05 1645362306.M417932P37919.mail18.scom.ca,S=2333,W=2413:2,S > -rw------- 1 vmail vmail uarch 2.2K Feb 20 08:05 1645362306.M569866P37919.mail18.scom.ca,S=2287,W=2366:2,S > -rw------- 1 vmail vmail uarch 2.3K Feb 20 08:05 1645362307.M184339P37919.mail18.scom.ca,S=2371,W=2452:2,S > -rw------- 1 vmail vmail uarch 22K Feb 20 08:05 1645362307.M706237P37919.mail18.scom.ca,S=22281,W=22692:2,S > -rw------- 1 vmail vmail uarch 189K Feb 20 08:05 1645362313.M247319P37919.mail18.scom.ca,S=193354,W=196073:2,S > -rw------- 1 vmail vmail uarch 182K Feb 20 08:05 1645362313.M68190P39861.mail18.scom.ca,S=186263,W=188880:2,S > > fyi > > > > > > > > > > Happy Monday !!! > Thanks - paul > > Paul Kudla > > > > 004-1009 Byron Street South > Whitby, Ontario - Canada > L1N 4S3 > > Toronto 416.642.7266 > Main 1.866.411.7266 > Fax 1.888.892.7266 > On 4/17/2022 5:19 AM, dovecot-bounces at dovecot.org wrote: > > > On 2022-04-17 05:32, Turritopsis Dohrnii Teo En Ming wrote: > > Subject: Both Roundcube and Squirrelmail webmail cannot search for > anything + cannot open many emails because there are more than 200,000 > emails in my Inbox > > Good day from Singapore, > > I am using Virtualmin/Webmin web hosting control panel in my Virtual > Private Servers (VPS) in Germany. > > [1st issue] > ==========> > Both Roundcube and Squirrelmail webmail cannot search for anything > because there are more than 200,000 emails in my Inbox. > > Roundcube webmail doesn't show any error. Search results simply don't > show up after a long time. > > However, I got the following error in Squirrelmail webmail (using > Firefox web browser). > > ERROR: Connection dropped by IMAP server. > > Query: SEARCH CHARSET ISO-8859-1 ALL SUBJECT "keyword" > > [2nd issue] > ===========> > Both Roundcube and Squirrelmail webmail cannot open many emails > because there are more than 200,000 emails in my inbox. > > Roundcube webmail gives the following error: > > SERVER ERROR! > Could not load message from server. > > Squirrelmail webmail gives the following error: > > ERROR: Connection dropped by IMAP server. > Query: FETCH 1 BODY[HEADER] > > My IMAP server is Dovecot version 2.2.36. > > Virtualmin version is 6.17. > Webmin version is 1.990. > Usermin version is 1.834. > Authentic theme version is 19.85.1. > > However, using Samsung Email app on my Samsung Galaxy A32 5G Android > phone, I am able to open emails it seems. Only Roundcube and > Squirrelmail webmail appears to have issues opening emails. Using > Samsung Email app, I have deleted around 6,000 useless emails so far > but still unable to search and open emails. Looks like I need to > delete more and more useless emails. > > How do I solve the above problems? > > I am looking forward to your reply. > > Thank you very much. > > Mr. Turritopsis Dohrnii Teo En Ming, 44 years old as of 17 April 2022 > Sunday, is a TARGETED INDIVIDUAL living in Singapore. He is an IT > Consultant with a Systems Integrator (SI)/computer firm in Singapore. > He is an IT enthusiast. > > > > > > -----BEGIN EMAIL SIGNATURE----- > > The Gospel for all Targeted Individuals (TIs): > > [The New York Times] Microwave Weapons Are Prime Suspect in Ills of > U.S. Embassy Workers > > Link: > https://www.nytimes.com/2018/09/01/science/sonic-attack-cuba-microwave.html > > ******************************************************************************************** > > Singaporean Targeted Individual Mr. Turritopsis Dohrnii Teo En Ming's > Academic Qualifications as at 14 Feb 2019 and refugee seeking attempts > at the United Nations Refugee Agency Bangkok (21 Mar 2017), in Taiwan > (5 Aug 2019) and Australia (25 Dec 2019 to 9 Jan 2020): > > [1] https://tdtemcerts.wordpress.com/ > > [2] https://tdtemcerts.blogspot.sg/ > > [3] https://www.scribd.com/user/270125049/Teo-En-Ming > > -----END EMAIL SIGNATURE----- > > > Hi there, > > Try to increase ulimit for Dovecot and Solr service as well as the default OS one. > > Also, you might want to consider opening a bug report in Roundcube git repository, if you weren't lucky to have response from here, Aleksander is active over there. > > Good luck. > > Zakaria > >