Hello,
I'm currently benchmarking new hardware aimed to serve around 70k users
For now, our IMAP server have 13k users.
To run imaptest, I've spwan some bench clients.
Each bench client can run imaptest with 1000 clients.
More than 1000 clients will load CPU of this bench client
imaptest command (command are chosen from usage stat on our other IMAP
servers):
imaptest host=xxxxx port=xxx userfile=userfile mbox=/root/dovecot-crlf
pass=sxxxx seed=123 clients=1000 select=194 uidfetch=94 noop=70
status=82 append=49 fetch=276 list=12 store=19 expunge=22
msubs=4 search=4 logout=1 delete=81 no_pipelining
With one bench client, everything runs smoothly.
# ps aux | grep dovecot | awk '{print $11,$12,$13,$14,$15,$16,$17,$18}'
| sort | uniq -c
? ?? 1 anvil: [221 connections] (anvil)
?? 1 auth: [13 wait, 0 passdb, 0 userdb] (auth)
?? 1 dovecot/config
?? 1 dovecot/imap
? 84 dovecot/imap-login
?? 1 dovecot/log
? 20 dovecot/pop3-login
?? 1 grep dovecot
?? 1 stats: [1307 connections] (stats)
When a second instance bench instance start imaptest, clients of first
and second instance begin to stall :
?1400 stalled for 20 secs in command: 1 LOGIN"fakeuser644 at
mailbench...." "password"
And :
# ps aux | grep dovecot | awk '{print $11,$12,$13,$14,$15,$16,$17,$18}'
| sort | uniq -c
?? 1 anvil: [221 connections] (anvil)
?? 1 auth: [1227 wait, 0 passdb, 0 userdb] (auth)
?? 1 dovecot/config
?? 1 dovecot/imap
? 37 dovecot/imap-login
?? 1 dovecot/log
? 20 dovecot/pop3-login
?? 1 grep dovecot
?? 1 stats: [680 connections] (stats)
Every auth go in wait, number of connection decreases.
Using mysql or a password file give same results.
I have used different values for service_count with also no success.
I think my use of imaptest could be false.
My understanding of service auth is limited for now because I'm quite
new to Dovecot (I have previously worked with Cyrus).
Thank you for every hints.
Isma?l Tanguy
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20220207/eed8c823/attachment.htm>
Hey, please refer to: https://doc.dovecot.org/admin_manual/login_processes/ We are using high-performance mode and it is serving 30k users with no problems. Best, Justas On 2022-02-07 17:33, ismael.tanguy at univ-brest.fr wrote:> Hello, > > I'm currently benchmarking new hardware aimed to serve around 70k users > For now, our IMAP server have 13k users. > > To run imaptest, I've spwan some bench clients. > Each bench client can run imaptest with 1000 clients. > More than 1000 clients will load CPU of this bench client > > imaptest command (command are chosen from usage stat on our other IMAP > servers): > > imaptest host=xxxxx port=xxx userfile=userfile mbox=/root/dovecot-crlf > pass=sxxxx seed=123 clients=1000 select=194 uidfetch=94 noop=70 > status=82 append=49 fetch=276 list=12 store=19 expunge=22 > msubs=4 search=4 logout=1 delete=81 no_pipelining > > > With one bench client, everything runs smoothly. > > # ps aux | grep dovecot | awk '{print $11,$12,$13,$14,$15,$16,$17,$18}' | sort | uniq More than 1000 clients will load CPU of this bench client-c > ? ?? 1 anvil: [221 connections] (anvil) > ?? 1 auth: [13 wait, 0 passdb, 0 userdb] (auth) > ?? 1 dovecot/config > ?? 1 dovecot/imap > ? 84 dovecot/imap-login > ?? 1 dovecot/log > ? 20 dovecot/pop3-login > ?? 1 grep dovecot > ?? 1 stats: [1307 connections] (stats) > > > When a second instance bench instance start imaptest, clients of first > and second instance begin to stall : > > ?1400 stalled for 20 secs in command: 1 LOGIN"fakeuser644 at mailbench...." "password" > > And : > > # ps aux | grep dovecot | awk '{print $11,$12,$13,$14,$15,$16,$17,$18}' | sort | uniq -c > ?? 1 anvil: [221 connections] (anvil) > ?? 1 auth: [1227 wait, 0 passdb, 0 userdb] (auth) > ?? 1 dovecot/config > ?? 1 dovecot/imap > ? 37 dovecot/imap-login > ?? 1 dovecot/log > ? 20 dovecot/pop3-login > ?? 1 grep dovecot > ?? 1 stats: [680 connections] (stats) > > Every auth go in wait, number of connection decreases. > > Using mysql or a password file give same results. > > I have used different values for service_count with also no success. > > I think my use of imaptest could be false. > My understanding of service auth is limited for now because I'm quite > new to Dovecot (I have previously worked with Cyrus). > > Thank you for every hints. > > > Isma?l Tanguy > > >
>>>>> "ismael" == ismael tanguy at univ-brest fr <ismael.tanguy at univ-brest.fr> writes:ismael> I'm currently benchmarking new hardware aimed to serve around ismael> 70k users For now, our IMAP server have 13k users. This doesn't help us help you. Is this a new rasperry Pi 4? Is it a Dual CPU AMD Rzyzen with 128gb of memory and fast NVMe disks? What is your system setup? ismael> To run imaptest, I've spwan some bench clients. Are these tests run from remote hosts? What kind of network are you using? ismael> Each bench client can run imaptest with 1000 clients. ismael> More than 1000 clients will load CPU of this bench client ismael> imaptest command (command are chosen from usage stat on our other IMAP servers): ismael> imaptest host=xxxxx port=xxx userfile=userfile mbox=/root/dovecot-crlf ismael> pass=sxxxx seed=123 clients=1000 select=194 uidfetch=94 noop=70 ismael> status=82 append=49 fetch=276 list=12 store=19 expunge=22 ismael> msubs=4 search=4 logout=1 delete=81 no_pipelining ismael> With one bench client, everything runs smoothly. ismael> # ps aux | grep dovecot | awk '{print $11,$12,$13,$14,$15,$16,$17,$18}' | sort | uniq -c ismael> ? ?? 1 anvil: [221 connections] (anvil) ismael> ?? 1 auth: [13 wait, 0 passdb, 0 userdb] (auth) ismael> ?? 1 dovecot/config ismael> ?? 1 dovecot/imap ismael> ? 84 dovecot/imap-login ismael> ?? 1 dovecot/log ismael> ? 20 dovecot/pop3-login ismael> ?? 1 grep dovecot ismael> ?? 1 stats: [1307 connections] (stats) ismael> When a second instance bench instance start imaptest, clients ismael> of first and second instance begin to stall : ismael> ?1400 stalled for 20 secs in command: 1 LOGIN "fakeuser644 at mailbench...." "password" So how is your dovecot authentication setup? Are you using a mysql backend? LDAP? Where is the server you're querying against? Are you running mysql on the same server you're running dovecot on? Are you running multiple dovecot servers with dovecot director in front of them to help spread the load and to offer resilience if/when a backend server fails? ismael> And : ismael> # ps aux | grep dovecot | awk '{print $11,$12,$13,$14,$15,$16,$17,$18}' | sort | uniq -c ismael> ?? 1 anvil: [221 connections] (anvil) ismael> ?? 1 auth: [1227 wait, 0 passdb, 0 userdb] (auth) ismael> ?? 1 dovecot/config ismael> ?? 1 dovecot/imap ismael> ? 37 dovecot/imap-login ismael> ?? 1 dovecot/log ismael> ? 20 dovecot/pop3-login ismael> ?? 1 grep dovecot ismael> ?? 1 stats: [680 connections] (stats) ismael> Every auth go in wait, number of connection decreases. ismael> Using mysql or a password file give same results. Where is mysql located? ismael> I have used different values for service_count with also no success. Post your configuration details. ismael> I think my use of imaptest could be false. It could be. Are you thinking that 2000 users will all be logging into the system at the same time? ismael> My understanding of service auth is limited for now because ismael> I'm quite new to Dovecot (I have previously worked with ismael> Cyrus). Can't really give you any hints until you tell us more about your setup. John