Duc Anh Do
2021-Jan-27 09:20 UTC
Infinity loop when run "doveadm quota get -A" from Dovecot Director with 500 users
Hi all,
I have one Dovecot Director, two Dovecot Backends and one LDAP server with
about 500 users. I would like to run *doveadm quota get -A* from the
Director.
In each Backend, there is no problem when run the command:
# doveadm quota get -A
user1 User quota STORAGE 0 1048576000000000
0
user1 User quota MESSAGE 0 -
0
?
user500 User quota STORAGE 0 1048576000000000
0
user500 User quota MESSAGE 0 -
0
However, when I run from the Director, the command might stuck in an
infinity loop (I have to terminate to quit):
# doveadm quota get -A
user1 User quota STORAGE 0 1048576000000000
0
user1 User quota MESSAGE 0 -
0
?
user49 User quota STORAGE 0 1048576000000000 0
user49 User quota MESSAGE 0 - 0
user66 User quota STORAGE 0 1048576000000000 0
user66 User quota MESSAGE 0 - 0
^Cdoveadm(user86): Error: doveadm server failure
doveadm: Error: Failed to iterate through some users
doveadm: Error: backend2.local:24245: Command quota get failed for user53:
EOF
doveadm: Error: backend1.local:24245: Command quota get failed for user66:
EOF
doveadm: Error: Aborted
This problem occurs in both Dovecot 2.2.36 and Dovecot 2.3.11, 2.3.13 (I
build Dovecot from source). It's ok for me to get quota of one user from
the Director:
# doveadm quota get -u user1
Quota name Type Value Limit %
User quota STORAGE 0 1048576000000000 0
User quota MESSAGE 0 - 0
And if there's only one Backend, *doveadm quota get -A* from the Director
works well too.
After investigating, I found the infinity loop:
File src/doveadm/doveadm-mail-server.c:
static void doveadm_server_flush_one(struct doveadm_server *server)
{
unsigned int count = array_count(&server->queue);
do {
io_loop_run(current_ioloop);
} while (array_count(&server->queue) == count &&
doveadm_server_have_used_connections(server) &&
!DOVEADM_MAIL_SERVER_FAILED());
}
In case there're many Backends, I see only global variable *current_ioloop*
is used to notify in the callback function. Might this be a race condition?
I understand there's a workaround to do my work:
- Run *doveadm user '*'* to get all users
- Loop through all users and run *doveadm quota get -u xxx*
Thanks,
Anh Do
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20210127/c61952d8/attachment-0001.html>
Duc Anh Do
2021-Jan-29 02:49 UTC
Infinity loop when run "doveadm quota get -A" from Dovecot Director with 500 users
Hi all, Because I think this is a race condition so instead of using only one *current_ioloop*: - I create ioloop for each Backend in the list of Director - I think connections from a Director to same Backend are synchronous so I don't create ioloop for each connection - I think ioloop must be destroyed in correct order so I use a linked list to manage them, create then push and pop then destroy (sorry, I don't know any existing structure in source of Dovecot that I can re-use) I tested my patch with both "doveadm quota get -A" and "doveadm quota get -u xxx" many times. No error occurs (timeout leak, segment fault... etc). If you are interested in my patch, any comment is highly appreciated. I modified source files which might be shared with other doveadm commands so I'm not sure it's safe 100%. Thanks, Anh Do On Wed, 27 Jan 2021 at 16:20, Duc Anh Do <doducanh2710 at gmail.com> wrote:> Hi all, > > I have one Dovecot Director, two Dovecot Backends and one LDAP server with > about 500 users. I would like to run *doveadm quota get -A* from the > Director. > In each Backend, there is no problem when run the command: > # doveadm quota get -A > user1 User quota STORAGE 0 1048576000000000 > 0 > user1 User quota MESSAGE 0 - > 0 > ? > user500 User quota STORAGE 0 1048576000000000 > 0 > user500 User quota MESSAGE 0 - > 0 > > However, when I run from the Director, the command might stuck in an > infinity loop (I have to terminate to quit): > # doveadm quota get -A > user1 User quota STORAGE 0 1048576000000000 > 0 > user1 User quota MESSAGE 0 - > 0 > ? > user49 User quota STORAGE 0 1048576000000000 0 > user49 User quota MESSAGE 0 - 0 > user66 User quota STORAGE 0 1048576000000000 0 > user66 User quota MESSAGE 0 - 0 > ^Cdoveadm(user86): Error: doveadm server failure > doveadm: Error: Failed to iterate through some users > doveadm: Error: backend2.local:24245: Command quota get failed for user53: > EOF > doveadm: Error: backend1.local:24245: Command quota get failed for user66: > EOF > doveadm: Error: Aborted > > This problem occurs in both Dovecot 2.2.36 and Dovecot 2.3.11, 2.3.13 (I > build Dovecot from source). It's ok for me to get quota of one user from > the Director: > # doveadm quota get -u user1 > Quota name Type Value Limit % > User quota STORAGE 0 1048576000000000 0 > User quota MESSAGE 0 - 0 > And if there's only one Backend, *doveadm quota get -A* from the Director > works well too. > > After investigating, I found the infinity loop: > File src/doveadm/doveadm-mail-server.c: > static void doveadm_server_flush_one(struct doveadm_server *server) > { > unsigned int count = array_count(&server->queue); > > do { > io_loop_run(current_ioloop); > } while (array_count(&server->queue) == count && > doveadm_server_have_used_connections(server) && > !DOVEADM_MAIL_SERVER_FAILED()); > } > > In case there're many Backends, I see only global variable > *current_ioloop* is used to notify in the callback function. Might this > be a race condition? > I understand there's a workaround to do my work: > > - Run *doveadm user '*'* to get all users > - Loop through all users and run *doveadm quota get -u xxx* > > > Thanks, > Anh Do >-- Thanks, Duc Anh ---------------------------------------------------------------------------------------------------- Email: doducanh2710 at gmail.com Skype: ducanh.do88 Mobile: +84975730526 -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://dovecot.org/pipermail/dovecot/attachments/20210129/72a400a9/attachment-0001.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: fix-doveadm-quota-director-20210128-2055.patch Type: application/octet-stream Size: 4483 bytes Desc: not available URL: <https://dovecot.org/pipermail/dovecot/attachments/20210129/72a400a9/attachment-0001.obj>