Dafan Zhai
2012-Sep-05 08:53 UTC
[Dovecot] Memory leak by getting the virtual size of a IMAP folder
Hi everyone, I am writing a dovecot statistic plugin, which calls the 'mailbox_get_metadata' function with MAILBOX_METADATA_VIRTUAL_SIZE as the 2nd parameter. enum mailbox_status_items metadata_items = MAILBOX_METADATA_VIRTUAL_SIZE; struct mailbox_metadata metadata; mailbox_get_metadata(mailbox, metadata_items, &metadata); but Valgrind finds a memory leak when this function is called: ---------------------------------------------------snip-------------------------------------------------------------- ==10304== 12,288 bytes in 3 blocks are definitely lost in loss record 74 of 76 ==10304== at 0x40222A4: calloc (in /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so) ==10304== by 0x416EE9F: block_alloc (mempool-alloconly.c:237) ==10304== by 0x416F16C: pool_alloconly_create (mempool-alloconly.c:140) ==10304== by 0x409C07D: mail_search_build_init (mail-search-build.c:187) ==10304== by 0x40C77F1: index_mailbox_get_metadata (index-status.c:200) ==10304== by 0x4067102: maildir_mailbox_get_metadata (maildir-storage.c:486) ==10304== by 0x409F78B: mailbox_get_metadata (mail-storage.c:1298) ==10304== by 0x490492D: mailbox_status_update (statistic-plugin.c:310) ==10304== by 0x4904E2E: statistic_mail_save (statistic-plugin.c:426) ==10304== by 0x41A054D: notify_contexts_mail_save (notify-plugin.c:61) ==10304== by 0x41A104F: notify_save_finish (notify-storage.c:143) ==10304== by 0x409EE94: mailbox_save_finish (mail-storage.c:1673) ---------------------------------------------------snip-------------------------------------------------------------- I have looked into the source code, and found that from the 'mail_search_build_init' function call in lib-storage/index/index-status.c:200 a pool is created, but the pool is not freed in the mailbox_search_deinit function call in lib-storage/index/index-status.c:218. This may be the reason of the memory leak. But I do not know how to free the pool. It seems to me that doveadm is having the same memory leak because it is using 'mail_search_build_init' too. ---------------------------------------------------snip-------------------------------------------------------------- # valgrind --leak-check=full doveadm mailbox status -u testuser1 vsize INBOX ==10457== Memcheck, a memory error detector ==10457== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al. ==10457== Using Valgrind-3.6.1 and LibVEX; rerun with -h for copyright info ==10457== Command: doveadm mailbox status -u testuser1 vsize INBOX ==10457=INBOX vsize=41643319 ==10457===10457== HEAP SUMMARY: ==10457== in use at exit: 4,356 bytes in 3 blocks ==10457== total heap usage: 440 allocs, 437 frees, 539,124 bytes allocated ==10457===10457== 4,096 bytes in 1 blocks are definitely lost in loss record 3 of 3 ==10457== at 0x40222A4: calloc (in /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so) ==10457== by 0x41AEE9F: ??? (in /usr/lib/dovecot/libdovecot.so.0.0.0) ==10457== by 0x41AF16C: pool_alloconly_create (in /usr/lib/dovecot/libdovecot.so.0.0.0) ==10457== by 0x40DC07D: mail_search_build_init (in /usr/lib/dovecot/libdovecot-storage.so.0.0.0) ==10457== by 0x8057EB8: doveadm_mail_mailbox_search_args_build (in /usr/bin/doveadm) ==10457== by 0x805855A: ??? (in /usr/bin/doveadm) ==10457== by 0x8053A7B: doveadm_mail_single_user (in /usr/bin/doveadm) ==10457== by 0x8053D2B: ??? (in /usr/bin/doveadm) ==10457== by 0x80541CF: doveadm_mail_try_run (in /usr/bin/doveadm) ==10457== by 0x805B863: main (in /usr/bin/doveadm) ==10457===10457== LEAK SUMMARY: ==10457== definitely lost: 4,096 bytes in 1 blocks ==10457== indirectly lost: 0 bytes in 0 blocks ==10457== possibly lost: 0 bytes in 0 blocks ==10457== still reachable: 260 bytes in 2 blocks ==10457== suppressed: 0 bytes in 0 blocks ==10457== Reachable blocks (those to which a pointer was found) are not shown. ==10457== To see them, rerun with: --leak-check=full --show-reachable=yes ==10457===10457== For counts of detected and suppressed errors, rerun with: -v ==10457== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 122 from 35) ---------------------------------------------------snip-------------------------------------------------------------- Dovecot version: 2.1.9 OS: Linux 3.0.30-dist i686 maildir:~/Maildir:LAYOUT=fs:INBOX=~/Maildir/INBOX I have applied the following patches from the dovecot 2.1 branch fixing memory leaks: http://hg.dovecot.org/dovecot-2.1/rev/7bdbca7b0913 http://hg.dovecot.org/dovecot-2.1/rev/963482677c0b It would be nice if you could give me some hints how I can address this issue? Thanks in advance! Greetings, Dafan
Dafan Zhai
2012-Sep-06 16:19 UTC
[Dovecot] Memory leak by getting the virtual size of a IMAP folder
Hi I think I have fixed this bug. The 'virtual_size_add_new' function in src/lib-storage/index/index-status.c creates a 'search_args' object and forgets to free it at the end. The function does call the 'mailbox_search_deinit' function but 'mailbox_search_deinit' only frees the 'search_ctx' object and reduce the refcount of 'search_args' by one but that doesn't cause 'search_args' to be freed because its refcount was 2. So an extra 'mail_search_args_unref' must be called. Timo, it would be nice if you could write a small comment if the attached patch is the right approach to address this issue. Dafan On 09/05/2012 10:53 AM, Dafan Zhai wrote:> Hi everyone, > > I am writing a dovecot statistic plugin, which calls the > 'mailbox_get_metadata' function with MAILBOX_METADATA_VIRTUAL_SIZE as > the 2nd parameter. > > enum mailbox_status_items metadata_items = MAILBOX_METADATA_VIRTUAL_SIZE; > struct mailbox_metadata metadata; > mailbox_get_metadata(mailbox, metadata_items, &metadata); > > but Valgrind finds a memory leak when this function is called: > ---------------------------------------------------snip-------------------------------------------------------------- > > ==10304== 12,288 bytes in 3 blocks are definitely lost in loss record 74 > of 76 > ==10304== at 0x40222A4: calloc (in > /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so) > ==10304== by 0x416EE9F: block_alloc (mempool-alloconly.c:237) > ==10304== by 0x416F16C: pool_alloconly_create (mempool-alloconly.c:140) > ==10304== by 0x409C07D: mail_search_build_init (mail-search-build.c:187) > ==10304== by 0x40C77F1: index_mailbox_get_metadata (index-status.c:200) > ==10304== by 0x4067102: maildir_mailbox_get_metadata > (maildir-storage.c:486) > ==10304== by 0x409F78B: mailbox_get_metadata (mail-storage.c:1298) > ==10304== by 0x490492D: mailbox_status_update (statistic-plugin.c:310) > ==10304== by 0x4904E2E: statistic_mail_save (statistic-plugin.c:426) > ==10304== by 0x41A054D: notify_contexts_mail_save (notify-plugin.c:61) > ==10304== by 0x41A104F: notify_save_finish (notify-storage.c:143) > ==10304== by 0x409EE94: mailbox_save_finish (mail-storage.c:1673) > ---------------------------------------------------snip-------------------------------------------------------------- > > > I have looked into the source code, and found that from the > 'mail_search_build_init' function call in > lib-storage/index/index-status.c:200 a pool is created, but the pool is > not freed in the mailbox_search_deinit function call in > lib-storage/index/index-status.c:218. > This may be the reason of the memory leak. But I do not know how to free > the pool. > > It seems to me that doveadm is having the same memory leak because it is > using 'mail_search_build_init' too. > ---------------------------------------------------snip-------------------------------------------------------------- > > # valgrind --leak-check=full doveadm mailbox status -u testuser1 vsize > INBOX > ==10457== Memcheck, a memory error detector > ==10457== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al. > ==10457== Using Valgrind-3.6.1 and LibVEX; rerun with -h for copyright info > ==10457== Command: doveadm mailbox status -u testuser1 vsize INBOX > ==10457=> INBOX vsize=41643319 > ==10457=> ==10457== HEAP SUMMARY: > ==10457== in use at exit: 4,356 bytes in 3 blocks > ==10457== total heap usage: 440 allocs, 437 frees, 539,124 bytes allocated > ==10457=> ==10457== 4,096 bytes in 1 blocks are definitely lost in loss record 3 of 3 > ==10457== at 0x40222A4: calloc (in > /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so) > ==10457== by 0x41AEE9F: ??? (in /usr/lib/dovecot/libdovecot.so.0.0.0) > ==10457== by 0x41AF16C: pool_alloconly_create (in > /usr/lib/dovecot/libdovecot.so.0.0.0) > ==10457== by 0x40DC07D: mail_search_build_init (in > /usr/lib/dovecot/libdovecot-storage.so.0.0.0) > ==10457== by 0x8057EB8: doveadm_mail_mailbox_search_args_build (in > /usr/bin/doveadm) > ==10457== by 0x805855A: ??? (in /usr/bin/doveadm) > ==10457== by 0x8053A7B: doveadm_mail_single_user (in /usr/bin/doveadm) > ==10457== by 0x8053D2B: ??? (in /usr/bin/doveadm) > ==10457== by 0x80541CF: doveadm_mail_try_run (in /usr/bin/doveadm) > ==10457== by 0x805B863: main (in /usr/bin/doveadm) > ==10457=> ==10457== LEAK SUMMARY: > ==10457== definitely lost: 4,096 bytes in 1 blocks > ==10457== indirectly lost: 0 bytes in 0 blocks > ==10457== possibly lost: 0 bytes in 0 blocks > ==10457== still reachable: 260 bytes in 2 blocks > ==10457== suppressed: 0 bytes in 0 blocks > ==10457== Reachable blocks (those to which a pointer was found) are not > shown. > ==10457== To see them, rerun with: --leak-check=full --show-reachable=yes > ==10457=> ==10457== For counts of detected and suppressed errors, rerun with: -v > ==10457== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 122 from 35) > ---------------------------------------------------snip-------------------------------------------------------------- > > > Dovecot version: 2.1.9 > OS: Linux 3.0.30-dist i686 > maildir:~/Maildir:LAYOUT=fs:INBOX=~/Maildir/INBOX > > I have applied the following patches from the dovecot 2.1 branch fixing > memory leaks: > http://hg.dovecot.org/dovecot-2.1/rev/7bdbca7b0913 > http://hg.dovecot.org/dovecot-2.1/rev/963482677c0b > > > It would be nice if you could give me some hints how I can address this > issue? > > Thanks in advance! > > Greetings, > Dafan-- Follow us on Facebook: <http://www.facebook.com/pages/Securepoint-GmbH/132451210137395> Follow us on Twitter: <http://twitter.com/SecurepointStat> --------------------------------------------------------------------- Securepoint GmbH | Entwicklung Dafan Zhai Salzstr. 1 D-21335 Lueneburg http://www.securepoint.de Tele: ++49 4131 2401-0 Fax: ++49 4131 2401-50 Lueneburg HRB 1776 --------------------------------------------------------------------- CONFIDENTIALITY : This e-mail and any attachments are confidential and may be privileged. If you are not a named recipient, please notify the sender immediately and do not disclose the contents to another person, use it for any purpose or store or copy the information in any medium. GEHEIMHALTUNGSPFLICHT : Dieses E-Mail und alle damit verbundenen Anlagen sind vertraulich und d?rfen nur bestimmten Personen zug?nglich gemacht werden. Sofern Sie nicht zu den angegebenen Empf?ngern geh?ren, benachrichtigen Sie bitte unverz?glich den Absender. Der Inhalt darf weder an Dritte weitergegeben noch zu anderen Zwecken verwendet werden. Die Informationen d?rfen auch nicht auf einem Datentr?ger gespeichert oder auf einen Datentr?ger kopiert werden. -------------- next part -------------- A non-text attachment was scrubbed... Name: dovecot-2.1.9-lib-storage-virtual_size_add_new-memory-leak.patch Type: text/x-patch Size: 873 bytes Desc: not available URL: <http://dovecot.org/pipermail/dovecot/attachments/20120906/3112cff1/attachment-0004.bin>
Timo Sirainen
2012-Sep-11 16:15 UTC
[Dovecot] Memory leak by getting the virtual size of a IMAP folder
On 6.9.2012, at 19.19, Dafan Zhai wrote:> I think I have fixed this bug. The 'virtual_size_add_new' function in src/lib-storage/index/index-status.c creates a 'search_args' object and forgets to free it at the end. The function does call the 'mailbox_search_deinit' function but 'mailbox_search_deinit' only frees the 'search_ctx' object and reduce the refcount of 'search_args' by one but that doesn't cause 'search_args' to be freed because its refcount was 2. So an extra 'mail_search_args_unref' must be called. > > Timo, it would be nice if you could write a small comment if the attached patch is the right approach to address this issue.Yes, committed to hg.