I have the following in my log: Mar 21 14:46:59 bubba dovecot: replicator: Panic: data stack: Out of memory when allocating 1073741864 bytes Mar 21 14:46:59 bubba dovecot: replicator: Error: Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0x97c90) [0x7f4638a7cc90] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x97d6e) [0x7f4638a7cd6e] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f4638a14322] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x95e3f) [0x7f4638a7ae3f] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x9608b) [0x7f4638a7b08b] -> /usr/local/lib/dovecot/libdovecot.so.0(+0xb15f8) [0x7f4638a965f8] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x93c42) [0x7f4638a78c42] -> /usr/local/lib/dovecot/libdovecot.so.0(buffer_write+0x74) [0x7f4638a78fb4] -> dovecot/replicator(replicator_queue_push+0x13b) [0x40519b] -> dovecot/replicator() [0x4049c2] -> dovecot/replicator() [0x4040a8] -> dovecot/replicator() [0x404275] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52) [0x7f4638a907d2] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xe7) [0x7f4638a91d17] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c) [0x7f4638a9086c] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f4638a90a28] -> /usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f4638a1dfc3] -> dovecot/replicator(main+0x17a) [0x40342a] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f4638640ec5] -> dovecot/replicator() [0x4034c5] Mar 21 14:47:00 bubba dovecot: replicator: Fatal: master: service(replicator): child 15650 killed with signal 6 (core dumped) -- Daniel
Can you provide us gdb bt full dump? gdb /usr/libexec/dovecot/replicator /path/to/core on some systems, it's /usr/lib/dovecot/replicator Aki On 21.03.2017 23:48, Daniel Miller wrote:> I have the following in my log: > > Mar 21 14:46:59 bubba dovecot: replicator: Panic: data stack: Out of > memory when allocating 1073741864 bytes > Mar 21 14:46:59 bubba dovecot: replicator: Error: Raw backtrace: > /usr/local/lib/dovecot/libdovecot.so.0(+0x97c90) [0x7f4638a7cc90] -> > /usr/local/lib/dovecot/libdovecot.so.0(+0x97d6e) [0x7f4638a7cd6e] -> > /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f4638a14322] -> > /usr/local/lib/dovecot/libdovecot.so.0(+0x95e3f) [0x7f4638a7ae3f] -> > /usr/local/lib/dovecot/libdovecot.so.0(+0x9608b) [0x7f4638a7b08b] -> > /usr/local/lib/dovecot/libdovecot.so.0(+0xb15f8) [0x7f4638a965f8] -> > /usr/local/lib/dovecot/libdovecot.so.0(+0x93c42) [0x7f4638a78c42] -> > /usr/local/lib/dovecot/libdovecot.so.0(buffer_write+0x74) > [0x7f4638a78fb4] -> dovecot/replicator(replicator_queue_push+0x13b) > [0x40519b] -> dovecot/replicator() [0x4049c2] -> dovecot/replicator() > [0x4040a8] -> dovecot/replicator() [0x404275] -> > /usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52) > [0x7f4638a907d2] -> > /usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xe7) > [0x7f4638a91d17] -> > /usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c) > [0x7f4638a9086c] -> > /usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) > [0x7f4638a90a28] -> > /usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x13) > [0x7f4638a1dfc3] -> dovecot/replicator(main+0x17a) [0x40342a] -> > /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) > [0x7f4638640ec5] -> dovecot/replicator() [0x4034c5] > Mar 21 14:47:00 bubba dovecot: replicator: Fatal: master: > service(replicator): child 15650 killed with signal 6 (core dumped) >
Where would I find the core file? I'm not finding anything obvious. The replicator path is /usr/local/libexec/dovecot/replicator Daniel On 3/22/2017 12:52 AM, Aki Tuomi wrote:> Can you provide us gdb bt full dump? > > gdb /usr/libexec/dovecot/replicator /path/to/core > > on some systems, it's /usr/lib/dovecot/replicator > > Aki > > On 21.03.2017 23:48, Daniel Miller wrote: >> I have the following in my log: >> >> Mar 21 14:46:59 bubba dovecot: replicator: Panic: data stack: Out of >> memory when allocating 1073741864 bytes >> Mar 21 14:46:59 bubba dovecot: replicator: Error: Raw backtrace: >> /usr/local/lib/dovecot/libdovecot.so.0(+0x97c90) [0x7f4638a7cc90] -> >> /usr/local/lib/dovecot/libdovecot.so.0(+0x97d6e) [0x7f4638a7cd6e] -> >> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f4638a14322] -> >> /usr/local/lib/dovecot/libdovecot.so.0(+0x95e3f) [0x7f4638a7ae3f] -> >> /usr/local/lib/dovecot/libdovecot.so.0(+0x9608b) [0x7f4638a7b08b] -> >> /usr/local/lib/dovecot/libdovecot.so.0(+0xb15f8) [0x7f4638a965f8] -> >> /usr/local/lib/dovecot/libdovecot.so.0(+0x93c42) [0x7f4638a78c42] -> >> /usr/local/lib/dovecot/libdovecot.so.0(buffer_write+0x74) >> [0x7f4638a78fb4] -> dovecot/replicator(replicator_queue_push+0x13b) >> [0x40519b] -> dovecot/replicator() [0x4049c2] -> dovecot/replicator() >> [0x4040a8] -> dovecot/replicator() [0x404275] -> >> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52) >> [0x7f4638a907d2] -> >> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xe7) >> [0x7f4638a91d17] -> >> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c) >> [0x7f4638a9086c] -> >> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) >> [0x7f4638a90a28] -> >> /usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x13) >> [0x7f4638a1dfc3] -> dovecot/replicator(main+0x17a) [0x40342a] -> >> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) >> [0x7f4638640ec5] -> dovecot/replicator() [0x4034c5] >> Mar 21 14:47:00 bubba dovecot: replicator: Fatal: master: >> service(replicator): child 15650 killed with signal 6 (core dumped) >>
Think I got it:
#0 0x00007fddaf597c37 in __GI_raise (sig=sig at entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007fddaf59b028 in __GI_abort () at abort.c:89
#2 0x00007fddaf9c0c86 in default_fatal_finish (type=<optimized out>,
status=status at entry=0) at failures.c:201
#3 0x00007fddaf9c0d6e in i_internal_fatal_handler (ctx=0x7fff7197d000,
format=<optimized out>, args=<optimized out>)
at failures.c:670
#4 0x00007fddaf958322 in i_panic (format=format at entry=0x7fddafa047b8
"data stack: Out of memory when allocating %lu bytes")
at failures.c:275
#5 0x00007fddaf9bee3f in mem_block_alloc (min_size=536870912) at
data-stack.c:341
#6 0x00007fddaf9bf08b in t_malloc_real (size=size at entry=536870912,
permanent=permanent at entry=true) at data-stack.c:396
#7 0x00007fddaf9bf0ca in t_malloc (size=size at entry=536870912) at
data-stack.c:442
#8 0x00007fddaf9da5f8 in pool_data_stack_realloc (pool=<optimized out>,
mem=0x7fdd99f59038, old_size=268435449, new_size=536870912)
at mempool-datastack.c:126
#9 0x00007fddaf9bcc42 in p_realloc (new_size=<optimized out>,
old_size=<optimized out>, mem=<optimized out>, pool=<optimized
out>)
at mempool.h:88
#10 buffer_alloc (buf=buf at entry=0xf849a8, size=536870912) at buffer.c:36
#11 0x00007fddaf9bcfb4 in buffer_check_limits (data_size=32,
pos=268435424, buf=0xf849a8) at buffer.c:75
#12 buffer_write (_buf=0xf849a8, pos=268435424, data=0xfa0420,
data_size=32) at buffer.c:187
#13 0x000000000040519b in array_append_i (count=1, data=0xfa0420,
array=<synthetic pointer>) at ../../../src/lib/array.h:168
#14 replicator_queue_handle_sync_lookups (user=0xfa3fc0, queue=0xf92260)
at replicator-queue.c:278
---Type <return> to continue, or q <return> to quit---
#15 replicator_queue_push (queue=0xf92260, user=0xfa3fc0) at
replicator-queue.c:296
#16 0x00000000004049c2 in dsync_callback (reply=DSYNC_REPLY_OK,
state=0xf84590 "AQAAAHX+sC3O3gNSjAoAAOEwx0RQoclMAwAAAAQ",
'A'
<repeats 20 times>, "DAAAAEu2cNfSIZk0oMgAAO8QcX0yhyUwDAAAAlg",
'A'
<repeats 20 times>, "IAAADFjqMEUPIEU+IlAACdtazLUaHJTAcAAAAI",
'A'
<repeats 20 times>, "BwAAAEfYPTqXct9VYRMAAJ21"...,
context=0xfa1560)
at replicator-brain.c:121
#17 0x00000000004040a8 in dsync_callback (client=0xfa14a0,
state=<optimized out>, reply=DSYNC_REPLY_OK) at dsync-client.c:65
#18 0x0000000000404275 in dsync_input_line (line=0xf84988 "+",
client=0xfa14a0) at dsync-client.c:132
#19 dsync_input (client=0xfa14a0) at dsync-client.c:153
#20 0x00007fddaf9d47d2 in io_loop_call_io (io=0xfa15a0) at ioloop.c:599
#21 0x00007fddaf9d5d17 in io_loop_handler_run_internal
(ioloop=ioloop at entry=0xf8c720) at ioloop-epoll.c:223
#22 0x00007fddaf9d486c in io_loop_handler_run
(ioloop=ioloop at entry=0xf8c720) at ioloop.c:648
#23 0x00007fddaf9d4a28 in io_loop_run (ioloop=0xf8c720) at ioloop.c:623
#24 0x00007fddaf961fc3 in master_service_run (service=0xf8c5c0,
callback=callback at entry=0x404720 <client_connected>)
at master-service.c:641
#25 0x000000000040342a in main (argc=1, argv=0xf8c390) at replicator.c:112
Daniel
On 3/22/2017 9:11 AM, Daniel Miller wrote:> Where would I find the core file? I'm not finding anything obvious.
>
> The replicator path is /usr/local/libexec/dovecot/replicator
>
> Daniel
>
> On 3/22/2017 12:52 AM, Aki Tuomi wrote:
>> Can you provide us gdb bt full dump?
>>
>> gdb /usr/libexec/dovecot/replicator /path/to/core
>>
>> on some systems, it's /usr/lib/dovecot/replicator
>>
>> Aki
>>
>> On 21.03.2017 23:48, Daniel Miller wrote:
>>> I have the following in my log:
>>>
>>> Mar 21 14:46:59 bubba dovecot: replicator: Panic: data stack: Out
of
>>> memory when allocating 1073741864 bytes
>>> Mar 21 14:46:59 bubba dovecot: replicator: Error: Raw backtrace:
>>> /usr/local/lib/dovecot/libdovecot.so.0(+0x97c90) [0x7f4638a7cc90]
->
>>> /usr/local/lib/dovecot/libdovecot.so.0(+0x97d6e) [0x7f4638a7cd6e]
->
>>> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f4638a14322]
->
>>> /usr/local/lib/dovecot/libdovecot.so.0(+0x95e3f) [0x7f4638a7ae3f]
->
>>> /usr/local/lib/dovecot/libdovecot.so.0(+0x9608b) [0x7f4638a7b08b]
->
>>> /usr/local/lib/dovecot/libdovecot.so.0(+0xb15f8) [0x7f4638a965f8]
->
>>> /usr/local/lib/dovecot/libdovecot.so.0(+0x93c42) [0x7f4638a78c42]
->
>>> /usr/local/lib/dovecot/libdovecot.so.0(buffer_write+0x74)
>>> [0x7f4638a78fb4] ->
dovecot/replicator(replicator_queue_push+0x13b)
>>> [0x40519b] -> dovecot/replicator() [0x4049c2] ->
dovecot/replicator()
>>> [0x4040a8] -> dovecot/replicator() [0x404275] ->
>>> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52)
>>> [0x7f4638a907d2] ->
>>>
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xe7)
>>>
>>> [0x7f4638a91d17] ->
>>> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c)
>>> [0x7f4638a9086c] ->
>>> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x38)
>>> [0x7f4638a90a28] ->
>>> /usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x13)
>>> [0x7f4638a1dfc3] -> dovecot/replicator(main+0x17a) [0x40342a]
->
>>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)
>>> [0x7f4638640ec5] -> dovecot/replicator() [0x4034c5]
>>> Mar 21 14:47:00 bubba dovecot: replicator: Fatal: master:
>>> service(replicator): child 15650 killed with signal 6 (core dumped)
>>>