Aki Tuomi
2016-Oct-17 21:45 UTC
Dict proxy client returning empty string instead of multiline string
> > > >>>>> On 10/16/2016 11:16 PM, Pierre Jaury wrote: > > > >>>>>> Hello, > > > >>>>>> > > > >>>>>> I am using a dict proxy for my sieve extdata plugin to access some > > > >>>>>> fields from an SQLite database (autoreply text and other > > > >>>>>> database-configured items). > > > >>>>>> > > > >>>>>> All tests are performed against version 2.2.25. > > > >>>>>> > > > >>>>>> $ dovecot --version > > > >>>>>> 2.2.25 (7be1766) > > > >>>>>> > > > >>>>>> My configuration looks like: > > > >>>>>> > > > >>>>>> dict { > > > >>>>>> sieve = sqlite:/etc/dovecot/pigeonhole-sieve.dict > > > >>>>>> } > > > >>>>>> > > > >>>>>> [...] > > > >>>>>> sieve_extdata_dict_uri = proxy::sieve > > > >>>>>> > > > >>>>>> I am able to read pretty much any attribute without any issue, except > > > >>>>>> when the value contains a special character like "\r" or "\n". By using > > > >>>>>> the doveadm dict client, I narrowed it down to the dictionary management > > > >>>>>> part (either server, protocol or client). > > > >>>>>> > > > >>>>>> I was suspecting escaping functions from "lib/strescape.c" (mostly > > > >>>>>> str_tabescape and its counterpart, used by "lib-dict/client.c"), so I > > > >>>>>> monitored socket communications. It seems that escaping is done properly > > > >>>>>> on the server and the socket is not an issue either. > > > >>>>>> > > > >>>>>> The following strace dump results from running "doveadm dict get" > > > >>>>>> against the dict socket: > > > >>>>>> > > > >>>>>> connect(8, {sa_family=AF_UNIX, sun_path="..."}, 110) = 0 > > > >>>>>> fstat(8, {st_mode=S_IFSOCK|0777, st_size=0, ...}) = 0 > > > >>>>>> [...] > > > >>>>>> write(8, "H2\t0\t0\tadmin at domain.tld\tsieve\n", 30) = 30 > > > >>>>>> [...] > > > >>>>>> read(8, "Otest\1r\1ntest\n", 8192) = 14 > > > >>>>>> > > > >>>>>> Indeed "\1r" and "\1n" are the escape sequences used by > > > >>>>>> "lib/strescape.c". I went deeped and debugged the call to "dict_lookup" > > > >>>>>> performed by doveadm. Indeed the client gets the proper string from the > > > >>>>>> socket and to my surprise, it is properly unescaped. > > > >>>>>> > > > >>>>>> Then, in "client_dict_lookup" ("lib-dict/dict-client.c"), the call to > > > >>>>>> "p_strdup" returns an empty string (null byte set at the target address). > > > >>>>>> > > > >>>>>> Before the call to the dict "->lookup" attribute (client_dict_lookup): > > > >>>>>> > > > >>>>>> RAX: 0x7ffff73a37c0 (push r14) > > > >>>>>> RBX: 0x6831b8 ("priv/reply_body") > > > >>>>>> RCX: 0x7fffffffe240 --> 0x682a60 --> 0x6831b8 ("priv/reply_body") > > > >>>>>> RDX: 0x6831b8 ("priv/reply_body") > > > >>>>>> RSI: 0x683288 --> 0x7ffff7653120 --> 0x7ffff73ea620 ([...]) > > > >>>>>> RDI: 0x690ad0 --> 0x7ffff7400713 --> 0x75250079786f7270 ('proxy') > > > >>>>>> > > > >>>>>> 0x7ffff73a1f10 <dict_lookup+32>: mov rcx,r11 (value_r) > > > >>>>>> 0x7ffff73a1f13 <dict_lookup+35>: mov rdx,r8 (key) > > > >>>>>> 0x7ffff73a1f16 <dict_lookup+38>: mov rsi,r10 (pool) > > > >>>>>> 0x7ffff73a1f19 <dict_lookup+41>: mov rdi,r9 (dict) > > > >>>>>> 0x7ffff73a1f1c <dict_lookup+44>: add rsp,0x8 > > > >>>>>> => 0x7ffff73a1f20 <dict_lookup+48>: jmp rax > > > >>>>>> > > > >>>>>> Before the call to p_strdup in "client_dict_lookup": > > > >>>>>> > > > >>>>>> RSI: 0x6832d8 ("test\r\ntest") (lookup.result.value) > > > >>>>>> RDI: 0x683288 --> 0x7ffff7653120 --> [...] (pool) > > > >>>>>> RAX: 0x0 (result) > > > >>>>>> > > > >>>>>> 0x7ffff73a384f: nop > > > >>>>>> 0x7ffff73a3850: mov rsi,QWORD PTR [rsp+0x8] > > > >>>>>> 0x7ffff73a3855: mov rdi,r14 > > > >>>>>> => 0x7ffff73a3858: call 0x7ffff736d3c0 <p_strdup at plt> > > > >>>>>> 0x7ffff73a385d: mov QWORD PTR [r13+0x0],rax > > > >>>>>> 0x7ffff73a3861: mov rsi,QWORD PTR [rsp+0x18] > > > >>>>>> 0x7ffff73a3866: xor rsi,QWORD PTR fs:0x28 > > > >>>>>> 0x7ffff73a386f: mov eax,ebx > > > >>>>>> > > > >>>>>> After the call: > > > >>>>>> > > > >>>>>> 0x7ffff73a3850: mov rsi,QWORD PTR [rsp+0x8] > > > >>>>>> 0x7ffff73a3855: mov rdi,r14 > > > >>>>>> 0x7ffff73a3858: call 0x7ffff736d3c0 <p_strdup at plt> > > > >>>>>> => 0x7ffff73a385d: mov QWORD PTR [r13+0x0],rax > > > >>>>>> 0x7ffff73a3861: mov rsi,QWORD PTR [rsp+0x18] > > > >>>>>> 0x7ffff73a3866: xor rsi,QWORD PTR fs:0x28 > > > >>>>>> 0x7ffff73a386f: mov eax,ebx > > > >>>>>> 0x7ffff73a3871: jne 0x7ffff73a38da > > > >>>>>> > > > >>>>>> RSI: 0x0 > > > >>>>>> RDI: 0x6832d8 --> 0x0 > > > >>>>>> RAX: 0x6832d8 --> 0x0 (result) > > > >>>>>> > > > >>>>>> It is worth noting that I can reproduce the exact same execution flow > > > >>>>>> with a non-multiline result string (lookup.result.value) that is > > > >>>>>> properly copied by "p_strdup" and returned in RAX, then displayed by > > > >>>>>> doveadm. > > > >>>>>> > > > >>>>>> I am not familiar with the pooling mechanism hidden behind the call to > > > >>>>>> p_strdump and not quite sure why this behaviour is emerging. Maybe I am > > > >>>>>> even miles away from an understanding of the issue here, but it sounds > > > >>>>>> to me like something is wrong in the way "p_strdup" performs the copy. > > > >>>>>> > > > >>>>>> Hope this helps, > > > >>>>>> kaiyou. > > > >>>>>> > > > >>>>>> > > > >>>>> >Fixed with https://github.com/dovecot/core/commit/4f051c3082080b9d69ef12c3720c683cff34b0da Aki Tuomi
Pierre Jaury
2016-Oct-18 12:36 UTC
Dict proxy client returning empty string instead of multiline string
Hello, I can confirm the issue is fixed. Do you have a policy to backport the patch at least to the latest stable? Regards, On 10/17/2016 11:45 PM, Aki Tuomi wrote:>>>>>>>>> On 10/16/2016 11:16 PM, Pierre Jaury wrote: >>>>>>>>>> Hello, >>>>>>>>>> >>>>>>>>>> I am using a dict proxy for my sieve extdata plugin to access some >>>>>>>>>> fields from an SQLite database (autoreply text and other >>>>>>>>>> database-configured items). >>>>>>>>>> >>>>>>>>>> All tests are performed against version 2.2.25. >>>>>>>>>> >>>>>>>>>> $ dovecot --version >>>>>>>>>> 2.2.25 (7be1766) >>>>>>>>>> >>>>>>>>>> My configuration looks like: >>>>>>>>>> >>>>>>>>>> dict { >>>>>>>>>> sieve = sqlite:/etc/dovecot/pigeonhole-sieve.dict >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> [...] >>>>>>>>>> sieve_extdata_dict_uri = proxy::sieve >>>>>>>>>> >>>>>>>>>> I am able to read pretty much any attribute without any issue, except >>>>>>>>>> when the value contains a special character like "\r" or "\n". By using >>>>>>>>>> the doveadm dict client, I narrowed it down to the dictionary management >>>>>>>>>> part (either server, protocol or client). >>>>>>>>>> >>>>>>>>>> I was suspecting escaping functions from "lib/strescape.c" (mostly >>>>>>>>>> str_tabescape and its counterpart, used by "lib-dict/client.c"), so I >>>>>>>>>> monitored socket communications. It seems that escaping is done properly >>>>>>>>>> on the server and the socket is not an issue either. >>>>>>>>>> >>>>>>>>>> The following strace dump results from running "doveadm dict get" >>>>>>>>>> against the dict socket: >>>>>>>>>> >>>>>>>>>> connect(8, {sa_family=AF_UNIX, sun_path="..."}, 110) = 0 >>>>>>>>>> fstat(8, {st_mode=S_IFSOCK|0777, st_size=0, ...}) = 0 >>>>>>>>>> [...] >>>>>>>>>> write(8, "H2\t0\t0\tadmin at domain.tld\tsieve\n", 30) = 30 >>>>>>>>>> [...] >>>>>>>>>> read(8, "Otest\1r\1ntest\n", 8192) = 14 >>>>>>>>>> >>>>>>>>>> Indeed "\1r" and "\1n" are the escape sequences used by >>>>>>>>>> "lib/strescape.c". I went deeped and debugged the call to "dict_lookup" >>>>>>>>>> performed by doveadm. Indeed the client gets the proper string from the >>>>>>>>>> socket and to my surprise, it is properly unescaped. >>>>>>>>>> >>>>>>>>>> Then, in "client_dict_lookup" ("lib-dict/dict-client.c"), the call to >>>>>>>>>> "p_strdup" returns an empty string (null byte set at the target address). >>>>>>>>>> >>>>>>>>>> Before the call to the dict "->lookup" attribute (client_dict_lookup): >>>>>>>>>> >>>>>>>>>> RAX: 0x7ffff73a37c0 (push r14) >>>>>>>>>> RBX: 0x6831b8 ("priv/reply_body") >>>>>>>>>> RCX: 0x7fffffffe240 --> 0x682a60 --> 0x6831b8 ("priv/reply_body") >>>>>>>>>> RDX: 0x6831b8 ("priv/reply_body") >>>>>>>>>> RSI: 0x683288 --> 0x7ffff7653120 --> 0x7ffff73ea620 ([...]) >>>>>>>>>> RDI: 0x690ad0 --> 0x7ffff7400713 --> 0x75250079786f7270 ('proxy') >>>>>>>>>> >>>>>>>>>> 0x7ffff73a1f10 <dict_lookup+32>: mov rcx,r11 (value_r) >>>>>>>>>> 0x7ffff73a1f13 <dict_lookup+35>: mov rdx,r8 (key) >>>>>>>>>> 0x7ffff73a1f16 <dict_lookup+38>: mov rsi,r10 (pool) >>>>>>>>>> 0x7ffff73a1f19 <dict_lookup+41>: mov rdi,r9 (dict) >>>>>>>>>> 0x7ffff73a1f1c <dict_lookup+44>: add rsp,0x8 >>>>>>>>>> => 0x7ffff73a1f20 <dict_lookup+48>: jmp rax >>>>>>>>>> >>>>>>>>>> Before the call to p_strdup in "client_dict_lookup": >>>>>>>>>> >>>>>>>>>> RSI: 0x6832d8 ("test\r\ntest") (lookup.result.value) >>>>>>>>>> RDI: 0x683288 --> 0x7ffff7653120 --> [...] (pool) >>>>>>>>>> RAX: 0x0 (result) >>>>>>>>>> >>>>>>>>>> 0x7ffff73a384f: nop >>>>>>>>>> 0x7ffff73a3850: mov rsi,QWORD PTR [rsp+0x8] >>>>>>>>>> 0x7ffff73a3855: mov rdi,r14 >>>>>>>>>> => 0x7ffff73a3858: call 0x7ffff736d3c0 <p_strdup at plt> >>>>>>>>>> 0x7ffff73a385d: mov QWORD PTR [r13+0x0],rax >>>>>>>>>> 0x7ffff73a3861: mov rsi,QWORD PTR [rsp+0x18] >>>>>>>>>> 0x7ffff73a3866: xor rsi,QWORD PTR fs:0x28 >>>>>>>>>> 0x7ffff73a386f: mov eax,ebx >>>>>>>>>> >>>>>>>>>> After the call: >>>>>>>>>> >>>>>>>>>> 0x7ffff73a3850: mov rsi,QWORD PTR [rsp+0x8] >>>>>>>>>> 0x7ffff73a3855: mov rdi,r14 >>>>>>>>>> 0x7ffff73a3858: call 0x7ffff736d3c0 <p_strdup at plt> >>>>>>>>>> => 0x7ffff73a385d: mov QWORD PTR [r13+0x0],rax >>>>>>>>>> 0x7ffff73a3861: mov rsi,QWORD PTR [rsp+0x18] >>>>>>>>>> 0x7ffff73a3866: xor rsi,QWORD PTR fs:0x28 >>>>>>>>>> 0x7ffff73a386f: mov eax,ebx >>>>>>>>>> 0x7ffff73a3871: jne 0x7ffff73a38da >>>>>>>>>> >>>>>>>>>> RSI: 0x0 >>>>>>>>>> RDI: 0x6832d8 --> 0x0 >>>>>>>>>> RAX: 0x6832d8 --> 0x0 (result) >>>>>>>>>> >>>>>>>>>> It is worth noting that I can reproduce the exact same execution flow >>>>>>>>>> with a non-multiline result string (lookup.result.value) that is >>>>>>>>>> properly copied by "p_strdup" and returned in RAX, then displayed by >>>>>>>>>> doveadm. >>>>>>>>>> >>>>>>>>>> I am not familiar with the pooling mechanism hidden behind the call to >>>>>>>>>> p_strdump and not quite sure why this behaviour is emerging. Maybe I am >>>>>>>>>> even miles away from an understanding of the issue here, but it sounds >>>>>>>>>> to me like something is wrong in the way "p_strdup" performs the copy. >>>>>>>>>> >>>>>>>>>> Hope this helps, >>>>>>>>>> kaiyou. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> > > Fixed with https://github.com/dovecot/core/commit/4f051c3082080b9d69ef12c3720c683cff34b0da > > Aki Tuomi >
Aki Tuomi
2016-Oct-18 12:42 UTC
Dict proxy client returning empty string instead of multiline string
This will be fixed on next release, 2.2.26. We won't backport it to latest stable, sorry. Aki On 18.10.2016 15:36, Pierre Jaury wrote:> Hello, > > I can confirm the issue is fixed. Do you have a policy to backport the > patch at least to the latest stable? > > Regards, > > On 10/17/2016 11:45 PM, Aki Tuomi wrote: >>>>>>>>>> On 10/16/2016 11:16 PM, Pierre Jaury wrote: >>>>>>>>>>> Hello, >>>>>>>>>>> >>>>>>>>>>> I am using a dict proxy for my sieve extdata plugin to access some >>>>>>>>>>> fields from an SQLite database (autoreply text and other >>>>>>>>>>> database-configured items). >>>>>>>>>>> >>>>>>>>>>> All tests are performed against version 2.2.25. >>>>>>>>>>> >>>>>>>>>>> $ dovecot --version >>>>>>>>>>> 2.2.25 (7be1766) >>>>>>>>>>> >>>>>>>>>>> My configuration looks like: >>>>>>>>>>> >>>>>>>>>>> dict { >>>>>>>>>>> sieve = sqlite:/etc/dovecot/pigeonhole-sieve.dict >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> [...] >>>>>>>>>>> sieve_extdata_dict_uri = proxy::sieve >>>>>>>>>>> >>>>>>>>>>> I am able to read pretty much any attribute without any issue, except >>>>>>>>>>> when the value contains a special character like "\r" or "\n". By using >>>>>>>>>>> the doveadm dict client, I narrowed it down to the dictionary management >>>>>>>>>>> part (either server, protocol or client). >>>>>>>>>>> >>>>>>>>>>> I was suspecting escaping functions from "lib/strescape.c" (mostly >>>>>>>>>>> str_tabescape and its counterpart, used by "lib-dict/client.c"), so I >>>>>>>>>>> monitored socket communications. It seems that escaping is done properly >>>>>>>>>>> on the server and the socket is not an issue either. >>>>>>>>>>> >>>>>>>>>>> The following strace dump results from running "doveadm dict get" >>>>>>>>>>> against the dict socket: >>>>>>>>>>> >>>>>>>>>>> connect(8, {sa_family=AF_UNIX, sun_path="..."}, 110) = 0 >>>>>>>>>>> fstat(8, {st_mode=S_IFSOCK|0777, st_size=0, ...}) = 0 >>>>>>>>>>> [...] >>>>>>>>>>> write(8, "H2\t0\t0\tadmin at domain.tld\tsieve\n", 30) = 30 >>>>>>>>>>> [...] >>>>>>>>>>> read(8, "Otest\1r\1ntest\n", 8192) = 14 >>>>>>>>>>> >>>>>>>>>>> Indeed "\1r" and "\1n" are the escape sequences used by >>>>>>>>>>> "lib/strescape.c". I went deeped and debugged the call to "dict_lookup" >>>>>>>>>>> performed by doveadm. Indeed the client gets the proper string from the >>>>>>>>>>> socket and to my surprise, it is properly unescaped. >>>>>>>>>>> >>>>>>>>>>> Then, in "client_dict_lookup" ("lib-dict/dict-client.c"), the call to >>>>>>>>>>> "p_strdup" returns an empty string (null byte set at the target address). >>>>>>>>>>> >>>>>>>>>>> Before the call to the dict "->lookup" attribute (client_dict_lookup): >>>>>>>>>>> >>>>>>>>>>> RAX: 0x7ffff73a37c0 (push r14) >>>>>>>>>>> RBX: 0x6831b8 ("priv/reply_body") >>>>>>>>>>> RCX: 0x7fffffffe240 --> 0x682a60 --> 0x6831b8 ("priv/reply_body") >>>>>>>>>>> RDX: 0x6831b8 ("priv/reply_body") >>>>>>>>>>> RSI: 0x683288 --> 0x7ffff7653120 --> 0x7ffff73ea620 ([...]) >>>>>>>>>>> RDI: 0x690ad0 --> 0x7ffff7400713 --> 0x75250079786f7270 ('proxy') >>>>>>>>>>> >>>>>>>>>>> 0x7ffff73a1f10 <dict_lookup+32>: mov rcx,r11 (value_r) >>>>>>>>>>> 0x7ffff73a1f13 <dict_lookup+35>: mov rdx,r8 (key) >>>>>>>>>>> 0x7ffff73a1f16 <dict_lookup+38>: mov rsi,r10 (pool) >>>>>>>>>>> 0x7ffff73a1f19 <dict_lookup+41>: mov rdi,r9 (dict) >>>>>>>>>>> 0x7ffff73a1f1c <dict_lookup+44>: add rsp,0x8 >>>>>>>>>>> => 0x7ffff73a1f20 <dict_lookup+48>: jmp rax >>>>>>>>>>> >>>>>>>>>>> Before the call to p_strdup in "client_dict_lookup": >>>>>>>>>>> >>>>>>>>>>> RSI: 0x6832d8 ("test\r\ntest") (lookup.result.value) >>>>>>>>>>> RDI: 0x683288 --> 0x7ffff7653120 --> [...] (pool) >>>>>>>>>>> RAX: 0x0 (result) >>>>>>>>>>> >>>>>>>>>>> 0x7ffff73a384f: nop >>>>>>>>>>> 0x7ffff73a3850: mov rsi,QWORD PTR [rsp+0x8] >>>>>>>>>>> 0x7ffff73a3855: mov rdi,r14 >>>>>>>>>>> => 0x7ffff73a3858: call 0x7ffff736d3c0 <p_strdup at plt> >>>>>>>>>>> 0x7ffff73a385d: mov QWORD PTR [r13+0x0],rax >>>>>>>>>>> 0x7ffff73a3861: mov rsi,QWORD PTR [rsp+0x18] >>>>>>>>>>> 0x7ffff73a3866: xor rsi,QWORD PTR fs:0x28 >>>>>>>>>>> 0x7ffff73a386f: mov eax,ebx >>>>>>>>>>> >>>>>>>>>>> After the call: >>>>>>>>>>> >>>>>>>>>>> 0x7ffff73a3850: mov rsi,QWORD PTR [rsp+0x8] >>>>>>>>>>> 0x7ffff73a3855: mov rdi,r14 >>>>>>>>>>> 0x7ffff73a3858: call 0x7ffff736d3c0 <p_strdup at plt> >>>>>>>>>>> => 0x7ffff73a385d: mov QWORD PTR [r13+0x0],rax >>>>>>>>>>> 0x7ffff73a3861: mov rsi,QWORD PTR [rsp+0x18] >>>>>>>>>>> 0x7ffff73a3866: xor rsi,QWORD PTR fs:0x28 >>>>>>>>>>> 0x7ffff73a386f: mov eax,ebx >>>>>>>>>>> 0x7ffff73a3871: jne 0x7ffff73a38da >>>>>>>>>>> >>>>>>>>>>> RSI: 0x0 >>>>>>>>>>> RDI: 0x6832d8 --> 0x0 >>>>>>>>>>> RAX: 0x6832d8 --> 0x0 (result) >>>>>>>>>>> >>>>>>>>>>> It is worth noting that I can reproduce the exact same execution flow >>>>>>>>>>> with a non-multiline result string (lookup.result.value) that is >>>>>>>>>>> properly copied by "p_strdup" and returned in RAX, then displayed by >>>>>>>>>>> doveadm. >>>>>>>>>>> >>>>>>>>>>> I am not familiar with the pooling mechanism hidden behind the call to >>>>>>>>>>> p_strdump and not quite sure why this behaviour is emerging. Maybe I am >>>>>>>>>>> even miles away from an understanding of the issue here, but it sounds >>>>>>>>>>> to me like something is wrong in the way "p_strdup" performs the copy. >>>>>>>>>>> >>>>>>>>>>> Hope this helps, >>>>>>>>>>> kaiyou. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >> Fixed with https://github.com/dovecot/core/commit/4f051c3082080b9d69ef12c3720c683cff34b0da >> >> Aki Tuomi >>