> On 19 Jan, 2018, at 4:39, Aki Tuomi <aki.tuomi at dovecot.fi> wrote: > > > > On 19.01.2018 04:35, Adam Weinberger wrote: >> Since upgrading to 2.3.0 / 0.5.0.1, incoming PDFs are getting mangled. >> It seems to be happening when I use vnd.dovecot.filter. When I comment >> out the block, things come through fine. >> >> My filter block looks like this: >> require "vnd.dovecot.filter"; >> filter "bogofilter_filter"; >> >> if header :contains "X-Bogosity" [ >> "Spam, tests=bogofilter, spamicity=1.00", >> "Spam, tests=bogofilter, spamicity=0.99" >> ] { >> fileinto "spam/totally"; >> stop; >> } >> elsif header :contains "X-Bogosity" "Spam," { >> fileinto "spam/probably"; >> stop; >> } >> elsif header :contains "X-Bogosity" "Unsure," { >> fileinto "spam/maybe"; >> stop; >> } >> >> filter/bogofilter_filter looks like this: >> #!/bin/sh >> cat /dev/stdin | /usr/local/bin/bogofilter -p -e -c /path/to/bogofilter.cf >> >> Images seem to come through fine. I didn't have this problem before the >> upgrade. It could easily be a problem with bogofilter, but I figured >> that I'd check here first to see if anybody else has an >> attachment-mangling issue with vnd.dovecot.filter. >> >> # Adam >> >> >> -- >> Adam Weinberger >> adamw at adamw.org >> http://www.adamw.org > It might work better with > > exec /usr/local/bin/bogofilter -p -e -c /path/to/bogofilter.cf > > AkiAfter looking into this some more, what's happening is that messages are getting truncated. Regardless of whether it's an attachment or just a long message body, the message gets truncated. To show this, I made a fileinto :copy rule to save a message copy before it gets sent off to bogofilter. You can see from the below pastings that the message is definitely getting truncated. When I replace my filter script with "cat /dev/stdin", it works fine. When I send the message to bogofilter on the command-line, it works fine. Something about the combination of vnd.dovecot.filter + bogofilter is causing the messages to truncate, and I don't know whether it's from truncating on the send or receive of the filter. Can you give me any advice? 1) $ jot 200000 1 > numbers.txt $ du -a . | grep numbers 1260 ./numbers-sent.txt 1248 ./numbers-received.txt 2) root at imap:~# ll test-* 125 -rw------- 1 root wheel 123K Jan 20 09:35 test-afterbogo.msg 149 -rw------- 1 root wheel 136K Jan 20 09:35 test-beforebogo.msg root at imap:~# tail -20 test-beforebogo.msg IAowMDAwMTAxMTUzIDAwMDAwIG4gCjAwMDAxMDEyMDYgMDAwMDAgbiAKMDAwMDEwMTIzMiAwMDAw MCBuIAowMDAwMTAxMjc0IDAwMDAwIG4gCjAwMDAxMDEyOTMgMDAwMDAgbiAKdHJhaWxlcgo8PCAv U2l6ZSAxOSAvUm9vdCAxMiAwIFIgL0luZm8gMSAwIFIgL0lEIFsgPGM2YTE5OTc3MWRmODZjODk0 YzJhNDIwZDUzZjU3YzAzPgo8YzZhMTk5NzcxZGY4NmM4OTRjMmE0MjBkNTNmNTdjMDM+IF0gPj4K c3RhcnR4cmVmCjEwMTQ1NwolJUVPRgo--Apple-Mail=_6BE3438D-2544-4A5B-8D67-44E4A7E33BA5 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii; delsp=yes; format=flowed Adam Weinberger adamw at adamw.org http://www.adamw.org --Apple-Mail=_6BE3438D-2544-4A5B-8D67-44E4A7E33BA5-- root at imap:~# tail -20 test-afterbogo.msg CkGE66EHXl45NsGeItQOxE5P4ONNu/mZPUXdOF/pOOBI9ACqvQxFvW162vb9cG7hXCioruIo+2ES Wtn+UT2t6Yv2YgpTlOhTXD5riRQ4vlEWepl6tvU6ZOj+Hhpm5INo8B2Nz6XHiU2Ohz333pyvZj6M 9cQeY0Oi8pzB5qzHlH16DZiMeth/a66YRspD4uvuvziGSiLxxx2ghYaFOL6VhVuI9YOhuyZrfZ3C drtt27RNfwboY6iERO0zZKE0dzqDx3WcaxkzABLu6vgjBo6OeG5O4nwidEzSUciJU41Yh1q9rjWQ z9GBrRJxpSOhzZQubWwQR3xJfGAOjgDDcHQJ56jNpGdVjoG3pvgu5lEKxmKwTsHGrxkuh5WT4g/O dXsiK8IvxjVJmx0oKF3OAtdvGUxACg/3onJx6uiKFRCI/GL4AtahLiEUHXPVcXDSSkJ0XY9lxXxY Ucu+H5xcbSVe98Eh12HxXpHFQjdBvJf79XM0kYhi/KHyGMCqrYZNBsRg6lOucyClarTvwKC6GfwB 2RTwhE+nCnmIgx3A4JBrYA92Qy0wSCrO6VyWt0D3eiZYEseLIflww0Q6IoPuK89g7Y7QdIH3MRQ7 QghQ7WWefQ9A4NE5Hoj0To5G6QBOsfmT2ROmEACdIlYdiH1qblfCh8inz6Ox16XdP941NC4f8PPz isEgdId5LlPyJDi9fRgIdURTYzqwnk4YhRKtUnDNDbDojLk3qbPs9XTfVCXb6lKfkDKgnECsautd Rj8ikTwlPC/eA4vDE0AAtB4Erm3AaTC8wh7/AJMn8dnYKLty7ZmZIU1smt7OHvxPRfpWtE9HsnwH JFPp9UguEIhweW3AAAAHAYeguJjkDyI8mJqCpIZtWpDk3rwEFVUE5HqjonCeumwGF5N+NJr1cKTx DGnpOo8NPGEIri13UJySpHjpqZxXe+/D+/T/APAr/pPT/ndn8DahqidSDjkL5V6YyNaU4eT2H4PV MKRUpASwDr1UCmBtXBKra8rH8EfvHUTgZyp9gPxnB4RTxr9k9wrwejJKRG3MPKcHVDDOaVQxC30D TNR6wJy5UEHY6A8lfbNyJEUBo4VN74JrYgzjSOgQatEZ474e14GgvHJFNhFm2Usa3iI7yG/M95gc XEKI8I9TIg087R/s+uPjTR0AaHy359TDzr0861/J5fx28TEIO4Jsh7AB8Y1wvymNfwLnF6Bgj8nf 1taKM9U8NO0nWHRgFTWaNrRyRV37BhnjLrmxFHXtjxm6iaKuxfI+KIO89G1M7rSa89ZghRuqA/Kz IZNNURgeCvjOp5SHqq1snTUjcFJ4XZNA4SwE50lTNBajww6boqGlslH0UJz3wGd9p9nIv0NIhED0 CQm9W7mGJ9zgfcB1G6 -- Adam Weinberger adamw at adamw.org http://www.adamw.org
> On 20 Jan, 2018, at 10:05, Adam Weinberger <adamw at adamw.org> wrote: > > >> On 19 Jan, 2018, at 4:39, Aki Tuomi <aki.tuomi at dovecot.fi> wrote: >> >> >> >> On 19.01.2018 04:35, Adam Weinberger wrote: >>> Since upgrading to 2.3.0 / 0.5.0.1, incoming PDFs are getting mangled. >>> It seems to be happening when I use vnd.dovecot.filter. When I comment >>> out the block, things come through fine. >>> >>> My filter block looks like this: >>> require "vnd.dovecot.filter"; >>> filter "bogofilter_filter"; >>> >>> if header :contains "X-Bogosity" [ >>> "Spam, tests=bogofilter, spamicity=1.00", >>> "Spam, tests=bogofilter, spamicity=0.99" >>> ] { >>> fileinto "spam/totally"; >>> stop; >>> } >>> elsif header :contains "X-Bogosity" "Spam," { >>> fileinto "spam/probably"; >>> stop; >>> } >>> elsif header :contains "X-Bogosity" "Unsure," { >>> fileinto "spam/maybe"; >>> stop; >>> } >>> >>> filter/bogofilter_filter looks like this: >>> #!/bin/sh >>> cat /dev/stdin | /usr/local/bin/bogofilter -p -e -c /path/to/bogofilter.cf >>> >>> Images seem to come through fine. I didn't have this problem before the >>> upgrade. It could easily be a problem with bogofilter, but I figured >>> that I'd check here first to see if anybody else has an >>> attachment-mangling issue with vnd.dovecot.filter. >>> >>> # Adam >>> >>> >>> -- >>> Adam Weinberger >>> adamw at adamw.org >>> http://www.adamw.org >> It might work better with >> >> exec /usr/local/bin/bogofilter -p -e -c /path/to/bogofilter.cf >> >> Aki > > After looking into this some more, what's happening is that messages are > getting truncated. Regardless of whether it's an attachment or just a > long message body, the message gets truncated. > > To show this, I made a fileinto :copy rule to save a message copy before > it gets sent off to bogofilter. > > You can see from the below pastings that the message is definitely > getting truncated. When I replace my filter script with "cat /dev/stdin", > it works fine. When I send the message to bogofilter on the command-line, > it works fine. Something about the combination of vnd.dovecot.filter + > bogofilter is causing the messages to truncate, and I don't know whether > it's from truncating on the send or receive of the filter. > > Can you give me any advice? > > 1) > $ jot 200000 1 > numbers.txt > $ du -a . | grep numbers > 1260 ./numbers-sent.txt > 1248 ./numbers-received.txt > > 2) > root at imap:~# ll test-* > 125 -rw------- 1 root wheel 123K Jan 20 09:35 test-afterbogo.msg > 149 -rw------- 1 root wheel 136K Jan 20 09:35 test-beforebogo.msgThe more I look into it, the more it looks to me like pigeonhole is somehow losing the last 4-6K of messages over 100K. When my filter script is: cat /dev/stdin | tee /tmp/input | bogofilter[...] | tee /tmp/output Then /tmp/output is the full message, but what pigeonhole gets back is truncated at by about 4K. Same with [...] bogofilter[...] > /tmp/output; cat /tmp/output The message is truncated there too. I've tried everything I can think of to make sure it's not a buffering problem: [...] bogofilter[...] | cat [...] bogofilter[...] | dd bs=1k [...] bogofilter[...] ; a=$?; sleep 2; exit $a But everything I do comes up with the same problem. But a simple "cat /dev/stdin" and the message comes through fine. Your help is really appreciated. I'm in the process of buying a house, so the mangled attachments is not coming at a great time :-) # Adam -- Adam Weinberger adamw at adamw.org http://www.adamw.org
Op 1/20/2018 om 11:01 PM schreef Adam Weinberger:>> On 20 Jan, 2018, at 10:05, Adam Weinberger <adamw at adamw.org> wrote: >> >> >>> On 19 Jan, 2018, at 4:39, Aki Tuomi <aki.tuomi at dovecot.fi> wrote: >>> >>> >>> >>> On 19.01.2018 04:35, Adam Weinberger wrote: >>>> Since upgrading to 2.3.0 / 0.5.0.1, incoming PDFs are getting mangled. >>>> It seems to be happening when I use vnd.dovecot.filter. When I comment >>>> out the block, things come through fine. >>>> >>>> My filter block looks like this: >>>> ?????? require "vnd.dovecot.filter"; >>>> ?????? filter "bogofilter_filter"; >>>> >>>> ??????? if header :contains "X-Bogosity" [ >>>> ??????????????? "Spam, tests=bogofilter, spamicity=1.00", >>>> ??????????????? "Spam, tests=bogofilter, spamicity=0.99" >>>> ??????? ] { >>>> ??????????? fileinto "spam/totally"; >>>> ??????????? stop; >>>> ??????? } >>>> ??????? elsif header :contains "X-Bogosity" "Spam," { >>>> ??????????? fileinto "spam/probably"; >>>> ??????????? stop; >>>> ??????? } >>>> ??????? elsif header :contains "X-Bogosity" "Unsure," { >>>> ??????????? fileinto "spam/maybe"; >>>> ??????????? stop; >>>> ??????? } >>>> >>>> filter/bogofilter_filter looks like this: >>>> ??????? #!/bin/sh >>>> ??????? cat /dev/stdin | /usr/local/bin/bogofilter -p -e -c >>>> /path/to/bogofilter.cf >>>> >>>> Images seem to come through fine. I didn't have this problem before >>>> the >>>> upgrade. It could easily be a problem with bogofilter, but I figured >>>> that I'd check here first to see if anybody else has an >>>> attachment-mangling issue with vnd.dovecot.filter. >>>> >>>> # Adam >>>> >>>> >>>> -- >>>> Adam Weinberger >>>> adamw at adamw.org >>>> http://www.adamw.org >>> It might work better with >>> >>> exec /usr/local/bin/bogofilter -p -e -c /path/to/bogofilter.cf >>> >>> Aki >> >> After looking into this some more, what's happening is that messages >> are getting truncated. Regardless of whether it's an attachment or >> just a long message body, the message gets truncated. >> >> To show this, I made a fileinto :copy rule to save a message copy >> before it gets sent off to bogofilter. >> >> You can see from the below pastings that the message is definitely >> getting truncated. When I replace my filter script with "cat >> /dev/stdin", it works fine. When I send the message to bogofilter on >> the command-line, it works fine. Something about the combination of >> vnd.dovecot.filter + bogofilter is causing the messages to truncate, >> and I don't know whether it's from truncating on the send or receive >> of the filter. >> >> Can you give me any advice? >> >> 1) >> $ jot 200000 1 > numbers.txt >> $ du -a . | grep numbers >> 1260??? ./numbers-sent.txt >> 1248??? ./numbers-received.txt >> >> 2) >> root at imap:~# ll test-* >> 125 -rw-------? 1 root? wheel?? 123K Jan 20 09:35 test-afterbogo.msg >> 149 -rw-------? 1 root? wheel?? 136K Jan 20 09:35 test-beforebogo.msg > > The more I look into it, the more it looks to me like pigeonhole is > somehow losing the last 4-6K of messages over 100K. > > When my filter script is: > cat /dev/stdin | tee /tmp/input | bogofilter[...] | tee /tmp/output > Then /tmp/output is the full message, but what pigeonhole gets back is > truncated at by about 4K. > > Same with > [...] bogofilter[...] > /tmp/output; cat /tmp/output > The message is truncated there too. > > I've tried everything I can think of to make sure it's not a buffering > problem: > [...] bogofilter[...] | cat > [...] bogofilter[...] | dd bs=1k > [...] bogofilter[...] ; a=$?; sleep 2; exit $a > > But everything I do comes up with the same problem. > > But a simple "cat /dev/stdin" and the message comes through fine. > > Your help is really appreciated. I'm in the process of buying a house, > so the mangled attachments is not coming at a great time :-)We are looking into it. Regards, Stephan.