Op 1/20/2018 om 11:01 PM schreef Adam Weinberger:>> On 20 Jan, 2018, at 10:05, Adam Weinberger <adamw at adamw.org> wrote: >> >> >>> On 19 Jan, 2018, at 4:39, Aki Tuomi <aki.tuomi at dovecot.fi> wrote: >>> >>> >>> >>> On 19.01.2018 04:35, Adam Weinberger wrote: >>>> Since upgrading to 2.3.0 / 0.5.0.1, incoming PDFs are getting mangled. >>>> It seems to be happening when I use vnd.dovecot.filter. When I comment >>>> out the block, things come through fine. >>>> >>>> My filter block looks like this: >>>> ?????? require "vnd.dovecot.filter"; >>>> ?????? filter "bogofilter_filter"; >>>> >>>> ??????? if header :contains "X-Bogosity" [ >>>> ??????????????? "Spam, tests=bogofilter, spamicity=1.00", >>>> ??????????????? "Spam, tests=bogofilter, spamicity=0.99" >>>> ??????? ] { >>>> ??????????? fileinto "spam/totally"; >>>> ??????????? stop; >>>> ??????? } >>>> ??????? elsif header :contains "X-Bogosity" "Spam," { >>>> ??????????? fileinto "spam/probably"; >>>> ??????????? stop; >>>> ??????? } >>>> ??????? elsif header :contains "X-Bogosity" "Unsure," { >>>> ??????????? fileinto "spam/maybe"; >>>> ??????????? stop; >>>> ??????? } >>>> >>>> filter/bogofilter_filter looks like this: >>>> ??????? #!/bin/sh >>>> ??????? cat /dev/stdin | /usr/local/bin/bogofilter -p -e -c >>>> /path/to/bogofilter.cf >>>> >>>> Images seem to come through fine. I didn't have this problem before >>>> the >>>> upgrade. It could easily be a problem with bogofilter, but I figured >>>> that I'd check here first to see if anybody else has an >>>> attachment-mangling issue with vnd.dovecot.filter. >>>> >>>> # Adam >>>> >>>> >>>> -- >>>> Adam Weinberger >>>> adamw at adamw.org >>>> http://www.adamw.org >>> It might work better with >>> >>> exec /usr/local/bin/bogofilter -p -e -c /path/to/bogofilter.cf >>> >>> Aki >> >> After looking into this some more, what's happening is that messages >> are getting truncated. Regardless of whether it's an attachment or >> just a long message body, the message gets truncated. >> >> To show this, I made a fileinto :copy rule to save a message copy >> before it gets sent off to bogofilter. >> >> You can see from the below pastings that the message is definitely >> getting truncated. When I replace my filter script with "cat >> /dev/stdin", it works fine. When I send the message to bogofilter on >> the command-line, it works fine. Something about the combination of >> vnd.dovecot.filter + bogofilter is causing the messages to truncate, >> and I don't know whether it's from truncating on the send or receive >> of the filter. >> >> Can you give me any advice? >> >> 1) >> $ jot 200000 1 > numbers.txt >> $ du -a . | grep numbers >> 1260??? ./numbers-sent.txt >> 1248??? ./numbers-received.txt >> >> 2) >> root at imap:~# ll test-* >> 125 -rw-------? 1 root? wheel?? 123K Jan 20 09:35 test-afterbogo.msg >> 149 -rw-------? 1 root? wheel?? 136K Jan 20 09:35 test-beforebogo.msg > > The more I look into it, the more it looks to me like pigeonhole is > somehow losing the last 4-6K of messages over 100K. > > When my filter script is: > cat /dev/stdin | tee /tmp/input | bogofilter[...] | tee /tmp/output > Then /tmp/output is the full message, but what pigeonhole gets back is > truncated at by about 4K. > > Same with > [...] bogofilter[...] > /tmp/output; cat /tmp/output > The message is truncated there too. > > I've tried everything I can think of to make sure it's not a buffering > problem: > [...] bogofilter[...] | cat > [...] bogofilter[...] | dd bs=1k > [...] bogofilter[...] ; a=$?; sleep 2; exit $a > > But everything I do comes up with the same problem. > > But a simple "cat /dev/stdin" and the message comes through fine. > > Your help is really appreciated. I'm in the process of buying a house, > so the mangled attachments is not coming at a great time :-)We are looking into it. Regards, Stephan.
Op 1/21/2018 om 4:34 PM schreef Stephan Bosch:> Op 1/20/2018 om 11:01 PM schreef Adam Weinberger: >>> On 20 Jan, 2018, at 10:05, Adam Weinberger <adamw at adamw.org> wrote: >>> >>> >>>> On 19 Jan, 2018, at 4:39, Aki Tuomi <aki.tuomi at dovecot.fi> wrote: >>>> >>>> >>>> >>>> On 19.01.2018 04:35, Adam Weinberger wrote: >>>>> Since upgrading to 2.3.0 / 0.5.0.1, incoming PDFs are getting mangled. >>>>> It seems to be happening when I use vnd.dovecot.filter. When I comment >>>>> out the block, things come through fine. >>>>> >>>>> My filter block looks like this: >>>>> ?????? require "vnd.dovecot.filter"; >>>>> ?????? filter "bogofilter_filter"; >>>>> >>>>> ??????? if header :contains "X-Bogosity" [ >>>>> ??????????????? "Spam, tests=bogofilter, spamicity=1.00", >>>>> ??????????????? "Spam, tests=bogofilter, spamicity=0.99" >>>>> ??????? ] { >>>>> ??????????? fileinto "spam/totally"; >>>>> ??????????? stop; >>>>> ??????? } >>>>> ??????? elsif header :contains "X-Bogosity" "Spam," { >>>>> ??????????? fileinto "spam/probably"; >>>>> ??????????? stop; >>>>> ??????? } >>>>> ??????? elsif header :contains "X-Bogosity" "Unsure," { >>>>> ??????????? fileinto "spam/maybe"; >>>>> ??????????? stop; >>>>> ??????? } >>>>> >>>>> filter/bogofilter_filter looks like this: >>>>> ??????? #!/bin/sh >>>>> ??????? cat /dev/stdin | /usr/local/bin/bogofilter -p -e -c >>>>> /path/to/bogofilter.cf >>>>> >>>>> Images seem to come through fine. I didn't have this problem before >>>>> the >>>>> upgrade. It could easily be a problem with bogofilter, but I figured >>>>> that I'd check here first to see if anybody else has an >>>>> attachment-mangling issue with vnd.dovecot.filter. >>>>> >>>>> # Adam >>>>> >>>>> >>>>> -- >>>>> Adam Weinberger >>>>> adamw at adamw.org >>>>> http://www.adamw.org >>>> It might work better with >>>> >>>> exec /usr/local/bin/bogofilter -p -e -c /path/to/bogofilter.cf >>>> >>>> Aki >>> After looking into this some more, what's happening is that messages >>> are getting truncated. Regardless of whether it's an attachment or >>> just a long message body, the message gets truncated. >>> >>> To show this, I made a fileinto :copy rule to save a message copy >>> before it gets sent off to bogofilter. >>> >>> You can see from the below pastings that the message is definitely >>> getting truncated. When I replace my filter script with "cat >>> /dev/stdin", it works fine. When I send the message to bogofilter on >>> the command-line, it works fine. Something about the combination of >>> vnd.dovecot.filter + bogofilter is causing the messages to truncate, >>> and I don't know whether it's from truncating on the send or receive >>> of the filter. >>> >>> Can you give me any advice? >>> >>> 1) >>> $ jot 200000 1 > numbers.txt >>> $ du -a . | grep numbers >>> 1260??? ./numbers-sent.txt >>> 1248??? ./numbers-received.txt >>> >>> 2) >>> root at imap:~# ll test-* >>> 125 -rw-------? 1 root? wheel?? 123K Jan 20 09:35 test-afterbogo.msg >>> 149 -rw-------? 1 root? wheel?? 136K Jan 20 09:35 test-beforebogo.msg >> The more I look into it, the more it looks to me like pigeonhole is >> somehow losing the last 4-6K of messages over 100K. >> >> When my filter script is: >> cat /dev/stdin | tee /tmp/input | bogofilter[...] | tee /tmp/output >> Then /tmp/output is the full message, but what pigeonhole gets back is >> truncated at by about 4K. >> >> Same with >> [...] bogofilter[...] > /tmp/output; cat /tmp/output >> The message is truncated there too. >> >> I've tried everything I can think of to make sure it's not a buffering >> problem: >> [...] bogofilter[...] | cat >> [...] bogofilter[...] | dd bs=1k >> [...] bogofilter[...] ; a=$?; sleep 2; exit $a >> >> But everything I do comes up with the same problem. >> >> But a simple "cat /dev/stdin" and the message comes through fine. >> >> Your help is really appreciated. I'm in the process of buying a house, >> so the mangled attachments is not coming at a great time :-) > We are looking into it.Problem found; working on a fix. Regards, Stephan.
Op 1/22/2018 om 8:58 PM schreef Stephan Bosch:> Op 1/21/2018 om 4:34 PM schreef Stephan Bosch: >> Op 1/20/2018 om 11:01 PM schreef Adam Weinberger: >>>> On 20 Jan, 2018, at 10:05, Adam Weinberger <adamw at adamw.org> wrote: >>>> >>>> >>>>> On 19 Jan, 2018, at 4:39, Aki Tuomi <aki.tuomi at dovecot.fi> wrote: >>>>> >>>>> >>>>> >>>>> On 19.01.2018 04:35, Adam Weinberger wrote: >>>>>> Since upgrading to 2.3.0 / 0.5.0.1, incoming PDFs are getting mangled. >>>>>> It seems to be happening when I use vnd.dovecot.filter. When I comment >>>>>> out the block, things come through fine. >>>>>> >>>>>> My filter block looks like this: >>>>>> ?????? require "vnd.dovecot.filter"; >>>>>> ?????? filter "bogofilter_filter"; >>>>>> >>>>>> ??????? if header :contains "X-Bogosity" [ >>>>>> ??????????????? "Spam, tests=bogofilter, spamicity=1.00", >>>>>> ??????????????? "Spam, tests=bogofilter, spamicity=0.99" >>>>>> ??????? ] { >>>>>> ??????????? fileinto "spam/totally"; >>>>>> ??????????? stop; >>>>>> ??????? } >>>>>> ??????? elsif header :contains "X-Bogosity" "Spam," { >>>>>> ??????????? fileinto "spam/probably"; >>>>>> ??????????? stop; >>>>>> ??????? } >>>>>> ??????? elsif header :contains "X-Bogosity" "Unsure," { >>>>>> ??????????? fileinto "spam/maybe"; >>>>>> ??????????? stop; >>>>>> ??????? } >>>>>> >>>>>> filter/bogofilter_filter looks like this: >>>>>> ??????? #!/bin/sh >>>>>> ??????? cat /dev/stdin | /usr/local/bin/bogofilter -p -e -c >>>>>> /path/to/bogofilter.cf >>>>>> >>>>>> Images seem to come through fine. I didn't have this problem before >>>>>> the >>>>>> upgrade. It could easily be a problem with bogofilter, but I figured >>>>>> that I'd check here first to see if anybody else has an >>>>>> attachment-mangling issue with vnd.dovecot.filter. >>>>>> >>>>>> # Adam >>>>>> >>>>>> >>>>>> -- >>>>>> Adam Weinberger >>>>>> adamw at adamw.org >>>>>> http://www.adamw.org >>>>> It might work better with >>>>> >>>>> exec /usr/local/bin/bogofilter -p -e -c /path/to/bogofilter.cf >>>>> >>>>> Aki >>>> After looking into this some more, what's happening is that messages >>>> are getting truncated. Regardless of whether it's an attachment or >>>> just a long message body, the message gets truncated. >>>> >>>> To show this, I made a fileinto :copy rule to save a message copy >>>> before it gets sent off to bogofilter. >>>> >>>> You can see from the below pastings that the message is definitely >>>> getting truncated. When I replace my filter script with "cat >>>> /dev/stdin", it works fine. When I send the message to bogofilter on >>>> the command-line, it works fine. Something about the combination of >>>> vnd.dovecot.filter + bogofilter is causing the messages to truncate, >>>> and I don't know whether it's from truncating on the send or receive >>>> of the filter. >>>> >>>> Can you give me any advice? >>>> >>>> 1) >>>> $ jot 200000 1 > numbers.txt >>>> $ du -a . | grep numbers >>>> 1260??? ./numbers-sent.txt >>>> 1248??? ./numbers-received.txt >>>> >>>> 2) >>>> root at imap:~# ll test-* >>>> 125 -rw-------? 1 root? wheel?? 123K Jan 20 09:35 test-afterbogo.msg >>>> 149 -rw-------? 1 root? wheel?? 136K Jan 20 09:35 test-beforebogo.msg >>> The more I look into it, the more it looks to me like pigeonhole is >>> somehow losing the last 4-6K of messages over 100K. >>> >>> When my filter script is: >>> cat /dev/stdin | tee /tmp/input | bogofilter[...] | tee /tmp/output >>> Then /tmp/output is the full message, but what pigeonhole gets back is >>> truncated at by about 4K. >>> >>> Same with >>> [...] bogofilter[...] > /tmp/output; cat /tmp/output >>> The message is truncated there too. >>> >>> I've tried everything I can think of to make sure it's not a buffering >>> problem: >>> [...] bogofilter[...] | cat >>> [...] bogofilter[...] | dd bs=1k >>> [...] bogofilter[...] ; a=$?; sleep 2; exit $a >>> >>> But everything I do comes up with the same problem. >>> >>> But a simple "cat /dev/stdin" and the message comes through fine. >>> >>> Your help is really appreciated. I'm in the process of buying a house, >>> so the mangled attachments is not coming at a great time :-) >> We are looking into it. > Problem found; working on a fix.Fix pending: https://github.com/stephanbosch/dovecot-core/commits/fix-program-client-big-io This is a feature branch of 9 commits containing the fix and also cleanups and reliability improvements. This is of course based on master. Regards, Stephan.