Am 12.06.2020 um 02:03 schrieb Ralph Seichter:> * Andreas Born: > >> There exists one problem: at this stage of mail reception you have no >> body content nor header information on which a milter may perform >> deeper analysis, only envelope data. > > I am not sure what you mean by "this stage of mail reception", or whatI meant the different stages when receiving mails over SMTP: (very short and incomplete, I know): 1. MTA is connecting via SMTP, TLS, etc. 2. Identification (EHLO), Authentication, Protocol Extensions etc. 3. MTA send envelope information (MAIL TO, RCPT TO) 4. MTA sends message header and body (DATA, .) 5. Connection close (QUIT) or repeat from 3. for another mail 6. enqueuing mail(s) 7. Local Delivery I was referring to what you wrote with: >>> "Better to reject the offending message with a 5xx status code [...]" You surely refer to the 5xx status codes from SMTP, and to reject the mail while receiving it via SMTP, instead of sending a DSN later on? So the sender knows that the mail was not accepted, and that it MUST NOT try to resend the mail again (as with 4xx status codes). You further write: > For example: Postfix supports both before-queue filters and after-queue filters. Milter-regex[1] supports both multi-header and body checks. Of course, and there is nothing wrong with it. It just runs into the issue I tried to describe: incomplete SMTP implementations from MTAs. Pre-queue filtering happens, before the mail was accepted to be queued. So a before-queue milter can trigger an 5xx status code to reject the mail. This code can be sent in response to steps 2, 3 or 4. According to the smtp specs. But for many years it was code of practice to send error/rejection codes latest after the RCPT TO command, and at this time the milter, independent of what software you use, has no information about email header or content. Rejecting a mail AFTER the DATA command (when the content becomes available) was discouraged because of incorrect behaving MTAs. (e.g. generating backscatter, or even treating the mail as successfully sent) Maybe, and I really hope so, this problem no longer exists. I will immediately reconfigure my mail system, if rejecting mails after DATA will be safe and reliable nowadays. /andreas
On Fri, 12 Jun 2020, Andreas Born wrote:> Maybe, and I really hope so, this problem no longer exists. I will > immediately reconfigure my mail system, if rejecting mails after DATA will be > safe and reliable nowadays.In particular, bots don't hang around for the DATA response. Any MTA that ignores SMTP responses for the DATA step would also ignore common conditions like full mailbox. Such brokeness and failure to follow RFC is by itself grounds to reject the mail until the MTA software is fixed. One blacklist operator actually uses this as a criteria for blacklisting (Section: Tracking use of QUIT) http://wiki.junkemailfilter.com/index.php/Spam_DNS_Lists I issue post-DATA return codes, and I have yet, in decades of use, had problems with legitimate senders. Joseph Tam <jtam.home at gmail.com>
* Andreas Born:> I meant the different stages when receiving mails over SMTP [...]I am well aware of the technical details of SMTP. Your comment is unclear to me because the OP did not make any limitations on when he wants to counter spam, so why would we artificially limit ourselves in this discussion? SMTP allows rejecting messages even after the whole body has been received (end of DATA phase in SMTP, EOM phase in the milter protocol).> for many years it was code of practice to send error/rejection codes > latest after the RCPT TO command [...]Can you name examples? I have never used a milter so restricted, nor have I seen one being used.> Rejecting a mail AFTER the DATA command (when the content becomes > available) was discouraged because of incorrect behaving > MTAs. (e.g. generating backscatter, or even treating the mail as > successfully sent)Again I am stumped by what you write. SMTP rejection does not generate backscatter. Frankly, I see no use in discussing broken MTAs or milters, real or imagined. Using milter-regex and other milters with Postfix works fine as I described it. What you wrote therefore makes little sense to me. -Ralph
First thing first, this isn't necessary a Dovecot related thread and
using a challenge-response system like the one suggested by the
initiator ("click here if you're not yet another bloody SEO guru")
is
plain wrong for several reasons, having said that:
On 12-06-2020 11:56, Andreas Born wrote:> Am 12.06.2020 um 02:03 schrieb Ralph Seichter:
>> * Andreas Born:
[...]>> For example: Postfix supports both before-queue filters and
>> after-queue filters. Milter-regex[1] supports both multi-header and
>> body checks.
>
> Of course, and there is nothing wrong with it. It just runs into the
> issue I tried to describe: incomplete SMTP implementations from MTAs.
>
> Pre-queue filtering happens, before the mail was accepted to be
> queued. So a before-queue milter can trigger an 5xx status code to
> reject the mail. This code can be sent in response to steps 2, 3 or 4.
> According to the smtp specs. But for many years it was code of
> practice to send error/rejection codes latest after the RCPT TO
> command, and at this time the milter, independent of what software you
> use, has no information about email header or content. Rejecting a
> mail AFTER the DATA command (when the content becomes available) was
> discouraged because of incorrect behaving MTAs. (e.g. generating
> backscatter, or even treating the mail as successfully sent)
$ telnet server 25
Trying x.x.x.x...
Connected to server
Escape character is '^]'.
220-server ESMTP Postfix <=== Postscreen trap here ;)
220 server ESMTP Postfix
HELO client.domain.com
250 server
MAIL FROM:<>
250 2.1.0 Ok
RCPT TO:<victim at domain.com>
250 2.1.5 Ok
DATA
354 End data with <CR><LF>.<CR><LF>
From: Me
To: You
Subject: Test
SA GTube string here
.
550 5.7.1 Blocked, see you later.
QUIT
221 2.0.0 Bye
Connection closed by foreign host.
In this case the rejection comes after DATA, a content filter should be
able to return either 4xx or 5xx *after* swallowing the entire email.
> Maybe, and I really hope so, this problem no longer exists. I will
> immediately reconfigure my mail system, if rejecting mails after DATA
> will be safe and reliable nowadays.
Rejecting or deferring after DATA is perfectly fine these days. If the
sending MTA, acting as a client in the SMTP conversation, doesn't behave
properly to 5xx after DATA, it's not the recipient's MTA problem, the
sender is broken and there's nothing the receiving MTA can do about it.
Make it their problem, not yours.
--
Adi Pircalabu
Am 12.06.2020 um 04:28 schrieb Joseph Tam:> On Fri, 12 Jun 2020, Andreas Born wrote: > >> Maybe, and I really hope so, this problem no longer exists. I will >> immediately reconfigure my mail system, if rejecting mails after DATA >> will be safe and reliable nowadays. > > In particular, bots don't hang around for the DATA response. > > Any MTA that ignores SMTP responses for the DATA step would also ignore > common conditions like full mailbox.? Such brokeness and failure to > follow RFC is by itself grounds to reject the mail until the MTA software > is fixed.Ok, that really makes sense.> One blacklist operator actually uses this as a criteria for blacklisting > > ????(Section: Tracking use of QUIT) > ????http://wiki.junkemailfilter.com/index.php/Spam_DNS_ListsThat's helpful, thanks!> I issue post-DATA return codes, and I have yet, in decades of use, had > problems with legitimate senders.I too had never problems with legitimate senders. (of course, they don't get rejected) But I had -long ago- problems with large providers like gmx, hotmail or yahoo. E.g., sending rejected mails again and again, even sending DSNs to their (forged and forwarded) senders. But as said, long ago. So times have changed, that's great. / Andreas
Am 12.06.2020 um 04:43 schrieb Adi Pircalabu:> First thing first, this isn't necessary a Dovecot related thread andyeah, and i'm sorry for that. Just thought to give some information on an issue that actually turns out to be completely out-dated.> using a challenge-response system like the one suggested by the > initiator ("click here if you're not yet another bloody SEO guru") is > plain wrong for several reasons,full ack> $ telnet server 25 > Trying x.x.x.x... > Connected to server > Escape character is '^]'. > 220-server ESMTP Postfix <=== Postscreen trap here ;) > 220 server ESMTP Postfix > HELO client.domain.com > 250 server > MAIL FROM:<> > 250 2.1.0 Ok > RCPT TO:<victim at domain.com> > 250 2.1.5 Ok > DATA > 354 End data with <CR><LF>.<CR><LF> > From: Me > To: You > Subject: Test > > SA GTube string here > . > 550 5.7.1 Blocked, see you later. > QUIT > 221 2.0.0 Bye > Connection closed by foreign host. > > In this case the rejection comes after DATA, a content filter should be > able to return either 4xx or 5xx *after* swallowing the entire email.I know and I already tried out such a filtering long before. With lots of problems and long discussions to mailadmins of large providers. The result: specifications and reality do not always match. :-)>> Maybe, and I really hope so, this problem no longer exists. I will >> immediately reconfigure my mail system, if rejecting mails after DATA >> will be safe and reliable nowadays. > > Rejecting or deferring after DATA is perfectly fine these days. If the > sending MTA, acting as a client in the SMTP conversation, doesn't behave > properly to 5xx after DATA, it's not the recipient's MTA problem, the > sender is broken and there's nothing the receiving MTA can do about it. > Make it their problem, not yours.Okay, makes sense! / Andreas
Am 12.06.2020 um 04:43 schrieb Ralph Seichter:> * Andreas Born: > >> I meant the different stages when receiving mails over SMTP [...] > > I am well aware of the technical details of SMTP. Your comment is > unclear to me because the OP did not make any limitations on when he > wants to counter spam, so why would we artificially limit ourselves in > this discussion?THe OP wanted to send an additional mail back to possible unwanted senders. That's even worse than a DSN in my opinion. You suggested to reject mail directly during the smtp process. Full ack! I just talked about problems that existed in the past, that caused additional problems when trying to reject a mail after the SMTP DATA command. So my response was not about any limitation denoted by the OP, but about your suggestion that a prequeue-milter can even scan mail header and body as well before "hard rejecting" the mail. This had some conflicts in earlier times, but it turnes out, that this problem is out-dated and no more existent.> SMTP allows rejecting messages even after the whole > body has been received (end of DATA phase in SMTP, EOM phase in the > milter protocol).Yes. And it always has been. But i knew that this was not always respected by MTAs, even by some large email providers I had contact to.>> for many years it was code of practice to send error/rejection codes >> latest after the RCPT TO command [...] > > Can you name examples?postfix.org itself had some information in its postconf section. There ist still an information about smtpd_relay_reject, that some clients mis-behave on error codes before RCTP TO (the other way around), and in the past there (somewhere) has been an info about issues sending them after DATA, too, because of mis-behaving clients. That was discussed in many postfix-setup-tutorials also. Reject on RCPT TO. But okay, many years ago. I thought few thing don't change, some others do. But i never heard that this issue was ever solved. So my fault. Friends? ;-)> I have never used a milter so restricted, nor > have I seen one being used.Might be. But default configurations and most old tutorials have no smtpd_data_restrictions set up. And smtpd_data_end_restrictions did not even exist last time I changed my config. :)>> Rejecting a mail AFTER the DATA command (when the content becomes >> available) was discouraged because of incorrect behaving >> MTAs. (e.g. generating backscatter, or even treating the mail as >> successfully sent) > > Again I am stumped by what you write. SMTP rejection does not generate > backscatter.Indeed; not, when Client (MTA) and Server (MX) behave correctly. But some MTAs did not. Maybe they do NOW, so everything is fine. Mis-Behaving MTAs did generate DSNs on rejections after DATA, they sent it (under special circumstances like forwarding mails or multiple recipients) to their forged envelope-sender, so this is backscatter in my opinion. But they should not, because the mail was immediately rejected. Other mail-providers did not respect these recjections as permanent as they should do, because they didn't really evaluate error responses after DATA, the treated it as underlying transmission Problem, therefore trying to resend... And other providers did never generate an info to the user, that their mail did not arrive. A legal problem, when the recipient was a corporation and the mail a false positive etc..> Frankly, I see no use in discussing broken MTAs or milters, > real or imagined. Using milter-regex and other milters with Postfix > works fine as I described it. What you wrote therefore makes little > sense to me.I agree, there is really no use in discussing broken Software; at least as long, as the resulting problems do not restrict/limit your configuration possibilities or otherwise causing additional Spam, e.g. It was not about broken Milters, not at all. It was about broken MTAs. And about correctly working MX-Server like postfix, which had to be configured in such a way, that different problems were circumvented. Like broken MTAs generating spam because of your MX Server behaving correctly by sending 5xx after DATA. You can do anything right, and at the same time a bad client can make things bad. These problems really existed in the past, and I apologize if i'm completed out-of-date regarding these issues. / Andreas
+1, and while you SHOULD try to block before DATA when you can, to reduce loads on servers, and not tie up the SMTP connections any longer than you need, for rules bases on content, it is perfectly well to send a 5XX after data, and while there may be still some 'broken' MTA's and clients that don't properly wait for the response code after DATA, that isn't your problem.. The only complexity comes from when multiple recipients have different policies for acceptance based on DATA content. Which is why many systems still do content filtering AFTER acceptance. There ARE ways to address it in-stream, and there ARE now methods to return a variable acceptance after DATA, and there are resource intensive DATA handlers that are best used out of band (eg, after acceptance) but in general, the more you can stop inband, the better it is and less potential back scatter. But yeah.. this thread doesn't really belong on the dovecot list, so 'nuff said ;) On 2020-06-11 7:43 p.m., Adi Pircalabu wrote:> Rejecting or deferring after DATA is perfectly fine these days. If the > sending MTA, acting as a client in the SMTP conversation, doesn't behave > properly to 5xx after DATA, it's not the recipient's MTA problem, the > sender is broken and there's nothing the receiving MTA can do about it. > Make it their problem, not yours.-- "Catch the Magic of Linux..." ------------------------------------------------------------------------ Michael Peddemors, President/CEO LinuxMagic Inc. Visit us at http://www.linuxmagic.com @linuxmagic A Wizard IT Company - For More Info http://www.wizard.ca "LinuxMagic" a Registered TradeMark of Wizard Tower TechnoServices Ltd. ------------------------------------------------------------------------ 604-682-0300 Beautiful British Columbia, Canada This email and any electronic data contained are confidential and intended solely for the use of the individual or entity to which they are addressed. Please note that any views or opinions presented in this email are solely those of the author and are not intended to represent those of the company.