Paul Kudla (SCOM.CA Internet Services Inc.)
2022-Sep-26 11:25 UTC
new feature: sieve forward plugin
Ok this in itself is a issue however forwards should be fully received by the server and then resent to get around this issue. I use the mapping feature & the database to handle forwards in postfix which appears to work without any issues bypassing sieve. #Postgres Stuff mydestination = pgsql:/usr/home/postfix/config/pgsql-mydestination.cf #virtual_maps=pgsql:/usr/home/postfix/config/pgsql-virtual.cf alias_maps=pgsql:/usr/home/postfix/config/pgsql-aliases.cf #sender_canonical_maps = pgsql:/usr/home/postfix/config/pgsql-canonical.cf with # cat pgsql-mydestination.cf # /etc/postfix/pgsql-mydestination.cf ##################################### # pgsql config file for local domain (like sendmail's sendmail.cw) # lookups on postfix # comments are ok. # # the user name and password to log into the pgsql server hosts = pg.scom.ca:5433 user password # the database name on the servers dbname = scom_billing # the table name table = email_users # select_field = domain where_field = domain and # cat /usr/home/postfix/config/pgsql-aliases.cf # **** pgsql-virtual.cf *** # # pgsql config file for alias lookups on postfix # comments are ok. # # the user name and password to log into the pgsql server hosts = pg.scom.ca:5433 user password # the database name on the servers dbname = scom_billing # the table name table = email_users # #Select source email address alias (ie sales@ etc aliases ) where_field = source #Select destination email account address (final delivery) select_field = destination #Account Status (1=good) additional_conditions = and status = '1' So when using the above postfix receives it, remaps it and resends the email as its own thus fixing any spf issues along the way as it is sent by the local server. I understand that forwarding in a sieve script might over ride this and cause an spf failure, in that case (and i have not tried) then the sieve script should somehow deliver local and then resend? remapping the address through postfix would be the better approach. this would mark the email as coming from the local sending server and the spf record sent down the line would reflect that. spf verification would have already been verified by incoming postfix so you are not passing along something that got rejected in the first place? (The comments below are a little off topic but not really - spf issues are starting to become more of an issue as of late, it seems upstream providers are more interested in collecting customer data then just simply processing it and moving it along.) I recently went through this with microsoft outlook 365 please refer to microsoft [Case #:32804718] - Ticket 32800372 / 32795526 spf records : they refered me to a few links https://docs.microsoft.com/en-us/microsoft-365/troubleshoot/antispam/sender-rewriting-scheme sender-rewriting-scheme (which can not be turned off by microsoft) basically takes the from address of the origional email and resets it to the account email address thus when forwarded onwards the email only see's the microsft email address, the origional is dropped and does not even get passed in the headers. & https://docs.microsoft.com/en-us/microsoft-365/security/office-365-security/high-risk-delivery-pool-for-outbound-messages?view=o365-worldwide the above link was microsoft's explanation on why they refused to fix their spf record. when forwarding by a rule in microsoft (the only way to preserve the senders address email), microsoft receives it and then reque's the email through another server (in principal this is correct) but the forwarding server is considered insecure because microsoft fails their own spf record on purpose. Apparently google is now also using unverified (or insecure) servers setup the same way. why i have no idea? Microsoft replied with the ticket after 3 months of messing around that they would not fix their spf record. so basically microsoft was by design blocking their own outgoing server with an spf fail. Caption from microsoft ticket. We know nothing will make this better but to fix things on our side as you have said, but based on what we understood and what you have submitted to us, we understand the IP addresses you have mentioned are not listed Exchange Online's SPF record, however this is by design as we do not publish addresses for all traffic coming from our servers. In particular, traffic being relayed through Exchange Online may be treated this way. In order to ensure that mail flow is secure and being processed correctly, we recommend you use connectors when routing mail between Exchange Online and any on premise or partner organizations. With your permission, I am archiving the ticket # 32804718. that being said when i did a dig on outlook dig txt spf.protection.outlook.com i got back spf.protection.outlook.com. 600 IN TXT "v=spf1 ip4:40.92.0.0/15 ip4:40.107.0.0/16 ip4:52.100.0.0/14 ip4:104.47.0.0/17 ip6:2a01:111:f400::/48 ip6:2a01:111:f403::/49 ip6:2a01:111:f403:8000::/50 ip6:2a01:111:f403:c000::/51 ip6:2a01:111:f403:f000::/52 include:spfd.protection.outlook.com -all" all microsoft had to do was change ip4:40.92.0.0/15 to ip4:40.92.0.0/14 and the offending microsoft servers would have passed I was forced to whitelist this in my spf hoping this would not let anything bad through. I now have to track spf bounces daily from microsoft in case they change something else upstream that messes up my customer. Whats the point of spf if it does not get used correctly in the config record? Typical microsoft, however google & bell.ca (canadian isp) are having similiar issues when a customer of mine sends an email to 6 different people at the same time (just a normal email list nothing fancy), bell.ca (for example) seems to have that destination forwarded to google and is bouncing back as an spf error anyways (and this is on a normal send) So in the case of bell.ca's customer forwarding his/her account to google somewhere along the line the origional spf record (from my server) is being passed along upstream and when bell.ca rewrites it with their own something is getting scrambled and google thinks it is coming from my origional server but because it is going through bell.ca and forwarding they are obviously having the same problem as described here? Happy Monday !!! Thanks - paul Paul Kudla Scom.ca Internet Services <http://www.scom.ca> 004-1009 Byron Street South Whitby, Ontario - Canada L1N 4S3 Toronto 416.642.7266 Main?1.866.411.7266 Fax?1.888.892.7266 Email?paul at scom.ca On 9/26/2022 6:22 AM, Marc wrote:> As this page[1] describes a more often occurring problem of forwarding messages from servers that are not included in the spf records. Maybe there should be a plugin that offers this forward functionality. Something like > > get the spf records of the sender > check if there is a -all > then apply the sender substitution. > > https://doc.dovecot.org/configuration_manual/sieve/configuring_auto_forward_sender_address/ > > >
Hi Paul, I appreciate the huge response! :)> > Ok this in itself is a issue however forwards should be fully received > by the server and then resent to get around this issue. > > I use the mapping feature & the database to handle forwards in postfix > which appears to work without any issues bypassing sieve. >But this creates a dependency between your dovecot mail server and your outgoing. You should keep things simple.> #Postgres StuffI am using ldap ;)> > So when using the above postfix receives it, remaps it and resends the > email as its own thus fixing any spf issues along the way as it is sent > by the local server.Yes always. I do not see anything that distinguishes between senders that have or have not set spf. It is still nice to receive the forwarded message unchanged (think of internal delivery)> I understand that forwarding in a sieve script might over ride this and > cause an spf failure, in that case (and i have not tried) then the sieve > script should somehow deliver local and then resend?I have no idea, but this code forwards, so it should be possible to change the envelope here at this stage. # rule:[Forward] if false # true { redirect :copy "shit at gmail.com"; }> remapping the address through postfix would be the better approach.No, because postfix needs to know, and you create complicated relationships between outgoing mail servers and your mail server. It would not surprise me if your solution is also using much more resources, because your solution constantly has to verify. Less cpu cycles is better for the environment. ;)> this would mark the email as coming from the local sending server and > the spf record sent down the line would reflect that. > > spf verification would have already been verified by incoming postfix so > you are not passing along something that got rejected in the first > place?No, not really relevant.> > https://docs.microsoft.com/en-us/microsoft-365/security/office-365- > security/high-risk-delivery-pool-for-outbound-messages?view=o365- > worldwide > > the above link was microsoft's explanation on why they refused to fix > their spf record. >I see a manual about "Outbound delivery pools" no explanaition of why spf should not have '-all'> > Apparently google is now also using unverified (or insecure) servers > setup the same way. > > why i have no idea?The only reason for using ~all, is because they do not know what servers are sending outgoing email. Afaik this is just a statement of being incompetent. Google does not give a fuck if others need to clean up their mess.> Microsoft replied with the ticket after 3 months of messing around that > they would not fix their spf record. >Microsoft and Google are probably the companies you do 'business' with that have broken more laws than anyone else you know. Recently I have seen that Bill Gates is proud of his slogan "Why you should hire lazy people". Which undoubtedly is the cause of much of the shit I have with windows/exchange/outlook. So forgive me if I do not really take Microsoft or Google as an example nor standard.