Pkg exim4 users - Mar 2009 - Exim4 with local network as well as smarthost (longish post)

The standard debian exim configurations do not seem to cater
for a small local network behind a NAT router.

Most mail needs to go to an (isp) smarthost with FROM headers
rewritten. But local mail within the network needs to be delivered
directly with headers unchanged.

This already presents a problem to exim4 because, as I understand it,
envelope rewriting can only be done globally.

My local network is small and no single machine is necessarily
running at all times. Thus a dedicated local network mail machine
is not an option. Instead each machine runs its own copy of exim4
and accepts email directly from its local peers.

The local hosts each have the local network hostnames in /etc/hosts:
these are not registered dns names and are aliases as usual for
reserved 10.0.0.*, 192.168.*.* or 172.16.*.* IP4 addresses.

As a total novice at exim4, I have cobbled together a solution which
appears to work, except that I have found no way to rewrite the envelope
headers when sending to the smarthost. As I understand it, bounce 
messages are sent to the envelope address rather than to the FROM or 
REPLY-TO headers of the message itself. Thus a smarthost trying to 
bounce will encounter an invalid address,or just conceivably will bounce 
to the wrong place.

I describe my configuration below for two reasons:

1) to ask for a better solution from more experienced people;
2) in case it might be useful to others, despite the minor (?) problem.


If the global rewrite limitation on envelope headers really is
the problem that it seems to be, perhaps upstream might consider
a modification in some future version.

-------------------------------------------------------------------------

The description here is using a monolithic configuration.

First I define a new macro LOCAL_NETWORK in exim4.conf.localmacros.

LOCAL_NETWORK is a regular expression used to recognise the local network
names (held in /etc/hosts). I tried to get exim4 to examine /etc/hosts
directly, but eventually resorted to writing the regular expression
manually:

--------------- exim4.conf.localmacros -------------------------

# Define macros for local network here

LOCAL_NETWORKS = 
\N^(?:mouse\d[^.]?|cw\d|w?(?:cat|tiger|lion))(?:\.rodent)?$\N

# Since LOCAL_NETWORKS appears in string lists with default separator :,
# those lists need default changed (or ::/: above)

#Since main rewriting rules are global we must turn them off:
NO_EAA_REWRITE_REWRITE = true
#To compensate, must add them to right transport

REMOTE_NETS_REWRITE1 = \
*@+local_domains "${lookup{${local_part}}lsearch{/etc/email-addresses}\
                    {$value}fail}" frs
REMOTE_NETS_REWRITE2 = \
*@ETC_MAILNAME "${lookup{${local_part}}lsearch{/etc/email-addresses}\
                    {$value}fail}" frs

REMOTE_NETS_REWRITES = REMOTE_NETS_REWRITE1 : REMOTE_NETS_REWRITE2

-----------------------------------------------------------------------

I had to edit update-exim4.conf.conf manually:

----------------- update-exim4.conf.conf ----------------
#...

dc_eximconfig_configtype=''smarthost''
dc_other_hostnames=''mousebag.rodent''
dc_local_interfaces=''''
dc_readhost=''mouse1.rodent''
dc_relay_domains=''''
dc_minimaldns=''true''
dc_relay_nets=''''
dc_smarthost=''smtp.isp.com''
CFILEMODE=''644''
dc_use_split_config=''false''
dc_hide_mailname=''false''
dc_mailname_in_oh=''true''
dc_localdelivery=''mail_spool''
---------------------------------------------------

dpkg-reconfigure exim4 would not let me have smarthost and
dc_hide_mailname=''false'' together. This last was to turn off
that
troublesome global envelope rewriting.

local_host_blacklist had
!192.168.0.0/17
to allow the incoming mail from local peers.
{ The local network has a wifi subnet. }

I had to edit exim4.conf.template:

First I added a new initial router to catch the local network mail:

----------------------------------------------------------------
  begin routers

#### LOCAL_NETWORKS is an RE from exim4.conf.localmacros ###

.ifdef LOCAL_NETWORKS
local_net:
   debug_print = "R: local_net for $domain"
   driver  = manualroute
   domains = <; LOCAL_NETWORKS
   route_list = * $domain byname
   condition = ${if match_domain{$domain}{+local_domains}{no}{yes}}
   transport = smtp_lnet
.endif

------------------------------------------------------------------
"byname" above ensures that the local network names can be translated
from /etc/hosts. The condition makes sure that the local names on the
host machine pass through.

It uses a dedicated transport smtp_lnet defined below:

--------------------------------------------------

  begin transports

# transport to avoid from rewrite

.ifdef LOCAL_NETWORKS
smtp_lnet:
   debug_print= "T: smtp_lnet for $local_part@$domain"
   driver = smtp
.endif

----------------------------------------------------------

But since the global header rewrite has been turned off, I had
to hack the next best thing

----------------------------------------------------------------
# local nets hack
.ifdef LOCAL_NETWORKS
REMOTE_SMTP_HEADERS_REWRITE=REMOTE_NETS_REWRITES
.elifdef HIDE_MAILNAME
REMOTE_SMTP_HEADERS_REWRITE=*@+local_domains $1 at DCreadhost frs : 
*@ETC_MAILNAME $1 at DCreadhost frs
.endif

.ifdef HIDE_MAILNAME
REMOTE_SMTP_RETURN_PATH=${if 
match_domain{$sender_address_domain}{+local_domains}{${sender_address_local_part}@DCreadhost}{${if
match_domain{$sender_address_domain}{ETC_MAILNAME}{${sender_address_local_part}@DCreadhost}fail}}}

.endif
# end of local nets hack

-------------------------------------------------------------------

That gets picked up in the remote_smtp_smarthost transport where
it causes the appropriate non-envelope headers to be re-written.

====================================================================
Warning. I have not considered TLS or other configuration options
which may well be broken as the above stands.

ael

ael wrote:
> The standard debian exim configurations do not seem to cater
> for a small local network behind a NAT router.
Actually it''s not the job of an MTA to do this. I have an instance of 
exim setup behind a NAT router and it works flawless (including 
spamfiltering and what have you). Just make sure you route port 25 and 
whatever else to the proper internal IP. Provided you don''t sabotage 
yourself...
> Most mail needs to go to an (isp) smarthost with FROM headers
> rewritten. But local mail within the network needs to be delivered
> directly with headers unchanged.
> This already presents a problem to exim4 because, as I understand it,
> envelope rewriting can only be done globally.
I do not know of any rewriting of the headers that needs to be done to 
send email within the network as well as to the outside. Beyond that 
which the standard exim4-config configurations options may decide upon. 
I believe the only thing that might cause a header rewrite is the "hide 
machine" name option, but I am not sure.

In your case you might prefer to have the machine name not hidden.
> My local network is small and no single machine is necessarily
> running at all times. Thus a dedicated local network mail machine
That''s what I mean by sabotaging yourself, you really should have one 
machine intended to run all the time as the mailserver. It''s not that 
hard. Then configuring exim is rather easy. Exim will do a good job 
queuing email and deliver it whenever a machine comes up.

That machine will be the gateway and is configured as an internet facing 
mailserver, all other machines are configured to use the gateway machine 
as a smarthost for outgoing mail. But those machines could have a local 
MTA to submit email to of course. The gateway machine can be told to 
route the appropriate incoming emails to the right machine. Based on the 
recipient address, i.e. email name at machine1.example.com goes to the 
machine named machine1.

But you could also use a central mailstore where people can collect 
there email using imap/pop3. In that way they all can use a uniform 
email address (name at example.com), where the machine name is left out. 
Which I think is a better idea.
> headers when sending to the smarthost. As I understand it, bounce 
> messages are sent to the envelope address rather than to the FROM or 
As a rule I''d try to avoid any rewriting of headers unless
it''s really
necessary.

Best regards,
Jeroen

Jeroen van Aart wrote:
> ael wrote:
>> The standard debian exim configurations do not seem to cater
>> for a small local network behind a NAT router.
> 
> Actually it''s not the job of an MTA to do this. I have an instance
of
> exim setup behind a NAT router and it works flawless (including 
> spamfiltering and what have you). Just make sure you route port 25 and 
> whatever else to the proper internal IP. Provided you don''t
sabotage
> yourself...
> 
>> Most mail needs to go to an (isp) smarthost with FROM headers
>> rewritten. But local mail within the network needs to be delivered
>> directly with headers unchanged.
> 
>> This already presents a problem to exim4 because, as I understand it,
>> envelope rewriting can only be done globally.
> 
> I do not know of any rewriting of the headers that needs to be done to 
> send email within the network as well as to the outside. 
That''s the point: I *don''t* want any rewriting for mail within
the local
network, but it *must* be done for mail to the smarthost. If the 
envelope rewriting is global, then I can''t satisfy both requirements.
Or
am I missing something?

> Beyond that 
> which the standard exim4-config configurations options may decide upon. 
> I believe the only thing that might cause a header rewrite is the
"hide
> machine" name option, but I am not sure.
> 
> In your case you might prefer to have the machine name not hidden.
> 
>> My local network is small and no single machine is necessarily
>> running at all times. Thus a dedicated local network mail machine
> 
> That''s what I mean by sabotaging yourself, you really should have
one
> machine intended to run all the time as the mailserve
It is not hard, but it is a ridiculous waste of energy: this is a 
*small* domestic network. I did say explicitly above that this sort of 
solution was not appropriate here.
> That machine will be the gateway and is configured as an internet facing 
> mailserver, 
Yes, I understand that approach: I read the archives and searched 
elsewhere and saw that sort of solution before posting. For a small 
office or business, it would be very sensible. But a profligate waste of 
precious resources in a lightly used domestic system.

Surely Debian should cater for all sorts of users in as flexible a way 
as possible?

But thanks for the reply: you seem to be confirming my feeling that this 
is an inherent limitation of the current version of exim.

ael

ael wrote:
> It is not hard, but it is a ridiculous waste of energy: this is a 
> *small* domestic network. I did say explicitly above that this sort of 
> solution was not appropriate here.
That sort of solution is appropriate almost anywhere. And at least in a 
small domestic network. You can easily find energy efficient systems, 
such as micro or pico ITX, or even laptops, which would serve well to be 
an always on mailserver. I have an old 800 mHz IBM thinkpad which has 
been on for almost 400 days and it barely makes a dent in the energy 
bill. My home mailserver is a G4 based system though, which similarly is 
very energy efficient.

I''d argue your solution will waste more resources than you are trying
to
save. If not in energy than at least in man hours.

Or are we talking about a 3rd world place with *very* limited resources?
> Surely Debian should cater for all sorts of users in as flexible a way 
> as possible?
It''s all open source, you can make it do whatever you want, within the 
limitations of the hardware. Just remember this excerpt from the GNU 
General Public License: "...without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE..."
> But thanks for the reply: you seem to be confirming my feeling that this is
an inherent limitation of the current version of exim.
Since you said that you are a total novice I do not believe you''re 
actually correct.

You can still use the simpler and more manageable and robust solution of 
having 1 central gateway server even if it is not always on. Email 
intended for delivery will just queue up at the sender''s MTA until you 
turn on your gateway machine and then deliveries will occur. Likewise 
mail will queue until other machines in your network are turned on.

Greetings,
Jeroen

ael wrote:
> The standard debian exim configurations do not seem to cater
> for a small local network behind a NAT router.
   That is true.  The Debian exim4 maintainers have tried to create a 
default configuration which can be tweaked by those debconf questions to 
fit the needs of the vast majority of people.

   For you and I, we are not the vast majority.  I have very similar 
needs to what you are describing, and received few responses on this 
list when I asked very similar questions in January.  I was advised to 
go to the mailing lists maintained by the upstream Exim developers.

> Most mail needs to go to an (isp) smarthost with FROM headers
> rewritten. But local mail within the network needs to be delivered
> directly with headers unchanged.
   I accomplished this, but had to rewrite some of the Exim rules in 
order to do it.

   I have 3 machines in my home network.  Here is a quick diagram:

ISP  <-->  gateway/router____ desktop
                      \  \____ fileserver
                       \______ webserver

This is a temporary setup, and will later become:

ISP <--> webserver <--> gateway/router <--> desktop <-->
fileserver


   I wanted to get ready for the future setup by configuring "desktop"
and "fileserver" to use "webserver" as their smarthost (in
the first
diagram).  I wanted any message sent _from_ a local machine _to_ a local 
machine to be correctly routed by the smarthost back to the LAN, with no 
headers rewritten.  I also needed any message sent (from any machine) 
outside the home network to have their headers rewritten.
   The current set of debconf questions do not allow for such a setup, 
since 99+% of users will either be using a single machine or will not 
have such picky needs for their home network email arrangement.  Adding 
such configuration options would require some significant changes to the 
current Debian exim4 configuration files, and additional debconf 
questions.  The Debian exim4 maintainers are adamant that their are 
already too many debconf questions, and they will resist (or, more 
likely, reject) any requests for additional support for rare cases like 
ours.

   If you feel my situation is similar enough to yours, I would be happy 
to share the changes I made to get local emails to pass through 
"webserver" unmodified, but external emails having their headers 
rewritten so that the reply address is the correct email address of my 
ISP account.

> This already presents a problem to exim4 because, as I understand it,
> envelope rewriting can only be done globally.
   Actually, exim4 is VERY configurable.  Your complaint is not against 
exim4, but against the default configuration provided by the Debian Exim 
maintainers.
   In my case, I decided to keep as much as possible from the 
configuration provided by the Debian team because (1) I have never 
configured Exim from scratch and (2) I have little desire to become an 
expert Exim configurer just to set up a little 3-machine home network!

   Please don''t blame the Debian team for their choices in preparing
the
default configuration.  What they have done is almost miraculous:  with 
a few short questions, they have made it possible for the overwhelming 
majority of people to use Exim without having to write an Exim 
configuration from scratch.  That was their goal, and they have 
succeeded.  It leaves people like us being forced to read all (or most) 
of the documentation trying to figure out how to get what we want, but 
you cannot expect them to handle every conceivable usage of Exim!

> My local network is small and no single machine is necessarily
> running at all times. Thus a dedicated local network mail machine
> is not an option. Instead each machine runs its own copy of exim4
> and accepts email directly from its local peers.
   That makes sense.  I _do_ leave one machine on all of the time, but I 
built it from scratch to use very low power components.  That is 
"webserver" (see above), and since it will be running 24/7/365, I
expect
parts to fail, so I designed it to be extremely easy to service, both in 
terms of hardware and software.
   If you''re not going to be using your machines constantly, then it
may
make more sense to shut them down -- especially if you will not be 
needing them for days at a time.

> The local hosts each have the local network hostnames in /etc/hosts:
> these are not registered dns names and are aliases as usual for
> reserved 10.0.0.*, 192.168.*.* or 172.16.*.* IP4 addresses.
   I also am not running DNS on the home network, though I may 
eventually do so.  (Just for the experience, more than anything.)  At 
the moment, I also rely on /etc/hosts, and that was part of the 
rewriting I had to do in order to prevent header rewrites when sending 
to home network destinations.

> I describe my configuration below for two reasons:
   To be honest, I haven''t worked on this since January.  I cannot 
recall from memory what my changes were, but you seem to have taken a 
much different approach from what I did.
   As I mentioned before, if you think my situation is similar enough to 
your own, I''ll be glad to go back over the notes I took and share the 
details of my own setup.  We may both end up headed for the upstream 
mailing list to further tweak our setups for what we really want, 
though.  For now, my setup is working, but I still have questions about 
whether I did it right, or whether there are better ways to do the same 
thing.


Dave W.

Dave Witbrodt wrote:
>> Most mail needs to go to an (isp) smarthost with FROM headers
>> rewritten. But local mail within the network needs to be delivered
>> directly with headers unchanged.
> 
>    I accomplished this, but had to rewrite some of the Exim rules in 
> order to do it.
> 
----
>    If you feel my situation is similar enough to yours, I would be happy 
> to share the changes I made to get local emails to pass through 
> "webserver" unmodified, but external emails having their headers 
> rewritten so that the reply address is the correct email address of my 
> ISP account.
Dave, would you be willing to share it with the list? I have been 
looking for such a solution also for quite a time. The only point where 
I disagree with you is, that I don''t think this is a very rare setup.
Just think of a company where every person should be able to communicate 
with every other person locally but only a few official e-mail addresses 
(like office at example.com or sales at example.com) should be used for 
communication with the outside world.

Thanks a lot in advance,

jutta

Jeroen van Aart wrote:
> ael wrote:
>> It is not hard, but it is a ridiculous waste of energy: this is a 
>> *small* domestic network. I did say explicitly above that this sort of 
>> solution was not appropriate here.
> 
> That sort of solution is appropriate almost anywhere. And at least in a 
> small domestic network. You can easily find energy efficient systems, 
> such as micro or pico ITX, or even laptops, which would serve well to be 
> an always on mailserver. 
Yes, I know about hardware and what is available. I have also done 
electronic design. But again. it is just silly to have to put a special 
system together just to cater for rare internal emails. But equally 
silly that a linux network can''t easily accomplice such a simple task 
without special effort.

I don''t know whether it was something in the way that I phrased the 
original email (I was trying to be concise) that is causing people to be 
so defensive. I was not trying to criticise: just explaining that I 
couldn''t seem satisfy what appeared to be simple requirement which I 
would expect to be quite common in small domestic situations. And asking 
for comments on my particular solution which, of course, tried to retain 
the debconf functionality.

I wasn''t expecting the answer "change the requirement" !

I know very well that this sort of case was far from Philip Hazel''s 
concerns when he designed exim at Cantab, so it is not surprising that
problems can arise for tiny networks.

One of the things I love about linux is that it is so flexible that it 
caters for almost any conceivable system from tiny heritage systems to 
full blown supercomputers. We just seem to be lacking a bit of 
flexibility at the low end with exim4 as it is today. Perhaps.

I haven''t examined the exim source, but I can guess that rewriting the 
envelope is done at RCPT time and that it would involve major 
reorganization to allow selective rewriting later.  I am far too much of 
a novice at the subtleties of email to think of modifying exim4 just 
now, nor do I have the time unfortunately.

ael

Jutta Zalud wrote:
> Dave Witbrodt wrote:
> 
>>> Most mail needs to go to an (isp) smarthost with FROM headers
>>> rewritten. But local mail within the network needs to be delivered
>>> directly with headers unchanged.
>>    I accomplished this, but had to rewrite some of the Exim rules in 
>> order to do it.
>>
> ----
> 
>>    If you feel my situation is similar enough to yours, I would be
happy
>> to share the changes I made to get local emails to pass through 
>> "webserver" unmodified, but external emails having their
headers
>> rewritten so that the reply address is the correct email address of my 
>> ISP account.
> 
> Dave, would you be willing to share it with the list? I have been 
> looking for such a solution also for quite a time. The only point where 
> I disagree with you is, that I don''t think this is a very rare
setup.
> Just think of a company where every person should be able to communicate 
> with every other person locally but only a few official e-mail addresses 
> (like office at example.com or sales at example.com) should be used for 
> communication with the outside world.
   I don''t feel strongly enough about this to pursue the debate very 
much farther, but your reasoning doesn''t convince me that a very high 
percentage of machines running Exim need a configuration that the Debian 
Exim Maintainers haven''t provided for with their debconf questions.  My
guess is that the overwhelming majority of the machines where Exim is 
installed have owners that don''t even realize Exim is present.
   Having narrowed down the percentage to the people who know it''s 
present, most of them (if they are using Debian or a derivative of 
Debian) are served by the default configuration + debconf questions.  Of 
this group of people, if they are even aware that they have needs that 
are not served by the Exim configuration provided by the Debian 
maintainers, then they probably already have experience configuring 
MTA''s... so they dump Exim and install Postfix or Sendmail.  (Not 
because Exim is inferior, but because they have experience with some 
other MTA.)

   In short, Debian users who pursue usage of Exim as far as you and I 
are talking about are rare extremists!  ;)

   Having said that, I am posting the information you requested in a 
separate email... with a subject that hopefully will catch some more 
attention:

       Exim4 home network configuration with smarthost
       but no local header rewrites

Look for that one for my response to your question.


DW