freebsd stable - Jun 2004 - 5.2.1 with 40000 virus scanned mails / day on Dell hardware?(Hardware suggestions)

Hello,

I need to setup 3 servers on different locations as mail gateway with 
about 40k mail per day each. the servers runs MailScanner, Clamav and 
Spamassassin. The customer has Dell hardware and only buys Dell hardware.

Has anybody experience with Dell servers which runs under FreeBSD. I 
think about systems with Dual Xeon 3.0, 4 GB Ram and fast SCSI Hot Plug 
Raid 5 which should be strong enough to handle that amount of mails 
forwarding them to a Notes server.

Many thanks,

-- 
Best regards / Mit freundlichen Gruessen,

Karl M. Joch

> I need to setup 3 servers on different locations as mail gateway with
> about 40k mail per day each. the servers runs MailScanner, Clamav and
> Spamassassin. The customer has Dell hardware and only buys Dell
> hardware.
> Has anybody experience with Dell servers which runs under FreeBSD. I
> think about systems with Dual Xeon 3.0, 4 GB Ram and fast SCSI Hot
> Plug Raid 5 which should be strong enough to handle that amount of
> mails forwarding them to a Notes server.
Wow!  That's overkill, and then some.  :)  I don't have direct
experience with Dell servers (we're actively replacing those with
white-box systems), but me experience with a mail gateways follows:

We're running a dual-AthlonMP 2600+ (1.8 GHz) system with 3.5 GB DDR
RAM, and 3x200 GB WD IDE drives in RAID 5 acting as mail gateway. 
Software includes FreeBSD 5.2.1-p8, Postfix, Amavisd-new,
SpamAssassin, and ClamAV.

This server handles virus / spam filtering for 15 domains, and about
25,000 unique mail accounts (15,000 staff, 10,000 students, give or
take a bit).  We get a hell of a lot of messages per day (although I
haven't tracked the exact number or trends yet, but could extrapolate
around the 50,000 mark), and I've yet to see the server CPU load get
above 5 - 10 % and the system load rarely goes above 2.0.  The mail
queue rarely gets above 200 messages, and most of those are bounce
messages to non-existent or uncooperative servers.  (This server is
also being used to test Courier-IMAP and Cyrus-IMAP with SquirrelMail,
so there's the odd spike during testing, but everyday use never taxes
the system).

We're actually planning on replacing this with a non-RAID system with
a single AthlonXP or Athlon64 so we can use this server as part of the
back-end of the new mail system coming in next month.

-- 
Freddie Cash, CCNT CCLP        Helpdesk / Network Support Tech.
School District 73             (250) 377-HELP [377-4357]
fcash-ml@sd73.bc.ca

* Karl M. Joch <k.joch@ctseuro.com> (20040623
17:45):
> I need to setup 3 servers on different locations as mail gateway with 
> about 40k mail per day each. the servers runs MailScanner, Clamav and 
> Spamassassin. The customer has Dell hardware and only buys Dell hardware.
> 
> Has anybody experience with Dell servers which runs under FreeBSD. I 
> think about systems with Dual Xeon 3.0, 4 GB Ram and fast SCSI Hot Plug 
> Raid 5 which should be strong enough to handle that amount of mails 
> forwarding them to a Notes server.
If you are looking for 5.2.1, you are on the wrong list.

Anyway, a Pentium III with 256 Mb RAM handles virus scanning and
spam tagging for ~ 60k mails a day without any problem.

Your biggest bottleneck might be the Notes server.

-- 
olive

Hi.

On Wed, Jun 23, 2004 at 05:45:18PM +0200, Karl M. Joch
wrote:
> I need to setup 3 servers on different locations as mail gateway with 
> about 40k mail per day each. the servers runs MailScanner, Clamav and 
> Spamassassin. The customer has Dell hardware and only buys Dell hardware.
> 
> Has anybody experience with Dell servers which runs under FreeBSD. I 
> think about systems with Dual Xeon 3.0, 4 GB Ram and fast SCSI Hot Plug 
> Raid 5 which should be strong enough to handle that amount of mails 
> forwarding them to a Notes server.
Our current setup for exim + amavisd-new incl. SpamAssassin and clamav 
consists of a bunch of boxes with 2.4 GHz Xeon, 2 Gigs of RAM and 3ware 
SATA RAID 10 on 4 10k rpm disks. Each of these machines handles about 
half a million mails per day (though we currently have enough machines 
to load them with only about 300,000 mails per day).

With only 40k mails per day you should not see any problems, but you 
should consider using RAID 1 or RAID 10 because for this task I/O is the 
bottleneck, not so much CPU.

I disabled swapping completely btw.

- Oliver

-- 
| Oliver Brandmueller | Offenbacher Str. 1  | Germany       D-14197 Berlin |
| Fon +49-172-3130856 | Fax +49-172-3145027 | WWW:   http://the.addict.de/ |
|               Ich bin das Internet. Sowahr ich Gott helfe.               |
| Eine gewerbliche Nutzung aller enthaltenen Adressen ist nicht gestattet! |

On Wed, Jun 23, 2004 at 05:45:18PM +0200, Karl M. Joch
wrote:
> I need to setup 3 servers on different locations as mail gateway with 
> about 40k mail per day each. the servers runs MailScanner, Clamav and 
> Spamassassin. The customer has Dell hardware and only buys Dell hardware.
  I've got a couple of 2650s (2x2.4GHz and 2x3GHz Xeons) doing that fairly
well.  It is front-ending ~30K email accounts.
> Has anybody experience with Dell servers which runs under FreeBSD. I 
> think about systems with Dual Xeon 3.0, 4 GB Ram and fast SCSI Hot Plug 
> Raid 5 which should be strong enough to handle that amount of mails 
> forwarding them to a Notes server.
  We're running 5.2.1 (-p8 now), mostly for the bge0 support (at the time,
4.9+ wasn't recognizing the built-in Broadcom BCM5703 Gigabit Ethernet).
We're also running with HT, and since it passed the initial stress testing
we decided to go with it.

  We end up with ~100K emails per day, and we manage to keep up.  When the
spammers are normal, things are Ok, but every now and then one decides to
totally bomb us and we'll end up with queues 40K-75K deep that'll take
half
a day to clear.  95% of the time that is totally fine, but 5% of the time
(not real statistics) we get tied to the anvil and tossed into the deep end
of the pool and we wanted the extra pony-power.  We've spent a long time
and have a lot of students over the years one the one machine so we get hit
with a lot of email for people that aren't here anymore, which can act to
increase the user-count if you're not careful.

  Typical CPU utilization is generally low, so we don't end up CPU bound
unless we get bombed (and then the load gets up into the 6-7 range).

  I don't think that disk I/O ends up being a big factor, but we have 2G of
RAM in those to try and keep things in cache and that certainly works
pretty good for us.  We tend to have ~10K of bogus unreturnable-to-sender
spam backlogged and that can cause the queue-runners to pile up, and the
mailscanner perl processes with clamav+spamassassin in them are huge.

  This is not a mail-server, it is only a MX so unless we're backlogged
we're probably running out of cache on not really hitting the disk much.


  My $.02: Anytime you act as a MX relay, be sure you have some way of
knowing what are legitimate users or not so you can give the smaller the
5xx permanent failure messages rather then 4xx (or accepting, then trying
to bounce email to bogus addresses).  You'll save yourself a lot of
busywork that way and you're server will scale better.

> Has anybody experience with Dell servers which runs under FreeBSD. I 
> think about systems with Dual Xeon 3.0, 4 GB Ram and fast SCSI Hot Plug 
> Raid 5 which should be strong enough to handle that amount of mails 
> forwarding them to a Notes server.
 
I would use RAID 0+1 or just plain RAID 0 since these should be
cookie-cutter.

-- 
Joe Rhett
Senior Systems Engineer
Meer.net

Joe Rhett wrote:
>>Has anybody experience with Dell servers which runs under FreeBSD. I 
>>think about systems with Dual Xeon 3.0, 4 GB Ram and fast SCSI Hot Plug 
>>Raid 5 which should be strong enough to handle that amount of mails 
>>forwarding them to a Notes server.
> 
>  
> I would use RAID 0+1 or just plain RAID 0 since these should be
> cookie-cutter.
> 
I agree that a basic RAID may be desirable. We have a series of dual 
Xeon 2.4GHz with 1-2Gig RAM processing maybe 1/2 million pieces of email 
a day. These machines in particular do spamassassin and run in parallel 
(6 machines behind a load balancer). During mail bombs the machines show 
signs of sweating under the perl load and there is the recurring issus 
of either softupdates causing backtraces or some other memory paging 
event causing them - again only under the severest of loads. We do not 
use a RAID solution on those so using a simple strip may help avoid the 
situation we are seeing (5.2.1-Release-P8). We also use FreeBSD on our 
primary MX machines (again a load-balanced series of 3 machines) and 
these are not running any perl processes - simply sendmail with 
extensive access lists. These machines run flawlessly and combined 
handle some 2 million+ emails a day.

Sven