I am trying to diagnose a problem whereby a virus scanner (clam antivirus) is taking too long to scan attachments on a mail server. We have an attachment limitation of 20MB and an attachment of 7-20MB can take over 3 minutes to scan. This often causes the sending mail server to timeout and resend the mail. In this case, my mail gateway is is a dual 3.06GHz Xeon with 1GB of ram and 2 36GB 15krpm drives in a raid-1 on a smart array 6i (cciss) controller. I am running FreeBSD 5.4-RELEASE-p1. Systat -vmstat reports the disk mirror is 100% busy at all times on this machine, with an average of around 300 tps at 15KB/t. This seems wrong to me, as these numbers are maintained even when the system doesn't otherwise appear busy. We don't seem to be swamped by log writes. How can I tell what's generating these disk writes? At the moment the 100% disk utilization is the only thing I can see that would cause the scanning delay. The machine overall is sluggish with file operations. -Will -- Will Saxon Systems Programmer, Network Services University of Florida Department of Housing Phone: (352) 392-2171 x10148 Email: wills@housing.ufl.edu
On Sun, 16 Oct 2005 15:16:34 +0200, Will Saxon <WillS@housing.ufl.edu> wrote:> I am trying to diagnose a problem whereby a virus scanner (clam > antivirus) is taking too long to scan attachments on a mail server. We > have an attachment limitation of 20MB and an attachment of 7-20MB can > take over 3 minutes to scan. This often causes the sending mail server > to timeout and resend the mail. > > In this case, my mail gateway is is a dual 3.06GHz Xeon with 1GB of ram > and 2 36GB 15krpm drives in a raid-1 on a smart array 6i (cciss) > controller. I am running FreeBSD 5.4-RELEASE-p1. > > Systat -vmstat reports the disk mirror is 100% busy at all times on this > machine, with an average of around 300 tps at 15KB/t. This seems wrong > to me, as these numbers are maintained even when the system doesn't > otherwise appear busy. We don't seem to be swamped by log writes. How > can I tell what's generating these disk writes? At the moment the 100% > disk utilization is the only thing I can see that would cause the > scanning delay. The machine overall is sluggish with file operations.How many e-mails do you process every second/minute? Do you use softupdates are is the filesystem mounted 'sync'. Change the mailserver to first accept the message, then scan it, then deliver it. In that case you have much more control over how many messages are scanned at the same time, etc.. Ronald. -- Ronald Klop Amsterdam, The Netherlands
In the last episode (Oct 16), Will Saxon said:> I am trying to diagnose a problem whereby a virus scanner (clam > antivirus) is taking too long to scan attachments on a mail server. > We have an attachment limitation of 20MB and an attachment of 7-20MB > can take over 3 minutes to scan. This often causes the sending mail > server to timeout and resend the mail. > > In this case, my mail gateway is is a dual 3.06GHz Xeon with 1GB of > ram and 2 36GB 15krpm drives in a raid-1 on a smart array 6i (cciss) > controller. I am running FreeBSD 5.4-RELEASE-p1. > > Systat -vmstat reports the disk mirror is 100% busy at all times on > this machine, with an average of around 300 tps at 15KB/t. This seems > wrong to me, as these numbers are maintained even when the system > doesn't otherwise appear busy. We don't seem to be swamped by log > writes. How can I tell what's generating these disk writes? At the > moment the 100% disk utilization is the only thing I can see that > would cause the scanning delay. The machine overall is sluggish with > file operations.Are you swapping? Check either "vmstat 1" or top output. You can also tell top to display blocking I/O requests per process by hitting "m", then ask it to sort by I/O by hitting "ototal<enter>". -- Dan Nelson dnelson@allantgroup.com
At 9:16 AM -0400 2005-10-16, Will Saxon wrote:> In this case, my mail gateway is is a dual 3.06GHz Xeon with 1GB of ram > and 2 36GB 15krpm drives in a raid-1 on a smart array 6i (cciss) > controller. I am running FreeBSD 5.4-RELEASE-p1. > > Systat -vmstat reports the disk mirror is 100% busy at all times on this > machine, with an average of around 300 tps at 15KB/t.Note that RAID-1 is the second worst-case for mail server performance -- it accelerates reads (if you have mirror load-balancing), but all writes are required to be held until complete on both disks. The only worse case would be RAID-5, where you have to write (or re-write) an entire RAID block at once, plus the parity information. For mail servers, you really want to watch your synchronous meta-data updates. FreeBSD is a good choice here, if you've got Soft Updates enabled (I think that FreeBSD 5.x does that by default). But, you also want to watch your directory sizes. If the directory size gets too large, then it takes too long to lock the directory against any other updates, scan through the entire directory to make sure there aren't any collisions, create/delete the file, then unlock the directory -- a process which has to be done every time a file is created or deleted. This is why most modern mail servers use a "hashed queue" scheme, so that you can greatly increase the chances of multiple processes working simultaneously without stepping all over each others toes. However, with regards to directory size issues, keep in mind that even if the directory does not currently have 100,000 files in it, if it ever had 100k files in it in the past, it's still got all those empty directory slots laying around and that still slows things down a lot. If you suspect that this may have happened in the past, you need to stop the offending program, move the old directories aside, create new directories with the same ownership/permissions, then restart the program. And don't forget to make sure to clean out the old directories you had moved aside, either by creating some manual queue runners, or whatever. In your case, while the MTA may be configured in a way to avoid most of these issues, the anti-virus scanning solution may not. So, you may need to find a way to go in and deal with this. If you want to find out how all these issues affect the MTA, you need to read the book "sendmail Performance Tuning" by Nick Christenson (see <http://www.jetcafe.org/npc/book/sendmail/>). Once you read this book, you will hopefully have a better idea of how these same issues may affect your anti-virus scanning solution, and what you may need to do about it. I also recommend the slides from Nick's "Performance Tuning Sendmail Systems" paper at <http://www.jetcafe.org/npc/doc/performance_tuning.pdf>, as well as my own slides on the same general subject at <http://www.shub-internet.org/brad/papers/sendmail-tuning/>.> This seems wrong > to me, as these numbers are maintained even when the system doesn't > otherwise appear busy. We don't seem to be swamped by log writes.How can you be sure? How are you logging information today? Is that being logged to a separate filesystem on a separate disk system?> How > can I tell what's generating these disk writes? At the moment the 100% > disk utilization is the only thing I can see that would cause the > scanning delay. The machine overall is sluggish with file operations.You have a certain amount of information available to you from tools like vmstat and iostat, as well as systat. However, in order to understand how to use them to see where your problems really lie, you need information such as provided in Nick's book. You should also read other books on overall system performance tuning. The O'Reilly book on this subject (see <http://www.oreilly.com/catalog/spt2/>) is a good start, even though it is a few years old. -- Brad Knowles, <brad@stop.mail-abuse.org> "Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety." -- Benjamin Franklin (1706-1790), reply of the Pennsylvania Assembly to the Governor, November 11, 1755 SAGE member since 1995. See <http://www.sage.org/> for more info.
-----Original Message----- From: Ronald Klop [mailto:ronald-freebsd8@klop.yi.org] Sent: Sun 2005-10-16 12:40 To: Will Saxon; stable@freebsd.org Cc: Subject: Re: Disk 100% busy> On Sun, 16 Oct 2005 15:16:34 +0200, Will Saxon <WillS@housing.ufl.edu> > wrote: > > > I am trying to diagnose a problem whereby a virus scanner (clam > > antivirus) is taking too long to scan attachments on a mail server. We > > have an attachment limitation of 20MB and an attachment of 7-20MB can > > take over 3 minutes to scan. This often causes the sending mail server > > to timeout and resend the mail. > > > > In this case, my mail gateway is is a dual 3.06GHz Xeon with 1GB of ram > > and 2 36GB 15krpm drives in a raid-1 on a smart array 6i (cciss) > > controller. I am running FreeBSD 5.4-RELEASE-p1. > > > > Systat -vmstat reports the disk mirror is 100% busy at all times on this > > machine, with an average of around 300 tps at 15KB/t. This seems wrong > > to me, as these numbers are maintained even when the system doesn't > > otherwise appear busy. We don't seem to be swamped by log writes. How > > can I tell what's generating these disk writes? At the moment the 100% > > disk utilization is the only thing I can see that would cause the > > scanning delay. The machine overall is sluggish with file operations.> How many e-mails do you process every second/minute? > Do you use softupdates are is the filesystem mounted 'sync'. > Change the mailserver to first accept the message, then scan it, then > deliver it. In that case you have much more control over how many messages > are scanned at the same time, etc..I completely forgot that I had the partition mounted 'sync'. That might explain things a bit, huh. I am using qmail - the author indicates that softupdates is not recommended. However, I am going to give it a shot and see if I start losing mail as he suggests may happen. Thanks for answering! -Will
> -----Original Message----- > From: Mike Jakubik [mailto:mikej@rogers.com] > Sent: Monday, October 17, 2005 5:21 PM > To: Will Saxon > Cc: Ronald Klop; stable@freebsd.org > Subject: RE: Disk 100% busy > > > On Sun, October 16, 2005 1:53 pm, Will Saxon wrote: > > > I completely forgot that I had the partition mounted > 'sync'. That might > > explain things a bit, huh. > > > Do NOT mount the partition async, you are asking for > filesystem corruption.I was not planning to. I was thinking though that without softupdates and with sync, the disk would stay a lot busier. Since I have enabled softupdates, the disk is not nearly as busy. I may lose some mail if the machines panics for some reason, but I think that is not likely.> > > I am using qmail - the author indicates that softupdates is not > > recommended. However, I am going to give it a shot and see > if I start > > losing mail as he suggests may happen. > > Do yourself a big favor, and switch from this historic piece > of software > to something like Postfix.BSD has been around since the 70s, any suggestions for something more modern? Thanks for the response. -Will
On Tue, October 18, 2005 12:15 pm, Will Saxon wrote:>> BSD has been around since the 70s, any suggestions for something > more modern?But FreeBSD is in active development. Qmail is not and has not been for around 7 years.
On Mon, 17 Oct 2005, Brad Knowles wrote:> Note that RAID-1 is the second worst-case for mail server performance > -- it accelerates reads (if you have mirror load-balancing), but all writes > are required to be held until complete on both disks. The only worse case > would be RAID-5, where you have to write (or re-write) an entire RAID block > at once, plus the parity information.Coming late into the thread... What is a good raid level for a maildir IMAP server? RAID 10 (or 0+1 as others call it).
At 1:34 PM -0500 2005-11-03, Francisco wrote:> On Mon, 17 Oct 2005, Brad Knowles wrote: > >> Note that RAID-1 is the second worst-case for mail server performance -- >> it accelerates reads (if you have mirror load-balancing), but all writes >> are required to be held until complete on both disks. The only worse >> case would be RAID-5, where you have to write (or re-write) an entire >> RAID block at once, plus the parity information. > > Coming late into the thread... > What is a good raid level for a maildir IMAP server? RAID 10 (or 0+1 as > others call it).Yes, RAID 1+0 is generally considered to be the best. But keep in mind that you always want to stripe the mirrors and not mirror the stripes. If you do the former, then if one of the disks die then the mirror is broken but the stripe is still okay. If you do the latter, then if a disk dies then the stripe is broken, and then the mirror is also broken. -- Brad Knowles, <brad@stop.mail-abuse.org> "Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety." -- Benjamin Franklin (1706-1790), reply of the Pennsylvania Assembly to the Governor, November 11, 1755 SAGE member since 1995. See <http://www.sage.org/> for more info.
--- Francisco Reyes <francisco@natserv.net> wrote:> On Thu, 3 Nov 2005, michael meltzer wrote: > > > > > Controller: > > http://www.3ware.com/products/serial_ata2-9000.asp > > 16 port muili-lane, with BU and 265meg, cheaper > than > > most SCSI controller > > From what I gather, the ARECA controllers have > significantly better > performance. >http://www.tomshardware.com/storage/20051031/index.html problem is they only did a 8 banger at raid 5. http://www.gamepc.com/labs/view_content.asp?id=serioussataii&page=1 they did both 0 and 5 on a 8 banger Bottom line is they are the looking the same, AT least to my eye. the bigger issue for me is cabling, the sata connectors are a nightmare and a real PITA to work with, they fall off if you breath on the them wrong way or are doing any work in the box. It gets more fun when the "almost" fall off. 16 of them is evil :-) Multilane is the way to go for better reliability and to save me some problem now and in the future. The other issue is support, Vinod hangs out on the list and does a dam good job of supporting people. Better than most suport contracts, credit for good work and putting money were my mouth is, assuming the 16 port is not vaporware, ship in 30+- days, I use 6 on this job, it the right thing to do. -mjm __________________________________ Yahoo! FareChase: Search multiple travel sites in one click. http://farechase.yahoo.com