I am looking at ways to harden my asterisk install to prevent computer related issues from happening. I am concerned about about disk write cache. That seems to be a major source of hard drive corruption on power failure. Hard Drive corruption is simply unacceptable for the 99.999% uptime requirements of my Asterisk install that needs to be as reliable as a proprietary PBX. Of course I will be using redundant power supplies, raid 1 and use a UPS. None of those things mean much if the power cords accidentally get pulled from the back of the server. Unlikely as it may be I have to consider ALL possibilities. So is disabling the write cache a good way to reduce the risk of hard drive corruption for an Asterisk server? I am not too concerned about the reduced performance/lifetime of hardrives with write cache disabled since Asterisk is not a very write intensive environment. Even with lot's of voicemail going on. Any other recommendations/links for increasing the reliability of Asterisk servers?
Raid card with an onboard battery backup. PaulH -- Paul Hales Technical Manager AsteriskIT www.asteriskit.com.au bus: 03 8320 8100 mob: 0434 673 529 shadowym wrote:> > I am looking at ways to harden my asterisk install to prevent computer > related issues from happening. I am concerned about about disk write cache. > That seems to be a major source of hard drive corruption on power failure. > Hard Drive corruption is simply unacceptable for the 99.999% uptime > requirements of my Asterisk install that needs to be as reliable as a > proprietary PBX. > > Of course I will be using redundant power supplies, raid 1 and use a UPS. > None of those things mean much if the power cords accidentally get pulled > from the back of the server. Unlikely as it may be I have to consider ALL > possibilities. > > So is disabling the write cache a good way to reduce the risk of hard drive > corruption for an Asterisk server? I am not too concerned about the reduced > performance/lifetime of hardrives with write cache disabled since Asterisk > is not a very write intensive environment. Even with lot's of voicemail > going on. > > Any other recommendations/links for increasing the reliability of Asterisk > servers? > _______________________________________________ > --Bandwidth and Colocation provided by Easynews.com -- > > Asterisk-Users mailing list > To UNSUBSCRIBE or update options visit: > http://lists.digium.com/mailman/listinfo/asterisk-users >
shadowym wrote:> Any other recommendations/links for increasing the reliability of Asterisk > servers?Separate the various use cases of the filesystem into different volumes with LVM. The parts that are not written to except during upgrades like /usr should be mounted read-only, and the various read/write sections like /var/spool/asterisk and /var/log should be on separate volumes also. This keeps any corruption experienced localized to a small area, and keeps your binaries unquestionably safe from a power outage and the only step needed is automated detection and cleanup of the read/write volumes. Write barriers allow you to keep write-cache turned on in the drives, and sends a command to the drive to reply when the data hits the platters. Also if the drive doesn't support that, various techniques are used to verify the data is on the media and not in drive cache. Contact your distro support company and ask them if write-barriers have been implemented in the drivers for the drive controllers you are using, and if not, then either buy the hardware that does support that or sponsor them to update the driver to support write-barriers. Also contact your distro support company to see if they have any recommendations for the setup you want. Mike
Hi, shadowym wrote:> I am looking at ways to harden my asterisk install to prevent computer > related issues from happening. I am concerned about about disk write cache. > That seems to be a major source of hard drive corruption on power failure. > Hard Drive corruption is simply unacceptable for the 99.999% uptime > requirements of my Asterisk install that needs to be as reliable as a > proprietary PBX.Things to consider: - Use compactflash to boot and run asterisk, add disk only for voicemail - Run the entire setup from a ram disk, make commit/rollback facilities to write to disk - Extra servers are cheap - you could use LinuxHA to failover the server. Florian
These days you don't have to worry much about your write cache unless
you're running application where once single byte changed will affect
whole file.
Look at it this way, the only corruption will occur is whatever the
files were open by asterisk at the time of the crash. And only up to the
point where the file was last open. As far as I know asterisk does not
keep cdr or log files open so you would loose only the data that was
written at the time of the power failure.
Any journaling file system (ext3, resierfs, xfs, etc) will easily handle
any power failure event. Your files will not be corrupt but could miss
some of the data.
At the most you will loose 10-50 cdr entries written to you log files.
If you post CDR to a remote SQL database then you asterisk install and
linux is more or less static and will not be affected by the power
failure.
What you need to do is minimise the writes to hard disk's:
1 - Send syslog to remote server and do not do ANY syslogs
Or keep the circular buffer in memory if you have plenty of it.
2 - Send CDR's to SQL server (or log to ramdisk and send to remote
server every few minutes via SSH)
3 - Do not record any calls (or do that somewhere else)
4 - Stop any services that write/read data on regular intervals.
If you have no writes you have nothing to worry about during power
failure and journaling file system will take care of the rest.
Keep your partition size really small so that fsck will not take much
time.
You have to be realistic, you cannot achieve 99.999% uptime. That's 5
minutes per year downtime.
You will have more or less 100% until your first hardware failure.
Even if you have all the hardware components pre-purchased it will still
take you 2-12 hours to detect, diagnose and fix the fault if you lucky.
So your 5 minuets
If the business is demanding 99.999% then it should be prepared to
invest into the hardware.
I would recommend a cluster or even better a fault tolerant server.
Those are expensive but you can pretty much rule out the hardware
failure and swap all of the failed components while the system is
running (cpu, memory, hdd, etc).
Look at Stratus or NEC FT servers if you need hardware redundancy.
They're expensive but will give you the hardware reliability you need.
Or get a traditional PABX :)
> -----Original Message-----
> From: asterisk-users-bounces@lists.digium.com [mailto:asterisk-users-
> bounces@lists.digium.com] On Behalf Of shadowym
> Sent: Tuesday, 13 June 2006 10:34
> To: asterisk-users@lists.digium.com
> Subject: [Asterisk-Users] Hard drive write cache
>
>
> I am looking at ways to harden my asterisk install to prevent computer
> related issues from happening. I am concerned about about disk write
> cache.
> That seems to be a major source of hard drive corruption on power
failure.> Hard Drive corruption is simply unacceptable for the 99.999% uptime
> requirements of my Asterisk install that needs to be as reliable as a
> proprietary PBX.
>
> Of course I will be using redundant power supplies, raid 1 and use a
UPS.> None of those things mean much if the power cords accidentally get
pulled> from the back of the server. Unlikely as it may be I have to consider
ALL> possibilities.
>
> So is disabling the write cache a good way to reduce the risk of hard
> drive
> corruption for an Asterisk server? I am not too concerned about the
> reduced
> performance/lifetime of hardrives with write cache disabled since
Asterisk> is not a very write intensive environment. Even with lot's of
voicemail> going on.
>
> Any other recommendations/links for increasing the reliability of
Asterisk> servers?
> _______________________________________________
> --Bandwidth and Colocation provided by Easynews.com --
>
> Asterisk-Users mailing list
> To UNSUBSCRIBE or update options visit:
> http://lists.digium.com/mailman/listinfo/asterisk-users
>[1] Ask any Norstar user how often Flash takes a big shit when it losespower. lol...snap! ....aaahhh the memories. Makes the Asterisk burn seem minor by comparision.
> The cold hard truth is that if Asterisk cannot achieve 99.999% uptime > without becoming much more expensive that a traditional PBX then it isnot> a > viable alternative. Even elcheapo Key systems are rated for fivenines.> That is what the telco world requires unless your just using Asteriskin> your basement as a hobby or as a one man company.Well, you can pretty much guarantee 100% software uptime with asterisk. The main causes of crashes of the working system are.... users. If it works... don't touch it, do not logon to it... forget about it. Create a minimalistic root system with busybox, have everything on CF on IDE adapter, user UPS with shutdown to protect the CF (as they're prone to failures on power loss) and you have yourself a VERY stable system. You can use JFFS2 on block device to reduce the wear on CF but you will not need it if you're not writing anything on CF (or have 2 CF's and md them together.) I have never seen PBX with guarantee of 99.999%. None of the manufacturers will commit to that unless it is a highly redundant system, but by then it's not elcheapo. About fanless PC's.. A stock standard intel fan would lust longer then you think unless its located in dusty and damp place. I have a p3 that's been running for 5 years non-stop and it was still going strong.. Half the capacitors started to leak on the motherboard but the fan was still spinning. :) Now that's reliability!> Redundant Servers is moving into the realm of non-competitive with > Traditional PBX IMHO. >More or less true. Any 100-200 extension highly redundant PBX system will costs you more or less the same money.
>99.999%I suspect you will see this drop as "traditional" PBX'es start to use commodity parts. My Mitel ICP 3300 has a Maxtor 10 gig hard drive in it (same as an Xbox!)
They have a 4 port switch, but not PoE. It's decommissioned but we haven't taken it out of the rack yet, if I'll remember when we derack it I'll snap a pic. AFAIC, there is nothing in them that cannot be duplicated in a decent Asterisk setup, and in fact the featureset that we have fleshed out in our Asterisk setup stands head and shoulders above the Mitel featureset with the only exception of voice recognition, which is actually quite good (and extremely expensive - the cost for the voice recognition as a single feature is equivalent to the cost of our ENTIRE Asterisk rollout, including phones.) -----Original Message----- From: shadowym [mailto:shadowym@hotmail.com] Sent: Wednesday, June 14, 2006 2:25 PM To: 'Asterisk Users Mailing List - Non-Commercial Discussion' Subject: RE: [Asterisk-Users] Hard drive write cache Hmmmmm.....that's interesting to know. I'll bet they boot from CF but I could be wrong. Any chance you can get some photo's of the inside of that thing? Those Mitels have a built in PoE switch do they not? Anything else special about them that cannot be duplicated in an Asterisk Server? Not that a couple 4 port PoE switches inside is not possible but I haven't heard of anyone doing that.> -----Original Message----- > From: Colin Anderson [mailto:ColinA@landmarkmasterbuilder.com] > Sent: Wednesday, June 14, 2006 7:19 AM > To: 'Asterisk Users Mailing List - Non-Commercial Discussion' > Subject: RE: [Asterisk-Users] Hard drive write cache > > >99.999% > > I suspect you will see this drop as "traditional" PBX'es > start to use commodity parts. My Mitel ICP 3300 has a Maxtor > 10 gig hard drive in it (same as an Xbox!) > >_______________________________________________ --Bandwidth and Colocation provided by Easynews.com -- Asterisk-Users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
How about having the asterisk config on CF or USB drives and the OS, Asterisk on a Linux LiveCD. That way you can mail out PBX upgrades to the customer they pop it into the CD drive and reboot. Config for DHCP boot, /etc/asterisk (or /etc entirely) on a USB dongle would work great. -- Matthew S. Crocker Vice President Crocker Communications, Inc. Internet Division PO BOX 710 Greenfield, MA 01302-0710 http://www.crocker.com
On Thu, Jun 15, 2006 at 08:28:57PM -0400, Matthew Crocker wrote:> > How about having the asterisk config on CF or USB drives and the OS, > Asterisk on a Linux LiveCD. That way you can mail out PBX upgrades > to the customer they pop it into the CD drive and reboot.Actually, the minute they pop out the CD drive, the system stops working. Unless the system runs completely from RAM.> Config > for DHCP boot, /etc/asterisk (or /etc entirely) on a USB dongle > would work great.It means that every small config change will have to require a reboot. And an attented one. No remote upgrades. Also note that separating code from config is not as trivial as it sounds. For instance, much of extensions.conf is actually code, and should belong under /usr/share on such a system (using #include). -- Tzafrir Cohen sip:tzafrir@local.xorcom.com icq#16849755 iax:tzafrir@local.xorcom.com +972-50-7952406 tzafrir.cohen@xorcom.com http://www.xorcom.com