I had my eye on the Tyan dual-Opteron mobos for awhile. I tried to find
a posting *anywhere* sharing experiences with these boards under Linux.
No such luck. So placing myself under the heading "Where Angles Fear to
Tread," I went ahead and built a system anyway. Here's what I've
learned.
The specs:
Tyan Thunder K8SE S2892, BIOS 1.01
2x Opteron 270, 2Ghz Dual-Core, retail package with fan
2GB RAM (4x SuperTalent 512MB P/N D32RB12P3)
3Ware 9500S-4LP 4 port SATA controller
4x 200GB Segate ST3200826AS 7200.8 SATA (attached to 3Ware controller)
1x 80GB Wester Digital WD800JD-00LSA0 SATA (attached to onboard controller)
Antec Truepower 2.0 550W PSU
Chenming 901A-0-0 RTL Case
Supermicro 5-drive SATA HD Cage
CentOS 4.1 x86_64
I thought I bought a plenty large case. Turns out the CPU heat sink at
the front of the motherboard sticks up far enough to interfere with the
drive bays at the bottom of the case. I had carefully mounted the CPUs,
heat sinks and memory before mounting the motherboard. When I tried to
insert the motherboard, the front heat sink hit the shelf that supports
one of the two internal hard drive cages that come with the Chenming
case. I had to peel off the heat sink to get the board in then reinstall
it -- something I didn't want to do. If the case was 9" wide instead of
8", this probably wouldn't be an issue.
The heat sink sticks up far enough to keep me from reinstalling the
internal hard drive cages. Not a problem since I'm using the Supermicro
SATA cage, but it bugs me to lose expansion possibilities. Can anyone
recommend a "low-rise" Opteron heat sink?
I hadn't yet purchased an optical drive and so was planning on
booting the install program from a USB key. However I could not get the
BIOS to recognize the key. I put 'removable drives' at the top of the
boot preferences but it refused to recognize the key. I ended up doing a
PXE boot install instead.
The install hung once forcing me to restart. But after that the install
process was the fastest I've ever seen, taking about seven minutes to
copy everything.
Booting the first time, I was happy to see that everything was
recognized by the system, all three network ports, the Nvidia SATA
controller and even the 3Ware controller.
However, the boot messages showed:
warning: many lost ticks.
Your time source seems to be instable or some driver is hogging
interrupts
and several repeating messages:
powernow-k8: error - out of sync...
Also, /proc/cpuinfo showed each cpu as running at about 1004Mhz with
about 900 bogomips. Not what I expected.
I decided to press ahead and added a couple of drives the the 3Ware,
created a unit, formatted it and began benchmarking the array. The
"powernow" messages continued to occur and then the 3Ware driver
started
issuing error messages. Eventually the system crashed and stopped
responding.
Googling on the powernow error message let me know that the kernel
wizards had started to fix some powernow bugs starting with about kernel
2.6.10. Of course the stock CentOS 4.1 kernel is 2.6.9-11.
So I downloaded, compiled and installed kernel 2.6.12.3. On rebooting,
the error messages went away and the 3Ware worked without complaint. Now
/proc/cpuinfo showed each cpu running at 2009.267 Mhz with 4014.08
bogomips. Ah, much better! In addition, the system came up in with NUMA
enabled. Looks like the Red Hat kernel had it turned off by default.
I benchmarked disk array performance using Bonnie++ version 1.03a. I ran
six benchmarks using 50GB of data, three using 16k blocks and three
using 64k blocks. During the raid 0 testing I had two instances of
Setiathome running. During the raid 5 testing I had three instances of
Setiathome running. Here are the results:
Raid 0, 64k Stripes:
bonnie++ 1.03a ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec
%CP
Tyan 50G:16k 49317 99 138307 62 +++++ +++ 46765 80 183180 22 88.9
0
Tyan 50G:16k 48911 98 148421 67 77099 21 46018 79 182136 21 89.8
0
Tyan 50G:16k 49188 98 143615 65 77381 21 46158 78 181181 21 90.5
0
Tyan 50G:64k 49372 99 146417 67 77185 21 45828 78 179758 21 76.7
0
Tyan 50G:64k 48585 98 145376 66 76580 21 45609 78 171888 21 76.3
0
Tyan 50G:64k 46093 92 134903 58 67851 18 45200 77 172103 21 75.6
0
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
Tyan 16 2524 97 +++++ +++ +++++ +++ 2542 97 +++++ +++ 8670 99
Tyan 16 2385 98 +++++ +++ +++++ +++ 2579 97 +++++ +++ 8408 98
Tyan 16 2627 97 +++++ +++ +++++ +++ 2437 92 +++++ +++ 8671 100
Tyan 16 1582 98 +++++ +++ +++++ +++ 1647 98 +++++ +++ 8488 100
Tyan 16 1399 85 +++++ +++ +++++ +++ 1632 98 +++++ +++ 8476 99
Tyan 16 1654 95 +++++ +++ +++++ +++ 1636 98 +++++ +++ 8405 99
Writes at about 140MB/Sec, reads at about 180MB/Sec. Very nice.
Raid 5, 64k Stripes:
bonnie++ 1.03a ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
Tyan 50G:16k 33394 68 35889 16 26513 7 44096 75 157248 19 84.1 0
Tyan 50G:16k 32601 67 36058 16 26623 7 43964 75 156970 19 83.3 0
Tyan 50G:16k 37892 77 35702 16 26960 7 44390 75 157084 19 83.5 0
Tyan 50G:64k 36168 74 35560 17 27003 8 43463 75 155179 20 69.1 0
Tyan 50G:64k 32713 68 36187 16 26580 7 43922 76 156752 20 69.2 0
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
Tyan 16 1561 98 +++++ +++ +++++ +++ 1694 97 +++++ +++ 8428 97
Tyan 16 1672 98 +++++ +++ +++++ +++ 1712 97 +++++ +++ 8751 99
Tyan 16 1542 98 +++++ +++ +++++ +++ 1355 81 +++++ +++ 8986 100
Tyan 16 2534 96 +++++ +++ +++++ +++ 2408 96 +++++ +++ 7945 98
Tyan 16 2382 97 +++++ +++ +++++ +++ 2380 97 +++++ +++ 7989 99
Whoa! Writes drop to about 35MB/Sec under raid 5, about 25% of the raid 0
performance. In fact, it was taking so long that I aborted the last test. Just
to be
sure that setiathome wasn't interfering, I killed seti and reran a smaller
test.
Writes increased to about 50MB/Sec without seti running. I was expecting some
hit for
raid 5 but not this much. Guess I'll stick with the hot rsync backups to
other hosts
scheme that I'm using now.
As a final test, I restarted the 50GB benchmark under raid5 and popped out one
of the
drives. There was a pause while I guess the 3Ware controller decided that the
drive
actually went off-line. The writing then continued to the remaining three disks.
Checking the controller showed that both the drive and the unit were
'DEGRADED.'
After popping the disk back in, I had to remove and re-add the drive to the
array
configuration then start the rebuild process. At this point the drive showed
'DEGRADED' and the unit showed 'REBUILDING.' The 3Ware drive
logged appropriate
messages when the disk was removed and replaced. I didn't bother finishing
the
rebuild, but it looked like it would take a couple of hours to finish if I let
the
benchmark continue to run.
The 3Ware controller is pretty cool. As I said, the driver (3w-9xxx) is
included in the 2.6 kernel. 3Ware provides a simple CLI utility for
management. They also have a GUI tool but I didn't bother running it.
You can create, remove and verify 'units' on a running system. The
associated /dev entries are dynamically added and removed as you make
changes. Very nice.
In summary, the Tyan S2892 plus the 3Ware controller runs well under CentOS 4.1
although you will have to update to a more current kernel than the one provided
by
Red Hat. Watch your clearance at the front of the motherboard when selecting a
case.
Comments and corrections are encouraged.
Kirk Bocek
On Thu, 2005-07-28 at 13:30 -0700, Kirk Bocek wrote:> Tyan Thunder K8SE S2892, BIOS 1.01 > ... > Antec Truepower 2.0 550W PSUIs that the 550EPS12V? Or just the regular 550? The former has an 8-pin (4x2) SSI server connector. The latter has only the 4-pin (2x2) P4 connector. Don't confuse either with the 6-pin (3x2) SSI Workstation (WS) connector, now known as a PCIe connector. And no, that's not the 6-pin (6x1) "AUX" connector. [ Confused yet? ;-]> Chenming 901A-0-0 RTL CaseIsn't that an ATX case, and _not_ an EATX/SSI EEB case?> I thought I bought a plenty large case.There are (not including newer BTX, or older formats): FlexATX: 9.0" x 7.5" MicroATX: 9.6" x 9.6" ATX: 12.0" x 9.6" Ext ATX: 12.0" x 13.0" SSI EEB: 12.0" x 13.0" + additional height/spacing/thermal Most of the time, an EATX case will fit an SSI EEB mainboard, but sometimes you don't have the headroom. Although some ATX cases may physically fit an EATX, they often don't have the stand-offs required, and definitely not the height required.> Turns out the CPU heat sink at > the front of the motherboard sticks up far enough to interfere with the > drive bays at the bottom of the case. I had carefully mounted the CPUs, > heat sinks and memory before mounting the motherboard. When I tried to > insert the motherboard, the front heat sink hit the shelf that supports > one of the two internal hard drive cages that come with the Chenming > case. I had to peel off the heat sink to get the board in then reinstall > it -- something I didn't want to do. If the case was 9" wide instead of > 8", this probably wouldn't be an issue.Welcome to SSI EEB. ;-> "Drive overhang" is typical of most ATX cases, and will quickly interfere with EATX and especially SSI EEB mainboards.> The heat sink sticks up far enough to keep me from reinstalling the > internal hard drive cages. Not a problem since I'm using the Supermicro > SATA cage, but it bugs me to lose expansion possibilities. Can anyone > recommend a "low-rise" Opteron heat sink?The problem is that you need to use a case designed for SSI EEB mainboards. You have a case designed for ATX, possibly EATX. You can tell if its EATX if you're last 2-3" of your mainboard has standoffs/screws. If it's "overhanging" without them, then you only have an ATX case.> Booting the first time, I was happy to see that everything was > recognized by the system, all three network ports, the Nvidia SATA > controller and even the 3Ware controller. > However, the boot messages showed: > warning: many lost ticks. > Your time source seems to be instable or some driver is hogging > interrupts > and several repeating messages: > > powernow-k8: error - out of sync... > Also, /proc/cpuinfo showed each cpu as running at about 1004Mhz with > about 900 bogomips. Not what I expected.Turn off PowerNOW in the BIOS. It's what's slowing down your CPUs. Everything I've read says do _not_ use it on a server.> So I downloaded, compiled and installed kernel 2.6.12.3. On rebooting, > the error messages went away and the 3Ware worked without complaint. Now > /proc/cpuinfo showed each cpu running at 2009.267 Mhz with 4014.08 > bogomips. Ah, much better! In addition, the system came up in with NUMA > enabled. Looks like the Red Hat kernel had it turned off by default. > I benchmarked disk array performance using Bonnie++ version 1.03a. I ran > six benchmarks using 50GB of data, three using 16k blocks and three > using 64k blocks. During the raid 0 testing I had two instances of > Setiathome running. During the raid 5 testing I had three instances of > Setiathome running. Here are the results: > > Raid 0, 64k Stripes: > > bonnie++ 1.03a ------Sequential Output------ --Sequential Input- --Random- > -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- > Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP > Tyan 50G:16k 49317 99 138307 62 +++++ +++ 46765 80 183180 22 88.9 0 > Tyan 50G:16k 48911 98 148421 67 77099 21 46018 79 182136 21 89.8 0 > Tyan 50G:16k 49188 98 143615 65 77381 21 46158 78 181181 21 90.5 0 > Tyan 50G:64k 49372 99 146417 67 77185 21 45828 78 179758 21 76.7 0 > Tyan 50G:64k 48585 98 145376 66 76580 21 45609 78 171888 21 76.3 0 > Tyan 50G:64k 46093 92 134903 58 67851 18 45200 77 172103 21 75.6 0 > ------Sequential Create------ --------Random Create-------- > -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- > files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP > Tyan 16 2524 97 +++++ +++ +++++ +++ 2542 97 +++++ +++ 8670 99 > Tyan 16 2385 98 +++++ +++ +++++ +++ 2579 97 +++++ +++ 8408 98 > Tyan 16 2627 97 +++++ +++ +++++ +++ 2437 92 +++++ +++ 8671 100 > Tyan 16 1582 98 +++++ +++ +++++ +++ 1647 98 +++++ +++ 8488 100 > Tyan 16 1399 85 +++++ +++ +++++ +++ 1632 98 +++++ +++ 8476 99 > Tyan 16 1654 95 +++++ +++ +++++ +++ 1636 98 +++++ +++ 8405 99 > > Writes at about 140MB/Sec, reads at about 180MB/Sec. Very nice. > > Raid 5, 64k Stripes: > > bonnie++ 1.03a ------Sequential Output------ --Sequential Input- --Random- > -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- > Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP > Tyan 50G:16k 33394 68 35889 16 26513 7 44096 75 157248 19 84.1 0 > Tyan 50G:16k 32601 67 36058 16 26623 7 43964 75 156970 19 83.3 0 > Tyan 50G:16k 37892 77 35702 16 26960 7 44390 75 157084 19 83.5 0 > Tyan 50G:64k 36168 74 35560 17 27003 8 43463 75 155179 20 69.1 0 > Tyan 50G:64k 32713 68 36187 16 26580 7 43922 76 156752 20 69.2 0 > ------Sequential Create------ --------Random Create-------- > -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- > files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP > Tyan 16 1561 98 +++++ +++ +++++ +++ 1694 97 +++++ +++ 8428 97 > Tyan 16 1672 98 +++++ +++ +++++ +++ 1712 97 +++++ +++ 8751 99 > Tyan 16 1542 98 +++++ +++ +++++ +++ 1355 81 +++++ +++ 8986 100 > Tyan 16 2534 96 +++++ +++ +++++ +++ 2408 96 +++++ +++ 7945 98 > Tyan 16 2382 97 +++++ +++ +++++ +++ 2380 97 +++++ +++ 7989 99 > > Whoa! Writes drop to about 35MB/Sec under raid 5, about 25% of the raid 0 > performance. In fact, it was taking so long that I aborted the last test. Just to be > sure that setiathome wasn't interfering, I killed seti and reran a smaller test. > Writes increased to about 50MB/Sec without seti running. I was expecting some hit for > raid 5 but not this much. Guess I'll stick with the hot rsync backups to other hosts > scheme that I'm using now.Or consider RAID-10. Send some benchmarks of that. ;->> The 3Ware controller is pretty cool. As I said, the driver (3w-9xxx) is > included in the 2.6 kernel. 3Ware provides a simple CLI utility for > management. They also have a GUI tool but I didn't bother running it. > You can create, remove and verify 'units' on a running system. The > associated /dev entries are dynamically added and removed as you make > changes. Very nice.Make sure you have the latest 3Ware driver and firmware for the 9500S series. Also consider the "tweaks" on 3Ware's site.> In summary, the Tyan S2892 plus the 3Ware controller runs well under CentOS 4.1 > although you will have to update to a more current kernel than the one provided by > Red Hat. Watch your clearance at the front of the motherboard when selecting a case.Or just get an SSI EEB case in the first place for an SSI EEB mainboard. ;-> -- Bryan J. Smith b.j.smith at ieee.org --------------------------------------------------------------------- It is mathematically impossible for someone who makes more than you to be anything but richer than you. Any tax rate that penalizes them will also penalize you similarly (to those below you, and then below them). Linear algebra, let alone differential calculus or even ele- mentary concepts of limits, is mutually exclusive with US journalism. So forget even attempting to explain how tax cuts work. ;->
Kirk Bocek wrote:>I had my eye on the Tyan dual-Opteron mobos for awhile. I tried to find >a posting *anywhere* sharing experiences with these boards under Linux. >No such luck. So placing myself under the heading "Where Angles Fear to >Tread," I went ahead and built a system anyway. Here's what I've learned. > >The specs: >Tyan Thunder K8SE S2892, BIOS 1.01 >2x Opteron 270, 2Ghz Dual-Core, retail package with fan >2GB RAM (4x SuperTalent 512MB P/N D32RB12P3) >3Ware 9500S-4LP 4 port SATA controller >4x 200GB Segate ST3200826AS 7200.8 SATA (attached to 3Ware controller) >1x 80GB Wester Digital WD800JD-00LSA0 SATA (attached to onboard controller) >Antec Truepower 2.0 550W PSU >Chenming 901A-0-0 RTL Case >Supermicro 5-drive SATA HD Cage >CentOS 4.1 x86_64 > >I thought I bought a plenty large case. Turns out the CPU heat sink at >the front of the motherboard sticks up far enough to interfere with the >drive bays at the bottom of the case. I had carefully mounted the CPUs, >heat sinks and memory before mounting the motherboard. When I tried to >insert the motherboard, the front heat sink hit the shelf that supports >one of the two internal hard drive cages that come with the Chenming >case. I had to peel off the heat sink to get the board in then reinstall >it -- something I didn't want to do. If the case was 9" wide instead of >8", this probably wouldn't be an issue. > >The heat sink sticks up far enough to keep me from reinstalling the >internal hard drive cages. Not a problem since I'm using the Supermicro >SATA cage, but it bugs me to lose expansion possibilities. Can anyone >recommend a "low-rise" Opteron heat sink? > >Monarch lists 3 1U all-copper-no-fan heat sinks that I bet would work. Please note the speculative nature of that recommendation !!!! I have been aiming for an Opteron based system for a while now, that's why I joined this list :-). <snip> -- William A. Mahaffey III --------------------------------------------------------------------- Remember, ignorance is bliss, but willful ignorance is LIBERALISM !!!!
Kay Diederichs
2005-Jul-29 07:59 UTC
[CentOS] Re: Tyan Thunder K8SE S2892 Report / here: Tyan S4882
Kirk Bocek wrote: ...> However, the boot messages showed: > > warning: many lost ticks. > Your time source seems to be instable or some driver is hogging > interrupts > > and several repeating messages: > > powernow-k8: error - out of sync... > > Also, /proc/cpuinfo showed each cpu as running at about 1004Mhz with > about 900 bogomips. Not what I expected. > > I decided to press ahead and added a couple of drives the the 3Ware, > created a unit, formatted it and began benchmarking the array. The > "powernow" messages continued to occur and then the 3Ware driver started > issuing error messages. Eventually the system crashed and stopped > responding. > > Googling on the powernow error message let me know that the kernel > wizards had started to fix some powernow bugs starting with about kernel > 2.6.10. Of course the stock CentOS 4.1 kernel is 2.6.9-11. > > So I downloaded, compiled and installed kernel 2.6.12.3. On rebooting, > the error messages went away and the 3Ware worked without complaint. Now > /proc/cpuinfo showed each cpu running at 2009.267 Mhz with 4014.08 > bogomips. Ah, much better! In addition, the system came up in with NUMA > enabled. Looks like the Red Hat kernel had it turned off by default....> Comments and corrections are encouraged. > > Kirk BocekThis is what I get with the standard (2.6.9-11ELsmp) kernel on a quad-opteron (2.6GHz) Tyan S4882 - actually the Tyan TX46 ( http://www.tyan.com/products/html/tx46b4882.html ): powernow-k8: Found 4 AMD Athlon 64 / Opteron processors (version 1.00.09b) powernow-k8: 0 : fid 0x12 (2600 MHz), vid 0x6 (1400 mV) powernow-k8: 1 : fid 0x10 (2400 MHz), vid 0x8 (1350 mV) powernow-k8: 2 : fid 0xe (2200 MHz), vid 0xa (1300 mV) powernow-k8: 3 : fid 0xc (2000 MHz), vid 0xc (1250 mV) powernow-k8: 4 : fid 0xa (1800 MHz), vid 0xe (1200 mV) powernow-k8: 5 : fid 0x2 (1000 MHz), vid 0x12 (1100 mV) powernow-k8: cpu_init done, current fid 0x12, vid 0x6 powernow-k8: 0 : fid 0x12 (2600 MHz), vid 0x6 (1400 mV) powernow-k8: 1 : fid 0x10 (2400 MHz), vid 0x8 (1350 mV) powernow-k8: 2 : fid 0xe (2200 MHz), vid 0xa (1300 mV) powernow-k8: 3 : fid 0xc (2000 MHz), vid 0xc (1250 mV) powernow-k8: 4 : fid 0xa (1800 MHz), vid 0xe (1200 mV) powernow-k8: 5 : fid 0x2 (1000 MHz), vid 0x12 (1100 mV) powernow-k8: cpu_init done, current fid 0x12, vid 0x6 powernow-k8: 0 : fid 0x12 (2600 MHz), vid 0x6 (1400 mV) powernow-k8: 1 : fid 0x10 (2400 MHz), vid 0x8 (1350 mV) powernow-k8: 2 : fid 0xe (2200 MHz), vid 0xa (1300 mV) powernow-k8: 3 : fid 0xc (2000 MHz), vid 0xc (1250 mV) powernow-k8: 4 : fid 0xa (1800 MHz), vid 0xe (1200 mV) powernow-k8: 5 : fid 0x2 (1000 MHz), vid 0x12 (1100 mV) powernow-k8: cpu_init done, current fid 0x12, vid 0x6 powernow-k8: 0 : fid 0x12 (2600 MHz), vid 0x6 (1400 mV) powernow-k8: 1 : fid 0x10 (2400 MHz), vid 0x8 (1350 mV) powernow-k8: 2 : fid 0xe (2200 MHz), vid 0xa (1300 mV) powernow-k8: 3 : fid 0xc (2000 MHz), vid 0xc (1250 mV) powernow-k8: 4 : fid 0xa (1800 MHz), vid 0xe (1200 mV) powernow-k8: 5 : fid 0x2 (1000 MHz), vid 0x12 (1100 mV) powernow-k8: cpu_init done, current fid 0x12, vid 0x6 ACPI: (supports S0 S1 S4 S5) which clearly shows that the kernel detects the powernow features nicely. Actually powernow adjusts the CPU speed dynamically according to the requirements - an idle server runs all CPUs at 1000MHz; whereas at higher loads the speed goes up to 2600MHz automagically (I grepped MHz in /proc/cpuinfo). All of this is no different when using the 2.6.12 kernel. I have not found a single problem with the system. One question of my own: I am pretty much confused by the multitude of BIOS options which influence the handling of the memory: memory hole remapping, SRAT table and such. Is there anybody who can say definitely what works best on the standard RH kernel? I tried a number of possibilities but did not find any differences in lmbench benchmarks. The existence of "NUMA" is only reported by the 2.6.12 kernel, not by the RH one. Currently running with memory remapping "SOFTWARE", SRAT table "DISABLED" and everything else at "AUTO". Kay