CentOS - Jun 2005 - Opteron Mobo Suggestions

I've been planning to build a dual Opteron server for awhile. I'd like 
to get people's suggestions on a suitable motherboard.

I've looked at the Tyan K8SE (S2892) and K8SRE (S2891) but would like to 
find more Linux-specific experiences with these boards.

Some features I expect are at least 4 SATA (SATA-300?) ports, serial 
console support in the BIOS, USB 2.0 and IEEE-1394 ports, low-end 
on-board video.

Anyone care to share their experiences?

Kirk Bocek

From: Kirk Bocek <t004 at kbocek.com>
> I've been planning to build a dual Opteron server for awhile.
> I'd like to get people's suggestions on a suitable motherboard.
I assume you read 2004 November Sys Admin on the Opteron, including
avoiding cheap mainboard designs?
  http://www.samag.com/documents/sam0411b/  

Specifically figure 6 for a 2-way mainboard:  
  http://www.samag.com/documents/s=9408/sam0411b/0411b_f6.htm  
> I've looked at the Tyan K8SE (S2892) and K8SRE (S2891) but would
> like to find more Linux-specific experiences with these boards.
On 2-way, you want one with an AMD8131 (or AMD8132) HyperTransport
PCI-X 1.0 (2.0) tunnel.

On 4-way, you want one with at least 2 such tunnels.

I assume you want 2-way.  The S2891 is a nice, 2-way board for 1/2U.
The S2892 is nice if you want more of a traditional SSI EEB mainboard.
They are true NUMA and don't cut corners on the CPU and memory.

_But_, unfortunately _both_ mainboards put _all_ I/O on CPU #1.
That's not ideal.

As such, you might be better of going with the S2895 which puts the
nForce Pro 2050 and 1 GbE NIC on CPU #2, at least if you have a
PCIe storage or NIC cards.  It looks more like a workstation board, but
the fact that 1 embedded GbE NIC _and_ 1 PCIe slot is on CPU #2
helps improve processor affinity for I/O (especially under Solaris, but
even Linux to an extent as well).

Tyan continues to disappoint me by putting _all_ I/O on 1 CPU.
The only option where they don't is the nForce Pro 2200+2050
combination on the S2895 where the 2050 goes on CPU #2.
Tyan should put the AMD8131 on CPU #2 on the 2891/2892
so it doesn't contend with the nForce 2200 on CPU #1 and
its slots/peripherals (and there is no 2050).
> Some features I expect are at least 4 SATA (SATA-300?) ports,
???  Why "dumb" SATA channels  ???

Why not a 3Ware Escalade 8506-4+ or a LSI Logic MegaRAID 300-8X?
> serial console support in the BIOS,
Any PhoenixBIOS will typically give you that.
> USB 2.0 and IEEE-1394 ports,
On a server?  The last thing you want is any storage over those two.
I've been ripping out a lot of external storage solutions that use
USB 2.0 or IEEE-1394 because they put server stability at risk.
> low-end on-board video.
Most low-footprint mainboards have video on-mainboard these days.
Typically just glued to a PCI bus.
> Anyone care to share their experiences?
Just curious why you want "dumb" SATA channels, let alone external
storage over USB or FireWire on a _server_.



--
Bryan J. Smith   mailto:b.j.smith at ieee.org

From: "Bryan J. Smith <b.j.smith at ieee.org>" <thebs413 at
earthlink.net>
> Tyan should put the AMD8131 on CPU #2 on the 2891/2892
> so it doesn't contend with the nForce 2200 on CPU #1 and
> its slots/peripherals (and there is no 2050).
I should have pointed out this is illustrated in Figure 7:  
  http://www.samag.com/documents/s=9408/sam0411b/0411b_f7.htm  


--
Bryan J. Smith   mailto:b.j.smith at ieee.org

From: Kirk Bocek <t004 at kbocek.com>
> Yea, but who *is* this guy? :)
I heard he is a jerk.  ;-ppp
> Don't really need physical 4-way.
> I'm planning on 2-way dual core.
On a server where throughput is king, everything is I/O.
Dual-core just gives you more computing.
> I mostly just want to see how two or three striped SATA drives will 
> perform. If the on-board ports don't perform, I can always add a 
> separate controller later. My understanding is that NCQ support isn't 
> here yet for Linux but that it should provide a boost.
Do you understand what NCQ does?
It allows an intelligent drive to queue for the bus,
because the bus is dumb.

So what do you think an intelligent strorage controller does?
(light bulb!)

3Ware Escalade 7000/8000s are the undesputed king of queuing in ATA.
You can't beat real-time ASIC+SRAM queuing!  It can service requests
faster than anything.

That's why 3Ware calls it a "storage switch," because it's
like having
a "layer 3 switch" instead of a "layer 3 router"
(traditional
microcontroller+DRAM).  And it's _ideal_ for RAID-0, 1 and 10.
> Not for primary storage. Mostly just future proofing here.
Throw in a $10 card.  Besides, they all come with it these days.
> I've been meaning to setup a backup system using external hard
> drives instead of tape. That wouldn't fly over USB 1.1.
Put in an out-of-band GbE card or iSCSI HBA instead.  Trust me on
this, you do _not_ want USB or FireWire.
> So, Brian, having given me all that good information (thanks again), 
> what specific models do you suggest? What are *you* running on?
I like the HP DL385 (2-way) and DL585 (4-way) servers.  They use the
reference AMD designs -- including (2) AMD813x's in the DL585:  
  http://www.samag.com/documents/s=9408/sam0411b/0411b_f5.htm  

Do you need 4-way for computation?  Or throughput?
If the latter, then dual-core on 2-way is not going to give you anything.


--
Bryan J. Smith   mailto:b.j.smith at ieee.org

From: Kirk Bocek <t004 at kbocek.com>
> Yes, I understand. I also don't have an unlimited budget
> If I can get most of the benefit without needing a $300-800
> add-on, that's a big plus.
You're talking about an $500-600 mainboard.
Then you're talking about $500-800 _per_ dual-core CPU.
And then you're gotta factor in the Registered ECC RAM costs.

And you're worried about a $125 (3Ware Escalade 8006-2) or
$300 (3Ware Escalade 8506-4) controller?  Furthermore, the
SATA on the nForce MCP is typically only a single, 133MBps
PCIe x1 channel.  You'll get 533MBps out of the 8506-4
in a PCI slot at 64-bit @ 66MHz.
> I conceptually understand iSCSI, but how would I use a GbE
> NIC in a backup solution?
What do you think iSCSI is?  It's storage over IP, but just with
an additional a host processor in the GbE to off-load overhead.

If you don't have the money for a HBA GbE, then just do
straight GbE.  Build an out-of-band IP network segmented off
on a 2nd GbE connection in each system.

[ HINT:  I'm exploring the issues of real-time backup and
discuss near-line storage, virtual tape libraries and, in a
future article after that, out-of-band networks and "cheap"
SANs.  All from the standpoint of a sysadmin, and not so
much network/storage engineering. ]
> Don't forget that a backup solution would depend on 
> hot-pluggability. That's why I was thinking of USB/IEEE-
> 1394.
Mistake on a server.  You don't want to hot-plug
consumer interconnects.  One issue, bam!  Kernel panic.

Use an out-of-band IP network over a 2nd GbE connection
to what you need elsewhere on the network.  I have been
recommending this where cost is a factor.
> Cool, I will take a look at HP's offerings. Do you have any
> motherboards alone that you would recommend?
It's difficult because of the limited designs.  A lot of
mainboards "cut corners" for cost considerations.  Even
the Tyan 4885 puts _all_ I/O on _one_ CPU.  The HP
DL585 follows the AMD reference.



--
Bryan J. Smith   mailto:b.j.smith at ieee.org

From: Ugo Bellavance <ugob at camo-route.com>
> I don't really know, but you could get a pretty cheap NAS with a
gigabit
> nic to do your backups instead of using USB/Firewire.
And then attach any tape or other media library to that.

There is this farce out there among sysadmins that they have to do
backups to tape or other media in "real-time" during a time-limited
"window."  No, no, no, no, NO!

REALITY:
You only need to send the data in real-time during the window to
a "buffering" system that actually then commits it to tape or other
media at _any_ time it chooses.

BONUSES:
- I can send just a "differential" to that system, instead of a full
filesystem dump (even if I want a "full" backup ;-).
- I can restore from that "near-line" system whenever I want
(assuming it is still on-line)
- many others ...

This is what has now evolved into the concept of a "virtual tape
library."  A near-line storage device that buffers between end
systems that have limited backup windows and the tape (or
other media) you may or may not need to always commit to.

Be sure to get 2005 September Sys Admin.  ;->


--
Bryan J. Smith   mailto:b.j.smith at ieee.org

From: Peter Arremann <loony at loonybin.org>
> Bryan,
> there are sometimes more important things than max performance... 
> Upgradability i.e. ... If you buy a complete server its one thing, but if 
> you're offering parts and people build there own they often do consider
> upgradability. If you buy a server that right now should run fine with 1
CPU
> but you want to upgrade later to a second one, then the Tyan design is for 
> you. This way you can run all your I/O still and still not have to pay
extra
> for the second CPU up front. The Tyan website even mentions this somewhere
as
> the reason for the design decision.
And I understand this.
And I understand why Tyan might offer it on _some_ models.

But not _all_ models.  ;->
That's what I can't believe.

--
Bryan J. Smith   mailto:b.j.smith at ieee.org

From: Peter Arremann <loony at loonybin.org>
> Better believe it - cause that's what they are doing :-) 
Actually, I can believe it (I just wish that had *1* good option).
Tyan's focus is cost and economies-of-scale.
You'll find the same of SuperMicro, among a few others.

When you start looking at HP and Sun's Opteron designs,
then the mainboard cost is only a portion of the system cost.
So they are more focus on delivering the optimal solution.
Because doing so only affects the end price less than 10%,
whereas it might affect the mainboard cost 50%+.
> Tyan doesn't lay out that many different boards from scratch...
> just look at most of their stuff (S2892, S2885, S2882) - the
> CPU/memory area is identical, so all they did was take the old
> design, mod around with the chipset and PCI/PCI-X/PCI Express
> slots and they got a new one... 
Actually, other than EMF and the support microelectronics, it's
simply a matter of deciding how many traces you want, which
may result in more PCB layers.

Tyan is really focused on cost.
They have an older, "cheap" 2-way Opteron mainboard that
doesn't even put DDR channels on CPU #2.

And several manufacturers are even worse than Tyan.
Especially the non-server focused vendors.
SuperMicro and Tyan are better than most.

But still not the quality in mainboard design you'll get
from select Tier-1 OEMs like HP and Sun.


--
Bryan J. Smith   mailto:b.j.smith at ieee.org

From: Chris Mauritz <chrism at imntv.com>
> If you have a task that's I/O bound, then perhaps Bryan's concerns
will
> impact your decision on a motherboard. For my purposes, I'm mostly cpu 
> bound,
Then it doesn't apply.
> it really doesn't mean a hell of a lot to me that one cpu is 
> "stuck" with mundane I/O tasks.  And as cpus get faster and
faster,
> this becomes less of an issue (if it's really an issue at all for most 
> tasks).
That's not it at all.  It has nothing to do with CPU performance.
It has everything to do with interconnect and the processor affinity
with regards to I/O and memory mapped transfers.
> Also, as Bryan mentions, the S2895 splits the I/O up a bit 
> better and the newer revs of that board support dual core Opterons
> too.
Technically, all mainboards should support dual-core Opterons.
But the market reality is why give consumers a BIOS update when
you can charge for a new mainboard?  ;->
 
Dual-core A64/Opterons don't change the interconnect one bit.
Dual-core Xeons/P4s are a whole different story.
 
> So if you're in an I/O heavy environment you could choose that
> board rather than the S2882.
Assume you could make use of one of the PCIe x16 slots for a storage
or other communications device, as well as the 2nd NIC.


--
Bryan J. Smith   mailto:b.j.smith at ieee.org

I had trouble with Tyan boards and BIOS revisions not getting Processor 
numbers and speeds correct.
We also had early life failures with Asus boards, and since switched to 
MSI, we now use the MSI 9161

K8D Master 3 
http://www.msi.com.tw/program/products/server/svr/pro_svr_detail.php?UID=560

 without any problems..

P.


Kirk Bocek wrote:
> I've been planning to build a dual Opteron server for awhile. I'd
like
> to get people's suggestions on a suitable motherboard.
>
> I've looked at the Tyan K8SE (S2892) and K8SRE (S2891) but would like 
> to find more Linux-specific experiences with these boards.
>
> Some features I expect are at least 4 SATA (SATA-300?) ports, serial 
> console support in the BIOS, USB 2.0 and IEEE-1394 ports, low-end 
> on-board video.
>
> Anyone care to share their experiences?
>
> Kirk Bocek
>
>
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos

From: Peter Arremann <loony at loonybin.org>
> Actually, support for the 9500s has gotten quite a lot better over the past
> few months - you might want to give them another try. A college of mine 
> mis-ordered a few of them when they first came out and at the beginning the
> results were less than stallar. By now (FC4 and CentOS 4U1) the drivers are
> stable enough that the tests I ran could no longer crash the box.
That is good to here.  My actual experience had been limited, but positive.
But given the slew of issues, I just stopped recommending the 9500S until
I heard otherwise.



--
Bryan J. Smith   mailto:b.j.smith at ieee.org

From: Peter Farrow <peter at farrows.org>
> I had trouble with Tyan boards and BIOS revisions not getting Processor 
> numbers and speeds correct.
> We also had early life failures with Asus boards, and since switched to 
> MSI, we now use the MSI 9161
> K8D Master 3 
>
http://www.msi.com.tw/program/products/server/svr/pro_svr_detail.php?UID=560
> without any problems..
Ack, no block diagram in MSI's manual.

It's an AMD8131+AMD8111, no HyperTransport tunnel for any video channels.
So you can probably assume AMD8111 bridge attached to AMD8131 tunnel to
CPU #0.  Since there is only one high-speed tunnel, you really can't get
more
optimal than adding another HT tunnel chip.

I guess I'm just looking for the "holy grail" in (2) AMD813x's
on a 2-way board.


--
Bryan J. Smith   mailto:b.j.smith at ieee.org