Xen users - Feb 2012 - entry-level XCP server using Intel desktop components

xen-users:

I''m considering building an entry-level Xen Cloud Platform server using
Intel desktop components.


Should I choose a processor and motherboard that support Intel VT-d 
technology?  For example:

http://ark.intel.com/search/advanced/?s=t&VTD=true&FamilyText=2nd%20Generation%20Intel%C2%AE%20Core%E2%84%A2%20i5%20Processors&Sockets=1155

http://www.intel.com/support/motherboards/desktop/sb/CS-030922.htm


Specifically, the i5-2500S processor and DQ67SW motherboard?

http://ark.intel.com/products/52211/Intel-Core-i5-2500S-Processor-%286M-Cache-2_70-GHz%29

http://www.intel.com/content/www/us/en/motherboards/desktop-motherboards/desktop-board-dq67sw.html


TIA,

David

David,

I would recommend you use as many VT features as you can. You also need to
consider whether you need more than one socket, for your first server board. A
2-socket board will allow you to gain experience with NUMA/Quickpath. Either
way, you should get a board that has a serial port (or a serial header on the
board), for both debugging and to learn how to administer XCP via a serial port.

We use the Intel S1200BTL as a good single-socket server board. It has many VT
features. The multi-socket boards we have used also use registered memory, so I
can''t recommend them as a simple multi-socket board.

Sincerely,

John
----


On Feb 10, 2012, at 1:25 AM, David Christensen wrote:
> xen-users:
> 
> I''m considering building an entry-level Xen Cloud Platform server
using Intel desktop components.
> 
> 
> Should I choose a processor and motherboard that support Intel VT-d
technology?  For example:
> 
>
http://ark.intel.com/search/advanced/?s=t&VTD=true&FamilyText=2nd%20Generation%20Intel%C2%AE%20Core%E2%84%A2%20i5%20Processors&Sockets=1155
> 
> http://www.intel.com/support/motherboards/desktop/sb/CS-030922.htm
> 
> 
> Specifically, the i5-2500S processor and DQ67SW motherboard?
> 
>
http://ark.intel.com/products/52211/Intel-Core-i5-2500S-Processor-%286M-Cache-2_70-GHz%29
> 
>
http://www.intel.com/content/www/us/en/motherboards/desktop-motherboards/desktop-board-dq67sw.html
> 
> 
> TIA,
> 
> David
> 
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
----
What is the formal meaning of the one-line program
#include "/dev/tty"

J.P. McDermott			building 12
Code 5542			john.mcdermott@nrl.navy.mil
Naval Research Laboratory	voice: +1 202.404.8301
Washington, DC 20375, US	fax:   +1 202.404.7942

Why on earth would you need to administer XCP via a serial port?!?


On Fri, Feb 10, 2012 at 7:36 AM, John McDermott CIV
<john.mcdermott@nrl.navy.mil> wrote:
> David,
>
> I would recommend you use as many VT features as you can. You also need to
consider whether you need more than one socket, for your first server board. A
2-socket board will allow you to gain experience with NUMA/Quickpath. Either
way, you should get a board that has a serial port (or a serial header on the
board), for both debugging and to learn how to administer XCP via a serial port.
>
> We use the Intel S1200BTL as a good single-socket server board. It has many
VT features. The multi-socket boards we have used also use registered memory, so
I can''t recommend them as a simple multi-socket board.
>
> Sincerely,
>
> John
> ----
>
>
> On Feb 10, 2012, at 1:25 AM, David Christensen wrote:
>
>> xen-users:
>>
>> I''m considering building an entry-level Xen Cloud Platform
server using Intel desktop components.
>>
>>
>> Should I choose a processor and motherboard that support Intel VT-d
technology?  For example:
>>
>>
http://ark.intel.com/search/advanced/?s=t&VTD=true&FamilyText=2nd%20Generation%20Intel%C2%AE%20Core%E2%84%A2%20i5%20Processors&Sockets=1155
>>
>> http://www.intel.com/support/motherboards/desktop/sb/CS-030922.htm
>>
>>
>> Specifically, the i5-2500S processor and DQ67SW motherboard?
>>
>>
http://ark.intel.com/products/52211/Intel-Core-i5-2500S-Processor-%286M-Cache-2_70-GHz%29
>>
>>
http://www.intel.com/content/www/us/en/motherboards/desktop-motherboards/desktop-board-dq67sw.html
>>
>>
>> TIA,
>>
>> David
>>
>> _______________________________________________
>> Xen-users mailing list
>> Xen-users@lists.xensource.com
>> http://lists.xensource.com/xen-users
>
> ----
> What is the formal meaning of the one-line program
> #include "/dev/tty"
>
> J.P. McDermott                  building 12
> Code 5542                       john.mcdermott@nrl.navy.mil
> Naval Research Laboratory       voice: +1 202.404.8301
> Washington, DC 20375, US        fax:   +1 202.404.7942
>
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users

On Fri, Feb 10, 2012 at 9:17 AM, Scott Damron <sdamron@gmail.com>
wrote:
> Why on earth would you need to administer XCP via a serial port?!?
It''s nice to have truly out of band access to systems. This way you
can make networking configuration changes with confidence, boot into
different kernels, or collect a server''s last gasp of air if it
crashes. Given that serial terminal servers are affordable even on the
OP''s budget, it seems a no brainer:
http://www.fragmentationneeded.net/2011/11/dirt-cheap-terminal-servers.html

Unfortunately, most motherboards can''t do *full* serial redirection,
where the POST details, memory check, BIOS settings, etc... are all
available over the COM port, so we''re often stuck with serial
management of the bootloader and beyond only. The boards that can do
full serial redirection tend to be expensive "server" boards.

For the OP, I''d keep an eye open for Intel boards that feature AMT:
http://en.wikipedia.org/wiki/Intel_Active_Management_Technology
This is a neat setup that with a VNC server independent of the OS. It
allows full KVM access to the system even when it hasn''t booted yet,
and also allows booting of the system off of remote media (an CD
image, for example). AMT is generally available on Intel''s
"desktop"
grade motherboards.

I believe that AMT used to allow the management traffic to live only
in a particular VLAN, but that feature has been removed. This may
introduce a security risk to your Dom0, so you might consider
installing additional NICs for use by the OSes and leave the onboard
NIC dedicated to the management feature. ...And plug it into a trusted
"management" subnet.

An inexpensive alternative to IP KVM is to snag an old RILOE board
from ebay. These are basically video cards with an network interface
for remote display. They require an external cable to plug them into
the server''s keyboard port.

On Thu, 2012-02-09 at 22:25 -0800, David Christensen
wrote:
> xen-users:
> 
> I''m considering building an entry-level Xen Cloud Platform server
using
> Intel desktop components.
> 
> 
> Should I choose a processor and motherboard that support Intel VT-d 
> technology?  For example:
> 
>
http://ark.intel.com/search/advanced/?s=t&VTD=true&FamilyText=2nd%20Generation%20Intel%C2%AE%20Core%E2%84%A2%20i5%20Processors&Sockets=1155
> 
> http://www.intel.com/support/motherboards/desktop/sb/CS-030922.htm
> 
> 
> Specifically, the i5-2500S processor and DQ67SW motherboard?
> 
>
http://ark.intel.com/products/52211/Intel-Core-i5-2500S-Processor-%286M-Cache-2_70-GHz%29
> 
>
http://www.intel.com/content/www/us/en/motherboards/desktop-motherboards/desktop-board-dq67sw.html
> 
> 
> TIA,
> 
> David
> 
> 
David,
I''m in the midst of implementing an entry-level XCP system for virtual
desktop infrastructure for my organization.  I ended up purchasing a
Dell PowerEdge C5000 chassis with 8 x C5220 "sleds".  This is the
Dell''s
new "clould" chassis, code-named Viking, I believe.  Each of the 8
sleds
can support a single Intel Xeon E3 processor - I purchased the 1270
model, which runs at 3.4ish GHz.  Each sled will also take up to 32 GB
of RAM.  They have two on-board GigE connections, and can support either
4 x 2.5" HDs or 2 x 3.5" HDs, and an on-board SDHC card slot.  The
C5000
chassis is 3 rack units.  Very nice chassis.  Unfortunately when you
purchase the chassis you have to purchase a minimum of 4 sleds, so you
really can''t get it for less than $10K or so.  But, maybe it gives you
some help as to what hardware I''m using.  I was going for something
more
dense and less expensive than the blades and/or 1U systems that we
usually purchase so that I can run relatively inexpensive virtual
desktops in my datacenter.  For me, this was just the ticket.

-Nick



--------
This e-mail may contain confidential and privileged material for the sole use of
the intended recipient.  If this email is not intended for you, or you are not
responsible for the delivery of this message to the intended recipient, please
note that this message may contain SEAKR Engineering (SEAKR)
Privileged/Proprietary Information.  In such a case, you are strictly prohibited
from downloading, photocopying, distributing or otherwise using this message,
its contents or attachments in any way.  If you have received this message in
error, please notify us immediately by replying to this e-mail and delete the
message from your mailbox.  Information contained in this message that does not
relate to the business of SEAKR is neither endorsed by nor attributable to
SEAKR.

On Fri, Feb 10, 2012 at 08:17:22AM -0600, Scott Damron
wrote:
> Why on earth would you need to administer XCP via a serial port?!?
Logging, mostly.   What caused that crash last night?  Was it a hardware 
problem?  a software problem?   With most KVM over IP setups, you have 
no idea.  With some (like the KVM over IP capabilities of some DRAC 
cards)  you can get a little video of what the monitor says right before 
a crash, but aside from the fact you can''t cut and paste, a lot of
times
the monitor is on some management GUI and doesn''t get the backtrace.
(also, the ''video before a crash'' feature is usually only
available on
the higher end models, and like I said, I''ve never seen it work
particularly well.)

At scale, serial ports have the additional advantage of being grepable.   
One place I worked we had thousands of servers logging to a central place.
I could quickly use grep and similar tools to find servers with bad ram,
bad hard drives, and similar problems.  

If you setup serial properly and linux crashes, the backtrace goes
to the serial port.  Setting up text logging of a serial console is
trivial (I use conserver for this, but there are a thousand solutions 
that will work)  

If you only have a few servers, you can setup a ''buddy system''
so that
server A watches console for server B and server B watches the console of 
server A for nothing more than the cost of a usb-> serial dongle and a 
null modem adaptor.  (I used the onboard db9 as the serial console for that
server and the usb-> serial to log the console of the other server in the 
pair.)

Once you have more than a few servers, you can get nice, new opengear brand
48 port console servers for under $1500.  You can go on ebay and get used
48 port cyclades ts3000 console servers for around $200 each. 

Personally, I think it''s extremely irresponsable to put any server in 
production without a serial console.   Without a serial console, those
irritating ''it reboots once a week''  errors keep coming back
until you
decide to chuck the hardware.  With a serial console, you have a fighting
chance of figuring out the problem after the first crash.

In testing, it''s less important, but it''s still pretty nice
for those times
when you are trying to figure out of the instability is because you
cheaped out on test hardware (and we''ve all done that)  or because
there
is a genuine problem with the product.

xen-users:

Thank you everyone for the replies and discussion.  :-)


On 02/10/2012 05:36 AM, John McDermott CIV wrote:
 > I would recommend you use as many VT features as you can.

As I understanding it, VT-d is Intel''s current top-shelf virtualization
technology.  All three of the Intel DQ67* motherboards have it.


 > ... get a board that has a serial port (or a serial header on the 
board), for both debugging and to learn how to administer XCP via a 
serial port.

All three DQ67*''s have 1 serial header.


David

On 02/10/2012 07:21 AM, Chris Marget wrote:
> For the OP, I''d keep an eye open for Intel boards that feature
AMT:
> http://en.wikipedia.org/wiki/Intel_Active_Management_Technology
> This is a neat setup that with a VNC server independent of the OS. It
> allows full KVM access to the system even when it hasn''t booted
yet,
> and also allows booting of the system off of remote media (an CD
> image, for example). AMT is generally available on Intel''s
"desktop"
> grade motherboards.
All three of the Intel DQ67* boards have AMT 7.0.

> I believe that AMT used to allow the management traffic to live only
> in a particular VLAN, but that feature has been removed. This may
> introduce a security risk to your Dom0, so you might consider
> installing additional NICs for use by the OSes and leave the onboard
> NIC dedicated to the management feature. ...And plug it into a trusted
> "management" subnet.
I''ll keep that in mind if/when I deploy an Internet-facing XCP server.


Thanks!

David