CentOS - Jul 2005 - Fighting some install issues with a new box

Folks-

I have had a machine dropped in my lap that I am trying to get CentOS
4.1 to run on as a first pass (the hope is for it to eventually run
Rocks -- http://www.rocksclusters.org , which uses CentOS 4.x as its
underlying OS).

The machine has 2 Opteron 250DP (2.4GHz) with 4 GB of RAM. It is using a
Tyan Thunder K8WE (S2895A2NRF) motherboard with an nVidia nForce chipset
-- http://www.tyan.com/products/html/thunderk8we.html. It is configured
to boot from an SATA drive (80GB) and then use 3Ware 8506-8 card with
8x250GB SATA drives in a big raid 5 as a data store.

So, CentOS appears to recognize the nics and find the boot drive well
enough to go through a complete install. Now this is where things get
"interesting" .. the install process hung for a very long period of
time
during the "installing grub" phase (this was a cd install, so no post
or
anything, done off-line). It finally rebooted, but it just hangs at the
"GRUB Loading stage2..."

If I boot the machine using the 4.1 install cd in rescue mode, I can see
the filesystem (the OS is installed). /etc/grub.conf doesn't look
unreasonable. One oddball thing is that I cannot see the 3ware card at
all (doesn't show up in lspci output).

Any thoughts? Anyone set up a similar beastie?

Thanks.

Sean

Sean O'Connell <oconnell at soe.ucsd.edu>
wrote:
> The machine has 2 Opteron 250DP (2.4GHz) with 4 GB of RAM.
> It is using a Tyan Thunder K8WE (S2895A2NRF) motherboard\
> with an nVidia nForce chipset
> -- http://www.tyan.com/products/html/thunderk8we.html.
Yes, it is a seriously sweet board!
> It is configured to boot from an SATA drive (80GB) and then
> use 3Ware 8506-8 card with 8x250GB SATA drives in a big
raid
> 5 as a data store.
Okay, someone should be shot on that one.  You should be able
to use the 8506-8 "out-of-the-box" for boot, no "intermedia"
disk required.

Option 1:  *DEAD*SIMPLE*SOLUTION*

YANK OUT THE SATA DISK ON THE NVIDIA CHIPSET AND USE _ONLY_
THE 3WARE CARD!  NOW RE-INSTALL.

Option 2:  "Complex solution" (and why this happened)

I'm sure part of the problem is the load order of the nv_sata
and 3w-xxxx driver.  If the nv_sata loads first, which uses
the generic SCSI interafces, and it will be /dev/sda.  The
3w-xxxx volumes will /dev/sdb on-ward.  You need to tell
Linux the _exact_order_ of the host adapters that will map to
SCSI cards.  So ...

A.  Decide who gets to boot first -- nVidia chipset SATA or
3Ware Escalade SATA.  Then ...

B.  BIOS -- The S2895 uses Phoenix ServerBIOS and will let
you select the _exact_ card/[S]ATA channel that gets first
boot.  I.e., it will even "see" the 3Ware BIOS, and list that
under the Boot selection.  Make sure you decide which one,
and then set that in the BIOS.

C.  INITRD/GRUB -- You now need to setup /etc/modprobe.conf
in Linux to do the same.  E.g., if you select the nv_sata to
boot first, /etc/modprobe.conf should have:   
    alias scsi_hostadapter nv_sata
    alias scsi_hostadapter1 3w-xxxx

Now you'll need to remake the "initrd-*.img" for the kernel
with "mkinitrd" -- e.g.,
  mkinitrd /boot/initrd-`uname -r`-NVfirst.img `uname -r`

This will generate a new initrd file of
/boot/initrd-(kernel-rev)-NVfirst.img.  Now create another
GRUB entry to use it and boot it.  I.e., you can copy the
existing entry, just change the "initrd" line to the new
filename for the new entry.

D.  GRUB MAP/INIT -- Verify your GRUB map file
(/boot/grub/device.map) says the following ...
  (hd0)  /dev/sda
  (hd1)  /dev/sdb

[ hd2 if you have /dev/sdc, etc... ]

And then run: 
  grub-install /dev/sda

NOTE:  You may need to _reboot_ into the "Rescue Mode"
_after_ modifying /etc/modprobe.conf so it reads the order of
SCSI adapters.  And even then, I'm not sure it will work,
because the driver might assume the nv_sata is always first.

Which is why I recommend the "dead simple solution."
Install with _only_ the 3Ware Escalade attached drives, and
then the installer _should_ setup /etc/modprobe.conf with
"alias scsi_hostadapter 3w-xxxx" and the initrd will be
correct.

Also make sure you set the 3Ware to _always_ boot before the
on-board ATA/SATA in the Phoenix ServerBIOS.
> One oddball thing is that I cannot see the 3ware card at
> all (doesn't show up in lspci output).
Of course!  Because it's very likely the nVidia SATA is
loading _first_!  That's the problem.
> Any thoughts? Anyone set up a similar beastie?
All-the-time.  If I have a 3Ware card, there is *0* reason to
use the on-board SATA channels.  If you do, make sure you put
the 3Ware as the _first_ card in the /etc/modprobe.conf with
a "scsi_hostadapter" alias.

Otherwise any other SCSI card might be assumed to be first,
like the nVidia SATA with its nv_sata driver (which appears
as SCSI).

-- Bryan

P.S.  Remember, if you don't want to use RAID-5 for system
volumes, you _can_ configure your 3Ware card with _multiple_
volumes.  E.g.,
  (2) RAID-1 /dev/sda for "System"
  (6) RAID-5 /dev/sdb for "Data"

In fact, this is what I do normally.  Sometimes I'll do (4)
RAID-10 and (4) RAID-5, if I want some faster RAID-10 storage
for some data, using the RAID-5 volume for lesser and/or more
"read-only" access.


-- 
Bryan J. Smith                 mailto:b.j.smith at ieee.org
Sent from Yahoo Mail (please excuse any missing headers)