CentOS - Nov 2005 - boot with more scsi card

hi,
we've got a server with a 8 port 3ware card and 2 ide system disks. now
we'd like to replace the ide disks with scsi disks or sata disks (these
also recognized as scsi in the kernel). but we can't boot from it. the
problem are twofold. first in the normal case the first scsi host scsi0
id the 3ware card, but grub only see the first 8 disk so if the system
disk are sdi and sdj the one can't boot from it. on the other hand if
one disk is removed from the 3ware card or the kernel reorder the scsi
disk and sdi become sdh which is also a problem both for the bios and
for the kernel. if i change the order in bios, the kernel still use as
3ware scsi0. if i use scsihosts kernel param or load the scsi driver
(through modprobe.conf) in a reverse order then the kernel hang forever
during the initialization of the 3ware driver.
any tip?
thanks in advance.
yours.

centos 4.2
kernel-2.6.9-22.0.1.EL
3w_xxxx 1.26.00.039

-- 
  Levente                               "Si vis pacem para bellum!"

On Thu, 2005-11-24 at 14:51 +0100, Farkas Levente wrote:
> hi,
> we've got a server with a 8 port 3ware card and 2 ide system disks.
Are you saying you have 2 IDE disks on the 3Ware?
Or they are connected to the mainboard's ATA channels?
> now we'd like to replace the ide disks with scsi disks or sata disks 
> (these also recognized as scsi in the kernel).
On which device?  Are you saying you want to remove the IDE disks from
the mainboard's ATA channels, and use the 3Ware volumes to boot?

Or other?
> but we can't boot from it.
If its the case where you want to remove the IDE disks from the
mainboard's ATA channels and let the 3Ware boot, then your problem is
the BIOS Fixed Disk mapping to Linux device.

1) That is handled by the file "/boot/grub/device.map".

E.g., your current /boot/grub/device.map probably has something like:  
  (fd0)     /dev/fd0
  (hd0)     /dev/hda
  (hd1)     /dev/hdc
  (hd2)     /dev/sda

BIOS Disk 80h is /dev/hda
BIOS Disk 81h is /dev/hdc
BIOS Disk 82h is /dev/sda

If you remove /dev/hda and /dev/hdc -- *OR* tell your BIOS to boot off-
chipset channels/cards first (like the 3Ware card) -- then the BIOS Disk
80h will become /dev/sda.  You have to tell GRUB this in its mapping or
it won't boot.

So you need to change your file "/boot/grub/device.map" to:  
  (fd0)     /dev/fd0
  (hd0)     /dev/sda
And, optionally if you still leave the IDE disks attached to the ATA
channels, but just changed the boot order in the BIOS:
  (hd1)     /dev/hda
  (hd2)     /dev/hdc

2) You'll _also_ need to re-install the GRUB loader into /dev/sda
with:  
  grub-install /dev/sda

The easiest way to do that is to use a Rescue disk (e.g., CD #1 of your
CentOS release).

3)  Lastly, you'll need to modify your main /boot/grub/grub.conf entries
to reflect that root is in /dev/sda now, instead of /dev/hda -- as well
as in /etc/fstab (if you're not using Ext2/3 filesystem labels).
> the problem are twofold. first in the normal case the first scsi host
> scsi0 id the 3ware card, but grub only see the first 8 disk so if the
> system disk are sdi and sdj the one can't boot from it. on the other
> hand if one disk is removed from the 3ware card or the kernel reorder
> the scsi disk and sdi become sdh which is also a problem both for the
> bios and for the kernel.
If you're going to use the 3Ware card, why not put the 2 disks on port 0
and 1 -- instead of port 6 and 7?

But even then, the BIOS _should_ enumerate the first 2 disks as BIOS
disk 80h and 81h _if_ no other drives are in the system.  In that case,
it's just a matter of editing the /boot/grub/device.map with:  
  (fd0)     /dev/fd0
  (hd0)     /dev/sdi
  (hd1)     /dev/sdj
And running:
  grub-install /dev/sdi
And well as /boot/grub/grub.conf and /etc/fstab changes as appropriate.

Or do you already have the first port 0-5 populated?  If so, then let's
assume they are a 6 disk RAID-5 volume of /dev/sda, and then you have
the port 6 and 7 disks as /dev/sdi and /dev/sdj:  
  (fd0)     /dev/fd0
  (hd0)     /dev/sda
  (hd1)     /dev/sdi
  (hd2)     /dev/sdj
Running:
  grub-install /dev/sda
While putting in hd(1,x) and /dev/sdi in /boot/grub/grub.conf because
it's booting from /dev/sdi (even though GRUB is on /dev/sda for the
MBR).
> if i change the order in bios, the kernel still use as
> 3ware scsi0. if i use scsihosts kernel param or load the scsi driver
> (through modprobe.conf) in a reverse order then the kernel hang forever
> during the initialization of the 3ware driver.
It hangs?  Or you get a kernel panic?

You should get a kernel panic because the BIOS disk doesn't map to the
proper Linux /dev/.

Again, I need you to _clarify_ and give _exact_info_ of ...

1.  What is your _exact_ configuration of your mainboard ATA channels
(channel, disk, etc...) -- current as well as what you want to change?

2.  What is your _exact_ configuration of your 3Ware card (ports, disk,
etc...) -- again, current as well as what you want to change?

3.  Where is your / and/or /boot filesystems, on what disk?

4.  What you have setup to boot first in your BIOS -- current as well as
what you want to change?
>From that, I will give you an _exact_ /boot/grub/device.map file, and
tell you how to configure /boot/grub/grub.conf.  It also helps to have a
Rescue Disk (or CentOS CD #1) around.
> any tip?
You have to get:
A)  The BIOS order
B)  The /boot/grub/device.map and /boot/grub/grub.conf
C)  The GRUB MBR/bootstrap

_Exactly_ mapping correctly.  That means if you change just 1, you have
to change _all_ 3.  Otherwise, it won't work.
> thanks in advance.
> yours.
> 
> centos 4.2
> kernel-2.6.9-22.0.1.EL
> 3w_xxxx 1.26.00.039
-- 
Bryan J. Smith   b.j.smith at ieee.org   http://thebs413.blogspot.com
-------------------------------------------------------------------
For everything else *COUGH*commercials*COUGH* there's
"ManningCard"