freebsd stable - Jun 2007 - Booting Supermicro 6015V-M3 on SAS drive with LSI 1068E

I have this machine and I'm just trying to get it to boot from an SAS 
drive.... that's all. I'm not trying to use the LSI 1068E for RAID.

See here if you're curious about specs:
http://www.supermicro.com/products/system/1U/6015/SYS-6015V-M3.cfm

Booting from the FreeBSD 6.2 install CD is not a problem; and the install 
kernel is, interestingly, able to see the drives attached to the LSI 1068E 
controller and use them. 

The problem is that the machine's BIOS can not seem to see the drives
plugged
into the LSI 1068E (raid controller), and thus those drives can not be 
selected in the BIOS for booting. "operating system not found". 

It seems the only way to get drives bootable from the BIOS that are plugged 
into the LSI is to put them into a raid. Once a logical disk is configured in 
the RAID bios one can see in the system bios "PCI SCSI: Software
Raid".
However, even if i did want to use this RAID device, FreeBSD doesn't 
see/support it (or, it doesn't support FreeBSD).

As an experiment I configured the LSI raid to contain one drive (RAID0) 
(stripe being the size of the drive). I then installed FreeBSD on this drive,
and, wonderfully enough, it worked. The BIOS was set to use the "PCI SCSI: 
Software raid", and the MBR was picked up from the single drive, and
FreeBSD
took over from there. I don't know how safe this is, but it seemed to still 
work after multiple reboots, and installing a bunch of software. Problems 
arose however when i installed a gmirror label onto the drive. Suddenly there 
was no more logical drive in the RAID bios on my next reboot; so i was back 
to "operating system not found": so it seems the LSI RAID also uses
the same
area as gmirror to store metadata. We want the gmirror on the SAS drives but 
not the LSI RAID... 

Other interesting ideas I had included trying to create a boot CD-ROM that 
would instruct the boot loader to do whatever it needs to do to load gmirror 
and then continue booting (using vfs.root.mountfrom?). Another idea was using 
a USB flash key for this. (Not really sure how to create a bootable CDRom to 
do this, but i'm sure it's possible. Any pointers here might be
helpful.)

Another idea is it seems we could possibly switch a couple of the cables 
currently plugged into the LSI ports to the ESB2 SATA ports (this board has 
both the LSI 1068E controller supporting SAS/SATA, and an ESB2 controler 
supporting SATA) and use gmirrored SATA drives for booting. This is just a 
bit of a  pain because the existing cables probably aren't long enough...
and
everything is very tightly packed. Also it means we have to use a couple of 
SATA drives. And it makes my boss nervous to start moving cables... (though 
perhaps not as nervous as using a USB flash to boot).

Anyhow, anyone played with this hardware? Any recommendations on how to handle 
this seemingly weird situation?

On Wed, Jun 06, 2007 at 09:20:55AM -0400, T. Middleton
wrote:
> It seems the only way to get drives bootable from the BIOS that are plugged
> into the LSI is to put them into a raid. Once a logical disk is configured
in
> the RAID bios one can see in the system bios "PCI SCSI: Software
Raid".
> However, even if i did want to use this RAID device, FreeBSD doesn't 
> see/support it (or, it doesn't support FreeBSD).
>
> As an experiment I configured the LSI raid to contain one drive (RAID0) 
> (stripe being the size of the drive). I then installed FreeBSD on this
drive,
> and, wonderfully enough, it worked. The BIOS was set to use the "PCI
SCSI:
> Software raid", and the MBR was picked up from the single drive, and
FreeBSD
> took over from there. I don't know how safe this is, but it seemed to
still
> work after multiple reboots, and installing a bunch of software.
I see this same behaviour with one of my Promise SATA RAID 0/1
controllers.  If I put a disk on the controller and *do not* define an
array (e.g. drives are in JBOD), FreeBSD can see the drive (as ad4) and
install onto it, but the Promise BIOS will not boot the disk.  The
system BIOS boot order is Floppy, CD-ROM, IDE, then SCSI (which is
what these controllers claim to be from a BIOS perspective).  The "Boot
Other" option is also set to Yes.

The Promise controller also states something about "not installing
interrupt handler", which I'm pretty sure is needed for the controller
to be used to boot the disk.  I forget which interrupt it is, but the
method is fairly normal (seen it on Adaptec controllers too).

Not until I go into the Promise BIOS and define a RAID array (I picked
RAID 0 with only 1 disk associated with it) will the controller boot the
disk.  FreeBSD then comes up and sees two available disks: ad4 (the
standalone disk in JBOD) and ar0 (the RAID 0 array).  I'm positive this
part is normal behaviour (for FreeBSD), but the fact I have to define an
array to get the disk bootable is just silly.

Then again, I suppose most people buying RAID controllers are doing so
to use the actual benefits of RAID, so I'll likely be branded as someone
who's crying over spilled milk.  If I had two disks in the system and
was using RAID 1, safe to say the instant I defined the RAID 1 array,
the system would boot and FreeBSD would see both disks and ar0.

-- 
| Jeremy Chadwick                                    jdc at parodius.com |
| Parodius Networking                           http://www.parodius.com/ |
| UNIX Systems Administrator                      Mountain View, CA, USA |
| Making life hard for others since 1977.                  PGP: 4BD6C0CB |