Sounds weird but now 2 drives will not work. I have an 80GB drive that has sxcr 50 on a 20gb slice (/) 160GB drive that I drop zfs on the whole drive. Blah blah blah, doing stuff making dir like zones/home users and what not. Reboot, system wont go past the check for IDE devices. you know IDE Primary Master DVD IDE Primary Slave None IDE Secondary Master None IDE Secondary Slave None IDE Thrid Master : 80GB SATA IDE Fourth Master : 160GB SATA Nope can''t get into BIOS. Unplug Drive, right into BIOS, system boots fine minus 160GB/zfs drive. I''m thinking ok, maybe the drive went bad (crap) Stick in a brand new 250GB SATA drive, System boots up, I re-install sxcr50 blah blah blah create 250GB zfs make some home/zones/brandz installed cento38 and did some playing in there. Reboot to make a change..........DAMMIT! Is ZFS writing something to the drive that would make a BIOS wonder what the hell its doing? Thanks ;) This message posted from opensolaris.org
Justin Conover wrote:> Sounds weird but now 2 drives will not work....> Is ZFS writing something to the drive that would make a BIOS wonder what the hell its doing?Yes, an EFI label. If instead you give zfs a slice (it can contain the whole disk) that should be sufficient to work around this issue. We saw this on pre-May 2006 versions of the bios in the Ultra 20. cheers, James C. McPherson -- Solaris kernel software engineer Sun Microsystems
Is there anything I can do to get my drives back? If there plugged in, I can''t boot the box, can''t get into the BIOS or boot from CD/DVD. I''ll try putting them in another box tonight when I get home. Thanks, This message posted from opensolaris.org
On 11/20/06, Justin Conover <justin.conover at gmail.com> wrote:> Is there anything I can do to get my drives back? If there plugged > in, I can''t boot the box, can''t get into the BIOS or boot from CD/DVD. > I''ll try putting them in another box tonight when I get home.An unsupported and dubious but useful workaround is to have your case lid off and the disks not connected. Power-on the system, and once the ATA identification routine has completed, quickly connect the disks. This is easy on an ultra20 because it has the Sun "spud bracket" for which allows for hotplug ops. Once you''re past the ATA identification routine the EFI bios *shouldn''t* matter to the system and you should be able to get operational. No guarantees provided, no warranties given ... your mileage may vary..... don''t come crying to me if it doesn''t work! I''m just passing on what worked for me with an ultra20 :) good luck, James -- Solaris kernel software engineer, system admin and troubleshooter http://www.jmcp.homeunix.com/blog Find me on LinkedIn @ http://www.linkedin.com/in/jamescmcpherson
>Is there anything I can do to get my drives back? If there plugged in, I can''t boot the box, can''t get into the BIOS or boot from CD/DVD. I''m surprised you can''t get into the BIOS; what are the drives connected too? You may need to disable any onboard RAID which can see the disks and you may want to disable the ability to boot from the disks. (I had a similar issue which was solved by making sure the disk in question no longer appeared in the boot sequence so it was not probed at POST) Casper
On Mon, Nov 20, 2006 at 01:41:12PM +0100, Casper.Dik at Sun.COM wrote:> > >Is there anything I can do to get my drives back? If there plugged in, I can''t boot the box, can'' > t get into the BIOS or boot from CD/DVD. > > > I''m surprised you can''t get into the BIOS; what are the drives connected > too? > > You may need to disable any onboard RAID which can see the disks and > you may want to disable the ability to boot from the disks. > > (I had a similar issue which was solved by making sure the disk in > question no longer appeared in the boot sequence so it was not probed > at POST)I''m also running into this situation; AWARD BIOS hangs forever at the splash screen after creating a ZFS mirror across two entire SATA disks on snv_50. Unplugging the SATA drives corrects the BIOS hang problem. Being unfamilar with EFI label adoption trends in the PC world, is it common for BIOS''es not to recognize EFI labels? Have others found that upgrading BIOS images seems to fix this problem?
>Being unfamilar with EFI label adoption trends in the PC world, is >it common for BIOS''es not to recognize EFI labels? Have others found >that upgrading BIOS images seems to fix this problem?We have found quite a few, including our own, with this issue. It''s not "not recognizing"; it''s hanging/crashing when finding a label they don''t recognize. I.e., no appropriate error checking. Casper
Well, I was able to pull the drives and put them in a different box and it booted. Linux box so I formated the EFI with ext3 and put them in the other box and now it boots. Thanks, This message posted from opensolaris.org
I had the same thing happen to me twice on my x86 box. I installed ZFS (RaidZ) on my enclosure with four drives and upon reboot the bios hangs upon detection of the newly EFI''d drives. I''ve already RMA''d 4 drives to seagate and the new batch was frozen as well. I was suspecting my enclosure, but I was suspicious when it only went bye bye after installing ZFS. This is a problem since how can anyone use ZFS on a PC??? My motherboard is a newly minted AM2 w/ all the latest firmware. I disabled boot detection on the sata channels and it still refuses to boot. I had to purchase an external SATA enclosure to fix the drives. This seems to me to be a serious problem. I put build 47 and 50 on there with the same issue. This message posted from opensolaris.org
On 29-Nov-06, at 9:30 AM, David Elefante wrote:> I had the same thing happen to me twice on my x86 box. I installed > ZFS (RaidZ) on my enclosure with four drives and upon reboot the > bios hangs upon detection of the newly EFI''d drives. ... This > seems to me to be a serious problem.Indeed. Yay for PC BIOS. --Toby
On 11/30/06, David Elefante <elefante72 at hotmail.com> wrote:> I had the same thing happen to me twice on my x86 box. I > installed ZFS (RaidZ) on my enclosure with four drives and > upon reboot the bios hangs upon detection of the newly EFI''d > drives. I''ve already RMA''d 4 drives to seagate and the new > batch was frozen as well. I was suspecting my enclosure, > but I was suspicious when it only went bye bye after installing ZFS. > > This is a problem since how can anyone use ZFS on a PC??? > My motherboard is a newly minted AM2 w/ all the latest > firmware. I disabled boot detection on the sata channels and > it still refuses to boot. I had to purchase an external SATA > enclosure to fix the drives. This seems to me to be a serious > problem. I put build 47 and 50 on there with the same issue.Yes, this is a serious problem. It''s a problem with your motherboard bios, which is clearly not up to date. The Sun Ultra-20 bios was updated with a fix for this issue back in May. Until you have updated your bios, you will need to destroy the EFI labels, write SMI labels to the disks, and create slices on those disks which are the size that you want to devote to ZFS. Then you can specify the slice name when you run your zpool create operation. This has been covered in the ZFS discussion lists several times, and a quick google search should have found the answer for you. James C. McPherson -- Solaris kernel software engineer, system admin and troubleshooter http://www.jmcp.homeunix.com/blog Find me on LinkedIn @ http://www.linkedin.com/in/jamescmcpherson
>This is a problem since how can anyone use ZFS on a PC??? My motherboard is a newly minted AM2 w/all the latest firmware. I disabled boot detection on the sata channels and it still refuses to b oot. I had to purchase an external SATA enclosure to fix the drives. This seems to me to be a ser ious problem. I put build 47 and 50 on there with the same issue. A serious problem *IN YOUR BIOS*. You will need to format the disks, at ordinary PC (fdisk) labels and on those create Solaris partitions and give those to ZFS. Casper
I had this happen on three different motherboards. So it seems that there should be a procedure in the documentation that states if your BIOS doesn''t support EFI labels than you need to write ZFS to a partition (slice) not the overlay, causing the BIOS to hang on reading the drive on boot up. Most PC bios do not support EFI at this point, so this can impact the larger community. Having that documentation would have saved me 30 hours at least, and I only hope that you take this as positive feedback and integrate it into the doc set. I have ZFS working on my Ultra 20 just fine, and that is what confused me when I was working with my x86 box. It says that EFI is not supported on IDE disks (SATA drive), but I''m assuming that this has changed.>From the sol9 doc set:Restrictions of the EFI Disk Label Keep the following restrictions in mind when determining whether to use disks greater than 1 terabyte is appropriate for your environment: * The SCSI driver, ssd, currently only supports up to 2 terabytes. If you need greater disk capacity than 2 terabytes, use a volume management product like Solaris Volume Manager to create a larger device. * Layered software products intended for systems with EFI-labeled disks might be incapable of accessing a disk with an EFI disk label. * A disk with an EFI disk label is not recognized on systems running previous Solaris releases. * The EFI disk label is not supported on IDE disks. * You cannot boot from a disk with an EFI disk label. * You cannot use the Solaris Management Console''s Disk Manager Tool to manage disks with EFI labels. Use the format utility or the Solaris Management Console''s Enhanced Storage Tool to manage disks with EFI labels, after you use the format utility to partition the disk. * The EFI specification prohibits overlapping slices. The whole disk is represented by cxtydz. This message posted from opensolaris.org
On Nov 29, 2006, at 10:41, Casper.Dik at Sun.COM wrote:>> This is a problem since how can anyone use ZFS on a PC??? My >> motherboard is a newly minted AM2 w/ all the latest firmware. I >> disabled boot detection on the sata channels and it still refuses >> to boot. I had to purchase an external SATA enclosure to fix the >> drives. This seems to me to be a serious problem. I put build 47 >> and 50 on there with the same issue. > > A serious problem *IN YOUR BIOS*. > > You will need to format the disks, at ordinary PC (fdisk) labels and > on those create Solaris partitions and give those to ZFS.take a look at the EFI/GPT discussion here (apple): http://www.roughlydrafted.com/RD/Home/7CC25766-EF64-4D85-AD37- BCC39FBD2A4F.html I suspect a lack of an MBR could cause some BIOS implementations to barf .. does our fdisk put an MBR on the disk? if so, does the EFI vdev labeling invalidate the MBR? Jonathan
>I suspect a lack of an MBR could cause some BIOS implementations to >barf ..Why? Zeroed disks don''t have that issue either. What appears to be happening is more that raid controllers attempt to interpret the data in the EFI label as the proprietary "hardware raid" labels. At least, it seems to be a problem with internal RAIDs only. In my experience, removing the disks from the boot sequence was not enough; you need to disable the disks in the BIOS. The SCSI disks with EFI labels in the same system caused no issues at all; but the disks connected to the on-board RAID did have issues. So what you need to do is: - remove the controllers from the probe sequence - disable the disks Casper
David Elefante wrote:> I had this happen on three different motherboards. So it seems that there should be a procedure in the documentation that states if your BIOS doesn''t support EFI labels than you need to write ZFS to a partition (slice) not the overlay, causing the BIOS to hang on reading the drive on boot up. Most PC bios do not support EFI at this point, so this can impact the larger community. > > Having that documentation would have saved me 30 hours at least, and I only hope that you take this as positive feedback and integrate it into the doc set. I have ZFS working on my Ultra 20 just fine, and that is what confused me when I was working with my x86 box. It says that EFI is not supported on IDE disks (SATA drive), but I''m assuming that this has changed. > >>From the sol9 doc set:You shouldn''t be using the Solaris *9* doc set. Use the Solaris 10 docs, specifically: Solaris 10 System Administrator Collection >> System Administration Guide: Devices and File Systems >> 11. Managing Disks (Overview) >> What''s New in Disk Management in the Solaris 10 Release? http://docs.sun.com/app/docs/doc/817-5093 and Solaris 10 System Administrator Collection >> Solaris ZFS Administration Guide >> 4. Managing ZFS Storage Pools >> Components of a ZFS Storage Pool http://docs.sun.com/app/docs/doc/819-5461 Also, please pester your mobo vendor to get with the times... :-) -- richard
On Nov 29, 2006, at 13:24, Casper.Dik at Sun.COM wrote:> >> I suspect a lack of an MBR could cause some BIOS implementations to >> barf .. > > Why? > > Zeroed disks don''t have that issue either. >you''re right - I was thinking that a lack of an MBR with a GPT could be causing problems, but actually it looks like we do write a protective MBR in efi_write() - so it''s either going to be the GPT header at LBA1 or backwards compatibility with the version 1.00 spec that the BIOS vendors aren''t dealing with correctly. Proprietary BIOS RAID signatures does sound quite plausible as a common cause for problems. Digging a little deeper, I''m thinking some of our EFI code might be a little old .. in efi_partition.h we''ve got the following defined for dk_gpt and dk_part: 161 /* Solaris library abstraction for EFI partitons */ 162 typedef struct dk_part { 163 diskaddr_t p_start; /* starting LBA */ 164 diskaddr_t p_size; /* size in blocks */ 165 struct uuid p_guid; /* partion type GUID */ 166 ushort_t p_tag; /* converted to part''n type GUID */ 167 ushort_t p_flag; /* attributes */ 168 char p_name[EFI_PART_NAME_LEN]; /* partition name */ 169 struct uuid p_uguid; /* unique partition GUID */ 170 uint_t p_resv[8]; /* future use - set to zero */ 171 } dk_part_t; 172 173 /* Solaris library abstraction for an EFI GPT */ 174 #define EFI_VERSION102 0x00010002 175 #define EFI_VERSION100 0x00010000 176 #define EFI_VERSION_CURRENT EFI_VERSION100 177 typedef struct dk_gpt { 178 uint_t efi_version; /* set to EFI_VERSION_CURRENT */ 179 uint_t efi_nparts; /* number of partitions below */ 180 uint_t efi_part_size; /* size of each partition entry */ 181 /* efi_part_size is unused */ 182 uint_t efi_lbasize; /* size of block in bytes */ 183 diskaddr_t efi_last_lba; /* last block on the disk */ 184 diskaddr_t efi_first_u_lba; /* first block after labels */ 185 diskaddr_t efi_last_u_lba; /* last block before backup labels */ 186 struct uuid efi_disk_uguid; /* unique disk GUID */ 187 uint_t efi_flags; 188 uint_t efi_reserved[15]; /* future use - set to zero */ 189 struct dk_part efi_parts[1]; /* array of partitions */ 190 } dk_gpt_t; which looks lke we''re using the EFI Version 1.00 spec and looking at cmd/zpool/zpool_vdev.c we call efi_write() which does the label and writes the PMBR at LBA0 (first 512B block), the EFI header at LBA1 and should reserve the next 16KB for other partition tables .. [now we really should be using EFI version 1.10 with the -001 addendum (which is what 1.02 morphed into about 5 years back) or version 2.0 in the UEFI space .. but that''s a separate discussion, as the address boundaries haven''t really changed for device labels.] in uts/common/fs/zfs/vdev_label.c we define the zfs boot block 500 501 /* 502 * Initialize boot block header. 503 */ 504 vb = zio_buf_alloc(sizeof (vdev_boot_header_t)); 505 bzero(vb, sizeof (vdev_boot_header_t)); 506 vb->vb_magic = VDEV_BOOT_MAGIC; 507 vb->vb_version = VDEV_BOOT_VERSION; 508 vb->vb_offset = VDEV_BOOT_OFFSET; 509 vb->vb_size = VDEV_BOOT_SIZE; which gets written down at the 8KB boundary after we start usable space from LBA34: 857 vtoc->efi_parts[0].p_start = vtoc->efi_first_u_lba; [note: 17KB isn''t typically well aligned for most logical volumes .. it would probably be better to start writing data at LBA1024 so we stay well aligned for logical volumes with stripe widths up to 512KB and avoid the R/M/W misalignment that can occur there .. currently with a 256KB vdev label, I believe we start the data portion out on LBA546 which seems like a problem] and then we apparently store a backup vtoc right before the backup partition table entries and backup GPT: 858 vtoc->efi_parts[0].p_size = vtoc->efi_last_u_lba + 1 - 859 vtoc->efi_first_u_lba - resv; this next bit is interesting since we should probably define a GUID for ZFS partitions that points to the ZFS vdev label instead of using V_USR 860 861 /* 862 * Why we use V_USR: V_BACKUP confuses users, and is considered 863 * disposable by some EFI utilities (since EFI doesn''t have a backup 864 * slice). V_UNASSIGNED is supposed to be used only for zero size 865 * partitions, and efi_write() will fail if we use it. V_ROOT, V_BOOT, 866 * etc. were all pretty specific. V_USR is as close to reality as we 867 * can get, in the absence of V_OTHER. 868 */ 869 vtoc->efi_parts[0].p_tag = V_USR; 870 (void) strcpy(vtoc->efi_parts[0].p_name, "zfs"); and here we define the backup vdev label on the last usable LBA before our standard 8MB(?) reservation (16384 blocks) at the end of the disk and do the efi_write(): 871 872 vtoc->efi_parts[8].p_start = vtoc->efi_last_u_lba + 1 - resv; 873 vtoc->efi_parts[8].p_size = resv; 874 vtoc->efi_parts[8].p_tag = V_RESERVED; 875 876 if (efi_write(fd, vtoc) != 0) I''m thinking we should really define a GUID for ZFS and maybe do some better provisioning at the front end of the disk to be better aligned for full stripe write conditions .. with EFI we could use from LBA34 to LBA1023 for vdev labels and other stuff to start write aligning out on LBA1024. There also looks like a error(?) in the EFI reservation bits at the tail end of the disk since I thought the EFI spec only needed 16KB for the backup partitions and 512B for the GPT header .. not 16384 * 512B blocks .. for what it''s worth that''s also been in the format utility for a while now, so I could be missing something on the methodology for the 8MB reservation at the tail end of the disk. .je
Dave which BIOS manufacturers and revisions? that seems to be more of the problem as choices are typically limited across vendors .. and I take it you''re running 6/06 u2 Jonathan On Nov 30, 2006, at 12:46, David Elefante wrote:> Just as background: > > I attempted this process on the following: > > 1. Jetway amd socket 734 (vintage 2005) > 2. Asus amd socket 939 (vintage 2005) > 3. Gigabyte amd socket am2 (vintage 2006) > > All with the same problem. I disabled the onboard nvidia nforce > 410/430 > raid bios in the bios in all cases. Now whether it actually does > not look > for a signature, I do not know. I''m attempting to make this box > into an > iSCSI target for my ESX environments. I can put W3K and SanMelody > on there, > but it is not as interesting and I am attempting to help the Solaris > community. > > I am simply making the business case that over three major vendors > boards > and the absolute latest (gigabyte), the effect was the same. > > As a workaround I can make slice 0 1 cyl and slice 1 1-x, and the > zpool on > the rest of the disk and be fine with that. So on a PC with zpool > create > there should be a warning for pc users that most likely if they use > the > entire disk, the resultant EFI label is likely to cause lack of > bootability. > > I attempted to hotplug the sata drives after booting, and Nevada 51 > came up > with scratch space errors and did not recognize the drive. In any > case I''m > not hotplugging my drives every time. > > The given fact is that PC vendors are not readily adopting EFI bios > at this > time, the millions of PC''s out there are vulnerable to this. And > if x86 > Solaris is to be really viable, this community needs to be > addressed. Now I > was at Sun 1/4 of my entire life and I know the politics, but the > PC area is > different. If you tell the customer to go to the mobo vendor to > fix the > bios, they will have to find some guy in a bunker in Taiwan. Not > likely. > Now I''m at VMware actively working on consolidating companies into x86 > platforms. The simple fact that the holy war between AMD and Intel > has > created processors that a cheap enough and fast enough to cause > disruption > in the enterprise space. My new dual core AMD processor is > incredibly fast > and the entire box cost me $500 to assemble. > > The latest Solaris 10 documentation (thx Richard) has use the > entire disk > all over it. I don''t see any warning in here about EFI labels, in > fact > these statements discourage putting ZFS in a slice.: > > > ZFS applies an EFI label when you create a storage pool with whole > disks. > Disks can be labeled with a traditional Solaris VTOC label when you > create a > storage pool with a disk slice. > > Slices should only be used under the following conditions: > > * > > The device name is nonstandard. > * > > A single disk is shared between ZFS and another file system, > such as > UFS. > * > > A disk is used as a swap or a dump device. > > Disks can be specified by using either the full path, such as > /dev/dsk/c1t0d0, or a shorthand name that consists of the device > name within > the /dev/dsk directory, such as c1t0d0. For example, the following > are valid > disk names: > > * > > c1t0d0 > * > > /dev/dsk/c1t0d0 > * > > c0t0d6s2 > * > > /dev/foo/disk > > ZFS works best when given whole physical disks. Although constructing > logical devices using a volume manager, such as Solaris Volume Manager > (SVM), Veritas Volume Manager (VxVM), or a hardware volume manager > (LUNs or > hardware RAID) is possible, these configurations are not > recommended. While > ZFS functions properly on such devices, less-than-optimal > performance might > be the result. > > -----Original Message----- > From: casper at holland.sun.com [mailto:casper at holland.sun.com] On > Behalf Of > Casper.Dik at Sun.COM > Sent: Wednesday, November 29, 2006 1:24 PM > To: Jonathan Edwards > Cc: David Elefante; zfs-discuss at opensolaris.org > Subject: Re: [zfs-discuss] Re: system wont boot after zfs > > >> I suspect a lack of an MBR could cause some BIOS implementations to >> barf .. > > Why? > > Zeroed disks don''t have that issue either. > > What appears to be happening is more that raid controllers attempt > to interpret the data in the EFI label as the proprietary > "hardware raid" labels. At least, it seems to be a problem > with internal RAIDs only. > > In my experience, removing the disks from the boot sequence was > not enough; you need to disable the disks in the BIOS. > > The SCSI disks with EFI labels in the same system caused no > issues at all; but the disks connected to the on-board RAID > did have issues. > > So what you need to do is: > > - remove the controllers from the probe sequence > - disable the disks > > Casper > > -- > No virus found in this incoming message. > Checked by AVG Free Edition. > Version: 7.5.430 / Virus Database: 268.15.2/560 - Release Date: > 11/30/2006 > 3:41 PM > > > -- > No virus found in this outgoing message. > Checked by AVG Free Edition. > Version: 7.5.430 / Virus Database: 268.15.2/560 - Release Date: > 11/30/2006 > 3:41 PM > >
>All with the same problem. I disabled the onboard nvidia nforce 410/430 >raid bios in the bios in all cases. Now whether it actually does not look >for a signature, I do not know. I''m attempting to make this box into an >iSCSI target for my ESX environments. I can put W3K and SanMelody on there, >but it is not as interesting and I am attempting to help the Solaris >community.>I am simply making the business case that over three major vendors boards >and the absolute latest (gigabyte), the effect was the same.Have you tried disabling the disks in the BIOS? Select the disk in the BIOS and set the type to "NONE". That will prevent the disks from being accessed by the BIOS. They will still be usable in Solaris.>The given fact is that PC vendors are not readily adopting EFI bios at this >time, the millions of PC''s out there are vulnerable to this. And if x86 >Solaris is to be really viable, this community needs to be addressed. Now I >was at Sun 1/4 of my entire life and I know the politics, but the PC area is >different. If you tell the customer to go to the mobo vendor to fix the >bios, they will have to find some guy in a bunker in Taiwan. Not likely. >Now I''m at VMware actively working on consolidating companies into x86 >platforms. The simple fact that the holy war between AMD and Intel has >created processors that a cheap enough and fast enough to cause disruption >in the enterprise space. My new dual core AMD processor is incredibly fast >and the entire box cost me $500 to assemble.There''s a lot of hidden EFI out there, reportedly. Casper
Our client is looking for a Senior SAN Administrator for a Full-Time position located in Philadelphia, PA. Job Description: This position will have systems administration responsibilities of the company?s SAN infrastructure. Department is Financial Automation - Technical Services - Enterprise Systems and Storage Department. Reports to Director - Enterprise Systems and Storage Group. Responsibilities: *Design, configure, implement, maintain, and support all SAN hardware, software and network infrastructure. *Perform architectural design and implementation based on current and future SAN requirements. *Monitor and track health of SAN and storage infrastructure. *Create and distribute utilization, capacity, trending, and analysis reports. *Provide routine maintenance and upgrades to established service levels. *Deploy and maintain SAN monitoring tools. *Collaborate with the Enterprise systems administration team to deploy storage and backup solutions. Regards, Gwen Miller This message posted from opensolaris.org