zfs discuss - Aug 2011 - Wrong rpool used after reinstall!

I am having a problem after a new install of Solaris 10. The installed rpool
works fine when I have only those disks connected. When I connect disks from
an rpool I created during a previous installation, my newly installed rpool
is ignored even though the BIOS (x86) is set to boot only from the new rpool
drives. When the system starts it uses the old rpool! How can I get around
this? I want to use my new install and then import the old rpool under a new
name, and clean things up.

> I am having a problem after a new install of Solaris 10. The installed
> rpool
> works fine when I have only those disks connected. When I connect
> disks from
> an rpool I created during a previous installation, my newly installed
> rpool
> is ignored even though the BIOS (x86) is set to boot only from the new
> rpool
> drives. When the system starts it uses the old rpool! How can I get
> around
> this? I want to use my new install and then import the old rpool under
> a new
> name, and clean things up.
Boot on the install CD, import the old rpool (leave the new rpool drives
unplugged) and export it again. Once you have rebooted on the new rpool, you can
reimport the old rpool with a new name (rpool import someid newname). Keep in
mind that reimporting an rpool may render it useless for booting, at least
that''s not optimal.

Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 97542685
roy at karlsbakk.net
http://blogg.karlsbakk.net/
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er
et element?rt imperativ for alle pedagoger ? unng? eksessiv anvendelse av
idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og
relevante synonymer p? norsk.

Hi Roy, things got alot worse since my first email. I don''t know what
happened but I can''t import the old pool at all. It shows no errors but
when
I import it I get a kernel panic from assertion failed: zvol_get_stats(os,
nv) which looks like is fixed by patch 6801926 which is applied in Solaris
10u9. But I cannot boot update 9 on this box! I tried Solaris Express, none
of those will run right either. They all go into maintenance mode. The last
thing I can boot is update 8 and that is the one with the ZFS bug.

I have 200G of files I deliberately copied to this new mirror and now I
can''t get at them! Any ideas?

Thanks.

Jim

On Wed, Aug 3, 2011 at 8:38 AM, Anonymous Remailer (austria)
<mixmaster at remailer.privacy.at> wrote:
>
> Hi Roy, things got alot worse since my first email. I don''t know
what
> happened but I can''t import the old pool at all. It shows no
errors but when
> I import it I get a kernel panic from assertion failed: zvol_get_stats(os,
> nv) which looks like is fixed by patch 6801926 which is applied in Solaris
> 10u9. But I cannot boot update 9 on this box! I tried Solaris Express, none
> of those will run right either. They all go into maintenance mode. The last
> thing I can boot is update 8 and that is the one with the ZFS bug.
If they go into maintenance mode but could recognize the disk, you
should still be able to do zfs stuff (zpool import, etc). If you''re
lucky you''d only miss the GUI
>
> I have 200G of files I deliberately copied to this new mirror and now I
> can''t get at them! Any ideas?
Another thing you can try (albeit more complex) is use another OS
(install, or even use a linux Live CD), install virtualbox or similar
on it, pass the disk as raw vmdk, and use solaris express on the VM.
You should be able to at least import the pool and recover the data.

-- 
Fajar

You wrote:
> >
> > Hi Roy, things got alot worse since my first email. I don''t
know what
> > happened but I can''t import the old pool at all. It shows no
errors but when
> > I import it I get a kernel panic from assertion failed:
zvol_get_stats(os,
> > nv) which looks like is fixed by patch 6801926 which is applied in
Solaris
> > 10u9. But I cannot boot update 9 on this box! I tried Solaris Express,
none
> > of those will run right either. They all go into maintenance mode. The
last
> > thing I can boot is update 8 and that is the one with the ZFS bug.
> 
> If they go into maintenance mode but could recognize the disk, you
> should still be able to do zfs stuff (zpool import, etc). If
you''re
> lucky you''d only miss the GUI
Thank you for your comments. This is pretty frustrating.

Unfortunately I''m hitting another bug I saw on the net. Once the
Express
Live CD or text installer falls back to maintenance mode, I keep getting a
msg bash: /usr/bin/hostname command not found (from memory, may not be
exact). All ZFS commands fail with this message. I don''t know what
causes
this and I''m surprised since Solaris 10 update 8 works mostly fine on
the
same hardware. I would expect support to get wider not less but the opposite
seems to be happening because I can''t install update 9 on this machine.
> > I have 200G of files I deliberately copied to this new mirror and now
I
> > can''t get at them! Any ideas?
> 
> Another thing you can try (albeit more complex) is use another OS
> (install, or even use a linux Live CD), install virtualbox or similar
> on it, pass the disk as raw vmdk, and use solaris express on the VM.
> You should be able to at least import the pool and recover the data.
I didn''t think I would be able to use raw disks with exported pools in
a VM
but your comment is interesting. I will consider it. The host is bootable,
it just immediately panics upon trying to import the 2nd pool. Is there some
way I can force a normal boot where my root pool is mounted but it
doesn''t
mount the other pool? If so I could install VirtualBox and try your
suggestion without moving the drives to another box.

Aren''t there any ZFS recovery tools, or is ZFS just not expected to
break?

I mentioned I can boot the update 9 installer, but it fails when trying to
read media from the DVD because of a lack of nvidia drivers (documented
limitation) I wonder if I can do some kind of network or jumpstart
install. I have no other Solaris Intel boxes (and this post should explain
some of the reasons why) but I have several Solaris Sparc boxes. I
haven''t
gone through the doc on net/jumpstart installs, there is alot to read. But
maybe this would get update 9 on that box and maybe it could import the pool.

Are there any other Solaris based live CD or DVD I could try that have
known, good ZFS support? I will need something where I can scp/rcp/rsync the
data off that box, assuming that I can import it somehow.

Thanks,
Jim

Thanks for your comments so far. I''ll try to put everything I know into
this
post now that I have signed up at the forums.

Solaris 10, update 8 Intel, 500G ZFS root mirror rpool.

I recently received two 320G drives and realized from reading this list it
would have been better if I would have done the install on the small drives
but I didn''t have them at the time. I added the two 320G drives and
created
tank mirror.

I moved some data from other sources to the tank and then decided to go
ahead and do a new install. In preparation for that I moved all the data I
wanted to save onto the rpool mirror and then installed Solaris 10 update 8
again on the 320G drives.

When my system rebooted after the installation, I saw for some reason it
used my tank pool as root. I realize now since it was originally a root pool
and had boot blocks this didn''t help. Anyway I shut down, changed the
boot
order and then booted into my system. It paniced when trying to access the
tank and instantly rebooted. I had to go through this several times until I
caught a glimpse of one of the first messages:

assertion failed: zvol_get_stats(os, nv)

Here is what my system looks like when I boot into failsafe mode.


# zpool import
  pool: rpool
    id: 16453600103421700325
 state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:

        rpool        ONLINE
          mirror     ONLINE
            c0t2d0s0 ONLINE
            c0t3d0s0 ONLINE

  pool: tank
    id: 12861119534757646169
 state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:

        tank         ONLINE
          mirror     ONLINE
            c0t0d0s0 ONLINE
            c0t1d0s0 ONLINE

# zpool import tank
cannot import ''tank'': pool may be in use from other system
use ''-f'' to import anyway

When I do

# zpool import -f tank

I get a kernal panic and the screen flashes and the machine quickly reboots
before I can see what''s written, but I did see

   assertion failed: zvol_get_stats(os, nv)

and there seems to be a patch 6801926 associated with it. I tried the
following:

1. fmdump -ev. Either the command is not found when I bring my system up in
the Solaris failsafe mode, or when the root pool which is ok gets mounted on
/a the command is also not found. I did chroot /a /bin/sh and then the
command is found but I get error messages about not having certain files
available in /var, probably because I have /var on a separate filesystem. I
do not know how to mount things as if the system came up normally and if I
boot in the normal mode although it can import the root pool the tank pool
was also a root pool at one time and it seems to mount it and then panic and
reboot. I think it would be a lot easier to do whatever we can do if I could
get the system to boot into normal mode without importing tank. Can anyone
please explain in very clear and simple directions how to do that please?

2. Solaris 10 update 9 does boot but won''t install, presumably because
of not having a CD driver for my nvidia chipset, although the install does start
it gives a message something like "the CD is not a Solaris OS CD".. I
tried single user shell and it also panics when I zpool import -f tank. This is
very discouraging because according to the Solaris 10 update 9 patch list,
6801926 is supposed to be preapplied. I don''t understand why the update
8 DVD works on my system but update 9 doesn''t. That is just weird.

3. Solaris 11 Express falls into maintenance mode and doesn''t go any
further. I get an error message something like bash: /usr/bin/hostname command
not found, and I can''t do anything useful.

4. OpenIndiana Server doesn''t come all the way up. I get the Solaris
5.11 message and then it does nothing.

5. Milax falls into maintenance mode like Sol 11 Express and then panics when I
try to import tank.

I have Solaris on Sparc boxes available if it would help to do a net install
or jumpstart. I have never done those and it looks complicated, although I
think I may be able to get to the point in the u9 installer on my Intel box
where it asks me whether I want to install from DVD etc. But I may be wrong, and
anyway the single user shell in the u9 DVD also panics when I try to import tank
so maybe that won''t help.

I have only 4 sata ports on this Intel box so I have to keep pulling cables
to be able to boot from a DVD and then I won''t have all my drives
available. I cannot move these drives to any other box because they are
consumer drives and my servers all have ultras.

I have about 200G of data on the tank pool, about 100G or so I don''t
have
anywhere else. I created this pool specifically to make a "safe place"
to
store data that I had accumulated over several years and didn''t have
organized yet. Can anyone PLEASE help me get this data back!

Thank you.

Jim
-- 
This message posted from opensolaris.org

On 08/ 4/11 01:29 AM, Stuart James Whitefish wrote:
> I have Solaris on Sparc boxes available if it would help to do a net
install
> or jumpstart. I have never done those and it looks complicated, although I
> think I may be able to get to the point in the u9 installer on my Intel box
where it asks me whether I want to install from DVD etc. But I may be wrong, and
anyway the single user shell in the u9 DVD also panics when I try to import tank
so maybe that won''t help.
>
Put your old drive in a USB enclosure and connect it to another system in order
to read back the data.

> I have only 4 sata ports on this Intel box so I have to keep pulling cables
> to be able to boot from a DVD and then I won''t have all my drives
> available. I cannot move these drives to any other box because they are
> consumer drives and my servers all have ultras.
Most modern boards will be boot from a live USB stick.

-- 
Ian.

Jim wrote:
>> But I may be wrong, and anyway the single user shell in the u9 DVD also
panics when I try to import tank so maybe that won''t help.
Ian wrote:
> Put your old drive in a USB enclosure and connect it
> to another system in order to read back the data.
Given that update 9 can''t import the pool is this really worth trying?
I would have to buy the enclosures, if I had them already I would have tried it
in
desperation.

Jim wrote:
> > I have only 4 sata ports on this Intel box so I have to keep pulling
cables to be
> > able to boot from a DVD and then I won''t have all my drives
available. I cannot
> > move these drives to any other box because they are consumer drives
and my
> > servers all have ultras.
Ian wrote:
> Most modern boards will be boot from a live USB
> stick.
True but I haven''t found a way to get an ISO onto a USB that my system
can boot from it. I was using DD to copy the iso to the usb drive. Is there some
other way?

This is really frustrating. I haven''t had any problems with Linux
filesystems but I heard ZFS was safer. It''s really ironic that I lost
access to so much data after moving it to ZFS. Isn''t there any way to
get them back on my newly installed U8 system? If I disconnect this pool the
system starts fine. Otherwise my questions above in my summary post might be key
to getting this working.

Thanks,
Jim
-- 
This message posted from opensolaris.org

On 8/3/2011 5:47 PM, Ian Collins wrote:
>  On 08/ 4/11 01:29 AM, Stuart James Whitefish wrote:
>> I have Solaris on Sparc boxes available if it would help to do a net 
>> install
>> or jumpstart. I have never done those and it looks complicated, 
>> although I
>> think I may be able to get to the point in the u9 installer on my 
>> Intel box where it asks me whether I want to install from DVD etc. 
>> But I may be wrong, and anyway the single user shell in the u9 DVD 
>> also panics when I try to import tank so maybe that won''t
help.
>>
> Put your old drive in a USB enclosure and connect it to another system 
> in order to read back the data.
>
>
I''m curious - would it work to boot from a live CD, go to shell, and 
deport/import/rename the old rpool, then boot normally?
>> I have only 4 sata ports on this Intel box so I have to keep pulling 
>> cables
>> to be able to boot from a DVD and then I won''t have all my
drives
>> available. I cannot move these drives to any other box because they are
>> consumer drives and my servers all have ultras.
>
> Most modern boards will be boot from a live USB stick.
>
-- 


-----------------------------------------------------------------------------------
Brian Wilson, Solaris SE, UW-Madison DoIT
Room 3114 CS&S            608-263-8047
bfwilson(a)doit.wisc.edu
''I try to save a life a day. Usually it''s my own.'' -
John Crichton
-----------------------------------------------------------------------------------

I''m opening a new thread since the original subject was not as helpful
and I saw a similar problem mentioned in May of this year (2011) and others
going back to 2009. New thread is found at
http://opensolaris.org/jive/thread.jspa?threadID=140899
-- 
This message posted from opensolaris.org

On Thu, Aug 04, 2011 at 03:52:39AM -0700, Stuart James Whitefish
wrote:
> Jim wrote:
> 
> >> But I may be wrong, and anyway the single user shell in the u9 DVD
also panics when I try to import tank so maybe that won''t help.
> 
> Ian wrote:
> 
> > Put your old drive in a USB enclosure and connect it
> > to another system in order to read back the data.
> 
> Given that update 9 can''t import the pool is this really worth
trying?
> I would have to buy the enclosures, if I had them already I would have
tried it in
> desperation.
> 
> Jim wrote:
> 
> > > I have only 4 sata ports on this Intel box so I have to keep
pulling cables to be
> > > able to boot from a DVD and then I won''t have all my
drives available. I cannot
> > > move these drives to any other box because they are consumer
drives and my
> > > servers all have ultras.
> 
> Ian wrote:
> 
> > Most modern boards will be boot from a live USB
> > stick.
> 
> True but I haven''t found a way to get an ISO onto a USB that my
system can boot from it. I was using DD to copy the iso to the usb drive. Is
there some other way?

Maybe give http://unetbootin.sourceforge.net/ a try.

Bill


> 
> This is really frustrating. I haven''t had any problems with Linux
filesystems but I heard ZFS was safer. It''s really ironic that I lost
access to so much data after moving it to ZFS. Isn''t there any way to
get them back on my newly installed U8 system? If I disconnect this pool the
system starts fine. Otherwise my questions above in my summary post might be key
to getting this working.
> 
> Thanks,
> Jim
> -- 
> This message posted from opensolaris.org
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

On Fri, 5 Aug 2011, Bill wrote:
>>
>> True but I haven''t found a way to get an ISO onto a USB that
my system can boot from it. I was using DD to copy the iso to the usb drive. Is
there some other way?
>
> Maybe give http://unetbootin.sourceforge.net/ a try.
This package seems to list support for most x86 OSs EXCEPT for 
*Solaris.

Bob
-- 
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

On 08/ 4/11 10:52 PM, Stuart James Whitefish wrote:
> Ian wrote:
>> Put your old drive in a USB enclosure and connect it
>> to another system in order to read back the data.
> Given that update 9 can''t import the pool is this really worth
trying?
I would use a newer (express maybe) system.
>> Most modern boards will be boot from a live USB
>> stick.
> True but I haven''t found a way to get an ISO onto a USB that my
system can boot from it. I was using DD to copy the iso to the usb drive. Is
there some other way?
Recent OpenSolaris based builds have a handy utility usbcopy.
> This is really frustrating. I haven''t had any problems with Linux
filesystems but I heard ZFS was safer. It''s really ironic that I lost
access to so much data after moving it to ZFS. Isn''t there any way to
get them back on my newly installed U8 system? If I disconnect this pool the
system starts fine. Otherwise my questions above in my summary post might be key
to getting this working.
>
If you have support, badger them.  Otherwise use a newer system.

-- 
Ian.

----- Original Message -----
> From: Brian Wilson <brian.wilson at doit.wisc.edu>
> To: zfs-discuss at opensolaris.org
> Cc: 
> Sent: Thursday, August 4, 2011 2:57:26 PM
> Subject: Re: [zfs-discuss] Wrong rpool used after reinstall!
> 
> I''m curious - would it work to boot from a live CD, go to shell,
and
> deport/import/rename the old rpool, then boot normally?
Hi Brian,

No, it doesn''t work. Kernel panic still happens.
>>  Most modern boards will be boot from a live USB stick.
I think this quote was from Ian. Yes they will, but only Solaris 11 Express and
OpenIndiana seem to have USB bootable installers.

I installed Solaris 11 this way. You can see from the screen shot I posted the
error still occurs.

The data is still inaccessible. I was hope somebody from Oracle would say
something but I don''t see any emails.

Jim

Hi Ian,
> I would use a newer (express maybe) system.
I did and it panics. Posted screenshot last week.

> Recent OpenSolaris based builds have a handy utility usbcopy.
Thanks, I used that to create a bootable Solaris 11 Express. Hiroshi did a great
job!
>>  This is really frustrating. I haven''t had any problems with
Linux
> filesystems but I heard ZFS was safer. It''s really ironic that I
lost access
> to so much data after moving it to ZFS. Isn''t there any way to get
them back
> on my newly installed U8 system? If I disconnect this pool the system
starts
> fine. Otherwise my questions above in my summary post might be key to
getting
> this working.
>> 
> If you have support, badger them.? Otherwise use a newer system.
What''s newer than Sol 11 Express?

I would like to run a supported system but I don''t have supported
hardware. I don''t really need new
hardware at the moment, but I do need my 200G of missing data back.

Thanks,

Jim