zfs discuss - Jan 2009 - Error 16: Inconsistent filesystem structure after a change in the system

I''ve hit this bug on my home machine couple of times and finally
decided to log
it since I''ve spend 2 days configuring my "OpenSolaris 2008.11
snv_101b_rc2
X86" and after installing the SUNWsmbfskr package I ended up in the
grub> menu.
The package contains a necessary module for CIFS. After pkg install
SUNWsmbfskr, I rebooted the box. There was no grub menu anymore but only the
grub prompt. I managed to boot it manually using another BE. I was unable to
use the original BE getting this error message during the manual boot:
bootfs rpool/ROOT/rp01 
OK
kernel$ /platform/i86pc/kernel/$ISADIR/unix -B $ZFS-BOOTFS
OK
module$ /platform/i86pc/$ISADIR/boot_archive
after this command I got the error: Error 16: Inconsistent filesystem
structure.
I had to use a different BE to boot up the system.
I''ve tried to repair boot_archive from the rp01 BE but unsuccessfully
every
time ending up in the grub menu forced to manually boot the system using a
different BE.
My unsuccessful method to rebuild boot_archive:
# mkdir /a
# bootadm mount rp01 /a
# rm /a/platform/i86pc/boot_archive
# bootadm update-archive -f -R /a
Creating boot_archive for /a
updating /a/platform/i86pc/amd64/boot_archive
updating /a/platform/i86pc/boot_archive
# beadm unmount rp01
# reboot

rafal at panda:~$ beadm list
BE            Active Mountpoint Space   Policy Created          
--            ------ ---------- -----   ------ -------          
opensolaris   -      -          8.23M   static 2008-12-16 21:04 
opensolaris-1 -      -          121.78M static 2008-12-20 12:40 
opensolaris-2 -      -          8.21M   static 2008-12-21 01:57 
opensolaris-3 -      -          67.31M  static 2008-12-21 15:12 
opensolaris-4 -      -          306.0K  static 2008-12-21 20:46 
opensolaris-5 -      -          165.07M static 2008-12-21 20:49 
rp01          -      -          140.59M static 2008-12-22 21:38 
rp02          NR     /          3.48G   static 2008-12-23 00:06 
update        -      -          13.59M  static 2008-12-20 12:25 

To reproduce the bug on my system is very simple just change/install/un-install
anything in the kernel directory or try to create, activate a new BE using the
beadm command and you will end up in the grub prompt (no grub menu available).
To rescue from this stage only a manual boot or an Osol2008.11 Live CD.

Could you please provide me with a method to recover the rp01 BE. I''ve
spend to
much time configuring it and don''t want to loose the changes. But the
method
must be different then the one I provided above as the above one
doesn''t work.

Also trying to create a new BE from the rp01 BE doesn''t work:
rafal at panda:~$ pfexec su -
Sun Microsystems Inc.   SunOS 5.11      snv_101b        November 2008
You have new mail.
root at panda:~# export BE_PRINT_ERR=true
root at panda:~# beadm create -e rp01 rp03
be_copy: failed to snapshot BE (rpool/ROOT/rp01 at rp03): dataset is busy
Unable to create rp03.
Mount busy.
root at panda:~# 

As you can see the rp01 BE suppose to be a non-active BE name but somehow
it''s
not.

I''ve checked the disk and there are no errors:
rafal at panda:~$ zpool status -xv
all pools are healthy
rafal at panda:~$ zpool status
  pool: rpool
 state: ONLINE
 scrub: scrub completed after 2h51m with 0 errors on Mon Dec 29 02:58:19 2008
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          c3d0s0    ONLINE       0     0     0

errors: No known data errors
rafal at panda:~$

The next day I recovered the system meaning I have the grub menu and the system
boots up
fine automatically without any manual intervention. After that the update
manager recognized that on pkg.opensolaris.org there is an update for my
system. It was the entire package v0.5.11-0.101. So I clicked the update all
button, a new window popped up "Create new boot environment". This
should
create an update clone of the active BE. After a reboot the same story. No grub
menu available, only the grub> prompt. The system didn''t boot
automatically
anymore. 
The error message is the same: Error 16: Inconsistent filesystem structure
-- 
This message posted from opensolaris.org

I recovered the system and created the opensolaris-12 BE. The system was working
fine. I had the grub menu, it was fully recovered.
At this stage I decided to create a new BE but leave the opensolaris-12 BE as an
active BE and manually boot to the opensolaris-13 BE.
So the situation looked like this:
beadm list
opensolaris-12 R     /          3.73G   static 2009-01-03 14:44 
opensolaris-13 N      -          120.74M static 2009-01-03 14:51 
Running the opensolaris-13 BE I installed 2 packages: SUNWsmbs SUNWsmbskr
responsible for the solaris CIFS service. I reboot the box and it still was
fine. I had the grub menu and the boot_archive wasn''t corrupted. After
the reboot I configured the solaris CIFS service and exchanged some data between
the opensolaris and a laptop via a network. I''ve been doing this for
3h. Then I decided to reboot the box again to see if it''s still OK. I
used init 6, but this time I ended up in the grub> menu. When I manually
wanted to boot it looked like this:
bootfs rpool/ROOT/opensolaris-13
OK
kernel$ /platform/i86pc/kernel/$ISADIR/unix -B $ZFS-BOOTFS
Error 16: Inconsistent filesystem structure
I had to use rpool/ROOT/opensolaris-12 and it was fine.
I didn''t even get to load the boot_archive module. Usually it breaks
during the boot_archive module load.
I''m unable to use the opensolaris-13 BE and I''ve lost the CIFS
configuration as I can''t boot the BE.

Can someone wise help me to narrow down this bug.
Thanks in advance
Rafal
-- 
This message posted from opensolaris.org

Hey Rafal,

this sounds like missing GANG block support in GRUB. Checkout putback  
log for snv_106 (afaik), there''s a bug where grub fails like this.

Cheers,
Spity

On 3.1.2009, at 21:11, Rafal Pratnicki wrote:
> I recovered the system and created the opensolaris-12 BE. The system  
> was working fine. I had the grub menu, it was fully recovered.
> At this stage I decided to create a new BE but leave the  
> opensolaris-12 BE as an active BE and manually boot to the  
> opensolaris-13 BE.
> So the situation looked like this:
> beadm list
> opensolaris-12 R     /          3.73G   static 2009-01-03 14:44
> opensolaris-13 N      -          120.74M static 2009-01-03 14:51
> Running the opensolaris-13 BE I installed 2 packages: SUNWsmbs  
> SUNWsmbskr responsible for the solaris CIFS service. I reboot the  
> box and it still was fine. I had the grub menu and the boot_archive  
> wasn''t corrupted. After the reboot I configured the solaris CIFS  
> service and exchanged some data between the opensolaris and a laptop  
> via a network. I''ve been doing this for 3h. Then I decided to
reboot
> the box again to see if it''s still OK. I used init 6, but this
time
> I ended up in the grub> menu. When I manually wanted to boot it  
> looked like this:
> bootfs rpool/ROOT/opensolaris-13
> OK
> kernel$ /platform/i86pc/kernel/$ISADIR/unix -B $ZFS-BOOTFS
> Error 16: Inconsistent filesystem structure
> I had to use rpool/ROOT/opensolaris-12 and it was fine.
> I didn''t even get to load the boot_archive module. Usually it
breaks
> during the boot_archive module load.
> I''m unable to use the opensolaris-13 BE and I''ve lost the
CIFS
> configuration as I can''t boot the BE.
>
> Can someone wise help me to narrow down this bug.
> Thanks in advance
> Rafal
> -- 
> This message posted from opensolaris.org
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss