zfs crypto discuss - Apr 2008 - [Bug 1055] New: zpool destroy on a pool has an encrypted dataset (keyscope= pool) causes panic

http://defect.opensolaris.org/bz/show_bug.cgi?id=1055

           Summary: zpool destroy on a pool has an encrypted dataset
                    (keyscope=pool) causes panic
    Classification: Development
           Product: zfs-crypto
           Version: unspecified
          Platform: Other
        OS/Version: Solaris
            Status: NEW
          Severity: critical
          Priority: P1
         Component: other
        AssignedTo: darrenm at opensolaris.org
        ReportedBy: hua.tang at sun.com
         QAContact: hua.tang at sun.com
                CC: zfs-crypto-discuss at opensolaris.org
   Estimated Hours: 0.0


panic[cpu2]/thread=ffffff021522b760: assertion failed: key->zk_refcnt == 0
(0x1
== 0x0), file: ../../common/fs/zfs/zio_crypt.c, line: 81

ffffff0005795ab0 genunix:assfail3+b9 ()
ffffff0005795ae0 zfs:zio_crypt_key_free+a1 ()
ffffff0005795b30 zfs:spa_keystore_fini+c4 ()
ffffff0005795b70 zfs:spa_unload+60 ()
ffffff0005795bd0 zfs:spa_export_common+1ba ()
ffffff0005795bf0 zfs:spa_destroy+19 ()
ffffff0005795c20 zfs:zfs_ioc_pool_destroy+1e ()
ffffff0005795ca0 zfs:zfsdev_ioctl+162 ()
ffffff0005795ce0 genunix:cdev_ioctl+48 ()
ffffff0005795d20 specfs:spec_ioctl+86 ()
ffffff0005795da0 genunix:fop_ioctl+7b ()
ffffff0005795eb0 genunix:ioctl+174 ()
ffffff0005795f00 unix:brand_sys_syscall32+292 ()

I have seen this panic several times. But I couldn''t located the root
cause
because after this panic the test machine couldn''t be recovered.
>From the test journal file, I suspects the panic is caused by
''zpool destroy''
on a pool who has an encrypted dataset with keyscope=pool.

My test machine couldn''t be boot. It hanged.

[1;1SunOS Release 5.11 Version zfs-crypto-gate 64-bit                          
Copyright 1983-2008 Sun Microsystems, Inc.  All rights reserved.                
Use is subject to license terms.                                                
DEBUG enabled

I booted up from failsafe. The pool created in my test case was already
destroyed. 
# zpool import -d /a/var/tmp/zfs-crypto
no pools available to import

But there was /a/etc/zfs/zpool.cache.
# ls /a/etc/zfs
zpool.cache

After deleting zpool.cache, the test machine wasn''t booted up. Still
hanged.

This panic affected the testing very seriously because each test case should
destroy the pool and the test machine couldn''t be recover.

Is there any workarounds?

-- 
Configure bugmail: http://defect.opensolaris.org/bz/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

http://defect.opensolaris.org/bz/show_bug.cgi?id=1055


Darren J Moffat <darrenm at opensolaris.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |INVALID




--- Comment #1 from Darren J Moffat <darrenm at opensolaris.org> 
2008-04-09 01:36:40 ---
I have seen this but I believe I have also fixed it.

The new code is:

static boolean_t
zio_crypt_key_free(zio_crypt_key_t *key, boolean_t lock_held)
{
        if (key == NULL)
                return (TRUE);

        if (!lock_held)
                mutex_enter(&key->zk_reflock);
        if (key->zk_refcnt != 0)
                return (FALSE);

        ASSERT3U(key->zk_refcnt, ==, 0);
        if (key->zk_key.ck_length != 0) {


Note that the ASSERT that was trigged in the panic has now moved to after the
check that zk_refcnt is 0 and the locking has been improved.

I''m closing this as the originally reported case can''t happen
with the new code
- the new code will be in this weeks build.

-- 
Configure bugmail: http://defect.opensolaris.org/bz/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

http://defect.opensolaris.org/bz/show_bug.cgi?id=1055


Grace <hua.tang at sun.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |REOPENED
         Resolution|INVALID                     |




--- Comment #2 from Grace <hua.tang at sun.com>  2008-04-10 21:11:21 ---
I saw the panic again with the build ''darrenm 2008-04-10
[zfs-crypto-gate]''.

panic[cpu0]/thread=30018154380: assertion failed: key->zk_refcnt == 0 (0x1
=0x0), file: ../../common/fs/zfs/zio_crypt.c, line: 81

000002a100c91490 genunix:assfail3+98 (7aa953b8, 1, 7aa953d0, 0, 7aa95340, 51)
  %l0-3: 00000000f4eef4ee 00000000f4eef400 0000000000000000 00000000018a2000
  %l4-7: 0000000000000000 0000000001853800 000000000135f800 0000000000000000
000002a100c91550 zfs:zio_crypt_key_free+34 (6001166bce0, 7aa95000, 7aa95000,
7aa95000, 0, 1346c00)
  %l0-3: 0000000000000001 0000060011774100 0000000000000001 0000000000000001
  %l4-7: 00000000000045ec 0000030000048240 0000000000000003 0000000001346c00
000002a100c91600 zfs:spa_keystore_fini+a0 (60012087388, 2a100c916b8,
60012087380, 60012087380, 60012087380, 30004fecd40)
  %l0-3: 0000060011bc9640 0000000000000002 0000000000000007 000000007aa8e800
  %l4-7: 0000000000000002 0000030004fed248 0000030004fed242 0000000000000000
000002a100c916c0 zfs:spa_unload+54 (30004fecd40, 0, 7aa8d400, 60011bc95e0, 9,
7aa7cb60)
  %l0-3: 0000060011a3f0a0 000000007aa8bc00 0000000000000008 000000007aa8b540
  %l4-7: 0000030004ff1bc0 0000000000000008 0000000000000001 000000007aa8e000
000002a100c91770 zfs:spa_export_common+210 (30036028000, 1, 0, 7aa7cc8b,
30004fecd40, 700feed8)
  %l0-3: 0000000000000000 0000000000000000 000000007aa8c400 000000007aa7cc8b
  %l4-7: 0000000000000015 000000007aa7cc00 0000000000000001 000000007aa8c400
000002a100c91820 zfs:zfsdev_ioctl+1c4 (7aa65cd4, 700f7b18, ffbf9768, 1000,
600119a7bf0, 30036028000)
  %l0-3: 0000000000000048 00000000700f7b20 00000000700f7800 0000000000000009
  %l4-7: 0000000000000003 000000000000000c 0000000000000001 00000000700f7b28
000002a100c918d0 genunix:fop_ioctl+58 (60010b61a80, 5a03, ffbf9768, 100003, 0,
2a100c91adc)
  %l0-3: 00000000000000c0 000000000139756c 0000030005816080 0000000000000008
  %l4-7: 0000000000000000 00000000018d8c00 0000000000000001 0000030034e87000
000002a100c91990 genunix:ioctl+16c (3, 5a03, ffbf9768, c4754, 600119732c0,
ff2f5664)
  %l0-3: 0000000000000003 0000000000000003 0000000000000001 0000000000000000
  %l4-7: 000000001cdb39c3 000000000135bc00 0000000000000000 0000000000000000

panic: entering debugger (continue to save dump)
Type  ''go'' to resume

-- 
Configure bugmail: http://defect.opensolaris.org/bz/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

http://defect.opensolaris.org/bz/show_bug.cgi?id=1055





--- Comment #3 from Grace <hua.tang at sun.com>  2008-04-15 01:15:49 ---
I was able to reproduce this bug manually.

# zpool create -o keysource=passphrase,prompt tank /export/home/testfileEnter
passphrase for ''tank'': 
Enter again: 
# zfs create -o keyscope=dataset -o keysource=passphrase,prompt -o
encryption=on tank/fs
Enter passphrase for ''tank/fs'': 
Enter again: 
# zpool key -u tank
# zpool key -l tank
Enter passphrase for ''tank'': 
# echo $?
0
# zfs create -o encryption=on tank/fs_2
# zfs key -u tank/fs_2
# echo $?
0
# zfs get keystatus tank/fs_2
NAME       PROPERTY   VALUE        SOURCE
tank/fs_2  keystatus  available    -
# zfs key -l tank/fs_2
Key Load Error in ''tank/fs_2''.: Key already loaded.
# echo $?
255
# zpool key -u tank

It panic''ed here.
Please see if this bug has relation to bug 1204 and 1205.

-- 
Configure bugmail: http://defect.opensolaris.org/bz/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

http://defect.opensolaris.org/bz/show_bug.cgi?id=1055


Darren J Moffat <darrenm at opensolaris.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|REOPENED                    |RESOLVED
         Resolution|                            |FIXINSOURCE




--- Comment #4 from Darren J Moffat <darrenm at opensolaris.org> 
2008-04-22 17:24:35 ---
Please verify again with binaries built post 2008-04-22

-- 
Configure bugmail: http://defect.opensolaris.org/bz/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

http://defect.opensolaris.org/bz/show_bug.cgi?id=1055


Grace <hua.tang at sun.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |REOPENED
         Resolution|FIXINSOURCE                 |




--- Comment #5 from Grace <hua.tang at sun.com>  2008-04-23 23:35:15 ---
I verified with build 2008-04-24. The panic was still there each time I run the
test suite.

-- 
Configure bugmail: http://defect.opensolaris.org/bz/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

http://defect.opensolaris.org/bz/show_bug.cgi?id=1055





--- Comment #6 from Darren J Moffat <darrenm at opensolaris.org> 
2008-04-30 09:00:51 ---
This should be fixed.

-- 
Configure bugmail: http://defect.opensolaris.org/bz/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

http://defect.opensolaris.org/bz/show_bug.cgi?id=1055


Darren J Moffat <darrenm at opensolaris.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|REOPENED                    |CLOSED
         Resolution|                            |FIXED




--- Comment #7 from Darren J Moffat <darrenm at opensolaris.org> 
2008-05-02 09:03:45 ---
No longer reproducible it appears to be really fixed and integrated this time!

-- 
Configure bugmail: http://defect.opensolaris.org/bz/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.