freebsd stable - May 2017 - zpool imported twice with different names (was Re: Fwd: ZFS)

On Wed, May 17, 2017 at 8:27 PM Paul Kraus <paul at kraus-haus.org> wrote:
>
> > On May 17, 2017, at 5:12 PM, Eric A. Borisch <eborisch at
gmail.com> wrote:
> >
> > Short version:
> >
> > It's not the -f causing the problem, it's the parsing and
(double)
> > importing via /boot/zfs/zpool.cache that is.
>
> I was unclear? _how_ do you get the double entry in zpool.cache _except_
> by using the -f option with a zpool that is already imported ?
>
To be clear, I'm using "already imported" to me == shows up in
'zpool list'
of some running OS, not described in a config file somewhere. The Oracle
description [1] that goes along with 'import -f' is: "Do not
attempt to
import a pool that is active on one system to another system. ZFS is not a
native cluster, distributed, or parallel file system and cannot provide
concurrent access from multiple, different hosts." This sounds much more
like like my description (shows in a running zpool list) than described by
a config/cache file.

Steps to recreate: (I did it in < 5 minutes in a VM.) Arabic numerals are
user actions, Roman numerals are zpool state.

1) Install FreeBSD root-on-zfs onto 'Alpha'

I. Alpha exists on disk, and is active whenever your computer is running

1a) Decide at some point later you would like to rename your root pool...
so you

2) Boot off live CD

II. Alpha exists on disk, but is not active (imported)

3) zpool import alpha beta (because that's how you rename pools)
3a) zpool helpfully points out that it *may be in use* on another system,
and use -f to import it anyway. Realize you are at the system that was
using it, but in a completely different environment, and decide that it is
safe to import.

III. Alpha still exists on disk, and is still not active

4) zpool import -f alpha beta succeeds without any complaints.

IV. Alpha no longer exists, Beta exists (with the same GUID as Alpha) AND
is active. This is an 'atomic' operation from user's point of view,
and at
no point does zpool list show more than one pool

5) zpool list shows only beta; scrubs fine, life is good..

V. Beta exists on disk and is active and healthy. Alpha is no more. (Except
as a ghost in a cache file)

6) Reboot into OS.

Kernel messages mention booting off of beta/<bootfs path>. Good, good...

VI. Beta exists at this point certainly, as boot progresses. The ghost of
Alpha is still lurking, however.

Before we drop to a prompt, the zpool.cache is parsed, and Alpha springs
back into existence beside Beta. NOTE: This is where I'm suggesting a cache
file shouldn't trump the on-disk metadata (name mismatch) and running state
(GUID collision) and lead to corruption.

VII. Alpha and Beta exist together violating the Pauli exclusion principle
as it applies to zpools.

7) Despair. Try to export one or the other. No good. Errors start to show
up in zpool status. Rebooting fails in various colorful ways. Trying to
import the pool again via Live CD is equally fruitless.

VIII. Your zpool is cooked. By a config file. Hope you weren't using ZFS
for integrity.

I would maintain that it is the combination of the import of an
already
> imported zpool which causes the double entry in the zpool.cache in
> combination with a recent rename of the zpool.
>
> Would (did) the import without the force fail ? The last host this zpool
> was imported on was _this_ host, so the ?is this zpool used by another
> host? check would not stop such an import.

Yes, it fails without the -f, and helpfully suggests using the -f to import
it. I don't think it's already imported in a way that most people
(including Oracle) describe it.

Or, is the zpool rename not updating the name in the zpool.cache file ?


As the zpool rename is occurring in the live cd system; the cache file
(living on the filesystems of the pool to be renamed) isn't accessible
during the import/rename step, and isn't known about by the active OS. I
suspect using a temporary and updating the cache file post-rename
pre-reboot would work, just like deleting the cache file before reboot
worked.

I still, however, maintain an errant cache file shouldn't trump both
on-disk state (my name... is Beta) and  in-memory state (but I already have
that GUID) and bring down a pool.

  - Eric

[1] http://docs.oracle.com/cd/E19253-01/819-5461/gazuf/index.html
>

On Wed, May 17, 2017 at 9:38 PM, Eric A. Borisch <eborisch at gmail.com>
wrote:
> 4) zpool import -f alpha beta succeeds without any complaints.
>
> IV. Alpha no longer exists, Beta exists (with the same GUID as Alpha) AND
> is active. This is an 'atomic' operation from user's point of
view, and at
> no point does zpool list show more than one pool
>
> 5) zpool list shows only beta; scrubs fine, life is good..
>
> V. Beta exists on disk and is active and healthy. Alpha is no more.
> (Except as a ghost in a cache file)
>
A quick test revealed one other tidbit this morning; if I 'zpool export
beta' at this point in the process (still in live CD OS), everything is OK
on reboot into the installed OS.

For whoever wants to investigate handling/hand-off between
bootloader/kernel/fully booted, the problem appears when a:

1) root pool is renamed, but not 'zpool export'-ed after rename via an
'external' OS (live CD)
** and **
2) A stale zpool.cache file exists on the pool referring to the original
name.

 - Eric