zfs discuss - Jul 2011 - race condition between zpool status and zio_suspend()

hello,

I encounter a deadlock condition in zfs recently,  by creating a pool using
one lun of a disk array through fibre channel?unplug the fiber cable when
there is a dd writing to the pool?then do zpool status -v immediately, that
command will hang forever.

After reading the soure code and doing some experiments, I think there is a
race condition between zpool status and zio_suspend.



ffffff0008512c40 fffffffffbc2e330                0   0  60 ffffff0213995c88
 PC: _resume_from_idle+0xf1    THREAD: txg_sync_thread()
 stack pointer for thread ffffff0008512c40: ffffff00085129b0
 [ ffffff00085129b0 _resume_from_idle+0xf1() ]
   swtch+0x145()
   cv_wait+0x61()
   zio_wait+0x5d()
   dsl_pool_sync+0xe1()
   spa_sync+0x38d()
   txg_sync_thread+0x247()
   thread_start+8()


ffffff01e255a8c0 ffffff01efbb5008 ffffff01d743da80   1  59 ffffff01cdd02d9c
 PC: _resume_from_idle+0xf1    CMD: zpool status -v test
 stack pointer for thread ffffff01e255a8c0: ffffff0007bf1a70
 [ ffffff0007bf1a70 _resume_from_idle+0xf1() ]
   swtch+0x145()
   cv_wait+0x61()
   spa_config_enter+0x86()
   spa_vdev_state_enter+0x3c()
   spa_vdev_set_common+0x37()
   spa_vdev_setpath+0x22()
   zfs_ioc_vdev_setpath+0x48()
   zfsdev_ioctl+0x15e()
   cdev_ioctl+0x45()
   spec_ioctl+0x5a()
   fop_ioctl+0x7b()
   ioctl+0x18e()
   _sys_sysenter_post_swapgs+0x149()


Normally zfs_ioc_vdev_setpath() will return EAGAIN when spa_suspended() is
true?but before zio_suspend() get called, spa_suspended() will evaluate to
false, so zpool status passes pool_status_check(), and wants to take spa
config lock in spa_config_enter() which is already held by txg_sync_thread()
in spa_sync(). And txg_sync_thread() seems will block forever? plug in the
fiber cable and  zpool clear -F   does not help.

Maybe zfs_ioc_vdev_setpath should not be called in this condition,  I made
the following change in zpool_vdev_name() in libzfs_pool.c?and it can solve
the problem, but I am not sure this will break  something or there are
better ways to solve the problem, so please help.

                               if ((newpath = devid_to_path(devid)) != NULL)
{
                                       /*
                                        * Update the path appropriately.
                                        */
-                                       set_path(zhp, nv, newpath);
+                                       if (strcmp(path, newpath) != 0)
+                                               set_path(zhp, nv, newpath);
                                       if (nvlist_add_string(nv,
                                           ZPOOL_CONFIG_PATH, newpath) == 0)

verify(nvlist_lookup_string(nv,


---------------
Best Regards

Zhiwen Zheng
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110711/071b5fe8/attachment.html>