Jason Gallagher - Sun Microsystems
2006-Oct-31 22:26 UTC
[zfs-discuss] zfs: zvols minor #''s changing and causing probs w/ volumes
Team,
**Please respond to me and my coworker listed in the Cc, since neither
one of us are on this alias**
QUICK PROBLEM DESCRIPTION:
Cu created a dataset which contains all the zvols for a particular
zone. The zone is then given access to all the zvols in the dataset
using a match statement in the zoneconfig (see long problem description
for details). After the initial boot of the zone everything appears
fine and the localzone zvol dev files match the globalzone zvol dev
files.
Upon the reboot of the box (following the initial that had no problems)
the Minor numbers of the zvols are different between the local zone and
the Global zone and some of the volumes are not mounted in the correct
location and some volumes can''t even be mounted.
LONG PROBLEM DESCRIPTION:
All the details are listed below, authored by the customer:
Here is a summary of the problem we are experiencing.
Just a quick little background. We had hoped to use ZFS for our
filesystems but as a result of our backup system not fully supporting
ZFS yet we are stuck with using UFS for now. In an effort to make
migrating to ZFS in the future that much easier and to be able to take
advantage of some of the other features ZFS gives us we have decided to
use ZFS volumes and create UFS filesystems on top of them.
We have created a dataset which contain all the zvols for a particular
zone. The zone is then given access to all the zvols in the dataset
using a match statement in the zoneconfig.
[/users/mdey]
mdey at tofu$ zonecfg -z biscotti info
zonepath: /zones/biscotti
autoboot: false
pool:
inherit-pkg-dir:
dir: /lib
inherit-pkg-dir:
dir: /platform
inherit-pkg-dir:
dir: /sbin
inherit-pkg-dir:
dir: /usr
inherit-pkg-dir:
dir: /opt
net:
address: 10.1.33.91
physical: hme0
device
match: /dev/zvol/rdsk/d1000pool/biscotti-vols/*
device
match: /dev/zvol/dsk/d1000pool/biscotti-vols/*
There are 4 volumes in the dataset
[/]
root at tofu# zfs list -r d1000pool/biscotti-vols
NAME USED AVAIL REFER MOUNTPOINT
d1000pool/biscotti-vols 400M 197G 49K none
d1000pool/biscotti-vols/vol1 11.2M 197G 11.2M -
d1000pool/biscotti-vols/vol2 10.7M 197G 10.7M -
d1000pool/biscotti-vols/vol3 11.0M 197G 11.0M -
d1000pool/biscotti-vols/vol4 10.5M 197G 10.5M -
The volumes are mounted in the zone via the zones vfstab
/dev/zvol/dsk/d1000pool/biscotti-vols/vol1
/dev/zvol/rdsk/d1000pool/biscotti-vols/vol1 /vol1 ufs 2
yes -
/dev/zvol/dsk/d1000pool/biscotti-vols/vol2
/dev/zvol/rdsk/d1000pool/biscotti-vols/vol2 /vol2 ufs 2
yes -
/dev/zvol/dsk/d1000pool/biscotti-vols/vol3
/dev/zvol/rdsk/d1000pool/biscotti-vols/vol3 /vol3 ufs 2
yes -
/dev/zvol/dsk/d1000pool/biscotti-vols/vol4
/dev/zvol/rdsk/d1000pool/biscotti-vols/vol4 /vol4 ufs 2
yes -
After the initial boot of the zone everything appears fine and the
localzone zvol dev files match the globalzone zvol dev files.
[/]
root at tofu# ls -lL /zones/biscotti/dev/zvol/rdsk/d1000pool/biscotti-vols
total 0
crw------- 1 root sys 256, 2 Oct 23 21:23 vol1
crw------- 1 root sys 256, 3 Oct 23 21:23 vol2
crw------- 1 root sys 256, 4 Oct 23 21:23 vol3
crw------- 1 root sys 256, 5 Oct 23 21:23 vol4
[/]
root at tofu# ls -lL /dev/zvol/rdsk/d1000pool/biscotti-vols
total 0
crw------- 1 root sys 256, 2 Oct 23 21:02 vol1
crw------- 1 root sys 256, 3 Oct 23 21:02 vol2
crw------- 1 root sys 256, 4 Oct 23 21:02 vol3
crw------- 1 root sys 256, 5 Oct 23 21:02 vol4
I login to the zone and create a file in each mount to keep track of
which volume is which.
[/]
root at biscotti# ls -l /vol?/vol?
-rw------- 1 root root 10485760 Oct 23 21:38 /vol1/vol1
-rw------- 1 root root 10485760 Oct 23 21:38 /vol2/vol2
-rw------- 1 root root 10485760 Oct 23 21:38 /vol3/vol3
-rw------- 1 root root 10485760 Oct 23 21:38 /vol4/vol4
I then create a new volume vol5 then shutdown the zone and then reboot
the box with an init 6
Upon the reboot of the box the Minor numbers of the zvols are different
between the local zone and the Global zone.
[/zones/biscotti/dev/zvol/rdsk/d1000pool/biscotti-vols]
root at tofu# ls -lL
total 0
crw------- 1 root sys 256, 2 Oct 23 21:23 vol1
crw------- 1 root sys 256, 3 Oct 23 21:23 vol2
crw------- 1 root sys 256, 4 Oct 23 21:23 vol3
crw------- 1 root sys 256, 5 Oct 23 21:23 vol4
crw------- 1 root sys 256, 6 Oct 23 21:44 vol5
[/zones/biscotti/dev/zvol/rdsk/d1000pool/biscotti-vols]
root at tofu# cd /dev/zvol/rdsk/d1000pool/biscotti-vols
[/dev/zvol/rdsk/d1000pool/biscotti-vols]
root at tofu# ls -lL
total 0
crw------- 1 root sys 256, 4 Oct 24 10:57 vol1
crw------- 1 root sys 256, 3 Oct 24 10:57 vol2
crw------- 1 root sys 256, 5 Oct 24 10:57 vol3
crw------- 1 root sys 256, 6 Oct 24 10:57 vol4
crw------- 1 root sys 256, 2 Oct 24 10:57 vol5
[/dev/zvol/rdsk/d1000pool/biscotti-vols]
root at tofu#
Upon bootup of the zone the problem becomes even more apparent as
volumes are not mounted in the correct location and some volumes can''t
even be mounted.
[/]
root at biscotti# ls -l
/vol?/vol?
-rw------- 1 root root 10485760 Oct 23 21:38 /vol2/vol2
-rw------- 1 root root 10485760 Oct 23 21:38 /vol3/vol1
-rw------- 1 root root 10485760 Oct 23 21:38 /vol4/vol3
[/]
root at biscotti# mount /vol1
mount: /dev/zvol/dsk/d1000pool/biscotti-vols/vol1 is already mounted or
/vol1 is busy
If I halt the zone then rm -rf /zones/biscotti/dev/zvol and then reboot
eh zone the problems are resolved.
[/]
root at tofu# ls -lL /dev/zvol/rdsk/d1000pool/biscotti-vols/
total 0
crw------- 1 root sys 256, 4 Oct 24 10:57 vol1
crw------- 1 root sys 256, 3 Oct 24 10:57 vol2
crw------- 1 root sys 256, 5 Oct 24 10:57 vol3
crw------- 1 root sys 256, 6 Oct 24 10:57 vol4
crw------- 1 root sys 256, 2 Oct 24 10:57 vol5
[/]
root at tofu# ls -lL /zones/biscotti/dev/zvol/rdsk/d1000pool/biscotti-vols/
total 0
crw------- 1 root sys 256, 4 Oct 24 11:07 vol1
crw------- 1 root sys 256, 3 Oct 24 11:07 vol2
crw------- 1 root sys 256, 5 Oct 24 11:07 vol3
crw------- 1 root sys 256, 6 Oct 24 11:07 vol4
crw------- 1 root sys 256, 2 Oct 24 11:07 vol5
I then login to the zone and everything appears fine.
[/]
root at biscotti# ls -l /vol?/vol?
-rw------- 1 root root 10485760 Oct 23 21:38 /vol1/vol1
-rw------- 1 root root 10485760 Oct 23 21:38 /vol2/vol2
-rw------- 1 root root 10485760 Oct 23 21:38 /vol3/vol3
-rw------- 1 root root 10485760 Oct 23 21:38 /vol4/vol4
=====================end summary of the
problem=========================
--
Jason Gallagher
Operating Systems Group Support Team
Sun Microsystems, Inc.
Working hours: Sat/Sun 6am - 6pm & Mon/Tue 8am - 5pm MNT
Phone: Call 1-800-USA4-SUN and reference your case id
e-mail: jvg at sun.com
Case info located at http://www.sun.com/service/online
Edward Pilatowicz
2006-Nov-01 01:53 UTC
[zfs-discuss] zfs: zvols minor #''s changing and causing probs w/ volumes
if your running solaris 10 or an early nevada build then it''s possible your hitting this bug (which i fixed in build 35): 4976415 devfsadmd for zones could be smarter when major numbers change if you''re running a recent nevada build then this could be a new issue. so what version of solaris are you running? thanks ed On Tue, Oct 31, 2006 at 03:26:06PM -0700, Jason Gallagher - Sun Microsystems wrote:> Team, > > **Please respond to me and my coworker listed in the Cc, since neither > one of us are on this alias** > > QUICK PROBLEM DESCRIPTION: > > Cu created a dataset which contains all the zvols for a particular > zone. The zone is then given access to all the zvols in the dataset > using a match statement in the zoneconfig (see long problem description > for details). After the initial boot of the zone everything appears > fine and the localzone zvol dev files match the globalzone zvol dev > files. > > Upon the reboot of the box (following the initial that had no problems) > the Minor numbers of the zvols are different between the local zone and > the Global zone and some of the volumes are not mounted in the correct > location and some volumes can''t even be mounted. > > > LONG PROBLEM DESCRIPTION: > > All the details are listed below, authored by the customer: > > > Here is a summary of the problem we are experiencing. > > Just a quick little background. We had hoped to use ZFS for our > filesystems but as a result of our backup system not fully supporting > ZFS yet we are stuck with using UFS for now. In an effort to make > migrating to ZFS in the future that much easier and to be able to take > advantage of some of the other features ZFS gives us we have decided to > use ZFS volumes and create UFS filesystems on top of them. > > We have created a dataset which contain all the zvols for a particular > zone. The zone is then given access to all the zvols in the dataset > using a match statement in the zoneconfig. > > [/users/mdey] > > mdey at tofu$ zonecfg -z biscotti info > > zonepath: /zones/biscotti > > autoboot: false > > pool: > > inherit-pkg-dir: > > dir: /lib > > inherit-pkg-dir: > > dir: /platform > > inherit-pkg-dir: > > dir: /sbin > > inherit-pkg-dir: > > dir: /usr > > inherit-pkg-dir: > > dir: /opt > > net: > > address: 10.1.33.91 > > physical: hme0 > > device > > match: /dev/zvol/rdsk/d1000pool/biscotti-vols/* > > device > > match: /dev/zvol/dsk/d1000pool/biscotti-vols/* > > > > There are 4 volumes in the dataset > > > > [/] > > root at tofu# zfs list -r d1000pool/biscotti-vols > > NAME USED AVAIL REFER MOUNTPOINT > > d1000pool/biscotti-vols 400M 197G 49K none > > d1000pool/biscotti-vols/vol1 11.2M 197G 11.2M - > > d1000pool/biscotti-vols/vol2 10.7M 197G 10.7M - > > d1000pool/biscotti-vols/vol3 11.0M 197G 11.0M - > > d1000pool/biscotti-vols/vol4 10.5M 197G 10.5M - > > > > The volumes are mounted in the zone via the zones vfstab > > > > /dev/zvol/dsk/d1000pool/biscotti-vols/vol1 > /dev/zvol/rdsk/d1000pool/biscotti-vols/vol1 /vol1 ufs 2 > yes - > > /dev/zvol/dsk/d1000pool/biscotti-vols/vol2 > /dev/zvol/rdsk/d1000pool/biscotti-vols/vol2 /vol2 ufs 2 > yes - > > /dev/zvol/dsk/d1000pool/biscotti-vols/vol3 > /dev/zvol/rdsk/d1000pool/biscotti-vols/vol3 /vol3 ufs 2 > yes - > > /dev/zvol/dsk/d1000pool/biscotti-vols/vol4 > /dev/zvol/rdsk/d1000pool/biscotti-vols/vol4 /vol4 ufs 2 > yes - > > > > After the initial boot of the zone everything appears fine and the > localzone zvol dev files match the globalzone zvol dev files. > > > > [/] > > root at tofu# ls -lL /zones/biscotti/dev/zvol/rdsk/d1000pool/biscotti-vols > > total 0 > > crw------- 1 root sys 256, 2 Oct 23 21:23 vol1 > > crw------- 1 root sys 256, 3 Oct 23 21:23 vol2 > > crw------- 1 root sys 256, 4 Oct 23 21:23 vol3 > > crw------- 1 root sys 256, 5 Oct 23 21:23 vol4 > > > > [/] > > root at tofu# ls -lL /dev/zvol/rdsk/d1000pool/biscotti-vols > > total 0 > > crw------- 1 root sys 256, 2 Oct 23 21:02 vol1 > > crw------- 1 root sys 256, 3 Oct 23 21:02 vol2 > > crw------- 1 root sys 256, 4 Oct 23 21:02 vol3 > > crw------- 1 root sys 256, 5 Oct 23 21:02 vol4 > > > > > > I login to the zone and create a file in each mount to keep track of > which volume is which. > > > > [/] > > root at biscotti# ls -l /vol?/vol? > > -rw------- 1 root root 10485760 Oct 23 21:38 /vol1/vol1 > > -rw------- 1 root root 10485760 Oct 23 21:38 /vol2/vol2 > > -rw------- 1 root root 10485760 Oct 23 21:38 /vol3/vol3 > > -rw------- 1 root root 10485760 Oct 23 21:38 /vol4/vol4 > > > > > > I then create a new volume vol5 then shutdown the zone and then reboot > the box with an init 6 > > > > Upon the reboot of the box the Minor numbers of the zvols are different > between the local zone and the Global zone. > > > > [/zones/biscotti/dev/zvol/rdsk/d1000pool/biscotti-vols] > > root at tofu# ls -lL > > total 0 > > crw------- 1 root sys 256, 2 Oct 23 21:23 vol1 > > crw------- 1 root sys 256, 3 Oct 23 21:23 vol2 > > crw------- 1 root sys 256, 4 Oct 23 21:23 vol3 > > crw------- 1 root sys 256, 5 Oct 23 21:23 vol4 > > crw------- 1 root sys 256, 6 Oct 23 21:44 vol5 > > > > [/zones/biscotti/dev/zvol/rdsk/d1000pool/biscotti-vols] > > root at tofu# cd /dev/zvol/rdsk/d1000pool/biscotti-vols > > > > [/dev/zvol/rdsk/d1000pool/biscotti-vols] > > root at tofu# ls -lL > > total 0 > > crw------- 1 root sys 256, 4 Oct 24 10:57 vol1 > > crw------- 1 root sys 256, 3 Oct 24 10:57 vol2 > > crw------- 1 root sys 256, 5 Oct 24 10:57 vol3 > > crw------- 1 root sys 256, 6 Oct 24 10:57 vol4 > > crw------- 1 root sys 256, 2 Oct 24 10:57 vol5 > > > > [/dev/zvol/rdsk/d1000pool/biscotti-vols] > > root at tofu# > > > > > > Upon bootup of the zone the problem becomes even more apparent as > volumes are not mounted in the correct location and some volumes can''t > even be mounted. > > > > > > [/] > > root at biscotti# ls -l > /vol?/vol? > > -rw------- 1 root root 10485760 Oct 23 21:38 /vol2/vol2 > > -rw------- 1 root root 10485760 Oct 23 21:38 /vol3/vol1 > > -rw------- 1 root root 10485760 Oct 23 21:38 /vol4/vol3 > > > > [/] > > root at biscotti# mount /vol1 > > mount: /dev/zvol/dsk/d1000pool/biscotti-vols/vol1 is already mounted or > /vol1 is busy > > > > If I halt the zone then rm -rf /zones/biscotti/dev/zvol and then reboot > eh zone the problems are resolved. > > > > > > [/] > > root at tofu# ls -lL /dev/zvol/rdsk/d1000pool/biscotti-vols/ > > total 0 > > crw------- 1 root sys 256, 4 Oct 24 10:57 vol1 > > crw------- 1 root sys 256, 3 Oct 24 10:57 vol2 > > crw------- 1 root sys 256, 5 Oct 24 10:57 vol3 > > crw------- 1 root sys 256, 6 Oct 24 10:57 vol4 > > crw------- 1 root sys 256, 2 Oct 24 10:57 vol5 > > > > [/] > > root at tofu# ls -lL /zones/biscotti/dev/zvol/rdsk/d1000pool/biscotti-vols/ > > total 0 > > crw------- 1 root sys 256, 4 Oct 24 11:07 vol1 > > crw------- 1 root sys 256, 3 Oct 24 11:07 vol2 > > crw------- 1 root sys 256, 5 Oct 24 11:07 vol3 > > crw------- 1 root sys 256, 6 Oct 24 11:07 vol4 > > crw------- 1 root sys 256, 2 Oct 24 11:07 vol5 > > > > I then login to the zone and everything appears fine. > > > > [/] > > root at biscotti# ls -l /vol?/vol? > > -rw------- 1 root root 10485760 Oct 23 21:38 /vol1/vol1 > > -rw------- 1 root root 10485760 Oct 23 21:38 /vol2/vol2 > > -rw------- 1 root root 10485760 Oct 23 21:38 /vol3/vol3 > > -rw------- 1 root root 10485760 Oct 23 21:38 /vol4/vol4 > =====================end summary of the > problem=========================> > -- > Jason Gallagher > Operating Systems Group Support Team > Sun Microsystems, Inc. > Working hours: Sat/Sun 6am - 6pm & Mon/Tue 8am - 5pm MNT > Phone: Call 1-800-USA4-SUN and reference your case id > e-mail: jvg at sun.com > > Case info located at http://www.sun.com/service/online > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
David I Radden
2006-Nov-01 02:27 UTC
[zfs-discuss] zfs: zvols minor #''s changing and causing probs w/ volumes
Thanks Ed. The ticket shows the customer running Solaris 10. Do you know if the fix will be incorporated in an S10 update or patch? Or possibly an S10 workaround made available? Thanks again! Dave Radden x74861 --- Edward Pilatowicz wrote On 10/31/06 18:53,:>if your running solaris 10 or an early nevada build then it''s >possible your hitting this bug (which i fixed in build 35): > > 4976415 devfsadmd for zones could be smarter when major numbers change > >if you''re running a recent nevada build then this could be a new issue. > >so what version of solaris are you running? > >thanks >ed > >On Tue, Oct 31, 2006 at 03:26:06PM -0700, Jason Gallagher - Sun Microsystems wrote: > > >>Team, >> >>**Please respond to me and my coworker listed in the Cc, since neither >>one of us are on this alias** >> >>QUICK PROBLEM DESCRIPTION: >> >>Cu created a dataset which contains all the zvols for a particular >>zone. The zone is then given access to all the zvols in the dataset >>using a match statement in the zoneconfig (see long problem description >>for details). After the initial boot of the zone everything appears >>fine and the localzone zvol dev files match the globalzone zvol dev >>files. >> >>Upon the reboot of the box (following the initial that had no problems) >>the Minor numbers of the zvols are different between the local zone and >>the Global zone and some of the volumes are not mounted in the correct >>location and some volumes can''t even be mounted. >> >> >>LONG PROBLEM DESCRIPTION: >> >>All the details are listed below, authored by the customer: >> >> >>Here is a summary of the problem we are experiencing. >> >>Just a quick little background. We had hoped to use ZFS for our >>filesystems but as a result of our backup system not fully supporting >>ZFS yet we are stuck with using UFS for now. In an effort to make >>migrating to ZFS in the future that much easier and to be able to take >>advantage of some of the other features ZFS gives us we have decided to >>use ZFS volumes and create UFS filesystems on top of them. >> >>We have created a dataset which contain all the zvols for a particular >>zone. The zone is then given access to all the zvols in the dataset >>using a match statement in the zoneconfig. >> >>[/users/mdey] >> >>mdey at tofu$ zonecfg -z biscotti info >> >>zonepath: /zones/biscotti >> >>autoboot: false >> >>pool: >> >>inherit-pkg-dir: >> >> dir: /lib >> >>inherit-pkg-dir: >> >> dir: /platform >> >>inherit-pkg-dir: >> >> dir: /sbin >> >>inherit-pkg-dir: >> >> dir: /usr >> >>inherit-pkg-dir: >> >> dir: /opt >> >>net: >> >> address: 10.1.33.91 >> >> physical: hme0 >> >>device >> >> match: /dev/zvol/rdsk/d1000pool/biscotti-vols/* >> >>device >> >> match: /dev/zvol/dsk/d1000pool/biscotti-vols/* >> >> >> >>There are 4 volumes in the dataset >> >> >> >>[/] >> >>root at tofu# zfs list -r d1000pool/biscotti-vols >> >>NAME USED AVAIL REFER MOUNTPOINT >> >>d1000pool/biscotti-vols 400M 197G 49K none >> >>d1000pool/biscotti-vols/vol1 11.2M 197G 11.2M - >> >>d1000pool/biscotti-vols/vol2 10.7M 197G 10.7M - >> >>d1000pool/biscotti-vols/vol3 11.0M 197G 11.0M - >> >>d1000pool/biscotti-vols/vol4 10.5M 197G 10.5M - >> >> >> >>The volumes are mounted in the zone via the zones vfstab >> >> >> >>/dev/zvol/dsk/d1000pool/biscotti-vols/vol1 >>/dev/zvol/rdsk/d1000pool/biscotti-vols/vol1 /vol1 ufs 2 >>yes - >> >>/dev/zvol/dsk/d1000pool/biscotti-vols/vol2 >>/dev/zvol/rdsk/d1000pool/biscotti-vols/vol2 /vol2 ufs 2 >>yes - >> >>/dev/zvol/dsk/d1000pool/biscotti-vols/vol3 >>/dev/zvol/rdsk/d1000pool/biscotti-vols/vol3 /vol3 ufs 2 >>yes - >> >>/dev/zvol/dsk/d1000pool/biscotti-vols/vol4 >>/dev/zvol/rdsk/d1000pool/biscotti-vols/vol4 /vol4 ufs 2 >>yes - >> >> >> >>After the initial boot of the zone everything appears fine and the >>localzone zvol dev files match the globalzone zvol dev files. >> >> >> >>[/] >> >>root at tofu# ls -lL /zones/biscotti/dev/zvol/rdsk/d1000pool/biscotti-vols >> >>total 0 >> >>crw------- 1 root sys 256, 2 Oct 23 21:23 vol1 >> >>crw------- 1 root sys 256, 3 Oct 23 21:23 vol2 >> >>crw------- 1 root sys 256, 4 Oct 23 21:23 vol3 >> >>crw------- 1 root sys 256, 5 Oct 23 21:23 vol4 >> >> >> >>[/] >> >>root at tofu# ls -lL /dev/zvol/rdsk/d1000pool/biscotti-vols >> >>total 0 >> >>crw------- 1 root sys 256, 2 Oct 23 21:02 vol1 >> >>crw------- 1 root sys 256, 3 Oct 23 21:02 vol2 >> >>crw------- 1 root sys 256, 4 Oct 23 21:02 vol3 >> >>crw------- 1 root sys 256, 5 Oct 23 21:02 vol4 >> >> >> >> >> >>I login to the zone and create a file in each mount to keep track of >>which volume is which. >> >> >> >>[/] >> >>root at biscotti# ls -l /vol?/vol? >> >>-rw------- 1 root root 10485760 Oct 23 21:38 /vol1/vol1 >> >>-rw------- 1 root root 10485760 Oct 23 21:38 /vol2/vol2 >> >>-rw------- 1 root root 10485760 Oct 23 21:38 /vol3/vol3 >> >>-rw------- 1 root root 10485760 Oct 23 21:38 /vol4/vol4 >> >> >> >> >> >>I then create a new volume vol5 then shutdown the zone and then reboot >>the box with an init 6 >> >> >> >>Upon the reboot of the box the Minor numbers of the zvols are different >>between the local zone and the Global zone. >> >> >> >>[/zones/biscotti/dev/zvol/rdsk/d1000pool/biscotti-vols] >> >>root at tofu# ls -lL >> >>total 0 >> >>crw------- 1 root sys 256, 2 Oct 23 21:23 vol1 >> >>crw------- 1 root sys 256, 3 Oct 23 21:23 vol2 >> >>crw------- 1 root sys 256, 4 Oct 23 21:23 vol3 >> >>crw------- 1 root sys 256, 5 Oct 23 21:23 vol4 >> >>crw------- 1 root sys 256, 6 Oct 23 21:44 vol5 >> >> >> >>[/zones/biscotti/dev/zvol/rdsk/d1000pool/biscotti-vols] >> >>root at tofu# cd /dev/zvol/rdsk/d1000pool/biscotti-vols >> >> >> >>[/dev/zvol/rdsk/d1000pool/biscotti-vols] >> >>root at tofu# ls -lL >> >>total 0 >> >>crw------- 1 root sys 256, 4 Oct 24 10:57 vol1 >> >>crw------- 1 root sys 256, 3 Oct 24 10:57 vol2 >> >>crw------- 1 root sys 256, 5 Oct 24 10:57 vol3 >> >>crw------- 1 root sys 256, 6 Oct 24 10:57 vol4 >> >>crw------- 1 root sys 256, 2 Oct 24 10:57 vol5 >> >> >> >>[/dev/zvol/rdsk/d1000pool/biscotti-vols] >> >>root at tofu# >> >> >> >> >> >>Upon bootup of the zone the problem becomes even more apparent as >>volumes are not mounted in the correct location and some volumes can''t >>even be mounted. >> >> >> >> >> >>[/] >> >>root at biscotti# ls -l >>/vol?/vol? >> >>-rw------- 1 root root 10485760 Oct 23 21:38 /vol2/vol2 >> >>-rw------- 1 root root 10485760 Oct 23 21:38 /vol3/vol1 >> >>-rw------- 1 root root 10485760 Oct 23 21:38 /vol4/vol3 >> >> >> >>[/] >> >>root at biscotti# mount /vol1 >> >>mount: /dev/zvol/dsk/d1000pool/biscotti-vols/vol1 is already mounted or >>/vol1 is busy >> >> >> >>If I halt the zone then rm -rf /zones/biscotti/dev/zvol and then reboot >>eh zone the problems are resolved. >> >> >> >> >> >>[/] >> >>root at tofu# ls -lL /dev/zvol/rdsk/d1000pool/biscotti-vols/ >> >>total 0 >> >>crw------- 1 root sys 256, 4 Oct 24 10:57 vol1 >> >>crw------- 1 root sys 256, 3 Oct 24 10:57 vol2 >> >>crw------- 1 root sys 256, 5 Oct 24 10:57 vol3 >> >>crw------- 1 root sys 256, 6 Oct 24 10:57 vol4 >> >>crw------- 1 root sys 256, 2 Oct 24 10:57 vol5 >> >> >> >>[/] >> >>root at tofu# ls -lL /zones/biscotti/dev/zvol/rdsk/d1000pool/biscotti-vols/ >> >>total 0 >> >>crw------- 1 root sys 256, 4 Oct 24 11:07 vol1 >> >>crw------- 1 root sys 256, 3 Oct 24 11:07 vol2 >> >>crw------- 1 root sys 256, 5 Oct 24 11:07 vol3 >> >>crw------- 1 root sys 256, 6 Oct 24 11:07 vol4 >> >>crw------- 1 root sys 256, 2 Oct 24 11:07 vol5 >> >> >> >>I then login to the zone and everything appears fine. >> >> >> >>[/] >> >>root at biscotti# ls -l /vol?/vol? >> >>-rw------- 1 root root 10485760 Oct 23 21:38 /vol1/vol1 >> >>-rw------- 1 root root 10485760 Oct 23 21:38 /vol2/vol2 >> >>-rw------- 1 root root 10485760 Oct 23 21:38 /vol3/vol3 >> >>-rw------- 1 root root 10485760 Oct 23 21:38 /vol4/vol4 >>=====================end summary of the >>problem=========================>> >>-- >>Jason Gallagher >>Operating Systems Group Support Team >>Sun Microsystems, Inc. >>Working hours: Sat/Sun 6am - 6pm & Mon/Tue 8am - 5pm MNT >>Phone: Call 1-800-USA4-SUN and reference your case id >>e-mail: jvg at sun.com >> >>Case info located at http://www.sun.com/service/online >> >>_______________________________________________ >>zfs-discuss mailing list >>zfs-discuss at opensolaris.org >>http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >> >>-- <http://www.sun.com> * David Radden * Technical Support Engineer - Operating Systems *Sun Microsystems * Phone: Phone 1-800-USA-4SUN **If Unavailable, option 0 for next available agent** Email: Dave.Radden at sun.com Working Hours: 0900-1800 Mon-Fri MST <http://www.sun.com/storage> -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20061031/67f70012/attachment.html>
Edward Pilatowicz
2006-Nov-01 05:03 UTC
[zfs-discuss] zfs: zvols minor #''s changing and causing probs w/ volumes
i think that this fix may be being backported as part of the brandz project backport, but i don''t think anyone is backporting it outside of that. you might want to add a new call record and open a subCR if you need this to be backported. the workaround is just what you''ve already discovered. delete any old nodes before booting the zone. ed On Tue, Oct 31, 2006 at 07:27:44PM -0700, David I Radden wrote:> Thanks Ed. The ticket shows the customer running Solaris 10. Do you > know if the fix will be incorporated in an S10 update or patch? Or > possibly an S10 workaround made available? > > Thanks again! > > > Dave Radden > x74861 > > --- > > Edward Pilatowicz wrote On 10/31/06 18:53,: > > >if your running solaris 10 or an early nevada build then it''s > >possible your hitting this bug (which i fixed in build 35): > > > > 4976415 devfsadmd for zones could be smarter when major numbers > > change > > > >if you''re running a recent nevada build then this could be a new issue. > > > >so what version of solaris are you running? > > > >thanks > >ed > > > >On Tue, Oct 31, 2006 at 03:26:06PM -0700, Jason Gallagher - Sun > >Microsystems wrote: > > > > > >>Team, > >> > >>**Please respond to me and my coworker listed in the Cc, since neither > >>one of us are on this alias** > >> > >>QUICK PROBLEM DESCRIPTION: > >> > >>Cu created a dataset which contains all the zvols for a particular > >>zone. The zone is then given access to all the zvols in the dataset > >>using a match statement in the zoneconfig (see long problem description > >>for details). After the initial boot of the zone everything appears > >>fine and the localzone zvol dev files match the globalzone zvol dev > >>files. > >> > >>Upon the reboot of the box (following the initial that had no problems) > >>the Minor numbers of the zvols are different between the local zone and > >>the Global zone and some of the volumes are not mounted in the correct > >>location and some volumes can''t even be mounted. > >> > >> > >>LONG PROBLEM DESCRIPTION: > >> > >>All the details are listed below, authored by the customer: > >> > >> > >>Here is a summary of the problem we are experiencing. > >> > >>Just a quick little background. We had hoped to use ZFS for our > >>filesystems but as a result of our backup system not fully supporting > >>ZFS yet we are stuck with using UFS for now. In an effort to make > >>migrating to ZFS in the future that much easier and to be able to take > >>advantage of some of the other features ZFS gives us we have decided to > >>use ZFS volumes and create UFS filesystems on top of them. > >> > >>We have created a dataset which contain all the zvols for a particular > >>zone. The zone is then given access to all the zvols in the dataset > >>using a match statement in the zoneconfig. > >> > >>[/users/mdey] > >> > >>mdey at tofu$ zonecfg -z biscotti info > >> > >>zonepath: /zones/biscotti > >> > >>autoboot: false > >> > >>pool: > >> > >>inherit-pkg-dir: > >> > >> dir: /lib > >> > >>inherit-pkg-dir: > >> > >> dir: /platform > >> > >>inherit-pkg-dir: > >> > >> dir: /sbin > >> > >>inherit-pkg-dir: > >> > >> dir: /usr > >> > >>inherit-pkg-dir: > >> > >> dir: /opt > >> > >>net: > >> > >> address: 10.1.33.91 > >> > >> physical: hme0 > >> > >>device > >> > >> match: /dev/zvol/rdsk/d1000pool/biscotti-vols/* > >> > >>device > >> > >> match: /dev/zvol/dsk/d1000pool/biscotti-vols/* > >> > >> > >> > >>There are 4 volumes in the dataset > >> > >> > >> > >>[/] > >> > >>root at tofu# zfs list -r d1000pool/biscotti-vols > >> > >>NAME USED AVAIL REFER MOUNTPOINT > >> > >>d1000pool/biscotti-vols 400M 197G 49K none > >> > >>d1000pool/biscotti-vols/vol1 11.2M 197G 11.2M - > >> > >>d1000pool/biscotti-vols/vol2 10.7M 197G 10.7M - > >> > >>d1000pool/biscotti-vols/vol3 11.0M 197G 11.0M - > >> > >>d1000pool/biscotti-vols/vol4 10.5M 197G 10.5M - > >> > >> > >> > >>The volumes are mounted in the zone via the zones vfstab > >> > >> > >> > >>/dev/zvol/dsk/d1000pool/biscotti-vols/vol1 > >>/dev/zvol/rdsk/d1000pool/biscotti-vols/vol1 /vol1 ufs 2 > >>yes - > >> > >>/dev/zvol/dsk/d1000pool/biscotti-vols/vol2 > >>/dev/zvol/rdsk/d1000pool/biscotti-vols/vol2 /vol2 ufs 2 > >>yes - > >> > >>/dev/zvol/dsk/d1000pool/biscotti-vols/vol3 > >>/dev/zvol/rdsk/d1000pool/biscotti-vols/vol3 /vol3 ufs 2 > >>yes - > >> > >>/dev/zvol/dsk/d1000pool/biscotti-vols/vol4 > >>/dev/zvol/rdsk/d1000pool/biscotti-vols/vol4 /vol4 ufs 2 > >>yes - > >> > >> > >> > >>After the initial boot of the zone everything appears fine and the > >>localzone zvol dev files match the globalzone zvol dev files. > >> > >> > >> > >>[/] > >> > >>root at tofu# ls -lL /zones/biscotti/dev/zvol/rdsk/d1000pool/biscotti-vols > >> > >>total 0 > >> > >>crw------- 1 root sys 256, 2 Oct 23 21:23 vol1 > >> > >>crw------- 1 root sys 256, 3 Oct 23 21:23 vol2 > >> > >>crw------- 1 root sys 256, 4 Oct 23 21:23 vol3 > >> > >>crw------- 1 root sys 256, 5 Oct 23 21:23 vol4 > >> > >> > >> > >>[/] > >> > >>root at tofu# ls -lL /dev/zvol/rdsk/d1000pool/biscotti-vols > >> > >>total 0 > >> > >>crw------- 1 root sys 256, 2 Oct 23 21:02 vol1 > >> > >>crw------- 1 root sys 256, 3 Oct 23 21:02 vol2 > >> > >>crw------- 1 root sys 256, 4 Oct 23 21:02 vol3 > >> > >>crw------- 1 root sys 256, 5 Oct 23 21:02 vol4 > >> > >> > >> > >> > >> > >>I login to the zone and create a file in each mount to keep track of > >>which volume is which. > >> > >> > >> > >>[/] > >> > >>root at biscotti# ls -l /vol?/vol? > >> > >>-rw------- 1 root root 10485760 Oct 23 21:38 /vol1/vol1 > >> > >>-rw------- 1 root root 10485760 Oct 23 21:38 /vol2/vol2 > >> > >>-rw------- 1 root root 10485760 Oct 23 21:38 /vol3/vol3 > >> > >>-rw------- 1 root root 10485760 Oct 23 21:38 /vol4/vol4 > >> > >> > >> > >> > >> > >>I then create a new volume vol5 then shutdown the zone and then reboot > >>the box with an init 6 > >> > >> > >> > >>Upon the reboot of the box the Minor numbers of the zvols are different > >>between the local zone and the Global zone. > >> > >> > >> > >>[/zones/biscotti/dev/zvol/rdsk/d1000pool/biscotti-vols] > >> > >>root at tofu# ls -lL > >> > >>total 0 > >> > >>crw------- 1 root sys 256, 2 Oct 23 21:23 vol1 > >> > >>crw------- 1 root sys 256, 3 Oct 23 21:23 vol2 > >> > >>crw------- 1 root sys 256, 4 Oct 23 21:23 vol3 > >> > >>crw------- 1 root sys 256, 5 Oct 23 21:23 vol4 > >> > >>crw------- 1 root sys 256, 6 Oct 23 21:44 vol5 > >> > >> > >> > >>[/zones/biscotti/dev/zvol/rdsk/d1000pool/biscotti-vols] > >> > >>root at tofu# cd /dev/zvol/rdsk/d1000pool/biscotti-vols > >> > >> > >> > >>[/dev/zvol/rdsk/d1000pool/biscotti-vols] > >> > >>root at tofu# ls -lL > >> > >>total 0 > >> > >>crw------- 1 root sys 256, 4 Oct 24 10:57 vol1 > >> > >>crw------- 1 root sys 256, 3 Oct 24 10:57 vol2 > >> > >>crw------- 1 root sys 256, 5 Oct 24 10:57 vol3 > >> > >>crw------- 1 root sys 256, 6 Oct 24 10:57 vol4 > >> > >>crw------- 1 root sys 256, 2 Oct 24 10:57 vol5 > >> > >> > >> > >>[/dev/zvol/rdsk/d1000pool/biscotti-vols] > >> > >>root at tofu# > >> > >> > >> > >> > >> > >>Upon bootup of the zone the problem becomes even more apparent as > >>volumes are not mounted in the correct location and some volumes can''t > >>even be mounted. > >> > >> > >> > >> > >> > >>[/] > >> > >>root at biscotti# ls -l > >>/vol?/vol? > >> > >>-rw------- 1 root root 10485760 Oct 23 21:38 /vol2/vol2 > >> > >>-rw------- 1 root root 10485760 Oct 23 21:38 /vol3/vol1 > >> > >>-rw------- 1 root root 10485760 Oct 23 21:38 /vol4/vol3 > >> > >> > >> > >>[/] > >> > >>root at biscotti# mount /vol1 > >> > >>mount: /dev/zvol/dsk/d1000pool/biscotti-vols/vol1 is already mounted or > >>/vol1 is busy > >> > >> > >> > >>If I halt the zone then rm -rf /zones/biscotti/dev/zvol and then reboot > >>eh zone the problems are resolved. > >> > >> > >> > >> > >> > >>[/] > >> > >>root at tofu# ls -lL /dev/zvol/rdsk/d1000pool/biscotti-vols/ > >> > >>total 0 > >> > >>crw------- 1 root sys 256, 4 Oct 24 10:57 vol1 > >> > >>crw------- 1 root sys 256, 3 Oct 24 10:57 vol2 > >> > >>crw------- 1 root sys 256, 5 Oct 24 10:57 vol3 > >> > >>crw------- 1 root sys 256, 6 Oct 24 10:57 vol4 > >> > >>crw------- 1 root sys 256, 2 Oct 24 10:57 vol5 > >> > >> > >> > >>[/] > >> > >>root at tofu# ls -lL /zones/biscotti/dev/zvol/rdsk/d1000pool/biscotti-vols/ > >> > >>total 0 > >> > >>crw------- 1 root sys 256, 4 Oct 24 11:07 vol1 > >> > >>crw------- 1 root sys 256, 3 Oct 24 11:07 vol2 > >> > >>crw------- 1 root sys 256, 5 Oct 24 11:07 vol3 > >> > >>crw------- 1 root sys 256, 6 Oct 24 11:07 vol4 > >> > >>crw------- 1 root sys 256, 2 Oct 24 11:07 vol5 > >> > >> > >> > >>I then login to the zone and everything appears fine. > >> > >> > >> > >>[/] > >> > >>root at biscotti# ls -l /vol?/vol? > >> > >>-rw------- 1 root root 10485760 Oct 23 21:38 /vol1/vol1 > >> > >>-rw------- 1 root root 10485760 Oct 23 21:38 /vol2/vol2 > >> > >>-rw------- 1 root root 10485760 Oct 23 21:38 /vol3/vol3 > >> > >>-rw------- 1 root root 10485760 Oct 23 21:38 /vol4/vol4 > >>=====================end summary of the > >>problem=========================> >> > >>-- > >>Jason Gallagher > >>Operating Systems Group Support Team > >>Sun Microsystems, Inc. > >>Working hours: Sat/Sun 6am - 6pm & Mon/Tue 8am - 5pm MNT > >>Phone: Call 1-800-USA4-SUN and reference your case id > >>e-mail: jvg at sun.com > >> > >>Case info located at http://www.sun.com/service/online > >> > >>_______________________________________________ > >>zfs-discuss mailing list > >>zfs-discuss at opensolaris.org > >>http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > >> > >> > > -- > <http://www.sun.com> * David Radden * > Technical Support Engineer - Operating Systems > *Sun Microsystems * > Phone: Phone 1-800-USA-4SUN > **If Unavailable, option 0 for next available agent** > Email: Dave.Radden at sun.com > Working Hours: 0900-1800 Mon-Fri MST > <http://www.sun.com/storage> >