Drew Tomlinson
2004-Dec-29 16:22 UTC
Multiple Vinum Problems Since Upgrade From 4.9 To 4.10
Since an upgrade from 4.9 to 4.10, I've had problems with vinum. The basic
problem is that upon reboot, two of my vinum drives show up as
"referenced" and
thus create the associated chaos. I've tried many things and fiddled
around
quite a bit so I can't say exactly what I've done. I can include all of
the
entries in the history file since Oct. 31 if that's a help but it would
be a
long list.
So prior to digging that deep, I will describe where I stand currently and
where I want to finish. Currently, I have one vinum volume that I use for
/usr. Fortunately this volume is up and running or I would really be in a
mess. Here's the 'vinum list' output in this state:
blacklamb# vinum
vinum -> list
2 drives:
D disk1 State: up Device /dev/da0s1h Avail:
0/8383 MB (0%)
D disk2 State: up Device /dev/da1s1h Avail:
0/8383 MB (0%)
1 volumes:
V usr State: up Plexes: 1 Size: 16 GB
1 plexes:
P usr.p0 S State: up Subdisks: 2 Size: 16 GB
2 subdisks:
S usr.p0.s0 State: up PO: 0 B Size: 8383 MB
S usr.p0.s1 State: up PO: 256 kB Size: 8383 MB
I want to add another volume and mount it on /ftp. After creating the
volume,
vinum sees it and it appears OK as indicated in this output:
vinum -> list
5 drives:
D disk1 State: up Device /dev/da0s1h Avail:
0/8383 MB (0%)
D disk2 State: up Device /dev/da1s1h Avail:
0/8383 MB (0%)
D ftp1 State: up Device /dev/ad0s1h Avail:
0/76319 MB (0%)
D ftp2 State: up Device /dev/ad1s1h Avail:
0/76319 MB (0%)
D ftp3 State: up Device /dev/da3s1h Avail:
0/114473 MB (0%)
2 volumes:
V usr State: up Plexes: 1 Size: 16 GB
V ftp State: up Plexes: 1 Size: 260 GB
2 plexes:
P usr.p0 S State: up Subdisks: 2 Size: 16 GB
P ftp.p0 C State: up Subdisks: 3 Size: 260 GB
5 subdisks:
S usr.p0.s0 State: up PO: 0 B Size: 8383 MB
S usr.p0.s1 State: up PO: 256 kB Size: 8383 MB
S ftp.p0.s0 State: up PO: 0 B Size: 74 GB
S ftp.p0.s1 State: up PO: 74 GB Size: 74 GB
S ftp.p0.s2 State: up PO: 149 GB Size: 111 GB
Next I 'quit' the vinum command line and fsck my newly created volume.
It
finishes successfully and I mount it:
blacklamb# df
Filesystem 1K-blocks Used Avail Capacity Mounted on
/dev/da0s1a 302350 102088 176074 37% /
/dev/vinum/usr 16639674 5433762 9874739 35% /usr
procfs 4 4 0 100% /proc
/dev/vinum/ftp 265119539 119729707 132133856 48% /ftp
I recheck with the 'vinum list' command and the output is the same as
above.
Now I cross my fingers and reboot to see if the volume comes up properly on
startup. However ftp3 (/dev/da3s1h) comes up 'referenced' and causes
problems.
Mounting root from ufs:/dev/da0s1a
dumpon: crash dumps to /dev/da1s1b (13, 131081)
vinum: loaded
vinum: reading configuration from /dev/ad1s1h
vinum: ftp.p0.s2 is crashed
vinum: ftp.p0 is corrupt
vinum: updating configuration from /dev/ad0s1h
vinum: updating configuration from /dev/da1s1h
vinum: updating configuration from /dev/da0s1h
vinum: /dev is mounted read-only, not rebuilding /dev/vinum
Warning: defective objects
D ftp3 State: referenced Device Avail: 0/0 MB
P ftp.p0 C State: corrupt Subdisks: 3 Size: 260 GB
S ftp.p0.s2 State: crashed PO: 149 GB Size: 111 GB
swapon: adding /dev/da1s1b as swap device
Automatic boot in progress...
/dev/da0s1a: FILESYSTEM CLEAN; SKIPPING CHECKS
/dev/da0s1a: clean, 100130 free (938 frags, 12399 blocks, 0.6%
fragmentation)
/dev/vinum/ftp: CANNOT READ: BLK 546979872
/dev/vinum/ftp: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
/dev/vinum/ftp: CANNOT WRITE: BLK 2800
/dev/vinum/ftp: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
/dev/vinum/usr: FILESYSTEM CLEAN; SKIPPING CHECKS
/dev/vinum/usr: clean, 11205903 free (152463 frags, 1381680 blocks, 0.9%
fragmentation)
THE FOLLOWING FILE SYSTEM HAD AN UNEXPECTED INCONSISTENCY:
/dev/vinum/ftp (/ftp)
File system preen failed, trying fsck -y . . .
** /dev/da0s1a
** Last Mounted on /
** Root file system
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
2053 files, 51045 used, 100130 free (938 frags, 12399 blocks, 0.6%
fragmentation)
** /dev/vinum/usr
** Last Mounted on /usr
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
401072 files, 5433771 used, 11205903 free (152463 frags, 1381680 blocks,
0.9% fragmentation)
** /dev/vinum/ftp
CANNOT READ: BLK 546979872
UNEXPECTED SOFT UPDATE INCONSISTENCY
CONTINUE? yes
THE FOLLOWING DISK SECTORS COULD NOT BE READ: 546979872, 546979873,
546979874, 546979875,
/dev/vinum/ftp: CANNOT FIGURE OUT FILE SYSTEM PARTITION
UNEXPECTED SOFT UPDATE INCONSISTENCY
WARNING: R/W mount of /ftp denied. Filesystem is not clean - run fsck
mount: /dev/vinum/ftp: Operation not permitted
Mounting /etc/fstab filesystems failed, startup aborted
Enter full pathname of shell or RETURN for /bin/sh:
I enter the shell and run 'vinum list':
vinum -> list
4 drives:
D ftp1 State: up Device /dev/ad0s1h Avail:
0/76319 MB (0%)
D ftp2 State: up Device /dev/ad1s1h Avail:
0/76319 MB (0%)
D disk1 State: up Device /dev/da0s1h Avail:
0/8383 MB (0%)
D disk2 State: up Device /dev/da1s1h Avail:
0/8383 MB (0%)
D ftp3 State: referenced Device Avail: 0/0 MB
2 volumes:
V usr State: up Plexes: 1 Size: 16 GB
V ftp State: up Plexes: 1 Size: 260 GB
2 plexes:
P usr.p0 S State: up Subdisks: 2 Size: 16 GB
P ftp.p0 C State: corrupt Subdisks: 3 Size: 260 GB
5 subdisks:
S usr.p0.s0 State: up PO: 0 B Size: 8383 MB
S usr.p0.s1 State: up PO: 256 kB Size: 8383 MB
S ftp.p0.s0 State: up PO: 0 B Size: 74 GB
S ftp.p0.s1 State: up PO: 74 GB Size: 74 GB
S ftp.p0.s2 State: crashed PO: 149 GB Size: 111 GB
Now I delete all references to the ftp volume by running these commands:
vinum -> rm -f ftp.p0.s2
vinum: removing ftp.p0.s2
Correcting length of ftp.p0: was 547043829, is 312602446
vinum: ftp.p0 is up
vinum -> rm -f ftp.p0.s1
vinum: removing ftp.p0.s1
Correcting length of ftp.p0: was 312602446, is 156301223
vinum -> rm -f ftp.p0.s0
vinum: removing ftp.p0.s0
vinum -> rm -f ftp.p0 vinum: removing ftp.p0
vinum: ftp is down
vinum -> rm -f ftp vinum: removing ftp
vinum -> rm -f ftp3
vinum -> rm -f ftp2
vinum -> rm -f ftp1
Yet the 'reference to ftp3 remains.
vinum -> list
1 drives:
D disk1 State: up Device /dev/da0s1h Avail:
0/8383 MB (0%)
D disk2 State: up Device /dev/da1s1h Avail:
0/8383 MB (0%)
D ftp3 State: referenced Device Avail: 0/0 MB
1 volumes:
V usr State: up Plexes: 1 Size: 16 GB
1 plexes:
P usr.p0 S State: up Subdisks: 2 Size: 16 GB
2 subdisks:
S usr.p0.s0 State: up PO: 0 B Size: 8383 MB
S usr.p0.s1 State: up PO: 256 kB Size: 8383 MB
Why does the list output show "1 drives:" when there are three listed
and two
working? Anyway, I've been here before and I quit vinum. Then I issue
these
commands:
# df
Filesystem 512-blocks Used Avail Capacity Mounted on
/dev/da0s1a 604700 204180 352144 37% /
/dev/vinum/usr 33279348 10867544 19749458 35% /usr
procfs 8 8 0 100% /proc
# umount /usr
# vinum stop
vinum: unloaded
vinum unloaded
# vinum start
vinum: loaded
vinum: reading configuration from /dev/da1s1h
vinum: updating configuration from /dev/da0s1h
Then I issue a 'vinum list' and all seems well.
# vinum
vinum -> list
2 drives:
D disk1 State: up Device /dev/da0s1h Avail:
0/8383 MB (0%)
D disk2 State: up Device /dev/da1s1h Avail:
0/8383 MB (0%)
1 volumes:
V usr State: up Plexes: 1 Size: 16 GB
1 plexes:
P usr.p0 S State: up Subdisks: 2 Size: 16 GB
2 subdisks:
S usr.p0.s0 State: up PO: 0 B Size: 8383 MB
S usr.p0.s1 State: up PO: 256 kB Size: 8383 MB
vinum ->
Since everything appears correct, this seems like a good place to do 'vinum
saveconfig'. Next I edit /etc/fstab so only /ftp is not mounted and
reboot.
The system reboots fine and the 'vinum list' is as it is immediately
above.
Running in full production mode, I create the ftp volume again:
vinum -> create -f vinum_ftp.conf
Dec 24 16:37:45 blacklamb /kernel: vinum: drive ftp1 is up
Dec 24 16:37:45 blacklamb /kernel: vinum: drive ftp2 is up
Dec 24 16:37:45 blacklamb /kernel: vinum: drive ftp3 is up
5 drives:
D disk1 State: up Device /dev/da0s1h Avail:
0/8383 MB (0%)
D disk2 State: up Device /dev/da1s1h Avail:
0/8383 MB (0%)
D ftp1 State: up Device /dev/ad0s1h Avail:
0/76319 MB (0%)
D ftp2 State: up Device /dev/ad1s1h Avail:
0/76319 MB (0%)
D ftp3 State: up Device /dev/da3s1h Avail:
0/114473 MB (0%)
2 volumes:
V usr State: up Plexes: 1 Size: 16 GB
V ftp State: up Plexes: 1 Size: 260 GB
2 plexes:
P usr.p0 S State: up Subdisks: 2 Size: 16 GB
P ftp.p0 C State: up Subdisks: 3 Size: 260 GB
5 subdisks:
S usr.p0.s0 State: up PO: 0 B Size: 8383 MB
S usr.p0.s1 State: up PO: 256 kB Size: 8383 MB
S ftp.p0.s0 State: up PO: 0 B Size: 74 GB
S ftp.p0.s1 State: up PO: 74 GB Size: 74 GB
S ftp.p0.s2 State: up PO: 149 GB Size: 111 GB
Dec 24 16:37:45 blacklamb /kernel: vinum: ftp.p0.s0 is up
Dec 24 16:37:45 blacklamb /kernel: vinum: ftp.p0.s1 is up
Dec 24 16:37:45 blacklamb /kernel: vinum: ftp.p0.s2 is up
Dec 24 16:37:45 blacklamb /kernel: vinum: ftp.p0 is up
Dec 24 16:37:45 blacklamb /kernel: vinum: ftp is up
After running fsck, I can mount and access the volume. The system
continues to
run fine until the next reboot where the same problems start all over
again. I
also have the same problem with another volume I wish to run called
"backup".
How can I fix my vinum volumes so they survive reboots?
Thanks for your help and Happy Holidays!
Drew