freebsd stable - Apr 2021 - Vinum deprecation for FreeBSD 14 - are there any remaining Vinum users?

Ed,
     On Thu, 25 Mar 2021 13:25:44 -0400 Ed Maste <emaste at freebsd.org>
wrote:
>Vinum is a Logical Volume Manager that was introduced in FreeBSD 3.0,
>and for FreeBSD 5 was ported to geom(4) as gvinum. gvinum has had no
>specific development at least as far back as 2010 and it is not clear
>how well it works today. There are open PRs with reports of panics
>upon removing disks, etc. And, it also imposes an ongoing cost as it
     First off, the "port" to geom(4) was incomplete in that gvinum is
somehow not restricted to the geom(4) device nodes presented to it, but
instead always grabs the entire physical device and node to do its own
label processing.
     Second, gvinum is completely incompatible with GPT partitioning
because, regardless of the device nodes given it to use, it always writes
and reads its own label to and from the ends of the physical drives.
That means it overwrites the GPT secondary partition table with its own
labels, which soon causes error/warning messages from the kernel about
a damaged/missing secondary partition table and recommending a recovery
of that damaged/missing partition table.  Doing the recovery then will
overwrite gvinum's labels, which is likely to cause a kernel panic or
worse.
     My memory on gvinum's compatibility with glabel(8) labels is fuzzy
at the present remove, but I seem to recall having encountered problems
there, too.  This is not unique, unfortunately, to gvinum(8).  For
example, using glabel(8) to label swap partitions, as well as
bsdlabel(8)ed partitions, can lead to many unexpected problems.  Such
inconsistencies should be researched and fixed.
     GPT labels allow a partition type of "freebsd-vinum".  I did not
try
to play with that one, but I suspect that it would not work correctly
because gvinum is somehow not limited to the GEOM device node for a
partition.  However, if you decide to keep gvinum(8) for some reason,
then this matter should be checked out in detail and its inconsistencies
fixed..
     At least w.r.t. gvinum's raid5, I can attest that the kernel panics
are real.  Before settling on ZFS raidz2 for my largest storage pool, I
experimented with gstripe(8), gmirror(8), graid3(8), and graid5(8) (from
sysutils/graid5).  All worked reasonably well, except for one operation,
namely, "stop".  Most/all such devices cannot actually be stopped
because
a stopped device does not *stay* stopped.  As soon as the GEOM device
node is destroyed, all disks are retasted, their labels, if any, are
recognized, and their corresponding device nodes are recreated and placed
back on line. :-(  All of this happens too quickly for even a series of
commands entered on one line to be able to unload the kernel module for
the device node type in question, so there is no practical way to stop
such a device once it has been started.  Because gvinum's raid5 was
always unbearably slow and also subject to kernel panics, I soon excluded
it from further consideration.  GEOM is one of the brightest gems of
modern FreeBSD design.  GEOM's native functions should not be corrupted
or ignored as a result of a botched attempt to "modernize" an old
monstrosity like gvinum, which was originally written for a system that
lacked GEOM and has not fit well into a modern system that *has* GEOM,
not to mention GPT partitioning.
      All of these specific, native GEOM second-level devices otherwise
work pretty much as advertised.  graid5(8), however, was recently marked
as deprecated, which is a real shame.  I would vote for finishing its man
page, which is very incomplete, and for adding a subcommand to do some
sort of scrub procedure like many hardware RAID5 controllers do.  There
are perfectly valid reasons to use these devices in some situations
instead of ZFS, e.g., better performance for temporary/disposable data,
especially for situations involving millions of very short files like
ccache(1) directory trees, portmaster(8)'s $WRKDIRPREFIX, and likely
others.  gvinum(8) appears to have been broken in several ways since
FreeBSD 5.0, is unmaintained as you wrote, and should be deprecated and
eliminated for the reasons given above.  The simple GEOM devices provide
much the same flexibility that gvinum was intended to provide without
the need to learn gvinum's peculiar configuration method.  Once one
understands how GEOM devices work and can be stacked, they are generally
very simple to use in contrast to gvinum, which remains broken in
multiple ways.
>must be updated when other work is done (such as the recent MAXPHYS
>work). I suspect that by now all users have migrated to either
>graid(8) or ZFS.
     graid(8) is not always a good option.  If you read its man page,
you will see that RAID5 is usually only supported as read-only devices,
where it is supported at all.  This can be helpful for recovering data
from a proprietary RAID device, but is not generally useful for actively
used and updated data.  IOW, it can be helpful in a potentially large
number of situations for some users, especially for data recovery, but
that is a different purpose from that for which the other GEOM RAID
commands were written to serve.  Again, IMO, gstripe(8), gmirror(8),
graid3(8), and graid5(8) from sysutils/graid5 should be improved, not
eliminated.  Anything gvinum(8) is *supposed* to be able to do, but often
cannot do without violating other system functions, can be done well by
these other GEOM commands.
     A special note is needed here regarding gcache(8) and graid3(8).  The
documentation of gcache parameters for sector size for physical devices
and gcache logical devices is very unclear, such that a user must have the
device nodes and space on them available to create test cases and do so,
whereas a properly documented gcache(8) would obviate the need to set up
such experiments.  There is similar lack of clarity in various size
specifications for blocks, sectors, records, etc. in many of the man pages
for native GEOM commands.
     While you are looking into this situation, please also consider
that deprecation and elimination of the veritably ancient ccd and
ccdconfig are long overdue.  Even the man pages state that these device
nodes are *not* robust and data can be easily lost.  The fact that NetBSD
separately maintains some version of ccd and ccdconfig should be
considered irrelevant in deciding to deprecate and/or eliminate the
supporting code from the source tree.
>
>I plan to add a deprecation notice after a short discussion period,
>assuming no reasonable justification is made to retain it. The notice
>would suggest graid and ZFS as alternatives, and would be merged in
>advance of FreeBSD 13.1. Then, gvinum would be removed in advance of
>FreeBSD 14.0.
>
>Please follow up if you have experience or input on vinum in FreeBSD,
>including past use but especially if you are still using it today and
>expect to continue doing so.
     Given the panics and other problems with gvinum(8), I cannot
recommend that anyone use it.  After experimenting with it, I ended up
using ZFS, gmirror(8), graid5(8), and gconcat(8) to meet my needs.
In sum, I recommend maintaining and enhancing to some degree the native
GEOM support and letting unfinished and/or unmaintained support for
gvinum(8), ccd(8) (and ccd(4)), and ccdconfig(8) be abandoned.  Please
reverse the deprecation for sysutils/graid5, which actually works as
specified, and complete its man page.  Please also add a scrub function
to it.  RAID5, whether by hardware or by software, has known limitations,
but for some purposes it is not only adequate, but is a better choice
than ZFS.  GEOM-based RAID support is also much for versatile and
flexible than hardware RAID, so let's keep it available as an option.
At the same time, the poorly supported, obsolete, and incompatible-with-
other-system-components stuff should rightly be eliminated from the source
tree.  The few known bugs with the native GEOM commands can and should be
fixed.


                                  Scott Bennett, Comm. ASMELG, CFIAG
**********************************************************************
* Internet:   bennett at sdf.org   *xor*   bennett at freeshell.org  *
*--------------------------------------------------------------------*
* "A well regulated and disciplined militia, is at all times a good  *
* objection to the introduction of that bane of all free governments *
* -- a standing army."                                               *
*    -- Gov. John Hancock, New York Journal, 28 January 1790         *
**********************************************************************

07.04.2021 12:49, Scott Bennett via freebsd-stable wrote:
>      At least w.r.t. gvinum's raid5, I can attest that the kernel
panics
> are real.  Before settling on ZFS raidz2 for my largest storage pool, I
> experimented with gstripe(8), gmirror(8), graid3(8), and graid5(8) (from
> sysutils/graid5).  All worked reasonably well, except for one operation,
> namely, "stop".  Most/all such devices cannot actually be stopped
because
> a stopped device does not *stay* stopped.  As soon as the GEOM device
> node is destroyed, all disks are retasted, their labels, if any, are
> recognized, and their corresponding device nodes are recreated and placed
> back on line. :-(  All of this happens too quickly for even a series of
> commands entered on one line to be able to unload the kernel module for
> the device node type in question, so there is no practical way to stop
> such a device once it has been started.
In fact, you can disable re-tasting with sysctl kern.geom.notaste=1,
stop an GEOM, clear its lables and re-enable tasting setting kern.geom.notaste=0
back.
>      A special note is needed here regarding gcache(8) and graid3(8).  The
> documentation of gcache parameters for sector size for physical devices
> and gcache logical devices is very unclear, such that a user must have the
> device nodes and space on them available to create test cases and do so,
> whereas a properly documented gcache(8) would obviate the need to set up
> such experiments.  There is similar lack of clarity in various size
> specifications for blocks, sectors, records, etc. in many of the man pages
> for native GEOM commands.
I found gcache(8) very nice at first, it really boosts UFS performance provided
you have extra RAM to dedicate to its cache. gcache can be stacked with gmirror
etc.
but I found it guilty to some obscure UFS-related panics. It seems there were
races or something.
No data loss, though as it is intended to be transparent for writing.

I was forced to stop using gcache for sake of stability and it's a shame.
For example, dump(8) speed-up due to gcache was 2x at least with big cache
comparing to dump -C32 without gcache.