freebsd stable - Dec 2020 - 12.2-STABLE: Something has broken the bash prompt

I updated my 12.2-STABLE system from circa December 10 to the last
stable/12 SVN commit and now my bash prompt is broken.

I use a prompt with properly delineated non-printing characters:

PS1="\[$(tput so)\]\u@\h\[$(tput se)\][\w] "

Suddenly bash is very confused about the size of the prompt.  To
reproduce the problem, set PS1 as above, type a few characters, hit
^A to go to the beginning of the line, type some more, see the mess.

A prompt without non-printing characters is perfectly fine.
The problem is evident with LC_CTYPE=C.UTF-8, but LC_CTYPE=C is
fine.

Yes, there have been recent updates to the bash port.  But I checked
several versions, and 5.0.18, 5.1, and 5.1.4 are equally affected.
And I was already running 5.1 before the problem appeared after I
updated base.

So I have reason to suspect that the breakage originates in base.
I've looked over the stable/12 commits starting December 10 and I
am very suspicious of yuripv's locale changes, in particular
"update wcwidth data from utf8proc" looks like a potential culprit.

Any insights?

-- 
Christian "naddy" Weisgerber                          naddy at
mips.inka.de

Christian Weisgerber wrote:
> I updated my 12.2-STABLE system from circa December 10 to the last
> stable/12 SVN commit and now my bash prompt is broken.
> 
> I use a prompt with properly delineated non-printing characters:
> 
> PS1="\[$(tput so)\]\u@\h\[$(tput se)\][\w] "
> 
> Suddenly bash is very confused about the size of the prompt.  To
> reproduce the problem, set PS1 as above, type a few characters, hit
> ^A to go to the beginning of the line, type some more, see the mess.
> 
> A prompt without non-printing characters is perfectly fine.
> The problem is evident with LC_CTYPE=C.UTF-8, but LC_CTYPE=C is
> fine.
> 
> Yes, there have been recent updates to the bash port.  But I checked
> several versions, and 5.0.18, 5.1, and 5.1.4 are equally affected.
> And I was already running 5.1 before the problem appeared after I
> updated base.
> 
> So I have reason to suspect that the breakage originates in base.
> I've looked over the stable/12 commits starting December 10 and I
> am very suspicious of yuripv's locale changes, in particular
> "update wcwidth data from utf8proc" looks like a potential
culprit.
> 
> Any insights?
Correct, my mistake -- I failed to see the (not so) subtle difference 
between values returned by wcwidth() and utf8proc_charwidth() for 
non-printable characters.  Will fix once the repos are back; for the 
moment you can drop the following file to 
tools/tools/locale/etc/final-maps/ and rebuild/reinstall ctype data in 
share/ctypedef/ (don't forget `make clean` there first, known issue):

https://people.freebsd.org/~yuripv/widths.txt