R devel - May 2022 - Regression in 4.2.0: unable to type special characters in Windows RGui

Hi Tomas,

On 09-05-2022 at 10:23, Tomas Kalibera wrote:
> 
> On 5/7/22 15:14, Cesko Voeten wrote:
>> Hi,
>>
>> Since upgrading to 4.2.0, I can't type special characters in RGui
anymore. Specifically affected are at least "'~^ which, on my
US-International keyboard layout, all rely on dead keys. This was not a problem
in R <= 4.1.3, although I can't say whether the reason is any change in R
itself, or the fact that 4.2.0 is the first version built against ucrt. OS is an
up-to-date Windows 10, language is Dutch. I tried toggling the
'experimental' Unicode support in Windows's Country & Region
settings, but it makes no difference. Is this a known regression?
> 
> Thanks, could you please try the latest R-patched snapshot build and test
it there? If there is still any problem, please report.
No problem, happy to test! First of all, I'm happy to say it works and I can
type deadkey characters again, but there are some issues and discrepancies
compared to other Windows apps:

  - Some accents don't work. For instance, pressing 'a produces ?, but
pressing 'a produces 'a rather than ?. From my testing, `^~ seem to work
as expected, but '" don't.
  - Nonexistent deadkey combinations are handled differently from native
Windows. For example, typing ~^ should output ~^, but produces nothing. Typing
~^a should output ~^a, but actually yields ~?. So it seems that the first ~ is
correctly translated to the tilde symbol, but simultaneously the second ^ is
being treated as a new deadkey press, which other Windows programs don't do
in this case.
  - Related to the above two issues, '& should produce '&, while
it only gives &. This again affects only '" --- something like
^& correctly produces ^&.

Also, I'm noticing only now that selecting typed text in RGui and pressing
Ctrl deletes that text, so trying to Ctrl+C text that I typed is kind of
awkward. But this might simply be a feature that I wasn't aware of before.

Thanks for your efforts! I hope these observations help.

Cesko
> 
> I've fixed handling of the dead keys in Rgui/GraphApp last week. It was
a long standing bug (lack of support) in GraphApp Unicode windows, which are
used when R is running in a multi-byte locale. As R 4.2 switched to UTF-8, this
code path started to be used on systems where it wasn't before, but sadly
this hasn't been discovered by testing before the release.
> 
> Technically, there were two bugs: one that dead keys could not be printed
in non-combining form in principle, and another that some accents were not
recognized. The former should be fixed fully. For the latter, I've added
support for a number of accent keys and I can add more if needed (and if there
is anyone who can test that).
> 
> Please test, in principle, there may be also other issues in the Unicode
windows not reported before, and it would be nice to get these things fixed at
least for R 4.1.
> 
> Thanks
> Tomas
> 
> 
> 
> 
> 
>>
>> Cesko Voeten
>>
>> ______________________________________________
>> R-devel at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-devel

On 5/9/22 19:44, Cesko Voeten wrote:
> Hi Tomas,
>
> On 09-05-2022 at 10:23, Tomas Kalibera wrote:
>>
>> On 5/7/22 15:14, Cesko Voeten wrote:
>>> Hi,
>>>
>>> Since upgrading to 4.2.0, I can't type special characters in
RGui
>>> anymore. Specifically affected are at least "'~^ which, on
my
>>> US-International keyboard layout, all rely on dead keys. This was 
>>> not a problem in R <= 4.1.3, although I can't say whether
the reason
>>> is any change in R itself, or the fact that 4.2.0 is the first 
>>> version built against ucrt. OS is an up-to-date Windows 10,
language
>>> is Dutch. I tried toggling the 'experimental' Unicode
support in
>>> Windows's Country & Region settings, but it makes no
difference. Is
>>> this a known regression?
>>
>> Thanks, could you please try the latest R-patched snapshot build and 
>> test it there? If there is still any problem, please report.
Hi Cesko,

thanks a lot for the testing.
>
> No problem, happy to test! First of all, I'm happy to say it works and 
> I can type deadkey characters again, but there are some issues and 
> discrepancies compared to other Windows apps:
>
> ?- Some accents don't work. For instance, pressing 'a produces ?,
but
> pressing 'a produces 'a rather than ?. From my testing, `^~ seem to
> work as expected, but '" don't.
I could reproduce this on the US International keyboard and it is fixed 
now. There may be missing support for more accents, so if you or anyone 
else find about them, please let me know.
> ?- Nonexistent deadkey combinations are handled differently from 
> native Windows. For example, typing ~^ should output ~^, but produces 
> nothing. Typing ~^a should output ~^a, but actually yields ~?. So it 
> seems that the first ~ is correctly translated to the tilde symbol, 
> but simultaneously the second ^ is being treated as a new deadkey 
> press, which other Windows programs don't do in this case.
Thanks, fixed now. It was another bug in the old code, plus a
"feature"
I added that by pressing e.g. the dead-key apostrophe twice you get it 
printed only once (this is what Linux does and I find it more natural). 
But I made it now to work as usual on Windows.
> ?- Related to the above two issues, '& should produce '&,
while it
> only gives &. This again affects only '" --- something like
^&
> correctly produces ^&.
It should be fixed by now as well.

The fixes are in 82332 (R-patched) and 82331 (R-devel). If you could 
test again after the snapshot build gets there, that would be great. It 
is easy to miss something.

When testing now, I've been using http://kbdlayout.info/ as a reference 
for several keyboard layouts on Windows, to find how to type different 
dead-keys. That can give you also names of the deadkeys, which can be 
easily communicated without having to copy-paste from the Rgui window. 
On some layouts, additional deadkeys are available with right Alt (AltGr).
>
> Also, I'm noticing only now that selecting typed text in RGui and 
> pressing Ctrl deletes that text, so trying to Ctrl+C text that I typed 
> is kind of awkward. But this might simply be a feature that I wasn't 
> aware of before.
Interesting. It isn't happening on my system, but I can't tell whether 
it is a "feature" under some circumstances. Maybe you can try to find 
out more about this: whether it is repeatable, whether it depends on 
some setting, whether it was happening also in R 4.1, etc.
> Thanks for your efforts! I hope these observations help.
They did, thanks!

Tomas
>
> Cesko
>
>>
>> I've fixed handling of the dead keys in Rgui/GraphApp last week. It
>> was a long standing bug (lack of support) in GraphApp Unicode 
>> windows, which are used when R is running in a multi-byte locale. As 
>> R 4.2 switched to UTF-8, this code path started to be used on systems 
>> where it wasn't before, but sadly this hasn't been discovered
by
>> testing before the release.
>>
>> Technically, there were two bugs: one that dead keys could not be 
>> printed in non-combining form in principle, and another that some 
>> accents were not recognized. The former should be fixed fully. For 
>> the latter, I've added support for a number of accent keys and I
can
>> add more if needed (and if there is anyone who can test that).
>>
>> Please test, in principle, there may be also other issues in the 
>> Unicode windows not reported before, and it would be nice to get 
>> these things fixed at least for R 4.1.
>>
>> Thanks
>> Tomas
>>
>>
>>
>>
>>
>>>
>>> Cesko Voeten
>>>
>>> ______________________________________________
>>> R-devel at r-project.org mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-devel