llvm dev - Oct 2021 - [llvm-lit] Old echo causes test failure on windows

At a minimum, I think we should stop using (and recommending) the bit-rotting
GnuWin32 tools, and instead use the tools that come with git-for-Windows.  Lit
currently checks for 5 tools (cmp, diff, echo, grep, sed) and all of these are
in the git-Windows set.  This will reduce the set of packages we depend on, even
if it doesn't reduce the set of actual tools we depend on.
--paulr

From: llvm-dev <llvm-dev-bounces at lists.llvm.org> On Behalf Of via
llvm-dev
Sent: Tuesday, October 5, 2021 10:30 AM
To: llvm-dev at lists.llvm.org
Subject: [llvm-dev] [llvm-lit] Old echo causes test failure on windows

Hi llvm-dev,

I wanted to bring discussion from D110986 to a wider audience.

After D110986 added a unicode character to the test,
llvm\test\tools\llvm-cxxfilt\delimiters.test
started failing in our (Sony's) downstream Windows CI build (and locally for
me) because the
unicode character was being converted to '?' at some stage in the test.
The upstream Windows bots
seemed to have no such trouble. I added a workaround in D111072 that we landed
because a Chromium
bot was running into the same issue.

Digging a little deeper it turns out that the echo that lit was picking up for
me (`echo (GNU
coreutils) 5.3.0` from GnuWin32) converts unicode characters to '?'.

llvm-lit has its own built in echo which handles unicode without an issue. This
is used in the workaround
D111072; lit's echo is used iff the echo command isn't used in a
pipeline.

N.B. `echo (GNU coreutils) 8.32` shipped with git for Windows also appears to
handle unicode without
an issue.

I have a few questions:
1) Is there a reason we cannot always use lit's echo, as Hans suggests in
D111072?
2) If we can't always use lit's echo, maybe we could add an
error/warning if a "bad" echo is
detected at build-config or test-running time?

Thanks,
Orlando
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20211005/97aabcc6/attachment.html>

That list that lit checks for looks out-of-date and insufficient these
days. Off the top of my head, I know for example that we need od, cd, rm,
and mkdir, and there are probably several others that are necessary.

On Tue, 5 Oct 2021 at 15:51, via llvm-dev <llvm-dev at lists.llvm.org>
wrote:
> At a minimum, I think we should stop using (and recommending) the
> bit-rotting GnuWin32 tools, and instead use the tools that come with
> git-for-Windows.  Lit currently checks for 5 tools (cmp, diff, echo, grep,
> sed) and all of these are in the git-Windows set.  This will reduce the set
> of packages we depend on, even if it doesn’t reduce the set of actual tools
> we depend on.
>
> --paulr
>
>
>
> *From:* llvm-dev <llvm-dev-bounces at lists.llvm.org> *On Behalf Of
*via
> llvm-dev
> *Sent:* Tuesday, October 5, 2021 10:30 AM
> *To:* llvm-dev at lists.llvm.org
> *Subject:* [llvm-dev] [llvm-lit] Old echo causes test failure on windows
>
>
>
> Hi llvm-dev,
>
>
>
> I wanted to bring discussion from D110986 to a wider audience.
>
>
>
> After D110986 added a unicode character to the test,
> llvm\test\tools\llvm-cxxfilt\delimiters.test
>
> started failing in our (Sony's) downstream Windows CI build (and
locally
> for me) because the
>
> unicode character was being converted to '?' at some stage in the
test.
> The upstream Windows bots
>
> seemed to have no such trouble. I added a workaround in D111072 that we
> landed because a Chromium
>
> bot was running into the same issue.
>
>
>
> Digging a little deeper it turns out that the echo that lit was picking up
> for me (`echo (GNU
>
> coreutils) 5.3.0` from GnuWin32) converts unicode characters to
'?'.
>
>
>
> llvm-lit has its own built in echo which handles unicode without an issue.
> This is used in the workaround
>
> D111072; lit's echo is used iff the echo command isn't used in a
pipeline.
>
>
>
> N.B. `echo (GNU coreutils) 8.32` shipped with git for Windows also appears
> to handle unicode without
>
> an issue.
>
>
>
> I have a few questions:
>
> 1) Is there a reason we cannot always use lit's echo, as Hans suggests
in
> D111072?
>
> 2) If we can't always use lit's echo, maybe we could add an
error/warning
> if a "bad" echo is
>
> detected at build-config or test-running time?
>
>
>
> Thanks,
>
> Orlando
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20211006/2945f095/attachment.html>

https://reviews.llvm.org/D84380 was supposed to remove the dependency
on GnuWin32 and use git-for-Windows instead.

Only the Clang doc still refers to GnuWin32:
https://clang.llvm.org/get_started.html
The Clang getting_started, at least the how to build part, feels
redundant with LLVM's. Maybe it should link to LLVM's instead?

I myself am using MSYS2 (not from Git-for-Windows) with its bin
directory in %PATH%.

Michael


Am Di., 5. Okt. 2021 um 09:51 Uhr schrieb via llvm-dev
<llvm-dev at lists.llvm.org>:
>
> At a minimum, I think we should stop using (and recommending) the
bit-rotting GnuWin32 tools, and instead use the tools that come with
git-for-Windows.  Lit currently checks for 5 tools (cmp, diff, echo, grep, sed)
and all of these are in the git-Windows set.  This will reduce the set of
packages we depend on, even if it doesn’t reduce the set of actual tools we
depend on.
>
> --paulr
>
>
>
> From: llvm-dev <llvm-dev-bounces at lists.llvm.org> On Behalf Of via
llvm-dev
> Sent: Tuesday, October 5, 2021 10:30 AM
> To: llvm-dev at lists.llvm.org
> Subject: [llvm-dev] [llvm-lit] Old echo causes test failure on windows
>
>
>
> Hi llvm-dev,
>
>
>
> I wanted to bring discussion from D110986 to a wider audience.
>
>
>
> After D110986 added a unicode character to the test,
llvm\test\tools\llvm-cxxfilt\delimiters.test
>
> started failing in our (Sony's) downstream Windows CI build (and
locally for me) because the
>
> unicode character was being converted to '?' at some stage in the
test. The upstream Windows bots
>
> seemed to have no such trouble. I added a workaround in D111072 that we
landed because a Chromium
>
> bot was running into the same issue.
>
>
>
> Digging a little deeper it turns out that the echo that lit was picking up
for me (`echo (GNU
>
> coreutils) 5.3.0` from GnuWin32) converts unicode characters to
'?'.
>
>
>
> llvm-lit has its own built in echo which handles unicode without an issue.
This is used in the workaround
>
> D111072; lit's echo is used iff the echo command isn't used in a
pipeline.
>
>
>
> N.B. `echo (GNU coreutils) 8.32` shipped with git for Windows also appears
to handle unicode without
>
> an issue.
>
>
>
> I have a few questions:
>
> 1) Is there a reason we cannot always use lit's echo, as Hans suggests
in D111072?
>
> 2) If we can't always use lit's echo, maybe we could add an
error/warning if a "bad" echo is
>
> detected at build-config or test-running time?
>
>
>
> Thanks,
>
> Orlando
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev