llvm dev - Jun 2020 - RFC: Adding support for the z/OS platform to LLVM and clang

As part of IBM’s ongoing efforts to improve the z/OS ecosystem, our 
current plans involve adding support for the z/OS platform to LLVM and 
Clang. Our goal is to have a viable C and C++ LLVM compiler and runtime 
library that generates code for, and runs on z/OS.
 
Long term, we expect to have a compiler and library that supports the 
platform more fully. We intend to support the native character encoding 
(EBCDIC), file system - z/OS UNIX System Services (z/OS UNIX) files and 
datasets, addressing modes (31-bit and 64-bit, and possibly 24-bit), 
different floating and fixed point formats, systems programming 
capabilities, language co-processors, and generating code output in the 
z/OS object file formats (GOFF and XOBJ).
 
The tentative plan is to initially support the z/OS Unix System Services 
(z/OS UNIX) interface with EBCDIC and ASCII 64-bit code generation. In 
particular, our intent would be to focus on:
1) C++ standard library support. We would be making changes to the library 
so it can work on z/OS.  We would need some design discussions with the 
community for issues such as character encodings.
2) EBCDIC source file input. We would need reviews at the Clang level when 
dealing with reading source files and dealing with multiple code pages.
3) GOFF object file output. We would need reviews in LLVM to add a new 
object file output format.
Our plans include setting up z/OS build bots and we will update the list 
when we have them ready to go.
 
To begin, we plan to add patches that would:
- Set the new triple for z/OS
- Make changes to the build recipes and tools (cmake, etc.) as needed to 
allow building for z/OS
Following that, we intend to start on the focus areas listed above:
 
1) Add patches to enable building and using the C++ standard library on 
z/OS. In particular, issues dealing with EBCDIC would need to be 
addressed. We would need to have functions in the headers (e.g. iostream) 
that work on ASCII encoded strings, and functions that work on EBCDIC 
encoded strings. These would need to work with the underlying system C 
library (e.g. printf) that provides the actual functionality. For example, 
currently, the z/OS C library has (at least) two sets of functions (ASCII 
and EBCDIC versions). The one used by the application is selected at 
compile time during the system header file processing which selects the 
correct function via mapping the programmer function name (e.g. printf) 
into one that the application will link to (e.g. __printf for EBCDIC and 
\174\174A00118 for ASCII). We would also add patches to disable 
functionality when on z/OS where there is no support for the 
functionality. For example, thread specific locales would be disabled when 
in a non-POSIX mode.
  Our intent is that follow on patches would incrementally add support in 
tandem with the compiler for features that require it and for other z/OS 
specifics such as various floating/fixed point formats.
 
2) Add patches to Clang to allow EBCDIC and ASCII (ISO-8859-1) encoded 
input source files. This would be done at the file open time to allow the 
rest of Clang to operate as if the source was UTF-8 and so require no 
changes downstream. Feedback on this plan is welcome from the Clang 
community.
  Our intent is that later patches would handle execution character set 
differences. Collaboration with the community here would be useful in 
areas such as adding in exec-charset and library selection options and 
strategies.
  Our intent is also to make changes to support any platform issues, 
processing native C header files, and idiosyncrasies on z/OS such as 
having no native strnlen function. We would update test tooling to handle 
character encoding issues as needed. Further design discussion will take 
place on the Clang mailing list.
 
3) Add patches to LLVM that will stub out GOFF object binary generation. 
We would not be generating assembly (HLASM in z/OS), and instead only 
generate the binary object directly for the initial round of changes. 
Assembly generation would follow in later stages once we have a working 
compiler on z/OS. Feedback on this plan of direction is appreciated.
  Our intent is that patches that incrementally add support for GOFF 
object generation such as code sections and records would follow. The next 
steps after support for the object file format would be handling the z/OS 
XPLINK calling convention. This would involve changes to both Clang and 
LLVM and we intend to follow the same style of functional component 
responsibility as is done for other platforms calling conventions. If we 
believe deviations from this is necessary, we plan on notifying the 
community and ensuring the reasons behind the deviations are valid and 
accepted.
 
Any feedback or comments are welcome.
 
Notice: IBM’s statements regarding its plans, directions, and intent are 
subject to change or withdrawal without notice at IBM’s sole discretion. 
Information regarding potential future products is intended to outline our 
general product direction and it should not be relied on in making a 
purchasing decision. The information mentioned regarding potential future 
products is not a commitment, promise, or legal obligation to deliver any 
material, code or functionality. Information about potential future 
products may not be incorporated into any contract. The development, 
release, and timing of any future features or functionality described for 
our products remains at our sole discretion.

Best regards,
Kai Nacke
IT Architect

IBM Deutschland GmbH
Vorsitzender des Aufsichtsrats: Sebastian Krause
Geschäftsführung: Gregor Pillen (Vorsitzender), Agnes Heftberger, Norbert 
Janzen, Markus Koerner, Christian Noll, Nicole Reimer
Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, 
HRB 14562 / WEEE-Reg.-Nr. DE 99369940

I'm not that familiar with the C++ library bits, but looks fine at first
glance.

For EBCDIC source files, what you've outlined matches what we've
discussed previously when the topic has come up on cfe-dev.

For GOFF object generation, we want to make sure that we can write reasonable
tests for patches at each stage.  If there's no assembly output, we have to
verify GOFF binary output.  And probably the only reasonable way to do that is
to first teach llvm-objdump to understand GOFF.

-Eli
> -----Original Message-----
> From: llvm-dev <llvm-dev-bounces at lists.llvm.org> On Behalf Of Kai
Peter
> Nacke via llvm-dev
> Sent: Wednesday, June 10, 2020 12:11 PM
> To: llvm-dev at lists.llvm.org
> Subject: [EXT] [llvm-dev] RFC: Adding support for the z/OS platform to LLVM
> and clang
>
> As part of IBM’s ongoing efforts to improve the z/OS ecosystem, our
> current plans involve adding support for the z/OS platform to LLVM and
> Clang. Our goal is to have a viable C and C++ LLVM compiler and runtime
> library that generates code for, and runs on z/OS.
>
> Long term, we expect to have a compiler and library that supports the
> platform more fully. We intend to support the native character encoding
> (EBCDIC), file system - z/OS UNIX System Services (z/OS UNIX) files and
> datasets, addressing modes (31-bit and 64-bit, and possibly 24-bit),
> different floating and fixed point formats, systems programming
> capabilities, language co-processors, and generating code output in the
> z/OS object file formats (GOFF and XOBJ).
>
> The tentative plan is to initially support the z/OS Unix System Services
> (z/OS UNIX) interface with EBCDIC and ASCII 64-bit code generation. In
> particular, our intent would be to focus on:
> 1) C++ standard library support. We would be making changes to the library
> so it can work on z/OS.  We would need some design discussions with the
> community for issues such as character encodings.
> 2) EBCDIC source file input. We would need reviews at the Clang level when
> dealing with reading source files and dealing with multiple code pages.
> 3) GOFF object file output. We would need reviews in LLVM to add a new
> object file output format.
> Our plans include setting up z/OS build bots and we will update the list
> when we have them ready to go.
>
> To begin, we plan to add patches that would:
> - Set the new triple for z/OS
> - Make changes to the build recipes and tools (cmake, etc.) as needed to
> allow building for z/OS
> Following that, we intend to start on the focus areas listed above:
>
> 1) Add patches to enable building and using the C++ standard library on
> z/OS. In particular, issues dealing with EBCDIC would need to be
> addressed. We would need to have functions in the headers (e.g. iostream)
> that work on ASCII encoded strings, and functions that work on EBCDIC
> encoded strings. These would need to work with the underlying system C
> library (e.g. printf) that provides the actual functionality. For example,
> currently, the z/OS C library has (at least) two sets of functions (ASCII
> and EBCDIC versions). The one used by the application is selected at
> compile time during the system header file processing which selects the
> correct function via mapping the programmer function name (e.g. printf)
> into one that the application will link to (e.g. __printf for EBCDIC and
> \174\174A00118 for ASCII). We would also add patches to disable
> functionality when on z/OS where there is no support for the
> functionality. For example, thread specific locales would be disabled when
> in a non-POSIX mode.
>   Our intent is that follow on patches would incrementally add support in
> tandem with the compiler for features that require it and for other z/OS
> specifics such as various floating/fixed point formats.
>
> 2) Add patches to Clang to allow EBCDIC and ASCII (ISO-8859-1) encoded
> input source files. This would be done at the file open time to allow the
> rest of Clang to operate as if the source was UTF-8 and so require no
> changes downstream. Feedback on this plan is welcome from the Clang
> community.
>   Our intent is that later patches would handle execution character set
> differences. Collaboration with the community here would be useful in
> areas such as adding in exec-charset and library selection options and
> strategies.
>   Our intent is also to make changes to support any platform issues,
> processing native C header files, and idiosyncrasies on z/OS such as
> having no native strnlen function. We would update test tooling to handle
> character encoding issues as needed. Further design discussion will take
> place on the Clang mailing list.
>
> 3) Add patches to LLVM that will stub out GOFF object binary generation.
> We would not be generating assembly (HLASM in z/OS), and instead only
> generate the binary object directly for the initial round of changes.
> Assembly generation would follow in later stages once we have a working
> compiler on z/OS. Feedback on this plan of direction is appreciated.
>   Our intent is that patches that incrementally add support for GOFF
> object generation such as code sections and records would follow. The next
> steps after support for the object file format would be handling the z/OS
> XPLINK calling convention. This would involve changes to both Clang and
> LLVM and we intend to follow the same style of functional component
> responsibility as is done for other platforms calling conventions. If we
> believe deviations from this is necessary, we plan on notifying the
> community and ensuring the reasons behind the deviations are valid and
> accepted.
>
> Any feedback or comments are welcome.
>
> Notice: IBM’s statements regarding its plans, directions, and intent are
> subject to change or withdrawal without notice at IBM’s sole discretion.
> Information regarding potential future products is intended to outline our
> general product direction and it should not be relied on in making a
> purchasing decision. The information mentioned regarding potential future
> products is not a commitment, promise, or legal obligation to deliver any
> material, code or functionality. Information about potential future
> products may not be incorporated into any contract. The development,
> release, and timing of any future features or functionality described for
> our products remains at our sole discretion.
>
> Best regards,
> Kai Nacke
> IT Architect
>
> IBM Deutschland GmbH
> Vorsitzender des Aufsichtsrats: Sebastian Krause
> Geschäftsführung: Gregor Pillen (Vorsitzender), Agnes Heftberger, Norbert
> Janzen, Markus Koerner, Christian Noll, Nicole Reimer
> Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart,
> HRB 14562 / WEEE-Reg.-Nr. DE 99369940
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

On Wed, Jun 10, 2020 at 3:11 PM Kai Peter Nacke via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> 2) Add patches to Clang to allow EBCDIC and ASCII (ISO-8859-1) encoded
> input source files. This would be done at the file open time to allow the
> rest of Clang to operate as if the source was UTF-8 and so require no
> changes downstream. Feedback on this plan is welcome from the Clang
> community.
>
Is there a statement that can be made with respect to accepting UTF-8
encoded source files in a z/OS hosted environment or is it implied that it
works with no changes (and there are no changes that will break this
functionality)?

Also, would these changes enable the consumption of non-UTF-8 encoded
source files on Clang as hosted on other platforms?
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200610/880dcad3/attachment.html>

On 2020-06-10, Hubert Tong via llvm-dev wrote:
>On Wed, Jun 10, 2020 at 3:11 PM Kai Peter Nacke via llvm-dev <
>llvm-dev at lists.llvm.org> wrote:
>
>> 2) Add patches to Clang to allow EBCDIC and ASCII (ISO-8859-1) encoded
>> input source files. This would be done at the file open time to allow
the
>> rest of Clang to operate as if the source was UTF-8 and so require no
>> changes downstream. Feedback on this plan is welcome from the Clang
>> community.
>>
>Is there a statement that can be made with respect to accepting UTF-8
>encoded source files in a z/OS hosted environment or is it implied that it
>works with no changes (and there are no changes that will break this
>functionality)?
>
>Also, would these changes enable the consumption of non-UTF-8 encoded
>source files on Clang as hosted on other platforms?
An 2018 thread has some discussions about non-UTF-8 encoding.
http://lists.llvm.org/pipermail/cfe-dev/2018-January/056745.html

Which case does z/OS want to support?

On 2020-06-10, Kai Peter Nacke via llvm-dev wrote:
>As part of IBM’s ongoing efforts to improve the z/OS ecosystem, our
>current plans involve adding support for the z/OS platform to LLVM and
>Clang. Our goal is to have a viable C and C++ LLVM compiler and runtime
>library that generates code for, and runs on z/OS.
>
>Long term, we expect to have a compiler and library that supports the
>platform more fully. We intend to support the native character encoding
>(EBCDIC), file system - z/OS UNIX System Services (z/OS UNIX) files and
>datasets, addressing modes (31-bit and 64-bit, and possibly 24-bit),
>different floating and fixed point formats, systems programming
>capabilities, language co-processors, and generating code output in the
>z/OS object file formats (GOFF and XOBJ).
>
>The tentative plan is to initially support the z/OS Unix System Services
>(z/OS UNIX) interface with EBCDIC and ASCII 64-bit code generation. In
>particular, our intent would be to focus on:
>1) C++ standard library support. We would be making changes to the library
>so it can work on z/OS.  We would need some design discussions with the
>community for issues such as character encodings.
I am a bit worried about introducing another encoding but there are
other subthreads discussing this matter.
>2) EBCDIC source file input. We would need reviews at the Clang level when
>dealing with reading source files and dealing with multiple code pages.
>3) GOFF object file output. We would need reviews in LLVM to add a new
>object file output format.
>Our plans include setting up z/OS build bots and we will update the list
>when we have them ready to go.
I think most contributors don't have access and probably don't have any
desire to buy z/OS devices if their changes somehow break z/OS. When a
commit caused problems on z/OS, is it the responsibility of the z/OS
community to fix:) ?

If the patch author wants to be nice and fix the issues, are z/OS
developers willing to provide VM access or other resources if such a
scenario arises? Are the binary formats experimental like experimental
targets? (Someone may disagree with experimental targets, but I believe
a new binary format has much larger impact than a new target and it
seems we will have two more)
>To begin, we plan to add patches that would:
>- Set the new triple for z/OS
>- Make changes to the build recipes and tools (cmake, etc.) as needed to
>allow building for z/OS
>Following that, we intend to start on the focus areas listed above:
>
>1) Add patches to enable building and using the C++ standard library on
>z/OS. In particular, issues dealing with EBCDIC would need to be
>addressed. We would need to have functions in the headers (e.g. iostream)
>that work on ASCII encoded strings, and functions that work on EBCDIC
>encoded strings. These would need to work with the underlying system C
>library (e.g. printf) that provides the actual functionality. For example,
>currently, the z/OS C library has (at least) two sets of functions (ASCII
>and EBCDIC versions). The one used by the application is selected at
>compile time during the system header file processing which selects the
>correct function via mapping the programmer function name (e.g. printf)
>into one that the application will link to (e.g. __printf for EBCDIC and
>\174\174A00118 for ASCII). We would also add patches to disable
>functionality when on z/OS where there is no support for the
>functionality. For example, thread specific locales would be disabled when
>in a non-POSIX mode.
>  Our intent is that follow on patches would incrementally add support in
>tandem with the compiler for features that require it and for other z/OS
>specifics such as various floating/fixed point formats.
>
>2) Add patches to Clang to allow EBCDIC and ASCII (ISO-8859-1) encoded
>input source files. This would be done at the file open time to allow the
>rest of Clang to operate as if the source was UTF-8 and so require no
>changes downstream. Feedback on this plan is welcome from the Clang
>community.
>  Our intent is that later patches would handle execution character set
>differences. Collaboration with the community here would be useful in
>areas such as adding in exec-charset and library selection options and
>strategies.
>  Our intent is also to make changes to support any platform issues,
>processing native C header files, and idiosyncrasies on z/OS such as
>having no native strnlen function. We would update test tooling to handle
>character encoding issues as needed. Further design discussion will take
>place on the Clang mailing list.
>
>3) Add patches to LLVM that will stub out GOFF object binary generation.
>We would not be generating assembly (HLASM in z/OS), and instead only
>generate the binary object directly for the initial round of changes.
>Assembly generation would follow in later stages once we have a working
>compiler on z/OS. Feedback on this plan of direction is appreciated.
>  Our intent is that patches that incrementally add support for GOFF
>object generation such as code sections and records would follow. The next
>steps after support for the object file format would be handling the z/OS
>XPLINK calling convention. This would involve changes to both Clang and
>LLVM and we intend to follow the same style of functional component
>responsibility as is done for other platforms calling conventions. If we
>believe deviations from this is necessary, we plan on notifying the
>community and ensuring the reasons behind the deviations are valid and
>accepted.
As someone who cares a lot on the lowlevel stuff, I have more questions
regarding this bullet point.

A MCGOFFStreamer inheriting MCObjectStreamer and possibly also a
MCXOBJStreamer? Do you expect GOFF and XOBJ will fit well in the current
MC framework and will not cause too much burden for developers who are
refactoring generic MC code?

Will z/OS reuse binary utilities (llvm-ar llvm-cov llvm-cxxfilt llvm-nm
llvm-objcopy llvm-objdump llvm-readobj llvm-size llvm-strings llvm-strip
llvm-symbolizer)? Are their any utilities where z/OS's counterparts
diverge enough (from commonplace platforms) that creating new utilities
may be better than reusing existing ones?
>Any feedback or comments are welcome.
>
>Notice: IBM’s statements regarding its plans, directions, and intent are
>subject to change or withdrawal without notice at IBM’s sole discretion.
>Information regarding potential future products is intended to outline our
>general product direction and it should not be relied on in making a
>purchasing decision. The information mentioned regarding potential future
>products is not a commitment, promise, or legal obligation to deliver any
>material, code or functionality. Information about potential future
>products may not be incorporated into any contract. The development,
>release, and timing of any future features or functionality described for
>our products remains at our sole discretion.
>
>Best regards,
>Kai Nacke
>IT Architect
>
>IBM Deutschland GmbH
>Vorsitzender des Aufsichtsrats: Sebastian Krause
>Geschäftsführung: Gregor Pillen (Vorsitzender), Agnes Heftberger, Norbert
>Janzen, Markus Koerner, Christian Noll, Nicole Reimer
>Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart,
>HRB 14562 / WEEE-Reg.-Nr. DE 99369940
>
>_______________________________________________
>LLVM Developers mailing list
>llvm-dev at lists.llvm.org
>https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

Eli Friedman <efriedma at quicinc.com> wrote on 10.06.2020 21:30:26:
> From: Eli Friedman <efriedma at quicinc.com>
> To: Kai Peter Nacke <kai.nacke at de.ibm.com>, "llvm-
> dev at lists.llvm.org" <llvm-dev at lists.llvm.org>
> Date: 10.06.2020 21:30
> Subject: [EXTERNAL] RE: [llvm-dev] RFC: Adding support for the z/OS 
> platform to LLVM and clang
> 
> I'm not that familiar with the C++ library bits, but looks fine at 
> first glance.
> 
> For EBCDIC source files, what you've outlined matches what we've 
> discussed previously when the topic has come up on cfe-dev.
> 
> For GOFF object generation, we want to make sure that we can write 
> reasonable tests for patches at each stage.  If there's no assembly 
> output, we have to verify GOFF binary output.  And probably the only
> reasonable way to do that is to first teach llvm-objdump to understand 
GOFF.

Yes, for sure. Our idea is to use llvm-readobj and llvm-objdump for tests.

Best regards,
Kai Nacke
IT Architect

IBM Deutschland GmbH
Vorsitzender des Aufsichtsrats: Sebastian Krause
Geschäftsführung: Gregor Pillen (Vorsitzender), Agnes Heftberger, Norbert 
Janzen, Markus Koerner, Christian Noll, Nicole Reimer
Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, 
HRB 14562 / WEEE-Reg.-Nr. DE 99369940

Fangrui Song <maskray at google.com> wrote on 11.06.2020 01:32:55:
> >Our plans include setting up z/OS build bots and we will update the 
list
> >when we have them ready to go.
> 
> I think most contributors don't have access and probably don't have
any
> desire to buy z/OS devices if their changes somehow break z/OS. When a
> commit caused problems on z/OS, is it the responsibility of the z/OS
> community to fix:) ?
> 
> If the patch author wants to be nice and fix the issues, are z/OS
> developers willing to provide VM access or other resources if such a
> scenario arises? Are the binary formats experimental like experimental
> targets? (Someone may disagree with experimental targets, but I believe
> a new binary format has much larger impact than a new target and it
> seems we will have two more)
I'm not sure if I really understand your concern. Sorry for that.
I assume that no LLVM developer has access to all supported hardware, so 
in
this respect z/OS is no different than any other target. Following the 
developer
policy (compiling the code and running the tests) should catch most
problems here. Of course, changing z/OS specific source code is a 
different
story. I expect that this is mostly limited to the Support library and 
should
pose no general problem.

The intention is to add support for GOFF format first. So far, we just 
implemented
the required MC interfaces and do not use something like an "experimental 
binary
format". As long as GOFF is not requested through the triple, the new 
format is not
used and therefore should not affect other platforms.
But I'm sure you have a specific scenario in mind. May be can elaborate a 
bit on it?
 
> >  Our intent is that patches that incrementally add support for GOFF
> >object generation such as code sections and records would follow. The 
next
> >steps after support for the object file format would be handling the 
z/OS
> >XPLINK calling convention. This would involve changes to both Clang and
> >LLVM and we intend to follow the same style of functional component
> >responsibility as is done for other platforms calling conventions. If 
we
> >believe deviations from this is necessary, we plan on notifying the
> >community and ensuring the reasons behind the deviations are valid and
> >accepted.
> 
> As someone who cares a lot on the lowlevel stuff, I have more questions
> regarding this bullet point.
> 
> A MCGOFFStreamer inheriting MCObjectStreamer and possibly also a
> MCXOBJStreamer? Do you expect GOFF and XOBJ will fit well in the current
> MC framework and will not cause too much burden for developers who are
> refactoring generic MC code?
We'll begin with GOFF first. This fits well with the current 
MCObjectStreamer
hierarchy and should therefore not cause a burden when refactoring MC 
code.
> Will z/OS reuse binary utilities (llvm-ar llvm-cov llvm-cxxfilt llvm-nm
> llvm-objcopy llvm-objdump llvm-readobj llvm-size llvm-strings llvm-strip
> llvm-symbolizer)? Are their any utilities where z/OS's counterparts
> diverge enough (from commonplace platforms) that creating new utilities
> may be better than reusing existing ones?
We reuse at least the binary utilities which are required for testing, 
e.g.
llvm-readobj and llvm-objdump are very valuable here. The z/OS UNIX System
Services are a certified UNIX, so the commonplace utilities have all a
familar interface. Of course, there are differences in detail.

Thanks for your feedback!
 
Best regards,
Kai Nacke
IT Architect

IBM Deutschland GmbH
Vorsitzender des Aufsichtsrats: Sebastian Krause
Geschäftsführung: Gregor Pillen (Vorsitzender), Agnes Heftberger, Norbert 
Janzen, Markus Koerner, Christian Noll, Nicole Reimer
Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, 
HRB 14562 / WEEE-Reg.-Nr. DE 99369940

Hubert Tong <hubert.reinterpretcast at gmail.com> wrote on 10.06.2020 
23:51:54:
> From: Hubert Tong <hubert.reinterpretcast at gmail.com>
> To: Kai Peter Nacke <kai.nacke at de.ibm.com>
> Cc: llvm-dev <llvm-dev at lists.llvm.org>
> Date: 10.06.2020 23:52
> Subject: [EXTERNAL] Re: [llvm-dev] RFC: Adding support for the z/OS 
> platform to LLVM and clang
> 
> On Wed, Jun 10, 2020 at 3:11 PM Kai Peter Nacke via llvm-dev <llvm-
> dev at lists.llvm.org> wrote:
> 2) Add patches to Clang to allow EBCDIC and ASCII (ISO-8859-1) encoded 
> input source files. This would be done at the file open time to allow 
the 
> rest of Clang to operate as if the source was UTF-8 and so require no 
> changes downstream. Feedback on this plan is welcome from the Clang 
> community.
> Is there a statement that can be made with respect to accepting 
> UTF-8 encoded source files in a z/OS hosted environment or is it 
> implied that it works with no changes (and there are no changes that
> will break this functionality)?
> 
> Also, would these changes enable the consumption of non-UTF-8 
> encoded source files on Clang as hosted on other platforms?
The intention is to use the auto-conversion feature from the
language environment. Currently, this platform feature does not
handle conversions of multi-byte encodings, so at this time
consumption of UTF-8 encoded source files is not possible.
For the same reason, this does not enable the consumption of
non-UTF-8 encoded source files on other platforms.

Best regards,
Kai Nacke
IT Architect

IBM Deutschland GmbH
Vorsitzender des Aufsichtsrats: Sebastian Krause
Geschäftsführung: Gregor Pillen (Vorsitzender), Agnes Heftberger, Norbert 
Janzen, Markus Koerner, Christian Noll, Nicole Reimer
Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, 
HRB 14562 / WEEE-Reg.-Nr. DE 99369940

> On Jun 10, 2020, at 15:11, Kai Peter Nacke via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> As part of IBM’s ongoing efforts to improve the z/OS ecosystem, our 
> current plans involve adding support for the z/OS platform to LLVM and 
> Clang. Our goal is to have a viable C and C++ LLVM compiler and runtime 
> library that generates code for, and runs on z/OS.
> 
> Long term, we expect to have a compiler and library that supports the 
> platform more fully. We intend to support the native character encoding 
> (EBCDIC), file system - z/OS UNIX System Services (z/OS UNIX) files and 
> datasets, addressing modes (31-bit and 64-bit, and possibly 24-bit), 
> different floating and fixed point formats, systems programming 
> capabilities, language co-processors, and generating code output in the 
> z/OS object file formats (GOFF and XOBJ).
> 
> The tentative plan is to initially support the z/OS Unix System Services 
> (z/OS UNIX) interface with EBCDIC and ASCII 64-bit code generation. In 
> particular, our intent would be to focus on:
> 1) C++ standard library support. We would be making changes to the library 
> so it can work on z/OS.  We would need some design discussions with the 
> community for issues such as character encodings.
> 2) EBCDIC source file input. We would need reviews at the Clang level when 
> dealing with reading source files and dealing with multiple code pages.
> 3) GOFF object file output. We would need reviews in LLVM to add a new 
> object file output format.
> Our plans include setting up z/OS build bots and we will update the list 
> when we have them ready to go.
> 
> To begin, we plan to add patches that would:
> - Set the new triple for z/OS
> - Make changes to the build recipes and tools (cmake, etc.) as needed to 
> allow building for z/OS
> Following that, we intend to start on the focus areas listed above:
> 
> 1) Add patches to enable building and using the C++ standard library on 
> z/OS. In particular, issues dealing with EBCDIC would need to be 
> addressed. We would need to have functions in the headers (e.g. iostream) 
> that work on ASCII encoded strings, and functions that work on EBCDIC 
> encoded strings. These would need to work with the underlying system C 
> library (e.g. printf) that provides the actual functionality. For example, 
> currently, the z/OS C library has (at least) two sets of functions (ASCII 
> and EBCDIC versions). The one used by the application is selected at 
> compile time during the system header file processing which selects the 
> correct function via mapping the programmer function name (e.g. printf) 
> into one that the application will link to (e.g. __printf for EBCDIC and 
> \174\174A00118 for ASCII).
I must admit I am wary of adding even more complexity to the libc++ headers. We
have a big problem of configuration explosion in libc++, and that will
unfortunately not help. If there are ways to support EDBCDIC non-intrusively,
that would be better. In all cases, let's talk. Ideally, we could have an
exploratory chat where someone from your side can show me what you're trying
to achieve before we go down the route of having a full patch posted to Phab.
> We would also add patches to disable 
> functionality when on z/OS where there is no support for the 
> functionality. For example, thread specific locales would be disabled when 
> in a non-POSIX mode.
>  Our intent is that follow on patches would incrementally add support in 
> tandem with the compiler for features that require it and for other z/OS 
> specifics such as various floating/fixed point formats.
At the risk of sounding grumpy, this scares me too. Conditionally removing parts
of libc++ is tricky. It basically creates a different "compile-time code
path" through #ifdefs every time we do that. Those are a source of bugs and
a maintenance burden, and I've been trying to remove as many as I can. For
example, you'll notice I'm fairly aggressive about removing workarounds
for old untested Apple platforms. It's much easier when the underlying
system just provides what the library needs.

But in all cases, let's have a chat to see the extent of the changes you
need and we can go from there. Also, I would suggest setting up build bots for
libc++ on z/OS from the very start -- one can only claim to support a system if
it's tested continuously, so patches will be received with a lot more
enthusiasm if they are tested somehow.

If you're on the CppLang Slack, feel free to drop me a line and we can have
an informal chat.

Cheers,
Louis
> 
> 2) Add patches to Clang to allow EBCDIC and ASCII (ISO-8859-1) encoded 
> input source files. This would be done at the file open time to allow the 
> rest of Clang to operate as if the source was UTF-8 and so require no 
> changes downstream. Feedback on this plan is welcome from the Clang 
> community.
>  Our intent is that later patches would handle execution character set 
> differences. Collaboration with the community here would be useful in 
> areas such as adding in exec-charset and library selection options and 
> strategies.
>  Our intent is also to make changes to support any platform issues, 
> processing native C header files, and idiosyncrasies on z/OS such as 
> having no native strnlen function. We would update test tooling to handle 
> character encoding issues as needed. Further design discussion will take 
> place on the Clang mailing list.
> 
> 3) Add patches to LLVM that will stub out GOFF object binary generation. 
> We would not be generating assembly (HLASM in z/OS), and instead only 
> generate the binary object directly for the initial round of changes. 
> Assembly generation would follow in later stages once we have a working 
> compiler on z/OS. Feedback on this plan of direction is appreciated.
>  Our intent is that patches that incrementally add support for GOFF 
> object generation such as code sections and records would follow. The next 
> steps after support for the object file format would be handling the z/OS 
> XPLINK calling convention. This would involve changes to both Clang and 
> LLVM and we intend to follow the same style of functional component 
> responsibility as is done for other platforms calling conventions. If we 
> believe deviations from this is necessary, we plan on notifying the 
> community and ensuring the reasons behind the deviations are valid and 
> accepted.
> 
> Any feedback or comments are welcome.
> 
> Notice: IBM’s statements regarding its plans, directions, and intent are 
> subject to change or withdrawal without notice at IBM’s sole discretion. 
> Information regarding potential future products is intended to outline our 
> general product direction and it should not be relied on in making a 
> purchasing decision. The information mentioned regarding potential future 
> products is not a commitment, promise, or legal obligation to deliver any 
> material, code or functionality. Information about potential future 
> products may not be incorporated into any contract. The development, 
> release, and timing of any future features or functionality described for 
> our products remains at our sole discretion.
> 
> Best regards,
> Kai Nacke
> IT Architect
> 
> IBM Deutschland GmbH
> Vorsitzender des Aufsichtsrats: Sebastian Krause
> Geschäftsführung: Gregor Pillen (Vorsitzender), Agnes Heftberger, Norbert 
> Janzen, Markus Koerner, Christian Noll, Nicole Reimer
> Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, 
> HRB 14562 / WEEE-Reg.-Nr. DE 99369940
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

Thanks for the feedback!

As summary,
- some concerns about added complexity and testability were expressed
- details of the encoding conversion in the clang frontend were discussed

For these kinds of discussion, it's best to have a look at the source, so 
I created the first Phabricator review: https://reviews.llvm.org/D82081 

Thanks again!

Best regards,
Kai Nacke
IT Architect

IBM Deutschland GmbH
Vorsitzender des Aufsichtsrats: Sebastian Krause
Geschäftsführung: Gregor Pillen (Vorsitzender), Agnes Heftberger, Norbert 
Janzen, Markus Koerner, Christian Noll, Nicole Reimer
Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, 
HRB 14562 / WEEE-Reg.-Nr. DE 99369940

"llvm-dev" <llvm-dev-bounces at lists.llvm.org> wrote on
10.06.2020 21:11:05:
> From: Kai Peter Nacke via llvm-dev <llvm-dev at lists.llvm.org>
> To: llvm-dev at lists.llvm.org
> Date: 10.06.2020 21:11
> Subject: [EXTERNAL] [llvm-dev] RFC: Adding support for the z/OS 
> platform to LLVM and clang
> Sent by: "llvm-dev" <llvm-dev-bounces at lists.llvm.org>
> 
> As part of IBM’s ongoing efforts to improve the z/OS ecosystem, our 
> current plans involve adding support for the z/OS platform to LLVM and 
> Clang. Our goal is to have a viable C and C++ LLVM compiler and runtime 
> library that generates code for, and runs on z/OS.
> 
> Long term, we expect to have a compiler and library that supports the 
> platform more fully. We intend to support the native character encoding 
> (EBCDIC), file system - z/OS UNIX System Services (z/OS UNIX) files and 
> datasets, addressing modes (31-bit and 64-bit, and possibly 24-bit), 
> different floating and fixed point formats, systems programming 
> capabilities, language co-processors, and generating code output in the 
> z/OS object file formats (GOFF and XOBJ).
> 
> The tentative plan is to initially support the z/OS Unix System Services 
> (z/OS UNIX) interface with EBCDIC and ASCII 64-bit code generation. In 
> particular, our intent would be to focus on:
> 1) C++ standard library support. We would be making changes to the 
library 
> so it can work on z/OS.  We would need some design discussions with the 
> community for issues such as character encodings.
> 2) EBCDIC source file input. We would need reviews at the Clang level 
when 
> dealing with reading source files and dealing with multiple code pages.
> 3) GOFF object file output. We would need reviews in LLVM to add a new 
> object file output format.
> Our plans include setting up z/OS build bots and we will update the list 
> when we have them ready to go.
> 
> To begin, we plan to add patches that would:
> - Set the new triple for z/OS
> - Make changes to the build recipes and tools (cmake, etc.) as needed to 
> allow building for z/OS
> Following that, we intend to start on the focus areas listed above:
> 
> 1) Add patches to enable building and using the C++ standard library on 
> z/OS. In particular, issues dealing with EBCDIC would need to be 
> addressed. We would need to have functions in the headers (e.g. 
iostream) 
> that work on ASCII encoded strings, and functions that work on EBCDIC 
> encoded strings. These would need to work with the underlying system C 
> library (e.g. printf) that provides the actual functionality. For 
example, 
> currently, the z/OS C library has (at least) two sets of functions 
(ASCII 
> and EBCDIC versions). The one used by the application is selected at 
> compile time during the system header file processing which selects the 
> correct function via mapping the programmer function name (e.g. printf) 
> into one that the application will link to (e.g. __printf for EBCDIC and 
> \174\174A00118 for ASCII). We would also add patches to disable 
> functionality when on z/OS where there is no support for the 
> functionality. For example, thread specific locales would be disabled 
when 
> in a non-POSIX mode.
>   Our intent is that follow on patches would incrementally add support 
in 
> tandem with the compiler for features that require it and for other z/OS 
> specifics such as various floating/fixed point formats.
> 
> 2) Add patches to Clang to allow EBCDIC and ASCII (ISO-8859-1) encoded 
> input source files. This would be done at the file open time to allow 
the 
> rest of Clang to operate as if the source was UTF-8 and so require no 
> changes downstream. Feedback on this plan is welcome from the Clang 
> community.
>   Our intent is that later patches would handle execution character set 
> differences. Collaboration with the community here would be useful in 
> areas such as adding in exec-charset and library selection options and 
> strategies.
>   Our intent is also to make changes to support any platform issues, 
> processing native C header files, and idiosyncrasies on z/OS such as 
> having no native strnlen function. We would update test tooling to 
handle 
> character encoding issues as needed. Further design discussion will take 
> place on the Clang mailing list.
> 
> 3) Add patches to LLVM that will stub out GOFF object binary generation. 
> We would not be generating assembly (HLASM in z/OS), and instead only 
> generate the binary object directly for the initial round of changes. 
> Assembly generation would follow in later stages once we have a working 
> compiler on z/OS. Feedback on this plan of direction is appreciated.
>   Our intent is that patches that incrementally add support for GOFF 
> object generation such as code sections and records would follow. The 
next 
> steps after support for the object file format would be handling the 
z/OS 
> XPLINK calling convention. This would involve changes to both Clang and 
> LLVM and we intend to follow the same style of functional component 
> responsibility as is done for other platforms calling conventions. If we 
> believe deviations from this is necessary, we plan on notifying the 
> community and ensuring the reasons behind the deviations are valid and 
> accepted.
> 
> Any feedback or comments are welcome.
> 
> Notice: IBM’s statements regarding its plans, directions, and intent are 
> subject to change or withdrawal without notice at IBM’s sole discretion. 
> Information regarding potential future products is intended to outline 
our 
> general product direction and it should not be relied on in making a 
> purchasing decision. The information mentioned regarding potential 
future 
> products is not a commitment, promise, or legal obligation to deliver 
any 
> material, code or functionality. Information about potential future 
> products may not be incorporated into any contract. The development, 
> release, and timing of any future features or functionality described 
for 
> our products remains at our sole discretion.
> 
> Best regards,
> Kai Nacke
> IT Architect
> 
> IBM Deutschland GmbH
> Vorsitzender des Aufsichtsrats: Sebastian Krause
> Geschäftsführung: Gregor Pillen (Vorsitzender), Agnes Heftberger, 
Norbert 
> Janzen, Markus Koerner, Christian Noll, Nicole Reimer
> Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht 
Stuttgart, 
> HRB 14562 / WEEE-Reg.-Nr. DE 99369940
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://urldefense.proofpoint.com/v2/url?
> 
u=https-3A__lists.llvm.org_cgi-2Dbin_mailman_listinfo_llvm-2Ddev&d=DwIGaQ&c=jf_iaSHvJObTbx-
> siA1ZOg&r=43FMMTMN1rMQYLfzcfWYI9JmFbjyCLLZVkpxUNJkDuQ&m=yMqyEWU-
> 
Y3yQdOvsfp9HBkSwlDeLE5A9ZydkH_73SZk&s=ymE6YLFHNgGyUlKieWzvyvY5QV4ycTojjrma8Uz_rhs&e>