On Fri, Jun 28, 2019 at 9:29 AM JF Bastien <jfbastien at apple.com> wrote:> > I think I now understand some of the disconnect you and I are having, and I think some of the pushback you’re getting from the community is the same. You’re talking about where you want to start with an LLVM libc. Many in the community (myself included) want to understand where we’ll get with this libc. At steady-state, what does it do? To a certain degree I don’t care about how you get to the steady state: sure the implementation approach is important, and which contributor cares about what parts is important in shaping that evolution, but at the end of the day what matters is where you get. > > So here’s what’s missing: there’s no goal. Right now, your proposal is “let’s do an LLVM libc, starting with what I care about, who’s interested?” > > That’s an OK place to start! You illustrated your needs, others chimed in with theirs, and now you know there’s some interest. However, you should take time now to come up with a plan. What’s this libc actually going to be? I ask a bunch of questions below that I think you need to answer as a next step. Others asked more questions which I didn’t echo, but which you should answer as well. What does this libc aspire to become?I apologize for the delay. I will try to address the above questions in this email. I will shortly follow up with answers to other questions. Below is a write up which I think would qualify as the "charter" for the new libc. It is also answering questions like, "where we’ll get with this libc?", "what's this libc actually going to be?" and similar ones. I have used libcxx.llvm.org landing page as a template to write it down. ############################################### "llvm-libc" C Standard Library ======================= llvm-libc is an implementation of the C standard library targeting C11 and above. It also provides platform specific extensions as relevant. For example, on Linux it also provides pthreads, librt and other POSIX extension libraries. Documentation =========== The llvm-libc project is still in the planning phase. Stay tuned for updates soon. Features and Goals =============== * C11 and upwards conformant. * A modular libc with individual pieces implemented in the "as a library" philosophy of the LLVM project. * Ability to layer this libc over the system libc. * Provide C symbols as specified by the standards, but take advantage and use C++ language facilities for the core implementation. * Provides POSIX extensions on POSIX compliant platforms. * Provides system-specific extensions as appropriate. For example, provides the Linux API on Linux. * Vendor extensions if and only if necessary. * Designed and developed from the start to work with LLVM tooling and testing like fuzz testing and sanitizer-supported testing. * ABI independent implementation as far as possible. * Use source based implementations as far possible rather than assembly. Will try to “fix” the compiler rather than use assembly language workarounds. Why a new C Standard Library? ======================== Implementing a libc is no small task and is not be taken lightly. A natural question to ask is, "why a new implementation of the C standard library?" There is no single answer to this question, but some of the major reasons are as follows: * Most libc implementations are monolithic. It is a non-trivial porting task to pick and choose only the pieces relevant to one's platform. The new libc will be developed with sufficient modularity to make picking and choosing a straightforward task. * Most libc implementations break when built with sanitizer specific compiler options. The new libc will be developed from the start to work with those specialized compiler options. * The new libc will be developed to support and employ fuzz testing from the start. * Most libc implementations use a good amount of assembly language, and assume specific ABIs (may be platform dependent). With the new libc implementation, we want to use normal source code as much as possible so that compiler-based changes to the ABI are easy. Moreover, as part of the LLVM project, we want to use this opportunity to fix performance related compiler bugs rather than using assembly workarounds. * A large hole in the llvm toolchain will be plugged with this new libc. With the broad platform expertise in the LLVM community, and the strong license and project structure, we think that the new libc will be more tunable and robust, without sacrificing the simplicity and accessibility typical of the LLVM project. Platform Support ============= llvm-libc development is still in the planning phase. However, we envision that it will support a variety of platforms in the coming years. Interested parties are encouraged to participate in the design and implementation, and add support for their favorite platforms. Current Status =========== llvm-libc development is still in the planning phase. Build Bots ======== Coming soon. Get involved! ========== First please review our Developer's Policy. Stay tuned for llvm-libc specific information. Design Documents ============== Coming soon.
On Fri, Jul 12, 2019 at 11:16 AM Siva Chandra via llvm-dev <llvm-dev at lists.llvm.org> wrote:> > On Fri, Jun 28, 2019 at 9:29 AM JF Bastien <jfbastien at apple.com> wrote: > > > > I think I now understand some of the disconnect you and I are having, and I think some of the pushback you’re getting from the community is the same. You’re talking about where you want to start with an LLVM libc. Many in the community (myself included) want to understand where we’ll get with this libc. At steady-state, what does it do? To a certain degree I don’t care about how you get to the steady state: sure the implementation approach is important, and which contributor cares about what parts is important in shaping that evolution, but at the end of the day what matters is where you get. > > > > So here’s what’s missing: there’s no goal. Right now, your proposal is “let’s do an LLVM libc, starting with what I care about, who’s interested?” > > > > That’s an OK place to start! You illustrated your needs, others chimed in with theirs, and now you know there’s some interest. However, you should take time now to come up with a plan. What’s this libc actually going to be? I ask a bunch of questions below that I think you need to answer as a next step. Others asked more questions which I didn’t echo, but which you should answer as well. What does this libc aspire to become? > > I apologize for the delay. I will try to address the above questions > in this email. I will shortly follow up with answers to other > questions. > > Below is a write up which I think would qualify as the "charter" for > the new libc. It is also answering questions like, "where we’ll get > with this libc?", "what's this libc actually going to be?" and similar > ones. I have used libcxx.llvm.org landing page as a template to write > it down. > > ############################################### > > "llvm-libc" C Standard Library > =======================> > llvm-libc is an implementation of the C standard library targeting C11 > and above.Any particular reason for C11 as opposed to C17? ~Aaron> It also provides platform specific extensions as relevant. > For example, on Linux it also provides pthreads, librt and other POSIX > extension libraries. > > Documentation > ===========> > The llvm-libc project is still in the planning phase. Stay tuned for > updates soon. > > Features and Goals > ===============> > * C11 and upwards conformant. > * A modular libc with individual pieces implemented in the "as a > library" philosophy of the LLVM project. > * Ability to layer this libc over the system libc. > * Provide C symbols as specified by the standards, but take advantage > and use C++ language facilities for the core implementation. > * Provides POSIX extensions on POSIX compliant platforms. > * Provides system-specific extensions as appropriate. For example, > provides the Linux API on Linux. > * Vendor extensions if and only if necessary. > * Designed and developed from the start to work with LLVM tooling and > testing like fuzz testing and sanitizer-supported testing. > * ABI independent implementation as far as possible. > * Use source based implementations as far possible rather than > assembly. Will try to “fix” the compiler rather than use assembly > language workarounds. > > Why a new C Standard Library? > ========================> > Implementing a libc is no small task and is not be taken lightly. A > natural question to ask is, "why a new implementation of the C > standard library?" There is no single answer to this question, but > some of the major reasons are as follows: > > * Most libc implementations are monolithic. It is a non-trivial > porting task to pick and choose only the pieces relevant to one's > platform. The new libc will be developed with sufficient modularity to > make picking and choosing a straightforward task. > * Most libc implementations break when built with sanitizer specific > compiler options. The new libc will be developed from the start to > work with those specialized compiler options. > * The new libc will be developed to support and employ fuzz testing > from the start. > * Most libc implementations use a good amount of assembly language, > and assume specific ABIs (may be platform dependent). With the new > libc implementation, we want to use normal source code as much as > possible so that compiler-based changes to the ABI are easy. Moreover, > as part of the LLVM project, we want to use this opportunity to fix > performance related compiler bugs rather than using assembly > workarounds. > * A large hole in the llvm toolchain will be plugged with this new > libc. With the broad platform expertise in the LLVM community, and the > strong license and project structure, we think that the new libc will > be more tunable and robust, without sacrificing the simplicity and > accessibility typical of the LLVM project. > > Platform Support > =============> > llvm-libc development is still in the planning phase. However, we > envision that it will support a variety of platforms in the coming > years. Interested parties are encouraged to participate in the design > and implementation, and add support for their favorite platforms. > > Current Status > ===========> > llvm-libc development is still in the planning phase. > > Build Bots > ========> > Coming soon. > > Get involved! > ==========> > First please review our Developer's Policy. Stay tuned for llvm-libc > specific information. > > Design Documents > ==============> > Coming soon. > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
I think this was mention before by JF, but in this new write up and testing point:> Designed and developed from the start to work with LLVM tooling and testing like fuzz testing and sanitizer-supported testing.I am also curious why language conformance testing is not mentioned? Are there ideas on that? ________________________________ From: llvm-dev <llvm-dev-bounces at lists.llvm.org> on behalf of Siva Chandra via llvm-dev <llvm-dev at lists.llvm.org> Sent: 12 July 2019 16:15 To: JF Bastien Cc: llvm-dev; Marshall Clow Subject: Re: [llvm-dev] A libc in LLVM On Fri, Jun 28, 2019 at 9:29 AM JF Bastien <jfbastien at apple.com> wrote:> > I think I now understand some of the disconnect you and I are having, and I think some of the pushback you’re getting from the community is the same. You’re talking about where you want to start with an LLVM libc. Many in the community (myself included) want to understand where we’ll get with this libc. At steady-state, what does it do? To a certain degree I don’t care about how you get to the steady state: sure the implementation approach is important, and which contributor cares about what parts is important in shaping that evolution, but at the end of the day what matters is where you get. > > So here’s what’s missing: there’s no goal. Right now, your proposal is “let’s do an LLVM libc, starting with what I care about, who’s interested?” > > That’s an OK place to start! You illustrated your needs, others chimed in with theirs, and now you know there’s some interest. However, you should take time now to come up with a plan. What’s this libc actually going to be? I ask a bunch of questions below that I think you need to answer as a next step. Others asked more questions which I didn’t echo, but which you should answer as well. What does this libc aspire to become?I apologize for the delay. I will try to address the above questions in this email. I will shortly follow up with answers to other questions. Below is a write up which I think would qualify as the "charter" for the new libc. It is also answering questions like, "where we’ll get with this libc?", "what's this libc actually going to be?" and similar ones. I have used libcxx.llvm.org landing page as a template to write it down. ############################################### "llvm-libc" C Standard Library ======================= llvm-libc is an implementation of the C standard library targeting C11 and above. It also provides platform specific extensions as relevant. For example, on Linux it also provides pthreads, librt and other POSIX extension libraries. Documentation =========== The llvm-libc project is still in the planning phase. Stay tuned for updates soon. Features and Goals =============== * C11 and upwards conformant. * A modular libc with individual pieces implemented in the "as a library" philosophy of the LLVM project. * Ability to layer this libc over the system libc. * Provide C symbols as specified by the standards, but take advantage and use C++ language facilities for the core implementation. * Provides POSIX extensions on POSIX compliant platforms. * Provides system-specific extensions as appropriate. For example, provides the Linux API on Linux. * Vendor extensions if and only if necessary. * Designed and developed from the start to work with LLVM tooling and testing like fuzz testing and sanitizer-supported testing. * ABI independent implementation as far as possible. * Use source based implementations as far possible rather than assembly. Will try to “fix” the compiler rather than use assembly language workarounds. Why a new C Standard Library? ======================== Implementing a libc is no small task and is not be taken lightly. A natural question to ask is, "why a new implementation of the C standard library?" There is no single answer to this question, but some of the major reasons are as follows: * Most libc implementations are monolithic. It is a non-trivial porting task to pick and choose only the pieces relevant to one's platform. The new libc will be developed with sufficient modularity to make picking and choosing a straightforward task. * Most libc implementations break when built with sanitizer specific compiler options. The new libc will be developed from the start to work with those specialized compiler options. * The new libc will be developed to support and employ fuzz testing from the start. * Most libc implementations use a good amount of assembly language, and assume specific ABIs (may be platform dependent). With the new libc implementation, we want to use normal source code as much as possible so that compiler-based changes to the ABI are easy. Moreover, as part of the LLVM project, we want to use this opportunity to fix performance related compiler bugs rather than using assembly workarounds. * A large hole in the llvm toolchain will be plugged with this new libc. With the broad platform expertise in the LLVM community, and the strong license and project structure, we think that the new libc will be more tunable and robust, without sacrificing the simplicity and accessibility typical of the LLVM project. Platform Support ============= llvm-libc development is still in the planning phase. However, we envision that it will support a variety of platforms in the coming years. Interested parties are encouraged to participate in the design and implementation, and add support for their favorite platforms. Current Status =========== llvm-libc development is still in the planning phase. Build Bots ======== Coming soon. Get involved! ========== First please review our Developer's Policy. Stay tuned for llvm-libc specific information. Design Documents ============== Coming soon. _______________________________________________ LLVM Developers mailing list llvm-dev at lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190712/f1429781/attachment.html>
On Fri, Jul 12, 2019 at 8:44 AM Sjoerd Meijer <Sjoerd.Meijer at arm.com> wrote:> I am also curious why language conformance testing is not mentioned? Are there ideas on that?Sorry, I split that part into another email: http://lists.llvm.org/pipermail/llvm-dev/2019-July/133867.html> ________________________________ > From: llvm-dev <llvm-dev-bounces at lists.llvm.org> on behalf of Siva Chandra via llvm-dev <llvm-dev at lists.llvm.org> > Sent: 12 July 2019 16:15 > To: JF Bastien > Cc: llvm-dev; Marshall Clow > Subject: Re: [llvm-dev] A libc in LLVM > > On Fri, Jun 28, 2019 at 9:29 AM JF Bastien <jfbastien at apple.com> wrote: > > > > I think I now understand some of the disconnect you and I are having, and I think some of the pushback you’re getting from the community is the same. You’re talking about where you want to start with an LLVM libc. Many in the community (myself included) want to understand where we’ll get with this libc. At steady-state, what does it do? To a certain degree I don’t care about how you get to the steady state: sure the implementation approach is important, and which contributor cares about what parts is important in shaping that evolution, but at the end of the day what matters is where you get. > > > > So here’s what’s missing: there’s no goal. Right now, your proposal is “let’s do an LLVM libc, starting with what I care about, who’s interested?” > > > > That’s an OK place to start! You illustrated your needs, others chimed in with theirs, and now you know there’s some interest. However, you should take time now to come up with a plan. What’s this libc actually going to be? I ask a bunch of questions below that I think you need to answer as a next step. Others asked more questions which I didn’t echo, but which you should answer as well. What does this libc aspire to become? > > I apologize for the delay. I will try to address the above questions > in this email. I will shortly follow up with answers to other > questions. > > Below is a write up which I think would qualify as the "charter" for > the new libc. It is also answering questions like, "where we’ll get > with this libc?", "what's this libc actually going to be?" and similar > ones. I have used libcxx.llvm.org landing page as a template to write > it down. > > ############################################### > > "llvm-libc" C Standard Library > =======================> > llvm-libc is an implementation of the C standard library targeting C11 > and above. It also provides platform specific extensions as relevant. > For example, on Linux it also provides pthreads, librt and other POSIX > extension libraries. > > Documentation > ===========> > The llvm-libc project is still in the planning phase. Stay tuned for > updates soon. > > Features and Goals > ===============> > * C11 and upwards conformant. > * A modular libc with individual pieces implemented in the "as a > library" philosophy of the LLVM project. > * Ability to layer this libc over the system libc. > * Provide C symbols as specified by the standards, but take advantage > and use C++ language facilities for the core implementation. > * Provides POSIX extensions on POSIX compliant platforms. > * Provides system-specific extensions as appropriate. For example, > provides the Linux API on Linux. > * Vendor extensions if and only if necessary. > * Designed and developed from the start to work with LLVM tooling and > testing like fuzz testing and sanitizer-supported testing. > * ABI independent implementation as far as possible. > * Use source based implementations as far possible rather than > assembly. Will try to “fix” the compiler rather than use assembly > language workarounds. > > Why a new C Standard Library? > ========================> > Implementing a libc is no small task and is not be taken lightly. A > natural question to ask is, "why a new implementation of the C > standard library?" There is no single answer to this question, but > some of the major reasons are as follows: > > * Most libc implementations are monolithic. It is a non-trivial > porting task to pick and choose only the pieces relevant to one's > platform. The new libc will be developed with sufficient modularity to > make picking and choosing a straightforward task. > * Most libc implementations break when built with sanitizer specific > compiler options. The new libc will be developed from the start to > work with those specialized compiler options. > * The new libc will be developed to support and employ fuzz testing > from the start. > * Most libc implementations use a good amount of assembly language, > and assume specific ABIs (may be platform dependent). With the new > libc implementation, we want to use normal source code as much as > possible so that compiler-based changes to the ABI are easy. Moreover, > as part of the LLVM project, we want to use this opportunity to fix > performance related compiler bugs rather than using assembly > workarounds. > * A large hole in the llvm toolchain will be plugged with this new > libc. With the broad platform expertise in the LLVM community, and the > strong license and project structure, we think that the new libc will > be more tunable and robust, without sacrificing the simplicity and > accessibility typical of the LLVM project. > > Platform Support > =============> > llvm-libc development is still in the planning phase. However, we > envision that it will support a variety of platforms in the coming > years. Interested parties are encouraged to participate in the design > and implementation, and add support for their favorite platforms. > > Current Status > ===========> > llvm-libc development is still in the planning phase. > > Build Bots > ========> > Coming soon. > > Get involved! > ==========> > First please review our Developer's Policy. Stay tuned for llvm-libc > specific information. > > Design Documents > ==============> > Coming soon. > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Aaron Ballman via llvm-dev <llvm-dev at lists.llvm.org> writes:>> * Ability to layer this libc over the system libc.Echoing others, this seems dubious to me. Why not build up small pieces at a time and write tests for them? This library doesn't need to support all existing programs out of the gate. I dont't think libc++ layered on top of existing standard C++ libraries, so why would libc need to?>> * Provide C symbols as specified by the standards, but take advantage >> and use C++ language facilities for the core implementation.Does this mean C programs would require a C++ runtime? If not, how will the project ensure that? -David
On Fri, Jul 12, 2019 at 02:54:40PM -0400, Siva Chandra wrote:> On Fri, Jun 28, 2019 at 9:29 AM JF Bastien <jfbastien at apple.com> > wrote: > > > > I think I now understand some of the disconnect you and I are > > having, and I think some of the pushback you’re getting from the > > community is the same. You’re talking about where you want to start > > with an LLVM libc. Many in the community (myself included) want to > > understand where we’ll get with this libc. At steady-state, what > > does it do? To a certain degree I don’t care about how you get to > > the steady state: sure the implementation approach is important, and > > which contributor cares about what parts is important in shaping > > that evolution, but at the end of the day what matters is where you > > get. > > > > So here’s what’s missing: there’s no goal. Right now, your proposal > > is “let’s do an LLVM libc, starting with what I care about, who’s > > interested?” > > > > That’s an OK place to start! You illustrated your needs, others > > chimed in with theirs, and now you know there’s some interest. > > However, you should take time now to come up with a plan. What’s > > this libc actually going to be? I ask a bunch of questions below > > that I think you need to answer as a next step. Others asked more > > questions which I didn’t echo, but which you should answer as well. > > What does this libc aspire to become? > > I apologize for the delay. I will try to address the above questions > in this email. I will shortly follow up with answers to other > questions. > > Below is a write up which I think would qualify as the "charter" for > the new libc. It is also answering questions like, "where we’ll get > with this libc?", "what's this libc actually going to be?" and similar > ones. I have used libcxx.llvm.org landing page as a template to write > it down. > > ############################################### > > "llvm-libc" C Standard Library > =======================> > llvm-libc is an implementation of the C standard library targeting C11 > and above. It also provides platform specific extensions as relevant. > For example, on Linux it also provides pthreads, librt and other POSIX > extension libraries. > > Documentation > ===========> > The llvm-libc project is still in the planning phase. Stay tuned for > updates soon. > > Features and Goals > ===============> > * C11 and upwards conformant. > * A modular libc with individual pieces implemented in the "as a > library" philosophy of the LLVM project.> * Ability to layer this libc over the system libc.I don't think this is really possible, without tooling designed specifically to do it (remapping symbols, etc.), which clang/LLVM, beign the compiler/tooling, *could* do. But even with the right tooling, you're going to find that it's *a lot* harder than you expect, likely almost impossible without making assumptions about the internals of the underlying libc that are not public contracts.> * Provide C symbols as specified by the standards, but take advantage > and use C++ language facilities for the core implementation.This was done in Fuchsia's fork of musl too, and was one of my major criticisms of it -- makes no sense except satisfying developers who want to use C++ for the sake of it being C++. It's very hard to do "freestanding" C++ that doesn't even rely on the underlying libc, and if you do rely on libc, it's a circular dependency. Moreover there's really *very little* in libc that benefits from C++ (much less a pure freestanding C++ with no library) for implementing it. And there's huge potential to get things wrong by using C++ in ways that have hidden failure cases/exceptions in places where the C interface you're implementing either cannot be allowed to fail, or where introducing the possibility of failure would be a huge QoI flaw.> * Provides POSIX extensions on POSIX compliant platforms. > * Provides system-specific extensions as appropriate. For example, > provides the Linux API on Linux. > * Vendor extensions if and only if necessary. > * Designed and developed from the start to work with LLVM tooling and > testing like fuzz testing and sanitizer-supported testing. > * ABI independent implementation as far as possible. > * Use source based implementations as far possible rather than > assembly. Will try to “fix” the compiler rather than use assembly > language workarounds. > > Why a new C Standard Library? > ========================> > Implementing a libc is no small task and is not be taken lightly. AIndeed.> natural question to ask is, "why a new implementation of the C > standard library?" There is no single answer to this question, but > some of the major reasons are as follows: > > * Most libc implementations are monolithic. It is a non-trivial > porting task to pick and choose only the pieces relevant to one's > platform. The new libc will be developed with sufficient modularity to > make picking and choosing a straightforward task.Have you given any thought to what it would mean to make this kind of porting practical? The reason we haven't done it in musl is because it's highly nontrivial. You have to either find an existing point amenable to abstraction that's reasonably common to all existing systems and hope it will apply to future ones too -- for musl, this means the concept of syscalls, which are presently assumed to be Linux ones but could be abstracted *somewhat* further, and might be in the future. If you can't find a suitable point amenable to abstraction that encompasses everything you want to support, then instead you end up making your own abstraction layer in between, and now you're stuck with the task of porting your abstraction layer to every new system you want to support. Now you have an extra layer of bloat, and haven't saved any significant amount of porting work. All of this aside, I agree that it would be rather nice to be "non-monolithic", especially for the parts of libc that are "pure library code" (not depending on any underlying system facilities) to be kept separate and easy to reuse in ports to weird/bare-metal/etc. systems. It'd also be nice for things like stdio that do depend on a system facility, but where the underlying system facility is understood to be "common" at a higher level than syscalls (actual functions on fds) to be able to work with arbitrary implementations of the underlying functions. The reason we didn't do this from the beginning in musl is namespacing; plain C symbols can't depend on symbols in the POSIX namespace.> * Most libc implementations break when built with sanitizer specific > compiler options. The new libc will be developed from the start to > work with those specialized compiler options.This is a nice goal but invites all sorts of circular dependency problems. At some point this will likely be possible with musl too, with the exception of certain components that need to operate at early entry time.> * The new libc will be developed to support and employ fuzz testing > from the start.> * Most libc implementations use a good amount of assembly language, > and assume specific ABIs (may be platform dependent). With the new > libc implementation, we want to use normal source code as much as > possible so that compiler-based changes to the ABI are easy. Moreover, > as part of the LLVM project, we want to use this opportunity to fix > performance related compiler bugs rather than using assembly > workarounds.This is particularly wrong about musl, where use of asm (especially extern asm files as opposed to inline asm) is mostly limited to places where something fundamentally can't be implemented without asm. We don't use asm as a workaround for poor compiler codegen, unless you count things like single-instruction math functions, where it would be really hard for a compiler to pattern-recognize the whole function and reduce it down to the instruction. (Note also that use of __builtin_* doesn't help here because it can create circular definitions if the compiler chooses not to inline the single instruction.)> * A large hole in the llvm toolchain will be plugged with this new > libc.I read this as a confirmation of my concerns from my previous post and tweets, that this looks like you're trying to make "LLVM libc" (or rather "Google libc") the first-class libc for use with clang/LLVM, radically altering the boundaries between tooling and platform, and relegating the existing libc implementations on LLVM's targets to second-class. If this is not the case, can you explain what guarantees we have that this is not what's going on?> With the broad platform expertise in the LLVM community, and the > strong license and project structure, we think that the new libc will > be more tunable and robust, without sacrificing the simplicity and > accessibility typical of the LLVM project.Tunable and robust are usually opposites; see also: uclibc. In summary, I think you're still massively underestimating what an undertaking this is, mistaken about various choices/tradeoffs and whether they make sense, and either not thinking about consequences on ecosystem/monoculture of tightly coupling library with tooling, or intentionally trying to bring about those consequences, contrary to what I see as the best interests of the communities affected. Rich
* Siva Chandra via llvm-dev <llvm-dev at lists.llvm.org> [2019-07-12 08:15:40 -0700]:> Below is a write up which I think would qualify as the "charter" for > the new libc. It is also answering questions like, "where we’ll get > with this libc?", "what's this libc actually going to be?" and similar > ones. I have used libcxx.llvm.org landing page as a template to write > it down.sorry, but this charter does not make much sense. i think you should get some c runtime developers involved.> > ############################################### > > "llvm-libc" C Standard Library > =======================> > llvm-libc is an implementation of the C standard library targeting C11 > and above. It also provides platform specific extensions as relevant. > For example, on Linux it also provides pthreads, librt and other POSIX > extension libraries. > > Documentation > ===========> > The llvm-libc project is still in the planning phase. Stay tuned for > updates soon. > > Features and Goals > ===============> > * C11 and upwards conformant. > * A modular libc with individual pieces implemented in the "as a > library" philosophy of the LLVM project.this is wrong on many levels: LLVM is a bad example of library design, as it has library safety and interface stability issues. libc is necessarily a library so of course it is implemented "as a library". "modular libc" does not make much sense: with satic linking you only link what you use (libc internal dependencies have to be minimized if you static link anyway), with dynamic linking, multiple modules is a huge mistake for other reasons (it creates internal abi between the components that is difficult to manage) if modularity means configurability, then you have a much bigger problem: maintenance and testing becomes harder and if components are actually interchangable then you need stable libc internal interfaces. (this is easy to do in existing libcs, just not done because it's a disaster, ..except in uclibc ..which is a disaster)> * Ability to layer this libc over the system libc. > * Provide C symbols as specified by the standards, but take advantage > and use C++ language facilities for the core implementation.this mistake was already made in bionic, why ppl want to rely on underspecified freestanding c++ language semantics and making assumptions about c++ implementation internals as well as dealing with subtle language incompatibilities when writing a libc is a mistery.> * Provides POSIX extensions on POSIX compliant platforms. > * Provides system-specific extensions as appropriate. For example, > provides the Linux API on Linux. > * Vendor extensions if and only if necessary. > * Designed and developed from the start to work with LLVM tooling and > testing like fuzz testing and sanitizer-supported testing.the difficulty of this is not in the libc, but various issues in the sanitizer libraries.. e.g. you don't want to fuzz test a libc with a fuzz runtime that depends on a c++ runtime (or any other external component that can call back to libc outside of the control of the fuzz runtime).> * ABI independent implementation as far as possible.if you mean call abi, then you get this for free, just use a portable language (such as c), which is what existing libc implementations do anyway. (glibc has a bit more target specifc asm than usual, but most of that has generic fallback code so easy to drop the asm) if you mean other abis then clarify.> * Use source based implementations as far possible rather than > assembly. Will try to “fix” the compiler rather than use assembly > language workarounds.same as above, all libcs do this already.> > Why a new C Standard Library? > ========================> > Implementing a libc is no small task and is not be taken lightly. A > natural question to ask is, "why a new implementation of the C > standard library?" There is no single answer to this question, but > some of the major reasons are as follows: > > * Most libc implementations are monolithic. It is a non-trivial > porting task to pick and choose only the pieces relevant to one's > platform. The new libc will be developed with sufficient modularity to > make picking and choosing a straightforward task.this does not make sense (see above).> * Most libc implementations break when built with sanitizer specific > compiler options. The new libc will be developed from the start to > work with those specialized compiler options.this does not make sense. (new libc does not make this easier: e.g. adding asan support to musl is about a week work and most issues are sanitizer related problems where you have to give up correctness or reliability for sanitizers to work)> * The new libc will be developed to support and employ fuzz testing > from the start.this does not make sense. (new libc does not make this easier)> * Most libc implementations use a good amount of assembly language, > and assume specific ABIs (may be platform dependent). With the new > libc implementation, we want to use normal source code as much as > possible so that compiler-based changes to the ABI are easy. Moreover, > as part of the LLVM project, we want to use this opportunity to fix > performance related compiler bugs rather than using assembly > workarounds.citation needed. (removing all unnecessary asm from musl and replacing it with intrinsics is a weekend project)> * A large hole in the llvm toolchain will be plugged with this new > libc. With the broad platform expertise in the LLVM community, and the > strong license and project structure, we think that the new libc will > be more tunable and robust, without sacrificing the simplicity and > accessibility typical of the LLVM project.for this hole to be filled in on linux you need distros to be able to rebuild everything against the new libc, otherwise you just have a new libc that nobody can use because applications have dependencies that have to be built against the same libc. in some contexts it is enough to have a libc with build scripts to build the dependencies.. but existing build scripts often don't work out of the box with a new libc. creating a software platform around a libc is a much bigger task than the libc itself, and without that the libc has limited value. it seems to me that none of the answers make much sense as is.> > Platform Support > =============> > llvm-libc development is still in the planning phase. However, we > envision that it will support a variety of platforms in the coming > years. Interested parties are encouraged to participate in the design > and implementation, and add support for their favorite platforms.the problem is not porting the new libc to a platfrom but porting the platform to a new libc. i.e. you need to worry a lot more about what's above the libc (all userspace software) than what's below it (tiny os kernel interface).> > Current Status > ===========> > llvm-libc development is still in the planning phase. > > Build Bots > ========> > Coming soon. > > Get involved! > ==========> > First please review our Developer's Policy. Stay tuned for llvm-libc > specific information. > > Design Documents > ==============> > Coming soon. > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
On Fri, Jul 12, 2019 at 12:29 PM Rich Felker via llvm-dev < llvm-dev at lists.llvm.org> wrote:> On Fri, Jul 12, 2019 at 02:54:40PM -0400, Siva Chandra wrote: > > * A large hole in the llvm toolchain will be plugged with this new > > libc. > > I read this as a confirmation of my concerns from my previous post and > tweets, that this looks like you're trying to make "LLVM libc" (or > rather "Google libc") the first-class libc for use with clang/LLVM, > radically altering the boundaries between tooling and platform, and > relegating the existing libc implementations on LLVM's targets to > second-class. > > If this is not the case, can you explain what guarantees we have that > this is not what's going on? >I'm surprised that you would think this given the rest of the llvm project. We have: * clang, yet all of llvm still builds with gcc, msvc, icc, edg, etc... * lldb, yet you can still debug llvm produced binaries with gdb, and even msvc's debugger. * lld, yet you can still link with gnu-ld, gold, link.exe, ld64. * libc++, yet clang still works with libstdc++, dinkumware, and the msvc stdlib. * libc++abi, yet libc++ still works with libsupc++ and other c++ abi libs. * libunwind, yet libc++abi works with other unwinders. * compiler-rt (builtins), yet you can still use libgcc. * compiler-rt (sanitizers), yet gcc uses them. It has been well proven that llvm project alternatives do not push out or harm non-llvm compatibility. We even cooperate with non-llvm projects on changes and new features where it makes sense. What makes you think a llvm libc would be any different? - Michael Spencer -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190712/76d1f4a4/attachment.html>
On Fri, Jul 12, 2019 at 4:19 PM Szabolcs Nagy via llvm-dev < llvm-dev at lists.llvm.org> wrote:> > * Designed and developed from the start to work with LLVM tooling and > > testing like fuzz testing and sanitizer-supported testing. > > the difficulty of this is not in the libc, but various > issues in the sanitizer libraries.. e.g. you don't want to > fuzz test a libc with a fuzz runtime that depends on a c++ > runtime (or any other external component that can call > back to libc outside of the control of the fuzz runtime). >FWIW, I was able to fuzz-test some functions in musl using the AFL fuzzer with only small changes (yes, this is not LLVM libFuzzer -- I haven't gotten around to tryingthat yet). It required only minor modifications to the musl makefile and to AFL. The existing "NOSSP_OBJS" listed in the musl Makefile were exactly the correct ones that needed to be excluded from fuzz coverage instrumentation. I compiled the rest of the library with "afl-gcc", and those files with plain "gcc". Additionally, a 2-line change was needed in AFL to avoid trying to recursively re-enter initialization while it was already in progress (startup initialization calls getenv which then tries to call back into initialization, because initialization hasn't completed yet). Other than initialization, the instrumentation doesn't call back into libc, so just suppressing that recursion is sufficient. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190712/68195da0/attachment.html>
On Fri, Jul 12, 2019 at 8:16 AM Siva Chandra via llvm-dev <llvm-dev at lists.llvm.org> wrote:> > On Fri, Jun 28, 2019 at 9:29 AM JF Bastien <jfbastien at apple.com> wrote: > > > > I think I now understand some of the disconnect you and I are having, and I think some of the pushback you’re getting from the community is the same. You’re talking about where you want to start with an LLVM libc. Many in the community (myself included) want to understand where we’ll get with this libc. At steady-state, what does it do? To a certain degree I don’t care about how you get to the steady state: sure the implementation approach is important, and which contributor cares about what parts is important in shaping that evolution, but at the end of the day what matters is where you get. > > > > So here’s what’s missing: there’s no goal. Right now, your proposal is “let’s do an LLVM libc, starting with what I care about, who’s interested?” > > > > That’s an OK place to start! You illustrated your needs, others chimed in with theirs, and now you know there’s some interest. However, you should take time now to come up with a plan. What’s this libc actually going to be? I ask a bunch of questions below that I think you need to answer as a next step. Others asked more questions which I didn’t echo, but which you should answer as well. What does this libc aspire to become? > > I apologize for the delay. I will try to address the above questions > in this email. I will shortly follow up with answers to other > questions. > > Below is a write up which I think would qualify as the "charter" for > the new libc. It is also answering questions like, "where we’ll get > with this libc?", "what's this libc actually going to be?" and similar > ones. I have used libcxx.llvm.org landing page as a template to write > it down. > > ############################################### > > "llvm-libc" C Standard Library > =======================> > llvm-libc is an implementation of the C standard library targeting C11 > and above. It also provides platform specific extensions as relevant. > For example, on Linux it also provides pthreads, librt and other POSIX > extension libraries. > > Documentation > ===========> > The llvm-libc project is still in the planning phase. Stay tuned for > updates soon. > > Features and Goals > ===============> > * C11 and upwards conformant. > * A modular libc with individual pieces implemented in the "as a > library" philosophy of the LLVM project. > * Ability to layer this libc over the system libc. > * Provide C symbols as specified by the standards, but take advantage > and use C++ language facilities for the core implementation. > * Provides POSIX extensions on POSIX compliant platforms. > * Provides system-specific extensions as appropriate. For example, > provides the Linux API on Linux. > * Vendor extensions if and only if necessary. > * Designed and developed from the start to work with LLVM tooling and > testing like fuzz testing and sanitizer-supported testing. > * ABI independent implementation as far as possible. > * Use source based implementations as far possible rather than > assembly. Will try to “fix” the compiler rather than use assembly > language workarounds. > > Why a new C Standard Library? > ========================> > Implementing a libc is no small task and is not be taken lightly. A > natural question to ask is, "why a new implementation of the C > standard library?" There is no single answer to this question, but > some of the major reasons are as follows: > > * Most libc implementations are monolithic. It is a non-trivial > porting task to pick and choose only the pieces relevant to one's > platform. The new libc will be developed with sufficient modularity to > make picking and choosing a straightforward task. > * Most libc implementations break when built with sanitizer specific > compiler options. The new libc will be developed from the start to > work with those specialized compiler options. > * The new libc will be developed to support and employ fuzz testing > from the start. > * Most libc implementations use a good amount of assembly language, > and assume specific ABIs (may be platform dependent). With the new > libc implementation, we want to use normal source code as much as > possible so that compiler-based changes to the ABI are easy. Moreover, > as part of the LLVM project, we want to use this opportunity to fix > performance related compiler bugs rather than using assembly > workarounds. > * A large hole in the llvm toolchain will be plugged with this new > libc. With the broad platform expertise in the LLVM community, and the > strong license and project structure, we think that the new libc will > be more tunable and robust, without sacrificing the simplicity and > accessibility typical of the LLVM project. >There are options like musl which is relatively new and might not have as much legacy. I am sure you have looked at various system C libraries out there for consideration, it would be interesting to share the insights on differences.> Platform Support > =============> > llvm-libc development is still in the planning phase. However, we > envision that it will support a variety of platforms in the coming > years. Interested parties are encouraged to participate in the design > and implementation, and add support for their favorite platforms. > > Current Status > ===========> > llvm-libc development is still in the planning phase. > > Build Bots > ========> > Coming soon. > > Get involved! > ==========> > First please review our Developer's Policy. Stay tuned for llvm-libc > specific information. > > Design Documents > ==============> > Coming soon. > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
On Fri, Jul 12, 2019 at 8:32 AM Aaron Ballman <aaron at aaronballman.com> wrote:> > llvm-libc is an implementation of the C standard library targeting C11 > > and above. > > Any particular reason for C11 as opposed to C17?Two reasons: 1. The C++17 standard refers to the C11 standard. 2. C11 is sufficiently modern while not closing doors for users requiring compliance with an "older" standards. That said, we could choose not to implement certain items removed in the C17 standard. An obvious example of such a candidate is `gets`.
Hello again, I have now committed a proposal for this new libc: https://reviews.llvm.org/D64939. This proposal is based on the charter I shared previously (see below). Though it is now committed, if there are any concerns still, I am open to iterating until the community in general is satisfied with the proposal. Thanks, Siva Chandra On Fri, Jul 12, 2019 at 8:15 AM Siva Chandra <sivachandra at google.com> wrote:> "llvm-libc" C Standard Library > =======================> > llvm-libc is an implementation of the C standard library targeting C11 > and above. It also provides platform specific extensions as relevant. > For example, on Linux it also provides pthreads, librt and other POSIX > extension libraries. > > Documentation > ===========> > The llvm-libc project is still in the planning phase. Stay tuned for > updates soon. > > Features and Goals > ===============> > * C11 and upwards conformant. > * A modular libc with individual pieces implemented in the "as a > library" philosophy of the LLVM project. > * Ability to layer this libc over the system libc. > * Provide C symbols as specified by the standards, but take advantage > and use C++ language facilities for the core implementation. > * Provides POSIX extensions on POSIX compliant platforms. > * Provides system-specific extensions as appropriate. For example, > provides the Linux API on Linux. > * Vendor extensions if and only if necessary. > * Designed and developed from the start to work with LLVM tooling and > testing like fuzz testing and sanitizer-supported testing. > * ABI independent implementation as far as possible. > * Use source based implementations as far possible rather than > assembly. Will try to “fix” the compiler rather than use assembly > language workarounds. > > Why a new C Standard Library? > ========================> > Implementing a libc is no small task and is not be taken lightly. A > natural question to ask is, "why a new implementation of the C > standard library?" There is no single answer to this question, but > some of the major reasons are as follows: > > * Most libc implementations are monolithic. It is a non-trivial > porting task to pick and choose only the pieces relevant to one's > platform. The new libc will be developed with sufficient modularity to > make picking and choosing a straightforward task. > * Most libc implementations break when built with sanitizer specific > compiler options. The new libc will be developed from the start to > work with those specialized compiler options. > * The new libc will be developed to support and employ fuzz testing > from the start. > * Most libc implementations use a good amount of assembly language, > and assume specific ABIs (may be platform dependent). With the new > libc implementation, we want to use normal source code as much as > possible so that compiler-based changes to the ABI are easy. Moreover, > as part of the LLVM project, we want to use this opportunity to fix > performance related compiler bugs rather than using assembly > workarounds. > * A large hole in the llvm toolchain will be plugged with this new > libc. With the broad platform expertise in the LLVM community, and the > strong license and project structure, we think that the new libc will > be more tunable and robust, without sacrificing the simplicity and > accessibility typical of the LLVM project. > > Platform Support > =============> > llvm-libc development is still in the planning phase. However, we > envision that it will support a variety of platforms in the coming > years. Interested parties are encouraged to participate in the design > and implementation, and add support for their favorite platforms. > > Current Status > ===========> > llvm-libc development is still in the planning phase. > > Build Bots > ========> > Coming soon. > > Get involved! > ==========> > First please review our Developer's Policy. Stay tuned for llvm-libc > specific information. > > Design Documents > ==============> > Coming soon.
On Thu, 15 Aug 2019 at 19:24, Siva Chandra via llvm-dev <llvm-dev at lists.llvm.org> wrote:> > Hello again, > > I have now committed a proposal for this new libc: > https://reviews.llvm.org/D64939. This proposal is based on the charter > I shared previously (see below). > > Though it is now committed, if there are any concerns still, I am open > to iterating until the community in general is satisfied with the > proposal.Hi Siva, Thanks for writing up this proposal and your work in getting it committed etc. For future cases where an RFC results in a proposal patch, it's helpful to respond to the RFC thread with a link (not everyone who is interested will catch it on llvm-commits). I'm looking forward to seeing the next steps for this effort. Best, Alex
Hello again, I would like to announce that we now have dedicated mailing lists for the llvm-libc project: 1. libc-dev - https://lists.llvm.org/cgi-bin/mailman/listinfo/libc-dev 2. libc-commits - https://lists.llvm.org/cgi-bin/mailman/listinfo/libc-commits A brief README.txt and LICENSE.txt have been added: https://github.com/llvm/llvm-project/tree/master/libc I will soon share or put out design docs and code for review, as suitable. Thanks, Siva Chandra On Thu, Aug 15, 2019 at 11:23 AM Siva Chandra <sivachandra at google.com> wrote:> Hello again, > > I have now committed a proposal for this new libc: > https://reviews.llvm.org/D64939. This proposal is based on the charter > I shared previously (see below). > > Though it is now committed, if there are any concerns still, I am open > to iterating until the community in general is satisfied with the > proposal. > > Thanks, > Siva Chandra > > On Fri, Jul 12, 2019 at 8:15 AM Siva Chandra <sivachandra at google.com> > wrote: > > "llvm-libc" C Standard Library > > =======================> > > > llvm-libc is an implementation of the C standard library targeting C11 > > and above. It also provides platform specific extensions as relevant. > > For example, on Linux it also provides pthreads, librt and other POSIX > > extension libraries. > > > > Documentation > > ===========> > > > The llvm-libc project is still in the planning phase. Stay tuned for > > updates soon. > > > > Features and Goals > > ===============> > > > * C11 and upwards conformant. > > * A modular libc with individual pieces implemented in the "as a > > library" philosophy of the LLVM project. > > * Ability to layer this libc over the system libc. > > * Provide C symbols as specified by the standards, but take advantage > > and use C++ language facilities for the core implementation. > > * Provides POSIX extensions on POSIX compliant platforms. > > * Provides system-specific extensions as appropriate. For example, > > provides the Linux API on Linux. > > * Vendor extensions if and only if necessary. > > * Designed and developed from the start to work with LLVM tooling and > > testing like fuzz testing and sanitizer-supported testing. > > * ABI independent implementation as far as possible. > > * Use source based implementations as far possible rather than > > assembly. Will try to “fix” the compiler rather than use assembly > > language workarounds. > > > > Why a new C Standard Library? > > ========================> > > > Implementing a libc is no small task and is not be taken lightly. A > > natural question to ask is, "why a new implementation of the C > > standard library?" There is no single answer to this question, but > > some of the major reasons are as follows: > > > > * Most libc implementations are monolithic. It is a non-trivial > > porting task to pick and choose only the pieces relevant to one's > > platform. The new libc will be developed with sufficient modularity to > > make picking and choosing a straightforward task. > > * Most libc implementations break when built with sanitizer specific > > compiler options. The new libc will be developed from the start to > > work with those specialized compiler options. > > * The new libc will be developed to support and employ fuzz testing > > from the start. > > * Most libc implementations use a good amount of assembly language, > > and assume specific ABIs (may be platform dependent). With the new > > libc implementation, we want to use normal source code as much as > > possible so that compiler-based changes to the ABI are easy. Moreover, > > as part of the LLVM project, we want to use this opportunity to fix > > performance related compiler bugs rather than using assembly > > workarounds. > > * A large hole in the llvm toolchain will be plugged with this new > > libc. With the broad platform expertise in the LLVM community, and the > > strong license and project structure, we think that the new libc will > > be more tunable and robust, without sacrificing the simplicity and > > accessibility typical of the LLVM project. > > > > Platform Support > > =============> > > > llvm-libc development is still in the planning phase. However, we > > envision that it will support a variety of platforms in the coming > > years. Interested parties are encouraged to participate in the design > > and implementation, and add support for their favorite platforms. > > > > Current Status > > ===========> > > > llvm-libc development is still in the planning phase. > > > > Build Bots > > ========> > > > Coming soon. > > > > Get involved! > > ==========> > > > First please review our Developer's Policy. Stay tuned for llvm-libc > > specific information. > > > > Design Documents > > ==============> > > > Coming soon. >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190916/ccdcb171/attachment.html>