On Mon, Jun 24, 2019 at 3:37 PM Jake Ehrlich <jakehehrlich at google.com> wrote:> disclaimer: I work at Google so don't take my +1 as an independent vote > forward. > > We would like to use this on Fuchsia and I am particularly interested in > creating a dynamic linking library for ELF with Roland McGrath's guidance. > We spoke about creating a library for writing dynamic linkers internally > and I don't see why this can't be upstreamed. >If dynamic linking support is added in a "as a library" fashion, so that it can easily be excluded if not required without affecting the rest of the system, I do not see any problems adding it.> On Fuchsia we critically need support for AArch64; What do you expect to > be architecture dependent? I struggled to think of where the architecture > and not the operating system was the issue. >I think syscalls are an example of being architecture specific? And, items like program startup and PIE loader are operating system/exe format specific? Just for my knowledge, why is answering these questions at a general level important?> On Mon, Jun 24, 2019 at 3:23 PM Siva Chandra via llvm-dev < > llvm-dev at lists.llvm.org> wrote: > >> Hello LLVM Developers, >> >> Within Google, we have a growing range of needs that existing libc >> implementations don't quite address. This is pushing us to start working on >> a new libc implementation. >> >> Informal conversations with others within the LLVM community has told us >> that a libc in LLVM is actually a broader need, and we are increasingly >> consolidating our toolchains around LLVM. Hence, we wanted to see if the >> LLVM project would be interested in us developing this upstream as part of >> the project. >> >> To be very clear: we don't expect our needs to exactly match everyone >> else's -- part of our impetus is to simplify things wherever we can, and >> that may not quite match what others want in a libc. That said, we do >> believe that the effort will still be directly beneficial and usable for >> the broader LLVM community, and may serve as a starting point for others in >> the community to flesh out an increasingly complete set of libc >> functionality. >> >> We are still in the early stages, but we do have some high-level goals >> and guiding principles of the initial scope we are interested in pursuing: >> >> >> 1. >> >> The project should mesh with the "as a library" philosophy of the >> LLVM project: even though "the C Standard Library" is nominally "a >> library," most implementations are, in practice, quite monolithic. >> 2. >> >> The libc should support static non-PIE and static-PIE linking. This >> means, providing the CRT (the C runtime) and a PIE loader for static >> non-PIE and static-PIE linked executables. >> 3. >> >> If there is a specification, we should follow it. The scope that we >> need includes most of the C Standard Library; POSIX additions; and some >> necessary, system-specific extensions. This does not mean we should (or >> can) follow the entire specification -- there will be some parts which >> simply aren't worth implementing, and some parts which cannot be safely >> used in modern coding practice. >> 4. >> >> Vendor extensions must be considered very carefully, and only >> admitted when necessary. Similar to Clang and libc++, it does seem >> inevitable that we will need to provide some level of compatibility with >> other vendors' extensions. >> 5. >> >> The project should be an exemplar of developing with LLVM tooling. >> Two examples are fuzz testing from the start, and sanitizer-supported >> testing. >> >> >> There are also few areas which we do not intend to invest in at this >> point: >> >> >> 1. >> >> Implement dynamic loading and linking support. >> 2. >> >> Support for more architectures (we'll start with just x86-64 for >> simplicity). >> >> >> For these areas, the community is of course free to contribute. Our hope >> is that, preserving the "as a library" design philosophy will make such >> extensions easy, and allow retaining the simplicity when these features >> aren't needed. >> >> We intend to build the new libc in a gradual manner. To begin with, the >> new libc will be a layer sitting between the application and the system >> libc. Eventually, when the implementation is sufficiently complete, it will >> be able to replace the system libc at least for some use cases and contexts. >> >> So, what do you think about incorporating this new libc under the LLVM >> project? >> >> Thank you, >> >> Siva Chandra and the rest of the Google LLVM contributors >> >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org >> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190625/64c56a75/attachment-0001.html>
Syscalls are operating system specific and architecture dependent so I think we'll want an abstraction layer around the fundamental operations the syscalls support anyway. Some things like open aren't even syscalls on all operating systems. There might be a generic syscall layer added that would be architecture and not operating system specific but even on x86_64 there are two different ways to do syscalls I think. Loading, startup, and linking are all both format and operating system specific and a few of these details involved are determined by the architecture but they're trivially abstracted away. why is answering these questions at a general level important? Because I wanted to make sure I understood the direction and the restriction stated. The restriction on what architecture will be used without stating a restriction on the operating system seemed like an odd statement. I'd very much like operating system abstractions to be considered right out of the gate and this seems like a bigger issue than the architecture to me. On Tue, Jun 25, 2019 at 3:47 PM Siva Chandra <sivachandra at google.com> wrote:> On Mon, Jun 24, 2019 at 3:37 PM Jake Ehrlich <jakehehrlich at google.com> > wrote: > >> disclaimer: I work at Google so don't take my +1 as an independent vote >> forward. >> >> We would like to use this on Fuchsia and I am particularly interested in >> creating a dynamic linking library for ELF with Roland McGrath's guidance. >> We spoke about creating a library for writing dynamic linkers internally >> and I don't see why this can't be upstreamed. >> > > If dynamic linking support is added in a "as a library" fashion, so that > it can easily be excluded if not required without affecting the rest of the > system, I do not see any problems adding it. > > >> On Fuchsia we critically need support for AArch64; What do you expect to >> be architecture dependent? I struggled to think of where the architecture >> and not the operating system was the issue. >> > > I think syscalls are an example of being architecture specific? And, items > like program startup and PIE loader are operating system/exe format > specific? > > Just for my knowledge, why is answering these questions at a general level > important? > > >> On Mon, Jun 24, 2019 at 3:23 PM Siva Chandra via llvm-dev < >> llvm-dev at lists.llvm.org> wrote: >> >>> Hello LLVM Developers, >>> >>> Within Google, we have a growing range of needs that existing libc >>> implementations don't quite address. This is pushing us to start working on >>> a new libc implementation. >>> >>> Informal conversations with others within the LLVM community has told us >>> that a libc in LLVM is actually a broader need, and we are increasingly >>> consolidating our toolchains around LLVM. Hence, we wanted to see if the >>> LLVM project would be interested in us developing this upstream as part of >>> the project. >>> >>> To be very clear: we don't expect our needs to exactly match everyone >>> else's -- part of our impetus is to simplify things wherever we can, and >>> that may not quite match what others want in a libc. That said, we do >>> believe that the effort will still be directly beneficial and usable for >>> the broader LLVM community, and may serve as a starting point for others in >>> the community to flesh out an increasingly complete set of libc >>> functionality. >>> >>> We are still in the early stages, but we do have some high-level goals >>> and guiding principles of the initial scope we are interested in pursuing: >>> >>> >>> 1. >>> >>> The project should mesh with the "as a library" philosophy of the >>> LLVM project: even though "the C Standard Library" is nominally "a >>> library," most implementations are, in practice, quite monolithic. >>> 2. >>> >>> The libc should support static non-PIE and static-PIE linking. This >>> means, providing the CRT (the C runtime) and a PIE loader for static >>> non-PIE and static-PIE linked executables. >>> 3. >>> >>> If there is a specification, we should follow it. The scope that we >>> need includes most of the C Standard Library; POSIX additions; and some >>> necessary, system-specific extensions. This does not mean we should (or >>> can) follow the entire specification -- there will be some parts which >>> simply aren't worth implementing, and some parts which cannot be safely >>> used in modern coding practice. >>> 4. >>> >>> Vendor extensions must be considered very carefully, and only >>> admitted when necessary. Similar to Clang and libc++, it does seem >>> inevitable that we will need to provide some level of compatibility with >>> other vendors' extensions. >>> 5. >>> >>> The project should be an exemplar of developing with LLVM tooling. >>> Two examples are fuzz testing from the start, and sanitizer-supported >>> testing. >>> >>> >>> There are also few areas which we do not intend to invest in at this >>> point: >>> >>> >>> 1. >>> >>> Implement dynamic loading and linking support. >>> 2. >>> >>> Support for more architectures (we'll start with just x86-64 for >>> simplicity). >>> >>> >>> For these areas, the community is of course free to contribute. Our hope >>> is that, preserving the "as a library" design philosophy will make such >>> extensions easy, and allow retaining the simplicity when these features >>> aren't needed. >>> >>> We intend to build the new libc in a gradual manner. To begin with, the >>> new libc will be a layer sitting between the application and the system >>> libc. Eventually, when the implementation is sufficiently complete, it will >>> be able to replace the system libc at least for some use cases and contexts. >>> >>> So, what do you think about incorporating this new libc under the LLVM >>> project? >>> >>> Thank you, >>> >>> Siva Chandra and the rest of the Google LLVM contributors >>> >>> _______________________________________________ >>> LLVM Developers mailing list >>> llvm-dev at lists.llvm.org >>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >>> >>-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190625/e8779abf/attachment.html>
On Tue, Jun 25, 2019 at 4:05 PM Jake Ehrlich <jakehehrlich at google.com> wrote:> Syscalls are operating system specific and architecture dependent so I > think we'll want an abstraction layer around the fundamental operations the > syscalls support anyway. Some things like open aren't even syscalls on all > operating >Right, syscalls are OS _and_ architecture dependent. So yes, one will have to build abstraction layers over fundamental operations in general.> systems. There might be a generic syscall layer added that would be > architecture and not operating system specific but even on x86_64 there are > two different ways to do syscalls I think. Loading, startup, and linking > are all both format and operating system specific and a few of these > details involved are determined by the architecture but they're trivially > abstracted away. > > why is answering these questions at a general level important? > > > Because I wanted to make sure I understood the direction and the > restriction stated. The restriction on what architecture will be used > without stating a restriction on the operating system seemed like an odd > statement. I'd very much like operating system abstractions to be > considered right out of the gate and this seems like a bigger issue than > the architecture to me. >Ah, I see what happened. So, we are definitely not restricting anything by design here. All we are saying is that we do not intend to contribute beyond x86_64 and Linux to begin with. The community is free to contribute and widen the scope as suitable. With respect to how exactly we want to build the abstractions, I am of the opinion that we have to go on a case by case basis. The scope of the project is so large that I think it is more meaningful to discuss designs at a more narrow level based on the area that is being worked on. Sure, we might end up discovering patterns down the road and choose to unify certain things eventually. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190625/fd0236c9/attachment.html>
On Tue, Jun 25, 2019 at 03:47:15PM -0700, Siva Chandra via llvm-dev wrote:> On Mon, Jun 24, 2019 at 3:37 PM Jake Ehrlich <jakehehrlich at google.com> > wrote: > > > disclaimer: I work at Google so don't take my +1 as an independent vote > > forward. > > > > We would like to use this on Fuchsia and I am particularly interested in > > creating a dynamic linking library for ELF with Roland McGrath's guidance. > > We spoke about creating a library for writing dynamic linkers internally > > and I don't see why this can't be upstreamed. > > > > If dynamic linking support is added in a "as a library" fashion, so that it > can easily be excluded if not required without affecting the rest of the > system, I do not see any problems adding it.It would be very cool to see an RTLD in llvm, especially one that could support multiple formats, like ELF and LLVM IR. If I understand llvm's Cross-DSO CFI support properly (which I may not), the entire address space for a dlopen()ed DSO (as in, where the DSO was loaded in memory) is added to the cfi-icall whitelist. An RTLD in llvm that can intelligently integrate with the sanitizers to provide the same level of granularity for dlopen()ed libraries as ELF::DT_NEEDED ones. One could also complete the required SafeStack integration with the RTLD, which requires bringing in the sanitizer framework, anyways. A portable libc and RTLD provided by llvm that can make integral use of the sanitizers, especially CFI and SafeStack, would be absolutely lovely. If not done within llvm, HardenedBSD needs to do it, anyways, in order to apply SafeStack and Cross-DSO CFI properly to both DSOs and applications. If not done within llvm, HardenedBSD will simply continue using and maintaining patches on top of the libc and RTLD we inherit from our upstream FreeBSD. Thanks, -- Shawn Webb Cofounder / Security Engineer HardenedBSD Tor-ified Signal: +1 443-546-8752 Tor+XMPP+OTR: lattera at is.a.hacker.sx GPG Key ID: 0xFF2E67A277F8E1FA GPG Key Fingerprint: D206 BB45 15E0 9C49 0CF9 3633 C85B 0AF8 AB23 0FB2 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190626/d0319763/attachment.sig>