Dmitry Golovin via llvm-dev
2017-Jan-19 21:56 UTC
[llvm-dev] Linking Linux kernel with LLD
Hi all! Recently there was a discussion on this mailing list about linking Linux and GRUB with LLD. I have actually tried to link Linux kernel with LLD and (with some modifications) it linked successfully (but didn't run well). Here is the list of modifications I had to do in order to link the kernel (I have used llvmlinux with clang and mainline with gcc, the results are similar): 1. LLD patches: - D28094 (Implemented support for R_386_PC8/R_386_8 relocations) - D28857 (Allow relative relocations to a absolute value defined in linker script) - [not actually used] D28612 (Added support for --emit-relocs) 2. Kernel configuration. I tried "make tinyconfig" and then changed to CONFIG_64BIT=y with menuconfig. I wanted the minimal kernel configuration for x86_64. After I get it working, I will try defconfig. 3. In arch/x86/Makefile changed the M16_CFLAGS to just "-m16". I think that cc-option should test if compiler supports '-m16' option and if it does, use it, if it doesn't use '-m32 -Wa,./arch/x86/boot/code16gcc.h' instead. I don't know why, but this test failed for me and LLD didn't accept this header, so I just simply changed the flag to '-m16' because I didn't want to investigate why did the test failed. It's just a quick workaround, but it works. 3. In arch/x86/boot/Makefile and arch/x86/realmode/rm/Makefile added "-m elf_i386" to LDFLAGS_setup.elf and LDFLAGS_realmode.elf accordingly. The problem is that the linker is invoked as "ld.lld -m elf_x86_64" and it ignores the OUTPUT_FORMAT from linker scripts. Being invoked as "ld.lld -m elf_x86_64 -m elf_i386" doesn't break anything and LLD even produces the right format of the binaries. Again a quick workaround, but it works. 4. In arch/x86/realmode/rm/Makefile removed "--emit-relocs" from LDFLAGS_realmode.elf. This is the problem with D28612 — I applied the patch to support --emit-relocs option, but LLD segfaulted. Reported here: https://llvm.org/bugs/show_bug.cgi?id=31579#c8 5. In arch/x86/kernel/vmlinux.lds.S commented out the "CONSTRUCTORS", because LLD doesn't support it. 6. In arch/x86/boot/setup.ld replaced 5*512 with precalculated value of 2560 because it doesn't seem that LLD supports math inside ASSERT in linker scripts. 7. In arch/x86/boot/setup.ld commented out the line ". = ASSERT(_end <= 0x8000, "Setup too big!");" because it seems to always fire this assertion, and the kernel is not going to be working anyway, so I just want it linked even if it will be unusable. 8. In arch/x86/lib/copy_user_64.S removed the line "bad_from_user:" (probably it was just a mistake), in arch/x86/kernel/vmlinux.lds.S removed semicolon after "*(.apicdrivers)" (also probably just a mistake). Finally the kernel was built and it obviously didn't run (only printed out "No setup signature found..." but this is some result as well). Probably, the result could be better if the --emit-relocs option didn't fail and CONSTRUCTORS were supported. I really don't know what to do about the assertion that I have commented out. I also tried using tools provided by LLVM instead of binutils (HOSTCC=clang HOSTLD=ld.lld CC=clang LD=ld.lld AR=llvm-ar NM=llvm-nm), it seems to work with llvmlinux, but clang will not work with the mainline. Clang doesn't seem to use integrated assembler, so I had to write a simple wrapper script around llvm-mc to use instead GNU as. I also have few patches to make llvm-objcopy to work with latest LLVM and support needed subset of flags, will publish it soon. You can find my workarounds as a patch (the kernel will not work with this applied!) and my kernel config attached. Regards, Dmitry -------------- next part -------------- A non-text attachment was scrubbed... Name: kernel-lld-workarounds.patch Type: text/x-diff Size: 3072 bytes Desc: not available URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170119/d0f7032b/attachment.patch> -------------- next part -------------- A non-text attachment was scrubbed... Name: kernel-config Type: application/octet-stream Size: 21013 bytes Desc: not available URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170119/d0f7032b/attachment.obj>
Wow, great work! I'm sure you are the first person who linked the Linux kernel using LLD. :) I think "no setup signature found" error is an error of the boot loader (probably grub). The resulting binary probably lacked magic bytes or something that a boot loader uses to identify bootable images. It should be fixable by investigating what is actually missing and fix it. On Thu, Jan 19, 2017 at 1:56 PM, Dmitry Golovin via llvm-dev < llvm-dev at lists.llvm.org> wrote:> Hi all! > > Recently there was a discussion on this mailing list about linking Linux > and GRUB with LLD. > > I have actually tried to link Linux kernel with LLD and (with some > modifications) it linked successfully (but didn't run well). > > Here is the list of modifications I had to do in order to link the kernel > (I have used llvmlinux with clang and mainline with gcc, the results are > similar): > > 1. LLD patches: > - D28094 (Implemented support for R_386_PC8/R_386_8 relocations) > - D28857 (Allow relative relocations to a absolute value defined in > linker script) > - [not actually used] D28612 (Added support for --emit-relocs) > > 2. Kernel configuration. I tried "make tinyconfig" and then changed to > CONFIG_64BIT=y with menuconfig. I wanted the minimal kernel configuration > for x86_64. After I get it working, I will try defconfig. > > 3. In arch/x86/Makefile changed the M16_CFLAGS to just "-m16". I think > that cc-option should test if compiler supports '-m16' option and if it > does, use it, if it doesn't use '-m32 -Wa,./arch/x86/boot/code16gcc.h' > instead. I don't know why, but this test failed for me and LLD didn't > accept this header, so I just simply changed the flag to '-m16' because I > didn't want to investigate why did the test failed. It's just a quick > workaround, but it works. > > 3. In arch/x86/boot/Makefile and arch/x86/realmode/rm/Makefile added "-m > elf_i386" to LDFLAGS_setup.elf and LDFLAGS_realmode.elf accordingly. The > problem is that the linker is invoked as "ld.lld -m elf_x86_64" and it > ignores the OUTPUT_FORMAT from linker scripts. Being invoked as "ld.lld -m > elf_x86_64 -m elf_i386" doesn't break anything and LLD even produces the > right format of the binaries. Again a quick workaround, but it works. > > 4. In arch/x86/realmode/rm/Makefile removed "--emit-relocs" from > LDFLAGS_realmode.elf. This is the problem with D28612 — I applied the patch > to support --emit-relocs option, but LLD segfaulted. Reported here: > https://llvm.org/bugs/show_bug.cgi?id=31579#c8 > > 5. In arch/x86/kernel/vmlinux.lds.S commented out the "CONSTRUCTORS", > because LLD doesn't support it. >We need to support those missing features.> 6. In arch/x86/boot/setup.ld replaced 5*512 with precalculated value of > 2560 because it doesn't seem that LLD supports math inside ASSERT in linker > scripts. >So this is the line. . = ASSERT(__end_init <= 5*512, "init sections too big!"); and LLD should be able to handle this pattern already, so I think you found a bug. We need to fix that. (The code to handle ASSERT is here <https://github.com/llvm-project/llvm-project/blob/master/lld/ELF/LinkerScript.cpp#L1396>, so if you are interested, you can take a look. It is a simple recursive-descendent parser, and because readAssert calls readExpr (as opposed to next() which returns the next token), it is intended to be able to handle an expression inside ASSERT.)> 7. In arch/x86/boot/setup.ld commented out the line ". = ASSERT(_end <> 0x8000, "Setup too big!");" because it seems to always fire this assertion, > and the kernel is not going to be working anyway, so I just want it linked > even if it will be unusable. >That's a weird issue... need to take a look deeper.> 8. In arch/x86/lib/copy_user_64.S removed the line "bad_from_user:" > (probably it was just a mistake), in arch/x86/kernel/vmlinux.lds.S removed > semicolon after "*(.apicdrivers)" (also probably just a mistake). > > Finally the kernel was built and it obviously didn't run (only printed out > "No setup signature found..." but this is some result as well). Probably, > the result could be better if the --emit-relocs option didn't fail and > CONSTRUCTORS were supported. I really don't know what to do about the > assertion that I have commented out. > > I also tried using tools provided by LLVM instead of binutils > (HOSTCC=clang HOSTLD=ld.lld CC=clang LD=ld.lld AR=llvm-ar NM=llvm-nm), it > seems to work with llvmlinux, but clang will not work with the mainline. > Clang doesn't seem to use integrated assembler, so I had to write a simple > wrapper script around llvm-mc to use instead GNU as. I also have few > patches to make llvm-objcopy to work with latest LLVM and support needed > subset of flags, will publish it soon. > > You can find my workarounds as a patch (the kernel will not work with this > applied!) and my kernel config attached. > > Regards, > Dmitry > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170119/e869e5f9/attachment.html>
On Thu, Jan 19, 2017 at 1:56 PM, Dmitry Golovin via llvm-dev < llvm-dev at lists.llvm.org> wrote:> Hi all! > > Recently there was a discussion on this mailing list about linking Linux > and GRUB with LLD. > > I have actually tried to link Linux kernel with LLD and (with some > modifications) it linked successfully (but didn't run well). >Awesome! Thanks for digging into this! When Michael, Davide, and I were debugging the issues with running the LLD-linked FreeBSD kernel, we found that there was really no alternative but to get familiar with the bootloader code and do printf debugging to understand what the bootloader didn't like. (even hooking into qemu's debugger integration didn't work; it is just too early for the usual debugger integration to work I think (interrupts and stuff not in a "sane" state etc.)). I'm happy to share more info about that experience if it would be useful to you. I'm guessing Ed (CC'd) also had some similar experiences getting the FreeBSD bootloaders working with LLD and might have some advice as well w.r.t. a good workflow and what tools to use.> > Here is the list of modifications I had to do in order to link the kernel > (I have used llvmlinux with clang and mainline with gcc, the results are > similar): > > 1. LLD patches: > - D28094 (Implemented support for R_386_PC8/R_386_8 relocations) > - D28857 (Allow relative relocations to a absolute value defined in > linker script) > - [not actually used] D28612 (Added support for --emit-relocs) > > 2. Kernel configuration. I tried "make tinyconfig" and then changed to > CONFIG_64BIT=y with menuconfig. I wanted the minimal kernel configuration > for x86_64. After I get it working, I will try defconfig. > > 3. In arch/x86/Makefile changed the M16_CFLAGS to just "-m16". I think > that cc-option should test if compiler supports '-m16' option and if it > does, use it, if it doesn't use '-m32 -Wa,./arch/x86/boot/code16gcc.h' > instead. I don't know why, but this test failed for me and LLD didn't > accept this header, so I just simply changed the flag to '-m16' because I > didn't want to investigate why did the test failed. It's just a quick > workaround, but it works. >These seem to be compiler flags. It is probably best to use GCC as the compiler for now (i.e. only swap out the linker). That will allow focusing on the LLD issues. Clang (and other LLVM tools) have their own issues with GNU compatibility, but things will be more difficult to debug when there are more moving pieces. -- Sean Silva> > 3. In arch/x86/boot/Makefile and arch/x86/realmode/rm/Makefile added "-m > elf_i386" to LDFLAGS_setup.elf and LDFLAGS_realmode.elf accordingly. The > problem is that the linker is invoked as "ld.lld -m elf_x86_64" and it > ignores the OUTPUT_FORMAT from linker scripts. Being invoked as "ld.lld -m > elf_x86_64 -m elf_i386" doesn't break anything and LLD even produces the > right format of the binaries. Again a quick workaround, but it works. > > 4. In arch/x86/realmode/rm/Makefile removed "--emit-relocs" from > LDFLAGS_realmode.elf. This is the problem with D28612 — I applied the patch > to support --emit-relocs option, but LLD segfaulted. Reported here: > https://llvm.org/bugs/show_bug.cgi?id=31579#c8 > > 5. In arch/x86/kernel/vmlinux.lds.S commented out the "CONSTRUCTORS", > because LLD doesn't support it. > > 6. In arch/x86/boot/setup.ld replaced 5*512 with precalculated value of > 2560 because it doesn't seem that LLD supports math inside ASSERT in linker > scripts. > > 7. In arch/x86/boot/setup.ld commented out the line ". = ASSERT(_end <> 0x8000, "Setup too big!");" because it seems to always fire this assertion, > and the kernel is not going to be working anyway, so I just want it linked > even if it will be unusable. > > 8. In arch/x86/lib/copy_user_64.S removed the line "bad_from_user:" > (probably it was just a mistake), in arch/x86/kernel/vmlinux.lds.S removed > semicolon after "*(.apicdrivers)" (also probably just a mistake). > > Finally the kernel was built and it obviously didn't run (only printed out > "No setup signature found..." but this is some result as well). Probably, > the result could be better if the --emit-relocs option didn't fail and > CONSTRUCTORS were supported. I really don't know what to do about the > assertion that I have commented out. > > I also tried using tools provided by LLVM instead of binutils > (HOSTCC=clang HOSTLD=ld.lld CC=clang LD=ld.lld AR=llvm-ar NM=llvm-nm), it > seems to work with llvmlinux, but clang will not work with the mainline. > Clang doesn't seem to use integrated assembler, so I had to write a simple > wrapper script around llvm-mc to use instead GNU as. I also have few > patches to make llvm-objcopy to work with latest LLVM and support needed > subset of flags, will publish it soon. > > You can find my workarounds as a patch (the kernel will not work with this > applied!) and my kernel config attached. > > Regards, > Dmitry > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170120/b7aee1f7/attachment.html>
One thing I remember is that Linux's build system does directly tamper with object files with custom tools, so those tools might be unhappy with LLD's binaries or just do the wrong thing (sometimes such tools have hardcoded assumptions etc.). One specific case I remember is that linux's copy_from_user/copy_to_user implementations use inline asm to build up a table which is sorted by a custom tool (I think this modification happens post-link even). http://lxr.free-electrons.com/source/arch/x86/include/asm/uaccess.h -- Sean Silva On Thu, Jan 19, 2017 at 1:56 PM, Dmitry Golovin via llvm-dev < llvm-dev at lists.llvm.org> wrote:> Hi all! > > Recently there was a discussion on this mailing list about linking Linux > and GRUB with LLD. > > I have actually tried to link Linux kernel with LLD and (with some > modifications) it linked successfully (but didn't run well). > > Here is the list of modifications I had to do in order to link the kernel > (I have used llvmlinux with clang and mainline with gcc, the results are > similar): > > 1. LLD patches: > - D28094 (Implemented support for R_386_PC8/R_386_8 relocations) > - D28857 (Allow relative relocations to a absolute value defined in > linker script) > - [not actually used] D28612 (Added support for --emit-relocs) > > 2. Kernel configuration. I tried "make tinyconfig" and then changed to > CONFIG_64BIT=y with menuconfig. I wanted the minimal kernel configuration > for x86_64. After I get it working, I will try defconfig. > > 3. In arch/x86/Makefile changed the M16_CFLAGS to just "-m16". I think > that cc-option should test if compiler supports '-m16' option and if it > does, use it, if it doesn't use '-m32 -Wa,./arch/x86/boot/code16gcc.h' > instead. I don't know why, but this test failed for me and LLD didn't > accept this header, so I just simply changed the flag to '-m16' because I > didn't want to investigate why did the test failed. It's just a quick > workaround, but it works. > > 3. In arch/x86/boot/Makefile and arch/x86/realmode/rm/Makefile added "-m > elf_i386" to LDFLAGS_setup.elf and LDFLAGS_realmode.elf accordingly. The > problem is that the linker is invoked as "ld.lld -m elf_x86_64" and it > ignores the OUTPUT_FORMAT from linker scripts. Being invoked as "ld.lld -m > elf_x86_64 -m elf_i386" doesn't break anything and LLD even produces the > right format of the binaries. Again a quick workaround, but it works. > > 4. In arch/x86/realmode/rm/Makefile removed "--emit-relocs" from > LDFLAGS_realmode.elf. This is the problem with D28612 — I applied the patch > to support --emit-relocs option, but LLD segfaulted. Reported here: > https://llvm.org/bugs/show_bug.cgi?id=31579#c8 > > 5. In arch/x86/kernel/vmlinux.lds.S commented out the "CONSTRUCTORS", > because LLD doesn't support it. > > 6. In arch/x86/boot/setup.ld replaced 5*512 with precalculated value of > 2560 because it doesn't seem that LLD supports math inside ASSERT in linker > scripts. > > 7. In arch/x86/boot/setup.ld commented out the line ". = ASSERT(_end <> 0x8000, "Setup too big!");" because it seems to always fire this assertion, > and the kernel is not going to be working anyway, so I just want it linked > even if it will be unusable. > > 8. In arch/x86/lib/copy_user_64.S removed the line "bad_from_user:" > (probably it was just a mistake), in arch/x86/kernel/vmlinux.lds.S removed > semicolon after "*(.apicdrivers)" (also probably just a mistake). > > Finally the kernel was built and it obviously didn't run (only printed out > "No setup signature found..." but this is some result as well). Probably, > the result could be better if the --emit-relocs option didn't fail and > CONSTRUCTORS were supported. I really don't know what to do about the > assertion that I have commented out. > > I also tried using tools provided by LLVM instead of binutils > (HOSTCC=clang HOSTLD=ld.lld CC=clang LD=ld.lld AR=llvm-ar NM=llvm-nm), it > seems to work with llvmlinux, but clang will not work with the mainline. > Clang doesn't seem to use integrated assembler, so I had to write a simple > wrapper script around llvm-mc to use instead GNU as. I also have few > patches to make llvm-objcopy to work with latest LLVM and support needed > subset of flags, will publish it soon. > > You can find my workarounds as a patch (the kernel will not work with this > applied!) and my kernel config attached. > > Regards, > Dmitry > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170120/224b3c81/attachment.html>