llvm dev - Dec 2016 - Offset too large on scattered relocations

CCing Nick Kledzik as I posed this question on IRC and Tim Northover
suggested you as a good resource for this.

I came across an error due to a scattered relocation offset being larger
than 2**24 and I was hoping to find more information on scattered
relocations. These are MachO specific, and Ive not been able to find any
documentation on them outside of source code.

I have a couple of immediate questions, but any info would be appreciated.

* Should a check be added to ARMMachObjectWriter::RecordARMScatteredRelocation
and RecordARMScatteredHalfRelocation to ensure that FixupOffset is within
bounds?

I see a check in the x86 back end that does precisely this and was curious
why a similar check was not in place for ARM:

  // Relocations are written out in reverse order, so the PAIR comes first.
  if (Type == MachO::GENERIC_RELOC_SECTDIFF ||
      Type == MachO::GENERIC_RELOC_LOCAL_SECTDIFF) {
    // If the offset is too large to fit in a scattered relocation,
    // we're hosed. It's an unfortunate limitation of the MachO format.
    if (FixupOffset > 0xffffff) {
      char Buffer[32];
      format("0x%x", FixupOffset).print(Buffer, sizeof(Buffer));
      Asm.getContext().reportError(Fixup.getLoc(),
                         Twine("Section too large, can't encode "
                                "r_address (") + Buffer +
                         ") into 24 bits of scattered "
                         "relocation entry.");
      return false;
    }

* Once we get here there seems to be nothing that can be done. Are there
ways of avoiding generation of scattered relocations? Ive had some success
in working around the problem by using llc's -relocation-model flag with
'static' and 'dynamic-no-pic' though I believe that is simply
due to a
small size reduction in the binary pulling the offset back into range of a
24bit value. Im also not sure that iOS will accept apps built with a static
relocation model.

Thanks for any additional info

Rob
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161206/f6b8f1ee/attachment.html>

> On Dec 6, 2016, at 5:09 PM, Rob Dalvik <rob.dalvik at gmail.com>
wrote:
> 
> CCing Nick Kledzik as I posed this question on IRC and Tim Northover
suggested you as a good resource for this.
> 
> I came across an error due to a scattered relocation offset being larger
than 2**24 and I was hoping to find more information on scattered relocations.
These are MachO specific, and Ive not been able to find any documentation on
them outside of source code.
> 
> I have a couple of immediate questions, but any info would be appreciated.
> 
> * Should a check be added to
ARMMachObjectWriter::RecordARMScatteredRelocation and
RecordARMScatteredHalfRelocation to ensure that FixupOffset is within bounds?
> 
> I see a check in the x86 back end that does precisely this and was curious
why a similar check was not in place for ARM:
> 
>   // Relocations are written out in reverse order, so the PAIR comes first.
>   if (Type == MachO::GENERIC_RELOC_SECTDIFF ||
>       Type == MachO::GENERIC_RELOC_LOCAL_SECTDIFF) {
>     // If the offset is too large to fit in a scattered relocation,
>     // we're hosed. It's an unfortunate limitation of the MachO
format.
>     if (FixupOffset > 0xffffff) {
>       char Buffer[32];
>       format("0x%x", FixupOffset).print(Buffer, sizeof(Buffer));
>       Asm.getContext().reportError(Fixup.getLoc(),
>                          Twine("Section too large, can't encode
"
>                                 "r_address (") + Buffer +
>                          ") into 24 bits of scattered "
>                          "relocation entry.");
>       return false;
>     }
> 
> * Once we get here there seems to be nothing that can be done. Are there
ways of avoiding generation of scattered relocations? Ive had some success in
working around the problem by using llc's -relocation-model flag with
'static' and 'dynamic-no-pic' though I believe that is simply
due to a small size reduction in the binary pulling the offset back into range
of a 24bit value. Im also not sure that iOS will accept apps built with a static
relocation model.
> 
> Thanks for any additional info
Yes, 32-bit arm mach-o object files run into size limitations because of how
relocations are encoded.  Some information is encoded in bits of the instruction
and some is encoded in bit fields in relocations.  If you try to make an arm .o
file larger than 16MB, you start hitting various limits.  Not all the limits are
enforced by the MC layer, resulting in bad .o files.

You could try not having debug info, or moving the __DWARF sections to the end
of the file, and you might last longer.  But the only sure fix is to keep all .o
file under 16MB.  The final product can be larger because resolved address are
32-bits and the linker inserts branch islands to let BL instructions branch
further.

-Nick


-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161206/30c7e443/attachment.html>

Thanks for the response Nick,

Do you think there is value in adding the check for FixupOffset > 0xffffff
into the ARM backend? The lack of that seems like it could silently record
incorrect offsets from the assignment later in
RecordARMScatteredRelocation():
  MRE.r_word0 = ((FixupOffset <<  0) |
                 (Type        << 24) |
                 (MovtBit     << 28) |
                 (ThumbBit    << 29) |
                 (IsPCRel     << 30) |
                 MachO::R_SCATTERED);

Rob

On Tue, Dec 6, 2016 at 8:38 PM, Nick Kledzik <kledzik at apple.com> wrote:
>
> On Dec 6, 2016, at 5:09 PM, Rob Dalvik <rob.dalvik at gmail.com>
wrote:
>
> CCing Nick Kledzik as I posed this question on IRC and Tim Northover
> suggested you as a good resource for this.
>
> I came across an error due to a scattered relocation offset being larger
> than 2**24 and I was hoping to find more information on scattered
> relocations. These are MachO specific, and Ive not been able to find any
> documentation on them outside of source code.
>
> I have a couple of immediate questions, but any info would be appreciated.
>
> * Should a check be added to
ARMMachObjectWriter::RecordARMScatteredRelocation
> and RecordARMScatteredHalfRelocation to ensure that FixupOffset is within
> bounds?
>
> I see a check in the x86 back end that does precisely this and was curious
> why a similar check was not in place for ARM:
>
>   // Relocations are written out in reverse order, so the PAIR comes first.
>   if (Type == MachO::GENERIC_RELOC_SECTDIFF ||
>       Type == MachO::GENERIC_RELOC_LOCAL_SECTDIFF) {
>     // If the offset is too large to fit in a scattered relocation,
>     // we're hosed. It's an unfortunate limitation of the MachO
format.
>     if (FixupOffset > 0xffffff) {
>       char Buffer[32];
>       format("0x%x", FixupOffset).print(Buffer, sizeof(Buffer));
>       Asm.getContext().reportError(Fixup.getLoc(),
>                          Twine("Section too large, can't encode
"
>                                 "r_address (") + Buffer +
>                          ") into 24 bits of scattered "
>                          "relocation entry.");
>       return false;
>     }
>
> * Once we get here there seems to be nothing that can be done. Are there
> ways of avoiding generation of scattered relocations? Ive had some success
> in working around the problem by using llc's -relocation-model flag
with
> 'static' and 'dynamic-no-pic' though I believe that is
simply due to a
> small size reduction in the binary pulling the offset back into range of a
> 24bit value. Im also not sure that iOS will accept apps built with a static
> relocation model.
>
> Thanks for any additional info
>
> Yes, 32-bit arm mach-o object files run into size limitations because of
> how relocations are encoded.  Some information is encoded in bits of the
> instruction and some is encoded in bit fields in relocations.  If you try
> to make an arm .o file larger than 16MB, you start hitting various limits.
> Not all the limits are enforced by the MC layer, resulting in bad .o files.
>
>
> You could try not having debug info, or moving the __DWARF sections to the
> end of the file, and you might last longer.  But the only sure fix is to
> keep all .o file under 16MB.  The final product can be larger because
> resolved address are 32-bits and the linker inserts branch islands to let
> BL instructions branch further.
>
> -Nick
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161213/6207a2e0/attachment.html>