llvm dev - May 2018 - How to add assembly instructions in CodeGen

Hello,

I want to add assembly instructions at certain points in a function. This
is X86 specific. So I am working in the lib/Target/X86 folder. I create a
`MachineFunctionPass` in that folder. I register it in the
X86TargetMachine.cpp in addPreEmitPass(). I use BuildMI to insert my own
assembly instructions in the MachineFunctionPass. This works and my
assembly instructions are inserted at desired places. However, this breaks
the alignment. So when I run the generated code, I get segmentation fault
(precisely in printf with XMM registers). Where should I add my pass?

My pass depends on the MachineBasicBlock information as well. Therefore, I
cannot add my pass too early in LLVM IR. What is the proper pass to add my
custom MachineFunctionPass? I tried addPreRegAlloc, but it failed due to
insufficient register allocation error or something on that line.

Can anybody please help me write a MachineFunctionPass where I can insert
assembly instruction without breaking the alignment? I am doing this for
X86_64.

Regards,
Soham Sinha
PhD Student, Department of Computer Science
Boston University
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180504/7d720e63/attachment.html>

On Sun, May 6, 2018 at 7:26 AM Soham Sinha via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hello,
> I want to add assembly instructions at certain points in a function. This
is X86 specific. So I am working in the lib/Target/X86 folder. I create a
`MachineFunctionPass` in that folder. I register it in the
X86TargetMachine.cpp in addPreEmitPass(). I use BuildMI to insert my own
assembly instructions in the MachineFunctionPass. This works and my
assembly instructions are inserted at desired places. However, this breaks
the alignment. So when I run the generated code, I get segmentation fault
(precisely in printf with XMM registers). Where should I add my pass?


It sounds like you're running into stack alignment issues. If you're
adding
data to the stack, you may need to work a little harder with maintaining
the state of the stack. This is not trivial to do especially if you're
emitting the assembly by the time you're at a MachineFunctionPass (because
register spilling and/or stack alignment information would have already
been done by the time you're in machine instruction lowering). What you may
need to do here is to either:

- hook into the preamble and stack re-alignment code specifically in X86
that would look at information from your pass. This is not trivial and I
don't recommend going down this path (I tried, but I lost the patience to
do it properly).

- when emitting the assembly instructions that involve pushing/popping from
the stack, that you're keeping track of the alignment of the stack
variables. This is what we do with XRay, when we're lowering the custom
event sleds.

- use pseudo-instructions and preserving those until lowering, where the
lowering
> My pass depends on the MachineBasicBlock information as well. Therefore,
I cannot add my pass too early in LLVM IR. What is the proper pass to add
my custom MachineFunctionPass? I tried addPreRegAlloc, but it failed due to
insufficient register allocation error or something on that line.
> Can anybody please help me write a MachineFunctionPass where I can insert
assembly instruction without breaking the alignment? I am doing this for
X86_64.


You can look at the XRay lowering for the PATCHABLE_EVENT_CALL lowering in
X86AsmPrinter as a guide for the lowering, but you might also want to see
how we're inserting these pseudo-instructions from the

I don't remember having to specify where the pass is defined, since it's
already in the assembly printing. So you might consider inserting these
pseudo-instructions a the MachineFunctionPass, which gets lowered
appropriately in the assembly printer. Unfortunately I don't think
there's
a generic way of doing this (yet) with the X86 back-end. There might be a
good case for making this easier, but right now these kinds of things
haven't been too important to fix yet.

Hope this helps!
-- 
Dean

One place to look might be in the MachineOutliner target hooks in X86InstrInfo
and AArch64InstrInfo. The MachineOutliner runs extremely late in the pass
pipeline so it might be a good place to look for some inspiration. Of course,
because this is *extremely late* it might not do *exactly* what you need. (e.g,
this is post-register allocation, post frame-lowering, etc.)

- Jessica
> On May 4, 2018, at 6:32 PM, Soham Sinha via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> Hello,
> 
> I want to add assembly instructions at certain points in a function. This
is X86 specific. So I am working in the lib/Target/X86 folder. I create a
`MachineFunctionPass` in that folder. I register it in the X86TargetMachine.cpp
in addPreEmitPass(). I use BuildMI to insert my own assembly instructions in the
MachineFunctionPass. This works and my assembly instructions are inserted at
desired places. However, this breaks the alignment. So when I run the generated
code, I get segmentation fault (precisely in printf with XMM registers). Where
should I add my pass?
> 
> My pass depends on the MachineBasicBlock information as well. Therefore, I
cannot add my pass too early in LLVM IR. What is the proper pass to add my
custom MachineFunctionPass? I tried addPreRegAlloc, but it failed due to
insufficient register allocation error or something on that line.
> 
> Can anybody please help me write a MachineFunctionPass where I can insert
assembly instruction without breaking the alignment? I am doing this for X86_64.
> 
> Regards,
> Soham Sinha
> PhD Student, Department of Computer Science
> Boston University
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

Hello Dean,

I looked at the XRay Instrumentation. That's a nice engineering effort. I
am sure you had your motivation to do this in CodeGen just like I wanted to
do. I don't understand all of your code but I get the idea that you are
adjusting the alignment with explicit bytes and no-op instructions. My
problem is also very much related to yours where my stack pointer ($rsp)
alignment breaks in printf.

Having said that, I am not sure whether I need the engineering effort that
you have pursued. I am trying to add function calls in some places of the
machine code. I followed X86_64 calling convention to do so. I saved
(pushed into stack) all the necessary registers (also tried saving all the
16 registers) and then filled up 3 arguments in rdi, rsi, rdx and then call
the desired function (and then pop the registers). Mathematically, saving
the 16 register should not break the alignment of the stack pointer. But
when I am trying to debug with gdb, I see that the alignment breaks
sometimes during the push operations of 16 registers, and it comes as
broken alignment in the printf function. I am very confused what can go
wrong here. This is why I was trying to rely on LLVM to maintain the
alignment.

Interestingly, at the start of the runOnMachineFunction, I check the
alignment of the function and also at the end of the runOnMachineFunction
(after my push, call function and pop). The alignment stays same as 4 (16
bytes). Therefore, I guess, the BuildMI function doesn't maintain the
alignment and doesn't even report the broken alignment through the
alignment variable of MachineFunction. I access the alignment through the
function, getAlignment. I think BuildMI should have cared about alignment
or at least update the alignment value.

I am afraid if I follow your path of instrumentation, again I might
ultimately face the same issue where I could not maintain the alignment.
Your effort is quite similar to what I am trying to do, but I am just
doing it in the MachineFunctionPass itself.

It's very non-trivial and tedious to change the internals of CodeGen
because the LLVM MC infrastructure is very much intertwined with the
Assembler. That makes compilation faster but instrumentation tougher. This
is why I wrote a MachineFunctionPass so that my instrumentation stays like
a module. I add my MachineFunctionPass at the end of addPreEmitPass phase
of X86.

I wish LLVM provided more modular ways of instrumentation just like it
provides similar instrumentation in the LLVM IR level.

Regards,
Soham Sinha
PhD Student, Department of Computer Science
Boston University

On Mon, May 7, 2018 at 1:20 AM, Dean Michael Berris <dean.berris at
gmail.com>
wrote:
> On Sun, May 6, 2018 at 7:26 AM Soham Sinha via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
> > Hello,
>
> > I want to add assembly instructions at certain points in a function.
This
> is X86 specific. So I am working in the lib/Target/X86 folder. I create a
> `MachineFunctionPass` in that folder. I register it in the
> X86TargetMachine.cpp in addPreEmitPass(). I use BuildMI to insert my own
> assembly instructions in the MachineFunctionPass. This works and my
> assembly instructions are inserted at desired places. However, this breaks
> the alignment. So when I run the generated code, I get segmentation fault
> (precisely in printf with XMM registers). Where should I add my pass?
>
>
> It sounds like you're running into stack alignment issues. If
you're adding
> data to the stack, you may need to work a little harder with maintaining
> the state of the stack. This is not trivial to do especially if you're
> emitting the assembly by the time you're at a MachineFunctionPass
(because
> register spilling and/or stack alignment information would have already
> been done by the time you're in machine instruction lowering). What you
may
> need to do here is to either:
>
> - hook into the preamble and stack re-alignment code specifically in X86
> that would look at information from your pass. This is not trivial and I
> don't recommend going down this path (I tried, but I lost the patience
to
> do it properly).
>
> - when emitting the assembly instructions that involve pushing/popping from
> the stack, that you're keeping track of the alignment of the stack
> variables. This is what we do with XRay, when we're lowering the custom
> event sleds.
>
> - use pseudo-instructions and preserving those until lowering, where the
> lowering
>
> > My pass depends on the MachineBasicBlock information as well.
Therefore,
> I cannot add my pass too early in LLVM IR. What is the proper pass to add
> my custom MachineFunctionPass? I tried addPreRegAlloc, but it failed due to
> insufficient register allocation error or something on that line.
>
> > Can anybody please help me write a MachineFunctionPass where I can
insert
> assembly instruction without breaking the alignment? I am doing this for
> X86_64.
>
>
> You can look at the XRay lowering for the PATCHABLE_EVENT_CALL lowering in
> X86AsmPrinter as a guide for the lowering, but you might also want to see
> how we're inserting these pseudo-instructions from the
>
> I don't remember having to specify where the pass is defined, since
it's
> already in the assembly printing. So you might consider inserting these
> pseudo-instructions a the MachineFunctionPass, which gets lowered
> appropriately in the assembly printer. Unfortunately I don't think
there's
> a generic way of doing this (yet) with the X86 back-end. There might be a
> good case for making this easier, but right now these kinds of things
> haven't been too important to fix yet.
>
> Hope this helps!
> --
> Dean
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180507/bf14ad80/attachment.html>

Hello Jessica,

I tried running my pass in the PreEmit phase in X86. It doesn't maintain
the alignment of stack pointer. I tried with PreRegAlloc phase and I got
into an error of insufficient register allocation. If I reduce my register
usage, source code can be compiled, but when the generated machine code is
run, I still get segmentation fault.

Regards,
Soham Sinha
PhD Student, Department of Computer Science
Boston University

On Mon, May 7, 2018 at 12:08 PM, Jessica Paquette <jpaquette at apple.com>
wrote:
> One place to look might be in the MachineOutliner target hooks in
> X86InstrInfo and AArch64InstrInfo. The MachineOutliner runs extremely late
> in the pass pipeline so it might be a good place to look for some
> inspiration. Of course, because this is *extremely late* it might not do
> *exactly* what you need. (e.g, this is post-register allocation, post
> frame-lowering, etc.)
>
> - Jessica
>
> > On May 4, 2018, at 6:32 PM, Soham Sinha via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
> >
> > Hello,
> >
> > I want to add assembly instructions at certain points in a function.
> This is X86 specific. So I am working in the lib/Target/X86 folder. I
> create a `MachineFunctionPass` in that folder. I register it in the
> X86TargetMachine.cpp in addPreEmitPass(). I use BuildMI to insert my own
> assembly instructions in the MachineFunctionPass. This works and my
> assembly instructions are inserted at desired places. However, this breaks
> the alignment. So when I run the generated code, I get segmentation fault
> (precisely in printf with XMM registers). Where should I add my pass?
> >
> > My pass depends on the MachineBasicBlock information as well.
Therefore,
> I cannot add my pass too early in LLVM IR. What is the proper pass to add
> my custom MachineFunctionPass? I tried addPreRegAlloc, but it failed due to
> insufficient register allocation error or something on that line.
> >
> > Can anybody please help me write a MachineFunctionPass where I can
> insert assembly instruction without breaking the alignment? I am doing this
> for X86_64.
> >
> > Regards,
> > Soham Sinha
> > PhD Student, Department of Computer Science
> > Boston University
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180507/4fb9b1fa/attachment.html>