llvm dev - Mar 2009 - [LLVMdev] MachO and ELF Writers/MachineCodeEmittersarehard-codedinto LLVMTargetMachine

On Mon, Mar 16, 2009 at 3:26 AM, Aaron Gray <aaronngray.lists at
googlemail.com
> wrote:
>  On Sun, Mar 15, 2009 at 10:52 PM, Aaron Gray <
> aaronngray.lists at googlemail.com> wrote:
>
>>  I like the idea of a generic MachineCodeWriter, although I prefer the
>>> name 'ObjectFileWriter'...
>>>
>>
>> Thats much more descriptive of the functionality.
>>
>
> Sorry, I disagree actually the MachineCodeEmitter or the
> 'MachineCodeWritter' does not do any file handling at all. Do look
at the
> code for the MachineCodeWritter and you will see it only writes to memory
> and if it reaches the end of the allotted memory I believe higher ordered
> logic reallocates a larget buffer and starts again from scratch. This could
> be avoided if they generated fixus for absolute memory references refering
> within the outputted code. Then a alloc function could be called before
> outputting say a 4 byte int and could realloc and copy code and when
finally
> written the fixups could be applied.
>
> I am also wondering about the efficiency of std::vector whether we could
> use that for the MachineCodeWriter, or whether we write out own code output
> stream/buffering ?
>
> I still think this is where the crux of the problem lies the upper logic is
> relatively simple compared to this buy looking at what you say it is
> important to get it right.
>

'ObjectCodeEmitter' looks like the right description to parallel the
MachineCodeEmitter. Its emitting object code to a data stream (which is an
object file section) and not direct to a file.

I will knock to gether an ObjectCodeEmitter that is call compatible with the
MachineCodeEmitter and wtites to a std::vector<byte>, so it could replace
the MachineCodeEmitter class generically in usage.

This needs alot of thought and to get things right, and provide the right
incremental patches to get this accepted.

Cheers,

Aaron
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20090316/a3560750/attachment.html>

> Sorry, I disagree actually the MachineCodeEmitter or the
> 'MachineCodeWritter' does not do any file handling at all. Do look
at the
> code for the MachineCodeWritter and you will see it only writes to memory
> and if it reaches the end of the allotted memory I believe higher ordered
> logic reallocates a larget buffer and starts again from scratch. This could
> be avoided if they generated fixus for absolute memory references refering
> within the outputted code. Then a alloc function could be called before
> outputting say a 4 byte int and could realloc and copy code and when
finally
> written the fixups could be applied.
IIRC the memory allocation is done in the MachineCodeEmitter, not the
higher level (see startFunction and finishFunction). The current
implementation has startFunction allocate some (arbitrary) reserve
size in the output vector, and if we the emitter runs out of space,
finishFunction returns a failure, causing the whole process to occur
again. This is icky.

It would be far better if the underlying buffer would grow
automatically (with an allocation method in the base class, as you
suggested), as code is emitted to it.
> 'ObjectCodeEmitter' looks like the right description to parallel
the
> MachineCodeEmitter. Its emitting object code to a data stream (which
> is an object file section) and not direct to a file.
I can live with that. Before you implement anything, can we try and
define the responsibilities of the various classes?

We have MachineCodeEmitter, which is responsible for actually emitting
bytes into a buffer for a function. Should it have methods for
emitting instructions/operands, or should it only work at the byte,
dword, etc. level?

ObjectCodeEmitter,  is responsible for emission of object 'files' into
a memory buffer. This includes handling of all object headers,
management of sections/segments, symbol and string tables and
relocations. The ObjectCodeEmitter should delegate all actual 'data
emission' to the MachineCodeEmitter.

ObjectCodeEmitter is a MachineFunctionPass. It does 'object wide'
setup in doInitialization and finalizes the object in doFinalize(?).
Each MachineFunction is emitted through the runOnFunction method,
which passes the MachineFunction to the MachineCodeEmitter. The
MachineCodeEmitter calls back to the ObjectCodeEmitter in order to
look up sections/segments, add globals to an unresolved globals list
etc.

I'm not too happy about the broken encapsulation here. I'd prefer to
find a better way to model this.

>> Sorry, I disagree actually the MachineCodeEmitter or the
>> 'MachineCodeWritter' does not do any file handling at all. Do
look at the
>> code for the MachineCodeWritter and you will see it only writes to
memory
>> and if it reaches the end of the allotted memory I believe higher
ordered
>> logic reallocates a larget buffer and starts again from scratch. This 
>> could
>> be avoided if they generated fixus for absolute memory references 
>> refering
>> within the outputted code. Then a alloc function could be called before
>> outputting say a 4 byte int and could realloc and copy code and when 
>> finally
>> written the fixups could be applied.
>
> IIRC the memory allocation is done in the MachineCodeEmitter, not the
> higher level (see startFunction and finishFunction). The current
> implementation has startFunction allocate some (arbitrary) reserve
> size in the output vector, and if we the emitter runs out of space,
> finishFunction returns a failure, causing the whole process to occur
> again. This is icky.
As I said the MachineCodeEmitter works exclusively with the JITEmitter class 
which inherits from the MachineCodeEmitter class and thats where the 
allocation is going on.
> It would be far better if the underlying buffer would grow
> automatically (with an allocation method in the base class, as you
> suggested), as code is emitted to it.
I think we leave the JITEmitter and MachineCodeEmitter alone as they work 
and its a pritty difficult thing to replace them without causing any 
regressions.
>> 'ObjectCodeEmitter' looks like the right description to
parallel the
>> MachineCodeEmitter. Its emitting object code to a data stream (which
>> is an object file section) and not direct to a file.
>
> I can live with that. Before you implement anything, can we try and
> define the responsibilities of the various classes?
Yes.
> We have MachineCodeEmitter, which is responsible for actually emitting
> bytes into a buffer for a function. Should it have methods for
> emitting instructions/operands, or should it only work at the byte,
> dword, etc. level?
It works at primatine byte, word, dword level only. the *CodeEmitter classes 
like the X86CodeEmitter class are responsible for generating machine code 
and use the MachineCodeEmitter class to do this.
> ObjectCodeEmitter,  is responsible for emission of object 'files'
into
> a memory buffer. This includes handling of all object headers,
> management of sections/segments, symbol and string tables and
It writes to sections in ELFWriter::ELFSection.SectionData which are 
std::vector<unsigned char>.
> relocations. The ObjectCodeEmitter should delegate all actual 'data
> emission' to the MachineCodeEmitter.
No it will have to parallel the MachineCodeEmitter as a generic alternative.
> ObjectCodeEmitter is a MachineFunctionPass. It does 'object wide'
> setup in doInitialization and finalizes the object in doFinalize(?).
> Each MachineFunction is emitted through the runOnFunction method,
> which passes the MachineFunction to the MachineCodeEmitter. The
> MachineCodeEmitter calls back to the ObjectCodeEmitter in order to
> look up sections/segments, add globals to an unresolved globals list
> etc.
>
No look at the ELFWriter which inherits from MachineFunctionPass and 
ELFCodeEmitter which inherits from the MachineCodeEmitter.

We need to override the behaviour of the MachineCodeEmitter creating the 
ObjectCodeEmitter which writes to the appropriate section in 
ELFWriter::ELFSection.SectionData.
> I'm not too happy about the broken encapsulation here. I'd prefer
to
> find a better way to model this.
Where ? How ?

Aaron