llvm dev - Apr 2013 - [LLVMdev] A new mechanism to compiler kernel modules using llvm: Defer type evaluation in clang?

Hi,

First of all, I didn't study on compiler too much, I'm a Linux kernel
developer,
Now I have one idea about compile kernel module by using llvm framework.

In Linux world, traditionally compile Linux kernel module only have one way:
compile it to machine code directly, deploy to target machine, then
run it in target machine.

This have some problem from long term view, especially kernel module
compatibility issue,
we know that Linux kernel change fast, when kernel change the data
structure exposed to external
kernel modules, then all ko depend on that data structure will need
recompile, one case is
kernel upgrade.
This is really a problem, we already suffered it many years, in whole
Linux world.
Linux distribution suffer it, third party kernel modules developers suffer it.
hardware driver developers also suffer it, we cannot avoid re-compile
kernel modules whenever new kernel
release if the structure changed, why? because kernel modules depend
on kernel header data structure, this is the point.

Is there have any method to fix it? let's try to decouple kernel
modules with kernel, by using llvm.
Please give comments on following mechanism, 3 steps:

1) kernel modules developer compile C file into high level IR, not to
machine code.

2) In install time, the high level IR file "link" with kernel header
data structure file, generate  low level
IR file, it could be .bc or .ll file in llvm form.

3) llvm compile the low level IR file into machine code in target
machine, then insert kernel modules, run it.

Using this method, kernel module code is not need to change when
kernel header file change, because it evaluate
structure type info at install time, not C code compile time. it fix
compatibility  issue in a clean way.
The key point of this method is we need defer structure type
evaluation into install time, Is Clang support
this "feature" at present?

How about this idea sounds? I really think llvm design is nature to
suit this idea, benefit from llvm's IR.
If we really implement this idea, it would be very valuable for whole
Linux community, trust me:)

I would appreciate if you can give me some technical comments on this design.
Thanks very much.

.jovi

Just out of curiosity, what would be the main benefit of this approach vs DKMS
which is already widely used ?

Le 28 avr. 2013 à 18:42, Jovi Zhang <bookjovi at gmail.com> a écrit :
> Hi,
> 
> First of all, I didn't study on compiler too much, I'm a Linux
kernel developer,
> Now I have one idea about compile kernel module by using llvm framework.
> 
> In Linux world, traditionally compile Linux kernel module only have one
way:
> compile it to machine code directly, deploy to target machine, then
> run it in target machine.
> 
> This have some problem from long term view, especially kernel module
> compatibility issue,
> we know that Linux kernel change fast, when kernel change the data
> structure exposed to external
> kernel modules, then all ko depend on that data structure will need
> recompile, one case is
> kernel upgrade.
> This is really a problem, we already suffered it many years, in whole
> Linux world.
> Linux distribution suffer it, third party kernel modules developers suffer
it.
> hardware driver developers also suffer it, we cannot avoid re-compile
> kernel modules whenever new kernel
> release if the structure changed, why? because kernel modules depend
> on kernel header data structure, this is the point.
> 
> Is there have any method to fix it? let's try to decouple kernel
> modules with kernel, by using llvm.
> Please give comments on following mechanism, 3 steps:
> 
> 1) kernel modules developer compile C file into high level IR, not to
> machine code.
> 
> 2) In install time, the high level IR file "link" with kernel
header
> data structure file, generate  low level
> IR file, it could be .bc or .ll file in llvm form.
> 
> 3) llvm compile the low level IR file into machine code in target
> machine, then insert kernel modules, run it.
> 
> Using this method, kernel module code is not need to change when
> kernel header file change, because it evaluate
> structure type info at install time, not C code compile time. it fix
> compatibility  issue in a clean way.
> The key point of this method is we need defer structure type
> evaluation into install time, Is Clang support
> this "feature" at present?
> 
> How about this idea sounds? I really think llvm design is nature to
> suit this idea, benefit from llvm's IR.
> If we really implement this idea, it would be very valuable for whole
> Linux community, trust me:)
> 
> I would appreciate if you can give me some technical comments on this
design.
> Thanks very much.
> 
> .jovi
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
-- Jean-Daniel

I am not sure this would work. It might be able to handle changes in
structure, but what about changes in the accessor functions?
I also don't see how this would be easier than compiling from source on the
target machine. You still need all the kernel headers on the target machine
because llvm with your patch would need them so that it uses the correct
structure.
The best approach for kernel stuff is to get the source code into mainline,
then the source code will be kept up to date for you.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20130429/8fabc493/attachment.html>

On Mon, Apr 29, 2013 at 3:11 PM, Jean-Daniel Dupas
<devlists at shadowlab.org> wrote:
> Just out of curiosity, what would be the main benefit of this approach vs
DKMS which is already widely used ?
>
Thanks Dupas.

I checked DKMS you mentioned, basically DKMS is just a ko and its
sources management tool.

It's not easy to deploy ko source into target machine, and it's more
harder to build ko in target machine,
this is a very common case, especially in embedded Linux, you really
cannot put source into target device,
for example, you didn't see any kernel module source in android phone,
right?

What I really want to see is kernel module source don't need to to
change when kernel changes, whatever kernel
upgrade or minor structure change, DKMS cannot meet this.

what's the benefit of that?

we can replace kernel as we want, and don't need to worry ko cannot
work in new kernel, this is the true decouple,
not the decouple implementation in DKMS claimed.
Imaging that you want upgrade the android kernel to kernel mainline by
yourself, you really could do this if using the mechanism
I said, but you definitely cannot achieve this currently or use DKMS.

Another case is fast kernel debugging, just replace kernel, to check
functionality or performance in new kernel, without ko recompile,
ko may provided by different vendor, we cannot contact every vendor to
request they deliver new ko to let us try our new kernel.

On 4/28/13 11:42 AM, Jovi Zhang wrote:
> Hi,
>
> First of all, I didn't study on compiler too much, I'm a Linux
kernel developer,
> Now I have one idea about compile kernel module by using llvm framework.
>
> In Linux world, traditionally compile Linux kernel module only have one
way:
> compile it to machine code directly, deploy to target machine, then
> run it in target machine.
>
> This have some problem from long term view, especially kernel module
> compatibility issue,
> we know that Linux kernel change fast, when kernel change the data
> structure exposed to external
> kernel modules, then all ko depend on that data structure will need
> recompile, one case is
> kernel upgrade.
> This is really a problem, we already suffered it many years, in whole
> Linux world.
> Linux distribution suffer it, third party kernel modules developers suffer
it.
> hardware driver developers also suffer it, we cannot avoid re-compile
> kernel modules whenever new kernel
> release if the structure changed, why? because kernel modules depend
> on kernel header data structure, this is the point.
The basic idea seems feasible, but I think this is a "devil in the 
details" sort of project.  As Joshua has pointed out, there are certain 
constructs you can't use (such as sizeof() in preprocessor #if 
statements), and I think you'd need to enhance the LLVM IR to represent 
unknown structure sizes symbolically.  There is also the fact that the 
Clang frontend generates architecture-specific code and that there are 
LLVM optimizations that use the size of structures for optimization.  
That is all stuff that your project would have to "fix."  You'd be
swimming against a strong current, so to speak.

More importantly, while you raise a number of limitations of linking 
native code for kernel modules, I don't think the Linux community sees 
this is a disadvantage.  Rather, they see it as a way of encouraging 
developers to open-source their drivers and have them included in the 
Linux kernel tree.  I'm not convinced that you'd get buy-in from the 
Linux community.

If this sort of project interests you, then I think you should aim for 
something smaller and that has more interest.  For example, I think the 
NaCL developers at Google were looking at ways to make the LLVM IR more 
suitable as a format for shipping NaCL plugins.  If that hasn't been 
done yet, it's an easier problem to tackle, and there is a group that is 
already interested in having it.

Finally, as a shameless plug, your project idea reminds me of the LLVA 
work that was done here at Illinois (which became the foundation for my 
own research on the Secure Virtual Architecture). That work aimed to 
replace the processor native instruction set with the LLVM IR to allow 
the processor to adapt without requiring manual recompilation of 
software.  You can find those papers on the llvm.org publications page 
if you're interested.

-- John T.


>
> Is there have any method to fix it? let's try to decouple kernel
> modules with kernel, by using llvm.
> Please give comments on following mechanism, 3 steps:
>
> 1) kernel modules developer compile C file into high level IR, not to
> machine code.
>
> 2) In install time, the high level IR file "link" with kernel
header
> data structure file, generate  low level
> IR file, it could be .bc or .ll file in llvm form.
>
> 3) llvm compile the low level IR file into machine code in target
> machine, then insert kernel modules, run it.
>
> Using this method, kernel module code is not need to change when
> kernel header file change, because it evaluate
> structure type info at install time, not C code compile time. it fix
> compatibility  issue in a clean way.
> The key point of this method is we need defer structure type
> evaluation into install time, Is Clang support
> this "feature" at present?
>
> How about this idea sounds? I really think llvm design is nature to
> suit this idea, benefit from llvm's IR.
> If we really implement this idea, it would be very valuable for whole
> Linux community, trust me:)
>
> I would appreciate if you can give me some technical comments on this
design.
> Thanks very much.
>
> .jovi
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

I sounds me that you are just substituting "compile C source file" for
"compile LLVM IR source file" or "compile LLVM IR .o file"
The problem is you wish to link in the linux kernel headers at
"install" time.
The problem I see is that the linux kernel headers are C source files,
so how is that going to work?
You need a C compiler to understand the Linux kernel headers.


On 28 April 2013 17:42, Jovi Zhang <bookjovi at gmail.com>
wrote:
> Hi,
>
> First of all, I didn't study on compiler too much, I'm a Linux
kernel developer,
> Now I have one idea about compile kernel module by using llvm framework.
>
> In Linux world, traditionally compile Linux kernel module only have one
way:
> compile it to machine code directly, deploy to target machine, then
> run it in target machine.
>
> This have some problem from long term view, especially kernel module
> compatibility issue,
> we know that Linux kernel change fast, when kernel change the data
> structure exposed to external
> kernel modules, then all ko depend on that data structure will need
> recompile, one case is
> kernel upgrade.
> This is really a problem, we already suffered it many years, in whole
> Linux world.
> Linux distribution suffer it, third party kernel modules developers suffer
it.
> hardware driver developers also suffer it, we cannot avoid re-compile
> kernel modules whenever new kernel
> release if the structure changed, why? because kernel modules depend
> on kernel header data structure, this is the point.
>
> Is there have any method to fix it? let's try to decouple kernel
> modules with kernel, by using llvm.
> Please give comments on following mechanism, 3 steps:
>
> 1) kernel modules developer compile C file into high level IR, not to
> machine code.
>
> 2) In install time, the high level IR file "link" with kernel
header
> data structure file, generate  low level
> IR file, it could be .bc or .ll file in llvm form.
>
> 3) llvm compile the low level IR file into machine code in target
> machine, then insert kernel modules, run it.
>
> Using this method, kernel module code is not need to change when
> kernel header file change, because it evaluate
> structure type info at install time, not C code compile time. it fix
> compatibility  issue in a clean way.
> The key point of this method is we need defer structure type
> evaluation into install time, Is Clang support
> this "feature" at present?
>
> How about this idea sounds? I really think llvm design is nature to
> suit this idea, benefit from llvm's IR.
> If we really implement this idea, it would be very valuable for whole
> Linux community, trust me:)
>
> I would appreciate if you can give me some technical comments on this
design.
> Thanks very much.
>
> .jovi
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

On Tue, Apr 30, 2013 at 12:46 AM, James Courtier-Dutton
<james.dutton at gmail.com> wrote:
> I sounds me that you are just substituting "compile C source
file" for
> "compile LLVM IR source file" or "compile LLVM IR .o
file"
> The problem is you wish to link in the linux kernel headers at
"install" time.
> The problem I see is that the linux kernel headers are C source files,
> so how is that going to work?
> You need a C compiler to understand the Linux kernel headers.
>
Thanks, not that complicated.
We indeed need to handle kernel header file in install time, but not
the raw header .h file,
likely it will be a raw text file include all structure offset/sizeof
info, this is the file I called
as kernel ABI file, its size is small in target machine.

whatever how this raw text generated, that's not the core part of design.

On Tue, Apr 30, 2013 at 12:31 AM, John Criswell <criswell at illinois.edu>
wrote:
> On 4/28/13 11:42 AM, Jovi Zhang wrote:
>>
>> Hi,
>>
>> First of all, I didn't study on compiler too much, I'm a Linux
kernel
>> developer,
>> Now I have one idea about compile kernel module by using llvm
framework.
>>
>> In Linux world, traditionally compile Linux kernel module only have one
>> way:
>> compile it to machine code directly, deploy to target machine, then
>> run it in target machine.
>>
>> This have some problem from long term view, especially kernel module
>> compatibility issue,
>> we know that Linux kernel change fast, when kernel change the data
>> structure exposed to external
>> kernel modules, then all ko depend on that data structure will need
>> recompile, one case is
>> kernel upgrade.
>> This is really a problem, we already suffered it many years, in whole
>> Linux world.
>> Linux distribution suffer it, third party kernel modules developers
suffer
>> it.
>> hardware driver developers also suffer it, we cannot avoid re-compile
>> kernel modules whenever new kernel
>> release if the structure changed, why? because kernel modules depend
>> on kernel header data structure, this is the point.
>
>
> The basic idea seems feasible, but I think this is a "devil in the
details"
> sort of project.  As Joshua has pointed out, there are certain constructs
> you can't use (such as sizeof() in preprocessor #if statements), and I
think
> you'd need to enhance the LLVM IR to represent unknown structure sizes
> symbolically.  There is also the fact that the Clang frontend generates
> architecture-specific code and that there are LLVM optimizations that use
> the size of structures for optimization.  That is all stuff that your
> project would have to "fix."  You'd be swimming against a
strong current, so
> to speak.
Thanks.
>From LLVM's point of view, the design I described could be interpreted
as:
     "Implementing Portable sizeof, offsetof and Variable Sized
Structures in LLVM"

It's a random LLVM notes at:
    http://nondot.org/sabre/LLVMNotes/
    http://nondot.org/sabre/LLVMNotes/SizeOf-OffsetOf-VariableSizedStructs.txt

Basically the note wrote in there will solve main part of my problem,
only another problem
is structure field name need pass into IR file, I don't think it's a
hard technical problem, we have choose
many way to solve this, even put structure field name as comments into IR file.

Look, it's not that complicated, what we need is implementing a
portable sizeof/offsetof in LLVM,
and most likely my design is only another use case for this "portable
sizeof/offset" feature.
>
> More importantly, while you raise a number of limitations of linking native
> code for kernel modules, I don't think the Linux community sees this is
a
> disadvantage.  Rather, they see it as a way of encouraging developers to
> open-source their drivers and have them included in the Linux kernel tree.
> I'm not convinced that you'd get buy-in from the Linux community.
Your opinion is same as Karen Shaeffer.
We cannot upstream all kernel modules source code into mainline, this
is not simply a
technical problem or community problem, I believe that every company
have its own
kernel module which cannot upstream, or not worthwhile, if the company
is working
on kernel module development. upstream kernel modules just is a small
part of all kernel
modules in Linux world, this is the truth we need admit it.
>
> If this sort of project interests you, then I think you should aim for
> something smaller and that has more interest.  For example, I think the
NaCL
> developers at Google were looking at ways to make the LLVM IR more suitable
> as a format for shipping NaCL plugins.  If that hasn't been done yet,
it's
> an easier problem to tackle, and there is a group that is already
interested
> in having it.
>
> Finally, as a shameless plug, your project idea reminds me of the LLVA work
> that was done here at Illinois (which became the foundation for my own
> research on the Secure Virtual Architecture). That work aimed to replace
the
> processor native instruction set with the LLVM IR to allow the processor to
> adapt without requiring manual recompilation of software.  You can find
> those papers on the llvm.org publications page if you're interested.
>
> -- John T.
>
>