llvm dev - Sep 2014 - [LLVMdev] [RFC] Exhaustive bitcode compatibility tests for IR features

From the discussion of bitcode backward compatibility on the list, it seems we
lack systematic way to test every existing IR features. It is useful to keep a
test that exercises all the IR features for the current trunk and we can freeze
that test in the form of bitcode for backward compatibility test in the future.
I am proposing to implement such a test, which should try to accomplish
following features:
1. Try to keep it in one file so it is easy to freeze and move to the next
version.
2. Try to exercise and verify as much features as possible, which should
includes all the globals, instructions, metadata and intrinsics (and more).
3. The test should be easily maintainable. It should be easy to fix when broken
or get updated when assembly gets updated.
I am going to implement such test with a lengthy LLVM assembly, in the form of
the attachment (which I only tests for global variable). It is going to be long,
but someone must do it first. Future updates should be much simper. In the test,
I started with a default global variable and enumerate all the possible
attributes by changing them one by one. I try to keep the variable declaration
as simple as possible so that it won’t be affected by some simple assembly level
changes (like changing the parsing order of some attributes, since this is
supposed to be a bitcode compatibility test, not assembly test). I try to make
the tests as thorough as possible but avoid large duplications. For example, I
will tests Linkage attribute in both GlobalVariable as well as Function, but
probably not enumerate all the types I want to test. I will keep the tests for
Types in a different section since it is going to be huge and it is orthogonal
to the tests of globals.
When making a release or some big changes in IR, we can freeze the test by
generating bitcode, change the RUN line so it runs llvm-dis directly, and
modified the CHECKs that corresponding to the change. Then we can move on with a
new version of bitcode tests. This will add some more works for people who would
like to make changes to IR (which might be one more reason to discourage them
from breaking the compatibility). I will make sure to update the docs for
changing IRs after I add this test.

Currently, there are individual bitcode tests in the llvm which are created when
IR or intrinsics get changed. This exhaustive test shouldn’t overlap with the
existing ones since this tests is focusing on keeping a working up-to-date
version of IR tests. Both approaches of bitcode tests can co-exists. For
example, for small updates, we can add specific test cases like the ones
currently to test auto-upgrade, while updating the exhaustive bitcode test to
incorporate the new changes. When making huge upgrades and major releases, we
can freeze the exhaustive test for future checks.

For the actual test cases, I think it should be trivial for globals,
instructions, types (Correct me  if I am wrong), but intrinsics can be very
tricky. I am not sure how much compatibility is guaranteed for intrinsics, but
they can’t not be checked through llvm-as then llvm-dis. Intrinsics, as far as I
know, are coded like normal functions, globals or metadata. My current plan is
to write a separate tool to check the intrinsics actually supported in the IR or
backend. Intrinsic function might be the easiest since the supported ones should
all be declared in Intrinsics*.td and can be check by calling getIntrinsicID()
after reading the bitcode. Intrinsics coded as globals (llvm.used) or metadata
(llvm.loop) can be more tricky. Maybe another .td file with hardcoded intrinsics
for these cases should be added just for the testing purpose (we can add a new
API to it later so that we don’t need to do string compares to figure out these
intrinsics). After we have another tool to test intrinsics (which can be merged
with llvm-dis to save a RUN command and execution time), the attached test will
just need to be updated like following (checking llvm.global_ctors for example):
; RUN: verify-intrinsics %s.bc | FileCheck -check-prefix=CHECK-INT %s

%0 = type { i32, void ()*, i8* }
@llvm.global_ctors = appending global [1 x %0] [%0 { i32 65535, void ()* @ctor,
i8* @data }]
; CHECK: @llvm.global_ctors = appending global [1 x %0] [%0 { i32 65535, void
()* @ctor, i8* @data }]
; CHECK-INT: @llvm.global_ctors int_global_ctors

Let me know if there is better proposal.

Steven

-------------- next part --------------
A non-text attachment was scrubbed...
Name: llvm-3.6.ll
Type: application/octet-stream
Size: 3796 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140918/f522acf9/attachment.obj>

On 19/09/2014 04:01, Steven Wu wrote:
>  From the discussion of bitcode backward compatibility on the list, it
seems we lack systematic way to test every existing IR features. It is useful to
keep a test that exercises all the IR features for the current trunk and we can
freeze that test in the form of bitcode for backward compatibility test in the
future. I am proposing to implement such a test, which should try to accomplish
following features:
> 1. Try to keep it in one file so it is easy to freeze and move to the next
version.
> 2. Try to exercise and verify as much features as possible, which should
includes all the globals, instructions, metadata and intrinsics (and more).
> 3. The test should be easily maintainable. It should be easy to fix when
broken or get updated when assembly gets updated.
> I am going to implement such test with a lengthy LLVM assembly, in the form
of the attachment (which I only tests for global variable). It is going to be
long, but someone must do it first. Future updates should be much simper. In the
test, I started with a default global variable and enumerate all the possible
attributes by changing them one by one. I try to keep the variable declaration
as simple as possible so that it won’t be affected by some simple assembly level
changes (like changing the parsing order of some attributes, since this is
supposed to be a bitcode compatibility test, not assembly test). I try to make
the tests as thorough as possible but avoid large duplications. For example, I
will tests Linkage attribute in both GlobalVariable as well as Function, but
probably not enumerate all the types I want to test. I will keep the tests for
Types in a different section since it is going to be huge and it is orthogonal
to the tests of globals.
> When making a release or some big changes in IR, we can freeze the test by
generating bitcode, change the RUN line so it runs llvm-dis directly, and
modified the CHECKs that corresponding to the change. Then we can move on with a
new version of bitcode tests. This will add some more works for people who would
like to make changes to IR (which might be one more reason to discourage them
from breaking the compatibility). I will make sure to update the docs for
changing IRs after I add this test.
>
> Currently, there are individual bitcode tests in the llvm which are created
when IR or intrinsics get changed. This exhaustive test shouldn’t overlap with
the existing ones since this tests is focusing on keeping a working up-to-date
version of IR tests. Both approaches of bitcode tests can co-exists. For
example, for small updates, we can add specific test cases like the ones
currently to test auto-upgrade, while updating the exhaustive bitcode test to
incorporate the new changes. When making huge upgrades and major releases, we
can freeze the exhaustive test for future checks.
>
> For the actual test cases, I think it should be trivial for globals,
instructions, types (Correct me  if I am wrong), but intrinsics can be very
tricky. I am not sure how much compatibility is guaranteed for intrinsics, but
they can’t not be checked through llvm-as then llvm-dis. Intrinsics, as far as I
know, are coded like normal functions, globals or metadata. My current plan is
to write a separate tool to check the intrinsics actually supported in the IR or
backend. Intrinsic function might be the easiest since the supported ones should
all be declared in Intrinsics*.td and can be check by calling getIntrinsicID()
after reading the bitcode. Intrinsics coded as globals (llvm.used) or metadata
(llvm.loop) can be more tricky. Maybe another .td file with hardcoded intrinsics
for these cases should be added just for the testing purpose (we can add a new
API to it later so that we don’t need to do string compares to figure out these
intrinsics). After we have another tool to !
 test intr
 i
nsics (which can be merged with llvm-dis to save a RUN command and execution
time), the attached test will just need to be updated like following (checking
llvm.global_ctors for example):
> ; RUN: verify-intrinsics %s.bc | FileCheck -check-prefix=CHECK-INT %s
>
> %0 = type { i32, void ()*, i8* }
> @llvm.global_ctors = appending global [1 x %0] [%0 { i32 65535, void ()*
@ctor, i8* @data }]
> ; CHECK: @llvm.global_ctors = appending global [1 x %0] [%0 { i32 65535,
void ()* @ctor, i8* @data }]
> ; CHECK-INT: @llvm.global_ctors int_global_ctors
>
> Let me know if there is better proposal.
Hi Steven,

thanks for taking care of this. I think bitcode compatibility is 
important and it believe the kind of test you propose takes a reasonable 
approach.

Cheers,
Tobias

Thanks Tobias. I will start writing more tests and the utility to check
intrinsics.
> On Sep 18, 2014, at 11:20 PM, Tobias Grosser <tobias at grosser.es>
wrote:
> 
> On 19/09/2014 04:01, Steven Wu wrote:
>> From the discussion of bitcode backward compatibility on the list, it
seems we lack systematic way to test every existing IR features. It is useful to
keep a test that exercises all the IR features for the current trunk and we can
freeze that test in the form of bitcode for backward compatibility test in the
future. I am proposing to implement such a test, which should try to accomplish
following features:
>> 1. Try to keep it in one file so it is easy to freeze and move to the
next version.
>> 2. Try to exercise and verify as much features as possible, which
should includes all the globals, instructions, metadata and intrinsics (and
more).
>> 3. The test should be easily maintainable. It should be easy to fix
when broken or get updated when assembly gets updated.
>> I am going to implement such test with a lengthy LLVM assembly, in the
form of the attachment (which I only tests for global variable). It is going to
be long, but someone must do it first. Future updates should be much simper. In
the test, I started with a default global variable and enumerate all the
possible attributes by changing them one by one. I try to keep the variable
declaration as simple as possible so that it won’t be affected by some simple
assembly level changes (like changing the parsing order of some attributes,
since this is supposed to be a bitcode compatibility test, not assembly test). I
try to make the tests as thorough as possible but avoid large duplications. For
example, I will tests Linkage attribute in both GlobalVariable as well as
Function, but probably not enumerate all the types I want to test. I will keep
the tests for Types in a different section since it is going to be huge and it
is orthogonal to the tests of globals.
>> When making a release or some big changes in IR, we can freeze the test
by generating bitcode, change the RUN line so it runs llvm-dis directly, and
modified the CHECKs that corresponding to the change. Then we can move on with a
new version of bitcode tests. This will add some more works for people who would
like to make changes to IR (which might be one more reason to discourage them
from breaking the compatibility). I will make sure to update the docs for
changing IRs after I add this test.
>> 
>> Currently, there are individual bitcode tests in the llvm which are
created when IR or intrinsics get changed. This exhaustive test shouldn’t
overlap with the existing ones since this tests is focusing on keeping a working
up-to-date version of IR tests. Both approaches of bitcode tests can co-exists.
For example, for small updates, we can add specific test cases like the ones
currently to test auto-upgrade, while updating the exhaustive bitcode test to
incorporate the new changes. When making huge upgrades and major releases, we
can freeze the exhaustive test for future checks.
>> 
>> For the actual test cases, I think it should be trivial for globals,
instructions, types (Correct me  if I am wrong), but intrinsics can be very
tricky. I am not sure how much compatibility is guaranteed for intrinsics, but
they can’t not be checked through llvm-as then llvm-dis. Intrinsics, as far as I
know, are coded like normal functions, globals or metadata. My current plan is
to write a separate tool to check the intrinsics actually supported in the IR or
backend. Intrinsic function might be the easiest since the supported ones should
all be declared in Intrinsics*.td and can be check by calling getIntrinsicID()
after reading the bitcode. Intrinsics coded as globals (llvm.used) or metadata
(llvm.loop) can be more tricky. Maybe another .td file with hardcoded intrinsics
for these cases should be added just for the testing purpose (we can add a new
API to it later so that we don’t need to do string compares to figure out these
intrinsics). After we have another tool to test intr
> i
> nsics (which can be merged with llvm-dis to save a RUN command and
execution time), the attached test will just need to be updated like following
(checking llvm.global_ctors for example):
>> ; RUN: verify-intrinsics %s.bc | FileCheck -check-prefix=CHECK-INT %s
>> 
>> %0 = type { i32, void ()*, i8* }
>> @llvm.global_ctors = appending global [1 x %0] [%0 { i32 65535, void
()* @ctor, i8* @data }]
>> ; CHECK: @llvm.global_ctors = appending global [1 x %0] [%0 { i32
65535, void ()* @ctor, i8* @data }]
>> ; CHECK-INT: @llvm.global_ctors int_global_ctors
>> 
>> Let me know if there is better proposal.
> 
> Hi Steven,
> 
> thanks for taking care of this. I think bitcode compatibility is important
and it believe the kind of test you propose takes a reasonable approach.
> 
> Cheers,
> Tobias
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140919/03b41888/attachment.html>

So the proposal is that during development new features are added to
test/Features/compatibility.ll (or some other name). When 3.6 is
released, we will

* assemble the file with llvm-as-3.6.
* Check in the .bc file as test/Features/Input/compatibility-3.6.bc
* Copy test/Features/compatibility.ll to
test/Features/compatibility-3.6.ll and change it to run llvm-dis
directly on the 3.6 bitcode.

And then when 4.1 is released we have a discussion on what we want to
drop from the old .bc files.

Correct? If so, sounds reasonable to me.


On 18 September 2014 22:01, Steven Wu <stevenwu at apple.com>
wrote:
> From the discussion of bitcode backward compatibility on the list, it seems
we lack systematic way to test every existing IR features. It is useful to keep
a test that exercises all the IR features for the current trunk and we can
freeze that test in the form of bitcode for backward compatibility test in the
future. I am proposing to implement such a test, which should try to accomplish
following features:
> 1. Try to keep it in one file so it is easy to freeze and move to the next
version.
> 2. Try to exercise and verify as much features as possible, which should
includes all the globals, instructions, metadata and intrinsics (and more).
> 3. The test should be easily maintainable. It should be easy to fix when
broken or get updated when assembly gets updated.
> I am going to implement such test with a lengthy LLVM assembly, in the form
of the attachment (which I only tests for global variable). It is going to be
long, but someone must do it first. Future updates should be much simper. In the
test, I started with a default global variable and enumerate all the possible
attributes by changing them one by one. I try to keep the variable declaration
as simple as possible so that it won’t be affected by some simple assembly level
changes (like changing the parsing order of some attributes, since this is
supposed to be a bitcode compatibility test, not assembly test). I try to make
the tests as thorough as possible but avoid large duplications. For example, I
will tests Linkage attribute in both GlobalVariable as well as Function, but
probably not enumerate all the types I want to test. I will keep the tests for
Types in a different section since it is going to be huge and it is orthogonal
to the tests of globals.
> When making a release or some big changes in IR, we can freeze the test by
generating bitcode, change the RUN line so it runs llvm-dis directly, and
modified the CHECKs that corresponding to the change. Then we can move on with a
new version of bitcode tests. This will add some more works for people who would
like to make changes to IR (which might be one more reason to discourage them
from breaking the compatibility). I will make sure to update the docs for
changing IRs after I add this test.
>
> Currently, there are individual bitcode tests in the llvm which are created
when IR or intrinsics get changed. This exhaustive test shouldn’t overlap with
the existing ones since this tests is focusing on keeping a working up-to-date
version of IR tests. Both approaches of bitcode tests can co-exists. For
example, for small updates, we can add specific test cases like the ones
currently to test auto-upgrade, while updating the exhaustive bitcode test to
incorporate the new changes. When making huge upgrades and major releases, we
can freeze the exhaustive test for future checks.
>
> For the actual test cases, I think it should be trivial for globals,
instructions, types (Correct me  if I am wrong), but intrinsics can be very
tricky. I am not sure how much compatibility is guaranteed for intrinsics, but
they can’t not be checked through llvm-as then llvm-dis. Intrinsics, as far as I
know, are coded like normal functions, globals or metadata. My current plan is
to write a separate tool to check the intrinsics actually supported in the IR or
backend. Intrinsic function might be the easiest since the supported ones should
all be declared in Intrinsics*.td and can be check by calling getIntrinsicID()
after reading the bitcode. Intrinsics coded as globals (llvm.used) or metadata
(llvm.loop) can be more tricky. Maybe another .td file with hardcoded intrinsics
for these cases should be added just for the testing purpose (we can add a new
API to it later so that we don’t need to do string compares to figure out these
intrinsics). After we have another tool to test intrinsics (which can be merged
with llvm-dis to save a RUN command and execution time), the attached test will
just need to be updated like following (checking llvm.global_ctors for example):
> ; RUN: verify-intrinsics %s.bc | FileCheck -check-prefix=CHECK-INT %s
>
> %0 = type { i32, void ()*, i8* }
> @llvm.global_ctors = appending global [1 x %0] [%0 { i32 65535, void ()*
@ctor, i8* @data }]
> ; CHECK: @llvm.global_ctors = appending global [1 x %0] [%0 { i32 65535,
void ()* @ctor, i8* @data }]
> ; CHECK-INT: @llvm.global_ctors int_global_ctors
>
> Let me know if there is better proposal.
>
> Steven
>

> On Sep 19, 2014, at 9:57 AM, Rafael Espíndola <rafael.espindola at
gmail.com> wrote:
> 
> So the proposal is that during development new features are added to
> test/Features/compatibility.ll (or some other name). When 3.6 is
> released, we will
> 
> * assemble the file with llvm-as-3.6.
> * Check in the .bc file as test/Features/Input/compatibility-3.6.bc
> * Copy test/Features/compatibility.ll to
> test/Features/compatibility-3.6.ll and change it to run llvm-dis
> directly on the 3.6 bitcode.
> 
> And then when 4.1 is released we have a discussion on what we want to
> drop from the old .bc files.
> 
> Correct? If so, sounds reasonable to me.
Correct. This is exactly what I mean.
> 
> On 18 September 2014 22:01, Steven Wu <stevenwu at apple.com> wrote:
>> From the discussion of bitcode backward compatibility on the list, it
seems we lack systematic way to test every existing IR features. It is useful to
keep a test that exercises all the IR features for the current trunk and we can
freeze that test in the form of bitcode for backward compatibility test in the
future. I am proposing to implement such a test, which should try to accomplish
following features:
>> 1. Try to keep it in one file so it is easy to freeze and move to the
next version.
>> 2. Try to exercise and verify as much features as possible, which
should includes all the globals, instructions, metadata and intrinsics (and
more).
>> 3. The test should be easily maintainable. It should be easy to fix
when broken or get updated when assembly gets updated.
>> I am going to implement such test with a lengthy LLVM assembly, in the
form of the attachment (which I only tests for global variable). It is going to
be long, but someone must do it first. Future updates should be much simper. In
the test, I started with a default global variable and enumerate all the
possible attributes by changing them one by one. I try to keep the variable
declaration as simple as possible so that it won’t be affected by some simple
assembly level changes (like changing the parsing order of some attributes,
since this is supposed to be a bitcode compatibility test, not assembly test). I
try to make the tests as thorough as possible but avoid large duplications. For
example, I will tests Linkage attribute in both GlobalVariable as well as
Function, but probably not enumerate all the types I want to test. I will keep
the tests for Types in a different section since it is going to be huge and it
is orthogonal to the tests of globals.
>> When making a release or some big changes in IR, we can freeze the test
by generating bitcode, change the RUN line so it runs llvm-dis directly, and
modified the CHECKs that corresponding to the change. Then we can move on with a
new version of bitcode tests. This will add some more works for people who would
like to make changes to IR (which might be one more reason to discourage them
from breaking the compatibility). I will make sure to update the docs for
changing IRs after I add this test.
>> 
>> Currently, there are individual bitcode tests in the llvm which are
created when IR or intrinsics get changed. This exhaustive test shouldn’t
overlap with the existing ones since this tests is focusing on keeping a working
up-to-date version of IR tests. Both approaches of bitcode tests can co-exists.
For example, for small updates, we can add specific test cases like the ones
currently to test auto-upgrade, while updating the exhaustive bitcode test to
incorporate the new changes. When making huge upgrades and major releases, we
can freeze the exhaustive test for future checks.
>> 
>> For the actual test cases, I think it should be trivial for globals,
instructions, types (Correct me  if I am wrong), but intrinsics can be very
tricky. I am not sure how much compatibility is guaranteed for intrinsics, but
they can’t not be checked through llvm-as then llvm-dis. Intrinsics, as far as I
know, are coded like normal functions, globals or metadata. My current plan is
to write a separate tool to check the intrinsics actually supported in the IR or
backend. Intrinsic function might be the easiest since the supported ones should
all be declared in Intrinsics*.td and can be check by calling getIntrinsicID()
after reading the bitcode. Intrinsics coded as globals (llvm.used) or metadata
(llvm.loop) can be more tricky. Maybe another .td file with hardcoded intrinsics
for these cases should be added just for the testing purpose (we can add a new
API to it later so that we don’t need to do string compares to figure out these
intrinsics). After we have another tool to test intrinsics (which can be merged
with llvm-dis to save a RUN command and execution time), the attached test will
just need to be updated like following (checking llvm.global_ctors for example):
>> ; RUN: verify-intrinsics %s.bc | FileCheck -check-prefix=CHECK-INT %s
>> 
>> %0 = type { i32, void ()*, i8* }
>> @llvm.global_ctors = appending global [1 x %0] [%0 { i32 65535, void
()* @ctor, i8* @data }]
>> ; CHECK: @llvm.global_ctors = appending global [1 x %0] [%0 { i32
65535, void ()* @ctor, i8* @data }]
>> ; CHECK-INT: @llvm.global_ctors int_global_ctors
>> 
>> Let me know if there is better proposal.
>> 
>> Steven
>>

This sounds like a good plan. Initially, the tests will be very
repetitious, but over time as we make changes they will diverge. For
example, before swapping the order of 'alias' and linkage in the IL, we
would've had:

@a = alias weak i8* @target
; CHECK: @a = alias weak i8* @target

This would've been copied to compatibility-3.N.ll for the previous release
before the change, and the change would have to update to the new llvm-dis
syntax:

@a = alias weak i8* @target
; CHECK: @a = weak alias i8* @target

So we need the duplication because we expect llvm-dis to slowly diverge
from the input that we would have fed to llvm-as-3.N.

On Fri, Sep 19, 2014 at 9:57 AM, Rafael Espíndola <
rafael.espindola at gmail.com> wrote:
> So the proposal is that during development new features are added to
> test/Features/compatibility.ll (or some other name). When 3.6 is
> released, we will
>
> * assemble the file with llvm-as-3.6.
> * Check in the .bc file as test/Features/Input/compatibility-3.6.bc
> * Copy test/Features/compatibility.ll to
> test/Features/compatibility-3.6.ll and change it to run llvm-dis
> directly on the 3.6 bitcode.
>
> And then when 4.1 is released we have a discussion on what we want to
> drop from the old .bc files.
>
> Correct? If so, sounds reasonable to me.
>
>
> On 18 September 2014 22:01, Steven Wu <stevenwu at apple.com> wrote:
> > From the discussion of bitcode backward compatibility on the list, it
> seems we lack systematic way to test every existing IR features. It is
> useful to keep a test that exercises all the IR features for the current
> trunk and we can freeze that test in the form of bitcode for backward
> compatibility test in the future. I am proposing to implement such a test,
> which should try to accomplish following features:
> > 1. Try to keep it in one file so it is easy to freeze and move to the
> next version.
> > 2. Try to exercise and verify as much features as possible, which
should
> includes all the globals, instructions, metadata and intrinsics (and more).
> > 3. The test should be easily maintainable. It should be easy to fix
when
> broken or get updated when assembly gets updated.
> > I am going to implement such test with a lengthy LLVM assembly, in the
> form of the attachment (which I only tests for global variable). It is
> going to be long, but someone must do it first. Future updates should be
> much simper. In the test, I started with a default global variable and
> enumerate all the possible attributes by changing them one by one. I try to
> keep the variable declaration as simple as possible so that it won’t be
> affected by some simple assembly level changes (like changing the parsing
> order of some attributes, since this is supposed to be a bitcode
> compatibility test, not assembly test). I try to make the tests as thorough
> as possible but avoid large duplications. For example, I will tests Linkage
> attribute in both GlobalVariable as well as Function, but probably not
> enumerate all the types I want to test. I will keep the tests for Types in
> a different section since it is going to be huge and it is orthogonal to
> the tests of globals.
> > When making a release or some big changes in IR, we can freeze the
test
> by generating bitcode, change the RUN line so it runs llvm-dis directly,
> and modified the CHECKs that corresponding to the change. Then we can move
> on with a new version of bitcode tests. This will add some more works for
> people who would like to make changes to IR (which might be one more reason
> to discourage them from breaking the compatibility). I will make sure to
> update the docs for changing IRs after I add this test.
> >
> > Currently, there are individual bitcode tests in the llvm which are
> created when IR or intrinsics get changed. This exhaustive test shouldn’t
> overlap with the existing ones since this tests is focusing on keeping a
> working up-to-date version of IR tests. Both approaches of bitcode tests
> can co-exists. For example, for small updates, we can add specific test
> cases like the ones currently to test auto-upgrade, while updating the
> exhaustive bitcode test to incorporate the new changes. When making huge
> upgrades and major releases, we can freeze the exhaustive test for future
> checks.
> >
> > For the actual test cases, I think it should be trivial for globals,
> instructions, types (Correct me  if I am wrong), but intrinsics can be very
> tricky. I am not sure how much compatibility is guaranteed for intrinsics,
> but they can’t not be checked through llvm-as then llvm-dis. Intrinsics, as
> far as I know, are coded like normal functions, globals or metadata. My
> current plan is to write a separate tool to check the intrinsics actually
> supported in the IR or backend. Intrinsic function might be the easiest
> since the supported ones should all be declared in Intrinsics*.td and can
> be check by calling getIntrinsicID() after reading the bitcode. Intrinsics
> coded as globals (llvm.used) or metadata (llvm.loop) can be more tricky.
> Maybe another .td file with hardcoded intrinsics for these cases should be
> added just for the testing purpose (we can add a new API to it later so
> that we don’t need to do string compares to figure out these intrinsics).
> After we have another tool to test intrinsics (which can be merged with
> llvm-dis to save a RUN command and execution time), the attached test will
> just need to be updated like following (checking llvm.global_ctors for
> example):
> > ; RUN: verify-intrinsics %s.bc | FileCheck -check-prefix=CHECK-INT %s
> >
> > %0 = type { i32, void ()*, i8* }
> > @llvm.global_ctors = appending global [1 x %0] [%0 { i32 65535, void
()*
> @ctor, i8* @data }]
> > ; CHECK: @llvm.global_ctors = appending global [1 x %0] [%0 { i32
65535,
> void ()* @ctor, i8* @data }]
> > ; CHECK-INT: @llvm.global_ctors int_global_ctors
> >
> > Let me know if there is better proposal.
> >
> > Steven
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140919/724e9404/attachment.html>

On Thu, Sep 18, 2014 at 7:01 PM, Steven Wu <stevenwu at apple.com> wrote:
> From the discussion of bitcode backward compatibility on the list, it
> seems we lack systematic way to test every existing IR features.

After reading the rest of the proposal, what you are suggesting doesn't
sound like a "systematic way to test every existing IR feature". You
are
missing a key part which is to ensure that the tests you produce actually
are exhaustive. Otherwise, you are just hoping, which is not an improvement
over the current state of affairs.

It seems like it would be much better to essentially put
http://llvm.org/docs/BitCodeFormat.html (and/or all other relevant
information about what constitutes a valid bitcode module) in a
machine-readable format (e.g. in TableGen, or YAML) and use that to
generate a bitcode file that exercises all the features of interest. It
would also be used to generate a verifier which checks that the bitcode
conforms to that schema: this allows automatically catching when changes
are introduced that require updating the "schema".

-- Sean Silva


> It is useful to keep a test that exercises all the IR features for the
> current trunk and we can freeze that test in the form of bitcode for
> backward compatibility test in the future. I am proposing to implement such
> a test, which should try to accomplish following features:
> 1. Try to keep it in one file so it is easy to freeze and move to the next
> version.
> 2. Try to exercise and verify as much features as possible, which should
> includes all the globals, instructions, metadata and intrinsics (and more).
> 3. The test should be easily maintainable. It should be easy to fix when
> broken or get updated when assembly gets updated.
> I am going to implement such test with a lengthy LLVM assembly, in the
> form of the attachment (which I only tests for global variable). It is
> going to be long, but someone must do it first. Future updates should be
> much simper. In the test, I started with a default global variable and
> enumerate all the possible attributes by changing them one by one. I try to
> keep the variable declaration as simple as possible so that it won’t be
> affected by some simple assembly level changes (like changing the parsing
> order of some attributes, since this is supposed to be a bitcode
> compatibility test, not assembly test). I try to make the tests as thorough
> as possible but avoid large duplications. For example, I will tests Linkage
> attribute in both GlobalVariable as well as Function, but probably not
> enumerate all the types I want to test. I will keep the tests for Types in
> a different section since it is going to be huge and it is orthogonal to
> the tests of globals.
> When making a release or some big changes in IR, we can freeze the test by
> generating bitcode, change the RUN line so it runs llvm-dis directly, and
> modified the CHECKs that corresponding to the change. Then we can move on
> with a new version of bitcode tests. This will add some more works for
> people who would like to make changes to IR (which might be one more reason
> to discourage them from breaking the compatibility). I will make sure to
> update the docs for changing IRs after I add this test.
>
> Currently, there are individual bitcode tests in the llvm which are
> created when IR or intrinsics get changed. This exhaustive test shouldn’t
> overlap with the existing ones since this tests is focusing on keeping a
> working up-to-date version of IR tests. Both approaches of bitcode tests
> can co-exists. For example, for small updates, we can add specific test
> cases like the ones currently to test auto-upgrade, while updating the
> exhaustive bitcode test to incorporate the new changes. When making huge
> upgrades and major releases, we can freeze the exhaustive test for future
> checks.
>
> For the actual test cases, I think it should be trivial for globals,
> instructions, types (Correct me  if I am wrong), but intrinsics can be very
> tricky. I am not sure how much compatibility is guaranteed for intrinsics,
> but they can’t not be checked through llvm-as then llvm-dis. Intrinsics, as
> far as I know, are coded like normal functions, globals or metadata. My
> current plan is to write a separate tool to check the intrinsics actually
> supported in the IR or backend. Intrinsic function might be the easiest
> since the supported ones should all be declared in Intrinsics*.td and can
> be check by calling getIntrinsicID() after reading the bitcode. Intrinsics
> coded as globals (llvm.used) or metadata (llvm.loop) can be more tricky.
> Maybe another .td file with hardcoded intrinsics for these cases should be
> added just for the testing purpose (we can add a new API to it later so
> that we don’t need to do string compares to figure out these intrinsics).
> After we have another tool to test intrinsics (which can be merged with
> llvm-dis to save a RUN command and execution time), the attached test will
> just need to be updated like following (checking llvm.global_ctors for
> example):
> ; RUN: verify-intrinsics %s.bc | FileCheck -check-prefix=CHECK-INT %s
>
> %0 = type { i32, void ()*, i8* }
> @llvm.global_ctors = appending global [1 x %0] [%0 { i32 65535, void ()*
> @ctor, i8* @data }]
> ; CHECK: @llvm.global_ctors = appending global [1 x %0] [%0 { i32 65535,
> void ()* @ctor, i8* @data }]
> ; CHECK-INT: @llvm.global_ctors int_global_ctors
>
> Let me know if there is better proposal.
>
> Steven
>
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140919/469e2f43/attachment.html>

> On Sep 19, 2014, at 3:25 PM, Sean Silva <chisophugis at gmail.com>
wrote:
> 
> 
> 
> On Thu, Sep 18, 2014 at 7:01 PM, Steven Wu <stevenwu at apple.com
<mailto:stevenwu at apple.com>> wrote:
> From the discussion of bitcode backward compatibility on the list, it seems
we lack systematic way to test every existing IR features.
> 
> After reading the rest of the proposal, what you are suggesting doesn't
sound like a "systematic way to test every existing IR feature". You
are missing a key part which is to ensure that the tests you produce actually
are exhaustive. Otherwise, you are just hoping, which is not an improvement over
the current state of affairs.
> 
I agree that my proposed test is not truly exhaustive but a truly exhaustive one
will be too large to checkin and pass around. Even trying to generating tests on
the fly will require two versions of LLVM for the compatibility test.
More importantly, the bitcode compatibility test is more than just encoding and
decoding the bitcode format, it needs to preserve the same meaning across the
versions. Other than the ability to generate exhaustive tests, there need a
mechanics to tell the set of IR and equal to the other set of IR.
I thought about auto generate tests (even using doxygen to keep up with IR
changes) but I cannot come up with a simple and elegant solution. If you have a
work-flow in mind, please suggest.
> It seems like it would be much better to essentially put
http://llvm.org/docs/BitCodeFormat.html
<http://llvm.org/docs/BitCodeFormat.html> (and/or all other relevant
information about what constitutes a valid bitcode module) in a machine-readable
format (e.g. in TableGen, or YAML) and use that to generate a bitcode file that
exercises all the features of interest. It would also be used to generate a
verifier which checks that the bitcode conforms to that schema: this allows
automatically catching when changes are introduced that require updating the
"schema".
> 
> -- Sean Silva
> 
>  
> It is useful to keep a test that exercises all the IR features for the
current trunk and we can freeze that test in the form of bitcode for backward
compatibility test in the future. I am proposing to implement such a test, which
should try to accomplish following features:
> 1. Try to keep it in one file so it is easy to freeze and move to the next
version.
> 2. Try to exercise and verify as much features as possible, which should
includes all the globals, instructions, metadata and intrinsics (and more).
> 3. The test should be easily maintainable. It should be easy to fix when
broken or get updated when assembly gets updated.
> I am going to implement such test with a lengthy LLVM assembly, in the form
of the attachment (which I only tests for global variable). It is going to be
long, but someone must do it first. Future updates should be much simper. In the
test, I started with a default global variable and enumerate all the possible
attributes by changing them one by one. I try to keep the variable declaration
as simple as possible so that it won’t be affected by some simple assembly level
changes (like changing the parsing order of some attributes, since this is
supposed to be a bitcode compatibility test, not assembly test). I try to make
the tests as thorough as possible but avoid large duplications. For example, I
will tests Linkage attribute in both GlobalVariable as well as Function, but
probably not enumerate all the types I want to test. I will keep the tests for
Types in a different section since it is going to be huge and it is orthogonal
to the tests of globals.
> When making a release or some big changes in IR, we can freeze the test by
generating bitcode, change the RUN line so it runs llvm-dis directly, and
modified the CHECKs that corresponding to the change. Then we can move on with a
new version of bitcode tests. This will add some more works for people who would
like to make changes to IR (which might be one more reason to discourage them
from breaking the compatibility). I will make sure to update the docs for
changing IRs after I add this test.
> 
> Currently, there are individual bitcode tests in the llvm which are created
when IR or intrinsics get changed. This exhaustive test shouldn’t overlap with
the existing ones since this tests is focusing on keeping a working up-to-date
version of IR tests. Both approaches of bitcode tests can co-exists. For
example, for small updates, we can add specific test cases like the ones
currently to test auto-upgrade, while updating the exhaustive bitcode test to
incorporate the new changes. When making huge upgrades and major releases, we
can freeze the exhaustive test for future checks.
> 
> For the actual test cases, I think it should be trivial for globals,
instructions, types (Correct me  if I am wrong), but intrinsics can be very
tricky. I am not sure how much compatibility is guaranteed for intrinsics, but
they can’t not be checked through llvm-as then llvm-dis. Intrinsics, as far as I
know, are coded like normal functions, globals or metadata. My current plan is
to write a separate tool to check the intrinsics actually supported in the IR or
backend. Intrinsic function might be the easiest since the supported ones should
all be declared in Intrinsics*.td and can be check by calling getIntrinsicID()
after reading the bitcode. Intrinsics coded as globals (llvm.used) or metadata
(llvm.loop) can be more tricky. Maybe another .td file with hardcoded intrinsics
for these cases should be added just for the testing purpose (we can add a new
API to it later so that we don’t need to do string compares to figure out these
intrinsics). After we have another tool to test intrinsics (which can be merged
with llvm-dis to save a RUN command and execution time), the attached test will
just need to be updated like following (checking llvm.global_ctors for example):
> ; RUN: verify-intrinsics %s.bc | FileCheck -check-prefix=CHECK-INT %s
> 
> %0 = type { i32, void ()*, i8* }
> @llvm.global_ctors = appending global [1 x %0] [%0 { i32 65535, void ()*
@ctor, i8* @data }]
> ; CHECK: @llvm.global_ctors = appending global [1 x %0] [%0 { i32 65535,
void ()* @ctor, i8* @data }]
> ; CHECK-INT: @llvm.global_ctors int_global_ctors
> 
> Let me know if there is better proposal.
> 
> Steven
> 
> 
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu <mailto:LLVMdev at cs.uiuc.edu>        
http://llvm.cs.uiuc.edu <http://llvm.cs.uiuc.edu/>
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
<http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140919/d2c508ad/attachment.html>