llvm dev - Jun 2019 - [CMake] External File Dependencies for Unit Tests

Thanks James!

Thanks for the help. I certainly see your point.
> In either case, if you wanted to change the test input, you'd need to
rerun CMake again
This is unfortunate, I would have hoped I could have CMake create
$(wildcard *.yaml) type of output in the make files. But it sounds like
this doesn't exist on make alternatives, so this wont work.

There is actually a function to get the Inputs directory,
http://llvm.org/doxygen/namespacellvm_1_1unittest.html. I think it is only
used in one test, though. Would you recommend that I do what is done in
unittest/ObjectYAML/MinidumpYAMLTest.cpp which just uses a string inside
the source file, or use an Inputs directory? Both are not great, as you
point out about using an Inputs directory.

Also, I'm not sure that I can create an object without invoking yaml2obj
itself, I can't seem to find a function to do this exposed by
libObjectYAML. If I do need to invoke yaml2obj, would you happen to know
how to tell CMake to build yaml2obj first, and also where would I find that
executable? Thanks so much, James.


On Mon, Jun 24, 2019 at 5:19 AM James Henderson <
jh7370.2008 at my.bristol.ac.uk> wrote:
> Hi Alex,
>
> Not answering your question directly, but I'd highly recommend against
> using CMake to bake the test outputs at the cmake step for several
> different reasons:
>
> 1) If yaml2obj changes, you'd have to rerun cmake to see whether the
new
> version would cause the tests to fail. This is not the default, so could
> easily lead to build-bot breakages.
> 2) It's not clear from your statement whether you're talking about
lit
> tests or gtest unit tests. If the former, there's no real precedence
for
> doing this. If the latter, you'd need to have your C++ code able to
detect
> the build output directory somehow, which feels like it would make the code
> at least unclear, and at worst unusable.
> 3) In either case, if you wanted to change the test input, you'd need
to
> rerun CMake again, rather than it just working. Runnning CMake is not
> cheap, especially on Windows for Visual Studio generators.
>
> As Jordan has mentioned offline, and assuming you're talking about
gtest,
> you're probably better off just calling the corresponding library
functions
> to generate the object from YAML. Otherwise, just use yaml2obj directly.
>
> If you want to explore your original suggestion further, and need help
> doing so, you probably need to give a bit more context as to what
you've
> tried and what didn't work about it.
>
> Regards,
>
> James
>
>
> On Sat, 22 Jun 2019 at 21:28, Alex Brachet-Mialot via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> Hi,
>>
>> I want to write unit tests for a project I am working on. I need to
>> create object files, preferably I want to have yaml files in the source
>> tree and have cmake generate object files using yaml2obj. Does anyone
know
>> what I would put in the CMakeLists.txt to get this behavior?
>>
>> For a concrete example if I didn’t explain clearly:
>>
>> In llvm/unittests/Object I would maybe have an Inputs directory, with a
>> basic-obj.yaml. In llvm/unittests/Object I would want a test case which
can
>> open Inputs/basic-obj.o.
>>
>> Thanks,
>> Alex
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190625/de2fa8ab/attachment.html>

Hi Alex,

Would you recommend that I do what is done in
> unittest/ObjectYAML/MinidumpYAMLTest.cpp which just uses a string inside
> the source file, or use an Inputs directory? Both are not great, as you
> point out about using an Inputs directory
>
I'd be inclined to go with the string in the source file approach. It keeps
the whole test localised to one place, making it easier to understand what
the test is actually testing.

Also, I'm not sure that I can create an object without invoking
yaml2obj
> itself, I can't seem to find a function to do this exposed by
libObjectYAML.
>
It looks to me like you'd need to move big chunks of code out of the
yaml2obj tool into the ObjectYAML library. The Minidump version was added
quite recently, and took the approach of doing all the layout in the
library, whereas for most other formats, the work is done in the executable
layer. I think it would be perfectly reasonable to do this code moving, and
would then allow you to use the library in your test setup, similar to how
minidump works.

If I do need to invoke yaml2obj, would you happen to know how to tell
CMake
> to build yaml2obj first, and also where would I find that executable?
>
I'm not a CMake expert by any means, but I think you would need to create a
dependency on yaml2obj from the test target, then add a step in the test's
CMakeLists.txt to run yaml2obj and generate your inputs. I'm still not
convinced that it's the right way to go though.

James

On Tue, 25 Jun 2019 at 06:33, Alex Brachet-Mialot <
alexbrachetmialot at gmail.com> wrote:
> Thanks James!
>
> Thanks for the help. I certainly see your point.
>
> > In either case, if you wanted to change the test input, you'd need
to
> rerun CMake again
> This is unfortunate, I would have hoped I could have CMake create
> $(wildcard *.yaml) type of output in the make files. But it sounds like
> this doesn't exist on make alternatives, so this wont work.
>
> There is actually a function to get the Inputs directory,
> http://llvm.org/doxygen/namespacellvm_1_1unittest.html. I think it is
> only used in one test, though. Would you recommend that I do what is done
> in unittest/ObjectYAML/MinidumpYAMLTest.cpp which just uses a string inside
> the source file, or use an Inputs directory? Both are not great, as you
> point out about using an Inputs directory.
>
> Also, I'm not sure that I can create an object without invoking
yaml2obj
> itself, I can't seem to find a function to do this exposed by
> libObjectYAML. If I do need to invoke yaml2obj, would you happen to know
> how to tell CMake to build yaml2obj first, and also where would I find that
> executable? Thanks so much, James.
>
>
> On Mon, Jun 24, 2019 at 5:19 AM James Henderson <
> jh7370.2008 at my.bristol.ac.uk> wrote:
>
>> Hi Alex,
>>
>> Not answering your question directly, but I'd highly recommend
against
>> using CMake to bake the test outputs at the cmake step for several
>> different reasons:
>>
>> 1) If yaml2obj changes, you'd have to rerun cmake to see whether
the new
>> version would cause the tests to fail. This is not the default, so
could
>> easily lead to build-bot breakages.
>> 2) It's not clear from your statement whether you're talking
about lit
>> tests or gtest unit tests. If the former, there's no real
precedence for
>> doing this. If the latter, you'd need to have your C++ code able to
detect
>> the build output directory somehow, which feels like it would make the
code
>> at least unclear, and at worst unusable.
>> 3) In either case, if you wanted to change the test input, you'd
need to
>> rerun CMake again, rather than it just working. Runnning CMake is not
>> cheap, especially on Windows for Visual Studio generators.
>>
>> As Jordan has mentioned offline, and assuming you're talking about
gtest,
>> you're probably better off just calling the corresponding library
functions
>> to generate the object from YAML. Otherwise, just use yaml2obj
directly.
>>
>> If you want to explore your original suggestion further, and need help
>> doing so, you probably need to give a bit more context as to what
you've
>> tried and what didn't work about it.
>>
>> Regards,
>>
>> James
>>
>>
>> On Sat, 22 Jun 2019 at 21:28, Alex Brachet-Mialot via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>>> Hi,
>>>
>>> I want to write unit tests for a project I am working on. I need to
>>> create object files, preferably I want to have yaml files in the
source
>>> tree and have cmake generate object files using yaml2obj. Does
anyone know
>>> what I would put in the CMakeLists.txt to get this behavior?
>>>
>>> For a concrete example if I didn’t explain clearly:
>>>
>>> In llvm/unittests/Object I would maybe have an Inputs directory,
with a
>>> basic-obj.yaml. In llvm/unittests/Object I would want a test case
which can
>>> open Inputs/basic-obj.o.
>>>
>>> Thanks,
>>> Alex
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190625/6b583e80/attachment.html>

On 25/06/2019 11:24, James Henderson via llvm-dev wrote:
> Hi Alex,
> 
>     Would you recommend that I do what is done in
>     unittest/ObjectYAML/MinidumpYAMLTest.cpp which just uses a string
>     inside the source file, or use an Inputs directory? Both are not
>     great, as you point out about using an Inputs directory
> 
> 
> I'd be inclined to go with the string in the source file approach. It 
> keeps the whole test localised to one place, making it easier to 
> understand what the test is actually testing.
> 
>     Also, I'm not sure that I can create an object without invoking
>     yaml2obj itself, I can't seem to find a function to do this exposed
>     by libObjectYAML.
> 
> 
> It looks to me like you'd need to move big chunks of code out of the 
> yaml2obj tool into the ObjectYAML library. The Minidump version was 
> added quite recently, and took the approach of doing all the layout in 
> the library, whereas for most other formats, the work is done in the 
> executable layer. I think it would be perfectly reasonable to do this 
> code moving, and would then allow you to use the library in your test 
> setup, similar to how minidump works.
[Disclaimer: I'm the person who added the minidump code.]

I would very much like to see this happen. Lldb has some unit tests 
which need to binary files to operate. The way that's handled right now 
is by shelling out to yaml2obj from inside the unit test (so, not from 
cmake). However, it would be infinitely better if we could just invoke a 
library function to do this conversion for us.

> 
>     If I do need to invoke yaml2obj, would you happen to know how to
>     tell CMake to build yaml2obj first, and also where would I find that
>     executable?
> 
> 
> I'm not a CMake expert by any means, but I think you would need to 
> create a dependency on yaml2obj from the test target, then add a step in 
> the test's CMakeLists.txt to run yaml2obj and generate your inputs.
I'm
> still not convinced that it's the right way to go though.
Running yaml2obj from the unit test like lldb does is probably a bit 
better than doing it from cmake. You can search for ReadYAMLObjectFile 
to see how it's done there. However, I wouldn't really recommend 
emulating that... :)

cheers,
pl