Hi Paul, Thank you for information about the [[:space:]] character-class. About performance I tested on Clang/LLVM test suite. I try to profile and the problem is that I used regular expressions a lot for supporting some new features and functions in your regex library are very slow . Regex library is very old and quite awkward, in my opinion. May be you will see some ways to improve performance, if some features are decided to include in LLVM FileCheck and I publish patches for each feature separately(I saw your comment in published patch). I see that there are a lot of opinions, may be it will be better to vote. I suggest vote in file https://docs.google.com/spreadsheets/d/1p8Hi_PH3Nd2kEtYveCwKXENmJGOzYWW6us0XK7eVvOw/edit?usp=sharing. There will be history and it's possible to check that nobody votes twice. Then somebody can stop voting at one moment and I will be able to understand what patches I should do and publish if there are some changes you would like. Thanks, Elena. -----Original Message----- From: Robinson, Paul [mailto:paul.robinson at sony.com] Sent: Friday, May 27, 2016 2:26 AM To: Elena Lepilkina <Elena.Lepilkina at synopsys.com>; llvm-dev at lists.llvm.org Subject: RE: [llvm-dev] RFC: FileCheck Enhancements> From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of > Elena Lepilkina via llvm-dev > Sent: Tuesday, May 24, 2016 6:51 AM > To: llvm-dev > Subject: [llvm-dev] RFC: FileCheck Enhancements > > Hi everyone, > > There was idea to add new directives to FileCheck: > 1. Directive to use some patterns as named template with or without parameters.Seems plausible, although I find the proposed syntax for a template call with parameters to be very awkward.> 2. CHECK-INCLUDE - Directive to include other file with checks to another. > 3. Expressions repeat for CHECK - If statement should be checked several times repeat modifiers {n}, {n,m} , {,n}, {n,}, *, + can be used. > 4. Repeat in regexs - Repeat with current number should become > available by using {n}, {n,m} , {,n}, {n,} 5. CHECK-LABEL-DAG - Not sequential order of labels.After reading through the google doc I see what you are trying to do. The motivation for CHECK-DAG is to reduce future test "churn" in case someone makes a change that influences the output order of something, but the order is not important to the test. The motivation is NOT to support instability in the output order across runs of the same compiler. Just so we're clear on that. In principle I could see how the order of blocks or entire functions could change over time, with no real relevance to a test, but how necessary is it really? I have seen work go into Clang/LLVM over the years to ensure stable output order, and my impression has been that these changes typically have little complexity or performance effect while significantly simplifying test effort.> 6. Check statement for words only - // CHECK-WORD, // CHECK-WORD-NEXT, // CHECK-WORD-SAME, // CHECK-WORD-DAG, // CHECK-WORD-NOT.I would expect a regex package to provide a word-break meta symbol, in which case this feature would be redundant. I admit I have not checked whether the package we use has this feature, or how it would be spelled.> 7. Wildcard for prefixes - If some statements should be checked regardless prefix, it should be used //{{*}}, //{{*}}-NEXT, //{{*}}-SAME and etc. > 8. Prefix with regular expressions - If statement should be checked if prefix matches some regular expression, it should be used {{regex}}:, {{regex}}-NEXT and etc. > > More information in file https://docs.google.com/document/d/1wAKNzU7-S2EeK1-aADwgP8dEiKfByKNazonybCQW3zs/edit?usp=sharing.I noticed the google doc stated that multi-line patterns are not supported. That's not actually the case, although it's a bit obscure: the [[:space:]] character-class will match EOL and allow you to write a multi-line CHECK.> > Now we have prototype with these features. It's tested on LLVM 3.8. > There was found unsupported before directive in old test. Bug about this - https://llvm.org/bugs/show_bug.cgi?id=27852. > > There is about 6% slowdown with new features when we tested them on 3.8.Is that a 6% slowdown just in FileCheck, or when running the entire Clang/LLVM test suite? Making the entire test run 6% more expensive seems like a lot. --paulr> > I see that there are some changes in FileCheck LLVM 3.9 with new features too. We can publish patch for 3.8 and it can be adapted for LLVM 3.9. Is it interesting for anyone? And how will be better to publish patch as for 3.8 or for 3.9? > > Thanks, > Elena.
Hi all, I' ll be glad to hear more opinions and may be some suggestions how to improve new features (may be there are ideas how template descriptions can be done simplier). After we try to accept your ideas and opinions to make new FileCheck features better. I will make changes and publish separate patches a month later, because I'll take a holiday in June. Thanks, Elena. -----Original Message----- From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of Elena Lepilkina via llvm-dev Sent: Friday, May 27, 2016 12:09 PM To: Robinson, Paul <paul.robinson at sony.com>; llvm-dev at lists.llvm.org Subject: Re: [llvm-dev] RFC: FileCheck Enhancements Hi Paul, Thank you for information about the [[:space:]] character-class. About performance I tested on Clang/LLVM test suite. I try to profile and the problem is that I used regular expressions a lot for supporting some new features and functions in your regex library are very slow . Regex library is very old and quite awkward, in my opinion. May be you will see some ways to improve performance, if some features are decided to include in LLVM FileCheck and I publish patches for each feature separately(I saw your comment in published patch). I see that there are a lot of opinions, may be it will be better to vote. I suggest vote in file https://docs.google.com/spreadsheets/d/1p8Hi_PH3Nd2kEtYveCwKXENmJGOzYWW6us0XK7eVvOw/edit?usp=sharing. There will be history and it's possible to check that nobody votes twice. Then somebody can stop voting at one moment and I will be able to understand what patches I should do and publish if there are some changes you would like. Thanks, Elena. -----Original Message----- From: Robinson, Paul [mailto:paul.robinson at sony.com] Sent: Friday, May 27, 2016 2:26 AM To: Elena Lepilkina <Elena.Lepilkina at synopsys.com>; llvm-dev at lists.llvm.org Subject: RE: [llvm-dev] RFC: FileCheck Enhancements> From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of > Elena Lepilkina via llvm-dev > Sent: Tuesday, May 24, 2016 6:51 AM > To: llvm-dev > Subject: [llvm-dev] RFC: FileCheck Enhancements > > Hi everyone, > > There was idea to add new directives to FileCheck: > 1. Directive to use some patterns as named template with or without parameters.Seems plausible, although I find the proposed syntax for a template call with parameters to be very awkward.> 2. CHECK-INCLUDE - Directive to include other file with checks to another. > 3. Expressions repeat for CHECK - If statement should be checked several times repeat modifiers {n}, {n,m} , {,n}, {n,}, *, + can be used. > 4. Repeat in regexs - Repeat with current number should become > available by using {n}, {n,m} , {,n}, {n,} 5. CHECK-LABEL-DAG - Not sequential order of labels.After reading through the google doc I see what you are trying to do. The motivation for CHECK-DAG is to reduce future test "churn" in case someone makes a change that influences the output order of something, but the order is not important to the test. The motivation is NOT to support instability in the output order across runs of the same compiler. Just so we're clear on that. In principle I could see how the order of blocks or entire functions could change over time, with no real relevance to a test, but how necessary is it really? I have seen work go into Clang/LLVM over the years to ensure stable output order, and my impression has been that these changes typically have little complexity or performance effect while significantly simplifying test effort.> 6. Check statement for words only - // CHECK-WORD, // CHECK-WORD-NEXT, // CHECK-WORD-SAME, // CHECK-WORD-DAG, // CHECK-WORD-NOT.I would expect a regex package to provide a word-break meta symbol, in which case this feature would be redundant. I admit I have not checked whether the package we use has this feature, or how it would be spelled.> 7. Wildcard for prefixes - If some statements should be checked regardless prefix, it should be used //{{*}}, //{{*}}-NEXT, //{{*}}-SAME and etc. > 8. Prefix with regular expressions - If statement should be checked if prefix matches some regular expression, it should be used {{regex}}:, {{regex}}-NEXT and etc. > > More information in file https://docs.google.com/document/d/1wAKNzU7-S2EeK1-aADwgP8dEiKfByKNazonybCQW3zs/edit?usp=sharing.I noticed the google doc stated that multi-line patterns are not supported. That's not actually the case, although it's a bit obscure: the [[:space:]] character-class will match EOL and allow you to write a multi-line CHECK.> > Now we have prototype with these features. It's tested on LLVM 3.8. > There was found unsupported before directive in old test. Bug about this - https://llvm.org/bugs/show_bug.cgi?id=27852. > > There is about 6% slowdown with new features when we tested them on 3.8.Is that a 6% slowdown just in FileCheck, or when running the entire Clang/LLVM test suite? Making the entire test run 6% more expensive seems like a lot. --paulr> > I see that there are some changes in FileCheck LLVM 3.9 with new features too. We can publish patch for 3.8 and it can be adapted for LLVM 3.9. Is it interesting for anyone? And how will be better to publish patch as for 3.8 or for 3.9? > > Thanks, > Elena._______________________________________________ LLVM Developers mailing list llvm-dev at lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Hi all, Voting may be a bad idea. I' ll be glad to hear more opinions here and may be some suggestions how to improve new features (may be there are ideas how template descriptions can be done simplier). After I try to accept your ideas and opinions to make new FileCheck features better. There was an idea to change regex library for support assertion \b. Are there any real plans to change regex library? I will make changes and publish separate patches a month later, because I'll take a holiday in June. Thanks, Elena. -----Original Message----- From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of Elena Lepilkina via llvm-dev Sent: Friday, May 27, 2016 12:09 PM To: Robinson, Paul <paul.robinson at sony.com>; llvm-dev at lists.llvm.org Subject: Re: [llvm-dev] RFC: FileCheck Enhancements Hi Paul, Thank you for information about the [[:space:]] character-class. About performance I tested on Clang/LLVM test suite. I try to profile and the problem is that I used regular expressions a lot for supporting some new features and functions in your regex library are very slow . Regex library is very old and quite awkward, in my opinion. May be you will see some ways to improve performance, if some features are decided to include in LLVM FileCheck and I publish patches for each feature separately(I saw your comment in published patch). I see that there are a lot of opinions, may be it will be better to vote. I suggest vote in file https://docs.google.com/spreadsheets/d/1p8Hi_PH3Nd2kEtYveCwKXENmJGOzYWW6us0XK7eVvOw/edit?usp=sharing. There will be history and it's possible to check that nobody votes twice. Then somebody can stop voting at one moment and I will be able to understand what patches I should do and publish if there are some changes you would like. Thanks, Elena. -----Original Message----- From: Robinson, Paul [mailto:paul.robinson at sony.com] Sent: Friday, May 27, 2016 2:26 AM To: Elena Lepilkina <Elena.Lepilkina at synopsys.com>; llvm-dev at lists.llvm.org Subject: RE: [llvm-dev] RFC: FileCheck Enhancements> From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of > Elena Lepilkina via llvm-dev > Sent: Tuesday, May 24, 2016 6:51 AM > To: llvm-dev > Subject: [llvm-dev] RFC: FileCheck Enhancements > > Hi everyone, > > There was idea to add new directives to FileCheck: > 1. Directive to use some patterns as named template with or without parameters.Seems plausible, although I find the proposed syntax for a template call with parameters to be very awkward.> 2. CHECK-INCLUDE - Directive to include other file with checks to another. > 3. Expressions repeat for CHECK - If statement should be checked several times repeat modifiers {n}, {n,m} , {,n}, {n,}, *, + can be used. > 4. Repeat in regexs - Repeat with current number should become > available by using {n}, {n,m} , {,n}, {n,} 5. CHECK-LABEL-DAG - Not sequential order of labels.After reading through the google doc I see what you are trying to do. The motivation for CHECK-DAG is to reduce future test "churn" in case someone makes a change that influences the output order of something, but the order is not important to the test. The motivation is NOT to support instability in the output order across runs of the same compiler. Just so we're clear on that. In principle I could see how the order of blocks or entire functions could change over time, with no real relevance to a test, but how necessary is it really? I have seen work go into Clang/LLVM over the years to ensure stable output order, and my impression has been that these changes typically have little complexity or performance effect while significantly simplifying test effort.> 6. Check statement for words only - // CHECK-WORD, // CHECK-WORD-NEXT, // CHECK-WORD-SAME, // CHECK-WORD-DAG, // CHECK-WORD-NOT.I would expect a regex package to provide a word-break meta symbol, in which case this feature would be redundant. I admit I have not checked whether the package we use has this feature, or how it would be spelled.> 7. Wildcard for prefixes - If some statements should be checked regardless prefix, it should be used //{{*}}, //{{*}}-NEXT, //{{*}}-SAME and etc. > 8. Prefix with regular expressions - If statement should be checked if prefix matches some regular expression, it should be used {{regex}}:, {{regex}}-NEXT and etc. > > More information in file https://docs.google.com/document/d/1wAKNzU7-S2EeK1-aADwgP8dEiKfByKNazonybCQW3zs/edit?usp=sharing.I noticed the google doc stated that multi-line patterns are not supported. That's not actually the case, although it's a bit obscure: the [[:space:]] character-class will match EOL and allow you to write a multi-line CHECK.> > Now we have prototype with these features. It's tested on LLVM 3.8. > There was found unsupported before directive in old test. Bug about this - https://llvm.org/bugs/show_bug.cgi?id=27852. > > There is about 6% slowdown with new features when we tested them on 3.8.Is that a 6% slowdown just in FileCheck, or when running the entire Clang/LLVM test suite? Making the entire test run 6% more expensive seems like a lot. --paulr> > I see that there are some changes in FileCheck LLVM 3.9 with new features too. We can publish patch for 3.8 and it can be adapted for LLVM 3.9. Is it interesting for anyone? And how will be better to publish patch as for 3.8 or for 3.9? > > Thanks, > Elena._______________________________________________ LLVM Developers mailing list llvm-dev at lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Hi all, Now all discussed enhancements are divided into separate patches. Moreover I have found mistake which reduces performance. Now it's fixed. List of patches (for those interested): Repeats in regular expressions - https://reviews.llvm.org/D22342 Including files - https://reviews.llvm.org/D22344 Expressions repeat for CHECK and CHECK-NEXT - https://reviews.llvm.org/D22345 CHECK-LABEL-DAG - https://reviews.llvm.org/D22348 CHECK-WORD - https://reviews.llvm.org/D22353 prefixes-regular expressions - https://reviews.llvm.org/D22401 pattern templates - https://reviews.llvm.org/D22403. Thanks, Elena.