thr3ads.net - llvm dev - [llvm-dev] Unable to parse command line more than once using llvm libraries? [Nov 2019]

If this information is useful, please help other people find it:
Share via:

LLVM Mailing List via llvm-dev

2019-Nov-18 14:18 UTC

[llvm-dev] Unable to parse command line more than once using llvm libraries?

Thanks,


I tried calling ResetAllOptionOccurrences after the run like this…




  // Compile the module TimeCompilations times to give better compile time
  // metrics.
  for (unsigned I = TimeCompilations; I; --I)
    if (int RetVal = compileModule(argv, Context))
      return RetVal;

  if (YamlFile)
    YamlFile->keep();

  cl::ResetAllOptionOccurrences();

  return 0;
}





Unfortunately it is still crashing out when the next build starts. Any other
thoughts?

> On 18 Nov 2019, at 10:13, James Henderson via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> I'm not entirely sure what it is that is causing your problem. It's
worth noting that the unit tests for CommandLine.cpp can call
cl::ParseCommandLineOptions multiple times within a process, although it looks
like the options parsed are explicitly removed at the end of each test to avoid
polluting the environment of the next test.
> 
> Probably the right thing to do is to call the
"ResetAllOptionOccurrences" function of the command line parser. This
should allow you to use it again cleanly. If you don't call that, you might
end up setting options multiple times, which could lead to unexpected behaviour.
Note that the comment in this function says that it is explicitly intended to
allow parsing different command lines multiple times.
> 
> Hope that helps.
> 
> James
> 
> On Sun, 17 Nov 2019 at 15:17, Carl Peto via llvm-dev <llvm-dev at
lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
> Hi,
> 
> I am running a server that does regular builds using llc. To begin with I
was doing a fork/exec to launch llc, passing the files to compile and other
switches as command line, etc. then waiting for the subprocess to complete. This
was fine but seemed an unnecessary overhead. After all llc is mostly a fairly
thin front end over the llc libraries, which do the actual work right?
> 
> So I took llc.cpp, and the llvm-c and llvm headers and llvm libraries and
linked them all into my server program (which is a C based server). I modified
llc.cpp to change...
> 
> int llcmain(int argc, char **argv) {
> ...
> }
> 
> ...to...
> 
> extern "C" int llcmain(int argc, char **argv) {
> ...
> }
> 
> 
> Now I am calling this function directly from my server code.  This works
fine for the first build, however any other builds fail in
> 
> CommandLine.cpp:1101
> 
> CommandLineParser::ParseCommandLineOptions
> 
> assert(hasOptions() && "No options specified!");
<--fails here
> 
> 
> I'm trying to work out why this is breaking, I am assuming there is
some global state left after processing that llc.cpp doesn't normally need
to worry about?
> 
> Is there a way to reset this global state cleanly? I'm assuming
there's something about command line processing that is written to only work
once, not repeatedly?
> 
> Thanks for any advice/help.
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191118/d2901a7f/attachment.html>

LLVM Mailing List via llvm-dev

2019-Nov-19 03:39 UTC

head link

[llvm-dev] Unable to parse command line more than once using llvm libraries?

I think I got to the bottom of this and it’s nearly working now.

Your comments were very helpful and stopped me going down many useless paths.
Especially the point about the unit tests re-running command line parsing
multiple times. Great point!


My issue actually came from this line at the start:

extern "C" int llcmain(int argc, char **argv) {
  InitLLVM X(argc, argv);
…



Doing a bit of debugging, after reading CommandLine.rst from start to finish
(and then realising that at least my copy was out of date because it didn’t
mention SubCommands :))… I was able to step into the code just before the assert
was triggered and realised somehow there were no command line options in
SubCommand…

(lldb) e TopLevelSubCommand->OptionsMap
(llvm::StringMap<llvm::cl::Option *, llvm::MallocAllocator>) $10 = {
  llvm::StringMapImpl = {
    TheTable = 0x0000000000000000
    NumBuckets = 0
    NumItems = 0
    NumTombstones = 0
    ItemSize = 16
  }
  Allocator = {}
}

Since they should all be set up in static variables, and the server program
isn’t going away, this is quite odd. The GlobalParser is also dead…
(lldb) e GlobalParser
(llvm::ManagedStatic<(anonymous namespace)::CommandLineParser,
llvm::object_creator<(anonymous namespace)::CommandLineParser>,
llvm::object_deleter<(anonymous namespace)::CommandLineParser> >) $12 =
{
  llvm::ManagedStaticBase = {
    Ptr = 0x0000000000000000
    DeleterFn = 0x0000000000000000
    Next = 0x0000000000000000
  }
}


I realised these managed static global variables had been deleted. This was
being done in llvm_shutdown which was called from the destructor of the InitLLVM
object.


I made a simple fix/hack:

extern "C" int llcmain(int argc, char **argv) {
  static InitLLVM X(argc, argv);


Now the InitLLVM object is only created once, never destroyed and the globals
stay in play.



Now I have to clean up the command line after each run. I’m trying
ResetAllOptionOccurrences() at the end of llcmain, that removes all options but
still seems to leave the positional parameters. I’ll probably figure it out
eventually. :)


Thanks for all your help.



> On 18 Nov 2019, at 14:18, LLVM Mailing List via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> Thanks,
> 
> 
> I tried calling ResetAllOptionOccurrences after the run like this…
> 
> 
> 
> 
>   // Compile the module TimeCompilations times to give better compile time
>   // metrics.
>   for (unsigned I = TimeCompilations; I; --I)
>     if (int RetVal = compileModule(argv, Context))
>       return RetVal;
> 
>   if (YamlFile)
>     YamlFile->keep();
> 
>   cl::ResetAllOptionOccurrences();
> 
>   return 0;
> }
> 
> 
> 
> 
> 
> Unfortunately it is still crashing out when the next build starts. Any
other thoughts?
> 
> 
>> On 18 Nov 2019, at 10:13, James Henderson via llvm-dev <llvm-dev at
lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>> 
>> I'm not entirely sure what it is that is causing your problem.
It's worth noting that the unit tests for CommandLine.cpp can call
cl::ParseCommandLineOptions multiple times within a process, although it looks
like the options parsed are explicitly removed at the end of each test to avoid
polluting the environment of the next test.
>> 
>> Probably the right thing to do is to call the
"ResetAllOptionOccurrences" function of the command line parser. This
should allow you to use it again cleanly. If you don't call that, you might
end up setting options multiple times, which could lead to unexpected behaviour.
Note that the comment in this function says that it is explicitly intended to
allow parsing different command lines multiple times.
>> 
>> Hope that helps.
>> 
>> James
>> 
>> On Sun, 17 Nov 2019 at 15:17, Carl Peto via llvm-dev <llvm-dev at
lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>> Hi,
>> 
>> I am running a server that does regular builds using llc. To begin with
I was doing a fork/exec to launch llc, passing the files to compile and other
switches as command line, etc. then waiting for the subprocess to complete. This
was fine but seemed an unnecessary overhead. After all llc is mostly a fairly
thin front end over the llc libraries, which do the actual work right?
>> 
>> So I took llc.cpp, and the llvm-c and llvm headers and llvm libraries
and linked them all into my server program (which is a C based server). I
modified llc.cpp to change...
>> 
>> int llcmain(int argc, char **argv) {
>> ...
>> }
>> 
>> ...to...
>> 
>> extern "C" int llcmain(int argc, char **argv) {
>> ...
>> }
>> 
>> 
>> Now I am calling this function directly from my server code.  This
works fine for the first build, however any other builds fail in
>> 
>> CommandLine.cpp:1101
>> 
>> CommandLineParser::ParseCommandLineOptions
>> 
>> assert(hasOptions() && "No options specified!");
<--fails here
>> 
>> 
>> I'm trying to work out why this is breaking, I am assuming there is
some global state left after processing that llc.cpp doesn't normally need
to worry about?
>> 
>> Is there a way to reset this global state cleanly? I'm assuming
there's something about command line processing that is written to only work
once, not repeatedly?
>> 
>> Thanks for any advice/help.
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191119/aa7ee833/attachment.html>

LLVM Mailing List via llvm-dev

2019-Nov-19 13:03 UTC

head link

[llvm-dev] Unable to parse command line more than once using llvm libraries?

The final piece of the puzzle, for anyone who is curious about this thread…

At the end of the llcmain function I had to do this…

  cl::ResetAllOptionOccurrences();
  InputFilename.reset();

  return 0;

ResetAllOptionOccurrences resets all options (-O3… etc.) but the positional
parameters are not reset because that function iterates through the options of
the option map, which only covers options with an argument name. Arguably this
is a bug but I’m betting this function is only really used in unit tests to
control test state and not in “real life” code (the same seems to be true of
ResetCommandLineParser in my branch), so I think it’s either not an issue or not
an issue worth fixing.

There is only one positional parameter for llc... <input bitcode>… which
corresponds to the Option within the llc.cpp file called InputFilename, so I
just reset that option and the command line parser is back to a clean state
after llcmain completes, and ready to be reused safely.

All seems to work as required now. Thanks for your help James!



> On 19 Nov 2019, at 03:39, LLVM Mailing List <carl-llvm-dev at
petosoft.com> wrote:
> 
> I think I got to the bottom of this and it’s nearly working now.
> 
> Your comments were very helpful and stopped me going down many useless
paths. Especially the point about the unit tests re-running command line parsing
multiple times. Great point!
> 
> 
> My issue actually came from this line at the start:
> 
> extern "C" int llcmain(int argc, char **argv) {
>   InitLLVM X(argc, argv);
> …
> 
> 
> 
> Doing a bit of debugging, after reading CommandLine.rst from start to
finish (and then realising that at least my copy was out of date because it
didn’t mention SubCommands :))… I was able to step into the code just before the
assert was triggered and realised somehow there were no command line options in
SubCommand…
> 
> (lldb) e TopLevelSubCommand->OptionsMap
> (llvm::StringMap<llvm::cl::Option *, llvm::MallocAllocator>) $10 = {
>   llvm::StringMapImpl = {
>     TheTable = 0x0000000000000000
>     NumBuckets = 0
>     NumItems = 0
>     NumTombstones = 0
>     ItemSize = 16
>   }
>   Allocator = {}
> }
> 
> Since they should all be set up in static variables, and the server program
isn’t going away, this is quite odd. The GlobalParser is also dead…
> (lldb) e GlobalParser
> (llvm::ManagedStatic<(anonymous namespace)::CommandLineParser,
llvm::object_creator<(anonymous namespace)::CommandLineParser>,
llvm::object_deleter<(anonymous namespace)::CommandLineParser> >) $12 =
{
>   llvm::ManagedStaticBase = {
>     Ptr = 0x0000000000000000
>     DeleterFn = 0x0000000000000000
>     Next = 0x0000000000000000
>   }
> }
> 
> 
> I realised these managed static global variables had been deleted. This was
being done in llvm_shutdown which was called from the destructor of the InitLLVM
object.
> 
> 
> I made a simple fix/hack:
> 
> extern "C" int llcmain(int argc, char **argv) {
>   static InitLLVM X(argc, argv);
> 
> 
> Now the InitLLVM object is only created once, never destroyed and the
globals stay in play.
> 
> 
> 
> Now I have to clean up the command line after each run. I’m trying
ResetAllOptionOccurrences() at the end of llcmain, that removes all options but
still seems to leave the positional parameters. I’ll probably figure it out
eventually. :)
> 
> 
> Thanks for all your help.
> 
> 
> 
> 
>> On 18 Nov 2019, at 14:18, LLVM Mailing List via llvm-dev <llvm-dev
at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>> 
>> Thanks,
>> 
>> 
>> I tried calling ResetAllOptionOccurrences after the run like this…
>> 
>> 
>> 
>> 
>>   // Compile the module TimeCompilations times to give better compile
time
>>   // metrics.
>>   for (unsigned I = TimeCompilations; I; --I)
>>     if (int RetVal = compileModule(argv, Context))
>>       return RetVal;
>> 
>>   if (YamlFile)
>>     YamlFile->keep();
>> 
>>   cl::ResetAllOptionOccurrences();
>> 
>>   return 0;
>> }
>> 
>> 
>> 
>> 
>> 
>> Unfortunately it is still crashing out when the next build starts. Any
other thoughts?
>> 
>> 
>>> On 18 Nov 2019, at 10:13, James Henderson via llvm-dev <llvm-dev
at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>>> 
>>> I'm not entirely sure what it is that is causing your problem.
It's worth noting that the unit tests for CommandLine.cpp can call
cl::ParseCommandLineOptions multiple times within a process, although it looks
like the options parsed are explicitly removed at the end of each test to avoid
polluting the environment of the next test.
>>> 
>>> Probably the right thing to do is to call the
"ResetAllOptionOccurrences" function of the command line parser. This
should allow you to use it again cleanly. If you don't call that, you might
end up setting options multiple times, which could lead to unexpected behaviour.
Note that the comment in this function says that it is explicitly intended to
allow parsing different command lines multiple times.
>>> 
>>> Hope that helps.
>>> 
>>> James
>>> 
>>> On Sun, 17 Nov 2019 at 15:17, Carl Peto via llvm-dev <llvm-dev
at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>>> Hi,
>>> 
>>> I am running a server that does regular builds using llc. To begin
with I was doing a fork/exec to launch llc, passing the files to compile and
other switches as command line, etc. then waiting for the subprocess to
complete. This was fine but seemed an unnecessary overhead. After all llc is
mostly a fairly thin front end over the llc libraries, which do the actual work
right?
>>> 
>>> So I took llc.cpp, and the llvm-c and llvm headers and llvm
libraries and linked them all into my server program (which is a C based
server). I modified llc.cpp to change...
>>> 
>>> int llcmain(int argc, char **argv) {
>>> ...
>>> }
>>> 
>>> ...to...
>>> 
>>> extern "C" int llcmain(int argc, char **argv) {
>>> ...
>>> }
>>> 
>>> 
>>> Now I am calling this function directly from my server code.  This
works fine for the first build, however any other builds fail in
>>> 
>>> CommandLine.cpp:1101
>>> 
>>> CommandLineParser::ParseCommandLineOptions
>>> 
>>> assert(hasOptions() && "No options specified!");
<--fails here
>>> 
>>> 
>>> I'm trying to work out why this is breaking, I am assuming
there is some global state left after processing that llc.cpp doesn't
normally need to worry about?
>>> 
>>> Is there a way to reset this global state cleanly? I'm assuming
there's something about command line processing that is written to only work
once, not repeatedly?
>>> 
>>> Thanks for any advice/help.
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>
>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>
>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
>> 
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191119/56dfc0a2/attachment.html>

llvm dev - Nov 2019 - Unable to parse command line more than once using llvm libraries?

[llvm-dev] Unable to parse command line more than once using llvm libraries?

[llvm-dev] Unable to parse command line more than once using llvm libraries?

[llvm-dev] Unable to parse command line more than once using llvm libraries?