llvm dev - Nov 2013 - [LLVMdev] Proposal: release MDNodes for source modules (LTO+debug info)

On Nov 14, 2013, at 8:19 AM, Rafael Espíndola <rafael.espindola at
gmail.com> wrote:
> But gold has at most 2 objects loaded at any time.
Are you sure about that? I haven't looked into it but while building
Chromium with LTO, I get:

../../third_party/gold/gold64: fatal error: out of file descriptors and
couldn't close any
clang: error: linker command failed with exit code 1 (use -v to see invocation)
ninja: build stopped: subcommand failed.
[secdev:~/chromium/src/out/Release] (master) s$ ulimit -Sn
10000

Looking in /proc, it has 10013 open file descriptors. With only 2 objects loaded
at a time, I'd expect many fewer file descriptors to be open. Maybe it only
has 2 objects in memory at once but keeps all the file descriptors open?

-- 
Stephen Checkoway

On 14 November 2013 18:01, Stephen Checkoway <s at pahtak.org>
wrote:
>
> On Nov 14, 2013, at 8:19 AM, Rafael Espíndola <rafael.espindola at
gmail.com> wrote:
>
>> But gold has at most 2 objects loaded at any time.
>
> Are you sure about that? I haven't looked into it but while building
Chromium with LTO, I get:
>
> ../../third_party/gold/gold64: fatal error: out of file descriptors and
couldn't close any
> clang: error: linker command failed with exit code 1 (use -v to see
invocation)
> ninja: build stopped: subcommand failed.
> [secdev:~/chromium/src/out/Release] (master) s$ ulimit -Sn
> 10000
>
> Looking in /proc, it has 10013 open file descriptors. With only 2 objects
loaded at a time, I'd expect many fewer file descriptors to be open. Maybe
it only has 2 objects in memory at once but keeps all the file descriptors open?
That is odd. I will debug it in a sec, but we have in the claim_file_hook:

  if (code_gen) {
    if (lto_codegen_add_module(code_gen, M)) {
      (*message)(LDPL_ERROR, "Error linking module: %s",
                 lto_get_error_message());
      return LDPS_ERR;
    }
  }

  lto_module_dispose(M);

In fact, with current gold we call get_view, so the plugin uses the
same fd as gold. It might actually be a bug with gold trying to cache
too many open files.

How are you trying to build it?

Cheers,
Rafael

On Nov 15, 2013, at 10:19 AM, Rafael Espíndola <rafael.espindola at
gmail.com> wrote:
> On 14 November 2013 18:01, Stephen Checkoway <s at pahtak.org> wrote:
>> 
>> On Nov 14, 2013, at 8:19 AM, Rafael Espíndola <rafael.espindola at
gmail.com> wrote:
>> 
>>> But gold has at most 2 objects loaded at any time.
>> 
>> Are you sure about that? I haven't looked into it but while
building Chromium with LTO, I get:
>> 
>> ../../third_party/gold/gold64: fatal error: out of file descriptors and
couldn't close any
>> clang: error: linker command failed with exit code 1 (use -v to see
invocation)
>> ninja: build stopped: subcommand failed.
>> [secdev:~/chromium/src/out/Release] (master) s$ ulimit -Sn
>> 10000
>> 
>> Looking in /proc, it has 10013 open file descriptors. With only 2
objects loaded at a time, I'd expect many fewer file descriptors to be open.
Maybe it only has 2 objects in memory at once but keeps all the file descriptors
open?
> 
> That is odd. I will debug it in a sec, but we have in the claim_file_hook:
> 
>  if (code_gen) {
>    if (lto_codegen_add_module(code_gen, M)) {
>      (*message)(LDPL_ERROR, "Error linking module: %s",
>                 lto_get_error_message());
>      return LDPS_ERR;
>    }
>  }
> 
>  lto_module_dispose(M);
> 
> In fact, with current gold we call get_view, so the plugin uses the
> same fd as gold. It might actually be a bug with gold trying to cache
> too many open files.
> 
> How are you trying to build it?
The standard Chromium build system was modified to add -flto -Os to cflags
(which I'm assuming in this case also gets passed to clang++) and -flto to
ldflags in chromium/src/build/common.gypi and I think some of the build files
for libraries that are built along with Chromium were modified to not use -flto
because they didn't work. I'm not the one who did that work,
unfortunately, so I can't say for sure exactly what all was modified.

-- 
Stephen Checkoway