On Tue, Mar 22, 2016 at 8:40 PM, Hal Finkel <hfinkel at anl.gov> wrote:> > ------------------------------ > > *From: *"Rui Ueyama" <ruiu at google.com> > *To: *"Hal Finkel" <hfinkel at anl.gov> > *Cc: *"David Blaikie" <dblaikie at gmail.com>, "llvm-dev" < > llvm-dev at lists.llvm.org>, "Bruce Hoult" <bruce at hoult.org> > *Sent: *Tuesday, March 22, 2016 2:36:34 PM > *Subject: *Re: [llvm-dev] Need help with code generation > > On Tue, Mar 22, 2016 at 7:36 PM, Rui Ueyama <ruiu at google.com> wrote: > >> I have a question. If there is a ELF verifier function that walks every >> part of an ELF file to verify that the file is sane, and if you can call >> that before calling LLD's function, are you guys happy with that? >> > > I'd like to get you guys opinion on this question. > > > I'll echo Rafael here. What does "sane" mean? If I define sane to mean, > "will not cause lld to exhibit undefined behavior if later run over the > same input", then this seems like the most efficient way of satisfying that > goal (perhaps staged to avoid cache thrashing), and I'd like to know how > much overhead it would add to run it in lld by default (even if we have an > option to disable it for absolute speed). >ELF is a documented file format so if you are not sure if something should be considered valid, you can take a look at the spec to determine whether it is valid or not. File validity is an independent concept from LLD and I think we can determine it according to the spec. I have a different opinion about how it could be implemented. We have no idea if it'd be faster if it is "embedded" to LLD, at least. And usually separating passes leads to cleaner and more readable code. Thanks again,> Hal > > > > >> On Tue, Mar 22, 2016 at 6:39 PM, Hal Finkel via llvm-dev < >> llvm-dev at lists.llvm.org> wrote: >> >>> >>> ------------------------------ >>> >>> *From: *"David Blaikie via llvm-dev" <llvm-dev at lists.llvm.org> >>> *To: *"Rafael Espíndola" <rafael.espindola at gmail.com> >>> *Cc: *"llvm-dev" <llvm-dev at lists.llvm.org>, "Bruce Hoult" < >>> bruce at hoult.org> >>> *Sent: *Tuesday, March 22, 2016 10:18:03 AM >>> *Subject: *Re: [llvm-dev] Need help with code generation >>> >>> >>> >>> On Tue, Mar 22, 2016 at 4:27 AM, Rafael Espíndola < >>> llvm-dev at lists.llvm.org> wrote: >>> >>>> > Maybe not, but it's not impossible either - browsers manage to harden >>>> themselves against malicious input and they operate in a far hostile >>>> environment with many more input formats than we do. >>>> >>>> It is important to note how different they are. Both Firefox and >>>> Chromium have people working just to try to make them more secure. >>>> Compare that with LLVM: One week ago I pointed out that your patch >>>> (r263521) introduces a crash. It still hasn't been reverted or even >>>> acknowledge yet. >>>> >>>> >>>> > I'm not trying to shift your personal goal, or to direct the features >>>> that you choose to put your time into, but I am interested in project >>>> policy. >>>> >>>> Why do you care about policy that is not followed? A policy saying >>>> llvm should not crash on any input is as relevant as one that says >>>> that clang should keep bootstrapping in under one second. >>>> >>> >>> It's pretty different when you say, essentially, that patches to address >>> these things are unlikely to be accepted. It doesn't seem surprising that >>> people wouldn't try to provide those patches and would choose not to use >>> the project if that's the expressed policy of the developers on the project >>> and doesn't line up with the needs of other people. >>> >>> >>> +1 >>> >>> -Hal >>> >>> >>> >>>> >>>> So, if we stick to reality, what we have is that lld (ELF and COFF) >>>> are already the most reliable parts of the toolchain. If not for Rui >>>> and I being upfront about it most people would not even know that you >>>> could crash it. So please, just let us keep working on the most >>>> reliable part of the toolchain. >>>> >>>> Cheers, >>>> Rafael >>>> _______________________________________________ >>>> LLVM Developers mailing list >>>> llvm-dev at lists.llvm.org >>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >>>> >>> >>> >>> _______________________________________________ >>> LLVM Developers mailing list >>> llvm-dev at lists.llvm.org >>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >>> >>> >>> >>> >>> -- >>> Hal Finkel >>> Assistant Computational Scientist >>> Leadership Computing Facility >>> Argonne National Laboratory >>> >>> _______________________________________________ >>> LLVM Developers mailing list >>> llvm-dev at lists.llvm.org >>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >>> >>> >> > > > > -- > Hal Finkel > Assistant Computational Scientist > Leadership Computing Facility > Argonne National Laboratory >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160322/83be663f/attachment.html>
----- Original Message -----> From: "Rui Ueyama" <ruiu at google.com> > To: "Hal Finkel" <hfinkel at anl.gov> > Cc: "David Blaikie" <dblaikie at gmail.com>, "llvm-dev" > <llvm-dev at lists.llvm.org>, "Bruce Hoult" <bruce at hoult.org> > Sent: Tuesday, March 22, 2016 2:57:45 PM > Subject: Re: [llvm-dev] Need help with code generation> On Tue, Mar 22, 2016 at 8:40 PM, Hal Finkel < hfinkel at anl.gov > > wrote:> > > From: "Rui Ueyama" < ruiu at google.com > > > > > > > To: "Hal Finkel" < hfinkel at anl.gov > > > > > > > Cc: "David Blaikie" < dblaikie at gmail.com >, "llvm-dev" < > > > llvm-dev at lists.llvm.org >, "Bruce Hoult" < bruce at hoult.org > > > > > > > Sent: Tuesday, March 22, 2016 2:36:34 PM > > > > > > Subject: Re: [llvm-dev] Need help with code generation > > >> > > On Tue, Mar 22, 2016 at 7:36 PM, Rui Ueyama < ruiu at google.com > > > > wrote: > > >> > > > I have a question. If there is a ELF verifier function that > > > > walks > > > > every part of an ELF file to verify that the file is sane, and > > > > if > > > > you can call that before calling LLD's function, are you guys > > > > happy > > > > with that? > > > > > > > > > I'd like to get you guys opinion on this question. > > > > > I'll echo Rafael here. What does "sane" mean? If I define sane to > > mean, "will not cause lld to exhibit undefined behavior if later > > run > > over the same input", then this seems like the most efficient way > > of > > satisfying that goal (perhaps staged to avoid cache thrashing), and > > I'd like to know how much overhead it would add to run it in lld by > > default (even if we have an option to disable it for absolute > > speed). >> ELF is a documented file format so if you are not sure if something > should be considered valid, you can take a look at the spec to > determine whether it is valid or not. File validity is an > independent concept from LLD and I think we can determine it > according to the spec.I understand your point. However, how does this compare in complexity to using the input to link against something trivial except that it requires all of its symbols?> I have a different opinion about how it could be implemented. We have > no idea if it'd be faster if it is "embedded" to LLD, at least. And > usually separating passes leads to cleaner and more readable code.\Yes, but how much code would it share with lld itself? And running it as a separate pass could be considerably more expensive on large inputs. Thanks again, Hal> > Thanks again, > > > Hal >> > > > On Tue, Mar 22, 2016 at 6:39 PM, Hal Finkel via llvm-dev < > > > > llvm-dev at lists.llvm.org > wrote: > > > > > >> > > > > > From: "David Blaikie via llvm-dev" < > > > > > > llvm-dev at lists.llvm.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > To: "Rafael Espíndola" < rafael.espindola at gmail.com > > > > > > > > > > > > > > > > > > > > > > Cc: "llvm-dev" < llvm-dev at lists.llvm.org >, "Bruce Hoult" < > > > > > > bruce at hoult.org > > > > > > > > > > > > > > > > > > > > > > Sent: Tuesday, March 22, 2016 10:18:03 AM > > > > > > > > > > > > > > > > > > > > > Subject: Re: [llvm-dev] Need help with code generation > > > > > > > > > > > > > > >> > > > > > On Tue, Mar 22, 2016 at 4:27 AM, Rafael Espíndola < > > > > > > llvm-dev at lists.llvm.org > wrote: > > > > > > > > > > > > > > >> > > > > > > > Maybe not, but it's not impossible either - browsers > > > > > > > > manage > > > > > > > > to > > > > > > > > harden themselves against malicious input and they > > > > > > > > operate > > > > > > > > in > > > > > > > > a > > > > > > > > far hostile environment with many more input formats > > > > > > > > than > > > > > > > > we > > > > > > > > do. > > > > > > > > > > > > > > > > > > > > >> > > > > > > It is important to note how different they are. Both > > > > > > > Firefox > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > > > > > > Chromium have people working just to try to make them > > > > > > > more > > > > > > > secure. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Compare that with LLVM: One week ago I pointed out that > > > > > > > your > > > > > > > patch > > > > > > > > > > > > > > > > > > > > > > > > > > > > (r263521) introduces a crash. It still hasn't been > > > > > > > reverted > > > > > > > or > > > > > > > even > > > > > > > > > > > > > > > > > > > > > > > > > > > > acknowledge yet. > > > > > > > > > > > > > > > > > > > > >> > > > > > > > I'm not trying to shift your personal goal, or to > > > > > > > > direct > > > > > > > > the > > > > > > > > features that you choose to put your time into, but I > > > > > > > > am > > > > > > > > interested in project policy. > > > > > > > > > > > > > > > > > > > > >> > > > > > > Why do you care about policy that is not followed? A > > > > > > > policy > > > > > > > saying > > > > > > > > > > > > > > > > > > > > > > > > > > > > llvm should not crash on any input is as relevant as one > > > > > > > that > > > > > > > says > > > > > > > > > > > > > > > > > > > > > > > > > > > > that clang should keep bootstrapping in under one second. > > > > > > > > > > > > > > > > > > > > >> > > > > > It's pretty different when you say, essentially, that > > > > > > patches > > > > > > to > > > > > > address these things are unlikely to be accepted. It > > > > > > doesn't > > > > > > seem > > > > > > surprising that people wouldn't try to provide those > > > > > > patches > > > > > > and > > > > > > would choose not to use the project if that's the expressed > > > > > > policy > > > > > > of the developers on the project and doesn't line up with > > > > > > the > > > > > > needs > > > > > > of other people. > > > > > > > > > > > > > > > > > > > > +1 > > > > > > > > > >> > > > > -Hal > > > > > > > > > >> > > > > > > So, if we stick to reality, what we have is that lld (ELF > > > > > > > and > > > > > > > COFF) > > > > > > > > > > > > > > > > > > > > > > > > > > > > are already the most reliable parts of the toolchain. If > > > > > > > not > > > > > > > for > > > > > > > Rui > > > > > > > > > > > > > > > > > > > > > > > > > > > > and I being upfront about it most people would not even > > > > > > > know > > > > > > > that > > > > > > > you > > > > > > > > > > > > > > > > > > > > > > > > > > > > could crash it. So please, just let us keep working on > > > > > > > the > > > > > > > most > > > > > > > > > > > > > > > > > > > > > > > > > > > > reliable part of the toolchain. > > > > > > > > > > > > > > > > > > > > >> > > > > > > Cheers, > > > > > > > > > > > > > > > > > > > > > > > > > > > > Rafael > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > > > > > > > > > > > > > > > > > > > > LLVM Developers mailing list > > > > > > > > > > > > > > > > > > > > > > > > > > > > llvm-dev at lists.llvm.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > > > > > > > > > > > > > > > > > > > >> > > > > > _______________________________________________ > > > > > > > > > > > > > > > > > > > > > LLVM Developers mailing list > > > > > > > > > > > > > > > > > > > > > llvm-dev at lists.llvm.org > > > > > > > > > > > > > > > > > > > > > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > > > > > > > > > > > > > >> > > > > -- > > > > > > > > > >> > > > > Hal Finkel > > > > > > > > > > > > > > > Assistant Computational Scientist > > > > > > > > > > > > > > > Leadership Computing Facility > > > > > > > > > > > > > > > Argonne National Laboratory > > > > > > > > > >> > > > > _______________________________________________ > > > > > > > > > > > > > > > LLVM Developers mailing list > > > > > > > > > > > > > > > llvm-dev at lists.llvm.org > > > > > > > > > > > > > > > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > > > > > > > > >> > -- >> > Hal Finkel > > > Assistant Computational Scientist > > > Leadership Computing Facility > > > Argonne National Laboratory >-- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160322/0bc86c03/attachment.html>
David Chisnall via llvm-dev
2016-Mar-23 11:00 UTC
[llvm-dev] Need help with code generation
On 22 Mar 2016, at 19:57, Rui Ueyama via llvm-dev <llvm-dev at lists.llvm.org> wrote:> > ELF is a documented file format so if you are not sure if something should be considered valid, you can take a look at the spec to determine whether it is valid or not. File validity is an independent concept from LLD and I think we can determine it according to the spec.Some folk in Cambridge (including Stephen Kell, who I think you met at dinner on Friday) are working on a formal model for ELF and of linker semantics. From what I’ve seen of their work, I’d echo Raphael’s comment: determining that a file is a valid ELF file is a long way from being trivial. I wouldn’t be surprised if it took longer to run than lld, and I’d rather see a linker that ran in 10 seconds on valid input and crashed on invalid than one that ran in 30 seconds on valid input and took 20 seconds to produce an error message telling me in what way my ELF file is invalid on invalid input. Hopefully, one of the outcomes of their research will be a tool that allows you to get this diagnostic if you’ve run a linker and had it crash. They’re also hoping to be able to validate that the output from a linker is a valid linking of the input, though that’s a bit further away as yet. David
Joerg Sonnenberger via llvm-dev
2016-Mar-23 13:33 UTC
[llvm-dev] Need help with code generation
On Wed, Mar 23, 2016 at 11:00:08AM +0000, David Chisnall via llvm-dev wrote:> On 22 Mar 2016, at 19:57, Rui Ueyama via llvm-dev <llvm-dev at lists.llvm.org> wrote: > > > > ELF is a documented file format so if you are not sure if something should be considered valid, you can take a look at the spec to determine whether it is valid or not. File validity is an independent concept from LLD and I think we can determine it according to the spec. > > Some folk in Cambridge (including Stephen Kell, who I think you met at > dinner on Friday) are working on a formal model for ELF and of linker > semantics. From what I’ve seen of their work, I’d echo Raphael’s > comment: determining that a file is a valid ELF file is a long way from > being trivial.But determining that an ELF file is valid is not required here. Code only has to make sure it is not obviously harmful. This means doing basic out-of-bounds checks the first time one of the data structures is processed, but it doesn't mean trying terribly hard to make sure that the content makes sense. That's a huge difference. Joerg