Hi Ed, pt., 17 kwi 2020 o 15:52 Ed Maste <emaste at freebsd.org> napisa?(a):> > On Fri, 17 Apr 2020 at 08:58, Marcin Wojtas <mw at semihalf.com> wrote: > > > > Hi, > > > > Together with our customers, Semihalf is interested in improving the status > > of security mitigations enablement in FreeBSD. > > Happy to hear that there's interest in this work! > > > 1. Are there any hard blockers, like missing features or bugs, that prevent > > enabling ASLR by default in the kernel and building the base system with > > -DWITH_PIE? > > I believe there are no showstopper issues but there are a some > prerequisites. One is that there are some applications that may > misbehave with randomization enabled. They would need to be > identified, and tagged (with the elfctl tool now in the base system).I was thinking if it is possible to come up with such wide test coverage to test every single application from the base system. Do you think it is achievable or should we rather follow the approach to do as many tests as possible, but rely on the community feedback to catch the corner cases (like the ntpd issue mentioned in this thread)? What about the ports?> > > 2. In case the enablement becomes eventually approved, will it be better to > > do it for all archs or focus only on the selected ones? > > There's a general and increasing preference of avoiding different > defaults per architecture. One issue though is that some options may > have much larger performance impacts on certain architectures - e.g. > position independent executables (PIE) on i386.Understood. If there is likely a performance trade-off, how about doing tests e.g. on i386 and armv7? In case of a big drop or other issues, could limiting ALSR/PIE to 64-bit-only be a considered solution?> > > 3. IMO it may be worth to benchmark/stress the system for the stability > > verification and perf comparison purpose. Do you think it may be reasonable > > to create a kind of reference matrix (archs vs tests)? Those could be done > > to evaluate the current state of the OS, but also for validating each > > proposed feature. I also think engaging the FreeBSD CI might be a huge help > > in such an effort. BTW, any particular tests / benchmarks come to your mind > > as useful in this case? > > Yes, benchmarking and testing are very important tasks on a path to > enabling these by default. I agree with the CI comment; we should > start with CI build + kyua runs with options turned on, in advance of > changing the default.Indeed I thought of kyua and measuring buildworld execution time for stressing the DUT and having the first comparison numbers for the low price. Do you think it is possible to get help here, i.e. is there a FreeBSD devops team, maintaining the Jenkins CI whose spare cycles could be used for this purpose? Or is this a field requiring external help from interested parties? Even before the automation, would it be useful to perform some runs on chosen archs?> > I would be interested in seeing macro-level benchmarking with > mitigations on/off - for example, I assume Firefox must have a > performance test suite that they use for tracking their own > performance changes during development, and we could use benchmarks > like that to see the impact of mitigations. Coming up with a full set > of appropriate benchmarks will be a useful endeavour.Yes, making use of something actively maintained would be great. Do you see a need for IO stressing/benchmarking for the discussed cases? Best regards, Marcin
Hi Ed, Any thoughts on the questions below? Best regards, Marcin pon., 20 kwi 2020 o 16:21 Marcin Wojtas <mw at semihalf.com> napisa?(a):> > Hi Ed, > > pt., 17 kwi 2020 o 15:52 Ed Maste <emaste at freebsd.org> napisa?(a): > > > > On Fri, 17 Apr 2020 at 08:58, Marcin Wojtas <mw at semihalf.com> wrote: > > > > > > Hi, > > > > > > Together with our customers, Semihalf is interested in improving the status > > > of security mitigations enablement in FreeBSD. > > > > Happy to hear that there's interest in this work! > > > > > 1. Are there any hard blockers, like missing features or bugs, that prevent > > > enabling ASLR by default in the kernel and building the base system with > > > -DWITH_PIE? > > > > I believe there are no showstopper issues but there are a some > > prerequisites. One is that there are some applications that may > > misbehave with randomization enabled. They would need to be > > identified, and tagged (with the elfctl tool now in the base system). > > I was thinking if it is possible to come up with such wide test > coverage to test every single application from the base system. Do you > think it is achievable or should we rather follow the approach to do > as many tests as possible, but rely on the community feedback to catch > the corner cases (like the ntpd issue mentioned in this thread)? > What about the ports? > > > > > > 2. In case the enablement becomes eventually approved, will it be better to > > > do it for all archs or focus only on the selected ones? > > > > There's a general and increasing preference of avoiding different > > defaults per architecture. One issue though is that some options may > > have much larger performance impacts on certain architectures - e.g. > > position independent executables (PIE) on i386. > > Understood. If there is likely a performance trade-off, how about > doing tests e.g. on i386 and armv7? In case of a big drop or other > issues, could limiting ALSR/PIE to 64-bit-only be a considered > solution? > > > > > > 3. IMO it may be worth to benchmark/stress the system for the stability > > > verification and perf comparison purpose. Do you think it may be reasonable > > > to create a kind of reference matrix (archs vs tests)? Those could be done > > > to evaluate the current state of the OS, but also for validating each > > > proposed feature. I also think engaging the FreeBSD CI might be a huge help > > > in such an effort. BTW, any particular tests / benchmarks come to your mind > > > as useful in this case? > > > > Yes, benchmarking and testing are very important tasks on a path to > > enabling these by default. I agree with the CI comment; we should > > start with CI build + kyua runs with options turned on, in advance of > > changing the default. > > Indeed I thought of kyua and measuring buildworld execution time for > stressing the DUT and having the first comparison numbers for the low > price. > > Do you think it is possible to get help here, i.e. is there a FreeBSD > devops team, maintaining the Jenkins CI whose spare cycles could be > used for this purpose? Or is this a field requiring external help from > interested parties? > > Even before the automation, would it be useful to perform some runs on > chosen archs? > > > > > I would be interested in seeing macro-level benchmarking with > > mitigations on/off - for example, I assume Firefox must have a > > performance test suite that they use for tracking their own > > performance changes during development, and we could use benchmarks > > like that to see the impact of mitigations. Coming up with a full set > > of appropriate benchmarks will be a useful endeavour. > > Yes, making use of something actively maintained would be great. Do > you see a need for IO stressing/benchmarking for the discussed cases? > > Best regards, > Marcin
On Mon, Apr 20, 2020 at 04:21:59PM +0200, Marcin Wojtas wrote:> Hi Ed, > > pt., 17 kwi 2020 o 15:52 Ed Maste <emaste at freebsd.org> napisa??(a): > > > > On Fri, 17 Apr 2020 at 08:58, Marcin Wojtas <mw at semihalf.com> wrote: > > > > > > Hi, > > > > > > Together with our customers, Semihalf is interested in improving the status > > > of security mitigations enablement in FreeBSD. > > > > Happy to hear that there's interest in this work! > > > > > 1. Are there any hard blockers, like missing features or bugs, that prevent > > > enabling ASLR by default in the kernel and building the base system with > > > -DWITH_PIE? > > > > I believe there are no showstopper issues but there are a some > > prerequisites. One is that there are some applications that may > > misbehave with randomization enabled. They would need to be > > identified, and tagged (with the elfctl tool now in the base system). > > I was thinking if it is possible to come up with such wide test > coverage to test every single application from the base system. Do you > think it is achievable or should we rather follow the approach to do > as many tests as possible, but rely on the community feedback to catch > the corner cases (like the ntpd issue mentioned in this thread)? > What about the ports?If we gate on full testing we'll never move forward. We had a GSoC project a few years ago to try to generate lame tests for each program, if someone picked that up, we could get better coverage fairly quickly, but it would still be far from complete. Our best bet is probably to make it easy for people to test and to try and recruit testers in the community (this is especially true for ports). -- Brooks -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 455 bytes Desc: not available URL: <http://lists.freebsd.org/pipermail/freebsd-security/attachments/20200423/bf508360/attachment.sig>
On Thu, 23 Apr 2020 at 11:38, Brooks Davis <brooks at freebsd.org> wrote:> > > I was thinking if it is possible to come up with such wide test > > coverage to test every single application from the base system. Do you > > think it is achievable or should we rather follow the approach to do > > as many tests as possible, but rely on the community feedback to catch > > the corner cases (like the ntpd issue mentioned in this thread)? > > What about the ports? > > If we gate on full testing we'll never move forward. We had a GSoC > project a few years ago to try to generate lame tests for each program, > if someone picked that up, we could get better coverage fairly > quickly, but it would still be far from complete.Indeed, having a basic smoke test for as much of the base system as possible is a good initial step. I suspect it won't take very long to have confidence in turning on options for the base system, but ports will be a much longer process. For ports I think the first thing that needs to happen is to have some infrastructure in ports itself to allow individual ports to indicate (via elfctl) that they are not compatible with certain options; with that in place it should be trivial to start marking individual ports.
On Mon, 20 Apr 2020 at 10:22, Marcin Wojtas <mw at semihalf.com> wrote:> > Indeed I thought of kyua and measuring buildworld execution time for > stressing the DUT and having the first comparison numbers for the low > price. > > Do you think it is possible to get help here, i.e. is there a FreeBSD > devops team, maintaining the Jenkins CI whose spare cycles could be > used for this purpose? Or is this a field requiring external help from > interested parties?There aren't a lot of spare cycles to go around, but putting automation in place so that tests like this can easily be performed is certainly something that's in the Jenkins team's domain.> Yes, making use of something actively maintained would be great. Do > you see a need for IO stressing/benchmarking for the discussed cases?In the fullness of time I think it's important, but my opinion is that it's really functional tests that we need, for enabling features in -CURRENT; we can work on benchmarking before and after changing a default.