Daniel Ebdrup Jensen
2020-Oct-21 22:07 UTC
FreeBSD Quarterly Status Report - Third Quarter 2020
FreeBSD Project Quarterly Status Report - Third Quarter 2020
Introduction
This report covers FreeBSD related projects for the period between July
and September, and is the third of four planned reports for 2020.
This quarter brings a good mix of additions and changes to the FreeBSD
Project and community, from a diverse number of teams and people
covering everything from architectures, continuous integration,
wireless networking and drivers, over drm, desktop and third-party
project work, as well as several team reports, along with many other
interesting subjects too numerous to mention.
As the world is still affected by the epidemic, we hope that this
report can also serve as a good reminder that there is good work that
can be done by people working together, even if we're apart.
We hope you'll be as interested in reading it, as we've been in
making
it. Daniel Ebdrup Jensen, on behalf of the quarterly team.
__________________________________________________________________
FreeBSD Team Reports
* FreeBSD Foundation
* FreeBSD Release Engineering Team
* Cluster Administration Team
* Continuous Integration
* Ports Collection
* FreeBSD Office team - 3rd quarter 2020 report
* FreeBSD Graphics Team status report
Projects
* FreeBSD on Microsoft HyperV and Azure
* Building FreeBSD on non-FreeBSD hosts
* Git Migration Working Group
* Linux compatibility layer update
* LLDB Debugger Improvements
* Lua usage in FreeBSD
* NFS over TLS implementation
* syzkaller on FreeBSD
Kernel
* DRM Drivers Update
* DTS Update
* DesignWare Ethernet adapter driver improvements
* Google Summer of Code'20 Project - eBPF XDP Hooks
* ENA FreeBSD Driver Update
* IPSec Extended Sequence Number (ESN) support
* NXP ARM64 SoC support
* Addition of PowerPC64LE Architecture
* ure - USB 3.0 Gigabit Ethernet Driver update
* Stateless hardware offloads for VXLANs
* Wireless updates
* ZSTD Compression in ZFS
Architectures
* CheriBSD 2020 Q3
* FreeBSD/RISC-V Project
Ports
* Update to grub-bhyve
* KDE on FreeBSD
Documentation
* DOCNG on FreeBSD
Third-Party Projects
* Potluck - Flavour & Image Repository for pot
__________________________________________________________________
FreeBSD Team Reports
Entries from the various official and semi-official teams, as found in
the Administration Page.
FreeBSD Foundation
Contact: Deb Goodkin <deb at FreeBSDFoundation.org>
The FreeBSD Foundation is a 501(c)(3) non-profit organization dedicated
to supporting and promoting the FreeBSD Project and community
worldwide. Funding comes from individual and corporate donations and is
used to fund and manage software development projects, conferences and
developer summits, and provide travel grants to FreeBSD contributors.
The Foundation purchases and supports hardware to improve and maintain
FreeBSD infrastructure and provides resources to improve security,
quality assurance, and release engineering efforts; publishes marketing
material to promote, educate, and advocate for the FreeBSD Project;
facilitates collaboration between commercial vendors and FreeBSD
developers; and finally, represents the FreeBSD Project in executing
contracts, license agreements, and other legal arrangements that
require a recognized legal entity.
Here are some highlights of what we did to help FreeBSD last quarter:
COVID-19 Impact to the Foundation
Like other organizations, we put policies in place for all of our staff
members to work from home. We also put a temporary ban on travel for
staff members. We are continuing our work supporting the community and
Project, but some of our work and responses may be delayed because of
changes in some of our priorities and the impact of limited childcare
for a few of our staff members.
Partnerships and Commercial User Support
We help facilitate collaboration between commercial users and FreeBSD
developers. We also meet with companies to discuss their needs and
bring that information back to the Project. Not surprisingly, the stay
at home orders, combined with our company ban on travel during Q3 made
in-person meetings non-existent. However, the team was able to continue
meeting with our partners and commercial users virtually. These
meetings help us understand some of the applications where FreeBSD is
used.
We are currently scheduling Zoom company meetings for Q4, please reach
out if you would like to schedule a meeting with us.
Fundraising Efforts
Last quarter we raised $192,874.43! Thank you to the individuals and
organizations that stepped in, to help fund our efforts. We'd like to
thank Arm for their large contribution last quarter, which helped bring
our 2020 fundraising effort to $521k. We hope other organizations will
follow their lead and give back to help us continue supporting FreeBSD.
These are trying times, and we deeply appreciate every donation that
has come in from $5 to $150,000. We're still here giving 110% to
supporting FreeBSD!
We are 100% funded by donations, and those funds go towards software
development work to improve FreeBSD, FreeBSD advocacy around the world,
keeping FreeBSD secure, continuous integration improvements, sponsoring
BSD-related and computing conferences (even the virtual events!), legal
support for the Project, and many other areas.
Please consider making a donation to help us continue and increase our
support for FreeBSD.
We also have the Partnership Program, to provide more benefits for our
larger commercial donors. Find out more information about the
partnership program and share with your companies!
OS Improvements
A number of FreeBSD Foundation grant recipients started, continued
working on, or completed projects during the third quarter. These
include:
* Ongoing WiFi and Linux KPI layer improvements.
* Linuxulator application compatibility.
* DRM / Graphics driver updates.
* Zstd compression for OpenZFS.
* Online RAID-Z expansion.
* Modernized LLDB target support for FreeBSD.
You can find more details about most of these projects in other
quarterly
reports.
Staff members also worked on a number of larger projects, including:
* Run-Time Dynamic Linker (rtld) and kernel ELF loader improvements.
* Rewritten UNIX domain socket locking.
* Build infrastructure.
* Open system call path handling support for O_BENEATH,
O_RESOLVE_BENEATH.
* arm64 support.
* Migration to a Git repository.
Many of these projects also have detailed entries in other quarterly
report
entries.
Staff members also put in significant effort in many ways other than
larger, individual projects. These include assisting with code reviews,
bug report triage, security report triage and advisory handling,
addressing syzkaller reports, and ongoing maintenance and bug fixes in
functional areas such as the tool chain, developer tools, virtual
memory kernel subsystem, low-level x86 infrastructure, sockets and
protocols, and others.
University of Waterloo Co-op
With the transition to working from home, the Foundation decided to
again take on three University of Waterloo Co-op students for the Fall
2020 term (September to December). Tiger returns for a second term,
joined by new students Yang and Zac. Projects for the term include more
work on ELF Tool Chain, application of Capsicum to additional
utilities, testing and integration of FreePBX and Asterisk VOIP
software, pkgbase, and exploring containerization tooling.
Continuous Integration and Quality Assurance
The Foundation provides a full-time staff member and funds projects on
improving continuous integration, automated testing, and overall
quality assurance efforts for the FreeBSD project.
During the third quarter of 2020, Foundation staff continued improving
and monitoring the Project's CI infrastructure, and working with
experts to fix the failing builds and the regressions found by tests.
The setting up of dedicated VM host for running tests is completed. New
feature developments and the CI staging environment is in progress. We
are also working with other teams in the Project for their testing
needs. For example, tests of non-x86 architectures now run
periodically, and improve the CI of the embedded systems. We are also
working with many external projects and companies to improve the CI
between their products and FreeBSD.
See the FreeBSD CI section of this report for completed work items and
detailed information.
Supporting FreeBSD Infrastructure
The Foundation provides hardware and support to improve the FreeBSD
infrastructure. Last quarter, we continued supporting FreeBSD hardware
located around the world. We coordinated efforts between the new NYI
Chicago facility and clusteradm to start working on getting the
facility prepared for some of the new FreeBSD hardware we are planning
on purchasing. NYI generously provides this for free to the Project. We
also worked on connecting with the new owners of the Bridgewater site,
where most of the FreeBSD infrastructure is located.
Some of the purchases we made for the Project last quarter to support
infrastructure includes:
* Spamhaus spam filtering software to limit the amount of spam on the
mailing lists.
* 5 application servers to run tasks like bugzilla, wiki, website,
cgi, Phabricator, host git, etc.
* 1 server to replace the old pkg server and provide a lot more IOPS
to avoid the slowdowns seen during peak times of the day where the
disks just cannot keep up with the request volume.
* 1 server for exp-runs to make them faster.
* 1 server to build packages more frequently.
FreeBSD Advocacy and Education
A large part of our efforts are dedicated to advocating for the
Project. This includes promoting work being done by others with
FreeBSD; producing advocacy literature to teach people about FreeBSD
and help make the path to starting using FreeBSD or contributing to the
Project easier; and attending and getting other FreeBSD contributors to
volunteer to run FreeBSD events, staff FreeBSD tables, and give FreeBSD
presentations.
The FreeBSD Foundation sponsors many conferences, events, and summits
around the globe. These events can be BSD-related, open source, or
technology events geared towards underrepresented groups. We support
the FreeBSD-focused events to help provide a venue for sharing
knowledge, to work together on projects, and to facilitate
collaboration between developers and commercial users. This all helps
provide a healthy ecosystem. We support the non-FreeBSD events to
promote and raise awareness of FreeBSD, to increase the use of FreeBSD
in different applications, and to recruit more contributors to the
Project. As is the case for most of us in this industry, COVID-19 has
put our in-person events on hold. In addition to attending virtual
events, we are continually working on new training initiatives and
updating our selection of how-to guides to facilitate getting more
folks to try out FreeBSD.
Check out some of the advocacy and education work we did last quarter:
* Launched our FreeBSD Fridays series of 101 classes. Topics included
an Introduction to FreeBSD, FreeBSD Installfest, Introduction to
Security, Introduction to ZFS and more. Videos of the past sessions
and a schedule of upcoming events can be found here.
* Attended and presented at OSI's State of the Source conference. The
event was held virtually, September 9-11, 2020.
* Launched the redesign of the FreeBSD Foundation Website.
* Announced the 20th Anniversary of the FreeBSD Foundation.
* Participated as an Admin for Google Summer of Code 2020
* Continued to promote the FreeBSD Office Hours series including
holding our own Foundation led office hours. Videos from the one
hour sessions can be found on the Project's YouTube Channel. You
can watch ours here.
* Interviewed members of the outgoing FreeBSD Core Team to get their
thoughts on their term.
* Began working with the FreeBSD Vendor Summit planning committee on
the November 2020 Vendor Summit.
* Promoted the Foundation's 20th Anniversary and our work to support
the FreeBSD Project in the It's FOSS Article. FreeBSD Foundation
Celebrates 20 Years of Promoting and Supporting FreeBSD Project.
* Authored a Beginners Guide to FreeBSD for Fosslife.
* Committed to sponsoring All Things Open as a media Sponsor.
* Committed to sponsoring the OpenZFS Developers Summit at the Bronze
level.
* Became an International RISC-V Member.
* Committed to giving a FreeBSD talk at the nerdear.la conference on
October 20th.
Keep up to date with our latest work in our
monthly newsletters.
Netflix provided an update on how and why they use FreeBSD in our
latest Contributor Case Study.
We help educate the world about FreeBSD by publishing the
professionally produced FreeBSD Journal. As we mentioned previously,
the FreeBSD Journal is now a free publication. Find out more and access
the latest issues at https://www.FreeBSDfoundation.org/journal/.
You can find out more about events we attended and upcoming events at
https://www.FreeBSDfoundation.org/news-and-events/.
Legal/FreeBSD IP
The Foundation owns the FreeBSD trademarks, and it is our
responsibility to protect them. We also provide legal support for the
core team to investigate questions that arise. We updated our Trademark
Usage Terms and Conditions on July 1, 2020.
Go to the FreeBSD Foundation's web site to find out how we support
FreeBSD and how we can help you!
### Other
We welcomed Andrew Wafaa and Kevin Bowling to our board of directors,
to help govern the Foundation and guide us with our strategic
direction. We have more information about our new board members on our
website.
__________________________________________________________________
FreeBSD Release Engineering Team
Links
FreeBSD 12.2-RELEASE schedule
URL: https://www.freebsd.org/releases/12.2R/schedule.html
FreeBSD 12.2 test builds
URL: https://www.freebsd.org/where.html#helptest
FreeBSD development snapshots
URL: https://download.freebsd.org/ftp/snapshots/ISO-IMAGES/
Contact: FreeBSD Release Engineering Team <re at FreeBSD.org>
The FreeBSD Release Engineering Team is responsible for setting and
publishing release schedules for official project releases of FreeBSD,
announcing code freezes and maintaining the respective branches, among
other things.
During the third quarter of 2020, the Release Engineering Team started
work on the 12.2-RELEASE cycle, the third release from the stable/12
branch.
As of this writing, two BETA builds have been released, with the
expectation there will be a third BETA build currently remaining on the
schedule.
The 12.2-RELEASE cycle will continue throughout October, with two RC
builds currently planned, and RC3 scheduled on an as-needed basis. The
12.2-RELEASE is so far scheduled for final release on October 27.
In addition to the 12.2-RELEASE, Glen Barber of the Release Engineering
Team finished work to the release build tools and scripts to prepare
for the conversion from Subversion to Git for the 13.0-RELEASE cycle.
There are no plans to merge these changes to stable branches at this
time; as discussed within the Git working group, we feel such a change
on a stable branch would be too intrusive to our user base as well as
downstream FreeBSD consumers. Development snapshot builds for
13.0-CURRENT have recently been built from the Git tree within the
project, and further snapshot builds for 12.x and 11.x will continue to
be built from Subversion.
Additionally throughout the quarter, several development snapshots
builds were released for the head, stable/12, and stable/11 branches.
Finally, the Release Engineering Team would like to thank Marius Strobl
for his time serving on the team; he had recently stepped down from the
Deputy RE Lead role due to constraints on his time. The Team welcomes
Colin Percival, who has accepted fulfilling this role.
Much of this work was sponsored by Rubicon Communications, LLC
(netgate.com) and the FreeBSD Foundation.
__________________________________________________________________
Cluster Administration Team
Links
Cluster Administration Team members
URL: https://www.freebsd.org/administration.html#t-clusteradm
Contact: Cluster Administration Team <clusteradm at FreeBSD.org>
The FreeBSD Cluster Administration Team consists of the people
responsible for administering the machines that the Project relies on
for its distributed work and communications to be synchronised. In this
quarter, the team has worked on the following:
* Work with the FreeBSD Foundation on hardware update for web
services, mirror and package building servers.
* Disable directory indexing on the package mirrors to resolve
performance issues of the machine.
+ This was later relaxed to allow indexing of the parent
directories but still disallow the large package directories.
* Ongoing systems administration work:
+ Accounts management for committers.
+ Backups of critical infrastructure.
+ Keeping up with security updates in 3rd party software.
Work in progress:
* Setup Malaysia (KUL) mirror.
* Setup Brazil (BRA) mirror.
* Review the service jails and service administrators operation.
* Infrastructure of building aarch64 and powerpc64 packages.
+ NVMe issues on PowerPC64 POWER9 blocking dual socket machine
from being used as pkg builder.
+ Drive upgrade test for pkg builders (SSDs) courtesy of the
FreeBSD Foundation.
+ Boot issues with Aarch64 reference machines.
* New NYI.net sponsored colocation space in Chicago-land area.
* Work with git working group for the git repository.
* Searching for more providers that can fit the requirements for a
generic mirrored layout or a tiny mirror.
__________________________________________________________________
Continuous Integration
Links
FreeBSD Jenkins Instance
URL: https://ci.FreeBSD.org
FreeBSD Hardware Testing Lab
URL: https://ci.FreeBSD.org/hwlab
FreeBSD CI artifact archive
URL: https://artifact.ci.FreeBSD.org
FreeBSD CI weekly report
URL: https://hackmd.io/@FreeBSD-CI
FreeBSD Jenkins wiki
URL: https://wiki.freebsd.org/Jenkins
Hosted CI wiki
URL: https://wiki.freebsd.org/HostedCI
3rd Party Software CI
URL: https://wiki.freebsd.org/3rdPartySoftwareCI
Tickets related to freebsd-testing@
URL: https://preview.tinyurl.com/y9maauwg
FreeBSD CI Repository
URL: https://github.com/freebsd/freebsd-ci
Contact: Jenkins Admin <jenkins-admin at FreeBSD.org>
Contact: Li-Wen Hsu <lwhsu at FreeBSD.org>
Contact: freebsd-testing Mailing List
Contact: IRC #freebsd-ci channel on EFNet
The FreeBSD CI team maintains the continuous integration system of the
FreeBSD project. The CI system firstly checks the committed changes can
be successfully built, then performs various tests and analysis over
the newly built results. The artifacts from those builds are archived
in the artifact server for further testing and debugging needs. The CI
team members examine the failing builds and unstable tests and work
with the experts in that area to fix the codes or adjust test
infrastructure. The details of these efforts are available in the
weekly CI reports.
During the third quarter of 2020, we continued working with the
contributors and developers in the project to fulfill their testing
needs and also keep collaborating with external projects and companies
to improve their products and FreeBSD.
Important changes:
* All !x86 -test builds now trigger a new build on 22:00 UTC daily;
this was not running very often because running all the tests in
qemu takes lots of time. The work on improving the test execution
speed and parallelism is in progress. The following is a list of
the jobs affected:
+ Test build for FreeBSD HEAD on ARMv7.
+ Test build for FreeBSD HEAD on AArch64.
+ Test build for FreeBSD HEAD on MIPS64.
+ Test build for FreeBSD HEAD on PowerPC64.
+ Test build for FreeBSD HEAD on RISC-V64.
* The build and test results will be sent to the dev-ci mailing list
soon. Feedback and help with analysis is very appreciated!
+ A builder dedicated to run jobs using provisioned VMs is
setup, this improves the stableness and reduces the execution
time.
+ The result of FreeBSD-head-amd64-test_zfs is changed after
OpenZFS importing; we encourage everyone to check and fix the
failing and skipped test cases.
New jobs added:
* CI build for FreeBSD HEAD on PowerPC64LE.
Work in progress:
* Collecting and sorting CI tasks and ideas here.
* Testing and merging pull requests in the the FreeBSD-ci repo.
* Designing and implementing pre-commit CI building and testing,
* Reduce the procedures of CI/test environment setting up for
contributors and developers.
* Setting up the CI stage environment and putting the experimental
jobs on it.
* Setting up public network access for the VM guest running tests.
* Implementing automatic tests on bare metal hardware.
* Adding drm ports building tests against -CURRENT.
* Planning to run ztest and network stack tests.
* Adding more external toolchain related jobs.
* Improving the hardware lab to be more mature and adding more
hardware.
* Helping more 3rd software get CI on FreeBSD through a hosted CI
solution.
* Working with hosted CI providers to have better FreeBSD support.
Please see freebsd-testing@ related tickets for more WIP information,
and don't hesitate to join the effort!
Sponsor: The FreeBSD Foundation
__________________________________________________________________
Ports Collection
Links
About FreeBSD Ports
URL: https://www.FreeBSD.org/ports/
Contributing to Ports
URL: https://www.freebsd.org/doc/en_US.ISO8859-1/articles/contributing/ports
-contributing.html
FreeBSD Ports Monitoring
URL: http://portsmon.freebsd.org/index.html
Ports Management Team
URL: https://www.freebsd.org/portmgr/index.html
Contact: Ren?? Ladan <portmgr-secretary at FreeBSD.org>
Contact: FreeBSD Ports Management Team <portmgr at FreeBSD.org>
The Ports Management Team is responsible for overseeing the overall
direction of the Ports Tree, building packages, and personnel matters.
Below is what happened in the last quarter.
We passed the landmark of 40,000 ports in the Ports Collection and are
now around 40,400 ports. The last quarter saw 9335 commits to the HEAD
branch and 481 commits to the 2020Q3 branch by respectively 167 and 63
committers. There are currently 2525 open problem reports of which 595
are unassigned. Compared to last quarter, this means a slight decrease
in activity and also a slight increase in open PRs.
During the last quarter we welcomed Rainer Hurling (rhurlin@) and said
goodbye to Kevin Lo (kevlo@) and Grzegorz Blach (gblach@).
The last three months saw new default versions for Perl (5.32),
PostgreSQL (12) and PHP (7.4). Various packages also got updated:
Firefox to 81.0.1, Chromium to 84.0.4147.135, Gnome to 3.36, Xorg to
1.20.9, Qt5 to 5.15.0, Emacs to 27.1, KDE Frameworks to 5.74.0 and pkg
itself to 1.15.8.
Never tired, antoine@ ran 30 exp-runs to test port version updates, on
such diverse matters as:
* Updating byacc in base to 20200330.
* Check balancing of sed "y" command.
* Use of brackets.
* Removing the now redundant "port" argument from USES=readline.
__________________________________________________________________
FreeBSD Office team - 3rd quarter 2020 report
Links
The FreeBSD Office project
URL: https://wiki.freebsd.org/Office
Contact: FreeBSD Office team ML <office at FreeBSD.org>
Contact: Dima Panov <fluffy at FreeBSD.org>
Contact: Li-Wen Hsu <lwhsu at FreeBSD.org>
The FreeBSD Office team works on a number of office-related software
suites and tools such as OpenOffice and LibreOffice.
Work during this quarter focused on providing the latest stable release
of LibreOffice suite and companion apps to all FreeBSD users.
* Alongside with updating old stable branch to latest 6.4.x releases,
current ports-tree now have a full-featured cutting-edge 7.0.1
bundle.
* Conservative users can keep 6.4.x stable version by switching to
use all-in-one editors/libreoffice6 port and even with i18n
language pack (off by default). It will be kept updated at least
till 7.1.0 version is released.
We are looking for people to help the project.
All unstable work with LibreOffice snapshots is staged in our WIP
repository.
The open bugs list contains all filed issues which need some attention.
Patches, comments and objections are always welcome in the mailing list
and bugzilla.
__________________________________________________________________
FreeBSD Graphics Team status report
Links
Project GitHub page
URL: https://github.com/FreeBSDDesktop
Contact: FreeBSD Graphics Team <x11 at freebsd.org>
Contact: Niclas Zeising <zeising at freebsd.org>
The FreeBSD X11/Graphics team maintains the lower levels of the FreeBSD
graphics stack. This includes graphics drivers, graphics libraries such
as the MESA OpenGL implementation, the X.org xserver with related
libraries and applications, and Wayland with related libraries and
applications.
There have been several updates to the FreeBSD graphics stack and
related libraries since the last report.
Most notably, MESA related ports were changed to use the meson build
system, instead of the autotools based one. This was needed since mesa
upstream has deprecated and removed the autotools build system, and
this paved the way for further mesa updates. While there was a need for
a few minor corrections after the initial update, this update has been
successful and made it possible to further update and improve the
FreeBSD mesa port.
There have also been several security fixes for xorg-server and libX11,
so these ports have been updated to fix these issues.
During the period, FreeBSD 12 was changed to improve the compatibility
with input devices using udev/evdev and libinput. This change removes
the need for local configuration and makes most mice, touchpads and
keyboards work out of the box. This change will be in the upcoming
FreeBSD 12.2 release.
There have also been several updates to various libraries, both in the
graphics and input stacks, and several userland drivers have been
updated. Libraries such as libdrm and libevdev have been updated to
include new FreeBSD support, developed by team members and added
upstream.
There has also been ongoing work to keep the various drm-kmod ports and
packages up to date, mostly in response to changes in various FreeBSD
versions.
We have also continued our regularly scheduled bi-weekly meetings.
People who are interested in helping out can find us on the
x11 at FreeBSD.org mailing list, or on our gitter chat. We are also
available in #freebsd-xorg on EFNet.
We also have a team area on GitHub where our work repositories can be
found.
__________________________________________________________________
Projects
Projects that span multiple categories, from the kernel and userspace
to the Ports Collection or external projects.
FreeBSD on Microsoft HyperV and Azure
Links
Microsoft Azure article on FreeBSD wiki
URL: https://wiki.freebsd.org/MicrosoftAzure
Microsoft HyperV article on FreeBSD wiki
URL: https://wiki.freebsd.org/HyperV
Contact: FreeBSD Integration Services Team <bsdic at microsoft.com>
Contact: Wei Hu <whu at FreeBSD.org>
Contact: Li-Wen Hsu <lwhsu at FreeBSD.org>
Li-Wen is working on the FreeBSD release code related to Azure for the
-CURRENT, 12-STABLE and 11-STABLE branches. The work-in-progress is
available here. The 11.4-RELEASE image on Azure Marketplace is
published. We are testing the releng/12.2 branch and 12.2-RELEASE image
will be published to Azure Marketplace soon after released.
This project is sponsored by The FreeBSD Foundation, with resources
provided by Microsoft.
__________________________________________________________________
Building FreeBSD on non-FreeBSD hosts
Links
Wiki
URL: https://wiki.freebsd.org/BuildingOnNonFreeBSD
Contact: Alex Richardson <arichardson at freebsd.org>
Until recently FreeBSD could only be built on a FreeBSD host. However,
many popular free CI tools only allow building on Linux or macOS and
therefore can not be used for building the FreeBSD base system.
Furthermore, it is sometimes useful to cross-build FreeBSD for a remote
machine or an emulator even if the build machine is not running
FreeBSD. The goal of this project is to allow building the base system
on Linux and macOS hosts.
I started this project in 2017 to allow building CheriBSD on the Linux
servers and desktops that many of us working on the CHERI project use.
The first few patches were upstreamed in 2018 (see the 2018q3 report)
and I merged the full set of patches to CheriBSD shortly after. Over
the past two years I have slowly been upstreaming the remaining patches
and finally committed the last required change in time for this report.
As of September 2020 it should be possible to use the buildworld and
buildkernel make targets to build a fully-functional FreeBSD
installation on macOS and Linux hosts. We use this in our continuous
integration system to build and test CheriBSD disk images for multiple
architectures. I have also committed a GitHub Actions configuration
upstream that takes approximately 10 minutes to build an amd64 kernel.
This will ensure that changes that break crossbuilding from Linux/macOS
can be detected easily.
Upstreaming the crossbuilding changes has resulted in various build
system cleanups. For example, we now no longer need to use lorder.sh
when building libraries which speeds up the linking step a bit. The
portability and bootstrapping changes should also make it easier to
upgrade from older versions since we no longer rely on host headers in
/usr/include matching those of the target system (e.g. when
bootstrapping localedef, etc.).
While this support for building on Linux and macOS should still be
considered experimental, it should work in many cases. If you would
like to give it a try, the following command line should successfully
build an amd64 world on Linux and macOS systems that have packages for
LLVM 10 (or newer) installed: MAKEOBJDIRPREFIX=/somewhere
./tools/build/make.py TARGET=amd64 TARGET_ARCH=amd64 buildworld Builds
must be performed using the ./tools/build/make.py wrapper script since
most Linux and macOS systems do not ship an appropriate version of
bmake. Please let me know if you encounter any issues.
Sponsor: DARPA
__________________________________________________________________
Git Migration Working Group
Links
Git conversion tooling repo
URL: https://github.com/freebsd/git_conv
FreeBSD-git mailing list
URL: https://lists.freebsd.org/mailman/listinfo/freebsd-git
Beta doc git repo
URL: https://cgit-beta.FreeBSD.org/doc
Beta ports git repo
URL: https://cgit-beta.FreeBSD.org/ports
Beta src git repo
URL: https://cgit-beta.FreeBSD.org/src
Contact: Ed Maste <emaste at FreeBSD.org>
Contact: Warner Losh <imp at FreeBSD.org>
Contact: Ulrich Sp??rlein <uqs at FreeBSD.org>
Work continues on FreeBSD's migration from Subversion to Git. Ulrich
has addressed all known issues with svn2git and has been able to work
around the inconsistent metadata and forced commit issues in the
Subversion history.
We still have additional documentation to write, and need to finish
installing commit hooks (e.g. restricting branch creation, or ensuring
appropriate data exists on cherry-pick commits).
We expect to open the beta repository to test commits before the end of
October. This is to allow testing of the commit hooks, and to allow
developers to test access and become familiar with git operation.
Commits in this repository will be deleted and the repository will be
recreated at least once prior to the final migration.
Those with an interest in the migration to Git are encouraged to
subscribe to the FreeBSD-git mailing list and test out the beta src,
ports, and/or doc repositories.
You are also welcome check out the wiki, issues, README and other
documentation at the Git conversion tooling repo.
We currently expect to transition the src and doc repositories in
mid-November. Additional investigation and experimentation with the
ports repository is still underway.
Sponsor: The FreeBSD Foundation (in part)
__________________________________________________________________
Linux compatibility layer update
Contact: Edward Tomasz Napierala <trasz at FreeBSD.org>
Contact: Mark Johnston <markj at FreeBSD.org>
Earlier Linuxulator work focused on code cleanups and improving
diagnostic tools. Work has now shifted from cleanups to fixing actual
applications. Current status is being tracked at Linux app status Wiki
page. Initial focus was on applications that don't involve X11, mostly
because they tend to be easier to test and debug, and the bug fixes are
not application-specific.
Foundation-sponsored work during this quarter included implementing a
devfs(5) workaround to fix gettynam(3) inside jail/chroot, and
workaround for the missing splice(2) syscall, which caused problems for
grep and autotools. The Linux version reported to userspace was bumped
to 3.10.0, which matches the kernel shipped with RHEL 7 and is
neccessary for IBM's DB2 database installation to succeed. The
BLKPBSZGET ioctl neccessary for Oracle database is supported now. There
is now support for kcov(4), neccessary for syzcaller; as well as a
number of fixes for issues reported by syzcaller, such as futex lock
leaks. There were also more cleanups, including moving some
Linuxulator-specific functionality related to error handling off from
the syscall's fast code paths. The sysutils/debootstrap port, which
provides an easy way to create Debian or Ubuntu jail, was updated to
version 1.0.123. Finally there were some improvements to the
documentation.
Most of those changes have been merged to FreeBSD 12-STABLE, in order
to ship with 12.2-RELEASE.
There is increased involvement from other developers; this includes
termios performance fixes, improved memfd support, implementing
CLOCK_MONOTONIC_RAW required for Steam, madvise improvements, new
compat.linux.use_emul_path sysctl. There is also ongoing work on
tracking down the causes of failures related to Steam and WebKit, with
fixes being first implemented in linuxulator-steam-utils.
Sponsor: The FreeBSD Foundation
__________________________________________________________________
LLDB Debugger Improvements
Links
Moritz Systems Project Description
URL: https://www.moritz.systems/blog/lldb-debugger-improvements-for-freebsd/
Git Repository
URL: https://github.com/moritz-systems/llvm-project
Contact: Kamil Rytarowski <kamil at moritz.systems>
Contact: Michal G??rny <mgorny at moritz.systems>
FreeBSD includes LLDB, the debugger in the LLVM family, in the base
system. At present it has some limitations in comparison with the GNU
GDB debugger, and does not yet provide a complete replacement. It
relies on an obsolete plugin model in LLDB that causes growing
technical debt. This project aims to bring LLDB closer to a fully
featured replacement for GDB, and therefore for FreeBSD to feature a
modern debugger for software developers.
The legacy monolithic target supports the executed application being
debugged in the same process space as the debugger. The modern LLDB
plugin approach, used on other supported targets, executes the target
process under a separate lldb-server process. This improves reliability
and simplifies the process / thread model in LLDB itself. In addition,
remote and local debugging will both be performed using the same
approach.
After the migration to the new process model is complete, the project
will include reviewing the results of LLDB's test suite and fixing
tests as time permits. The work is expected to be complete in 2020.
The project schedule is divided into three milestones, each taking
approximately one month:
1. Introduce new FreeBSD Remote Process Plugin for x86_64 with basic
support and upstream to LLVM. 2. Ensure and add the mandated features
in the project (process launch, process attach (pid), process attach
(name), userland core files, breakpoints, watchpoints, threads, remote
debugging) for FreeBSD/amd64 and FreeBSD/i386. 3. Iterate over the LLDB
tests. Detect, and as time permits, fix bugs. Ensure bug reports for
each non-fixed and known problem. Add missing man pages and update the
FreeBSD Handbook.
We are nearing the completion of the first milestone. The new plugin is
getting into shape, and it can already run simple single-threaded
programs. The supported features include single-stepping, breakpoints,
memory and register I/O on amd64. Both plugins are supported
simultaneously. The new plugin is used if FREEBSD_REMOTE_PLUGIN
environment variable is set to any value, or if lldb-server is spawned
directly. Otherwise, the old plugin is used for compatibility. Once the
new plugin matures, we are planning to enable it unconditionally on the
architectures that it is ported to.
Sponsor: The FreeBSD Foundation
__________________________________________________________________
Lua usage in FreeBSD
Contact: Ed Maste <emaste at FreeBSD.org>
Contact: Kyle Evans <kevans at FreeBSD.org>
Contact: Ryan Moeller <freqlabs at FreeBSD.org>
During this quarter, flua (FreeBSD Lua) was taught where to find base
.lua modules in order to support require of .lua modules to be provided
by the base system. flua also gained support for require of binary
modules.
A review for libjail bindings has also been submitted, pending review.
libjail is an essential component if one wants to be able to write jail
management utilities in flua.
People interested in working with Lua in FreeBSD are welcome to get in
contact to discuss other project ideas. To name a couple of potential
projects, some interesting modules that have not been started but could
prove useful (listed in no particular order):
* libcrypt
* libexpat
* libnv
* libxo
There is also a small list of scripts that would do well with a port to
flua:
* certctl(8)
__________________________________________________________________
NFS over TLS implementation
Contact: Rick Macklem <rmacklem at freebsd.org>
In an effort to improve NFS security, an internet draft which I expect
will become an RFC soon specifies the use of TLS 1.3 to encrypt all
data traffic on a Sun RPC connection used for NFS.
Although NFS has been able to use sec=krb5p to encrypt data on the
wire, this requires a Kerberos environment and, as such, has not been
widely adopted. It also required that encryption/decryption be done in
software, since only the RPC message NFS arguments are encrypted. Since
Kernel TLS is capable of using hardware assist to improve performance
and does not require Kerberos, NFS over TLS may be more widely adopted,
once implementations are available.
The coding for this project has now been completed. All required
changes to the NFS and kernel RPC code have been committed to -CURRENT.
The daemons are now believed to be complete, but will remain in
base/projects/nfs-over-tls until -CURRENT has an OpenSSL library with
the kernel TLS support incorporated in it. If this does not happen for
FreeBSD-13, hopefully the patched OpenSSL and the daemons can become
ports.
To support clients such as laptops, the daemons that perform the TLS
handshake may optionally handle client X.509 certificates from a site
local CA. There are now exports(5) options to require client(s) to
provide a valid X.509 certificate.
While setting up system(s) for testing is still a little awkward, the
documentation is now available for those who want to help with testing.
The main limitation in the current implementation is that it uses
TLS1.2 and not TLS1.3. This should change once the KERN_TLS rx patch
includes TLS1.3 support.
Third party testing would be appreciated.
__________________________________________________________________
syzkaller on FreeBSD
Contact: Mark Johnston <markj at FreeBSD.org>
See the syzkaller entry in the 2019q1 quarterly report for an
introduction to syzkaller.
syzkaller, especially the public syzbot instance, continues to find
bugs in the FreeBSD kernel. A number of these bugs have been fixed in
subsystems such as the VFS name cache, the TCP and SCTP stacks, pf(4),
the unix domain socket implementation, and the Linuxulator.
The FreeBSD Foundation sponsored some work to enable cross-OS fuzzing.
This makes it possible to fuzz the Linuxulator using syzkaller's Linux
target. This effort quickly found several bugs; once the support is
committed upstream we will hopefully be able to leverage syzbot to gain
continuous testing of the Linux system call interface in addition to
the native and 32-bit compatibility interfaces.
Some work was also done to enable running syzkaller in a FreeBSD jail,
with the eventual aim of making it easy to distribute binary images
containing everything required to immediately start running syzkaller
on a new host. Currently a number of setup steps are required, making
deployment somewhat painful.
Sponsor: The FreeBSD Foundation
__________________________________________________________________
Kernel
Updates to kernel subsystems/features, driver support, filesystems, and
more.
DRM Drivers Update
Links
drm-kmod
URL: https://github.com/freebsd/drm-kmod/
Contact: Emmanuel Vadot <manu at FreeBSD.Org>
The drm drivers for FreeBSD 13-CURRENT have been updated to match Linux
5.4.62 Then graphics/drm-current-kmod have been updated to follow this
LTS release of Linux.
For now graphics/drm-devel-kmod is also tracking this release but will
be updated to a later revision of Linux drm drivers in the near future.
A lot of linuxkpi code was removed from the ports or replaced with a
BSD licenced implementation.
Sponsor: The FreeBSD Foundation
__________________________________________________________________
DTS Update
Contact: Emmanuel Vadot <manu at FreeBSD.org>
DTS files (Device Tree Sources) were updated to be on par with Linux
5.8 for HEAD and 5.6 for the 12-STABLE branch.
__________________________________________________________________
DesignWare Ethernet adapter driver improvements
Links
WIP branch
URL: https://github.com/gonzoua/freebsd/tree/rk_eth
Contact: Oleksandr Tymoshenko <gonzo at FreeBSD.org>
DesignWare Ethernet adapter IP is used in Rockchip and Allwinner SoCs.
The driver was updated with following fixes:
* Initialize clocks instead of relying on u-boot to do the right
thing.
* Sense media type and adjust controller configuration accordingly.
* Add support for RMII PHY mode.
Yet uncommitted changes include performance optimisation by adding
support for multi-segment mbuf transmission. The next step is to try to
get more performance boost by using interrupt coalescence.
__________________________________________________________________
Google Summer of Code'20 Project - eBPF XDP Hooks
Links
Github diff link
URL: https://github.com/Ankurk99/freebsd/tree/ebpf-import
Project wiki
URL: https://wiki.freebsd.org/SummerOfCode2020Projects/eBPFXDPHooksl
Contact: Ankur Kothiwal <ankur at freebsd.org>
The eBPF eXpress Data Path (XDP) allows eBPF programs to be run to
filter received packets as early as possible, avoiding unnecessary
processing overhead before the filter is run. The goal of this project
is to extend an existing FreeBSD network driver (a virtual NIC like a
VirtIO if_vtnet) to be able to call into an eBPF program when
processing a newly received packet. In short, with XDP the driver must
PASS (accept and process normally), DROP, TX or REDIRECT the packet as
specified by the program. eBPF helper functions and maps for aiding in
packet filtering will also be implemented.
Implemented:
* Register a eBPF probe when an interface is registered with pfil.
* Activating eBPF probe.
* Create hooks and link them to the pfil head when the eBPF XDP probe
is activated and successfully list the XDP probes.
* Create a xdp_rx function which will pass the received packets to
the eBPF program where the packets can be further processed. This
function will return XDP actions: DROP and PASS.
* Register the xdp hook and link it to the pfil head.
* Write an eBPF program to process (currently drop and pass) ICMP
traffic - This is to test that the hook is working properly.
* Write a loader function to load the ICMP filter program to the
kernel.
Future Work:
* Currently we can only attach the XDP hook to PASS and DROP the
packets - The work on detaching the hook is left.
* The XDP action to "TX" and "REDIRECT" the packets.
Final Deliverables:
* Implemented XDP hook to pass and drop packets.
* Created a loader program to attach the eBPF program to the kernel.
* A test program to DROP ICMP filter.
This code was done under the Google Summer of Code 2020 under the
guidance
of Ryan Stone (rstone@). The eBPF implementation for FreeBSD is still a
work in progress and FreeBSD doesn't support eBPF yet. The basic
implementation for eBPF was a GSoC'18 project, and is still under
development. This project is based on that implementation so the XDP
implementation for FreeBSD can only be merged into the FreeBSD source
code once it supports eBPF.
Currently this code is a work in progress and is merged to Ryan Stone's
branch with support for the eBPF implementation.
Sponsor: Google Summer of Code
__________________________________________________________________
ENA FreeBSD Driver Update
Links
ENA README
URL: https://github.com/amzn/amzn-drivers/blob/master/kernel/fbsd/ena/README
Contact: Michal Krawczyk <mk at semihalf.com>
Contact: Artur Rojek <ar at semihalf.com>
Contact: Marcin Wojtas <mw at semihalf.com>
ENA (Elastic Network Adapter) is the smart NIC available in the
virtualized environment of Amazon Web Services (AWS). The ENA driver
supports multiple transmit and receive queues and can handle up to 100
Gb/s of network traffic, depending on the instance type on which it is
used.
Completed since the last update:
* Fix ENA compilation in case it is integrated into the kernel
binary.
* MFC of the ENA v2.2.0 driver to the FreeBSD 12.2.
Work in progress:
* Add feature that allows reading extra ENI (Elastic Network
Interface) metrics about exceeding BW/pps limits.
* Introduce full kernel RSS API support.
* Allow reconfiguration of the RSS indirection table and hash key.
* Evaluation and prototyping of the driver port to the iflib
framework.
Sponsor: Amazon.com Inc
__________________________________________________________________
IPSec Extended Sequence Number (ESN) support
Contact: Grzegorz Jaszczyk <jaz at semihalf.com>
Contact: Patryk Duda <pdk at semihalf.com>
Contact: Marcin Wojtas <mw at semihalf.com>
Extended Sequence Number (ESN) is IPSec extension defined in RFC4303
Section 2.2.1. It makes possible to implement high-speed IPSec
implementations where standard, 32-bit sequence number is not
sufficient. A key feature of the ESN is that only low order 32 bits of
sequence number are transmitted over the wire. High-order 32 bits are
maintained by sender and receiver. Additionally high-order bits are
included in the computation of Integrity Check Value (ICV) field.
Extended Sequence Number support contains following:
* Modification of existing anti-replay algorithm to fulfil ESN
requirements.
* Trigger soft lifetime expiration at 80% of UINT32_MAX when ESN is
disabled.
* Implement support for including ESN into ICV in cryptosoft engine
in both encrypt and authenticate mode (eg. AES-CBC and SHA256 HMAC)
and combined mode (eg. AES-GCM).
* Implement support for including ESN into ICV in AES-NI engine in
both encrypt and authenticate mode and combined mode.
Completed since the last update:
* Adjust implementation of crypto part to the reworked Open Crypto
Framework.
* Move the core ESN implementation from the crypto drivers to
netipsec layer.
* Make use of the newly introduced crp_aad mechanism for combined
modes.
* Introduce minor fixes and improvements.
TODO:
* Complete review process in Phabricator and merge patches in the
tree.
Sponsor: Stormshield
__________________________________________________________________
NXP ARM64 SoC support
Contact: Marcin Wojtas <mw at semihalf.com>
Contact: Artur Rojek <ar at semihalf.com>
Contact: Dawid Gorecki <dgr at semihalf.com>
The Semihalf team initiated working on FreeBSD support for the NXP
LS1046A SoC
LS1046A are quad-core 64-bit ARMv8 Cortex-A72 processors with
integrated packet processing acceleration and high speed peripherals
including 10 Gb Ethernet, PCIe 3.0, SATA 3.0 and USB 3.0 for a wide
range of networking, storage, security and industrial applications.
Completed since the last update:
* Upstreaming of the QorIQ SDHCI driver (r365054).
With above the current Semihalf upstreaming activity is complete.
The major out-of-tree supported components:
* DPAA network controller support.
* QSPI controller support.
They work on 11.2-RELEASE, but still require significant
effort to adopt to FreeBSD-CURRENT.
Sponsor: Alstom Group
__________________________________________________________________
Addition of PowerPC64LE Architecture
Links
Early notes
URL: https://lists.freebsd.org/pipermail/freebsd-ppc/2020-August/012043.html
Announcement
URL: https://lists.freebsd.org/pipermail/freebsd-ppc/2020-September/012098.h
tml
Contact: Brandon Bergren <bdragon at freebsd.org>
As of r366063, experimental support for little-endian PowerPC64
(PowerPC64LE) is available in -CURRENT for POWER8 and POWER9 machines.
In 2010, when FreeBSD was ported to PowerPC64, the average user would
have been using a G5 PowerMac, a purely big-endian machine.
While, at the time, a 32-bit PowerPC machine could run in
little-endian, as well as POWER6 and POWER7, in practice, the
complexities involved in managing it at the kernel level and lack of
firmware support made it infeasible to support.
When IBM designed POWER8, one main focus was to improve little-endian
support, and bring it up to parity with big-endian.
This improved support makes it practical to support a little-endian
operating environment on what is traditionally a primarily big-endian
platform.
In 2020, with POWER9 being affordable for many users thanks to the
Raptor Blackbird, semi-easy access to surplus POWER8 hardware, IBM
having a major future focus on POWER little-endian, and the decay of
big-endian support in modern video cards and graphical environments,
there is demand for a little-endian version of FreeBSD on POWER.
With FreeBSD/PowerPC64's transition in 2019 to the ELFv2 ABI as part of
the 2019q4 PowerPC on Clang effort, the last major barrier to a
little-endian port was eliminated.
Since nobody else was working on it, and I had the skillset required to
do the port, I decided to experiment one weekend with a little-endian
kernel to see how difficult it would be to port.
It turned out to be a lot more trivial than I was expecting. Three days
later I had console support in qemu, and after another week of
debugging, I had it fully up and running on hardware.
FreeBSD PowerPC64LE is now an experimental MACHINE_ARCH in base, and is
continuing to evolve at a rapid pace.
Big-endian PowerPC64 is still the preferred platform for the
foreseeable future, and will not be deprecated.
Sponsor: Tag1 Consulting, Inc.
__________________________________________________________________
ure - USB 3.0 Gigabit Ethernet Driver update
Links
svn commit: r365648
URL: https://svnweb.freebsd.org/changeset/base/365648
FreeBSD-SA-20:27.ure
URL: https://www.freebsd.org/security/advisories/FreeBSD-SA-20:27.ure.asc
D25809 major update to if_ure
URL: https://reviews.freebsd.org/D25809
Contact: John-Mark Gurney <jmg at FreeBSD.org>
The ure is a driver for handling the RealTek ethernet adapters,
including the RTL8153 USB 3.0 Gigabit ethernet adapters. It is used in
many ethernet dongles and docking stations.
Previous to this update, the driver was limited in speed. In my
testing, I was only able to get ~91Mbps. This limit was due to one
packet per USB transfer. USB has a limit of 8000 transfers per second
(1500 bytes/pkt * 8000 pkts/sec * 8bits/byte == 96 Mbps). This was
acceptable for fast ethernet (RTL8152, 100Mbps), but with the
additional support for Gigabit ethernet, it became a bottleneck.
The updates add sending and receiving multiple packets in a single USB
transfer, VLAN hardware tagging, and enable TCP and UDP checksum
offloading. This increased the speed on gigabit ethernet to ~940 Mbps.
In doing this work, a security vulnerability was discovered in the
driver. Due to improper setting of a device register, on some devices,
it caused packets to be fragmented when they shouldn't be and the
driver was unable to handle them correctly. This allowed an attacker,
who could generate large frames (say, ping packets, or large TCP
transfers), to inject arbitrary packets into the network stack. This
could allow the attacker to spoof traffic from other machines, and
bypass VLAN protections. See the SA for more information.
As part of this work, a script was created to run tests to validate
that basic functionality of the driver (w/o options) work properly, and
then iterate over each option to make sure that they function properly.
This will be released at some point in the future.
If you're interested in helping out, or testing it, let me know.
__________________________________________________________________
Stateless hardware offloads for VXLANs
Links
r365867
URL: https://svnweb.freebsd.org/base?view=revision&revision=r365867
r365868
URL: https://svnweb.freebsd.org/base?view=revision&revision=r365868
r365869
URL: https://svnweb.freebsd.org/base?view=revision&revision=r365869
r365870
URL: https://svnweb.freebsd.org/base?view=revision&revision=r365870
r365871
URL: https://svnweb.freebsd.org/base?view=revision&revision=r365871
RFC6935
URL: https://tools.ietf.org/html/rfc6935
Contact: Navdeep Parhar <np at FreeBSD.org>
VXLAN (Virtual eXtensible LAN) is a tunneling protocol in which Layer 2
traffic for a virtual LAN is encapsulated in UDP and transferred over
Layer 3 networks between VTEPs (VXLAN Tunnel End Points). Traffic on
the wire has two sets of networking headers: the headers for the
encapsulation and the headers of the traffic being encapsulated. VXLANs
are supported by if_vxlan(4) on FreeBSD.
Modern NICs commonly support header checksum insertion and
verification, TSO (TCP Segmentation Offload) on transmit, and RSS for
load distribution on receive. But the default is to operate on the
outermost headers. Some NICs can operate on the inner encapsulated
frames as well. The commits listed above allow if_vxlan(4) to take
advantage of such NICs.
r365867 and r365868 add new mbuf checksum flags and ifnet capabilities.
r365870 implements the kernel parts of the new capabilities and updates
if_vxlan(4) to make use of them. r365871 implements driver support for
the new capabilities in cxgbe(4).
VXLAN and other tunneling protocols that use UDP explicitly allow zero
checksum in the outer UDP header, even with IPv6. r365869 adds support
for configuring one UDP/IPv6 port where zero checksums are allowed.
This work was sponsored by Chelsio Communications and was implemented
and tested using T6 (Terminator 6) NICs supported by cxgbe(4). It is
available in 13.0-CURRENT (head) right now and will be available in
12-STABLE in the future.
VXLANs can be created as usual and will automatically have checksum and
TSO capabilities if the underlying physical interface supports VXLAN
stateless offloads. Use ifconfig to list, disable, and enable checksum
capabilities on the VXLAN interface. Use
https://bugs.freebsd.org/bugzilla/ to report bugs.
Future work:
* Direct call into a vxlan input routine from the driver's receive
routine.
* LRO support in if_vxlan(4).
* GENEVE support.
Sponsor: Chelsio Communications
__________________________________________________________________
Wireless updates
Links
The freebsd-wireless mailing list
URL: https://lists.freebsd.org/mailman/listinfo/freebsd-wireless
athp github repository
URL: https://github.com/erikarn/athp
Contact: Adrian Chadd <adrian at FreeBSD.org>
Contact: Bjoern A. Zeeb <bz at FreeBSD.org>
The following works happened in FreeBSD HEAD (some already in Q2) and
were merged for 12.2-BETA2 and include net80211 and driver updates for
better 11n and upcoming 11ac support.
In more detail, this includes an ath(4) update, some run(4) 11n
support, 11n for otus(4), A-MPDU, A-MSDU, A-MPDU+A-MSDU and Fast frames
options, scanning fixes, enhanced PRIV checks for jails, restored
parent device name printing, improvements for upcoming VHT support,
lots of under-the-hood infrastructure improvements, new device IDs, and
debug tools updates.
If you have a chance please test before the release.
Atheros 11ac driver athp
In the last three months the athp(4) port of the ath10k driver has
progressed well. Adrian reports the following important changes:
* Per-node transmit buffering was implemented, required for correct
hostap and QCA6174 behaviour.
* Issues with ignoring sending some management frames got fixed;
null-data frames were being filtered out and this caused
undesirable hostap behaviour.
* Transmit path refactoring reduced code duplication.
* A fix on firmware start / VAP running tracking no longer stops the
first VAP from coming active after VAP creation / ifconfig up.
* Correcting hostap mode PHY configuration now allows non-VHT
stations to associate and correctly exchange data with a VHT AP.
* Addition of a crypto key configuration cache in the driver ensures
the ieee80211_key details are available after the key is deleted;
net80211 would reuse or free the state before the driver task would
finish the firmware command.
Newer Intel Wireless device support
Initial work was done to integrate net80211 support in the LinuxKPI
compat layer to get the wireless parts going. In addition, upstreaming
code changes and working through problems and review started on two
sides. One was trying to get mostly compile time changes upstreamed to
the iwlwifi driver. The other is sorting out conflicting LinuxKPI
changes to not break the DRM graphics drivers. Bjoern hopes that with
some of that sorted out, he can soon go back to focus on the wireless
parts and produce a new snapshot.
rtw88 and brcmfmac
As the Intel driver port and LinuxKPI advance, both the rtw88, and to a
lower degree the brcmfmac, ports benefit from that. Bjoern lately also
got a brcmfmac PCIe card and started to port support for that. This for
the moment remains a free-time project.
Work by Bjoern was sponsored by: Rubicon Communications, LLC (d/b/a
"Netgate") and The FreeBSD Foundation
__________________________________________________________________
ZSTD Compression in ZFS
Contact: Allan Jude <allanjude at freebsd.org>
Zstandard (ZSTD) is a modern high-performance compression algorithm
designed to provide the compression ratios of gzip while offering much
better performance. ZSTD has been adopted in FreeBSD for a number of
other uses, including compressing kernel crash dumps, as a replacement
for gzip or bzip for compressing log files, and for future versions of
pkg(8).
This effort to complete the integration of ZSTD into ZFS is funded by
the FreeBSD Foundation.
During the third quarter the integrating of ZSTD into OpenZFS was
completed in the upstream OpenZFS repository, and the new OpenZFS 2.0
codebase was imported into 13-CURRENT. Completed milestones in this
project:
* Importing ZSTD 1.4.5 into OpenZFS, using the recent upstream zstd
features that make it easier to embed zstd in other projects.
* Changing the way compression levels are tracked and inherited.
* Save and restore the compression level via an embedded block
header.
* Also store the version of zstd used in the embedded block header,
for future-proofing. The checksum of a block may not match if zstd
is upgraded, since it may compress the block more.
* Add tests to ensure zstd compression and metadata survive ZFS
replication.
* Resolve possible negative interactions with L2ARC and ZFS Native
Encryption.
* Fix bug with L2ARC if the Compressed ARC feature is disabled.
* Improve the ZFS feature activation code, so that zstd cannot create
pools that will panic older versions of ZFS.
With these changes, upgraded pools can compress data with zstd
or zstd-fast, across a wide range of different compression levels. This
will allow the storage administrator to select the
performance-vs-compression tradeoff that best suits their needs.
Tasks remaining to be completed:
* Add a section to the FreeBSD Handbook ZFS chapter about zstd
* Create more documentation around selecting a suitable compression
level
* Finish support for ZSTD in the FreeBSD boot loader (Warner Losh
imp at freebsd.org)
Sponsor: The FreeBSD Foundation
__________________________________________________________________
Architectures
Updating platform-specific features and bringing in support for new
hardware platforms.
CheriBSD 2020 Q3
Links
Contact: Alex Richardson <arichardson at FreeBSD.org>
Contact: Andrew Turner <andrew at FreeBSD.org>
Contact: Brooks Davis <brooks at FreeBSD.org>
Contact: Edward Tomasz Napierala <trasz at FreeBSD.org>
Contact: George Neville-Neil <gnn at FreeBSD.org>
Contact: Jessica Clarke <jrtc27 at FreeBSD.org>
Contact: John Baldwin <jhb at FreeBSD.org>
Contact: Robert Watson <rwatson at FreeBSD.org>
Contact: Ruslan Bukin <br at FreeBSD.org>
CheriBSD extends FreeBSD to implement memory protection and software
compartmentalization features supported by the CHERI instruction-set
extensions. There are three architectural implementations of the CHERI
protection model: CHERI-MIPS, CHERI-RISC-V, and Arm's forthcoming
experimental Morello processor (due late 2021). CheriBSD is a research
operating system with a stable baseline implementation into which
various new research features have been, or are currently being,
merged:
* Arm Morello - We are preparing to open source our adaptation of
CheriBSD to Arm's Morello architecture. The Morello branch is being
updated to the most recent CheriBSD baseline, and patches are in
review for upstreaming to our open-source repository. CheriBSD
currently boots and runs statically linked CheriABI binaries on the
Morello simulator, and dynamic linking support is in progress, with
OS and toolchain bugs being worked on. We aim to make a first
CheriBSD/Morello snapshot available alongside other open-source
Morello software in mid-October 2020, however, our target for a
more mature and usable implementation is December 2020.
* Kernel spatial memory safety (pure-capability kernel) - The current
CheriBSD kernel is a hybrid C program where only pointers to
userspace are CHERI capabilities. This ensures that the kernel
follows the intent of the application runtime and cannot be used to
defeat bounds on application pointers. We have developed and will
soon merge a pure-capability kernel where all pointers in the
kernel are appropriately bounded capabilities. This vastly reduces
the opportunity for buffer overflows. This spatial memory safety
lays the groundwork for future work such as device driver
compartmentalization and kernel temporal safety.
* Userspace heap temporal memory safety (Cornucopia) - CHERI
capabilities provide the necessary features to enable robust and
efficient revocation of freed pointers. With Cornucopia we have
implemented a light-weight revocation framework providing
protection from use-after-reallocation bugs with an average cost
below 2%. We aim to bring these overheads down further over the
next year and merge this functionality into the mainline CheriBSD.
* We have been working on updating the arm64 bhyve from Politehnica
University of Bucharest to have it committed to FreeBSD. We have
been upstreaming initial changes to help support this.
* Baseline FreeBSD improvements - We are upstreaming (to FreeBSD)
various bug fixes and tweaks for PCIe support, and support for the
System MMU (SMMU) that will be present on the N1SDP and Morello
SoCs. We have upstreamed support for cross-building FreeBSD from
macOS and Linux (with some limitations; see separate entry on
crossbuilding). We have also fixed implementation bugs in the
RISC-V ABI.
CHERI Documentation and Exercises
* We have released [Capability Hardware Enhanced RISC Instructions:
CHERI Instruction-Set Architecture (Version
8)](https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-951.pdf).
Notable changes include promotion of CHERI-RISC-V to
non-experimental and discussion of Arm's Morello prototype.
* We have developed a set of [Adversarial CHERI Exercises and
Missions](https://ctsrd-cheri.github.io/cheri-exercises) to
introduce security researchers to CHERI protections.
__________________________________________________________________
FreeBSD/RISC-V Project
Links
Wiki
URL: https://wiki.freebsd.org/riscv
Contact: Mitchell Horne <mhorne at FreeBSD.org>
Contact: freebsd-riscv Mailing List
Contact: IRC #freebsd-riscv on freenode
The FreeBSD/RISC-V project is providing support for running FreeBSD on
the RISC-V Instruction Set Architecture.
This quarter saw several important bug fixes. A number of hangs in the
system were identified and addressed, and a bug in QEMU's
implementation of the Platform Level Interrupt Controller was fixed.
This fix is included in the new devel/qemu50 and devel/qemu-devel
ports.
The end result of these fixes is that the test suite can now be
reliably run to completion in QEMU. The entire run takes several hours,
so CI has been configured to run the job once a day. There is active
effort into reducing the time it takes to run the entire test suite.
A new u-boot port was created: sysutils/u-boot-qemu-riscv64. This
variant can be used as a secondary bootloader alongside OpenSBI to load
and launch FreeBSD's loader(8) from an EFI System Partition.
Next quarter will likely bring further fixes to address some of the
failing test cases.
__________________________________________________________________
Ports
Changes affecting the Ports Collection, whether sweeping changes that
touch most of the tree, or individual ports themselves.
Update to grub-bhyve
Links
grub-bhyve Git Repository
URL: https://gitlab.com/ctuffli/grub
Contact: Chuck Tuffli <chuck at freebsd.org>
bhyve is the hypervisor included in FreeBSD and other operating systems
used to run virtual machines. When not using a boot ROM (i.e. UEFI),
the user must load the guest operating system for bhyve. For
non-FreeBSD guests, the loader is a version of GNU GRUB (a.k.a the GNU
GRand Unified Bootloader) modified to interface with bhyve. This work
is an effort to both update the base GRUB code to the latest version as
well as improve the usability on FreeBSD.
The current grub-bhyve is based on an older version of GRUB (circa
2015) and thus is missing more recent additions such as XFS file system
and syslinux support. With the update, installing CentOS, for example,
now does not require the extra step of changing the default file system
to something other than XFS.
Internally, the code has been restructured to be its own "platform"
which should make it easier to keep in sync with upstream development.
The major improvement is the ability to automatically find and load the
GRUB configuration file from the guest disk image. With this change, it
is not necessary to create a device map file or specify which Linux
kernel or initrd image to use. More importantly, if the guest image
updates its GRUB configuration, for example after updating the kernel,
no changes are needed when invoking grub-bhyve. Note, this feature
requires a new "disk" option:
# grub-bhyve --disk=/zroot/vms/u18-mini/disk0.img --vm=u18-mini
The automatic configuration file detection works with both GRUB
configuration files (e.g. CentOS, Ubuntu) as well as syslinux
configuration (e.g. Alpine). For the adventurous, there is experimental
support for Fedora's BootLoaderSpec (a.k.a. blscfg) on the blscfg
branch of the grub-bhyve Git repository.
The code has been tested on a few Linux variants, but it would benefit
from wider testing (and bug reports!). The new version does not have a
Port but is easily built on FreeBSD. After cloning / downloading the
source, run:
$ PYTHON=python3.7 ./bootstrap $ MAKE=gmake ./configure
--with-platform=bhyve $ gmake
The resulting binary, grub-bhyve, will be in the grub-core/ directory.
If you have success or troubles with it, please let me know.
__________________________________________________________________
KDE on FreeBSD
Links
KDE FreeBSD
URL: https://freebsd.kde.org/
KDE Community FreeBSD
URL: https://community.kde.org/FreeBSD
Contact: Adriaan de Groot <kde at FreeBSD.org>
The KDE on FreeBSD project aims to package all of the software produced
by the KDE Community for the FreeBSD ports tree. The software includes
a full desktop environment called KDE Plasma, an IDE with the name
KDevelop, a PIM suite known as Kontact and hundreds of other
applications that can be used on any FreeBSD machine.
With the continuation of the ever-so-peculiar era of
almost-only-online, the KDE community has shifted gears and also gone
for online events. The yearly conference, Akademy, was conducted online
over video calls. Meanwhile, software continues to be released, so this
quarter the kde@ team:
* Put the beta of the next version of KDE Plasma, scheduled for
official release in October 2020, into the Area51 development tree.
Area51 is a fork of the FreeBSD ports tree where new development
for KDE ports happens.
* The monthly regular updates to the KDE Plasma desktop landed
on-time and safely.
* With three months in a quarter, there were also three releases of
KDE Frameworks 5, including a new framework for handling DAV jobs.
* The June applications update and its .1 release landed a bit late,
but brings with it the usual raft of updates to KDE applications
and libraries,
* A new Digikam release, which arrived in the ports tree on the day
of its release.
* A new KDevelop release arrived a day after its release. This update
contains a number of crash fixes for refactoring support.
* Qt was updated to Qt 5.15, the last in the Qt5 series and an LTS
version. Bugfix releases are expected, but the next major Qt will
be Qt 6.
On the infrastructure front, August saw some minor updates to CMake and
ninja.
As usual, kde@ continues to support the work of xorg@ and gnome@ in
maintaining the Free Desktop stack on FreeBSD, including XOrg, poppler,
and xdg-utils. A new MAINTAINER group, desktop@, has been created to
provide shared ownership of that shared stack.
With Python2 deprecation looming, the build system for QtWebEngine --
itself a fork of Chromium -- is becoming a pressing issue in Q4 and
will no doubt chew up a lot of time in the coming months.
__________________________________________________________________
Documentation
Noteworthy changes in the documentation tree, in manpages, or in
external books/documents.
DOCNG on FreeBSD
Links
DOCNG Website Repo
URL: https://gitlab.com/carlavilla/freebsd-hugo-website
DOCNG Documentation Repo
URL: https://gitlab.com/carlavilla/freebsd-hugo-documentation
DOCNG Share Repo
URL: https://gitlab.com/carlavilla/freebsd-hugo-data
Contact: Sergio Carlavilla <carlavilla at FreeBSD.org>
The Doc New Generation project aims to convert the website and all
existing documentation to Hugo/AsciiDoctor. Right now almost everything
is converted as you can see in the repositories.
The objective of using Hugo and AsciiDoctor is to reduce the learning
curve and let people to start quickly with our documentation system.
Other benefits of using Hugo is that we can use other technologies
aside from AsciiDoctor, like MarkDown, RST, Pandoc, etc.
The remaining tasks include:
* Finish the conversion of some books to AsciiDoctor.
* Get some tweaks in the CSS to be responsive.
* Add AsciiDoctor extensions to create an index of tables and
figures.
* Make a general review.
The dates for making the migration have yet to be discussed.
Patches, comments and objections are always welcome.
__________________________________________________________________
Third-Party Projects
Many projects build upon FreeBSD or incorporate components of FreeBSD
into their project. As these projects may be of interest to the broader
FreeBSD community, we sometimes include brief updates submitted by
these projects in our quarterly report. The FreeBSD project makes no
representation as to the accuracy or veracity of any claims in these
submissions.
Potluck - Flavour & Image Repository for pot
Links
Potluck Repository & Project
URL: https://potluck.honeyguide.net/
Potluck on github
URL: https://github.com/hny-gd/potluck
pot project
URL: https://pot.pizzamig.dev
Contact: <>
pot is a jail management tool that also supports orchestration through
nomad.
Potluck aims to be to FreeBSD and pot what Dockerhub is to Linux and
Docker: A repository of pot flavours and complete images for usage with
pot.
In the last quarter, an initial set of Nomad, Consul and Traefik images
has been created that are sufficient to run a simple virtual datacenter
out of the box.
A three-part article series explaining how to set this up is also
available now.
Furthermore, ready-made images suitable for scheduling via Nomad and
Consul in such an environment have been created, e.g. a BackupPC or a
Postfix Backup MX service.
Future plans include additional images and exposing more configuration
options in the existing images to allow a more flexible usage.
Beside general feedback and tests, additional flavours and patches are
very welcome!
Sponsors: Honeyguide GmbH & Honeyguide Group (Pty) Ltd ## Puppet
Puppet Puppet's FreeBSD slack channel Bolt Choria Puppet Team
puppet at FreeBSD.org
Since out last status report a few years ago, the puppet@ team
regularly updated the various Puppet ports to follow upstream releases
of Puppet 4, Puppet 5 and Puppet 6. Puppet 4 was removed when it
reached EOL.
More recently, an effort was made to enhance Facter 4 so that it can be
used as a drop-in replacement of Facter 3 on FreeBSD. Facter 4 is a
Ruby rewrite of Facter 3, the C++ rewrite of Facter 2 which was
initially in Ruby. As a consequence we have two ports for Facter:
sysutils/facter is the C++ implementation (Facter 3) and
sysutils/rubygems-facter is the Ruby implementation (updated from
Facter 2 to Facter 4 a few weeks ago). The Puppet 5 and Puppet 6 ports
already allow to choose which version of Facter to use. Facter 4 will
be the default version of Facter with Puppet 7 which is expected to be
released soon.
We are getting ready to add a port for Puppet 7 as sysutils/puppet7
when it is available, along with PuppetServer 7
(sysutils/puppetserver7), and PuppetDB 7 (databases/puppetdb7).
Regarding orchestration, most Marionette Collective ports have been
deprecated for a long time, and the last component sysutils/mcollective
is expected to be deprecated soon: Marionette Collective was not
shipped anymore with Puppet 6 and Bolt has been made available as a
lightweight replacement.
Bolt is already available in the ports tree as sysutils/rubygems-bolt),
but if you are using Marionette Collective, you are invited to look
into Choria which will reach the ports tree soon as sysutils/choria.
Choria is a direct evolution of Marionette Collective allowing a smooth
transition from MCollective. Once Choria is available in the ports
tree, Marionette Collective will be deprecated.
__________________________________________________________________
News Home | Status Home
Site Map | Legal Notices | ?? 1995-2020 The FreeBSD Project. All rights
reserved.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 618 bytes
Desc: not available
URL:
<http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20201022/0d990700/attachment-0001.sig>