Daniel Ebdrup Jensen
2021-Jul-24 07:07 UTC
[FreeBSD-Announce] FreeBSD Quarterly Status Report - Second Quarter 2021
Introduction This report covers FreeBSD related projects for the period between April and June, and is the second of four planned reports for 2021. Some of this reports highlights include but are not limited to work on an experimental installer, changes to pf, additional work on the Linuxulator, updates on the state of kernel sanitizers, coverage of the raidz expansion feature for ZFS, and some news about resource accounting. Yours, Daniel Ebdrup Jensen, with status hat on. ??????????????????????????????????????????????????????????????????????????????? Table of Contents ? FreeBSD Team Reports ? FreeBSD Foundation ? FreeBSD Release Engineering Team ? Cluster Administration Team ? Continuous Integration ? Ports Collection ? Documentation Engineering Team ? Projects ? BSDDialog - TUI Widgets ? Experimental installer ? Git Migration Working Group ? LLDB Debugger Improvements ? Performance Monitoring Counters ? syzkaller on FreeBSD ? Kernel ? NXP DPAA driver ? ENA FreeBSD Driver Update ? Graphics Driver Update from Linux 5.7 ? Intel Wireless driver support ? Kernel Sanitizers ? Linux compatibility layer update ? NXP LS1028A SoC support ? Marvell ARM64 SoCs support ? Multicast routing rework ? VFS path descriptors API ? Dummynet support for pf ? Ethernet support for pf ? pf syncookie support ? Microchip PolarFire SoC support ? Racct (Resource Accounting) Bug Fixes and Improvements ? OpenZFS RAIDZ Expansion update ? Microchip Switchtec support ? Ports ? Emacs Ports ? FreeBSD Erlang Ecosystem Ports update ? GCC in the Ports Tree (lang/gcc*) ? KDE on FreeBSD ? libglvnd on FreeBSD ? FreeBSD in Science ? Documentation ? FreeBSD Translations on Weblate ? Miscellaneous ? Third-Party Projects ? FreshPorts ? helloSystem ? Containers & FreeBSD: Pot, Potluck & Potman ??????????????????????????????????????????????????????????????????????????????? FreeBSD Team Reports Entries from the various official and semi-official teams, as found in the Administration Page. FreeBSD Foundation Contact: Deb Goodkin <deb at FreeBSDFoundation.org> The FreeBSD Foundation is a 501(c)(3) non-profit organization dedicated to supporting and promoting the FreeBSD Project and community worldwide. Funding comes from individual and corporate donations and is used to fund and manage software development projects, conferences and developer summits, and provide travel grants to FreeBSD contributors. The Foundation purchases and supports hardware to improve and maintain FreeBSD infrastructure and provides resources to improve security, quality assurance, and software engineering efforts; publishes marketing material to promote, educate, and advocate for the FreeBSD Project; facilitates collaboration between commercial vendors and FreeBSD developers; and finally, represents the FreeBSD Project in executing contracts, license agreements, and other legal arrangements that require a recognized legal entity. Here are some highlights of what we did to help FreeBSD last quarter: COVID-19 Impact to the Foundation Like most organizations, our team continued to work from home. Our temporary ban on travel for staff members remains in effect, but continues to not affect our output too much, since most conferences are still virtual. As the world continues to open up we will re-evaluate the travel ban. We continued supporting the community and Project through our regular channels. Partnerships and Commercial User Support We help facilitate collaboration between commercial users and FreeBSD developers. We also meet with companies to discuss their needs and bring that information back to the Project. Our temporary travel ban stayed in effect during Q2, so we continued meeting with corporate users virtually. If things look good for in-person meetings in the fall, then we?ll start incorporating those into an in-person and virtual meeting mix. Fundraising Efforts First, we?d like to say thank you to everyone who has given us a financial contribution this year! Last quarter we raised $70,410, which includes donations from organizations like Verisign, VMware, and Stormshield, as well as many individuals. We depend on these donations to fund our work supporting FreeBSD. Late last year we decided to put more funding into helping to improve FreeBSD. We hired a Sr. Software Developer to work on arm64 and a Project Coordinator to manage our projects and interface with the community. We also hired two of our part-time software developers full-time. The purpose of this was to provide more resources to step in to implement and improve major features in FreeBSD, review patches and bug reports, implement bug fixes, and support the security efforts. This ensures FreeBSD remains the innovative, secure, and reliable operating system that you rely on. You?ll find out how we used your donations for Q2 in our report, as well as individual reports throughout this status report. We are excited about our plans for 2021, which include more FreeBSD online advocacy and training, operating system course content, and the software development work mentioned above. While we are still in this pandemic, we?re working hard to help connect folks within the community with more virtual opportunities. Please consider making a donation to help us continue and increase our support for FreeBSD in 2021: https://www.freebsdfoundation.org/donate/. We also have the Partnership Program, to provide more benefits for our larger commercial donors. Find out more information and share with your companies! https://www.freebsdfoundation.org/FreeBSD-foundation-partnership-program/ OS Improvements During the second quarter Foundation staff and grant recipients committed 348 src tree changes, 19 ports tree changes, and 11 doc tree changes. This represents 40% of src commits which identify a sponsor. For ports commits it?s 15%, and 18% of doc commits. Foundation staff and grant recipients also contributed to a number of 3rd party projects. Two notable examples are the LLVM project?s LLDB debugger and the Syzkaller code-coverage-guided system call fuzzer. You can read about a number of Foundation projects in individual quarterly reports. Smaller projects and improvements include: ? Implement on-demand coredump generation by the kernel via ptrace (PT_COREDUMP) ? General kernel debugging improvements ? Remove obsolete kernel mcount profiling ? Nullfs and tmpfs bug fixes ? libc cleanup and improvements ? rtld dlerror and thread local variable fixes (reported by Julia developers) ? kqueue and POSIX timer fixes ? UFS bug fixes ? Capsicum socket operation improvements ? hwpmc (hardware performance profiling) maintenance and CPU support ? Cirrus-CI boot smoke test ? sndstat(4) schema updates ? AMD PCI passthrough fixes in vmm(4), see: commit 1, commit 2 and review ? Virtio 1.0 modern support in bhyve(8) As usual Foundation staff also supported the project with significant effort on code reviews and general bug triage and fixes. Also, Ka Ho added an article titled Use Language Servers for Development in the FreeBSD Src Tree. Continuous Integration and Quality Assurance The Foundation provides a full-time staff member and funds projects on improving continuous integration, automated testing, and overall quality assurance efforts for the FreeBSD project. During the second quarter of 2021, work on pre-commit tests and building release artifacts in the CI environment continued. A project using the netperf cluster to do network-related CI jobs is being planned. See the FreeBSD CI section of this report for completed work items and detailed information. Supporting FreeBSD Infrastructure The Foundation provides hardware and support to improve the FreeBSD infrastructure. Last quarter, we supported the test cluster at Sentex, purchased a few needed parts for infrastructure in general, and started working with the clusteradm team on a more efficient and improved hardware request/ purchase process. FreeBSD Advocacy and Education A large part of our efforts are dedicated to advocating for the Project. This includes promoting work being done by others with FreeBSD; producing advocacy literature to teach people about FreeBSD and help make the path to starting using FreeBSD or contributing to the Project easier; and attending and getting other FreeBSD contributors to volunteer to run FreeBSD events, staff FreeBSD tables, and give FreeBSD presentations. The FreeBSD Foundation sponsors many conferences, events, and summits around the globe. These events can be BSD-related, open source, or technology events geared towards underrepresented groups. We support the FreeBSD-focused events to help provide a venue for sharing knowledge, to work together on projects, and to facilitate collaboration between developers and commercial users. This all helps provide a healthy ecosystem. We support the non-FreeBSD events to promote and raise awareness of FreeBSD, to increase the use of FreeBSD in different applications, and to recruit more contributors to the Project. While we were still unable to attend in-person meetings due to Covid-19, we were able to attend virtual events and help organize the June 2021 FreeBSD Developer Summit. In addition to attending and planning virtual events, we are continually working on new training initiatives and updating our selection of how-to guides to facilitate getting more folks to try out FreeBSD. Check out some of the advocacy and education work we did last quarter: ? Participated as an Industry Partner for USENIX LISA21 ? Held two new FreeBSD Fridays: Introduction to BastilleBSD and How to Submit a Patch to FreeBSD. ? Organized content and promoted FreeBSD Day on June 18-19, 2021 ? Helped organize and run the June 2021 FreeBSD Developer Summit - videos are now posted on the wiki. ? Presented at the The 16th Open Source China Open Source World Summit on June 18 ? New blog posts on the Linxulator work we funded and What?s new in FreeBSD 13.0 ? New How To Guide on Git ? Continued to promote the FreeBSD Office Hours series Videos from the one hour sessions can be found on the Project?s YouTube Channel. See the Office Hours section of this report for more information. ? Committed to be a Silver Sponsor for EuroBSDcon Keep up to date with our latest work in our newsletters: https:// www.freebsdfoundation.org/news-and-events/newsletter/ We help educate the world about FreeBSD by publishing the professionally produced FreeBSD Journal. As we mentioned previously, the FreeBSD Journal is now a free publication. Find out more and access the latest issues at https:// www.FreeBSDfoundation.org/journal/. You can find out more about events we attended and upcoming events at https:// www.FreeBSDfoundation.org/news-and-events/. Legal/FreeBSD IP The Foundation owns the FreeBSD trademarks, and it is our responsibility to protect them. We also provide legal support for the core team to investigate questions that arise. Go to https://www.FreeBSDfoundation.org to find out how we support FreeBSD and how we can help you! ??????????????????????????????????????????????????????????????????????????????? FreeBSD Release Engineering Team Links: FreeBSD 13.0-RELEASE schedule URL: https://www.freebsd.org/releases/13.0R/ schedule/ FreeBSD 13.0-RELEASE announcement URL: https://www.freebsd.org/releases/13.0R/ announce/ FreeBSD releases URL: https://download.freebsd.org/ftp/releases/ISO-IMAGES/ FreeBSD development snapshots URL: https://download.freebsd.org/ftp/snapshots/ ISO-IMAGES/ Contact: FreeBSD Release Engineering Team, <re at FreeBSD.org> The FreeBSD Release Engineering Team is responsible for setting and publishing release schedules for official project releases of FreeBSD, announcing code freezes and maintaining the respective branches, among other things. During the second quarter of 2021, the Release Engineering Team completed work on the 13.0-RELEASE cycle, the first release from the stable/13 branch. During the release cycle, one additional BETA build and two additional RC builds were added to the schedule, however the release cycle went smoothly, overall. Additionally throughout the quarter, several development snapshots builds were released for the main, stable/13, and stable/12 branches. Development snapshot builds for stable/11 will no longer be available. Thank you to all that have helped test the 13.0 builds up until this point and have reported issues. As always, we strive for quality over quantity. Sponsor: Rubicon Communications, LLC ("Netgate") Sponsor: The FreeBSD Foundation ??????????????????????????????????????????????????????????????????????????????? Cluster Administration Team Contact: Cluster Administration Team <clusteradm at FreeBSD.org> Links: Cluster Administration Team members URL: https://www.freebsd.org/administration /#t-clusteradm The FreeBSD Cluster Administration Team consists of the people responsible for administering the machines that the Project relies on for its distributed work and communications to be synchronised. In this quarter, the team has worked on the following: ? Moved Phabricator (reviews.freebsd.org) to a faster machine ? Moved https://www.freebsd.org CGI backend to a faster machine ? Improved the network design of our primary cluster site ? Removed public IPv4 connectivity from VLANs not hosting public-facing services ? Tidied up the pkgbuild and pkgexp VLANs where the production and experimental package builders live. ? Moved developer reference machines and universe building machines to a different VLAN to better accommodate aarch64 and ppc64 machines ? Upgraded the machines running important internal FreeBSD.org services ? DNS, Kerberos, LDAP, LetsEncrypt.org certbot, etc ? FTP, Subversion, Git mirror seed ? Upgraded the developer reference machines and universe builders to a newer FreeBSD 14-CURRENT ? Upgraded package building machines to newer versions of FreeBSD 14-CURRENT ? Assisted postmaster with migrating the mailing lists from Mailman to mlmmj ? Recycled a particularly troublesome PowerPC64 package building machine with extreme prejudice (rest in pieces) ? Installed a new production PowerPC64 package builder ? Worked with Git migration working group for ports tree migration ? Ongoing day to day cluster management activity ? Putting out fires ? Babysitting pkgsync Work in progress: ? Move pkg-master.nyi to new hardware ? Improve to the package building infrastructure ? Review the service jails and service administrators operation ? Setup powerpc pkgbuilder/ref/universal machines ? Search for more providers that can fit the requirements for a generic mirrored layout or a tiny mirror ? Upgrading public-facing cluster services from 12.2-STABLE to 13.0-STABLE ? Installing a new cluster site in Japan ? Full mirror site (ftp, pkg, git, doc, dns,??) ? The network and machine resources for this mirror are generously sponsored by the Cloud and SDN Laboratory at BroadBand Tower, Inc., one of the Internet data center service providers in Tokyo, Japan, with 300+ Gbps international IP transit bandwidth ? Improvements to GeoDNS routing, particularly in Asia ? Working with doceng@ to improve https://www.freebsd.org and https:// docs.freebsd.org ? Improve the web service architecture ? Improve the cluster backup plan ??????????????????????????????????????????????????????????????????????????????? Continuous Integration Links: FreeBSD Jenkins Instance URL: https://ci.FreeBSD.org FreeBSD Hardware Testing Lab URL: https://ci.FreeBSD.org/hwlab FreeBSD CI artifact archive URL: https://artifact.ci.FreeBSD.org FreeBSD CI weekly report URL: https://hackmd.io/@FreeBSD-CI FreeBSD Jenkins wiki URL: https://wiki.freebsd.org/Jenkins Hosted CI wiki URL: https://wiki.freebsd.org/HostedCI 3rd Party Software CI URL: https://wiki.freebsd.org/3rdPartySoftwareCI Tickets related to freebsd-testing@ URL: https://preview.tinyurl.com/y9maauwg FreeBSD CI Repository URL: https://github.com/freebsd/freebsd-ci Contact: Jenkins Admin <jenkins-admin at FreeBSD.org> Contact: Li-Wen Hsu <lwhsu at FreeBSD.org> Contact: freebsd-testing Mailing List Contact: IRC #freebsd-ci channel on EFNet The FreeBSD CI team maintains the continuous integration system of the FreeBSD project. The CI system firstly checks the committed changes can be successfully built, then performs various tests and analysis over the newly built results. The artifacts from those builds are archived in the artifact server for further testing and debugging needs. The CI team members examine the failing builds and unstable tests and work with the experts in that area to fix the code or adjust test infrastructure. The details of these efforts are available in the weekly CI reports. During the second quarter of 2021, we continued working with the contributors and developers in the project to fulfil their testing needs and also keep collaborating with external projects and companies to improve their products and FreeBSD. Important changes: ? The build environment of main (-CURRENT) and stable/13 branches are changed to 13.0-RELEASE Retired jobs: ? GCC 6 build for main on amd64 Work in progress and open tasks: ? Designing and implementing pre-commit CI building and testing ? Designing and implementing use of CI cluster to build release artifacts as release engineering does ? Collecting and sorting CI tasks and ideas here ? Testing and merging pull requests in the FreeBSD-ci repo ? Reducing the procedures of CI/test environment setting up for contributors and developers ? Setting up the CI stage environment and putting the experimental jobs on it ? Setting up public network access for the VM guest running tests ? Implementing using bare metal hardware to run test suites ? Adding drm ports building tests against -CURRENT ? Planning to run ztest and network stack tests ? Adding more external toolchain related jobs ? Improving the hardware lab to be more mature and adding more hardware ? Helping more software get FreeBSD support in its CI pipeline (Wiki pages: 3rdPartySoftwareCI, HostedCI) ? Working with hosted CI providers to have better FreeBSD support Please see freebsd-testing@ related tickets for more WIP information, and don?t hesitate to join the effort! Sponsor: The FreeBSD Foundation ??????????????????????????????????????????????????????????????????????????????? Ports Collection Links: About FreeBSD Ports URL:https://www.FreeBSD.org/ports/ Contributing to Ports URL: https://docs.freebsd.org/en/articles/contributing/ ports-contributing/ FreeBSD Ports Monitoring URL: http://portsmon.freebsd.org/ Ports Management Team URL: https://www.freebsd.org/portmgr/ Ports Tarball URL: http://ftp.freebsd.org/pub/FreeBSD/ports/ports/ Contact: Ren? Ladan <portmgr-secretary at FreeBSD.org> Contact: FreeBSD Ports Management Team <portmgr at FreeBSD.org> The Ports Management Team is responsible for overseeing the overall direction of the Ports Tree, building packages, and personnel matters. Below is what happened in the last quarter. According to FreshPorts, we currently enjoy a little over 44,200 ports in the Ports Collection. These are accompanied by 2,532 PRs of which 526 are unassigned. During the last quarter, there were 10,428 commits on main by 164 committers and 618 commits on 2021Q2 by 59 committers. Compared to 2021Q1, the number of ports and PRs remained roughly the same and the main tree saw 10% more commits this quarter. During the last quarter, we welcomed Charlie Li (vishwin@) and Guangyuan Yang (ygy@) and said goodbye to jlaffaye@, jpaetzel@, kmoore@, lifanov@, miwi@ and shurd at . On the infrastructure side, a new USES, ansible, was added so that Ansible-related ports can more easily define plugins and modules. Several default versions were updated: ? Default version of PYTHON and PYTHON3 switched to 3.8 ? Default version of librsvg2 on PowerPC switched to rust Regarding the license framework, the badly maintained LEGAL was removed in favor of per-port LICENSEs and the BSD0CLAUSE license was added. In the options framework, pre-defined shared version control OPTIONS and descriptions were added. Some notable port updates: ? Firefox 89.0.2 ? Firefox-esr 78.11.0 ? Chromium 91.0.4472.114 ? Ruby 3.0.1 ? Xorg server 1.20.11 Finally, antoine@ ran 18 exp-runs to test updates for cmake, expat2, KDE, meson, poppler, rust, Xorg and the lapack family, to switch the default version of Python to 3.8, and to remove the outdated qtchooser. ??????????????????????????????????????????????????????????????????????????????? Documentation Engineering Team Links: FreeBSD Documentation Project URL: https://www.freebsd.org/docproj/ FreeBSD Documentation Project Primer for New Contributors URL: https:// docs.freebsd.org/en/books/fdp-primer/ Documentation Engineering Team URL: https://www.freebsd.org/administration/# t-doceng Contact: FreeBSD Doceng Team <doceng at FreeBSD.org> The doceng@ team is a body to handle some of the meta-project issues associated with the FreeBSD Documentation Project, more infomation on FreeBSD Doceng Team Charter. During the last quarter, we welcomed back Ceri Davies (ceri@) as a doc committer. Sergio Carlavilla (carlavilla@) and Danilo G. Baio (dbaio@) joined the doceng@ team. A new article was included in the FreeBSD Documentation by Ka Ho Ng (khng@), Use Language Servers for Development in the FreeBSD Src Tree. Much work was done regarding the transition to Hugo/Asciidoctor, but there are still pending issues in the FreeBSD Documentation Project after the migration. We thank everyone who is helping to find and fixing them. The TODO items wiki list was refreshed, and everyone is welcome to contribute to it. ??????????????????????????????????????????????????????????????????????????????? Projects Projects that span multiple categories, from the kernel and userspace to the Ports Collection or external projects. BSDDialog - TUI Widgets Links: BSDDialog Project URL: https://gitlab.com/alfix/bsddialog Dialog Project URL: https://invisible-island.net/dialog GPLinBase Wiki Page URL: https://wiki.freebsd.org/GPLinBase Contact: Alfonso Sabato Siciliano <alfonso.siciliano at email.com> The purpose of this project is to provide an utility and a library to build scripts and tools with UI Widgets in a terminal. The project is inspired by Dialog; however BSDDialog is released under the terms of the BSD-2-Clause License while Dialog is licensed as LGPL so a link of the project has been added to the "GPL Software in the Base System" wiki page. The aim is to provide full compatibility with the dialog utility (the challenge is to implement over seventy options and almost thirty widgets). The API compatibility with the library is not a priority because libbsddialog should meet FreeBSD?s needs, for example it provides the new mixedlist() function, it can take as argument a set of: checklists, radiolists and separators, so it could be useful for building a dialog4ports clone: easier to implement and not depending on non-permissive dependencies. BSDDialog is currently under development, no widget is really completed (autosizing, resizing and scrolling are missing), nevertheless runnable examples for the utility and the library are inside the example folders and described in the README. Finally, a curiosity: BSDDialog started from the MixerTUI code base, however the original code has been almost completely rewritten. ??????????????????????????????????????????????????????????????????????????????? Experimental installer Links: Installer repository URL: https://github.com/yangzhong-freebsd/lua-httpd Live ISO containing installer URL: https://github.com/yangzhong-freebsd/ISO Contact: Yang Zhong <yzhong at freebsdfoundation.org> bsdinstall is FreeBSD?s current installer. It is a terminal application with an unwieldy interface, and it asks some very obscure questions that are hard to understand for ordinary users. So, the purpose of this project is to create a graphical installer for FreeBSD that has a more streamlined interface, and to improve other aspects of the FreeBSD installation process. The experimental installer uses a web front-end: a web server runs locally from the installation media, and the user configures their install by filling out web forms on a browser also running on the installation media. A Web interface is flexible and accessible, so it suits an installer well. This interface can also support remote installs, where the server runs on the target and the install is configured through some other machine, though I have not done much work here. The installer also aims to have a modular design, where the user configuration options are written to an installation config file, that is then used for the actual installation. While bsdinstall already supports scripted installations, its config file format is very free-form. A more rigid config file design would make it easier to write other installation front-ends in the future. The installer is currently a rough proof-of-concept, but it can handle a basic installation with limited configuration. Help with testing would be appreciated; you can try the installer by downloading one of the releases in the ISO repository. Also, please email me with any thoughts on the design of the installer, or on useful features it should have. Sponsor: The FreeBSD Foundation ??????????????????????????????????????????????????????????????????????????????? Git Migration Working Group Links: Git for FreeBSD development wiki URL: https://wiki.freebsd.org/Git Git transition wiki URL: https://wiki.freebsd.org/GitTransition doc git repo web URL: https://cgit.FreeBSD.org/doc ports git repo web URL: https://cgit.FreeBSD.org/ports src (base system) git repo web URL: https://cgit.FreeBSD.org/src Committers guide Git primer URL: https://docs.freebsd.org/en/articles/ committers-guide/#git-primer Handbook Using Git appendix URL: https://docs.freebsd.org/en/books/handbook/ mirrors/#git Game of Trees URL: http://gameoftrees.org/ gitup URL: https://github.com/johnmehr/gitup Contact: Li-Wen Hsu <lwhsu at FreeBSD.org> Contact: Warner Losh <imp at FreeBSD.org> Contact: Ed Maste <emaste at FreeBSD.org> Contact: Ulrich Sp?rlein <uqs at FreeBSD.org> Contact: FreeBSD-git mailing list Contact IRC #gitcvt channel on EFnet The Git Working Group has been working on ports repository migration to Git, this task started at the end of the March, beginning with a final Subversion commit on March 31st to indicate that the conversion started. The whole migration completed on April 6th. Since 2021Q2, the ports quarterly branch is created in Git repository only. We continued working on portsnap and other ports infrastructure to accommodate git. We continued working on implementing and updating commit hooks. The work including helped change FreeBSD 13.0 release process to use Git. And we are sorting and making the infrastructure available to the public, as well as the documents. On June 8th, we worked with our ZFS developers to have better tracking of upstream OpenZFS development. The vendor/openzfs branch was renamed to vendor/ openzfs/legacy. Two new branches were imported directly from upstream, vendor/ openzfs/master and vendor/openzfs/zfs-2.1-release, and merged to main and stable/13. The details and the required action to correct the errors might result for the people tracking the old branch is available at https:// lists.freebsd.org/archives/freebsd-current/2021-June/000153.html The Git Working Group continues to track progress on two permissively-licensed git compatible tools: Gitup and Game of Trees. Gitup is a small, dependency-free tool to clone and update git repositories. It is used only to keep a local tree up-to-date, and has no support for local commits. Game of Trees is a version control client that is compatible with Git repositories. It provides a user interface and workflow that is distinct from that of Git. It is in no way intended to be a drop-in replacement for git, but can be used to develop software maintained in a Git repository. Gitup and Game of Trees are currently available as ports and packages. Future work will evaluate them as candidates for the base system. The core team began a new effort to investigate and evaluate new workflow changes in the June 2021 DevSummit. In the third quarter of 2021 we expect to complete the remaining migration tasks and create a new working group to help with workflow refresh. We?ve wound up our regular meetings, and the remaining migration tasks are being done by individuals (Li-Wen Hsu is mainly working on this). The new working group(s) will have people that participated in this working group as well as new people who will bring new perspectives to the process. Sponsor: The FreeBSD Foundation (in part) ??????????????????????????????????????????????????????????????????????????????? LLDB Debugger Improvements Links: Moritz Systems Project Description URL: https://www.moritz.systems/blog/ lldb-freebsd-cpu-target-support-and-userland-debugging-improvements/ Progress Report 1 URL: https://www.moritz.systems/blog/ freebsd-remote-process-plugin-on-non-x86-architectures/ Progress Report 2 URL: https://www.moritz.systems/blog/ freebsd-legacy-process-plugin-removed/ Progress Report 3 URL: https://www.moritz.systems/blog/ lldb-support-for-fork-and-vfork/ Progress Report 4 URL: link:https://www.moritz.systems/blog/ lldb-core-dump-support-improvements/ Git Repository URL: https://github.com/moritz-systems/llvm-project Contact: Kamil Rytarowski <kamil at moritz.systems> Contact: Micha? G?rny <mgorny at moritz.systems> The LLDB project builds on libraries provided by LLVM and Clang to provide a great modern debugger. It uses the Clang ASTs and the expression parser, LLVM JIT, LLVM disassembler, etc so that it provides an experience that ?just works?. It is also blazingly fast and more permissively licensed than GDB, the GNU Debugger. FreeBSD includes LLDB in the base system. At present, it has some limitations in comparison with the GNU GDB debugger, and does not yet provide a complete replacement. This project spanned from January 2021 to April 2021 and aimed to improve the compatibility of LLDB with FreeBSD and extend it with new features. The initial work was focused on finishing the port of the FreeBSD process plugin to a new client-server model. After porting all previously supported architectures we were able to remove the legacy plugin. Afterwards, we have implemented support for tracing fork(2) and vfork(2) syscalls using a model that is compatible with the follow-fork-mode setting from GDB. Finally, we have worked on improving support for core dumps in LLDB. We have used the newly introduced PT_COREDUMP ptrace(2) request to support creating a core dump of a stopped program without crashing it. During our work, we have fixed many bugs and improved the state of the test suite on amd64 and arm64 architectures. The introduced changes are expected to be shipped with LLDB 13.0. The overall experience of FreeBSD/LLDB developers and advanced users on this rock solid Operating System reached the state known from other environments. Furthermore, the FreeBSD-focused work also resulted in generic improvements, enhancing the LLDB support for Linux and NetBSD. Sponsor: The FreeBSD Foundation ??????????????????????????????????????????????????????????????????????????????? Performance Monitoring Counters Contact: Mitchell Horne <mhorne at FreeBSD.org> This quarter, some improvements were made to the pmc(3) library and hwpmc(4) kernel module. Background: Traditionally, PMC event codes have been defined manually in the sys/dev/hwpmc/pmc_events.h header and compiled statically into the kernel. These definitions provide a translation between an event?s name and its encoding and are leveraged by PMC tools like pmcstat(8) via the pmc(3) library. In 2018, a new source of truth for event definitions was introduced for the amd64 and i386 architectures, which is the pmu-events tables. These are a set of json files, maintained by the Linux kernel, which are compiled directly into libpmc and provide a more complete set of event definitions. The main result of this work was porting the use of the pmu-events definitions to arm64. Some minor refactoring to libpmc was performed, making porting to other platforms in the future easier. A few small bugs were found and fixed along the way; notably, the "instructions" alias for pmcstat(8) should work again on Intel CPUs. The ability was added to query the legacy event tables when pmu-events does not contain the desired event, restoring the ability to use the pmc.soft(3) events on x86. Finally, event definitions for newer AMD Zen3 CPUs were imported (credits: Greg V). An larger update of the PMU event definitions is planned for Q3 of 2021. This will bring us in sync with 3 years of updates for x86 CPUs and greatly increase the number of events available on arm64 platforms. Those who make regular or occasional use of FreeBSD?s PMC tools are encouraged to test their typical workflow and report any regressions. Sponsor: The FreeBSD Foundation ??????????????????????????????????????????????????????????????????????????????? syzkaller on FreeBSD Contact: Mark Johnston <markj at FreeBSD.org> Contact: Simran Kathpalia < skathpalia3 at gmail.com> See the syzkaller entry in the 2019q1 quarterly report for an introduction to syzkaller. Steady progress has been made on fixing bugs found by syzkaller, both by private instances and by the public syzbot instance. Bugs have been fixed in TCP, ktrace(2), kqueue(2), POSIX timers, fork(2), crypto(4), and in the VFS and UFS code. Work has also progressed on adding additional definition to syzkaller, so unfortunately the size of backlog remains more or less steady. Simran Kathpalia, a student participating in this year?s Google Summer Of Code (GSOC) program for FreeBSD, has made excellent progress in extending syzkaller to cover more of the FreeBSD kernel. She has added definitions for a number of system calls and has also been working on automating the generation of such definitions. Some work has been done towards making it easier to easier to set up syzkaller instances with minimal effort. For example, much of the work can be automated using a BastilleBSD template, which uses a jail to simplify configuration. While this example still requires some manual configuration and is not very flexible, it is much simpler than setting up an instance manually, and helps motivate some planned UI improvements and bug fixes for FreeBSD jails. Future work includes: ? Further effort to try and reduce the size of the syzkaller backlog. ? Completion of some proof-of-concept work to extend syzkaller to fuzz the Linuxulator subsystem. ? Additional system call and ioctl definitions. ? Automated fuzzing with kernel sanitizers enabled. Sponsor: The FreeBSD Foundation ??????????????????????????????????????????????????????????????????????????????? Kernel Updates to kernel subsystems/features, driver support, filesystems, and more. NXP DPAA driver Contact: Jakub Klama <jakub at conclusive.pl> The Conclusive Engineering team is working on support for the NXP Data Path Acceleration Architecture (DPAA) present in the NXP QorIQ Layerscape ARMv8 SoCs as well as PowerPC-based QorIQ SoCs. This work is focused on providing a clean-room, native implementation of DPAA driver and all of its associated components such as BMan, QMan, FMan and mEMAC for ARM targets such as LS1046A. First functional version of the driver with support for the 1Gbps and 10Gbps MACs should be completed in the 2021 Q3 timeframe. ??????????????????????????????????????????????????????????????????????????????? ENA FreeBSD Driver Update Links: ENA README URL: https://github.com/amzn/amzn-drivers/blob/master/kernel/fbsd/ ena/README Contact: Michal Krawczyk <mk at semihalf.com> Contact: Artur Rojek <ar at semihalf.com> Contact: Marcin Wojtas <mw at semihalf.com> ENA (Elastic Network Adapter) is the smart NIC available in the virtualized environment of Amazon Web Services (AWS). The ENA driver supports multiple transmit and receive queues and can handle up to 100 Gb/s of network traffic, depending on the instance type on which it is used. Completed since the last update: ? Update ENA driver to v2.4.0 ? Update ENA man page ? Restructure the logging system ? Hide sysctl nodes for unused IO queues ? Add support for the large LLQ headers ? Update HAL version Work in progress: ? Introduce full kernel RSS API support ? Allow reconfiguration of the RSS indirection table and hash key ? Prototype the driver port to the iflib framework ? MFC the ENA v2.4.0 driver to the FreeBSD 11/12/13-STABLE branches ? Support netmap on the c6gn/m6i AWS instance types ? Fix race between detach function and some of the procedural sysctl nodes ? Add new statistics counters Sponsor: Amazon.com Inc ??????????????????????????????????????????????????????????????????????????????? Graphics Driver Update from Linux 5.7 Links: WIP GitHub Repository URL: https://github.com/neelchauhan/drm-kmod/tree/ 5.7-wip Contact: Neel Chauhan <nc at FreeBSD.org> Contact: Hans Petter Selasky <hselasky at FreeBSD.org> Contact: Emmanuel Vadot <manu at FreeBSD.org> Contact: Mark Linimon <linimon at FreeBSD.org> We are attempting to update the drm-kmod graphics drivers to the Linux 5.7 code, based on the existing drm-kmod 5.5-wip branch. Right now, the current version of drm-kmod do not support newer GPU such as Intel?s Tiger Lake/Iris Xe, used on laptops such as the 2020 HP Spectre x360. This prevents us from supporting accelerated graphics on newer hardware containing Intel or AMD GPUs. Some tasks have already been completed, including: ? amdgpu bring-up ? i915kms console bring-up Some tasks need to be completed, including: ? i915kms Xorg bring-up (currently pagefaults on remap_io_sg() page address) ? amdgpu VA-API bring-up ? radeonkms bring-up We welcome help for the incomplete tasks, especially from users wanting to use newer hardware (or support FreeBSD-as-a-desktop in general). ??????????????????????????????????????????????????????????????????????????????? Intel Wireless driver support Links: iwlwifi status FreeBSD wiki page URL: https://wiki.freebsd.org/WiFi/Iwlwifi The Intel Wireless driver update project aims to bring support for newer chipsets along with mac80211 LinuxKPI compat code. The dual-licensed Intel driver code was ported in the past for the iwm(4) native driver and using the LinuxKPI compat framework allows us to use the driver directly with only very minor modifications. Multiple updates during development in the last year have shown that pulling in newer versions can be done in under 1-2 hours. After a break, part-time work resumed and the last LinuxKPI conflicts got resolved and most of the other LinuxKPI changes were committed to FreeBSD main. During the update process firmware crashes were unxpectedly encountered which got solved and the 802.11 compat code improved. The iwlwifi driver and firmware got updated from the iwlwifi-next git branch and the linux-firmware repository. Integration into FreeBSD main is pending for a policy reason. For the latest state of the development or code, please follow the referenced wiki page or the freebsd-wireless mailing list. I would love to thank everyone who has reviewed changes, did initial testing, helped with drm-kmod, helped in various other ways, answered questions, encouraged, or patiently waited for any results or running code. Sponsor: The FreeBSD Foundation ??????????????????????????????????????????????????????????????????????????????? Kernel Sanitizers Contact: Mark Johnston <markj at FreeBSD.org> Sanitizers are bug detection facilities which use a combination of instrumentation inserted by the compiler (LLVM in this case) and runtime state tracking to detect bugs in C code. They can automatically detect many types of C programming bugs, such as use-after-frees and uses of uninitialized variables, which may otherwise require substantial effort to identify. They are particularly effective in combination with regression testing suites or fuzzing tools such as syzkaller. Unlike tools such as Valgrind, software must be recompiled to enable a given sanitizer, but sanitizers can be used in the kernel. Kernels with sanitizers enabled incur a significant performance overhead from the runtime, in both CPU utilization and memory usage. Work has been ongoing to port a pair of kernel sanitizers from NetBSD to FreeBSD. These are the Kernel Address SANitizer (KASAN) and Kernel Memory SANitizer (KMSAN). The KASAN port is complete and has been committed to FreeBSD?s development branch, and a small LLVM patch to enable KASAN and KMSAN on FreeBSD has also been committed. KMSAN intercepts all memory accesses and verifies that they are valid using some hidden state saved for each memory allocation in the kernel; see kasan(9) for further details. It can be enabled in amd64 kernels by compiling the GENERIC-KASAN kernel configuration. The FreeBSD regression test suite currently passes with KASAN enabled. The KMSAN port is currently in progress and is nearing completion. KMSAN detects uses of uninitialized memory using the algorithms described in the original MemorySanitizer paper. In particular, it can detect instances of uninitialized kernel memory being copied out to userspace. Kernel subsystems may additionally call into the KMSAN runtime to verify the state of a given buffer. This can be used, for example, to add checks in the network stack for uninitialized bytes in egress packets. A number of bugs have been found using KMASN and the FreeBSD regression test suite; many have already been fixed (search for "KMSAN" in src commit logs for examples), and patches have been written for all others found so far. Future work includes: * Finishing the KMSAN port and committing it to the FreeBSD main branch. * Enabling CI jobs to run the test suite with KASAN and KMSAN enabled. * Adding syzbot configurations with KASAN and KMSAN enabled, and fixing bugs found it. * Reducing the scope of memory accesses which cannot be validated using KASAN or KMSAN today; this consists primarily of making use of the amd64 direct map optional in various parts of the kernel, and eliminating uses of UMA_ZONE_NOFREE. Sponsor: The FreeBSD Foundation ??????????????????????????????????????????????????????????????????????????????? Linux compatibility layer update Contact: Dmitry Chagin, <dchagin at FreeBSD.org> Contact: Edward Tomasz Napierala, <trasz at FreeBSD.org> The goal of this project is to improve FreeBSD?s ability to execute unmodified Linux binaries. Current support status of specific Linux applications is being tracked at the Linux app status Wiki page. The 32-bit syscalls which accept a timespec parameter now have their 64-bit timespec counterparts: clock_settime64(2), clock_gettime64(2), clock_nanosleep_time64(2), clock_getres_time64(2), futex_time64(2), pselect6_time64(2), rt_sigtimedwait_time64(2), utimensat_time64(2). The O_PATH flag to open(2) system call is now supported, as well as the AT_EMPTY_PATH flag to fstatat(2) (required for Qt5 applications in Ubuntu Focal), fchownat(2), linkat(2), and utimensat(2). The F_GETPIPE_SZ fcntl(2) is now supported. The ppoll(2) system call was reworked to fix POLLRDHUP semantics. The COMPAT_LINUX and COMPAT_LINUX32 kernel build options got removed; they were confusing and not quite functional. This doesn?t affect the usual setups which use Linuxulator loaded as a kernel module. The FreeBSD kernel can now generate coredumps for Linux processes, on both amd64 and arm64. The cores can be analyzed with Linux gdb(1). There were a number of fixes to ptrace(2) implementation; as a result, the Linux strace(1) doesn?t get confused with multithreaded programs. The kernel version was bumped to 4.4.0, as required for new Arch Linux binaries. There was a number of fixes to Linuxulator on arm64, ranging from ELF auxilliary vector (auxv) and Thread-Local Storage fixes to documentation improvements. There is ongoing work to make Linuxulator on arm64 on par with the amd64 one. There is also ongoing work on improving the vDSO functionality. The sysutils/debootstrap port, and its corresponding debootstrap package, now also work on arm64, not just x86. It is also much faster now. There have been some further improvements to the startup scripts. Sponsor: EPSRC ??????????????????????????????????????????????????????????????????????????????? NXP LS1028A SoC support Contact: Kornel Dul?ba <mindal at semihalf.com> Contact: Lukasz Hajec <lha at semihalf.com> Contact: Marcin Wojtas <mw at semihalf.com> The Semihalf team has been working on adding the FreeBSD support for the NXP LS1028A SoC. NXP LS1028A SoC is a dual-core 64-bit ARMv8 Cortex-A72 application processor with high-speed peripherals such as 2 Time-Sensitive Networking-capable (TSN) Ethernet controllers, quad-port TSN-enabled switch, PCIE, SD/MMC, USB3.0 and others. The newly introduced support includes: ? ENETC Ethernet and MDIO controllers drivers ? LS1028A support and other improvements in the FSL SDHCI driver ? LS1028A clockgen driver ? Generic PCI improvements: ? Add ofw interface support to PCI - commits 28c4e511c23f and 40429103cd0 ? Clean up and add proper support for mapping dts nodes to PCI devices in pci_host_generic_fdt - commits f0f7b0868a94 and ea52e815887b TODO: ? Upstream quad-port TSN switch support Sponsor: Alstom Group ??????????????????????????????????????????????????????????????????????????????? Marvell ARM64 SoCs support Contact: Zyta Szpak <zr at semihalf.com> Contact: Kornel Dul?ba <mindal at semihalf.com> Contact: Marcin Wojtas <mw at semihalf.com> The Semihalf team is working on improving the FreeBSD support for the Marvell Octeon TX2 CN913x and Armada 7k/8k SoC families. Marvell Armada 7k8k and Octeon TX2 CN913x SoC families are quad-core 64-bit ARMv8 Cortex-A72 processors with high speed peripherals including 10 Gb Ethernet, PCIe 3.0, SATA 3.0 and USB 3.0 for a wide range of networking, storage, security and industrial applications. Although the mentioned SoCs are mostly supported in FreeBSD HEAD, some pieces required improvements. Applied changes: ? Upstream PCIE improvements ? PCIE Designware driver (pci_dw) fixes: ? Correct setting of outbound I/O ATU window (commit 57dbb3c25936) ? Allow mapping ATU windows bigger than 4GB (commit 243000b19f8b) ? Generic improvements that enable proper user-space mapping and access of the PCI BARs - commits f2f1ab39c040 and 1c1ead9b94a1 ? SDHCI improvements ? 64-bit DMA operation (commit 7d8700bc291b) ? sdhci_xenon UHS support (commits 43e31350f8f6-4fa977f854e2) TODO: ? Improve and merge ICU support rework (D28803) Sponsor: Marvell ??????????????????????????????????????????????????????????????????????????????? Multicast routing rework Contact: Wojciech Macek <wma at semihalf.com> The Semihalf team has been working on reworking the existing ip_mroute module. The previous implementation was over 20 years old and didn?t use modern kernel features. A complete rework of locking mechanism was done to eliminate taking multiple locks for a single job. Some portions of code were modified to use lockless constructions, BW-meter API refactored to work in a single context and routing hot path was identified and made to work efficiently. All these fixes helped achieve over 5x speed boost in multicast routing. The newly introduced rework includes: ? mroute: fix race condition during mrouter shutting down race condition ? mrouter: do not loopback packets unconditionally loopback ? ip_mroute: refactor bw_meter API bw_meter ? ip_mroute: rework ip_mroute locking Sponsor: Stormshield ??????????????????????????????????????????????????????????????????????????????? VFS path descriptors API Contact: Konstantin Belousov <kib at FreeBSD.org> Unix VFS API, from a very high point of view, provides two kinds of interfaces. One is the path-based operations, typically manipulating metadata, like remove, rename, change permissions and similar. The second is the file-descriptor operations, typical are read, write, truncate, and other. The well-known open (2) syscall returns a file descriptor from the path. The file descriptor is a reliable handle for the file targeted by operations. Whatever other action was performed on the file meanwhile, rename, even delete, file descriptor still references the same file, and any operation done on it, affects that file. Until recently, there was no similar reliable way to take a handle on the file path, and then reliably use it for the operations of the first kind. Now, Linux introduced some facilities that make it possible, and FreeBSD tried to provide a compatible API. First, there is a way to get the file descriptor for a path. It is created with the open(2) syscall, when the O_PATH flag is specified. The returned descriptor cannot be used for normal file operations, it only designates a place in the filesystem namespace. Attempts to e.g. read or write using it results in EBADF. But the system also does not check the access rights on the target file while opening. It is enough for the process to read the content of the directory with the file name for open(O_PATH) to succeed. To actually use such descriptor, there is a new flag AT_EMPTY_PATH, understood by the \*at(2) set of syscalls. When supplied, and the path argument set to "" (empty path), \*at(2) syscalls operate on the file designated by the dirfd argument. It is not too exciting for e.g. fstatat(2), where already available fstat(2) operates on any kind of file descriptor. But consider linkat(2), which in combination with AT_EMPTY_PATH now allows to re-link any opened file descriptor to a named file (this requires root privilege). Other examples of available operations are chownat(2), chmodat(2), and others; see man pages. When a file descriptor and AT_EMPTY_PATH are supplied, the program knows for sure that the corresponding operation was done on the specific file, not subject to a race when our file was renamed, and some other file replaced it, providing a different file on the same path. With O_PATH, we get the same access rules as for normal chmod, not needing to have read or write permission for the file content. A way to translate the path descriptor to a normal file descriptor that can be read from or written to is to use openat(2) with O_EMPTY_PATH flag. The syscall checks the current access permissions and grants the requested mode by returning a normal descriptor. In Linux, there is no equivalent of the O_EMPTY_PATH flag. It seems, instead, that its implementation of our equivalent of fdescfs opens the underlying file on open(2) of /dev/fd/N name. This behavior is incompatible with the existing FreeBSD fdescfs exposed operation, so it cannot be changed for the default mount of fdescfs on /dev/fd. A new mount flag "nodup" was added for fdescfs which emulates Linux for descriptors backed by vnodes. See the fdescfs(5) man page for details. The described new Unix VFS API is used by Samba. Adding it to FreeBSD should allow our system to be still the first-grade platform for Samba, and was requested by Samba porters. Sponsor: The FreeBSD Foundation ??????????????????????????????????????????????????????????????????????????????? Dummynet support for pf Links: dummynet in-progress tree URL: https://github.com/kprovost/freebsd-src/tree/ netgate/dummynet Contact: Kristof Provost <kp at FreeBSD.org> The pf firewall currently relies on ALTQ for traffic shaping. ALTQ is not enabled in default kernel builds, and is not compatible with all network drivers (only drivers which implement if_start()). Work is in progress to add support for dummynet traffic shaping to pf. Dummynet is already used by ipfw for traffic shaping. As part of this work, automated tests are being added to dummynet, for both ipfw and pf. This requires extending dummynet to add vnet support, which was contributed by Tom Jones <thj at FreeBSD.org>. While this work is incomplete feedback on architecture and functionality is welcomed. TODO: ? Additional test cases ? Debug failure of the IPv6 queue test case ? Fix panic on vnet shutdown if there are still IPv6 packets queued. (These eventually get sent to ip6_input() with a now freed rcvif pointer. Badness ensues.) Sponsor: Rubicon Communications, LLC ("Netgate") ??????????????????????????????????????????????????????????????????????????????? Ethernet support for pf Links: pf-link in-progress tree URL: https://github.com/kprovost/freebsd-src/tree/ netgate/pf-link Work is ongoing to add basic support for Ethernet filtering to pf. This will allow layer 2 addresses to be used to tag packets for subsequent filtering or shaping in the existing pf code. The layer 2 code is strictly stateless. The intended use case for this is to improve pf?s capabilities in captive portal setups (i.e. allow/deny internet access based on client MAC addresses). TODO: ? (optional) anchor support ? move nvlist interface code into libpfctl ? audit nvlist code for bugs (several bugs were found in the recent nvlist alternatives to existing ioctl calls) ? (optional) VLAN ID filtering ? (optional) MAC address table support While this work is incomplete feedback on architecture and functionality is welcomed. Sponsor: Rubicon Communications, LLC ("Netgate") ??????????????????????????????????????????????????????????????????????????????? pf syncookie support Links: pf syncookies in-progress tree URL: https://github.com/kprovost/freebsd-src/ tree/modirum/pf-syncookie-cleanup Contact: Kristof Provost <kp at FreeBSD.org> SYN cookies are a mechanism to resist SYN flood denial of service attacks. The FreeBSD network stack has supported this since 2001. However, this does not prevent such an attack from flooding the pf state table. OpenBSD ported this mechanism from FreeBSD into their pf in 2018. This code is now being ported to FreeBSD?s pf. The work is still in progress, but the current version demonstrates the basic capability. Next steps include additional testing, extra automated test cases and implementing the adaptive mode. The adaptive mode requires extra care in FreeBSD?s pf, above what was done in OpenBSD, because of different locking strategies. Sponsor: Modirum MDPay ??????????????????????????????????????????????????????????????????????????????? Microchip PolarFire SoC support Contact: Jakub Klama <jakub at conclusive.pl> Contact: Wojciech Kloska <wojciech at conclusive.pl> The Conclusive Engineering team is working on support for the Microchip PolarFire FPGA SoC chip family. PolarFire SoC is based on five 64-bit hard RISC-V cores, out of which four are equipped with MMU and capable of running FreeBSD. The SoC also features an FPGA and various peripherals including Gigabit Ethernet, PCIe and multi-function 12.3Gbps SERDES transceivers. At the time of writing, the following tasks have been completed: ? Initial platform bring-up with SMP support ? Add support for creating "booti"-style images for RISC-V ? Ethernet support Work in progress: ? Clock and reset domain support ? USB driver ? PCI Express driver ? I2C driver ? SPI driver ? GPIO driver ? eNVM driver ??????????????????????????????????????????????????????????????????????????????? Racct (Resource Accounting) Bug Fixes and Improvements Links: racct PR for runj URL: https://github.com/samuelkarp/runj/pull/11 Contact: Cyril Zhang <cyril at freebsdfoundation.org> racct is a resource accounting system in the FreeBSD kernel. It keeps track of resource usage, such as memory use and CPU runtime. Additionally, it provides an easy interface for limiting the resource usage of entities such as users and, importantly, jails. For example, it may be of interest to set a limit on the number of CPUs that a jail can use. I have been discussing with other FreeBSD developers the prospect of adding racct support as an experimental feature to an OCI-compliant container runtime, runj by Samuel Karp. This would mimic Linux?s cgroups functionality in the OCI specification, which allows Linux containers to have limits on memory, CPU usage, etc. It also allows us to consider the possibility of adding FreeBSD-specific configuration to the OCI specification. As part of this work, I have been improving racct so that it is more functional for use with jails. Improvements include: ? A new, more accurate scheme for calculating total CPU usage of all processes in a jail ? Fixing a bug that incorrectly counted the runtime of all child processes in the parent?s runtime ? Fixing a bug that incorrectly decremented the count of persistent resources, such as shared memory, when a process exits ? Accounting for POSIX features like shared memory and semaphores, where previously only SysV features were accounted for ? Adding tests Feel free to get in touch. Sponsor: The FreeBSD Foundation ??????????????????????????????????????????????????????????????????????????????? OpenZFS RAIDZ Expansion update Contact: Matthew Ahrens <matt at mahrens.org> Links: OpenZFS Pull Request URL: https://github.com/openzfs/zfs/pull/12225 video from 2021 FreeBSD Developer Summit URL: https://youtu.be/3SUKJye54aI?t6166 slides from 2021 FreeBSD Developer Summit URL: https://docs.google.com/ presentation/d/1FeQgEwChrtNQBHfWSNsPK3Y53O5BnPh3Cz5nRa5GAQY/view news article from Ars Technica URL: https://arstechnica.com/gadgets/2021/06/ raidz-expansion-code-lands-in-openzfs-master/ Project This feature allows disks to be added one at a time to a RAID-Z group, expanding its capacity incrementally. This feature is especially useful for small pools (typically with only one RAID-Z group), where there isn?t sufficient hardware to add capacity by adding a whole new RAID-Z group (typically doubling the number of disks). FreeBSD?s ZFS implementation comes from the OpenZFS project. This work will be integrated into the OpenZFS repo, with support for FreeBSD and Linux. Status The work is functionally complete, and a pull request has been opened. Remaining Work Remaining work includes some code cleanup, design documentation, addressing test failures. We also need to solicit code reviewers and respond to code review feedback. Sponsor: The FreeBSD Foundation ??????????????????????????????????????????????????????????????????????????????? Microchip Switchtec support Contact: Jakub Klama <jakub at conclusive.pl> Contact: Pawe? Eichler <pawel at conclusive.pl> The Conclusive Engineering team is working on support for the Microchip Switchtec PCI Express storage switch family. Switchtec devices are PCI Express Gen3/Gen4/Gen5 packet switches which offer built-in NTB support for up to 48 NT partitions, extensive diagnostics and programmable firmware which can be used to implement additional functionality such as NVMe enclosure management. At the time of writing, following features are completed: ? NTB support ? Firmware management and diagnostics Work in progress: ? Event logging ? NVMe enclosure management This work in co-sponsored by AGILESTORAGE Europe GmbH. ??????????????????????????????????????????????????????????????????????????????? Ports Changes affecting the Ports Collection, whether sweeping changes that touch most of the tree, or individual ports themselves. Emacs Ports Links: PR255706 URL: https://bugs.freebsd.org255706 commit bb995aaf6e25e33b028fa4b32321864b48f49055 URL: https://cgit.freebsd.org/ ports/commit/?id=bb995aaf6e25e33b028fa4b32321864b48f49055 Contact: Emacs Ports Team <emacs at FreeBSD.org> The Emacs development port, editors/emacs-devel, continues to be updated twice per month to synchronize with the HEAD of upstream?s master branch. A noteworthy change from the first June 2021 update was the addition of a NATIVECOMP port option. NATIVECOMP adds support for compiling Emacs lisp to native code using libgccjit, which was first enabled in lang/gcc11-devel and is now in lang/gcc11. For more information about native compilation of Emacs lisp, see Gcc Emacs Wiki . ??????????????????????????????????????????????????????????????????????????????? FreeBSD Erlang Ecosystem Ports update Links: FreeBSD Erlang wiki URL: https://wiki.freebsd.org/Erlang Erlang/OTP language URL: https://erlang.org/ Elixir language URL: https://elixir-lang.org/ Gleam language URL: https://gleam.run/ Contact: FreeBSD Erlang mailing list <erlang at FreeBSD.org> The Erlang runtime system, commonly known as the BEAM, provides a runtime that is used by a number of programming languages and applications in the FreeBSD ports collection. Notable changes in 2021 include: ? adding OTP24, including support on amd64 architecture for JIT compilation, and dropping the previous NATIVE and HIPE options ? security fixes for the devel/rebar3 build tool ? adding a new language runtime for Gleam language ? more than 40 point release updates in the last quarter alone for the Erlang runtimes As the upstream Erlang OTP team have committed to supporting the 2 current releases, more and more point updates are arriving for OTP22-24, but not for the older Erlang runtime releases, which are now unlikely to get security and bug fixes. The Erlang team is planning to: ? deprecate in 2021Q3 any ports that are not compatible with OTP releases in the last 2 years ? remove the deprecated runtimes in 2021Q4 ? remove ports directly dependent on erlang- and elixir- languages, where they are more commonly installed via mix and rebar3 tools, the standard community build tool chain. ? update RabbitMQ to the next major release, which requires OTP23 minimum, and introduces support for the JIT ? bump the main lang/erlang runtime to OTP24 because JIT is awesome Additional testing and community contributions are welcomed, please reach out on the mailing list, especially if you are able to help testing of specific port updates. ??????????????????????????????????????????????????????????????????????????????? GCC in the Ports Tree (lang/gcc*) Links: GCC Project URL: https://gcc.gnu.org GCC 10 release series URL: https://gcc.gnu.org/gcc-10/ Contact: Gerald Pfeifer <gerald at pfeifer.com> With the great help of linimon@ GCC 10 became the default version of GCC in the Ports Collection bringing many improvements and fixes. Looking one step ahead, GCC 11 is now available as a port and even for use as GCC_DEFAULT via Mk/bsd.default-versions.mk . Modern GCC ports like this now feature support for powerpcle, and most related changes also made it it upstream. On the infrastructure side, USE_GCC now allows for a build time-only dependency, e.g. USE_GCC=yes:build . And in case you are developing other ports, USE_GCC=any is a thing of the past and USE_GCC no longer tries to use the old 4.2-based system compiler (/usr/bin/ gcc) even if present. Finally, after some two decades of maintaining FreeBSD?s lang/gcc* ports, the time has come to hand over the baton and largely step back. Please reach out if you are want to help - we?d hate to simply drop maintainership and would be happy to collaborate and transition. ??????????????????????????????????????????????????????????????????????????????? KDE on FreeBSD Links: KDE FreeBSD URL: https://freebsd.kde.org/ KDE Community FreeBSD URL: https://community.kde.org/FreeBSD Contact: Adriaan de Groot <kde at FreeBSD.org> The KDE on FreeBSD project aims to package all of the software produced by the KDE Community for the FreeBSD ports tree. The software includes a full desktop environment called KDE Plasma, graphics applications, instant messengers, a video-editing suite, as well as a tea timer and hundreds of other applications that can be used on any FreeBSD machine. The KDE team (kde@) is part of desktop@ and x11@ as well, building the software stack to make FreeBSD beautiful and usable as a daily-driver graphics-based desktop machine. Organisational The KDE-FreeBSD team has moved its primary communications channel (on IRC) in with the rest of the FreeBSD world. You can now find us on Libera.Chat, in the #freebsd-desktop channel. Why the rename? We?ve been #kde-freebsd since 2003 or so. However, changes in the IRC Network landscape have caused us to migrate to Libera.Chat and join up with the rest of the desktop folk???we have a lot of shared concerns, after all. Big Picture ? The new KDE Gear release???all the KDE software products that are not specifically KDE Frameworks or KDE Plasma, which release in a coordinated fashion on a regular schedule???arrived in ports the day it was released. Albert Astals Cid has a good blog post explaining the rationale behind KDE Gear. ? Wayland support has arrived for KDE Plasma. It is now possible to run a KDE Plasma Wayland environment on FreeBSD machines with a DRM-enabled graphics card (Intel iGPU or AMDGPU). ? There is slow-but-steady progress in reducing the dependencies of KDE Applications. Most FreeBSD ports are "batteries included" and that includes all the build-tools to create the application, but really, an IDE should not be a dependency of your photo-gallery application. ? X11 was made optional in some of the Qt ports; this opens up possibilities such as "Wayland-Only" Qt, or "Framebuffer-Only". X11 remains enabled by default. ? Qt6 is stuck in the "it builds, and parts of it run, but not enough" stage and has not landed in ports; it will probably wait until after the next quarterly is cut. Highlights ? Poppler, CMake, polkit .. these underpinnings of the Free Desktop on FreeBSD all received their regular updates. ? extproc/kdiff3 had some serious issues with 3-way-merges. This was reported by Ed Maste, and turned out to have fixes upstream. ? x11/konsole, the terminal emulator, would not accept /bin/sh as a shell; this is more of an issue on FreeBSD than on Linux where bash is a very common choice. This was resolved upstream and landed in ports. ? The regular KDE Frameworks releases (monthly) and KDE Plasma releases (monthly) and KDE Gear all were announced upstream and landed in FreeBSD ports without a fuss. Upstream CI continues to be updated with the latest FreeBSD ports so that CI reflects well what the KDE-on-FreeBSD team delivers to FreeBSD users. ??????????????????????????????????????????????????????????????????????????????? libglvnd on FreeBSD Links: libglvnd URL: https://gitlab.freedesktop.org/glvnd/libglvnd Contact: Kevin Bowling <kbowling at FreeBSD.org> libglvnd (GL Vendor-Neutral Dispatch) "is a vendor-neutral dispatch layer for arbitrating OpenGL API calls between multiple vendors. It allows multiple drivers from different vendors to coexist on the same filesystem, and determines which vendor to dispatch each API call to at runtime. Both GLX and EGL are supported, in any combination with OpenGL and OpenGL ES." Making this the default GL implementation in FreeBSD Ports unblocks a number of efforts and aligns us with future graphics stacks. It cleans up the ability to have multiple GL implementations on one system or image, regardless of what hardware is in use, by eliminating the need for one-or-the-other hard links for these libraries. This can in turn be used by Nvidia Optimus setups, or using your fancy PCIe interconnect to have multiple vendor GPUs for any reason. It will support future Mesa releases where a concurrent LTS (Long Term Support) branch install can be used to fill in drivers that will be removed from newer Mesa releases. Software like KDE and Firefox can use EGL contexts under X11 (with OpenGL) and Wayland, and in the case of upcoming Firefox releases it may match the speed of Google Chrome when doing so. The library uses various implementation details to minimize any performance impact from indirection, with platform specific support for aarch64, amd64, armv7, i386, and powerpc64 (ELFv2). Finally, it helps reduces over-linking with a new libOpenGL to provide OpenGL without X11 for headless servers or kms or Wayland environments. If there is any remaining fallout from this change please add kbowling@ to the PR watchers. Thank you to Jan Beich <jbeich at FreeBSD.org> for doing the ports infrastructure work and Matt Turner (Gentoo/freedesktop) for helping me understand the architecture and how to handle this change in a source-based distribution, as well as merging a change in libGLU upstream to support building without X11 dependency. ??????????????????????????????????????????????????????????????????????????????? FreeBSD in Science Links: Meta-port for Biostar Handbook tools URL: https://www.freshports.org/biology/ biostar-tools/ Simple, Portable Cluster Manager URL: https://acadix.biz/spcm.php Desktop-installer URL: https://acadix.biz/desktop-installer.php Contact: Jason Bacon <jwb at FreeBSD.org> Contact: Yuri Victorovich <yuri at FreeBSD.org> FreeBSD has long provided a solid foundation for scientific computing, with its remarkable reliability, network and storage performance, and the FreeBSD ports system which not only provides an extensive collection of scientific software, but also facilitates optimized installation by allowing the user to easily install from source with additional compiler flags. Last quarter saw the introduction of sysutils/spcm (Simple, Portable Cluster Manager), an integrated tool set for building and managing HPC (High Performance Computing) clusters based on FreeBSD. This quarter we saw rapid growth in the biology category, much of which culminated in the completion of the biology/biostar-tools metaport. This metaport installs virtually all of the software needed by bioinformatician in training. It is now easier for bioinformatics students to install and run the software they need on FreeBSD than on any other platform. Bioinformatics is the analysis of biological data such as gene and protein sequences. We also hit a milestone of 200 ports in the biology category, thanks to contributions from 21 maintainers. Software installation has always been and still is a major hurdle for many researchers. Far too often, researchers attempt "cave man" installations, following primitive instructions from the developers' site for manually running configure scripts and make. In many other cases, they struggle with low-quality application containers or third-party on package managers that are prone to failures and difficult to use. Problems installing the software they need often cause major delays in their research and often end in failure. The rapid growth of scientific software in the FreeBSD ports collection, coupled with the introduction of SPCM and numerous improvements to sysutils/desktop-installer, have removed many hurdles that could delay or prevent scientific discovery. ??????????????????????????????????????????????????????????????????????????????? Documentation Noteworthy changes in the documentation tree, in manpages, or in external books /documents. FreeBSD Translations on Weblate Links: Translate FreeBSD on Weblate URL: https://wiki.freebsd.org/Doc/Translation/ Weblate FreeBSD Weblate Instance URL: https://translate-dev.freebsd.org/ Contact: Danilo G. Baio <dbaio at FreeBSD.org> The Weblate project Documentation (Books and Articles) is open to the public. The Automatic Translation feature was executed in all languages, re-using translations from the former project (Docbook). There are still pending issues about the new translation process, and the status can be seen on IdeaList#Translation. Website translations weren?t released through Weblate yet. So we decided to hold it until we have more details about the working group that will redesign the Website and the Documentation Portal, to avoid re-work. ??????????????????????????????????????????????????????????????????????????????? FreeBSD Website Revamp - WebApps working group Contact: Sergio Carlavilla <carlavilla at FreeBSD.org> Working group in charge of creating the new FreeBSD Documentation Portal; and redesigning the FreeBSD main Website and its components. FreeBSD developers can follow and join the working group on the FreeBSD Slack channel #wg-www21. The work will be divided into four phases: 1. Redesign of the Documentation Portal Create a new design, responsive and with global search. (Work in progress) 2. Redesign of the Manual Pages on Web Scripts to generate the HTML pages using mandoc. (Not started) 3. Redesign of the Ports page on Web Ports scripts to create an applications portal. (Not started) 4. Redesign of the FreeBSD main Website New design, responsive and dark theme. (Not started) ??????????????????????????????????????????????????????????????????????????????? Miscellaneous Objects that defy categorization. Third-Party Projects Many projects build upon FreeBSD or incorporate components of FreeBSD into their project. As these projects may be of interest to the broader FreeBSD community, we sometimes include brief updates submitted by these projects in our quarterly report. The FreeBSD project makes no representation as to the accuracy or veracity of any claims in these submissions. FreshPorts Links: FreshPorts URL:https://freshports.org/ FreshPorts blog URL: https://news.freshports.org/ Contact: Dan Langille <dvl at FreeBSD.org> FreshPorts, and its sister site, FreshSource, have reported upon FreeBSD commits for 20 years. They cover all commits, but its primary focus is ports. FreshPorts tracks the commits and extracts data from the port Makefiles to create a database of information useful to both port developers and port users. For example, https://www.freshports.org/security/acme.sh/ shows the history of this port, back to its creation in May 2017. git The transition from subversion to git went superbly. Now the work is concentrating on branches. We are close to incorporating all branches (on src, doc, and ports) into the website. The website now queries the main repo every three minutes and pulls in updates. Commit processing under git is faster and more reliable. Creating the XML directly from git and not from parsing commit emails has its benefits. Help wanted Since the last report, Jethro Nederhof has been doing fantastic work bringing the website up to HTML5 and into the modern world. Nothing dramatic, from what the users see, as it has been mostly behind-the-scenes. We can always use more helpers. The website mostly runs itself and requires very little on-going maintenance (pkg upgrades, OS patche, etc). Most of the work is designing new features, fixing bugs, identifying issues, and discussion with users. Changes to the ports tree usually don?t affect the website much. Tasks include: ? There is a long list of issues for the website. ? The git_proc_commit project also has a set of issues, mostly in Pyton, some perhaps related to the website. ? Documentation outlining how the various projects fit together and how they work is required. ? I have some subversion repos which should be converted to git and uploaded to GitHub. This would allow people to work on the backend code (commit ingress and processing). ? The FreshSource website could be updated to modern standards and the repo converted to git. Thank you ??????????????????????????????????????????????????????????????????????????????? helloSystem Links: Documentation URL: https://hellosystem.github.io/ Contact: Simon Peter <probono at puredarwin.org> Contact: #helloSystem on irc.libera.chat, mirrored to #helloSystem:matrix.org on Matrix What is helloSystem? helloSystem is FreeBSD preconfigured as a desktop operating system with a focus on simplicity, elegance, and usability. Its design follows the ?Less, but better? philosophy. Q2 2021 Status ? Version 0.5.0 of helloSystem has been published. Installable Live ISO images are available. Download and changelog are at https://github.com/ helloSystem/ISO/releases/tag/r0.5.0 ? helloSystem was part of the Desktop panel at the FreeBSD Developer Summit 2021. A video is available at https://www.youtube.com/watch?v=3SUKJye54aI&t =11410s ? Work has started towards 0.6.0. Thanks for contributed features and bugfixes ? Improved spatial file manager ? Switch to KWin (if its dependencies can be reduced significantly for use outside of KDE Plasma) Experimental and release builds of the Live ISO are available at https:// github.com/helloSystem/ISO/releases. Contributing Help is wanted in a number of areas, especially in the areas of the FreeBSD core OS and kernel, and Qt/C++. ??????????????????????????????????????????????????????????????????????????????? Containers & FreeBSD: Pot, Potluck & Potman Links: Pot on github URL: https://github.com/pizzamig/pot Potluck Repository & Project URL: https://potluck.honeyguide.net/ Potluck on github URL: https://github.com/hny-gd/potluck Potman on github URL: https://github.com/grembo/potman Contact: Luca Pizzamiglio (Pot) <pizzamig at freebsd.org> Contact: Stephan Lichtenauer (Potluck) <sl at honeyguide.eu> Contact: Michael Gmelin (Potman) <grembo at FreeBSD.org> Pot is a jail management tool that also supports orchestration through Nomad. In the last quarter, v0.12.0 was released and many new improvements have been worked on since then. The main new feature under development at the moment is the possibility to create layered pot images for e.g. quickly adding applications in a build pipeline to a bigger base image which can then be deployed via Ansible or Nomad. Potluck aims to be to FreeBSD and Pot what Dockerhub is to Linux and Docker: A repository of Pot flavours and complete images for usage with Pot and in many cases Nomad. Here, all images have been built for FreeBSD 13.0 for the first time, FreeBSD 11.4 support has been removed and 12.2 images have been upgraded to latest package versions. Furthermore, new images like a PostgreSQL Patroni cluster have been added or the Nomad and Consul images have been extended to support clustering, encryption and a new Vault image. Also, a PoC has been done which shows that Potluck images can potentially easily be used with containerd and runj. Potman aims to simplify building Pot images with Vagrant and VirtualBox based on the Potluck approach, e.g. as part of a DevOps workflow for software development including testing and publishing them to a repository. Here, in the last quarter, usage and documentation has been improved. Future plans include finishing the layered Pot images, further Potman workflow features and more new and improved Potluck images. As always, feedback and patches are welcome. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 618 bytes Desc: not available URL: <http://lists.freebsd.org/pipermail/freebsd-announce/attachments/20210724/4c1d0703/attachment.sig>