Benjamin Kaduk
2016-Feb-04 04:35 UTC
FreeBSD Quarterly Status Report - Fourth Quarter 2015
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
FreeBSD Project Quarterly Status Report: October - December 2015
The fourth quarter of 2015 saw a great deal of activity for FreeBSD.
This is now the third quarter running for which I can say that this is
the largest report yet published! Many thanks to everyone who
proactively submitted topics and entries -- it is great to have more
complete coverage of ongoing development for the community to learn
about in these reports.
An experimental new Triage Team was formed this quarter to create a new
way for community members to participate, and to improve issue
management and productivity in general. Making more effective use of
automation and tooling can help to increase developer productivity and
the quality of FreeBSD, just as the adoption of Jenkins and continual
integration tooling catches regressions quickly and maintains the high
standards for the system.
Efforts to bring our BSD high standards to new architectures continue,
with impressive work on arm64 leading to its promotion to Tier-2 status
and a flurry of work bringing up the new RISC-V hardware architecture.
Software architecture is also under active development, including
system startup and service management. A handful of potential init
system replacements are mentioned in this report: launchd, relaunchd,
and nosh. Architectural changes originating both from academic research
(multipath TCP) and from the realities of industry (sendfile(2)
improvements) are also under way. It is heartening to see how FreeBSD
provides a welcoming platform for contributions from both research and
industry.
To all the readers, whether from academia or industry, hobbyist or
professional: I hope you are as excited as I am to read about all of
the progress and projects covered in this report, and the future of
FreeBSD!
--Ben Kaduk
__________________________________________________________________
The deadline for submissions covering the period from January to March
2016 is April 7, 2016.
__________________________________________________________________
FreeBSD Team Reports
* FreeBSD Release Engineering Team
* Issue Tracking (Bugzilla)
* The FreeBSD Core Team
* The FreeBSD Issue Triage Team
Projects
* CAM I/O Scheduler
* Encrypted Kernel Crash Dumps
* Jenkins Continuous Integration for FreeBSD
* Mellanox iSCSI Extensions for RDMA (iSER) Support
* MIPS: Ralink/Mediatek Support
* Multipath TCP for FreeBSD
* OpenBSM
* Raspberry Pi: VideoCore Userland Application Packaging
* RCTL Disk IO Limits
* Root Remount
* Routing Stack Update
* The Graphics Stack on FreeBSD
* The nosh Project
* UEFI Boot and Framebuffer Support
Kernel
* Chelsio iSCSI Offload Driver (Initiator and Target)
* FreeBSD Integration Services (BIS)
* FreeBSD Xen
* Improvements to the QLogic HBA Driver
* iMX.6 Video Output Support
* ioat(4) Driver Enhancements
* Kernel Vnode Cache Tuning
* Mellanox Drivers
* Minimal Kernel with PNP-Based Autoloading
* MMC Stack Under CAM Framework
* ntb_hw(4)/if_ntb(4) Driver Synced up to Linux
* Out of Memory Handler Rewrite
* sendfile(2) Improvements
* sysctl Enhancements
* Touchscreen Support for Raspberry Pi and Beaglebone Black
Architectures
* armv6 Hard Float Default ABI
* FreeBSD on Marvell Armada38x
* FreeBSD on Newer ARM Boards
* FreeBSD on SoftIron Overdrive 3000
* FreeBSD/arm64
* FreeBSD/RISC-V
* Improvements for ARMv6/v7 Support
Userland Programs
* Base System Build Improvements
* ELF Tool Chain Tools
* The LLDB Debugger
* Updates to GDB
Ports
* Bringing GitLab into the Ports Collection
* GNOME on FreeBSD
* IPv6 Promotion Campaign
* KDE on FreeBSD
* Linux Kernel as a Library Added to the Ports Collection
* LXQt on FreeBSD
* New Tools to Enhance the Porting Experience
* Node.js Modules
* Ports Collection
* Supporting Variants in the Ports Framework
* Xfce on FreeBSD
Documentation
* "FreeBSD Mastery: Specialty Filesystems" Early Access Version
Now
Available
* style(9) Enhanced to Allow C99 bool
Miscellaneous
* HardenedBSD
* NanoBSD Modernization
* relaunchd
* System Initialization and Service Management
* The FreeBSD Foundation
__________________________________________________________________
FreeBSD Release Engineering Team
Links
FreeBSD 10.3-RELEASE schedule
URL: https://www.freebsd.org/releases/10.3R/schedule.html
FreeBSD Development Snapshots
URL: http://ftp.freebsd.org/pub/FreeBSD/snapshots/ISO-IMAGES/
Contact: FreeBSD Release Engineering Team <re at FreeBSD.org>
The FreeBSD Release Engineering Team is responsible for setting and
publishing release schedules for official project releases of FreeBSD,
announcing code freezes, and maintaining the respective branches, among
other things.
During the last quarter of 2015, the Release Engineering team added
support for three additional FreeBSD/arm systems: BANANAPI, CUBIEBOARD,
and CUBIEBOARD2.
In addition to regular development snapshot builds for
FreeBSD 11.0-CURRENT and FreeBSD 10.2-STABLE, several changes and
enhancements were made to the release build code. Of note, the release
build code no longer produces MD5 checksums, in favor of SHA512.
Toward the end of the year, focus was primarily centered on the
upcoming FreeBSD 10.3 release cycle, which will begin in January 2016.
As always, help testing development snapshot builds is crucial to
producing quality releases, and we encourage testing development
snapshots whenever possible.
This project is sponsored by The FreeBSD Foundation.
__________________________________________________________________
Issue Tracking (Bugzilla)
Links
Bugzilla Home Page
URL: https://bugs.freebsd.org/bugzilla/
Contact: Bugmeisters <bugmeister at FreeBSD.org>
Contact: Kubilay Kocak <koobs at FreeBSD.org>
Contact: Mahdi Mokhtari <mokhi64 at gmail.com>
The bugmeister team has gained a new member, Mahdi Mokhtari
(mokhi64 at gmail.com). Mahdi has been contributing to the FreeBSD Project
for just over one month. After getting started by creating ports for
Chef-Server and MySQL 5.7 (with Bernard Spil's help), an introduction
to Kubilay Kocak led to guidance on appropriate projects, such as
Bugzilla development to help Bugmeister, the Bugzilla Triage team,
Developers, and the community by making issue tracking better. This is
how things are going so far:
Issue Tracking can be either "Defect Tracking for Systems" or
"Bug-Tracking for Systems". System Defect Tracking is to allow
individual or groups of developers to keep track of outstanding issues
in their product effectively. We use Bugzilla to manage issues for the
FreeBSD project.
We are pleased to announce some developments on our issue management
systems:
* We have made improvements to the AutoAssigner module (not yet
deployed) that was previously developed by Marcus von Appen to
assign port bugs to their maintainers by default, such as:
+ Improvements and bugfixes to port detection in the Summary:
field of issues, for automatic assignment to their maintainers
in a better way.
+ Refactoring code to make future development easier and faster
in a more modular way.
* We have developed a new module (FBSDAttachment), which automates
setting maintainer-approval flag values on attachments under most
conditions. This will improve time to resolution, consistency of
triage, and reduce manual effort by triagers and maintainers.
* We reported and upstreamed a number of bugs in Bugzilla, working
with the upstream Bugzilla developers.
Open tasks:
1. Major improvements to templates for usability and simplicity.
2. Further improvements to automation (for example, additional
processing of commit logs).
__________________________________________________________________
The FreeBSD Core Team
Contact: FreeBSD Core Team <core at FreeBSD.org>
Two major issues have occupied much of core's attention during the last
quarter: the reorganisation of the Security Team and the question of
whether to import GPLv3 licensed code into the source repository.
1. The idea of reorganizing the Security team was first proposed to
Core during a meeting at BSDCan this year by Gleb Smirnoff -- core
member and newly-appointed deputy Security Officer (SO). The
"Security Team", which previously could contain several people
(a
varying number over time, but more than two) has been refashioned
into just two roles: Security Officer and Deputy Security Officer.
Accordingly, the role of the SO team has been redefined to be the
controller of the distribution of security sensitive information
into and within the project: they are responsible for interfacing
with external bodies and individuals reporting security problems to
the project, and connecting those reports to the appropriate
individuals within the project with the technical expertise to
address the identified concerns. These changes will improve the
project's responsiveness to security alerts, help maintain security
on privileged information received in confidence before general
publication and, not least, reduce the work load on the security
officer. The SO team will continue to benefit from liasons with the
Core, Cluster Administration, and Release Engineering teams, and
will be assisted by a secretary; they will also be able to obtain
input and assistance in drafting security advisories from former
and potential future (Deputy) Security Officers.
Core would particularly like to thank the former members of the
Security Team group for their past contributions, now that the
Security Team role has been merged into the Security Officer's
responsibilities.
2. The other large question concerning Core is how to provide a modern
toolchain for all supported achitectures. Tier 1 architectures are
required to ship with a toolchain unencumbered by onerous license
terms. This is currently provided for i386 and arm64 by the LLVM
suite, including the Clang compiler, LLD and LLDB. However LLVM
support for other (Tier 2 or below) architectures is not yet of
sufficient quality to be viable, and the older but pre-existing
GPLv2 toolchain cannot support some of the interesting new
architectures such as arm64 and RISC-V. Pragmatically, in order for
the project to support these architectures, until LLVM support
arrives we must turn to the GNU project's GPLv3 licenced toolchain.
The argument here is whether to import GPLv3 licensed code into the
FreeBSD src repository with all of the obligations on patent terms
and source code redistribution that would entail, not only for the
FreeBSD project itself but for numerous downstream consumers of
FreeBSD code. Not having a toolchain readily available is a big
impediment to working on a new architecture.
One potential solution is to create a range of "GPLv3
toolchain"
base-system packages out of a completely separate source code
repository, for instance within the FreeBSD area on Github. These
would be distributed equivalently to the other base system binary
packages when that mechanism is introduced.
Core recognises that this is a decision with wide-ranging
consequences and will be producing a position paper for circulation
amongst all interested parties in order to judge community opinion
on the matter. Core welcomes feedback from all interested parties
on the subject.
Beyond these two big questions, Core has handled a number of other
items:
* Core approved the formation of a wiki-admin team to take over
managing the Wiki, to curate the Wiki content and work on
navigation and organization of existing technical content and to
evaluate new Wiki software with the aim of opening up the Wiki to
contributions from the public.
* An external review board has been assembled to look at the Code of
Conduct, including a mixture of project members and experts from
external groups. The review process is getting under way and Core
is awaiting their report.
* The standard documentation license was found to be unfit for its
purpose, and the doceng group had temporarily reverted to the
previous license while a new replacement was drafted. This new
license is now the default for new documentation submissions.
However, one factor emerging from this review was the difficulty of
maintaining correct authorial attributions for sections of
documentation, some of which may only be a few words long. Unlike
source code, blocks of documentation are frequently moved around
within individual files, or even between files. Consequently, Core
would like to introduce a Voluntary Contribution Agreement along
the lines of the one operated by the Apache Foundation. With this,
copyrights are signed over to the FreeBSD Foundation, with
individual contributions being recognised by recording names in a
general "Authors" file. This will be another alternative
alongside
the existing copyright mechanisms used in the project. Core is
interested to hear any opinions on the subject.
* Core approved the formation of a new "dev-announce" mailing
list,
which all FreeBSD committers should be members of. This will be a
low-traffic moderated list to contain important announcements,
heads-ups, warnings of code freezes, changes in policy and
notifications of events that affect the project as a whole.
* Around eight years ago, an attempt was made to import the OpenBSD
sensors framework. This was rejected at the time as potentially
blocking the development of a better designed framework. However,
no such development has occurred in the intervening time whilst the
sensors framework has been in use successfully by both OpenBSD and
FreeNAS. Despite some concerns about the efficiency of the
framework and potential impacts on power consumption and hence
battery lifetime, core is minded to approve the import, but wants
to consult with interested developers first.
* Core is exploring the legal ramifications for the project of the
"Right to Be Forgotten" established by the European Court of
Justice.
* Core is also seeking an alternative means for holding their regular
monthly conference calls. The current, paid-for, service has less
than satisfactory sound quality and reliability, and Core would
like to switch to a free video conferencing solution.
This quarter also saw a particularly large influx of new commit bit
requests, with on occasion, four votes running simultaneously. Please
welcome Kurt Lidl, Svatopluk Kraus, Michal Meloun, Jonathan Looney
(Juniper), Daisuke Aoyama, Phil Shafer (Juniper), Ravi Pokala
(Panasas), Anish Gupta and Mark Bloch (Mellanox) to the ranks of src
committers. In addition, core was delighted to restore commit
privileges for Eric Melville after a hiatus of many years.
No commit bits were taken in during the quarter. A non-committer
account was approved for Kevin Bowling of LimeLight Networks. Kevin
will be doing systems administration work with clusteradm, with
particular interest in the parts of the cluster that are now hosted in
LLNW's facilities. Deb Goodkin of the FreeBSD Foundation was added to
the developers mailing list: she was one of the few members of the
Foundation Board not already on the list, and having awareness of what
is going on in the developer community will help her to support the
project more effectively.
__________________________________________________________________
The FreeBSD Issue Triage Team
Contact: Bugmeister <bugmeister at FreeBSD.org>
Contact: Kubilay Kocak <koobs at FreeBSD.org>
Contact: Vladimir Krstulja <vlad-fbsd at acheronmedia.com>
Contact: Rodrigo N. Hernandez <rodrigo.FreeBSD at minasambiente.com.br>
By the end of the Q4 2015 period, Kubilay Kocak (koobs@) started an
initiative to form an experimental Bugzilla Triage Team. The main goals
of the team are to increase community involvement (addition/training of
new triagers) and enhance current procedures and tools, among others.
This experiment was started with the participation of Vladimir
(blackflow on irc/freenode) and Rodrigo (DanDare on irc/freenode), who
approached koobs@ with a desire to contribute and get more involved
with the FreeBSD Project. This experimental pilot project has the task
of setting up procedures for enhanced Issue (Problem Report) management
that include better classification and prioritization, eventually
leading to faster resolution of issues.
We are now happy to report on the progress of this experimental team:
* The #FreeBSD-bugs IRC channel has been set up on Freenode and we
are successfully using it to exchange information about triage
processes, ask for help, propose changes and discuss related
topics.
* We have identified the primary role of an Issue Triage Team to be
that of classification of problem reports of all kinds (currently
limited mostly to ports and obvious src issues) and facilitation of
issue assignment, which is making sure that the reported issues are
explained well, contain all the appropriate information (or as much
of it as possible), and are brought to attention of the people who
can act upon them.
* Vladimir and Rodrigo are successfully training in bug triage as
well as porting processes (Vladimir is also taking maintainership
of some ports).
* This experiment is benefiting from the introduction of newcomers to
issue tracking. It naturally resulted in a entire review of the
tracking process from its very elementary aspects. This "fresh
eyes" participation spotted minor details during the process,
giving the opportunity to scrutinize actual procedures on a number
of smaller points, followed by proposals on how to improve the
overall Issue Tracking and Management. The new ideas include both
organizational and technical ideas and solutions, such as new or
modified keywords or flags for better classification, the triage
workflow, and Bugzilla technical improvements, among others.
* An important goal is producing documentation about best practices
for using Bugzilla and issue management workflow. This
documentation should be aimed not only at people directly engaged
in issue triage tasks, but also at general users. Another relevant
point is that feedback from the triage team can be used to improve
Bugzilla in terms of adjusting existing features to best fit
FreeBSD's needs, and the development of new features (please see
Mahdi "Magic" Mokhtari's report on "Bugzilla
improvements").
* We are still collating ideas in preparation of setting up a Wiki
namespace for the overall topic of issue management, containing
information for all the parties involved in issue tracking: from
users (reporters) to maintainers and committers. The unorganized
brainstorming document is linked in this report.
Since the Issue Triage Team is very young, we expect more information
be available and more actions to be reported in the next status report.
Open tasks:
1. Set up the Wiki namespace and organize the brainstorming document
into a meaningful set of documents.
2. We are actively recruiting to grow our FreeBSD Triage Team. If you
are interested in participating and contributing to one of the most
important community-facing areas of the FreeBSD project, join
#freebsd-bugs on the freenode IRC and let us know!
Experience with issue tracking is desirable, but not required. No
prior internal project knowledge or technical skills are required,
just bring your communication skills and awesome attitude. Training
is provided.
__________________________________________________________________
CAM I/O Scheduler
Links
BSDCan Paper
URL: https://people.FreeBSD.org/~imp/bsdcan2015/iosched-v3.pdf
Phabricator Review
URL: https://reviews.FreeBSD.org/D4609
Contact: Warner Losh <imp at FreeBSD.org>
Reviews have begun on the CAM I/O scheduler that I wrote for Netflix.
It is anticipated that this process will be done in time for the
FreeBSD 11 branch.
Details about this work can be found in the linked BSDcan paper from
last year.
Briefly, the scheduler allows one to differentiate I/O types and limit
I/O based on the type and characteristics of the I/Os (including the
latency of recent requests relative to historical averages). This is
most useful when tuning system loads to SSD performance. Both a simple
default scheduler, the same that we use today in FreeBSD, as well as a
scheduler that can be well-tuned for system loads related to video
streaming will be included.
This project is sponsored by Netflix, Inc.
__________________________________________________________________
Encrypted Kernel Crash Dumps
Links
Technical Details
URL:
https://lists.FreeBSD.org/pipermail/freebsd-security/2015-December/008780.html
Patch Review
URL: https://reviews.FreeBSD.org/D4712
Contact: Konrad Witaszczyk <def at FreeBSD.org>
Kernel crash dumps contain information about currently running
processes. This can include sensitive data, for example passwords kept
in memory by a browser when a kernel panic occurred. An entity that can
read data from a dump device or a crash directory can also extract this
information from a core dump. To prevent this situation, the core dump
should be encrypted before it is stored on the dump device.
This project allows a kernel to encrypt a core dump during a panic. A
user can configure the kernel for encrypted dumps and save the core
dump after reboot using the existing tools, dumpon(8) and savecore(8).
A new tool decryptcore(8) was added to decrypt the core files.
A patch has been uploaded to Phabricator for review. The patch is
currently being updated to address the review comments, and should be
committed as soon as it is accepted. For more technical details, please
visit the FreeBSD-security mailing list archive or see the Phabricator
review.
__________________________________________________________________
Jenkins Continuous Integration for FreeBSD
Links
The Jenkins CI Server in the FreeBSD Cluster
URL: https://jenkins.FreeBSD.org
Portest Script
URL: https://github.com/Ultima1252/portest
Jenkins Workflow Plugin
URL: https://github.com/jenkinsci/workflow-plugin
Cloudbees
URL: https://cloudbees.com
Jenkins Phabricator Plugin
URL: https://github.com/uber/phabricator-jenkins-plugin
Phabricator Plugin Fixes
URL: https://github.com/uber/phabricator-jenkins-plugin/pull/110
Durable Task Plugin Fixes
URL: https://github.com/jenkinsci/durable-task-plugin/pull/14
Clang Scanbuild Plugin Fixes
URL: https://github.com/jenkinsci/clang-scanbuild-plugin/commits/master
Multiple SCMs Plugin Fixes
URL: https://github.com/jenkinsci/multiple-scms-plugin/commits/master
SCM Sync Configuration Plugin Fixes
URL:
https://github.com/jenkinsci/scm-sync-configuration-plugin/commits/master
Porting Jobs to the Workflow Plugin
URL:
https://lists.FreeBSD.org/pipermail/freebsd-testing/2016-January/001285.html
Akuma Fixes for FreeBSD
URL: https://github.com/kohsuke/akuma/pull/9
Kyua Fix for Invalid Characters
URL: https://github.com/jmmv/kyua/pull/148
Contact: Craig Rodrigues <rodrigc at FreeBSD.org>
Contact: Jenkins Administrators <jenkins-admin at FreeBSD.org>
Contact: FreeBSD Testing <FreeBSD-testing at freebsd.org>
The Jenkins Continuous Integration and Testing project has been helping
to improve the quality of FreeBSD. Since the last status report, we
have quickly found commits that caused build breakage or test failures.
FreeBSD developers saw these problems and quickly fixed them. Some of
the highlights include:
* Ricky Gallagher wrote a script named portest, which can take a
patch to the FreeBSD ports tree as input, and can generate a
sequence of commands to check out the ports tree from Subversion,
apply the patch, and then invoke poudriere to build the affected
part of the ports tree. Ricky consulted with Torsten Z?hlsdorff
during its development. This script will be used later to test
changes to the ports tree.
* Craig Rodrigues converted some Jenkins builds to use the Workflow
plugin. Workflow is a plugin written by Jesse Glick and other
developers at Cloudbees, the main company providing commercial
support for Jenkins. With this plugin, a Jenkins job can be written
in a Domain Specific Language (DSL) which is written in the Groovy
scripting language. Workflow scripts are meant to provide
sophisticated access to Jenkins functionality, in a simple
scripting language. As Jenkins jobs get more complicated and have
more interdependencies, using a DSL is easier for maintainability
instead of creating Jenkins jobs via menus.
Craig Rodrigues worked with Jesse Glick to identify and fix a
problem with the Durable Task plugin used by the workflow plugin.
This problem seemed to show up mostly on non-Linux platforms such
as OS X and FreeBSD.
* Eitan Adler worked with Craig Rodrigues to test a Jenkins plugin
written by Aiden Scandella at Uber which integrates Phabricator and
Jenkins. With this plugin, if someone submits a code review with
Phabricator's Differential tool, a Jenkins build with this code
change will be triggered. The Phabricator code review would then be
updated with the result of the build.
Eitan Adler and Craig Rodrigues had some initial success testing
this plugin using the FreeBSD docs repository, but this plugin
still has a lot of hardcoded dependencies specific to Uber's
environment which make it difficult to use out-of-the-box for
FreeBSD. Alexander Yerenkow submitted some patches upstream to fix
some of these problems, but this plugin still needs more work.
Craig Rodrigues thinks that it might be better to write a workflow
script to call Phabricator commands directly.
* Craig Rodrigues pushed fixes upstream to several plugins including:
+ SCM Sync configuration plugin
+ NodeLabel parameter plugin
+ Subversion plugin
+ Multiple SCMs plugin
+ Clang Scanbuild plugin
Craig Rodrigues was granted commit access to the SCM Sync
configuration plugin, Multiple SCMs plugin, and Clang Scanbuild
plugin.
* Li-Wen Hsu set up multiple builds using jails on machines located
at NYI and administered by the FreeBSD Cluster Administrators. One
of these builds targets 64-bit ARM.
* Michael Zhilin fixed the Akuma library for FreeBSD. The Akuma
library is used by Jenkins to determine what command-line arguments
were passed to a running process. To fix it, Michael invoked an
FreeBSD-specific sysctl() with KERN_PROC_ARGS to determine the
arguments for a running pid. This fix allows a running Jenkins
instance to restart itself after new plugins are installed.
* Julio Merino accepted a fix for Kyua from Craig Rodrigues to fix
writing out XML characters to test report files.
Open tasks:
1. Work more on using the workflow plugin for various builds.
2. Set up a build to test bmake's meta-mode.
3. Finish off integration with Phabricator.
4. People interested in helping out should join the
freebsd-testing at FreeBSD.org list.
__________________________________________________________________
Mellanox iSCSI Extensions for RDMA (iSER) Support
Links
GitHub repository
URL: https://github.com/sagigrimberg/iser-FreeBSD
Contact: Max Gurtovoy <maxg at mellanox.com>
Contact: Sagi Grimberg <sagig at mellanox.com>
Building on the new in-kernel iSCSI initiator stack released in FreeBSD
10.0 and the recently added iSCSI offload interface, Mellanox
Technologies has developed iSCSI extensions for RDMA (iSER) initiator
support to enable efficient data movement using the hardware offload
capabilities of Mellanox's 10, 40, 56, and 100 Gigabit Infiniband
(IB)/Ethernet adapters.
Remote Direct Memory Access (RDMA) has been shown to have great value
for storage applications. RDMA infrastructure provides benefits such as
zero-copy, CPU offload, reliable transport, fabric consolidation, and
many more. The iSER protocol eliminates some of the bottlenecks in the
traditional iSCSI/TCP stack, provides low latency and high throughput,
and is well suited for latency aware workloads.
This work includes a new ICL module that implements the iSER initiator.
The iSCSI stack is slightly modified to support some extra features
such as asynchronous IO completions, unmapped data buffers, and
data-transfer offloads. The user will be able to choose iSER as the
iSCSI transport with iscsictl.
The project is in the process of being merged to FreeBSD 11-CURRENT and
is expected to ship with FreeBSD 11.0.
This project is sponsored by Mellanox Technologies.
__________________________________________________________________
MIPS: Ralink/Mediatek Support
Links
Github Branch With Work in Progress
URL: https://github.com/sgalabov/FreeBSD/tree/local/sgalabov_mtk
Contact: Stanislav Galabov <sgalabov at gmail.com>
This project is aimed at adding FreeBSD support for Ralink/Mediatek's
family of WiFi router system-on-chip (SoC) devices based on MIPS
processors. These SoCs are commonly found in embedded network devices
such as WiFi routers. Having support for these SoCs would allow FreeBSD
to run on a number of additional low-cost devices, which could help
spread FreeBSD's popularity in the embedded systems world.
The project currently aims to support the following Ralink/Mediatek
chipsets: RT3050, RT3052, RT3350, RT3352, RT3662, RT3883, RT5350,
RT6855, RT6856, MT7620, MT7621, MT7628 and MT7688. The following
functionality (where applicable) is currently planned to be supported:
Interrupt controller, UART, GPIO, USB, PCI/PCIe, Ethernet, and SPI.
This project is sponsored by Smartcom - Bulgaria AD.
Open tasks:
1. Help with adding WiFi driver support (possibly to ral(4)) for the
above SoCs would be greatly appreciated.
2. Help with refactoring if_rt(4) to be usable on all of the above
SoCs would be appreciated.
3. Help wth testing target boards (e.g., WiFi routers) would be
appreciated.
__________________________________________________________________
Multipath TCP for FreeBSD
Links
MPTCP for FreeBSD Repository
URL: https://bitbucket.org/nw-swin/caia-mptcp-freebsd/
MPTCP for FreeBSD Project Website
URL: http://caia.swin.edu.au/urp/newtcp/mptcp/
Contact: Nigel Williams <njwilliams at swin.edu.au>
Multipath TCP (MPTCP) is an extension to TCP that allows for the use of
multiple network interfaces on a standard TCP session. The addition of
new addresses and scheduling of data across these occurs transparently
from the perspective of the TCP application.
The goal of this project is to deliver an MPTCP kernel patch that
interoperates with the reference MPTCP implementation, along with
additional enhancements to aid network research.
A v0.51 release has been tagged in our repository, with some minor
improvements over v0.5.
We have now removed much of the MPTCP code that was inside the
functions tcp_do_segment, tcp_output, and other code used for standard
TCP connections. The goal of this is to restrict the added MPTCP code
to just MPTCP connections, leaving regular TCP connections using the
existing code.
We are currently in the process of implementing a subflow socket buffer
upcall and event processing. These will handle changes in subflow
socket state, MP-signalling, and incoming data segments.
This also requires some re-working of the MP option processing,
particularly how incoming DSN maps are parsed and stored for use during
MP-layer reassembly.
We are also looking at how our changes might take advantage of the new
TCP stack modularisation enhancements to create subflow-specific TCP
functions.
This project is sponsored by The Cisco University Research Program Fund
at Community Foundation Silicon Valley, and The FreeBSD Foundation.
Open tasks:
1. Complete the implementations of subflow event processing and new
option parsing.
2. Update documentation and task lists.
__________________________________________________________________
OpenBSM
Links
OpenBSM: Open Source Basic Security Module (BSM) Audit Implementation
URL: http://www.openbsm.org
OpenBSM on GitHub
URL: https://github.com/openbsm/openbsm
FreeBSD Audit Handbook Chapter
URL: https://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/audit.html
Contact: Christian Brueffer <brueffer at FreeBSD.org>
Contact: Robert Watson <rwatson at FreeBSD.org>
Contact: TrustedBSD audit mailing list
<trustedbsd-audit at TrustedBSD.org>
OpenBSM is a BSD-licensed implementation of Sun's Basic Security Module
(BSM) API and file format. It is the user-space side of the CAPP Audit
implementations in FreeBSD and Mac OS X. Additionally, the audit trail
processing tools are expected to work on Linux.
Progress has been slow but steady this quarter, culminating in OpenBSM
1.2 alpha 4, the first release in three years. It features various bug
fixes and documentation improvements; the complete list of changes is
documented in the NEWS file on GitHub. The release was imported into
FreeBSD head and merged to FreeBSD 10-STABLE. As such, it will be part
of FreeBSD 10.3-RELEASE.
Open tasks:
1. Test the new release on different versions of FreeBSD, Mac OS X,
and Linux. In particular, testing on Mac OS X 10.9 (Mavericks) and
newer would be greatly appreciated.
2. Fix problems that have been reported via GitHub and the FreeBSD bug
tracker.
3. Implement features mentioned in the TODO list on GitHub.
__________________________________________________________________
Raspberry Pi: VideoCore Userland Application Packaging
Contact: Mika?l Urankar <mikael.urankar at gmail.com>
Contact: Oleksandr Tymoshenko <gonzo at FreeBSD.org>
The Raspberry Pi SoC consists of two parts: ARM and GPU (VideoCore).
Many interesting features like OpenGL, video playback, and HDMI
controls are implemented on the VideoCore side and can be accessed from
the OS through libraries provided by Broadcom (userland repo). These
libraries were ported to FreeBSD some time ago, so Mika?l created the
port misc/raspberrypi-userland for them. He also created a port for
omxplayer (a low-level video player that utilizes VideoCore APIs) and
is working on a port for Kodi (formerly XBMC), a more user-firendly
media player software with Raspberry Pi support.
__________________________________________________________________
RCTL Disk IO Limits
Contact: Edward Tomasz Napierala <trasz at FreeBSD.org>
An important missing piece of the RCTL resource limits mechanism was
the ability to limit disk throughput. This project aims to fill that
hole by making it possible to add RCTL rules for read bytes per second
(BPS), write BPS, read I/O operations per second (IOPS), and write
IOPS. It also adds a new throttling mechanism to delay process
execution when a limit is reached.
The project is at the late implementation stage. The major piece of
work left apart from testing is to integrate it with ZFS. The project
is expected to ship with FreeBSD 11.0.
This project is sponsored by The FreeBSD Foundation.
__________________________________________________________________
Root Remount
Links
Commit to Head
URL: https://svnweb.freebsd.org/base?view=revision&revision=290548
reboot(8) Manual Page Changes
URL:
https://svnweb.freebsd.org/base/head/sbin/reboot/reboot.8?r1=290548&r2=290547&pathrev=290548
Contact: Edward Tomasz Napierala <trasz at FreeBSD.org>
One of the long-missing features of FreeBSD was the ability to boot up
with a temporary rootfs, configure the kernel to be able to access the
real rootfs, and then replace the temporary root with the real one. In
Linux, this functionality is known as pivot_root. The reroot projects
provides similar functionality in a different, slightly more
user-friendly way: rerooting. Simply put, from the user point of view
it looks like the system performs a partial shutdown, killing all
processes and unmounting the rootfs, and then partial bringup, mounting
the new rootfs, running init, and running the startup scripts as usual.
The project is finished. All the relevant code has been committed to
FreeBSD 11-CURRENT and is expected to ship with FreeBSD 11.0.
This project is sponsored by The FreeBSD Foundation.
__________________________________________________________________
Routing Stack Update
Links
Initial Proposal
URL: http://wiki.freebsd.org/ProjectsRoutingProposal
Contact: Alexander Chernikov <melifaro at FreeBSD.org>
The projects/routing Subversion branch is a FreeBSD routing system
rework aimed at providing performance, scalability and the ability to
add advanced features to the routing stack.
The current packet output path suffers from excessive locking.
Acquiring and releasing four distinct contested locks is required to
convert a packet to a frame suitable to put on the wire. The first
project goal is to reduce the number of locks needed to just two
rmlock(9)s for the output path, which permits close-to-linear scaling.
Since September, one of the locks (used to protect link-level entries)
has been completely eliminated from the packet data path. A new routing
API was introduced, featuring better scalability and hiding routing
internals. Most of the consumers of the old routing API were converted
to use the new API.
__________________________________________________________________
The Graphics Stack on FreeBSD
Links
Graphics Stack Roadmap and Supported Hardware Matrix
URL: https://wiki.FreeBSD.org/Graphics
Ports Development Tree on GitHub
URL: https://github.com/FreeBSD/freebsd-ports-graphics
Contact: FreeBSD Graphics team <freebsd-x11 at freebsd.org>
Several important ports were updated: Mesa to 11.0.8, the X.Org server
to 1.17.4, libdrm to 2.4.65, as well as many applications and
libraries. The latest release of the X.Org server, 1.18, is being
tested in our Ports development tree.
On the kernel side, the i915 update is almost ready to land. There are
a couple known regressions for currently supported GPUs that we want to
fix before committing.
We started a discussion on the FreeBSD-x11@ mailing list to organize
future contributions to the kernel drivers. We have already received
some valuable comments. We are confident that future updates will
happen at a faster pace, thanks to several motivated people!
FOSDEM is held in Brussels on the 30th and 31st of January. We will
attend this conference. It will be a perfect time to see people again
from FreeBSD and from the XDC. On Sunday, we will give a talk about how
to contribute to the Graphics Stack.
Our blog is currently down because the service was discontinued. We
hope to get a dump of our data to put it back online elsewhere.
Unfortunately, there is no ETA for this item.
Open tasks:
1. See the "Graphics" wiki page for up-to-date information.
__________________________________________________________________
The nosh Project
Links
Introduction
URL:
http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh.html
FreeBSD binary packages
URL:
http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/freebsd-binary-packages.html
Installation How-To
URL:
http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/timorous-admin-installation-how-to.html
Roadmap
URL:
http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/roadmap.html
Commands
URL:
http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/commands.html
A Slightly Outdated User Guide
URL:
http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/guide/index.html
The Supervision Mailing List
URL: https://www.mail-archive.com/supervision at list.skarnet.org/
Contact: Jonathan de Boyne Pollard
<J.deBoynePollard-newsgroups at NTLWorld.COM>
The nosh project is a suite of system-level utilities for initializing,
running, and shutting down BSD systems, and for managing daemons,
terminals, and logging. It supersedes BSD init and the NetBSD rc.d
system, drawing inspiration from Solaris SMF for named milestones,
daemontools-encore for service control/status mechanisms, UCSPI, and
IBM AIX for separated service and system management. It comprises a
range of compatibility mechanisms, including shims for familiar
commands from other systems, and an automatic import mechanism that
takes existing configuration data from /etc/fstab,
/etc/rc.conf{,.local}, /etc/ttys, and elsewhere, applying them to its
native service definitions and creating additional native services. It
is portable (including to Linux) and composable, it provides a
migration path from the world of systemd Linux, and it does not require
new kernel APIs. It provides clean service environments, orderings and
dependencies between services, parallelized startup and shutdown
(including fsck), strictly size-capped and autorotated logging, the
service manager as a "subreaper", and uses kevent(2) for
event-driven
parallelism.
Since the last status report, in October 2015, the project has seen:
the complete replacement of its event-handling subsystem on Linux; the
introduction of tools for exporting cyclog/multilog logs via RFC 5426
to remote log handlers (such as logstash); and the switching of the
user-mode virtual terminal subsystem on BSD to using USB devices
directly, a more powerful device interface than sysmouse et al. because
it permits directly positioning touch devices for mice and other things
(thus permitting "mouse integration" under VirtualBox for those who
run
PC-BSD/FreeBSD on VirtualBox virtual machines), but sysmouse et al. can
still be used if desired.
In version 1.24, released shortly before publication of this report,
there are extensive additions for supporting a purely-ZFS system with
an empty /etc/fstab (as the PC-BSD 10.2 system installer creates), and
the ability to convert systemd unit files' process priority settings to
BSD's rtprio/idprio.
Version 1.24 also sees a large chunk taken out of the remainder of the
on-going project to create enough native service bundles and ancillary
utilities to entirely supplant the rc.d system. The progress of this
project has been open from the start, and can be followed on the nosh
roadmap web page. As of version 1.24, there are a mere 27 items
remaining out of the original target list of 157, with a 28th and a
29th (from PC-BSD 10.2) added. Items crossed off by version 1.24
include (amongst others) mfs support for /tmp, static ARP and
networking, persistent entropy for the randomness subsystem, pefs, and
hald.
The remaining items in the task list are mostly aimed at making the
overall system integration cleaner and friendlier to modern systems. We
are also interested in receiving suggestions, bug reports, and other
feedback from users. Try following the how-to guide and see how things
go!
Open tasks:
1. Add kernel support for passing a -b option to PID 1, and support
for a boot_bare variable in the loader, to allow "emergency"
(where
no shell dotfiles are loaded) and "rescue" mode bootstraps,
akin to
Linux. (History: the -b mechanism and idea date back to version
2.57d of Miquel van Smoorenburg's System 5 init clone, dated
1995-12-03, and was already known as "emergency boot" by 1997.)
2. Add support to FreeBSD's fsck(8) for outputting machine-readable
progress reports to a designated file descriptor, so that nosh can
provide progress bars for multiple fscks running in parallel. nosh
already provides this functionality on Linux, where fsck(8) does
provide machine-readable output.
3. Identify when the configuration import system needs to be
triggered, such as when bsdconfig alters configuration files, and
create the necessary hooks to import external configuration changes
into nosh.
__________________________________________________________________
UEFI Boot and Framebuffer Support
Contact: Ed Maste <emaste at FreeBSD.org>
A number of UEFI bug fixes were committed over the last quarter,
further improving compatibility with different UEFI implementations.
Specifically: on some implementations, FreeBSD failed to boot with an
"ExitBootServices() returned 0x8000000000000002" error. This has
been
fixed with a retry loop (as required by UEFI) in r292515 and r292338.
UEFI improvements from other developers have recently been committed or
are in progress. These include support for environment variables set on
the EFI loader command line, improved text console mode setting,
support for nvram variables, and root-on-ZFS support.
This project is sponsored by The FreeBSD Foundation.
Open tasks:
1. Test FreeBSD-CURRENT snapshots on a variety of UEFI
implementations.
2. Merge UEFI changes to stable/10 for FreeBSD 10.3-RELEASE.
__________________________________________________________________
Chelsio iSCSI Offload Driver (Initiator and Target)
Links
Commit Adding Hardware Acceleration Support
URL: https://svnweb.freebsd.org/changeset/base/292740
Contact: Navdeep Parhar <navdeep at chelsio.com>
A new driver, cxgbei, enabling hardware accelerated iSCSI with
Chelsio's T5- and T4-based offload-capable cards, has been committed to
head. Both Initiator and Target are supported. The wire traffic is
standard iSCSI (SCSI over TCP as per RFC 3720, etc.) so an
Initiator/Target using this driver will interoperate with all other
standards-compliant implementations.
Hardware assistance provided by the T5 and T4 ASICs includes:
* Complete TCP processing.
* iSCSI PDU identification and extraction from the byte oriented TCP
stream.
* Header and/or data digest generation and verification.
* Zero copy support for both transmit and receive.
This project is sponsored by Chelsio Communications.
Open tasks:
1. The cxgbei(4) man page is missing but will be committed shortly.
2. The driver is in advanced stage QA and will see some bugfixes and
performance enhancements in the very near future. MFC is possible
as soon as the QA cycle completes.
__________________________________________________________________
FreeBSD Integration Services (BIS)
Links
FreeBSD Virtual Machines on Microsoft Hyper-V
URL: https://wiki.FreeBSD.org/HyperV
Linux and FreeBSD Virtual Machines on Hyper-V
URL: https://technet.microsoft.com/en-us/library/dn531030.aspx
Contact: Dexuan Cui <decui at microsoft.com>
Contact: Hongjiang Zhang <honzhan at microsoft.com>
When FreeBSD virtual machines (VMs) run on Hyper-V, using Hyper-V
synthetic devices is recommended to get the best network and storage
performance and make full use of all the benefits that Hyper-V
provides. The collection of drivers that are required to run Hyper-V
synthetic devices in FreeBSD are known as FreeBSD Integration Services
(BIS). Some of the BIS drivers (like network and storage drivers) have
existed in FreeBSD 9.x and 10.x for years, but there are still some
performance and stability issues and bugs. Compared with Windows and
Linux VMs, the current BIS lacks some important features, such as
virtual Receive Side Scaling (vRSS) support in the Hyper-V network
driver and support for UEFI VM (boot from UEFI), among others.
We are now working more on the issues and performance tuning to make
FreeBSD VMs run better on Hyper-V and the Hyper-V based cloud platform
Azure.
Our work during 2015Q4 is documented below:
* Optimizing the VMBus driver and Hyper-V network driver for
performance:
+ Sent out patches to enable INTR_MPSAFE for the interrupt
handling thread, speed up relid-to-channel lookup in the
thread by map table, and optimize the VMBus ringbuffer
writable notification to the host.
+ Developing a patch to enable the virtual Receive Side Scaling
(vRSS) for Hyper-V network device driver. This will greatly
improve the network performance for SMP virtual machine (VM).
+ Sent out a patch to enable the Hyper-V timer, which will
improve the accuracy of timekeeping when FreeBSD VMs run on
Hyper-V.
* Fixing bugs and cleaning up the code:
+ Fixed a bug in checksum offloading (PR 203630 -- [Hyper-V]
[nat] [tcp] 10.2 NAT bug in TCP stack or hyperv netsvc driver)
in the Hyper-V network driver, making FreeBSD VM based NAT
gateways work more reliably.
+ Fixed a serialization issue in the initialization of VMBus
devices, fixing PR 205156 ([Hyper-V] NICs' (hn0, hn1) MAC
addresses can appear in an uncertain way across reboot).
+ Fixed a KVP (Key-Value Pair) issue (retrieving a key's value
can hang for an uncertain period of time).
+ Added ioctl support for SIOCGIFMEDIA for the Hyper-V network
driver, fixing PR 187006 ([Hyper-V] dynamic address (DHCP)
obtaining does not work on HYPER-V OS 2012 R2).
+ Sent out patches to add an interrupt counter for Hyper-V VMBus
interrupts (so the user can easily get statistical information
about VMBus interrupts), and fix the KVP daemon's poll timeout
(so the daemon will avoid unnecessary polling every 100
milliseconds.
+ Identified a TSC calibration issue: the i8254 PIT timer
emulation of Hyper-V is not fully reliable, so the Hyper-V
time counter should be used to calibrate the TSC. A patch was
drafted. With the patch, it looks the warning kernel message
(e.g., "calcru: runtime went backwards from 46204978 usec to
23362331 usec for pid 0 (kernel)") will go away, and the
time-based tracing of Dtrace will be more accurate.
* We plan to add support for UEFI VMs (Hyper-V Generation-2 VMs).
Currently some issues and to-do items were identified. For example,
we cannot use the i8254 PIT to calibrate the TSC because the i8254
PIT does not exist in a UEFI VM, and we need to add support for the
Hyper-V synthetic keyboard/mouse/framebuffer device.
* We are working on a disk detection issue: when a FreeBSD VM runs on
a Windows Server 2016 Technical Preview host, the VM will detect 16
disks when only one disk is configured for the VM. VMs running on
these hosts can fail to boot. A workaround patch was created and we
are trying to make a formal fix.
* We are tidying up some internal BIS test cases and plan to publish
them on github.
This project is sponsored by Microsoft.
__________________________________________________________________
FreeBSD Xen
Links
FreeBSD PVH DomU Wiki Page
URL: http://wiki.xen.org/wiki/FreeBSD_PVH
FreeBSD PVH Dom0 Wiki Page
URL: http://wiki.xen.org/wiki/FreeBSD_Dom0
FreeBSD/Xen HVMlite Implementation
URL:
http://xenbits.xen.org/gitweb/?p=people/royger/freebsd.git;a=shortlog;h=refs/heads/new_entry_point_v5
Contact: Roger Pau Monn? <royger at FreeBSD.org>
Contact: Wei Liu <wei.liu2 at citrix.com>
Xen is a hypervisor using a microkernel design, providing services that
allow multiple computer operating systems to execute on the same
computer hardware concurrently. Xen support for FreeBSD on x86 as a
guest was introduced in version 8, and ARM support is currently being
worked on. Support for running FreeBSD as an amd64 Xen host (Dom0) is
available in head.
The x86 work done during this quarter has been focused on rewriting the
PVH implementation inside of Xen, into what is now being called HVMlite
to differentiate it with the previous PVH implementation. The Xen side
of patches have already been committed to the Xen source tree, and will
be available in Xen 4.7, the next version. Work has also begun on
implementing HVMlite Dom0 support, although no patches have yet been
published.
HVMlite support for FreeBSD has not yet been committed, although an
initial implementation is available in a personal git repository. The
plan is to completely replace PVH with HVMlite on FreeBSD as soon as
HVMlite supports Dom0 mode.
Apart from this, Wei Liu is working on improving netfront performance
on FreeBSD. Initial patches have been posted to the FreeBSD review
system.
The x86 unmapped bounce buffer code has also been improved, and
unmapped IO support has been added to the blkfront driver.
This project is sponsored by Citrix Systems R&D.
Open tasks:
1. Finish HVMlite Dom0 support inside of Xen.
2. Deprecate and remove PVH support from Xen.
3. Remove PVH support from FreeBSD and switch to HVMlite.
4. Generalize the event channel code so it can be used on ARM.
5. Improve the performance of the various backends (netback, blkback).
__________________________________________________________________
Improvements to the QLogic HBA Driver
Contact: Alexander Motin <mav at FreeBSD.org>
The QLogic HBA driver, isp(4), received a substantial set of changes.
The primary goal was to make the Fibre Channel target role work well
with CTL, but many other things were also fixed/improved:
* Added support for modern 16Gbps 26xx FC cards.
* The firmware in ispfw(4) were updated to the latest versions.
* Target role support was fixed and tested for all FC cards from
ancient 1Gbps 22xx to modern 16Gbps 26xx.
* Port database handling was unified for target and initiator roles,
allowing an HBA port to play both roles at the same time.
* The maximal number of ports was increased from 256 to 1024.
* Multi-ID (NPIV) functionality was fixed/implemented, allowing 24xx
and above cards to provide up to 255 virtual FC ports per physical
port.
* Added support for 8-byte LUNs for 24xx and above cards.
The code is committed to FreeBSD head and stable/10 branches.
This project is sponsored by iXsystems, Inc..
Open tasks:
1. NVRAM data reading is hackish and requires rework.
2. FCoE support for 26xx cards was not tested yet.
__________________________________________________________________
iMX.6 Video Output Support
Links
Commit Adding Basic Video Support
URL: https://svnweb.FreeBSD.org/changeset/base/292574
Contact: Oleksandr Tymoshenko <gonzo at FreeBSD.org>
iMX.6 is a family of SoC used in multiple hobbyist ARM boards such as
the Hummingboard, RIoTboard, and Cubox. Most of these products have
HDMI output, but until recently, FreeBSD did not benefit from it. As of
r292574, there is basic video output support so you can use the console
on iMX6-based boards and probably run Xorg (not yet tested).
Due to the lack of some kernel functionality (see open tasks), the only
supported mode is 1024x768.
Open tasks:
1. Proper pixel clock initialization (relies on a clock framework).
2. More flexible video output path (support multiple IPUs and DIs).
__________________________________________________________________
ioat(4) Driver Enhancements
Links
Wikipedia on I/OAT
URL: https://en.wikipedia.org/wiki/I/O_Acceleration_Technology
Last quarter's ioat(4) report
URL:
https://www.FreeBSD.org/news/status/report-2015-07-2015-09.html#ioat%284%29-Driver-Import
Contact: Conrad Meyer <cem at FreeBSD.org>
I/OAT DMA engines are bulk memory operation offload engines built into
some Intel Server/Storage platform CPUs.
Several enhancements were made to the driver. It now avoids memory
allocation in locked paths, which should avoid deadlocking in memory
pressure scenarios. Support for Broadwell-EP devices has been added.
The "blockfill" operation and a non-contiguous 8 KB copy operation
have
been added to the API. The driver can recover from various programming
errors by resetting the hardware.
This project is sponsored by EMC / Isilon Storage Division.
Open tasks:
1. XOR and other advanced ("RAID") operation support.
__________________________________________________________________
Kernel Vnode Cache Tuning
Links
MFC to stable/10
URL: https://reviews.FreeBSD.org/rS292895
Contact: Kirk McKusick <mckusick at mckusick.com>
Contact: Bruce Evans <bde at FreeBSD.org>
Contact: Konstantin Belousov <kib at FreeBSD.org>
Contact: Peter Holm <pho at FreeBSD.org>
Contact: Mateusz Guzik <mjg at FreeBSD.org>
This completed project includes changes to better manage the vnode
freelist and to streamline the allocation and freeing of vnodes.
Vnode cache recycling was reworked to meet free and unused vnode
targets. Free vnodes are rarely completely free; rather, they are just
ones that are cheap to recycle. Usually they are for files which have
been stat'd but not read; these usually have inode and namecache data
attached to them. The free vnode target is the preferred minimum size
of a sub-cache consisting mostly of such files. The system balances the
size of this sub-cache with its complement to try to prevent either
from thrashing while the other is relatively inactive. The targets
express a preference for the best balance.
"Above" this target there are 2 further targets (watermarks)
related to
the recyling of free vnodes. In the best-operating case, the cache is
exactly full, the free list has size between vlowat and vhiwat above
the free target, and recycling from the free list and normal use
maintains this state. Sometimes the free list is below vlowat or even
empty, but this state is even better for immediate use, provided the
cache is not full. Otherwise, vnlru_proc() runs to reclaim enough
vnodes (usually non-free ones) to reach one of these states. The
watermarks are currently hard-coded as 4% and 9% of the available
space. These, and the default of 25% for wantfreevnodes, are too large
if the memory size is large. For example, 9% of 75% of MAXVNODES is
more than 566000 vnodes to reclaim whenever vnlru_proc() becomes
active.
The vfs.vlru_alloc_cache_src sysctl is removed. The new code frees
namecache sources as the last chance to satisfy the highest watermark,
instead of selecting source vnodes randomly. This provides good enough
behavior to keep vn_fullpath() working in most situations. Filesystem
layouts with deep trees, where the removed knob was required, are thus
handled automatically.
As the kernel allocates and frees vnodes, it fully initializes them on
every allocation and fully releases them on every free. These are not
trivial costs: it starts by zeroing a large structure, then initializes
a mutex, a lock manager lock, an rw lock, four lists, and six pointers.
Looking at vfs.vnodes_created, these operations are being done millions
of times an hour on a busy machine.
As a performance optimization, this code update uses the uma_init and
uma_fini routines to do these initializations and cleanups only as the
vnodes enter and leave the vnode zone. With this change, the
initializations are done kern.maxvnodes times at system startup, and
then only rarely again. The frees are done only if the vnode zone
shrinks, which never happens in practice. For those curious about the
avoided work, look at the vnode_init() and vnode_fini() functions in
sys/kern/vfs_subr.c to see the code that has been removed from the main
vnode allocation/free path.
__________________________________________________________________
Mellanox Drivers
Links
Hardware Information
URL: http://www.mellanox.com/page/ethernet_cards_overview
Commit Adding the Driver
URL: https://svnweb.FreeBSD.org/changeset/base/290650
Contact: Hans Petter Selasky <hselasky at FreeBSD.org>
The Mellanox FreeBSD team is proud to announce support for the
ConnectX-4 series of network cards in FreeBSD 11-current and FreeBSD
10-stable. These devices deliver top performance, with up to 100GBit/s
of raw transfer capacity, and support both Ethernet and Infiniband.
Currently, the Ethernet driver is ready for use and the Infiniband
support for ConnectX-4 is making good progress. We hope that it will be
complete before FreeBSD 11.0 is released. For more technical
information, refer to the mlx5en(4) manual page in 11-current. The new
driver for ConnectX-4 cards is called mlx5 and is put under /sys/dev
and not under /sys/ofed as was done for the previous mlx4 driver. The
mlx5en(4) kernel module is compiled by default in GENERIC kernels.
This project is sponsored by Mellanox Technologies.
__________________________________________________________________
Minimal Kernel with PNP-Based Autoloading
Links
Blog Post
URL:
http://bsdimp.blogspot.com/2016/01/details-on-coming-automatic-module.html
Contact: Warner Losh <imp at FreeBSD.org>
Work on automatically loading modules based on the plug-and-play data
from devices that are scanned and found to not already have a driver
attached is in progress. Digging this information out from kernel
modules, as well as tagging relevant bits of driver tables, has been
committed. PC Card, USB, and some PCI devices now have these markings.
This data is stored in a file that the kernel, boot loader, and
userland processes all can access.
When complete, a user will be able to run a minimal kernel (currently
checked in as the MINIMAL config). Devices necessary for booting will
be loaded by loader(8). Other devices may be loaded there, or early in
the boot (depending on which gives better performance). Users will
still be able to run more monolithic; configurations, as well as limit
which kernel modules are available as can be done today, though without
the convenience that automatic loading will provide. This work remains
ongoing.
Open tasks:
1. Go through all the simplebus drivers and add plug-and-play
information there. Some additional minor simplebus functionality is
needed. There is some work in progress for this.
2. Go through all the PCI drivers and add plug-and-play information to
them. Unlike PC Card or USB, the PCI bus does not have a stylized
table of PCI IDs, so each driver invents its own method, meaning
that the semi-mechanical conversion that was done with PC Card and
USB will not be possible. Instead, customized code for each driver
will be needed. Since a large number of drivers have their own
device tables, the work will be primarily writing a description of
the current table style.
3. Run-time parsing and loading is still needed.
__________________________________________________________________
MMC Stack Under CAM Framework
Links
Project Information
URL: https://bakulin.de/freebsd/mmccam.html
Source Code
URL: https://github.com/kibab/FreeBSD/tree/mmccam
Patch for Review
URL: https://reviews.FreeBSD.org/D4761
Contact: Ilya Bakulin <ilya at bakulin.de>
The goal of this project is to reimplement the existing MMC/SD stack
using the CAM framework. This will permit utilizing the well-tested CAM
locking model and debug features. It will also be possible to process
interrupts generated by the inserted card, which is a prerequisite for
implementing the SDIO interface.
The first version of the code was uploaded to Phabricator for review.
The new stack is able to attach to the SD card and bring it to an
operational state so it is possible to read and write to the card.
The only supported SD controller driver is ti_sdhci, which is used on
the BeagleBone Black. Modifying other SDHCI-compliant drivers should
not be difficult.
Open tasks:
1. Rework bus/target/LUN enumeration and the locking model. I do not
really understand the CAM locking and am likely to do it
incorrectly.
2. Modify the SDHCI driver on at least one x86 platform. This will
make development and collaboration easier.
3. Begin implementing SDIO-specific bits.
__________________________________________________________________
ntb_hw(4)/if_ntb(4) Driver Synced up to Linux
Links
Jon Mason's NTB wiki
URL: https://github.com/jonmason/ntb/wiki
Intel NTB whitepaper
URL:
https://www-ssl.intel.com/content/dam/www/public/us/en/documents/white-papers/xeon-c5500-c3500-non-transparent-bridge-paper.pdf
Contact: Conrad Meyer <cem at FreeBSD.org>
ntb_hw(4) is now up-to-date with the Linux NTB driver as of the
work-in-progress 4.4 kernel (and actually, contains some fixes that
haven't landed in the mainline Linux tree yet but will land in 4.5).
Only Back-to-back ("B2B") configurations are supported at this
time.
Going forward, newer hardware may only support the B2B configuration.
if_ntb(4) is mostly up-to-date with the Linux NTB netdevice driver.
Notably absent is support for changing the MTU at runtime.
This project is sponsored by EMC / Isilon Storage Division.
Open tasks:
1. Improving if_ntb(4) to avoid using the entire Base Address Register
(BAR) when very large BAR sizes are configured (e.g., 512 GB).
2. Improving pmap_mapdev(9) to somehow allocate only superpage
mappings for large BARs, on platforms that support superpages. (NTB
BARs can be as large as 512 GB.)
__________________________________________________________________
Out of Memory Handler Rewrite
Contact: Konstantin Belousov <kib at FreeBSD.org>
The Out of Memory (OOM) code is intended to handle the situation where
the system needs free memory to make progress, but no memory can be
reused. Most often, the situation is that to free memory, the system
needs more free memory. Consider a case where the system needs to
page-out dirty pages, but needs to allocate structures to track the
writes. OOM "solves" the problem by killing some selection of user
processes. In other words, it trades away system deadlock by suffering
a partial loss of user data. The assumption is that it is better to
kill a process and recover data in other processes than to lose
everything.
Free memory in the FreeBSD Virtual Memory (VM) system appears from two
sources. One is the voluntary reclamation of pages used by a process,
for example unmapping private anonymous regions, or the last unlink of
an otherwise unreferenced file with cached pages. Another source is the
pagedaemon, which forcefully frees pages which carry data, of course
after the data is moved to some other storage, like swap or file
blocks. OOM is triggered when the pagedaemon definitely cannot free
memory to satisfy the requests.
The old criteria to trigger the OOM action was a combination of low
free swap space and a low count of free pages (the latter is expressed
precisely with the paging targets constants, but this is not relevant
to the discussion). That test is mostly incorrect. For example, a low
free page state might be caused by a greedy consumer allocating all
pages freed by the page daemon in the current pass, but this does not
preclude the page daemon from producing more pages. Also, since
page-outs are asynchronous, the previous page daemon pass might not
immmediately produce any free pages, but they would appear some short
time later.
More seriously, low swap space does not necessarily indicate that we
are in trouble: lots of pages might not require swap allocations to be
freed, like clean pages or pages backed by files. The last notion is
serious, since swap-less systems were considered as having full swap.
Instead of trying to deduce the deadlock from looking at the current VM
state, the new OOM handler tracks the history of page daemon passes.
Only when several consecutive passes failed to meet the paging target
is an OOM kill considered necessary. The count of consequent failed
passes was selected empirically, by testing on small (32M) and large
(512G) machines. Auto-tuning of the counter is possible, but requires
some more architectural changes to the I/O subsystem.
Another issue was identified with the algorithm which selects a victim
process for OOM kill. It compared the counts of pages mapping entries
(PTEs) installed into the machine paging structures. For different
reasons, the machine-dependent VM code (pmap) may remove the pte for a
memory-resident page. Under some circumstances related to other
measures to prevent low memory deadlock, very large processes which
consume all system memory could have few or no ptes. The old OOM
selector ignored the process which caused the deadlock, killing
unrelated processes.
A new function, vm_pageout_oom_pagecount(), was written which applies a
reasonable heuristic to estimate the number of pages freed by killing
the given process. This eliminates the effect of selecting small
unrelated processes for OOM kill.
The rewrite was committed to head in r290917 and r290920.
This project is sponsored by The FreeBSD Foundation.
__________________________________________________________________
sendfile(2) Improvements
Links
Commit to Head
URL: https://svnweb.FreeBSD.org/base?view=revision&revision=293439
Slides
URL: http://www.slideshare.net/facepalmtarbz2/new-sendfile-in-english
Presentation (in Russian)
URL: https://events.yandex.ru/lib/talks/2682/
Contact: Gleb Smirnoff <glebius at FreeBSD.org>
The sendfile(2) system call was introduced in 1998 as an alternative to
a traditional read(2)/write(2) loop, speeding up server performance by
a factor of ten at the time. Since it was adopted by all major
operating systems, it is now used by any serious web server software.
Wherever there is high traffic, there is sendfile(2) under the hood.
Now, with FreeBSD 11, we are making the next revolutinary step in
serving traffic. sendfile(2) no longer blocks waiting on disk I/O.
Instead, it immediately returns control to the application, performing
the necessary I/O in the background. The original sendfile(2) waited
for the disk read operation to complete and then put the data that was
read into the socket, then returned to userspace. If a web server
served thousands of clients with thousands of requests, it was forced
to spawn extra contexts from which to run sendfile(2) to avoid stalls.
Alternatively, it could use special tricks like the SF_NODISKIO flag
that forces sendfile(2) to serve only content that is cached in memory.
Now, these tricks are in the past, and a web server can simply use
sendfile(2) as it would use write(2), without any extra care. The new
sendfile cuts out the overhead of extra contexts, short writes, and
extra syscalls to prepopulate the cache, bringing performance to a new
level.
The new syscall is built on top of two newly-introduced kernel
features. The first is an asynchronous VM pager interface and the
corresponding VOP_GETPAGES_ASYNC() file system method for UFS. The
second is the concept of "not ready" data in sockets. When
sendfile(2)
is called, first VOP_GETPAGES_ASYNC() is called, which dispatches I/O
requests for completion. Buffers with pages to be populated are put
into the socket buffer, but flagged as not-yet-ready. Control
immediately returns to the application. When the I/O is finished, the
buffers are marked as ready, and the socket is activated to continue
transmission.
Additional features of the new sendfile are new flags that provide the
application with extra control over the transmitted content. Now it is
possible to prevent caching of content in memory, which is useful when
it is known that the content is unlikely to be reused any time soon. In
such cases, it is better to let the associated storage be freed, rather
than putting the data in cache. It is also possible to specify a
readahead with every syscall, if the application can predict client
behavior.
The new sendfile(2) is a drop-in replacement, API and ABI compatible
with the old one. Applications do not even need to recompile to benefit
from the new implementation.
This work is a joint effort between two companies: NGINX, Inc., and
Netflix. There were many people involved in the project. At its initial
stage, before code was written, the idea of such an asynchronous
drop-in replacement was discussed amongst Gleb Smirnoff, Scott Long,
Konstantin Belousov, Adrian Chadd, and Igor Sysoev. The initial
prototype was coded by Gleb under the supervision of Kostik on the VM
parts of the patch, and under constant pressure from Igor, who demanded
that nginx be capable of running with the new sendfile(2) with no
modifications. The prototype demonstrated good performance and
stability and quickly went into Netflix production in late 2014. During
2015, the code matured and continued serving production traffic at
Netflix. Scott Long, Randall R. Stewart, Maksim Yevmenkin, and Andrew
Gallatin added their contributions to the code.
Now we are releasing the code behind our success to the FreeBSD
community, making it available to all FreeBSD users worldwide!
This project is sponsored by Netflix, and NGINX, Inc..
Open tasks:
1. SSL_sendfile() -- an extension to the new sendfile(2) that allows
uploading session keys to the kernel, and then using sendfile(2) on
an SSL-enabled socket.
__________________________________________________________________
sysctl Enhancements
Links
Wikipedia Entry on C99 Fixed-Width Integer Types
URL: https://en.wikipedia.org/wiki/C_data_types#Fixed-width_integer_types
sysctl(8) -t Submission PR
URL: https://bugs.FreeBSD.org/bugzilla/show_bug.cgi?id=203918
Contact: Conrad Meyer <cem at FreeBSD.org>
Contact: Ravi Pokala <rpokala at FreeBSD.org>
Contact: Marcelo Araujo <araujo at FreeBSD.org>
Support was added for fixed-width sysctls (signed and unsigned 8-bit,
16-bit, 32-bit, and 64-bit integers). The new KPIs are documented in
the sysctl(9) manual page. The sysctl(8) command line tool supports all
of the new types.
sysctl(8) gained the -t flag, which prints sysctl type information (the
original patch was submitted by Yoshihiro Ota). This support includes
the newly added fixed-width types.
This project is sponsored by EMC / Isilon Storage Division.
__________________________________________________________________
Touchscreen Support for Raspberry Pi and Beaglebone Black
Links
Beaglebone Black with 4DCAPE-43T Demo
URL: http://kernelnomicon.org/?p=534
Input Stack Plans
URL: https://wiki.FreeBSD.org/201510DevSummit/GraphicsStack
evdev Port
URL: https://wiki.FreeBSD.org/SummerOfCode2014/evdev_Touchscreens
Contact: Oleksandr Tymoshenko <gonzo at FreeBSD.org>
There are two working proof-of-concept drivers for the AM335x
touchscreen and for the official Raspberry Pi's touchscreen LCD.
Proper touchscreen support would consist of a userland event reading
API, a kernel event reporting API, and kernel hardware drivers for
specific devices. There is an ongoing effort to port the Linux evdev
API to FreeBSD so applications that use libraries like libinput or
tslib could be used without any major changes. Since it is not yet
complete, I created a naive evdev-like API for both kernel and tslib
and was able to run a demo on a Beaglebone Black with 4DCAPE-43T.
Once evdev makes it into the tree, both hardware drivers can be
modified to include "report events" portions and committed.
__________________________________________________________________
armv6 Hard Float Default ABI
Links
Blog Entry
URL:
http://bsdimp.blogspot.com/2015/12/hard-float-api-coming-soon-by-default.html
Contact: Warner Losh <imp at FreeBSD.org>
Work on moving armv6 from a "soft float" ABI (but still using
hardware
floating point) to a fully "hardware float" API moves forward. The
ability to have both soft and hard ABI libraries on the same system is
now functional. All armv6 and armv7 systems we support have hardware
floating point capabilities. We currently use the floating-point
hardware, but with a slightly un-optimal ABI, for compatibility with
older versions of FreeBSD. The ABI differences are only at the
userspace level -- the kernel does not care what floating-point ABI is
used, and both types of binaries can run at the same time.
The run-time linker now knows if a binary uses the hardware float ABI
or the software float ABI by examining some fields in the ELF header.
The linker uses different paths and config files for hard versus soft
binaries. The rc system has been enhanced to load the software float
paths. ldconfig now understands soft libraries in much the same way
that it understands 32-bit libraries on 64-bit systems. No additional
kernel support was necessary for this, apart from a minor patch to pass
the ELF header information to the binary, which has been in the tree
since last summer.
The experimental armv6hf MACHINE_ARCH will be retired after a
transition period. It will cease to mean anything different from armv6
after the build system changes go in. Support for building soft-float
ABI libraries will remain in the tree, to support the WITH_LIBSOFT
build option.
Open tasks:
1. Complete documentation needs to be written.
2. Hooks into the FreeBSD build system to generate soft float and
transition to hard float after a flag day need to be polished up
and committed.
3. A number of different upgrade/coexistence scenarios need to be
tested, and a full package run needs to be done to assess the
latest state of the ports tree. This work should be completed by
the end of January.
__________________________________________________________________
FreeBSD on Marvell Armada38x
Contact: Marcin Wojtas <mw at semihalf.com>
Contact: Michal Stanek <mst at semihalf.com>
Contact: Bartosz Szczepanek <bsz at semihalf.com>
Contact: Jan Dabros <jsd at semihalf.com>
FreeBSD has been ported to run on the Marvell Armada38x platform. This
SoC family boasts single/dual high-performance ARM Cortex-A9 CPUs.
The multi-user SMP system is fully working and has been tested on
Marvell DB-88F6288-GP and SolidRun ClearFog development boards.
The root filesystem can be hosted on a USB 3.0/2.0 drive or via NFS
using a PCIe network card. Experimental support is available for
on-chip Gigabit Ethernet (NETA).
Additional features:
* GIC+MPIC cascaded interrupts courtesy of INTRNG
* CESA dual-channel cryptographic engine
* USB 3.0 and 2.0
* PCIe 2.0
* I2C
* GPIO
* Watchdog
* RTC
The port is under community review and will be integrated into head
soon.
This project is sponsored by Stormshield, and Semihalf.
Open tasks:
1. Optimize performance of NETA and prepare for submission.
__________________________________________________________________
FreeBSD on Newer ARM Boards
Links
FreeBSD on Odroid-C1
URL: https://wiki.FreeBSD.org/FreeBSD/arm/Odroid-C1
Commit Adding Glue Driver
URL: https://svnweb.FreeBSD.org/changeset/base/291683
Contact: John Wehle <john at feith.com>
Contact: Ganbold Tsagaankhuu <ganbold at FreeBSD.org>
We made the changes required to support the Amlogic Meson Ethernet
controller on the Hardkernel ODROID-C1 board, which has an Amlogic
aml8726-m8b SoC. The main effort needed was to write a glue driver for
the Ethernet controller -- the Amlogic Meson Ethernet controller is
compatible with Synopsys DesignWare 10/100/1000 Ethernet MAC (if_dwc).
__________________________________________________________________
FreeBSD on SoftIron Overdrive 3000
Links
SoftIron Website
URL: http://softiron.co.uk/products/
Contact: Andrew Turner <andrew at FreeBSD.org>
The SoftIron Overdrive 3000 is an ARMv8 based server with an 8-core AMD
Opteron A1100 processor. The Overdrive 3000 has two 10Gbase-T Ethernet
ports, two PCI Express ports, and eight SATA ports. FreeBSD has been
updated to be able to boot on this hardware.
Support for the SATA device was added to the ahci(4) driver. Unlike on
x86, this is a Memory Mapped (mmio) device, and not on the PCI bus. To
support this, a new ahci mmio driver attachment has been added.
The generic PCIe driver has been updated to improve interrupt handling.
This includes supporting the interrupt-map devicetree property, and
supporting MSI and MSI-X interrupts on arm64.
Support for MSI and MSI-X interrupts has been added to the ARM Generic
Interrupt Controller v2 (gicv2) driver. This allows devices to use
these interrupts. This has been tested with a collection of PCIe NIC
hardware.
This project is sponsored by SoftIron Inc..
Open tasks:
1. Write a driver for the 10Gbase-T NIC.
__________________________________________________________________
FreeBSD/arm64
Links
FreeBSD arm64 Wiki Entry
URL: https://wiki.FreeBSD.org/arm64
Contact: Andrew Turner <andrew at FreeBSD.org>
Contact: Konstantin Belousov <kib at FreeBSD.org>
Contact: Ed Maste <emaste at FreeBSD.org>
Contact: Ed Schouten <ed at FreeBSD.org>
Support was added for kernel modules. This included adding the needed
relocation types to the in-kernel relocator, and updating the build
logic to build modules for arm64. CTF data is currently not generated
for modules due to a linker bug.
Shared page support was added. This allows gettimeofday(2) to be
implemented in userland by directly accessing the timer register. This
reduces the overhead of these calls as we no longer need to call into
the kernel. This also moves the signal trampoline code away from the
stack, allowing for the stack to become non-executable.
CloudABI support for arm64 was added. This included moving the
machine-independent code into a separate file to be shared among all
architectures. An issue in the arm64 kernel was found and fixed thanks
to the CloudABI test suite.
Self-hosted poudriere package builds have been tested. These complement
the previous build strategy of using qemu usermode emulation. With this
combination of self-hosted and qemu usermode building, many ports that
used to be broken on arm64 have been fixed, resulting in over 17,000
ports building for the architecture.
The machine-dependent portion of kernel support for single-stepping
userland binaries has been started. This will allow debuggers like lldb
to step through an application while debugging.
Many small fixes have been made to FreeBSD/arm64. These include fixing
stack tracing through exceptions, printing more information about "data
abort" kernel panics, cleaning up the atomic functions, supporting
multi-pass driver attachment, fixing userland stack alignment, cleaning
up early page table creation, fixing asynchronous software trap
handling, and enabling interrupts in exception handlers.
This project is sponsored by The FreeBSD Foundation, and ABT Systems
Ltd.
__________________________________________________________________
FreeBSD/RISC-V
Links
Project Wiki
URL: https://wiki.FreeBSD.org/riscv
Contact: Ruslan Bukin <br at bsdpad.com>
Contact: Ed Maste <emaste at FreeBSD.org>
Contact: Arun Thomas <arun.thomas at baesystems.com>
We have begun work on support for the RISC-V architecture.
RISC-V is a new ISA designed to support computer architecture research
and education that is now set to become a standard open architecture
for industry implementations.
A minimal set of changes needed to compile the kernel toolchain has
been committed, along with machine headers, run-time linker (rtld-elf)
support, and libc/libstand.
All development has been happening in a separate branch, with a goal of
moving development to head in a few weeks.
At present, FreeBSD/RISC-V boots to multiuser in the Spike simulator.
This project is sponsored by DARPA, AFRL, and HEIF5.
Open tasks:
1. We plan to commit the rest of userspace (i.e., libc), kernel
support, etc., in a few weeks.
__________________________________________________________________
Improvements for ARMv6/v7 Support
Contact: Dominik Ermel <der at semihalf.com>
Contact: Wojciech Macek <wma at semihalf.com>
Contact: Zbigniew Bodek <zbb at semihalf.com>
Numerous improvements for the ARMv6/v7 kernel and tools have been
developed by the Semihalf team. Those include:
* Fixes for KGDB support.
* Support for branch instructions in ptrace single stepping.
* Fixes for kernel minidumps.
* Improvements for LIBUSBBOOT.
* Support for Exynos EHCI in the loader.
* A fix for instruction single stepping in DDB.
* Support for hardware watchpoints, including watchpoints on SMP
systems.
* Single stepping using the ARM Debug Architecture.
* Support for gzip-compressed kernel modules in kldload.
* Backport of the new pmap VM code to FreeBSD 10-STABLE (not yet sent
to upstream).
Most of the introduced changes have been committed to head and more are
on the way.
This project is sponsored by Juniper Networks Inc., and Semihalf.
Open tasks:
1. Finish upstreaming the hardware watchpoints support.
__________________________________________________________________
Base System Build Improvements
Links
FreeBSD-Arch Post Describing Plans
URL:
https://lists.FreeBSD.org/pipermail/freebsd-arch/2015-December/017571.html
BSDCan 2014 META_MODE Presentation
URL: http://www.bsdcan.org/2014/schedule/events/460.en.html
WITH_FAST_DEPEND Details
URL: https://svnweb.FreeBSD.org/base?view=revision&revision=290433
WITH_CCACHE_BUILD Details
URL: https://svnweb.FreeBSD.org/base?view=revision&revision=290526
Contact: Bryan Drewery <bdrewery at FreeBSD.org>
Bryan Drewery (bdrewery@) has been working to improve the build
framework as well as buildworld build times. The build system has been
largely untouched by large-scale changes for many years. Most of the
effort has been on improving the recent META_MODE merge that was
presented at BSDCan 2014. This is a new build system that is not
currently enabled by default but brings many benefits. Beyond that,
some highlights of the work changing buildworld are:
* WITH_FAST_DEPEND, which avoids calling "mkdep" during the make
depend phase and instead generates dependency files during
compilation. The old scheme was pre-processing all source files
twice. The new version saves 16-35% in build times.
* WITH_CCACHE_BUILD adds built-in ccache support, avoiding many of
the historical pitfalls of changing CC in make.conf to use ccache.
* Many improvements for parallelization of the build.
* LIBADD improvements to ensure proper usage of this tool to replace
duplicate LDADD and DPADD statements. Further work is under way to
reduce overlinking.
* A lot of cleanup of improper framework usage.
* Ensuring that installing files from the build tree fails if the
destination directory is missing, rather than installing a file as
the directory name.
This project is sponsored by EMC / Isilon Storage Division.
Open tasks:
1. See the FreeBSD-arch mail for more information on planned work.
__________________________________________________________________
ELF Tool Chain Tools
Links
ELF Tool Chain Website
URL: http://elftoolchain.sourceforge.net
Contact: Ed Maste <emaste at FreeBSD.org>
The ELF Tool Chain project provides BSD-licensed implementations of
compilation tools and libraries for building and analyzing ELF objects.
The project began as part of FreeBSD but later became an independent
project in order to encourage wider participation from others in the
open-source developer community.
In the last quarter of 2015 the ELF Tool Chain tools were updated to a
snapshot of upstream Subversion revision 3272. Improvements include
better input file validation, RISC-V support, support for Xen ELF
notes, additional MIPS and ARM relocations, better performance, and bug
fixes.
The ELF Tool Chain project is planning a new release in the first
quarter of 2016, which will facilitate wider testing and use by
projects in addition to FreeBSD.
This project is sponsored by The FreeBSD Foundation.
Open tasks:
1. Add missing functionality (PE/COFF support) to elfcopy and migrate
the base system build.
2. Fix issues found by fuzzing inputs to the tools.
3. Add automatic support for separate debug files.
__________________________________________________________________
The LLDB Debugger
Links
FreeBSD LLDB Wiki Page
URL: https://wiki.FreeBSD.org/lldb
Contact: Ed Maste <emaste at FreeBSD.org>
LLDB is the debugger from the LLVM family of projects. Originally
developed for Mac OS X, it now also supports FreeBSD, NetBSD, Linux,
Android, and Windows. It builds on existing components in the larger
LLVM project, for example using Clang's expression parser and LLVM's
disassembler.
LLDB in the FreeBSD base system was upgraded to version 3.7.0 as part
of the Clang and LLVM upgrade, and it will similarly be upgraded again
to 3.8.0 for FreeBSD 11.0-RELEASE.
LLDB is now enabled by default on the amd64 and arm64 platforms. It is
now a functional basic debugger on arm64, after a number of fixes were
made in the last quarter to both LLDB and the FreeBSD kernel.
This project is sponsored by The FreeBSD Foundation.
Open tasks:
1. Rework the LLDB build to use LLVM and Clang shared libraries.
2. Port a remote debugging stub to FreeBSD.
3. Add support for local and core file kernel debugging.
4. Improve support on architectures other than amd64 and arm64.
__________________________________________________________________
Updates to GDB
Links
New 1:1-Only Thread Target for FreeBSD
URL: https://github.com/bsdjhb/gdb/tree/freebsd-threads
Contact: John Baldwin <jhb at FreeBSD.org>
The KGDB option is now on by default in the devel/gdb port.
Changes to support cross-debugging of crashdumps in libkvm were
committed to head in r291406.
A new thread target for FreeBSD that is suitable for merging upstream
has been written and lightly tested. However, it is not yet available
as an option in the port. This thread target uses ptrace(2) directly
rather than libthread_db and as such supports threads on all ABIs (such
as FreeBSD/i386 binaries on FreeBSD/amd64 and possibly Linux binaries,
though that is not yet tested). It also requires less-invasive changes
in the MD targets in GDB compared to the libthread_db-based target.
Open tasks:
1. Add a port option for the new 1:1-only thread target.
2. Test the new 1:1-only thread target.
3. Figure out why the powerpc kgdb targets are not able to unwind the
stack past the initial frame.
4. Add support for more platforms (arm, mips, aarch64) to upstream gdb
for both userland and kgdb.
5. Add support for debugging powerpc vector registers.
__________________________________________________________________
Bringing GitLab into the Ports Collection
Links
PR for the New Port
URL: https://bugs.FreeBSD.org/bugzilla/show_bug.cgi?id=202468
Installation Guide
URL:
https://github.com/t-zuehlsdorff/gitlabhq/blob/8-3-docu/doc/install/installation-freebsd.md
Upstream GitLab website
URL: https://github.com/gitlabhq/gitlabhq/
Contact: Torsten Z?hlsdorff <ports at toco-domains.de>
GitLab is a web-based Git repository manager with many features that is
used by more than 100,000 organizations including NASA and Alibaba. It
also is a very long-standing entry on the "Wanted Ports" list of
the
FreeBSD Wiki.
In the last quarter, there was steady progress in the project itself
and the porting. The current release of GitLab 8.3 is now based on
Rails 4.2, which obsoletes the need for around 50 new ports. Now there
are only 5 dependencies left to be committed!
While the new version of GitLab 8.3 eases the porting, there are big
changes since the last working port of GitLab 7.14. Nonetheless, it
could be expected to see the next working port in the first quarter of
2016.
This project is sponsored by anyMOTION GRAPHICS GmbH, D?sseldorf,
Germany.
Open tasks:
1. Update the patches from GitLab 7.14 to 8.3.
2. Update the documentation.
3. Provide an updated patch.
__________________________________________________________________
GNOME on FreeBSD
Links
FreeBSD Gnome Website
URL: http://www.FreeBSD.org/gnome
Devel Repository
URL: https://github.com/FreeBSD/freebsd-ports-gnome
Upstream Build Bot
URL: https://wiki.gnome.org/Projects/Jhbuild/FreeBSD
USE_GNOME Porter's Handbook Chapter
URL:
https://www.FreeBSD.org/doc/en_US.ISO8859-1/books/porters-handbook/using-gnome.html
Contact: FreeBSD GNOME Team <freebsd-gnome at FreeBSD.org>
The FreeBSD GNOME Team maintains the GNOME, MATE, and CINNAMON desktop
environments and graphical user interfaces for FreeBSD. GNOME 3 is part
of the GNU Project. MATE is a fork of the GNOME 2 desktop. CINNAMON is
a desktop environment using GNOME 3 technologies but with a GNOME 2
look and feel.
This quarter, due to limited available time there was not much
progress. This began to change in December, when work started on
porting MATE 1.12 and CINNAMON 2.8 to FreeBSD.
Open tasks:
1. The FreeBSD GNOME website is stale. Work is under way to improve
it.
2. Continue working on investigating the issues blocking GNOME 3.18.
__________________________________________________________________
IPv6 Promotion Campaign
Links
Wiki Page
URL: https://wiki.FreeBSD.org/IPv6PortsTODO
Contact: Torsten Z?hlsdorff <ports at toco-domains.de>
There are more and more machines on the internet that only support
IPv6. I manage some of them, and was regularly hit by missing IPv6
support when fetching the distfiles needed for building ports.
I did some research into the impact of missing IPv6 support on the
ports tree. The results are that 10,308 of 25,522 ports are not
fetchable when using IPv6. This renders, through dependencies, a total
of 17,715 ports unbuildable from IPv6-only systems. All you can do then
is wait and hope that distcache.FreeBSD.org caches the distfile. But
this will take some time, which might not be a luxury available when a
piece of software in use is hit by a security issue.
Based on the research, a promotion campaign for IPv6 was started. Some
volunteers will contact the relevant system administrators and try to
convince them to support IPv6. This will start in January 2016 and will
hopefully create some progress soon.
__________________________________________________________________
KDE on FreeBSD
Links
KDE on FreeBSD Website
URL: https://FreeBSD.kde.org/
Experimental KDE Ports Staging Area
URL: https://FreeBSD.kde.org/area51.php
KDE on FreeBSD Wiki
URL: https://wiki.FreeBSD.org/KDE
KDE/FreeBSD Mailing List
URL: https://mail.kde.org/mailman/listinfo/kde-FreeBSD
Development Repository for Integrating KDE Frameworks 5 and Plasma 5
URL: http://src.mouf.net/area51/log/branches/plasma5
Contact: KDE on FreeBSD team <kde at FreeBSD.org>
The KDE on FreeBSD team focuses on packaging and making sure that the
experience of KDE and Qt on FreeBSD is as good as possible.
The team kept busy during the last quarter of 2015. Quite a few big
updates were committed to the ports tree, and a few more are being
worked on in our experimental repository.
As in previous quarters, we would like to thank several people who have
contributed with machines, patches, and general help. Tobias Berner,
Guido Falsi (madpilot@), Adriaan de Groot, Ralf Nolden, Steve Wills
(swills@), and Josh Paetzel (jpaetzel@) have been essential to our
work.
The following big updates landed in the ports tree this quarter. In
many cases, we have also contributed patches to the upstream projects.
* CMake 3.4.0 and 3.4.1
* Calligra 2.9.1, the latest release of the integrated work
applications suite. Calligra had last been updated in the ports
tree at the end of 2013!
* PyQt4 4.11.4, QScintilla2 2.9.1 and SIP 4.17.
* PyQt5 5.5.1. Thanks to the work spearheaded by Guido Falsi and
Tobias Berner in the previous quarter, the PyQt5 ports have finally
been committed to the ports tree. Not only was this long-awaited on
its own, it allows other ports to be updated to their latest
versions.
* QtCreator 3.5.1 and 3.6.0.
* A couple of Qt5 packaging bugs were fixed: it should now be more
straightforward to use the Qt5 ports to build software outside the
ports tree, and it is now possible to build ports that require a
C++11 compiler and Qt5 on FreeBSD 9.x.
Work on updating the Qt5 ports to their latest version, as well as
porting KDE Frameworks 5 and Plasma 5 to FreeBSD, is well under way in
our experimental area51 repository. At the moment, it contains Qt5
5.5.1, KDE Frameworks 5.17.0, Plasma 5.5.1 and KDE Applications
15.12.0.
Users interested in testing those ports are encouraged to follow the
instructions in our website and report their results to our mailing
list. Qt5 5.5.1 is in our "qt-5.5" branch, and Plasma 5 and the
rest is
in the "plasma5" branch (which also contains Qt 5.5.1).
Open tasks:
1. Commit the Qt5 5.5.1 update.
2. Land the KDE Frameworks 5 and Plasma 5 ports in the tree.
3. Investigate what needs to be done to make QtWebEngine, the
Chromium-based replacement for QtWebKit, work on FreeBSD.
__________________________________________________________________
Linux Kernel as a Library Added to the Ports Collection
Links
Upstream LKL Github repository
URL: https://github.com/lkl/linux
Contact: Conrad Meyer <cem at FreeBSD.org>
LKL ("Linux Kernel as a Library") is a special
"architecture" of the
full Linux kernel that builds as a userspace library on various
platforms, including FreeBSD. One application of such a library is
using Linux filesystem drivers to implement a FUSE backend.
fusefs-lkl's lklfuse binary is such a FUSE filesystem. It can mount
ext4/3/2, XFS, and BTRFS read-write, using the native drivers from
Linux.
sysutils/fusefs-lkl can now be installed either from packages or ports,
providing access to these filesystems on FreeBSD via FUSE.
__________________________________________________________________
LXQt on FreeBSD
Links
FreeBSD LXQt Project
URL: https://wiki.FreeBSD.org/LXQt
LXQt Devel Repository
URL: https://www.assembla.com/spaces/lxqt/subversion/source
Contact: Olivier Duchateau <olivierd at FreeBSD.org>
LXQt is the Qt port of and the upcoming version of LXDE, the
Lightweight Desktop Environment. It is the product of the merge between
the LXDE-Qt and the Razor-qt projects.
The porting effort remains very much a work in progress: it needs some
components of Plasma 5, the new major KDE workspace.
Currently, only the 0.10 branch is functional. See our wiki page for a
complete list of applications.
We also sent updates for some components of LXDE, required for the LXQt
desktop:
* x11/menu-cache 1.0.1
* x11/lxmenu-data 0.1.4
Binary packages are available (only for test purposes) which are
regularly tested with the KDE development repository.
Open tasks:
1. Port libsysstat to BSD systems.
2. Fix some issues that need to be resolved, especially the shutdown
and reboot commands.
__________________________________________________________________
New Tools to Enhance the Porting Experience
Links
pytoport: Generate FreeBSD Ports from Python modules on PyPI
URL: https://github.com/FreeBSD/pytoport
bandar: Create Development Overlays for the Ports Tree
URL: https://github.com/bbqsrc/bandar
skog: Generate Visual Dependency Trees for FreeBSD Ports
URL: https://github.com/bbqsrc/skog-python
spdx-lookup: SPDX License List Query Tool
URL: https://github.com/bbqsrc/spdx-lookup-python
Contact: Brendan Molloy <brendan+freebsd at bbqsrc.net>
When I starting working on ports for FreeBSD in the last couple of
weeks, I found that my workflow was not as efficient as it could be
using just the available tools, so I made a few that could be useful to
the development community at large. All of these have been or will soon
be added to the Ports tree, so you can play with them today!
pytoport is a command-line application that generates a skeleton port
for a given PyPI package name. It attempts to generate the correct
dependencies, makes a good attempt at guessing the license using
spdx-lookup, and generates a pkg-descr. This made generating the
fifteen or so ports I was working on a complete breeze.
While doing this, however, I noticed that some ports were bringing in
dependencies that I did not expect, and I needed some way to visualise
this. skog builds a dependency tree from the depends lists output by
the Ports framework, and displays it on the command line (with extra
shiny output if you are using UTF-8). No more pesky example and
documentation dependencies being dragged in when you clearly toggled
that OPTION as far off as it would go.
While doing all of this, I found it cumbersome to be copying ports back
and forth between my small development tree living in git and the
larger upstream SVN tree I was using in poudriere. I built a tool
called bandar that takes advantage of the FUSE version of unionfs to
easily overlay my dev tree on the upstream tree, run lint checks,
poudriere, and generate archives with ease.
I am very impressed with how easy it was to build more tooling for
FreeBSD. I hope some of these tools will be of some use to you, and as
always, I'd love to hear your feedback!
Open tasks:
1. Improve skog to support searching a tree for a certain port.
2. Get the bandar port completed.
3. Continue to improve pytoport, adding trove support and better
dependency handling.
4. Patches welcome for all of the above!
__________________________________________________________________
Node.js Modules
Links
Node.js Modules Repository
URL: https://www.assembla.com/spaces/cozycloud/subversion/source
Contact: Olivier Duchateau <olivierd at FreeBSD.org>
Node.js is a platform built on Chrome's JavaScript runtime for easily
building fast, scalable network applications. It uses an event-driven,
non-blocking I/O model that makes it lightweight and efficient --
perfect for data-intensive real-time applications that run across
distributed devices.
The goal of this project is to make it easy to install the modules
available in the npm package registry.
Currently, the repository contains slightly fewer than 300 new ports,
in particular:
* Socket.IO, a library for realtime web applications
* Jison, a JavaSript parser generator
We have improved the USES framework:
* Users can define which version of Node.js will be installed through
/etc/make.conf.
* node-gyp is now well-integrated into the USES framework, via the
build argument.
* The pkg-plist is now automatically generated to make portlint
happy.
Each port is up-to-date.
Open tasks:
1. Update the pre-draft documentation.
2. Bring in grunt.js (and modules), the JavaScript task runner.
__________________________________________________________________
Ports Collection
Links
Ports Collection Landing Page
URL: http://www.FreeBSD.org/ports/
Contributor's Guide
URL:
https://www.freebsd.org/doc/en/articles/contributing/ports-contributing.html
Ports Monitoring Service
URL: http://portsmon.FreeBSD.org/index.html
Ports Management Team Website
URL: http://www.FreeBSD.org/portmgr/index.html
Portmgr on Facebook
URL: http://www.facebook.com/portmgr
Contact: Frederic Culot <portmgr-secretary at FreeBSD.org>
Contact: Frederic Culot <culot at FreeBSD.org>
Contact: FreeBSD Ports Management Team <portmgr at FreeBSD.org>
As of the end of the fourth quarter, the ports tree holds a bit more
than 25,000 ports, and the PR count is around 2,000. The activity on
the ports tree remains steady, with about 7,000 commits performed by
almost 120 active committers.
On the problem reports front, figures show an encouraging trend, with a
significant increase in the number of PRs fixed during Q4. Indeed,
almost 1,800 reports were fixed, which makes an increase of about 20%
compared to Q3.
In Q4, eight commit bits were taken in for safekeeping, following an
inactivity period of more than 18 months (lioux, lippe, simon, jhay,
max, sumikawa, alexey, sperber). Three new developers were granted a
ports commit bit (Kenji Takefu, Carlos Puga Medina, and Ian Lepore),
and one returning committer (miwi) had his commit bit reinstated.
Also related to the management of ports commit bits, nox's grants were
revoked, since the FreeBSD developers learned that Juergen Lock had
passed away.
On the management side, no changes were made to the portmgr team during
Q4.
On QA side 33 exp-runs were performed to validate sensitive updates or
cleanups. Amongst those noticeable changes are the update to GCC 4.9,
CMake to 3.4.1, PostgreSQL to 9.4, and ruby-gems to 2.5.0. Some
infrastructure changes included the usage of a WRKSRC different from
WRKDIR when NO_WRKSUBDIR is set, the removal of bsd.cpu.mk from sys.mk,
and the move of QT_NONSTANDARD to bsd.qt.mk.
Open tasks:
1. We would like to remind everyone that the ports tree is built and
run by volunteers, and any help is greatly appreciated. While Q4
saw a significant increase in the number of problem reports fixed,
we encourage all ports committers to have a look at the issues
reported by our users and try to fix as many as possible. Many
thanks to all who made a contribution during Q4, and keep up the
good work in 2016!
__________________________________________________________________
Supporting Variants in the Ports Framework
Links
Poudriere PoC with Variants
URL: https://github.com/bbqsrc/poudriere/compare/master...feature/variants
Ports Makefile PoC with Examples
URL: https://gist.github.com/bbqsrc/e7e3a54d84706485aa3a
Contact: Brendan Molloy <brendan+freebsd at bbqsrc.net>
I recently became involved with FreeBSD (as in, the last 2-3 weeks),
and found myself quickly involved with Ports development. What struck
me immediately was the difficulty in providing a Python package that
was depended upon by multiple versions of Python. As it turns out,
poudriere can currently only generate one package per port, meaning
that a Python version-neutral (compatible with 2.x and 3.x) port cannot
simultaneously be packaged for each variant at the same time.
I discussed the issue with Kubilay Kocak, who suggested that I look
into implementing a "variants protocol" within the Ports framework
and
the necessary changes to poudriere to allow a port to generate more
than one package.
Support for variants is strongly needed in Ports and provides
significant benefits.
* It would allow Python and other languages to provide packages for
dependencies for multiple language versions from the same port.
* It alleviates the need for so-called "slave ports", as a single
port could now have multiple generated packages from a single port.
* It would have a very small impact on the greater Ports ecosystem:
adding only two new variables, VARIANT and VARIANTS.
* It would provide a more consistent approach between different
packaging teams for handling variations.
For a simple example, editors/vim-lite could be folded into the
editors/vim port, while still generating a vim and vim-lite package.
For Python, VARIANTS can be derived from the already used USES flags
and generate compatible packages. py27-foobar and py34-foobar could now
be consistently generated by poudriere without issue.
Fortunately, this is not a wishful thinking piece. I dug in my heels
and have implemented a proof-of-concept implementation of variants in
the Ports framework, including the necessary modifications to poudriere
in order to support it. It was mildly upsettling to find that poudriere
is mostly written in Bourne shell scripts, but I pressed on
nonetheless.
I started with the prototype made by Baptiste Daroussin as a base, and
built from there. The poudriere PoC aims to limit changes as much as
possible to merely adding support for the new variants flags, while
also at the request of Kubilay Kocak making the logging output more
package-centric (as opposed to port-centric) as a result of these
changes.
This is a work in progress, and I would love to hear your feedback. I
have enjoyed my first few weeks working on FreeBSD, and I hope to stay
here for quite some time.
Open tasks:
1. Any constructive feedback on the implementation would be very
welcome!
2. Hopefully the code will be of sufficient quality to be considered
for formal review in the coming months.
__________________________________________________________________
Xfce on FreeBSD
Links
FreeBSD Xfce Project
URL: https://wiki.FreeBSD.org/Xfce
FreeBSD Xfce Repository
URL: https://www.assembla.com/spaces/xfce4/subversion/source
Contact: FreeBSD Xfce Team <xfce at FreeBSD.org>
Xfce is a free software desktop environment for Unix and Unix-like
platforms, such as FreeBSD. It aims to be fast and lightweight, while
still being visually appealing and easy to use.
During this quarter, the team has kept these applications up-to-date:
* audio/xfce4-pulseaudio-plugin 0.2.4
* multimedia/xfce4-parole 0.8.1
* x11/xfce4-whiskermenu-plugin 1.5.2
We also follow the unstable releases (available in our experimental
repository) of:
* x11/xfce4-dashboard 0.5.4
Open tasks:
1. Propose a patch to upstream to fix Xfdashboard with our version of
OpenGL (it currently coredumps).
__________________________________________________________________
"FreeBSD Mastery: Specialty Filesystems" Early Access Version Now
Available
Links
Book site
URL: https://www.michaelwlucas.com/nonfiction/fmsf
Early access version
URL: https://www.tiltedwindmillpress.com/?product=fmspf
Contact: Michael Lucas <mwlucas at michaelwlucas.com>
FreeBSD Mastery: Specialty Filesystems is now in copyediting. The ebook
should be available by the end of January at all major vendors, and the
print in February.
The book covers everything from removable media, to FUSE, NFSv4 ACLs,
iSCSI, CIFS, and more.
If you act really quickly, you can get the electronic early access
version at a 10% discount. You will get the final ebook when it comes
out as well. (This offer evaporates when the final version comes out.)
__________________________________________________________________
style(9) Enhanced to Allow C99 bool
Links
Bruce's Email Requesting bool be Added to style(9)
URL:
https://lists.FreeBSD.org/pipermail/svn-src-head/2015-December/079671.html
Differential Revision for the Change
URL: https://reviews.FreeBSD.org/D4384
Contact: Bruce Evans <brde at optusnet.com.au>
Contact: Conrad Meyer <cem at FreeBSD.org>
Use of bool is now allowed. It was allowed previously, as well, but now
it is really allowed. Party like it's 1999!
This project is sponsored by EMC / Isilon Storage Division.
Open tasks:
1. Specify style(9)'s opinion on iso646.h.
2. Fix intmax_t to be 128-bit on platforms where __int128_t is used.
__________________________________________________________________
HardenedBSD
Links
HardenedBSD Website
URL: https://hardenedbsd.org/
Introducing HardenedBSD's New Binary Updater
URL:
https://hardenedbsd.org/article/shawn-webb/2015-12-31/introducing-hardenedbsds-new-binary-updater
secadm Beta Published
URL:
https://hardenedbsd.org/article/shawn-webb/2015-11-22/introducing-secadm-030-beta-01
New Package Building Server
URL:
https://hardenedbsd.org/article/admin/2015-11-22/new-package-building-server
secadm
URL: https://github.com/HardenedBSD/secadm
HardenedBSD Haswell Support
URL:
https://github.com/HardenedBSD/hardenedBSD-playground/tree/hardened/experimental/master-i915
Nightly Builds for HardenedBSD Haswell Support
URL:
http://jenkins.hardenedbsd.org/builds/HardenedBSD-CURRENT-i915kms-amd64-LATEST/
Contact: Shawn Webb <shawn.webb at hardenedbsd.org>
Contact: Oliver Pinter <oliver.pinter at hardenedbsd.org>
HardenedBSD has been hard at work improving the performance and
stability of our security enhancements. Security flags are now
per-thread instead of per-process, removing some locking overhead. ASLR
for mmap(MAP_32BIT) requests has been refactored, but lib32 is now
disabled by default.
We have developed a new binary update utility, hbsd-update, akin to
freebsd-update. In addition to normal OS installs, it can also update
jails and ZFS Boot Environments (ZFS BEs). Updates are signed using
X.509 certificates.
secadm 0.3-beta has landed. It has been rewritten from scratch to be
more efficient. As part of the rewrite, the rule syntax has changed and
users must update their rulesets as described in the README.
Thanks to generous donations of a server from G2, Inc and hosting from
Automated Tendencies, we can now do full package builds in just 35
hours, down from 75 hours. This machine will also provide weekly binary
updates for the kernel and base system.
Owing partly to the needs of the developers, we have an experimental
branch that includes the work Jean-S?bastien P?dron has under way for
Haswell graphics support, on top of FreeBSD 11-current. Binary updates
are also provided for this branch.
Unfortunately, in order to focus our efforts on improving HardenedBSD,
we have had to pull back from submitting our ASLR patches to FreeBSD.
The past two years' efforts to address comments on the submission have
taken their toll, and the effort is no longer sustainable. We are proud
to be based on FreeBSD and believe that the whole community could
benefit from the security technologies we are developing. We hope that
someone else will be able to step forward and finish off the task of
integrating ASLR into FreeBSD.
This project is sponsored by Automated Tendencies, G2, Inc, and
SoldierX.
__________________________________________________________________
NanoBSD Modernization
Contact: Warner Losh <imp at FreeBSD.org>
This quarter's NanoBSD updates target three main areas. First, building
a NanoBSD image required root privileges. Second, building for embedded
platforms required detailed knowledge of the format required to boot.
Third, the exact image sizes needed to be known to produce an image.
When NanoBSD was written, FreeBSD's build system required root
privileges for the install step and onward. NanoBSD added to this by
creating a md(4) device in which to construct the image. Some
configurations of NanoBSD added further to this by creating a chroot in
which to cleanly build packages. NanoBSD solves the first problem using
the new NO_ROOT build option to create a meta file. NanoBSD also
augments this record as files are created and removed. The meta file is
then fed into makefs(8) to create a UFS image with the proper
permissions. The UFS image, and sometimes a DOS FAT partition, are then
passed to mkimg(1) to create the final SD image. The mtree manipulation
has been written as a separate script to allow it to move into the base
system where it could assist with other build orchestration tools
(though the move has not happened yet).
The detailed knowledge of how to build each embedded image (as well as
some of the base images for qemu) has always been hard to enshrine.
Crochet puts this knowledge into its builds. The FreeBSD release system
puts it into its system. NanoBSD, prior to the current work, provided
no way to access its knowledge of how to build images. The current
state of this project allows the user to set a simple image type and
have NanoBSD deal with all of the details needed to create that image
type. This includes using the u-boot ports and installing the right
files into a FAT partition so that FreeBSD can boot with ubldr(8),
creating the right boot1.elf file for powerpc64 qemu booting, or the
more familiar (though needlessly complicated) x86 setup. Previous
versions of NanoBSD required too much specialized knowledge from the
user. This work aims to concentrate the knowledge into a set of simple
scripts for any build orchestration system to use.
Finally, NanoBSD images in the past have needed very specific knowledge
of the target device. Part of this is a legacy of the BIOS
state-of-the-art a decade ago, which required very careful matching of
the image to the actual device in the deployed system. Although
relevant at the time, such systems are now vanishingly rare. Support
for them will be phased out (though given the flexibility of NanoBSD,
it can be moved to the few remaining examples in the tree and also
partially covered by the generic image scripts). Today, the typical use
case is to create an SD or microSD card image, and have the image
resize itself on boot. NanoBSD now supports that workflow.
In addition to these items, a number of minor improvements have been
made:
* Support for CPUTYPE-specialized builds. This includes both NanoBSD
support as well as important bug fixes in the base system.
* Support for marking MBR partitions as active.
* Support for more partition types.
Open tasks:
1. mkimg(8) needs to be augmented to create images for the i.MX6 and
Allwinner (and others) SoCs. These SoCs require a boot image to be
written after the MBR, but before the first partition starts.
2. The chroot functionality of some NanoBSD configurations has not yet
been migrated for non-privileged builds.
3. The functionality to manipulate mtree(8) files should be moved into
the base system for use by other build orchestration tools.
4. The script to create a bootable image from one or more trees of
files, as well as some creation of those trees, should be moved
into the base system for use with other build orchestration tools.
5. The growfs functionality works great for single images growing to
the whole disk. However, NanoBSD would prefer that the boot
FS/partition grow to approximately 1/2 the size of the media and
another identical (or close) partition be created for the
ping-ponging upgrades that NanoBSD is setup for. This needs to be
implemented in the growfs rc.d(8) script.
__________________________________________________________________
relaunchd
Links
Development tree on GitHub
URL: https://github.com/mheily/relaunchd
Contact: Mark Heily <mark at heily.com>
The relaunchd project provides a service management daemon that is
similar to the original launchd introduced in Apple OS X.
It is not limited to the original features of launchd, however:
interesting work is being done to add support for launching programs in
jails, passing socket descriptors from the host to a jail, and
launching programs within a preconfigured capsicum(4) sandbox.
Additionally, relaunchd uses UCL for its configuration files, so jobs
can be defined in JSON or other formats supported by UCL.
While there is still work to be done, most of the important features of
the original launchd have been implemented, and relaunchd has been made
available in the FreeBSD Ports Collection. It should still be
considered experimental and not ready for production use, but everyone
is welcome to try it, report issues, and contribute code or ideas for
improvement.
Open tasks:
1. Add support for restarting jobs if they crash.
2. Implement the cron(8) emulation feature.
3. Add support for monitoring files and directories for changes and
launching jobs when changes are detected.
4. Finish things that are incomplete, such as support for jails and
passing open socket descriptors to child processes.
5. Improve the documentation and provide more examples of usage.
__________________________________________________________________
System Initialization and Service Management
Links
A Comparison of init(8) and rc(8) Replacements
URL:
http://www.daemonspawn.org/2016/01/a-comparison-of-alternatives-to-init8.html
Contact: Mark Heily <mark at heily.com>
Contact: Jonathan de Boyne Pollard
<J.deBoynePollard-newsgroups at NTLWorld.COM>
Contact: Jordan Hubbard <jkh at FreeBSD.org>
There are three active projects to provide an alternative to the
traditional init(8) and rc(8) subsystems that manage the boot process
and system services. There are a number of reasons driving the desire
for change, including:
* Faster boot times, made possible by launching services in parallel
* Greater reliability, by ensuring that services are automatically
restarted if they terminate unexpectedly
* Simplified dependency management, using socket activation and
similar techniques
* The ability to launch services "on demand", and have them
self-terminate when idle
* Improved security, by removing the need to start common daemons as
the root user
Two of the projects, launchd and relaunchd, are based on the launchd(8)
API introduced by Apple in Mac OS X. The NextBSD project has ported the
original Apple source code by writing a Mach compatibility layer that
allows launchd to run on FreeBSD. The relaunchd project started from
scratch with the goal of creating a more modular, lightweight, and
portable implementation of the launchd API. The third project, nosh, is
a unique creation that borrows concepts from launchd, systemd, and
several other Unix operating systems.
While the FreeBSD Project has not made a decision to replace the
current init(8) and rc(8) subsystems, the existence and active
development of alternatives will continue to drive innovation in this
space.
Jordan Hubbard is the contact point for the NextBSD launchd, Jonathan
de Boyne Pollard is the contact point for nosh, and Mark Heily is the
contact point for relaunchd.
__________________________________________________________________
The FreeBSD Foundation
Links
Foundation Website
URL: http://www.FreeBSDFoundation.org/
FreeBSD Journal
URL: http://FreeBSDJournal.com/
Contact: Deb Goodkin <deb at FreeBSDFoundation.org>
The FreeBSD Foundation is a 501(c)(3) non-profit organization dedicated
to supporting and promoting the FreeBSD Project and community
worldwide. Funding comes from individual and corporate donations and is
used to fund and manage development projects, conferences and developer
summits, and provide travel grants to FreeBSD developers. The
Foundation purchases hardware to improve and maintain FreeBSD
infrastructure and publishes FreeBSD white papers and marketing
material to promote, educate, and advocate for the FreeBSD Project. The
Foundation also represents the FreeBSD Project in executing contracts,
license agreements, and other legal arrangements that require a
recognized legal entity.
Here are some highlights of what we did to help FreeBSD last quarter:
On the advocacy front, the Foundation attended and sponsored
EuroBSDcon, which took place Oct 1-4 (https://2015.eurobsdcon.org/) in
Stockholm, Sweden. Two days prior, during the developer summit, Deb
Goodkin ran a session on Recruiting to FreeBSD. The Foundation was also
very active during the event itself; in addition to Deb, we had Dru
Lavigne, Kirk McKusick, Erwin Lansing, Ed Maste, Hiroki Sato, Benedict
Reuschling, and Edward Tomasz Napiera?a attend the conference. Deb and
Ed gave a presentation on how the Foundation supports a BSD project.
Kirk gave a presentation on "a Brief History of the BSD Fast File
System," and he taught the two-day tutorial "Introduction to the
FreeBSD Open-Source Operating System."
Deb then attended the 2015 Grace Hopper Conference that was held in
Houston, TX, October 14-16. The conference is for women in computing
and most of the attendees were female computer science majors, female
software developers, and college professors. The Foundation was proud
to be a Silver Sponsor. The conference was very successful for us. Our
presence allowed us to raise awareness of the Project, help recruit
more women, and get more professors to include FreeBSD in their
curriculum.
George V. Neville-Neil traveled to Bangkok, Thailand to present talks
on DTrace, FreeBSD, and teaching with DTrace. The talks were presented
at Chulalongkorn University, which is the largest University in
Thailand with the largest engineering school. The first talk was the
practitioner's introduction to DTrace in which the technology, history
and usage is explained without diving into all the kernel subsystems.
The second was the sales pitch for teaching with Dtrace and with
FreeBSD. The pitch was well received and there were some very good
points made by the audience. The facts that the course materials are
both open source and hosted on github were also well received.
Kirk McKusick completed a 10-hour tutorial about FreeBSD for Pearson
Education in their "Live Lesson" program. In particular, there is a
great free snippet from that course comparing FreeBSD against Linux
here: http://youtu.be/dTpqALCwQ1Y?a. Find out more about the whole
session at:
http://click.linksynergy.com/fs-bin/click?id=NZS3W7D*uS0&subid=&offerid=163217.1&type=10&tmpid=3559&RD_PARM1=http%253A%252F%252Fwww.informit.com%252Fstore%252Fintroduction-to-the-freebsd-open-source-operating-system-9780134305868.
Anne Dickison resumed the Faces of FreeBSD series with interviews
featuring Michael Dexter and Erin Clark. She also continued to produce
and distribute FreeBSD materials for conferences, as well as advocating
for FreeBSD over our social channels.
George V. Neville-Neil headed up the latest Silicon Valley Vendor and
Developer Summit, November 2-3, at the NetApp campus in Sunnyvale,
California. Topics of discussion ranged over new developments in
persistent memory, the use of FreeBSD by a company that builds
rackscale systems, developments in our compiler and tool suite, as well
as others. Additional Foundation Board and Staff attending the summit
included: Deb Goodkin, Glen Barber, Justin T. Gibbs, Kirk McKusick, Ed
Maste, and Hiroki Sato. The complete schedule, and some of the slides,
are available on the FreeBSD Wiki
https://wiki.freebsd.org/201511VendorDevSummit .
Notes from the always lively "Have/Need/Want session" are available
at
https://wiki.freebsd.org/201511VendorDevSummit/HaveNeedWant .
While in the Bay Area, some Foundation members visited commercial users
of FreeBSD to help understand their needs, update them on the work the
Foundation is doing, and facilitate collaboration between them and the
Project.
We were a sponsor of the 2015 OpenZFS Developer Summit, which took
place October 19-20, in San Francisco, CA. Justin T. Gibbs and Kirk
McKusick attended the conference.
Justin T. Gibbs continued his semester long class teaching Intro to
Computer Science using FreeBSD at a middle school.
Ed Maste, Edward Tomasz Napiera?a, and Konstantin Belousov continue to
make progress on Foundation funded development projects. More
specifically:
* Ed Worked on a number of items relating to the tool chain: LLD
linker, ELF Tool Chain components, and LLDB debugger, and tested,
integrated, and merged outstanding UEFI work.
* Edward finished work on the reroot project as well as spending some
time on a certificate-transparency port. He also implemented a
prototype to support disk IO limit in RCTL.
* Konstantin rewrote the out of memory killer logic, which, in
particular, fixed FreeBSD operation on systems without swap,
especially systems with very little memory. The latter are becoming
more and more common with the popularity of embedded ARM platforms
where FreeBSD runs, but it also affects large systems which are
usually configured without swap. He also finalized and committed
the shared page support for the ARMv7 and ARMv8 systems. This
allows for a non-executable stack on ARMv7, and a much faster
userspace gettimeofday(2) for both, similar to x86.
Ed Maste presented a FreeBSD/arm64 talk and a hands-on demo at ARM
Techcon, which took place November 10-12, 2015, in Santa Clara, CA.
We continued publishing our monthly newsletters and acquiring new
company testimonials about using FreeBSD, including from Verisign and
Nginx.
Anne Dickison, Dru Lavigne, and Glen Barber represented the Foundation
at USENIX LISA '15, which took place November 3-8, in Washington D.C..
The Foundation had a booth in the Expo Hall and participated in a BoF.
Besides connecting with current community members, we spoke with
attendees who were interested in getting involved with the Project and
helped set them on the correct path. We also took the opportunity to
remind those who had not used FreeBSD in a while what they were
missing. Glen also attended the USENIX Release Engineering Summit,
which was co-located with LISA '15.
We published the Sept/Oct and Nov/Dec issues of the FreeBSD Journal.
George V. Neville-Neil and Robert Watson announced the release of their
TeachBSD initiative: http://teachbsd.org/. TeachBSD offers a set of
open source reusable course materials designed to allow others to teach
both university students and software practitioners FreeBSD operating
system fundamentals. The Foundation is proud to have partly sponsored
their efforts to teach the initial graduate level course on operating
systems with tracing at the University of Cambridge.
Deb Goodkin invited a representative from the Outreachy program to talk
at the Ottawa FreeBSD Developer Summit about the program and how we can
get involved.
Deb also started discussions with CS professors from the University of
Colorado, Boulder to offer some Intro to FreeBSD workshops.
Glen Barber continued wearing many hats to support to the Project. For
Release Engineering:
* Added support for building BANANAPI, CUBIEBOARD, and CUBIEBOARD2
arm images.
* Deprecated the use of MD5 checksums for verifying installation
media downloaded from the FreeBSD Project mirrors.
* Various miscellaneous updates and fixes to release build code.
* Continued providing regular development snapshot builds.
Under Systems Administration:
* Assisted the Admins team with migrating various services to two new
colocation facilities near Sunnyvale, generously provided by
RootBSD and LimeLight Networks.
* Moved email services for the Foundation to a new server.
Ed Maste attended the Reproducible Builds World Summit, which took
place in Athens, Greece, December 1-3, 2015.
We wrapped up our 2015 fundraising efforts with our End-of-Year
fundraising campaign by participating in #GivingTuesday, and continuing
with weekly email and social media requests for support of the
Foundation. Final fundraising numbers will be available in Q1 2016.
__________________________________________________________________
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQGgBAEBCgAGBQJWstP3AAoJECjZpvNk63USG3wMHArdbptYhXljtbfGKDUWPJjs
ip1eIqDxywocb8P0VZ/NqSdASyuWXyHZGPxjFhE9BIAvUKeTCJPsJLWPqZyiv2Qj
856dYOTyOnzkMbGaN9dPTdH2ymrsfRqQKVr1DP+7la6IdFpvlUZ2RwxBX7qPs73/
yMy640tfC+8VVuKt3EjZdbN6hBbeSoiZ0b8pJ+GP42CKhwSYW86wrtMSudjNRZzZ
lBYSJJ9RPh76Z6Qjcv8mBioEEl+XZyadzErM7rDNpiIRV4Ucr3ez24sUiGl30+4G
SpniAKmXBL+6wWm9jhWfm6obOQJecTc7ydc5zT0Qb8rXIBPsYGxOYxujM1zqWbqt
4TjAqnzkanTqOfZDMzvpMj8dBk9gUxjHloQ66pfNx6UeV0cCbSh6k/vIQdTbvGAn
NknGKyYQ8n0AQCHht3sXdOcGCca0gHq0uDEWR71woqTReXAjCdAEHNSE+XnnHWYa
lCgsuNe5PWtSfYuyMu95X863us7mShf8277hQzNlft8ILTY=7aRE
-----END PGP SIGNATURE-----