Benjamin Kaduk
2017-Sep-27 23:01 UTC
FreeBSD Quarterly Status Report - Second Quarter 2017
FreeBSD Quarterly Status Report - 2nd Quarter 2017
FreeBSD continues to defy the rumors of its demise.
Much of the development work done this quarter was not particularly
visible, especially the effort needed to ensure the upcoming 11.1
release has as few regressions as possible. Planning is also well under
way for the 10.4 maintenance release which will quickly follow it.
Further work focused on moving the arm architectures' support closer to
tier-1 status and improving documentation. In addition, large changes
were made to the src and ports trees.
These projects and others are further detailed below.
--Mark Linimon
__________________________________________________________________
The deadline for submissions covering the period from July to September
2017 is October 21, 2017.
__________________________________________________________________
FreeBSD Team Reports
* FreeBSD Release Engineering Team
* Ports Collection
* The FreeBSD Core Team
* The FreeBSD Foundation
* The Postmaster Team
Projects
* 64-bit Inode Numbers
* Capability-Based Network Communication for Capsicum/CloudABI
* Ceph on FreeBSD
* DTS Updates
Kernel
* Coda revival
* FreeBSD Driver for the Annapurna Labs ENA
* Intel 10G Driver Update
* pNFS Server Plan B
Architectures
* FreeBSD on Marvell Armada38x
* FreeBSD/arm64
Userland Programs
* DTC
* Using LLVM's LLD Linker as FreeBSD's System Linker
Ports
* A New USES Macro for Porting Cargo-Based Rust Applications
* GCC (GNU Compiler Collection)
* GNOME on FreeBSD
* KDE on FreeBSD
* New Port: FRRouting
* PHP Ports: Help Improving QA
* Rust
* sndio Support in the FreeBSD Ports Collection
* TensorFlow
* Updating Port Metadata for non-x86 Architectures
* Xfce on FreeBSD
Documentation
* Absolute FreeBSD, 3rd Edition
* Doc Version Strings Improved by Their Absence
* New Xen Handbook Section
Miscellaneous
* BSD Meetups at Rennes (France)
Third-Party Projects
* HardenedBSD
__________________________________________________________________
FreeBSD Team Reports
FreeBSD Release Engineering Team
Links
FreeBSD 11.1-RELEASE Schedule
URL: https://www.FreeBSD.org/releases/11.1R/schedule.html
FreeBSD Development Snapshots
URL: https://download.FreeBSD.org/ftp/snapshots/ISO-IMAGES/
Contact: FreeBSD Release Engineering Team <re at FreeBSD.org>
The FreeBSD Release Engineering Team is responsible for setting and
publishing release schedules for official project releases of FreeBSD,
announcing code freezes, and maintaining the respective branches, among
other things.
The FreeBSD 11.1-RELEASE cycle started on May 19, and continued as
scheduled. FreeBSD consumers are urged to test whenever possible to
help ensure the reliability and stability of the upcoming second
release from the stable/11 branch.
This project was sponsored by The FreeBSD Foundation.
__________________________________________________________________
Ports Collection
Links
About FreeBSD Ports
URL: https://www.FreeBSD.org/ports/
Contributing to Ports
URL:
https://www.freebsd.org/doc/en_US.ISO8859-1/articles/contributing/ports-contributing.html
FreeBSD Ports Monitoring
URL: http://portsmon.freebsd.org/index.html
Ports Management Team Website
URL: https://www.freebsd.org/portmgr/index.html
FreeBSD portmgr on Twitter (@freebsd_portmgr)
URL: https://twitter.com/freebsd_portmgr/
FreeBSD Ports Management Team on Facebook
URL: https://www.facebook.com/portmgr
FreeBSD Ports Management Team on Google+
URL: https://plus.google.com/communities/108335846196454338383
Contact: Ren? Ladan <portmgr-secretary at FreeBSD.org>
Contact: FreeBSD Ports Management Team <portmgr at FreeBSD.org>
This quarter, 2017Q2, broke the 30,000 ports landmark for the first
time. The PR count is currently just under 2,500, with almost 600 of
them unassigned. This quarter saw almost 7,400 commits from 171
committers. More PRs got closed this quarter than last quarter, but
also more PRs got sent in, both of which are good to see.
Over the past three months, we welcomed four new committers: Bradley T.
Hughes (bhughes@), Danilo G. Baio (dbaio@), Jochen Neumeister
(joneum@), and Richard Gallamore (ultima@). kan@ re-joined us as a
ports committer. One commit bit, that of bf@, was taken in for
safekeeping after a long period of inactivity.
On the management side, the Ports Management Team welcomed back bapt@,
who is working on several new features for the Ports Tree. The Ports
Management Team also had its annual real-life meeting during BSDCan.
On the infrastructure side, three new USES values were introduced:
* cargo, to ease the porting of Rust packages or binaries using the
cargo command (also covered separately in this report)
* groff, to handle a dependency on the groff document formatting
system, that has been removed from the base system for FreeBSD 12
* meson, to provide support for projects based on Meson
The default version of PostgreSQL switched from 9.3 to 9.5, and that of
Python3 from 3.5 to 3.6. The default generator for ports using cmake
has been switched to ninja.
Some major version updates are: pkg 1.10.1, Firefox 54.0.1, and
Chromium 59.0.3071.115.
Behind the scenes, antoine@ ran 36 exp-runs to test version updates,
make the CRAN ports platform-independent, test installing bsdgrep(1) as
/usr/bin/grep, test LLVM updates, test the ino64 project, and perform
Makefile cleanups.
__________________________________________________________________
The FreeBSD Core Team
Contact: FreeBSD Core Team <core at FreeBSD.org>
Core's activities during the second quarter culminated in the
introduction of two new initiatives during BSDCan:
* Extending FreeBSD Project Membership
* The FreeBSD Community Process
FreeBSD Project Members
FreeBSD Project Membership being extended to more than just committers
is a step that enables the Project to recognise and reward people who
support us in ways other than by writing code. People that organise
conferences or user groups; who are prominent supporters on social
media; who triage bug reports and who test changes; and many others who
contribute in various ways, are deserving of recognition for the
support that they give to the Project. Core hopes that this will both
encourage more people to volunteer their time and effort on behalf of
the Project, and encourage those who already do to stick with the
Project, if not become more deeply involved.
The naming for the new group of non-committer Project members took a
few tries to get right: having tried, and rejected, "Contributor"
and
then "Associate", Core took the view that since what they were
offerring was formal Project Membership, then that was the right thing
to call it. Committers thus become those Project Members with access to
commit to the Project's code repositories. Project Members receive an
@FreeBSD.org e-mail address, access to various Project hardware, access
to internal mailing lists and other communications channels, and
invitations to attend Developer Summits in their own right. Committers
in addition have commit rights in the Subversion repositories and
GitHub, and active Committers can vote in Core team elections.
The FreeBSD Community Process
This is an idea that has a long pedigree within other projects, and
FreeBSD is very consciously modelling its implementation on what has
worked elsewhere. When a significantly disruptive or wide-scale change
is proposed, we should have a formal mechanism for documenting the
change and what it implies. Interested parties can then respond and the
change can be evolved into the best fit for all users, or else it can
be found to be impracticable and withdrawn. The documentation of the
change will remain as a point of reference should the same or a similar
proposal come up in the future. Creating a more formal process should
help avoid endless sterile arguments about what needs to be done,
without anyone feeling they have sufficient investment in the idea nor
backing from the majority of the project to justify putting in the work
to achieve the desired result.
The very first FCP -- FCP 0 -- describes the process itself. At the
time of this writing, Core is voting on accepting the initial document,
which can be viewed in the Project's Github repository. Two new mailing
lists have been created: fcp at FreeBSD.org is the channel for receiving
notifications of new FCP proposals and discussing their content, whilst
fcp-editors at FreeBSD.org exists to provide help with the process of
drafting the FCP documents.
Other Core activities
Core is delighted to announce that Gordon Tetlow has joined the
Security Officer team, and will be working on managing the Security
Team caseload, freeing up other members to concentrate on the more
technical aspects of vulnerability remediation. In addition, Ed Maste
has joined the Security Team and is available to assist the Security
Officers where necessary.
Although Florian Smeets had to step down, the postmaster team has
recruited three new members and is now back up to strength.
Considering the desirability of a number of fixes that have been merged
into 10-STABLE since the 10.3 release, core has approved a 10.4 release
to occur shortly after the 11.1 release. This will be a normal
support-lifetime release, unlike the extended lifetime of the 10.3
release, so the overall support lifetime for the 10.x branch will not
be significantly extended.
During this quarter, Core has approved issuing three new commit bits.
Please welcome:
* Vladimir Kondratyev (wulf@)
* Ryan Libby (rlibby@)
* Kyle Evans (kevans@)
Also, during this quarter, we had one person give up their commit bit:
* Jordan Hubbard (jkh@)
It is always unsettling when one of the Project's founding members
decides to move on, but Jordan's interests have migrated away from
FreeBSD-related projects and he has decided to hang up his bit once and
for all.
Core would like to thank NTTA (formerly Verio) for providing hosting
for a cvsup mirror for many years, and also for their kind offer to
provide ongoing hosting for a machine in their Seattle facility. Since
we have no need for additional North America hosting, we have declined
their offer.
As usual, a number of questions have been raised about code licensing
and other matters related to intellectual property. Ed Maste has
registered "freebsd" on behalf of the FreeBSD Foundation on the
Mastodon social media network. The "Unlicense" is suitable for code
being imported into libc. We still have some code published under the
old 4-clause style BSD license, where the extra clause refers
specifically to the University of California. While UC has generally
approved removing that clause, we need to check with all copyright
holders before changing any remaining 4-clause licensing.
Core, along with the Security Team, are monitoring developments
concerning the "Stack Clash" vulnerability that hit the headlines
during June. Changes to the stack-guard mitigation system are underway
as a response to the proof-of-concept published by Qualys.
__________________________________________________________________
The FreeBSD Foundation
Links
FreeBSD Foundation Website
URL: https://www.FreeBSDFoundation.org/
FreeBSD Foundation Quarterly Newsletter
URL:
https://www.FreeBSDfoundation.org/wp-content/uploads/2017/06/FreeBSD-Foundation-Q2-2017-Update.pdf
Contact: Deb Goodkin <deb at FreeBSDFoundation.org>
Last quarter the Foundation was busy supporting the FreeBSD Project in
so many ways! We brought on two interns from the University of Waterloo
who were extremely productive, from working on a continuous integration
project to adding MSDOS FAT filesystem support to makefs. We continued
helping to accelerate OS changes with our internal staff of software
developers, as well as funding outside software development projects,
and continued promoting FreeBSD by participating in technology
conferences around the world. To encourage more commercial users to
donate to the Foundation, we launched a new partnership program. The
FreeBSD 11.1 release effort has been led by a full-time Foundation
employee, to continue keeping releases timely and reliable. Finally, we
led the effort to celebrate the newly declared FreeBSD Day, to help
raise awareness of FreeBSD around the world!
Below, you can read some of the highlights from our Q2 newsletter, and
find writeups throughout this status report from Foundation staff
members including Ed Maste, Kostik Belousov, and Glen Barber. Don't
forget, we are 100% funded by donations. Please take a moment to donate
now, so we can continue supporting the FreeBSD Project and community
worldwide!
Q2 Development Projects Summary
Our hard work continues into the 2nd quarter of 2017. Please take a
look at the highlights from our more recent Development Projects
summaries.
April: FreeBSD USB Mass Storage Target Project Update
The Foundation awarded a project grant to Edward Tomasz Napiera?a to
develop a USB mass storage target driver, using the FreeBSD CAM Target
Layer (CTL) as a backend. This project allows FreeBSD on an embedded
platform, such as a BeagleBone Black or Raspberry Pi Zero, to emulate a
USB mass storage target, commonly known as a USB flash stick. Read more
at
https://www.FreeBSDfoundation.org/blog/april-2017-development-projects-update/.
May: Foundation Brings on Co-Op Students
At the beginning of May we embarked on a new path in the FreeBSD
Foundation, with the hiring of co-operative education (co-op) students
from the University of Waterloo. The University of Waterloo is a
pioneer and leader in co-operative education, with 100% of Engineering
students and a majority of Computer Science students participating in
co-op programs. Read more at
https://www.FreeBSDfoundation.org/blog/may-2017-development-projects-update/.
June: FreeBSD Foundation 2017 Project Proposal Solicitation
(contributed by Ed Maste)
One of the ways the Foundation supports FreeBSD is by providing
development grants for work on individual projects. These allow
developers to propose projects they would like to undertake to improve
FreeBSD and request funding to perform that work. The Foundation is
always willing to receive proposals, but will occasionally issue a call
for proposals to highlight specific areas of focus and to be able to
collect and evaluate a group of proposals.
The proposal submission deadline was July 14, 2017, but as mentioned
above, people are welcome to submit proposals at any time.
Although proposals may address any FreeBSD subsystem or infrastructure,
we are particularly interested in receiving proposals related to:
* Improvements to the security of FreeBSD itself, or of applications
running on FreeBSD
* New test cases, improved test infrastructure, and quality assurance
* Improved software development tools
* Projects to improve community collaboration and communication
* Improving the FreeBSD "out of the box" experience for new users
on
various hardware platforms
* Establishing FreeBSD as a leader in advancing projects of shared
interest (such as ZFS, LLVM, or libarchive)
More details can be found at
https://www.FreeBSDfoundation.org/blog/FreeBSD-foundation-2017-project-proposal-solicitation/.
The full project proposal submission guidelines can be found at
http://cts.vresp.com/c/?FreeBSDFoundation/d364934d4d/TEST/1b229d9af7.
Please do not hesitate to contact proposals at FreeBSDfoundation.org with
any questions.
Announcing the New Partnership Program (contributed by Deb Goodkin)
I'm excited to announce our new FreeBSD Foundation Partnership Program!
Our work is 100% supported by donations from individuals and
organizations. With a spending budget of $1,500,000, we rely on large
donations from our commercial users to help us sustain and increase our
support. Recognizing the value of these donations, and putting together
a sustainable funding model, we wanted to institute benefits that
highlighted this support, and recognize these donors in productive
ways. Partnerships are an avenue to assist commercial users by helping
them get on board more quickly with FreeBSD, share their needs with the
community, and facilitate collaboration with FreeBSD developers. We
believe that building these relationships with commercial users will
contribute to keeping FreeBSD relevant and help provide a sustainable
and healthy ecosystem.
You can check out our updated donor pages to see how we are
acknowledging our Partners at
https://www.FreeBSDfoundation.org/donors/. You can also find out more
about this new program at
https://www.FreeBSDfoundation.org/FreeBSD-foundation-partnership-program/.
When I was in China last week, I had a chance to talk to a few
companies about our new partnership program, and it definitely
generated more interest in supporting our efforts.
We are continuing to reach out to commercial users for help that will
enable us to provide more outreach and support for FreeBSD. This
includes funding more projects to improve FreeBSD, providing FreeBSD
education and training, and recruiting more contributors to the
Project. We can only provide the above support with your donations, and
we need your help to connect us with your companies. Please consider
notifying your organization about our new Partnership Program and
helping to connect us with the appropriate contacts at your company.
Your donations will help us:
* Accelerate improvements and add new features to FreeBSD
* Support release engineering efforts full-time
* Create and provide FreeBSD educational and training material
* Provide face-to-face opportunities for developers to work together
* Improve and support FreeBSD infrastructure
We need your support to continue improving FreeBSD.
Q2 2017 Conference Recaps
From sponsoring events to attending conferences, the Foundation
continued its mission of advocacy in the second quarter of 2017. Over
the past few weeks, members of the Foundation team represented the
Project and the Foundation at events around the world. Below are just a
few of the conference recaps.
FOSSASIA 2017 (contributed by Philip Paeps)
The Foundation kindly funded part of my travel from Tokyo to Singapore
to attend FOSSASIA. I gave the "FreeBSD is not a Linux
Distribution"
presentation that Foundation board member George Neville-Neil wrote for
Open Source China in December. My presentation was well-attended, and I
got a lot of good questions from the primarily Linux-oriented audience.
Read more at
https://www.FreeBSDfoundation.org/blog/fossasia-2017-trip-report-philip-paeps/.
OSCON 2017 (contributed by Ed Maste)
I represented the FreeBSD Foundation at OSCON 2017, which took place
May 8-11, 2017, in Austin, TX:
https://conferences.oreilly.com/oscon/oscon-tx .
The Foundation booth was also staffed by FreeBSD committer Brad Davis
and Doug Mcintire from Netgate. We met up Wednesday morning to set up
the table. We were part of a "nonprofit pavilion" which consisted
of
eight or so tables, located between Open Camps and Operation Code.
To help attract booth traffic, I brought a Raspberry Pi 3, with a small
LCD display attached. As a demo, the Raspberry Pi showed a video of a
Gource rendering of changes to the FreeBSD source tree over time (see
example at https://www.youtube.com/watch?v=vZ8Sspua0Ks). Read more at
https://www.FreeBSDfoundation.org/blog/conference-recap-oscon-2017/.
Rootconf 2017 (contributed by Philip Paeps)
In mid-May I presented at Rootconf 2017 in Bangalore. Rootconf is
India's principal conference where systems and operations engineers
share real-world knowledge about building reliable systems:
https://rootconf.in/2017/.
As always, it was interesting to hear the difficulties people face
trying to run reliable systems on less reliable platforms. While many
of the presentations were very Linux-specific and not very exciting to
me, a couple of talks did catch my eye.
I particularly enjoyed the talk by Aruna Sankaranarayanan
(https://www.youtube.com/watch?v=XQJ7YhVoSWI&feature=youtu.be)
explaining how Mapbox takes advantage of Amazon's "spot
pricing"
mechanism by spawning and shutting down machines at different price
points to optimize for cost without compromising availability. Their
spotswap https://github.com/mapbox/spotswap/ software has been released
under a BSD license. It sounds as though it should be possible to port
this to FreeBSD with minimal effort. Read more at
https://www.FreeBSDfoundation.org/blog/rootconf-2017-trip-report-philip-paeps/.
BSDCan 2017/FreeBSD Developers Summit (contributed by Deb Goodkin)
One of our initiatives is to assist in providing face-to-face knowledge
sharing and development opportunities around the world. One way we do
this is by sponsoring BSD-related conferences and FreeBSD Developer and
Vendor Summits. We recently sponsored both BSDCan 2017 and the FreeBSD
Developer and Vendor Summit in Ottawa, Ontario, Canada, which took
place June 7-10, 2017. Many of our board and staff members attended the
summit and conference to run tutorials, give presentations, lead
sessions, work with developers, give demos, and share knowledge.
In addition, this year we were pleased to bring our new University of
Waterloo interns to the conference where they had the opportunity to
demonstrate some of their projects at the Foundation table. Read more
at
https://www.FreeBSDfoundation.org/blog/conference-recap-bsdcan-2017FreeBSD-developers-summit/.
Open Travel Grant Applications
The Foundation recognizes the importance of bringing members of the
FreeBSD community face-to-face to both further development of the
Project and spread the word about FreeBSD. Travel grants are available
to community members who need assistance with travel expenses for
attending conferences related to FreeBSD development and advocacy.
Please note: the travel grant policy has been recently updated. Please
carefully review it before submitting your application.
More information about travel grants is available at:
https://www.FreeBSDfoundation.org/what-we-do/grants/travel-grants/.
FreeBSD Day was June 19! (contributed by Anne Dickison)
June 19th was declared FreeBSD Day! Thank you to everyone who joined us
in honoring the FreeBSD Project's pioneering legacy and continuing
impact on technology. Find out more about FreeBSD Day and how we
celebrated here at
https://www.FreeBSDfoundation.org/blog/happy-FreeBSD-day/.
Upcoming Events
Find out about upcoming Foundation events at
https://www.FreeBSDfoundation.org/news-and-events/upcoming-events/.
FreeBSD Journal
The May/June 2017 Issue of the FreeBSD Journal is now available. Don't
miss articles on FreeBSD's Firewall Feast, CADETS: Blending Tracing and
Security on FreeBSD, Toward Oblivious Sandboxing with Capsicum, and
more. (https://www.FreeBSDfoundation.org/past-issues/security/)
Did you miss the March/April issue? Check out articles on CFEngine,
Puppet on FreeBSD, Vagrant, and more!
(https://www.FreeBSDfoundation.org/past-issues/configuration-management/)
As a recent addition of functionality, browser-based subscribers now
have the ability to download and share PDFs of the articles!
Sample Issue! If you've ever wanted to read through an entire issue of
the FreeBSD Journal, now's your chance. Download the sample issue from
https://mydigitalpublication.com/publication/?i=296880#{"issue_id":296880,"numpages":1,"page":1}
and be sure to share with your friends and colleagues. Not a subscriber?
Sign up today at https://www.FreeBSDfoundation.org/journal/.
More information about the Foundation's doings and goings-on can be
found in our own quarterly newsletter, linked above.
__________________________________________________________________
The Postmaster Team
Links
The Postmaster Team
URL: https://www.FreeBSD.org/administration.html#t-postmaster
Contact: David Wolfskill <dhw at FreeBSD.org>
Contact: Larry Rosenman <ler at FreeBSD.org>
Contact: Ryan Steinmetz <zi at FreeBSD.org>
Contact: Eygene Ryabinkin <rea at FreeBSD.org>
Contact: Remko Lodder <remko at FreeBSD.org>
Contact: Kurt Jaeger <pi at FreeBSD.org>
Postmaster handles the mail flow for the FreeBSD project.
Clusteradm provides us with four jails: mailman, mailarchive, mx1, and
mx2. In addition, there is some part of the setup running on
freefall.FreeBSD.org. The system uses postfix, mailman, spamassassin,
and some other tools from the ports tree to handle the mail flow. We
use a very small, non-public Subversion repository for parts of the
configuration.
During Q2, Larry Rosenman, Kurt Jaeger, Eygene Ryabinkin, Remko Lodder
and Ryan Steinmetz joined the Postmaster Team, and Florian Smeets left
the Postmaster Team.
Thanks to Florian for his long service in that role! David Wolfskill is
planning to leave the role as soon as the new team members are settled.
Vsevolod Stakhov plans to provide us with support to integrate rspamd
into the setup, as well.
The workload for the Postmaster Team is not high, but the complexity of
the setup has its own demands.
Open tasks:
1. We need to improve our internal documentation of workflows and
processes.
2. We should consider adding some monitoring to provide quarterly
numbers on the mail flow.
__________________________________________________________________
Projects
64-bit Inode Numbers
Links
Phabricator Review
URL: https://reviews.FreeBSD.org/D10439
Contact: Gleb Kurtsou <gleb at FreeBSD.org>
Contact: Konstantin Belousov <kib at FreeBSD.org>
Contact: Kirk McKusick <mckusick at FreeBSD.org>
The 64-bit inode project was completed and merged into FreeBSD 12 on
May 23, 2017. It extends the ino_t, dev_t, and nlink_t types to be
64-bit integers. It modifies the struct dirent layout to add a d_off
field, increases the size of d_fileno to 64 bits, increases the size of
d_namlen to 16 bits, and changes the required alignment of the
structure. It increases the struct statfs f_mntfromname[] and
f_mntonname[] array lengths from MNAMELEN to 1024.
ABI breakage is mitigated by providing compatibility using versioned
symbols, ingenious use of the existing padding in structures, and
employing various other tricks. Unfortunately, not everything can be
fixed, especially outside the base system. For instance, third-party
APIs which pass struct stat as parameters are broken in backward- and
forward-incompatible ways.
The ABI for kinfo-consuming sysctl MIBs is changed in a
backward-compatible way, but there is no general mechanism to handle
other sysctl MIBS which return structures where the layout has changed.
In our consideration, this breakage is either in management interfaces,
where we usually allow ABI slippage, or is not important.
The layout of struct xvnode changed, and no compatibility shims are
provided.
For struct xtty, the dev_t tty device member was reduced to be just
uint32_t. It was decided that maintaining ABI compatability in this
case is more useful than reporting a 64-bit dev_t value, for the sake
of pstat.
Updating note: strictly follow the instructions in UPDATING. Build and
install the new kernel with the COMPAT_FREEBSD11 option enabled, then
reboot, and only then install the new world.
Credits: The 64-bit inode project, also known as ino64, started life
many years ago as a project by Gleb Kurtsou (gleb). Kirk McKusick
(mckusick) then picked up and updated the patch, and acted as a
flag-waver. Feedback, suggestions, and discussions were carried out by
Ed Maste (emaste), John Baldwin (jhb), Jilles Tjoelker (jilles), and
Rick Macklem (rmacklem). Kris Moore (kris) performed an initial ports
investigation followed by an exp-run by Antoine Brodin (antoine).
Essential and all-embracing testing was done by Peter Holm (pho). The
heavy lifting of coordinating all these efforts and bringing the
project to completion were done by Konstantin Belousov (kib).
This project was sponsored by The FreeBSD Foundation (emaste, kib).
__________________________________________________________________
Capability-Based Network Communication for Capsicum/CloudABI
Links
ARPC: GRPC-Like RPC Library That Supports File Descriptor Passing
URL: https://github.com/NuxiNL/arpc
Flower: A Label-Based Network Backplane
URL: https://github.com/NuxiNL/flower
Contact: Ed Schouten <ed at nuxi.nl>
One of the weaknesses of Capsicum and CloudABI is that it is not easy
to develop applications that need to make outgoing network connections,
since system calls like connect() and sendto() are disabled. Though we
can sometimes work around this by ensuring that the sandboxed process
already possesses socket file descriptors on startup, this does not
allow the destination process to be restarted, moved to a different
network address, be load balanced, etc..
Coming up with a solution for this is quite important for me, as I am
currently working on making CloudABI work on top of Kubernetes,
Google's open source cluster management suite. The idea is that
Kubernetes will schedule CloudABI processes instead of Docker
containers. All of these CloudABI processes will have their
dependencies on other services in the cluster injected explicitly,
making internal communication very secure. All of this is intended to
work on FreeBSD as well, of course!
To solve this problem, I've been working on a daemon called Flower
(read: flow-er) that allows software to register services and connect
to them. Servers are identified by a set of labels with values (e.g.,
{datacenter: 'frankfurt', service: 'mysql'}). Clients can
connect these
servers by providing the corresponding label(s). Flower's security
model is capability-based, just like Capsicum. The ability to bind and
connect can be limited by permanently constraining labels to certain
values.
Flower has been designed not to act as a proxy. It does not copy any
data. It merely forwards existing socket file descriptors or creates
UNIX socket pairs and hands these out to its clients and servers. To
realize this, processes communicate with Flower using an RPC library
called ARPC. ARPC is a very simple clone of Google's GRPC, with the
special feature that messages (Protobufs) can have file descriptors
attached.
This project was sponsored by Nuxi, the Netherlands.
Open tasks:
1. Finish implementing the Flower code.
2. Integrate Flower with the Kubernetes/CloudABI runtime.
3. Release the Kubernetes/CloudABI runtime as open source software.
__________________________________________________________________
Ceph on FreeBSD
Links
Ceph Main Site
URL: http://ceph.com
Main Repository
URL: https://github.com/ceph/ceph
My FreeBSD Fork
URL: https://github.com/wjwithagen/ceph
Contact: Willem Jan Withagen <wjw at digiware.nl>
Ceph is a distributed object store and file system designed to provide
excellent performance, reliability and scalability.
* Object Storage
Ceph provides seamless access to objects using native language
bindings or radosgw, a REST interface that is compatible with
applications written for S3 and Swift.
* Block Storage
Ceph's RADOS Block Device (RBD) provides access to block device
images that are striped and replicated across the entire storage
cluster.
* File System
Ceph provides a POSIX-compliant network file system that aims for
high performance, large data storage, and maximum compatibility
with legacy applications.
I started looking into Ceph because the HAST solution with CARP and
ggate did not really do what I was looking for. I aim to run a Ceph
storage cluster of storage nodes that are running ZFS, with user
workstations running bhyve on RBD disks that are stored in Ceph.
Compiling for FreeBSD will now build most of the tools available in
Ceph.
The most important changes since the last report are:
* Ceph has released release candidate v12.1.0 (aka Luminous); the
corresponding packaging is sitting in my tree waiting for Luminous
to be actually released.
* ceph-fuse works, and allows mounting of cephfs filesystems. The
speed is not impressive, but it does work.
* rbd-ggate is available to create a Ceph rbd backed device.
rbd-ggate was submitted by Mykola Golub. It works in a rather
simple fashion: once a cluster is functioning, rbd import and
rbd-ggate map are used to create ggate-like devices backed by the
Ceph cluster.
Other improvements since the previous report:
* Some bugs in the init-ceph code (needed for rc.d) are being fixed.
* RBD and rados are functioning.
* The needed compatability code was written so that FreeBSD and Linux
daemons can operate together in a single cluster.
* More of the awkward dependancies on Linux-isms are deleted -- only
/bin/bash is there to stay.
The next forthcoming official release of Ceph is called Luminous
(v12.1.0). As soon as it is available from upstream, a port will be
provided for FreeBSD.
To get things running on a FreeBSD system, run pkg install
net/ceph-devel or clone https://github.com/wjwithagen/ceph, check out
the wip.freebsd.201707 branch, and build manually by running
./do_freebsd.sh in the checkout root.
Parts not (yet) included:
* KRBD -- but rbd-ggate is usable in its stead.
* BlueStore -- FreeBSD and Linux have different AIO APIs, and that
incompatibility needs to be resolved somehow. Additionally, there
is discussion in FreeBSD about aio_cancel not working for all
device types.
Open tasks:
1. Run integration tests to see if the FreeBSD daemons will work with
a Linux Ceph platform.
2. Investigate the keystore, which can be embedded in the kernel on
Linux and currently prevents building Cephfs and some other parts.
The first question is whether it is really required, or if only
KRBD requires it.
3. Scheduler information is not used at the moment, because the
schedulers work rather differently between Linux and FreeBSD. But
at a certain point in time, this will need some attention (in
src/common/Thread.cc).
4. Improve the FreeBSD init scripts in the Ceph stack, both for
testing purposes and for running Ceph on production machines. Work
on ceph-disk and ceph-deploy to make it more FreeBSD- and
ZFS-compatible.
5. Build a test cluster and start running some of the teuthology
integration tests on it. Teuthology wants to build its own libvirt,
and that does not quite work with all the packages FreeBSD already
has in place. There are many details to work out here.
6. Design a virtual disk implementation that can be used with bhyve
and attached to an RBD image.
__________________________________________________________________
DTS Updates
Contact: Emmanuel Vadot <manu at FreeBSD.org>
DTS (Device Tree Source) files provide a human-readable source
description of the hardware resources for a given computer system (such
as ARM- or MIPS-based embedded boards). The DTS source representation
must be compiled into a binary format in order to be linked into the
kernel and used to locate devices at runtime.
The DTS files in FreeBSD were updated to match the versions from Linux
4.11, to represent more modern devices and provide more accurate
representations.
__________________________________________________________________
Kernel
Coda revival
Links
GitHub Repository
URL: https://github.com/trasz/FreeBSD/tree/coda
Contact: Edward Tomasz Napiera?a <trasz at FreeBSD.org>
Coda is a distributed file system developed as a research project at
Carnegie Mellon University, descended from a older version of the
Andrew File System. It got dropped from FreeBSD some five years ago,
due to not having been adopted for a MPSAFE world. The focus for this
current project is to bring it back into sufficiently workable shape
that it could return to the kernel. It is currently in a working
condition. Work is underway to test it better, fix whatever issues are
found, and commit it to 12-CURRENT.
This project was sponsored by Chalmers University of Technology.
Open tasks:
1. Additional testing.
2. Update the userspace components (net/coda_client and
net/coda_server).
__________________________________________________________________
FreeBSD Driver for the Annapurna Labs ENA
Links
Enhanced Networking Guide
URL:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking.html
Contact: Marcin Wojtas <mw at semihalf.com>
Contact: Michal/ Krawczyk <mk at semihalf.com>
The ENA (Elastic Network Adapter) is a 25G SmartNIC developed by
Annapurna Labs and is based on a custom ARMv8 chip. This is a
high-performance networking card available in the AWS offerings. It
introduces enhancements in network utilization scalability on EC2
machines under the control of various operating systems, in particular
FreeBSD.
The goal of FreeBSD enablement is to provide top performance and a wide
range of monitoring and management features such as:
* multiple queue modes
* hardware offloads (rx and tx checksum)
* an admin queue
* asynchronous notifications
* robust hardware access
* a scalable number of MSI-X vectors
* hardware counters
* watchdog mechanism
* LRO
* RSS
The driver is available in the kernel source tree as of r318647.
This project was sponsored by Annapurna Labs -- an Amazon company.
Open tasks:
1. Add RSS configuration from userspace (via sysctls).
2. Add support for LLQ mechanisms.
__________________________________________________________________
Intel 10G Driver Update
Links
Commit Adding X553 ix/ixv Support
URL: https://reviews.FreeBSD.org/D11232
Contact: Chris Galazka <krzysztof.galazka at intel.com>
Contact: Jeb Cramer <jeb.j.cramer at intel.com>
The ix and ixv network interface drivers support a variety of Intel
network interfaces, with line speeds at 10 Gbit/second.
This quarter, the drivers gained support for the X553 network
interface, which is found on System-on-a-Chip devices based on the
Denverton platform. This update should allow FreeBSD to be more useful
on a new class of hardware platform.
Work is also underway to convert these drivers to use the iflib network
driver library, which should ease future maintenance of the drivers, as
well as the network subsystem as a whole.
__________________________________________________________________
pNFS Server Plan B
Links
Testing Instructions
URL: http://people.FreeBSD.org/~rmacklem/pnfs-planb-setup.txt
Contact: Rick Macklem <rmacklem at FreeBSD.org>
Parallel NFS (pNFS) is an extension to the NFSv4 protocol that allows
for file accesses within a single logical mount to be performed against
multiple file servers, with the potential for data access to occur in
parallel. The pNFS "layout" in use specifies how the division
occurs,
with metadata operations occurring against the main server, and bulk
data operations (read/write/setattr/etc.) occurring via a
layout-specific scheme between the client and the data servers.
My first attempt at a pNFS server using GlusterFS was a dud. It worked,
but performance was so poor that it was not usable. This attempt that I
call "Plan B", only uses FreeBSD, with one FreeBSD server handling
the
metadata operations and multiple FreeBSD servers configured to serve
data, is now ready for third-party testing. If testing by third parties
goes well, I anticipate the code will be merged into FreeBSD head in
time for FreeBSD 12. Fairly recent FreeBSD or Linux systems should be
usable as pNFS clients for testing. This server supports the File
Layout, which is supported by both of these clients.
There is no support for the Flex Files Layout or mirroring at this
time. I hope to use the Flex Files Layout to add mirroring support over
the next year or so. Striping is not supported, and I have no plans for
implementing this at the moment.
The patched FreeBSD sources may now be accessed for testing via either
Subversion or download of a gzipped tarball. They consist of a patched
kernel and nfsd and can be used on any FreeBSD 11 or later system.
Open tasks:
1. Testing by others will be needed, now that the code is available.
__________________________________________________________________
Architectures
FreeBSD on Marvell Armada38x
Contact: Marcin Wojtas <mw at semihalf.com>
Contact: Zbigniew Bodek <zbb at FreeBSD.org>
Work proceeds to finalize the process of bringing support for the
Marvell Armada38x platform into FreeBSD head.
The most important parts of the recent effort are:
* Add the network driver (NETA)
* Enable coherent busdma operation for all ARMv7 SoCs
* Add various low-level optimizations, such as L1 cache prefetch and
MBUS quirks
* Enable PL310 L2 cache controller
* Add SDHCI support
* Fixes for the e6000sw driver and a rework of its PHY handling
* Support multi-port PCIe operation
* Various fixes and enhancements of the common Marvell code
* Fix and enable support for performance counters (HWPMC)
This project was sponsored by Stormshield, Semihalf, and Netgate.
__________________________________________________________________
FreeBSD/arm64
Links
FreeBSD arm64 Wiki Page
URL: https://wiki.FreeBSD.org/arm64
Contact: Andrew Turner <andrew at FreeBSD.org>
Support for the Privilege Access Never (PAN) feature was added. This
stops the kernel from accessing userspace memory, except through
specific instructions. This helps security by only allowing access to
userspace via the correct accessor functions. This is enabled on all
supported CPUs that implement ARMv8.1 or later.
The pmap code now supports the Unprivileged execute-never (UXN) and
Privileged execute-never (PXN) bits in the page tables. These bits stop
userspace and the kernel, respectively, from executing instructions on
any marked page.
The performance of the pmap layer has been improved. Many of the cache
handling function calls have been removed. Some were needed early on to
work around other bugs that have now been fixed. The removal of these
calls has led to a large performance improvement.
The kernel now uses crc32c instructions where appropriate. These are an
optional set of instructions to perform crc32c checksumming quickly
without using a lookup table.c
The VM_MEMATTR_WRITE_THROUGH memory attribute is now supported. This is
used to allocate memory for the framebuffer. Previously, the kernel
would use cached memory; however, this leads to visual artifacts. The
write-through flag fixes these by writing data out to RAM.
The default linker on arm64 is now lld. This means that FreeBSD is able
to build itself with just the components in the base system, a big
milestone!
__________________________________________________________________
Userland Programs
DTC
Contact: Emmanuel Vadot <manu at FreeBSD.org>
The in-tree DTC (Device Tree Compiler) was switched to use the
BSD-licensed version by default. (The previous default DTC is licensed
under the GPL.) The current version supports overlays and is able to
compile every DTS (Device Tree Source) used by the FreeBSD arm
releases. The ports GPL version was updated to the latest release
(1.4.4). The in-tree GPL version is still present but the goal is to
remove it before FreeBSD 12.0.
__________________________________________________________________
Using LLVM's LLD Linker as FreeBSD's System Linker
Links
FreeBSD lld Wiki Page
URL: https://wiki.FreeBSD.org/LLD
FreeBSD/LLD Tracking PR (LLVM Bugzilla)
URL: http://llvm.org/pr23214
Exp-Run Request Using lld as /usr/bin/ld
URL: https://bugs.FreeBSD.org/214864
Contact: Rafael Esp?ndola <rafael.espindola at gmail.com>
Contact: Ed Maste <emaste at FreeBSD.org>
LLD is the linker in the LLVM family of projects. It is a
high-performance linker that supports the ELF, COFF and Mach-O object
formats. It is broadly compatible with the common linkers used for each
file format. For ELF this is the GNU Binary File Descriptor (BFD) ld
and GNU gold. However, LLD's authors are not constrained by strict
compatibility where it would hamper performance or desired
functionality.
LLD is now used as the default system linker for FreeBSD/arm64 and can
link a working kernel, kernel modules, and userland for FreeBSD/amd64.
LLD can also link a working kernel and modules (but not userland) for
FreeBSD/arm and FreeBSD/i386.
Work is ongoing to address ports that do not build with LLD as the
system linker (either by fixing the port, or configuring the port to be
linked by GNU ld).
For FreeBSD 12.0 we expect to use LLD as the system linker for the same
set of architectures that use Clang by default: 32- and 64-bit arm and
x86.
This project was sponsored by The FreeBSD Foundation.
Open tasks:
1. Fix libtool to detect LLD and pass the same command line arguments
as for GNU ld and gold.
2. Investigate the remaining amd64 and arm64 port build failures.
3. Investigate and improve LLD on i386 and arm, before the creation of
the stable/12 branch.
4. Investigate and improve LLD on all other architectures.
5. Extensive testing.
__________________________________________________________________
Ports
A New USES Macro for Porting Cargo-Based Rust Applications
Links
Rust Homepage
URL: https://www.rust-lang.org/
Cargo Homepage
URL: https://crates.io/
Alacritty Homepage
URL: https://github.com/jwilm/alacritty
Exa Homepage
URL: https://the.exa.website/
Ripgrep Homepage
URL: https://github.com/BurntSushi/ripgrep
Short Screencast About How to Use the USES=cargo Macro
URL: https://asciinema.org/a/SM2sOLi6iBUOmGWrxn5W1QI8U
Contact: Tobias Kortkamp <tobik at FreeBSD.org>
Support in the Ports Collection for applications written in the Rust
programming language that use Rust's package manager Cargo was added,
via a new USES=cargo setting. The work is based on the cargo module
from the OpenBSD ports tree.
This should significantly ease the porting of Rust applications, as
previously porters had to create their own tarball of the application's
dependencies or find other manual ways of bringing them in.
Several new ports were added that use it, for example:
* Alacritty, a GPU-accelerated terminal emulator
* Exa, a modern replacement for ls
* Ripgrep, a line-oriented search tool that combines the usability of
The Silver Searcher with the raw speed of GNU grep
Open tasks:
1. Add documentation for the new feature.
__________________________________________________________________
GCC (GNU Compiler Collection)
Links
GCC Homepage
URL: https://gcc.gnu.org
Issue Tracker Entry for the Update to GCC 6
URL: https://bugs.FreeBSD.org/bugzilla/show_bug.cgi?id=219275
GCC 5 Changelog
URL: https://gcc.gnu.org/gcc-5/changes.html
GCC 5 Porting Issues
URL: https://gcc.gnu.org/gcc-5/porting_to.html
Contact: Gerald Pfeifer <gerald at FreeBSD.org>
Contact: Andreas Tobler <andreast at FreeBSD.org>
The default version of GCC in the Ports Collection (the one requested
by USE_GCC=yes and various USES=compiler invocations) has been updated
from GCC 4.9.4 to GCC 5.4.
This new major version brings many new capabilities and improvements,
as well as some changes that may require adjustments. The latter
category includes many new compiler warnings, significant improvements
to inter-procedural optimizations, and link-time optimization.
The default mode for C is now -std=gnu11 instead of -std=gnu89. The C++
front end has full C++14 language support, including C++14 variable
templates, C++14 aggregates with non-static data member initializers,
C++14 extended constexpr, and more. The Standard C++ Library
(libstdc++) has full C++11 support and experimental full C++14 support.
It uses a new ABI by default.
The lang/gcc port now is a meta-port that pulls in the respective
lang/gccX port (based on the setting of $GCC_DEFAULT) and defines gcc,
g++, and gfortran as symlinks to the respective versioned binaries.
This is the end of a long journey establishing this infrastructure,
which is now similar that used by the python ports, for example. Having
the new infrastructure makes upgrading the default, as well as locally
adjusting the default version, a lot easier.
gcc8-devel has been added, and armv6hf support removed, and we made
adjustments for newer versions of FreeBSD. Also of note are various
cleanups and changes to improve the robustness of our packages and the
addition of support for aarch64 to many ports.
Thanks to dim@, jbeich@, tijl@, mat@, miwi@, linimon@ for assisting
with this work.
Open tasks:
1. The update of the default version of GCC from GCC 5.4 to GCC 6.4 is
stalled, unfortunately. The work on the GCC and insfrastructure
sides is complete, but unfortunately there are a number of broken
ports that need to be adjusted/fixed. Any help is very appreciated;
see PR 219275 for details.
__________________________________________________________________
GNOME on FreeBSD
Links
FreeBSD GNOME Website
URL: http://www.FreeBSD.org/gnome
Development Repository
URL: https://github.com/FreeBSD/FreeBSD-ports-gnome
Upstream Build Bot
URL: https://wiki.gnome.org/Projects/Jhbuild/FreeBSD
USE_GNOME Porter's Handbook Chapter
URL:
https://www.FreeBSD.org/doc/en_US.ISO8859-1/books/porters-handbook/using-gnome.html
Contact: FreeBSD GNOME Team <FreeBSD-gnome at FreeBSD.org>
The FreeBSD GNOME Team maintains the GNOME, MATE, and CINNAMON desktop
environments and graphical user interfaces for FreeBSD. GNOME 3 is part
of the GNU Project. MATE is a fork of the GNOME 2 desktop. CINNAMON is
a desktop environment using GNOME 3 technologies but with a GNOME 2
look and feel.
After a period of not much activity, this quarter we started a little
experiment in how we merge ports from the development repo to the
FreeBSD Ports Collection. Instead of merging everything in one big
commit, we have been updating the GNOME ports one at a time or in small
groups. For example, the GTK+ stack and the Evolution Suite were
updated as groups, and all the gnome-games components were done in one
commit. It might be a bit more work preparing and testing the updates,
but on the plus side, it easy to keep track of what is going on, and
allows us to pay attention to the details. It should also make it
easier to commit smaller changes.
This quarter started with the update of GTK+ 3 to 3.22.15, and the
underlying libraries to their latest stable versions. After the GTK+
update, work started on getting newer versions of other GNOME
applications updated.
The webkit2-gtk3 port was first updated to the 2.14 series and later to
2.16.3, which is the latest stable version. This step was needed
because 2.16 couldn't be built on FreeBSD 10.3 without some required
framework changes.
harfbuzz-icu was split off from the main harfbuzz port. This drops the
heavy icu dependency from the main harfbuzz port.
A longstanding GLib/gio bug was fixed that had previously caused
crashes of gnome-shell and other applications when share/applications
was modified, as happens on pkg install or deinstall.
Many of these updates are based on work previously done in the Gnome
development branch by Ruslan Makhmatkhanov, Gustau Perez and Koop Mast.
Open tasks:
1. Porting of Mutter/Gnome-shell/GDM 3.24 is complete. Unfortunately,
GDM is blocking the update because of a "handoff" bug to the
session after login.
2. Fix the printer submenu in gnome-control-center. As a workaround,
system-config-printer can be used to configure printers.
3. MATE 1.18 is being QA tested and should arrive in early July.
__________________________________________________________________
KDE on FreeBSD
Links
KDE on FreeBSD Website
URL: https://FreeBSD.kde.org/
KDE Ports Staging Area
URL: https://FreeBSD.kde.org/area51.php
KDE on FreeBSD Wiki
URL: https://wiki.FreeBSD.org/KDE
KDE/FreeBSD Mailing List
URL: https://mail.kde.org/mailman/listinfo/kde-FreeBSD
Development Repository
URL: https://github.com/FreeBSD/FreeBSD-ports-kde
KDE's Continous Integration Dashboard
URL: https://build.kde.org
Blog Post on Using the Ninja CMake Generator
URL: https://euroquis.nl/bobulate/?p=1600
Contact: KDE on FreeBSD Team <kde at FreeBSD.org>
The KDE on FreeBSD team focuses on packaging KDE and Qt, and making
sure that their experience on FreeBSD is as good as possible.
This quarter, in addition to the regular updates to the KDE, Qt, and
related ports, there have also been some changes behind the scenes: our
development repository has moved to GitHub, and FreeBSD is now part of
KDE's official continuous integration (CI infrastructure).
After the X.Org and GNOME ports teams, the KDE on FreeBSD team has
moved its development repository to GitHub. This should make it easier
for others to collaborate with us via pull requests, and by basing all
our changes on top of the official ports tree we also hope this reduces
the amount of conflicts and churn we need to deal with when landing big
updates across the tree. We would like to thank iXsystems for hosting
and supporting our area51 Subversion repository for many years.
FreeBSD has finally joined KDE's CI (Continuous Integration) system as
a tier-1 platform. KDE CI builds all the KDE sources -- 70 frameworks,
the KDE Plasma Desktop and a plethora of KDE Applications --
continuously, straight from KDE's git repositories. There is strong
commitment from upstream and the downstream KDE-FreeBSD team to reduce
the amount of patching in the KDE ports to as little as possible. The
first effects are being felt in expanding the set of unit tests to
include FreeBSD-specific situations, and in extending Qt to handle
FreeBSD filesystems better. In addition to the KDE sysadmins, we would
also like to extend our thanks to Adriaan de Groot, who is both a KDE
committer and part of our KDE on FreeBSD team, for spearheading these
efforts.
The following big updates landed in the ports tree this quarter:
* CMake was updated to 3.8.0 and 3.8.2
* KDE Frameworks was updated to 5.33, 5.34 and 5.35
* The Calligra office suite was updated to 3.0.1, the first release
in the ports tree to be based on KDE Frameworks 5, and the latest
stable release upstream
* The Konversation IRC client was updated to 1.7.2, the latest
upstream release and the first ports version based on KDE
Frameworks 5
* KchmViewer was updated to 7.7, which is based on KDE Frameworks 5
* LabPlot was updated to 2.3.0 and 2.4.0, and is now based on KDE
Frameworks 5
* QtCreator was upated to 4.2.2 and subsequently to 4.3.0
* py-sip was updated to 4.19.2, PyQt4 to 4.12 and PyQt5 to 5.7.1
* Several fixes for ARMv6 landed in the Qt4 and Qt5 ports -- thanks
to Mika?l Urankar
After several review rounds and exp-runs, Tobias Berner (tcberner@)
finally made the Ninja generator the default for CMake-based ports, so
that devel/ninja is used instead of (g)make in most cases. This should
make most builds faster, even if only by a small margin. Adriaan de
Groot also wrote a blog post about the change.
__________________________________________________________________
New Port: FRRouting
Links
FRRouting Home Page
URL: https://frrouting.org/
Contact: Olivier Cochard-Labb? <olivier at cochard.me>
FRRouting (FRR), a Quagga fork, is an IP routing protocol suite for
Linux and Unix platforms which includes protocol daemons for BGP,
IS-IS, OSPF and RIP (LPD and PIM support need to be fixed on FreeBSD).
FRR is a Linux Foundation Collaborative Project with contributors
including 6WIND, Architecture Technology Corporation, Big Switch
Networks, Cumulus Networks, LabN Consulting, NetDEF
(OpenSourceRouting), Orange, Volta Networks, and other companies.
This project was sponsored by Orange.
__________________________________________________________________
PHP Ports: Help Improving QA
Links
My Patreon Page
URL: https://www.patreon.com/TorstenZuehlsdorff
Contact: Torsten Z?hlsdorff <tz at FreeBSD.org>
As maintainer of the PHP ports, I first want to thank you all for the
great feedback and patches I receive, in many forms. You keep my life
interesting!
In the past few months I learned a lot about various configurations,
settings and bugs. Also, sadly, there are always PRs, patches and
emails left unanswered, because of missing time on my side.
I want to improve the situation by adding more automatic QA testing,
but I need help to do so. Please send me your non-standard
PHP-configurations or describe your exotic setups! These can be as
simple as changed default versions, like LibreSSL instead of OpenSSL or
the GCC version used for compiling. I, for example, always use another
PostgreSQL-version than the default (and always PHP 7.1). Of course,
this also covers port options set in an non-default way or setups that
change variables to allow for multiple PHP installations, etc..
I plan to test on all supported FreeBSD versions, so you only need to
mention if you are using an unsupported version.
Note: Since PHP 7.2 is coming (hopefully on schedule), I will test PHP
7.2 from the onset with all the provided configurations, too.
Open tasks:
1. Document the various configurations to be tested.
2. Setup the automatic QA infrastructure.
__________________________________________________________________
Rust
Links
Wiki Portal
URL: https://wiki.FreeBSD.org/Rust
Guide to Bootstrap Rust on FreeBSD
URL: https://gist.github.com/dumbbell/b587da50ef014078da9e732a4331ebad
Bug Report to Track Progress on Bootstrapping
URL: https://bugs.FreeBSD.org/bugzilla/show_bug.cgi?id=216143
Upstream Discussion of API/ABI-Breaking Changes
URL:
https://internals.rust-lang.org/t/pre-rfc-target-extension-dealing-with-breaking-changes-at-os-level/5289
Contact: FreeBSD Rust team <rust at FreeBSD.org>
Rust was updated to 1.18.0 and Cargo to 0.19.0, the latest versions at
the time of this writing.
lang/rust was enabled on FreeBSD/aarch64 and work has continued on
devel/cargo to achieve the same. We are also making slow progress to
add support for even more platforms.
Discussion has started upstream to support API/ABI-breaking changes
between major releases of operating systems. For instance, this is
required to be able to target both FreeBSD 11.x and 12.x, which have
ABI changes involving important structures. Once support is added
upstream, it will be possible to target a specific ABI and do
cross-compilation.
lang/rust-nightly was marked as broken for now. We need to revisit how
the port is built so we can use the x.py script as recommended by
upstream.
Tobias Kortkamp (tobik@) created the USES=cargo setting to make it easy
to add Rust applications to the Ports Collection. This is further
detailed in a separate entry in this quarterly status report.
The compiler, rustc, is crashing sometimes when there is a compilation
error. Therefore, there is a bit of work to do to improve its
stability.
There is some code duplication between the lang/rust* and devel/cargo
Makefiles. These all deserve a bit of cleanup, and it might be useful
to create a USES=rust Makefile helper.
Open tasks:
1. Bootstrap Rust on more platforms.
2. Investigate compiler crashes.
3. Investigate how to speed up lang/rust* compilation times.
__________________________________________________________________
sndio Support in the FreeBSD Ports Collection
Links
Sndio Homepage
URL: http://www.sndio.org
Sndio Paper
URL: https://www.openbsd.org/papers/asiabsdcon2010_sndio.pdf
Comprehensive and Biased Comparison of OpenBSD and FreeBSD (Section 17)
URL:
https://www.bsdfrog.org/pub/events/my_bsd_sucks_less_than_yours-AsiaBSDCon2017-paper.pdf
Contact: Tobias Kortkamp <tobik at FreeBSD.org>
sndio is a small audio and MIDI framework that is part of the OpenBSD
project. It provides a lightweight audio and MIDI server, sndiod. It
currently supports OpenBSD, FreeBSD, DragonFly BSD, and Linux.
The porting effort to FreeBSD and OSS started last year and the sndio
backend support in the FreeBSD Ports Collection can now be considered
good enough for daily use.
Sndio offers network transparency through sndiod, which provides an
easy way to share your audio devices with other machines/VMs/jails on
your network. However, applications and libraries need to support
playing and recording through it. To that end, I submitted several
patches to various ports over the course of the last year.
Here's a short selection of ports that now support sndio in the FreeBSD
Ports Collection:
* Most games, via audio/openal-soft, devel/sdl12, and devel/sdl20.
* GStreamer-based applications and WebKit-based browsers through two
new GStreamer plugins (audio/gstreamer1-plugins-sndio and
audio/gstreamer-plugins-sndio).
* Firefox, Firefox ESR, Seamonkey, Chromium, and Iridium. The
browsers currently lack or have a non-functional OSS backend. Sndio
support provides a BSD-native alternative to the ALSA and
PulseAudio backends.
* Video players like VLC, Totem, mpv, mplayer, etc..
* Audio players like Clementine, cmus, mpd, mpg123, siren, xmp, etc..
* SoX.
* Shairport Sync, through a newly implemented backend.
* JACK.
* PulseAudio, through audio/pulseaudio-module-sndio.
Open tasks:
1. Commit a backport of Kodi's new sndio backend to the Ports
Collection.
2. If you maintain or use an audio-related port, consider checking
whether it includes an sndio backend, and adding an SNDIO option.
Thanks to the OpenBSD developers, several open-source projects
already include one, so adding it might be very easy to do.
__________________________________________________________________
TensorFlow
Links
TensorFlow PR
URL: https://bugs.FreeBSD.org/bugzilla/show_bug.cgi?id=219609
Phabricator Review
URL: https://reviews.FreeBSD.org/D11194
Prebuilt Packages
URL: https://github.com/amutu/tf-FreeBSD-pkg
TensorFlow Upstream
URL: https://www.tensorflow.org
Contact: Jov <amutu at amutu.com>
As described on its website, "TensorFlow(TM) is an open source software
library for numerical computation using data flow graphs. Nodes in the
graph represent mathematical operations, while the graph edges
represent the multidimensional data arrays (tensors) communicated
between them. The flexible architecture allows you to deploy
computation to one or more CPUs or GPUs in a desktop, server, or mobile
device with a single API. TensorFlow was originally developed by
researchers and engineers working on the Google Brain Team within
Google's Machine Intelligence research organization for the purposes of
conducting machine learning and deep neural networks research, but the
system is general enough to be applicable in a wide variety of other
domains as well."
TensorFlow now is the most popular platform/library for machine
learning and AI. There are official binaries for Linux, Mac, Windows,
and Android, but no official support for FreeBSD. For the last several
months, I have done some work to make TensorFlow available on FreeBSD.
Some notable items include:
* bazel was patched to not depend on /proc at build time. bazel is a
build tool made by Google. It uses /proc to get path-to-self when
building C++ code, but mounting /proc is usually not allowed when
building as an unprivileged user.
* TensorFlow can now be built on FreeBSD 10.x by using clang38 as the
default bazel cross-build tool.
* Patch the bazel workspace files to allow TensorFlow to be built
using offline third-party dependencies. This work is needed because
the FreeBSD Ports framework does not allow network access except
during the fetch stage.
* Fix the build on FreeBSD i386.
* Make TensorFlow build with either Python 2 or Python 3.
* Update to the latest version, which is tensorflow-1.2.0.
TensorFlow can now be run on FreeBSD in CPU-only mode. Some functional
tests have been performed on some combinations of FreeBSD 10.3-RELEASE
and 11.0-RELEASE, amd64 and i386, and Python 2.7 and Python 3.6.
This port would not be possible without substantial assistance from
bapt@, lwhsu@, mat@, and koobs@ -- thank you for your advice, review,
and help! You are very nice and I learned a lot about FreeBSD and the
Ports framework from you.
Open tasks:
1. Review, test, comment, and most importantly, commit to the Ports
Collection.
2. Fix OpenCL (GPU acceleration) support on FreeBSD.
3. Port tensorflow-serving, which is a flexible, high-performance
serving system for machine learning models produced by TensorFlow.
4. Set up a CI for TensorFlow on FreeBSD and give early notice to
upstream when they break TensorFlow on FreeBSD.
__________________________________________________________________
Updating Port Metadata for non-x86 Architectures
Links
aarch64 Poudriere Machine
URL: http://thunderx1.nyi.FreeBSD.org/jail.html?mastername=110arm64-default
armv6 Poudriere Machine
URL: http://beefy8.nyi.FreeBSD.org/jail.html?mastername=head-armv6-default
Contact: Mark Linimon <linimon at FreeBSD.org>
I have been analyzing the error logs from ports builds for all non-x86
architectures, including both the logs published on the package build
cluster and also other builds of powerpc64 and sparc64.
From this analysis, I have marked almost all the failing ports as
either BROKEN or NOT_FOR/ONLY_FOR, as appropriate.
The intent of this work is not to make life harder for anyone, but
rather, in fact, the opposite. With these definitions in place, it is
possible to scan the poudriere bulk build output (the "Ignored
ports"
portion, in particular) and see quickly what ports are failing to build
and why. Previously, finding the exact reason why a build failed needed
some research (portsmon only analyzes failure messages on amd64).
Additionally, it is extremely difficult to work through several hundred
logs that simply say "failed to compile", "failed to
link", and so forth.
This is part of an effort to identify where we need further work to
bring sufficient Ports Collection support to, e.g., armv6 and aarch64
to bring them closer to true Tier-1 status.
To further facilitate locating patterns in the Poudriere output, I have
begun reworking some existing BROKEN/NOT_FOR/ONLY_FOR messages so that
they will sort more easily. This includes sorting the order in which
architectures appear in the lists.
Many people have been doing great work on fixing the individual ports.
I hope that my work makes their jobs somewhat easier.
__________________________________________________________________
Xfce on FreeBSD
Links
FreeBSD Xfce Project
URL: https://wiki.FreeBSD.org/Xfce
Ports Development Repository
URL: https://www.assembla.com/spaces/xfce4/subversion/source
Contact: FreeBSD Xfce Team <xfce at FreeBSD.org>
Contact: Olivier Duchateau <duchateau.olivier at gmail.com>
Xfce is a free software desktop environment for Unix and Unix-like
platforms such as FreeBSD. It aims to be fast and lightweight, while
still being visually appealing and easy to use.
During this quarter, we have kept these applications up-to-date:
* audio/xfce4-pulseaudio-plugin (0.2.5, PR219357)
* deskutils/xfce4-tumbler (0.1.32, PR219848)
* deskutils/xfce4-xkb-plugin (0.8.0, PR220071)
* sysutils/garcon (0.6.1, PR219928, and PR219334 for Mk/Uses/xfce.mk)
* textproc/xfce4-dict-plugin (0.8.0, PR220266)
* x11/xfce4-terminal (0.8.5.1, PR219312)
* x11/xfce4-whiskermenu-plugin (1.7.2, PR219347)
* x11-wm/xfce4-desktop (4.12.4, PR220290)
We have created a new Subversion tag (4.13) in order to follow the
unstable releases. The separate tag was necessary in order to support
changes in the USES=xfce infrastucture, and due to some incompatible
changes to the xfconf API. Ports following the unstable release are:
* deskutils/xfce4-tumbler (0.1.92.1)
* multimedia/xfce4-parole (0.9.2)
* sysutils/xfce4-settings (4.13.1)
* x11/libexo (0.11.3)
* x11/libxfce4menu (4.13.2)
* x11/libxfce4util (4.13.1)
* x11/xfce4-conf (4.13.2)
* x11/xfce4-dashboard (0.7.2)
* x11/xfce4-screenshooter (1.9.1)
* x11/xfce4-whiskermenu-plugin (2.1.2)
* x11-wm/xfce4-desktop (4.13.1)
* x11-wm/xfce4-panel (4.13.0)
* x11-wm/xfce4-session (4.13.0)
* x11-wm/xfce4-wm (4.13.0)
Open tasks:
1. Make the transition to Gtk3 smoother for end users.
__________________________________________________________________
Documentation
Absolute FreeBSD, 3rd Edition
Links
Status as of 30 June
URL: https://blather.michaelwlucas.com/archives/2972
Second Edition
URL: https://www.michaelwlucas.com/os/af2e
Trivial Updates
URL: https://twitter.com/search?q=%23af3e&src=typd
Contact: Michael Lucas <mwlucas at michaelwlucas.com>
I'm working on a third edition of Absolute FreeBSD. This will be a
nearly complete rewrite, thanks to the addition of little details like
ZFS, GPT, dma, GELI, new boot procedures, disk labeling, pkg(8),
blacklistd, jails, etc..
My current (delusional) plan is to have a first draft finished by the
end of October 2017, so we can have print copies for BSDCan 2018.
Open tasks:
1. Write the remaining 75% of the book.
__________________________________________________________________
Doc Version Strings Improved by Their Absence
Links
FreeBSD Documentation Project Primer
URL: https://www.freebsd.org/doc/en_US.ISO8859-1/books/fdp-primer/
Get Version Information from Subversion Metadata
URL:
https://svnweb.freebsd.org/doc/head/share/mk/doc.docbook.mk?r1=50233&r2=50232&pathrev=50233
Contact: Warren Block <wblock at FreeBSD.org>
In retrospect, our $FreeBSD$ strings in source files are kind of weird,
like a vestigial tail. The version control system stores all of that
information in metadata. Yet here we are, not only allowing the version
control system to alter our source files on every commit, but forcing
it to do so.
The reason for doing so is that the previous version control system did
it. Really.
Version control strings are a headache for translators using the new PO
toolchain. It is an ever-changing string that offers nothing to the
translation, yet can cause conflicts with earlier versions of itself.
We also had complaints about how the Handbook was always months out of
date. It was not, of course... but looking at just the version string
in the main, rarely-changing book.xml file gave that impression. We
fixed that problem last year, so the build system checks all the source
files for the latest commit, but it seems easier to not have to fix the
problem at all.
Of course, that was really only one aspect of an ongoing problem. Our
documentation build system was checking the version string in the
source file, not the metadata. In 1973, metadata, like cars not
composed chiefly of rust, had not yet been invented. I modified the
build system to extract the information from the metadata (and noted,
with some surprise, that this is a task at which Git is much better
than Subversion).
The next step was to remove the $FreeBSD$ strings from the source files
and remove the FreeBSD=%H property that forces Subversion, against its
better judgement, to substitute text in the actual contents of the
file. The version information is not lost. It lives in the metadata, so
retrieving it is as simple as svn info -- it does not need to be in the
source at all. However, as with anything that touches code or processes
which have not been touched in living memory, there was some debate
over this. At that point, I offered to remove the version strings from
the FreeBSD Documentation Project Primer book as a test.
The change allowed the zh_TW translation team to turn off the
FreeBSD=%H property on their translation and continue their work
without fighting with the version strings. Rendered versions of the
book still display the name of the last committer and the date and
revision number of the last commit, but all of that information comes
from metadata. As such, it is also more likely to be correct.
Since the change, there have not been any complaints, at least not to
me. In fairness, the removal of version strings from the FDP Primer
alone is a small change in a tiny corner of the project. Looking at it
another way, it might be that some things that seem to be necessary are
more about the comfort of familiarity than actual utility.
At present, this is strictly a change to the documentation build
toolchain and a single documentation book. However, there do not appear
to be any reason why it could not be extended to the rest of the
documents. It might even serve as tiny test of whether the expansion of
$FreeBSD$ tags is needed throughout the rest of the FreeBSD tree.
__________________________________________________________________
New Xen Handbook Section
Links
Handbook Section About FreeBSD as a Xen Host
URL:
https://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/virtualization-host-xen.html
Original Phabricator Review
URL: https://reviews.freebsd.org/D10774
Contact: Benedict Reuschling <bcr at FreeBSD.org>
FreeBSD supports the Xen hypervisor, with DomU (guest) support since
FreeBSD 8.0 and Dom0 (host) available since FreeBSD 11.0. The FreeBSD
Handbook was lacking instructions on how to run a Xen host and VMs. The
steps were outlined in the FreeBSD wiki, but needed some extra bits of
text from the upstream Xen wiki in order to form a complete guide. The
new handbook section briefly explains what Xen is, how it differs from
other hypervisors, and what features are currently available in
FreeBSD. It then goes on to describe how to set up the Dom0, as well as
detailing the guest VM support known as DomU.
Reviewers Nikolai Lifanov, Roger Pau Monn?, and Warren Block provided
valuable feedback on the initial version in Phabricator. Additional
corrections were made by Bj?rn Heidotting while translating the section
into German.
Open tasks:
1. More options for the Dom0 and DomU could be provided.
2. People should test these instructions on their hardware and provide
feedback. This would also help us get better testing of the Xen
port for FreeBSD.
__________________________________________________________________
Miscellaneous
BSD Meetups at Rennes (France)
Links
First Event
URL: https://www.meetup.com/fr-FR/Meetup-BSD-Rennes/events/239248155/
Second Event
URL: https://www.meetup.com/fr-FR/Meetup-BSD-Rennes/events/240202297/
Contact: Mathieu Kerjouan <contact at steepath.eu>
Two meetups dedicated to BSD systems were held in Rennes, France. The
first one was hosted in the OVH office in Rennes and included
presentations on multiple subjects: the non-technical history of
FreeNAS (presented by olivier@), how OVH is using ZFS, an introduction
to jails, and a use case for BGP/bird on FreeBSD.
The second meetup, also hosted in the OVH office, presented these
subjects: how to create a FreeBSD port (presented by jadawin@), how OVH
is using Finite State Machines for managing their storage system,
network high-availability with FreeBSD, and a jail tutorial by means of
a demonstration running 200 OSPF (using net/bird) routers using jails
and vnets on a small PC Engines APU2 system with only 4 CPU cores (1Ghz
AMD) and 4GB RAM).
This project was sponsored by OVH.
__________________________________________________________________
Third-Party Projects
Many projects build upon FreeBSD or incorporate components of FreeBSD
into their project. As these projects may be of interest to the broader
FreeBSD community, we sometimes include brief updates submitted by
these projects in our quarterly report. The FreeBSD project makes no
representation as to the accuracy or veracity of any claims in these
submissions.
HardenedBSD
Links
HardenedBSD Homepage
URL: https://hardenedbsd.org/
SafeStack
URL: http://clang.llvm.org/docs/SafeStack.html
HardenedBSD Tor Hidden Service
URL: http://t3a73imee26zfb3d.onion/
Projects HardenedBSD Would Like Help With
URL:
https://github.com/HardenedBSD/hardenedBSD/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22
Contact: Shawn Webb <shawn.webb at hardenedbsd.org>
Contact: Oliver Pinter <oliver.pinter at hardenedbsd.org>
HardenedBSD is a derivative of FreeBSD that gives special attention to
security-related enhancements and exploit-mitigation technologies. From
an initial focus on Address Space Layout Randomization (ASLR), it has
now branched out to explore additional exploit mitigation techniques.
It has been a long while since HardenedBSD's last entry in a quarterly
status report, back in 2015Q4. The intervening year saw HardenedBSD
gain new developers Bernard Spil and Franco Fichtner, import LibreSSL
and OpenNTPd into base as the default crypto library and NTP client,
respectively, and introduce the hbsd-update binary update mechanism for
the base system. The secadm application got a rewrite and Trusted Path
Execution (TPE). PIE is now enabled for the base system for arm64 and
amd64 as well as the bulk of the ports tree, and the ports tree also
gained RELRO and BIND_NOW. Integriforce (similar to NetBSD's verified
exec, veriexec) was introduced for the base system, as well as
SafeStack, a technology for protection against stack-based buffer
overflows that's developed by the Clang/LLVM community. SafeStack
relies and builds on top of Address Space Layout Randomization (ASLR),
and is strengthened by the presence of PaX NOEXEC. Certain high-profile
ports also have SafeStack enabled.
Extremely generous hardware donations from G2, Inc. have provided for
dedicated package building and binary update servers, as well as
development and test servers.
In March of 2017, we added Control Flow Integrity (CFI) to the base
system. CFI is an exploit mitigation technique that helps prevent
attackers from modifying the behavior of a program and jumping to
undefined or arbitrary memory locations. This type of technique is
gaining adoption across the industry -- Microsoft has implemented a
variant of CFI, which they term Control Flow Guard, or CFG, and the PaX
team has spent the last few years perfecting their Reuse Attack
Protector, RAP. Of these, RAP is the most complete and effective
implementation, followed by Clang's CFI. RAP would be a great addition
to HardenedBSD; however, it requires a GPLv3 toolchain and is
patent-pending.
CFI can be implemented either on a per-DSO basis, or across all DSOs in
a process. Currently only the former is implemented, but we are working
hard to enable cross-DSO CFI. As is the case for SafeStack, cross-DSO
CFI requires both ASLR and PaX NOEXEC in order to be effective. If an
attacker knows the memory layout of an application, the attacker might
be able to craft a data-only attack, modifying the CFI control data.
The behavior of several system control (sysctl) nodes has been tighened
up, limiting write access and introducing additional safety checks for
write accesses. Kernel module APIs received a similar treatment.
HardenedBSD's PaX SEGVGUARD implementation received a few updates to
make it more stable and performant.
As of March 2017, HardenedBSD is now accessible through a Tor hidden
service. The main website, binary updates, and package distribution are
all available over the hidden service.
We now maintain our own version of the drm-next branch for updated
graphics support. Binary updates are also provided for this branch.
HardenedBSD would like to thank all those who have generously donated
time, money, or other resources to the project.
This project was sponsored by SoldierX, and G2, Inc.
Open tasks:
1. Port SafeStack to arm64.
2. Integrate Cross-DSO CFI.
3. Add documentation to the HardenedBSD Handbook.
4. Start porting grsecurity's RBAC.
__________________________________________________________________
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 667 bytes
Desc: not available
URL:
<http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20170927/61f7a7b5/attachment.sig>