freebsd stable - Mar 2018 - [HEADS UP] - OFED/RDMA stack update

On Mon, Feb 26, 2018 at 02:21:39PM -0800, Navdeep Parhar
wrote:
> +freebsd-arch@
> 
> Hi Meny,
> 
> Can you please post the KPI/KBI analysis that you generated to some
> public location and provide a link here?  A straight MFC would be a
> major break of KPI/KBI in -STABLE and the options we're looking at are:
I put the report at
https://kib.kiev.ua/kib/ibcore_11_to_11_merged_compat_report.html
> 
> a) Ignore the breakage and let downstream consumers deal with the
> fallout.  This obviously isn't ideal in a -STABLE branch.
> 
> b) Provide compat shims to at least preserve the KPI.  One challenge is
> that the changes include functions with the same name but different
> signature/behavior.  See, for example, ib_create_cq in Meny's list once
> he publishes it.
Project did handled similar issues already.  One of the approaches is to
renname the ib_create_cq with the new signature to ib_create_cq_n12 and
check for (e.g.) _WANT_NEW_OFED symbol and to select one or another:
#ifdef _WANT_NEW_OFED
#define ib_create_cq(new args there) ib_create_cq_n21(new args there)
#else
#define ib_create_cq (ib_create_cq)
#endif

Then ULP that wants new KPI defines _WANT_NEW_OFED.
> 
> c) Have two versions of the OFED interfaces in 11-STABLE and not break
> existing downstream consumers at all.
It is possible to make them loadable simultaneously as modules, but it
is quite confusing to users, because Mellanox clearly wants mlx5_ib and
mlx4_ib to work only with new OFED, while cxgbe would use old OFED ?

Also, either we would need to mess with the ibcore.ko module name, or
with version.  I am not sure that our module handling is robust enough
to make the version trick possible.
> 
> I've reached out to users that I know of and know will be affected.
> If you use OFED and FreeBSD 11 this would be a good time to weigh
> in with your thoughts, ideas, concerns etc..

Hi,

The bsd_rdma_4_9_stable_11 projects branch is close to being merged into 
FreeBSD 11-stable. Mellanox plans to merge no later than 12:00 CEST TUE 
20th of March 2018, unless objections are received.

A compatibility header file has been created, ib_verbs_compat.h, which 
offers full source compatibility to existing OFED kernel applications, 
as a response to the raised conserns. User-space compatibility is 
maintained through library symbol versioning.

https://svnweb.freebsd.org/base/projects/bsd_rdma_4_9_stable_11/sys/ofed/include/rdma/ib_verbs_compat.h

An example client for this header file can be found here:

https://svnweb.freebsd.org/base/projects/bsd_rdma_4_9_stable_11/sys/contrib/rdma/krping_compat/

Currently the cxgb and cxgbe i-Warp drivers are not building, and will 
be stubbed from the kernel build before the branch is merged, unless 
Chelsio can add patches for these.

Here is a quick and dirty patch to make the bsd_rdma_4_9_stable_11 
branch build:
> diff --git a/sys/modules/Makefile b/sys/modules/Makefile
> index 6b005c854d7..b918a208f21 100644
> --- a/sys/modules/Makefile
> +++ b/sys/modules/Makefile
> @@ -530,7 +530,7 @@ _txp=               txp
>  .if ${MK_SOURCELESS_UCODE} != "no" && ${MACHINE_CPUARCH}
!= "arm" && \
>         ${MACHINE_ARCH:C/mips(el)?/mips/} != "mips" && \
>         ${MACHINE_ARCH} != "powerpc" &&
${MACHINE_CPUARCH} != "riscv"
> -_cxgbe=                cxgbe
> +#_cxgbe=               cxgbe
>  .endif
>  
>  .if ${MK_TESTS} != "no" || defined(ALL_MODULES)
> @@ -554,7 +554,7 @@ _vpo=               vpo
>  _sym=          sym
>  # intr_disable() is a macro, causes problems
>  .if ${MK_SOURCELESS_UCODE} != "no"
> -_cxgb=         cxgb
> +#_cxgb=                cxgb
>  .endif
>  .endif
--HPS