freebsd stable - Sep 2016 - X2APIC support

On Sun, Sep 04, 2016 at 06:49:43PM +0300, Andriy Gapon
wrote:
> On 04/09/2016 18:14, Konstantin Belousov wrote:
> > On Sun, Sep 04, 2016 at 11:19:16AM +0300, Andriy Gapon wrote:
> >> Kostik, I see one strange thing which is common to both successful
and
> >> unsuccessful configurations.  All "SMP: Added CPU..."
lines have "AP" in them.
> >> It seems like the platform does not tell explicitly tell which CPU
is the BSP,
> >> see cpu_add() function.  This can break quite a few assumption. 
And I am not
> >> even sure how the successful scenario works.
> >> Ah... I see that there is a backup code in cpu_mp_start() where
boot_cpu_id is
> >> set based on the current CPU's Local APIC ID.  I suspect then
that this
> >> information is incorrect in the failing case.
> >>
> > Well, there is no easy way to read the LAPIC Id of BSP before LAPICs
> > are initialized. BIOS might reprogram LAPIC Ids, so reading from
> > CPUID[1].EBX[31:24] might return incorrect data. Even more incorrect
> > it might be in the x2APIC state, since 8 bits are not enough for 32bit
> > x2APIC Id.
> 
> Hmm, I am not sure how what you are saying here is relevant to the problem.
> I believe that cpu_mp_start() is executed (on the BSP) after the BSP's
LAPIC is
> initialized.  So, the code should just work.
The order is madt_probe()->madt_probe_cpus()->madt_setup_local().

madt_probe() and madt_probe_cpus() are called from apic_init9) at
SI_SUB_TUNABLEs, and
madt_probe_cpus()->madt_add_cpu()->lapic_create()->
cpu_add() is how the SMP: ... lines are printed.

The madt_setup_local() code is called from apic_setup_local() at SI_SUB_CPU,
this is where APIC window is mapped or x2APIC mode is enabled by the call
to native_lapic_init().

You cannot get BSP LAPIC Id earlier than native_lapic_init() was executed.

mp_start()->cpu_mp_start() is called right after the madt_setup_local().
> 
> My theory was that the BSP's LAPIC ID was incorrectly programmed by
BIOS (firmware).
This is possible, of course.  But it would not affect "SMP: Added CPU
..."
lines.

On 04/09/2016 19:29, Konstantin Belousov wrote:
> This is possible, of course.  But it would not affect "SMP: Added CPU
..."
> lines.
Well, looking at the code it seems that only if mptable is used, then those
lines are expected to correctly identify a BSP.  With MADT there is no
information to identify the BSP and that is supposed to happen in
cpu_mp_start().


static void
madt_add_cpu(u_int acpi_id, u_int apic_id, u_int flags)
{
        struct lapic_info *la;

        /*
         * The MADT does not include a BSP flag, so we have to let the
         * MP code figure out which CPU is the BSP on its own.
         */
...

In other words, those "SMP: Added CPU ..." are truly a cosmetic issue.
And it's my guess (just a guess) that BSP LAPIC ID is incorrect in the
problematic configuration.

-- 
Andriy Gapon