Xen devel - Feb 2012 - Problems with hyperthreading in Windows HVM

Hi,

I have been trying to get Hyperthreading to work in a Windows HVM on
Xen 4.1.2 as described in ''xmexample.hvm''. I think I have set
it up
correctly, but I can''t seem to get it to work. There is not much
documentation on it and most topics are from years ago. I''m curious to
know to whether this functionality is still supposed to work.

I''m aware that Xen itself supports Hyperthreading, but my software is
not happy about seeing every VCPU from Xen as a physical core. I would
like to use hyperthreading opposed to disabling it since the software
in question does benefit of using Hyperthreading.

For reference this is the original Hyperthreading example from
''xmexample.hvm'':
#   Expose to the guest multi-core cpu instead of multiple processors
# Example for intel, expose a 8-core processor :
#cpuid=[''1:edx=xxx1xxxxxxxxxxxxxxxxxxxxxxxxxxxx,
#          ebx=xxxxxxxx00010000xxxxxxxxxxxxxxxx'',
#     ''4,0:eax=001111xxxxxxxxxxxxxxxxxxxxxxxxxx'']
#  - CPUID.1[EDX][HT] : Enable HT
#  - CPUID.1[EBX] : Number of vcpus * 2
#  - CPUID.4,0[EAX] : Number of vcpus * 2 - 1
#vcpus=8

I tried the example from xmexample.hvm on a 4-core system of which I
wanted to pass 2 physical cores (so 4 logical) to Windows:
cpuid =
[''1:edx=xxx1xxxxxxxxxxxxxxxxxxxxxxxxxxxx,ebx=xxxxxxxx00000100xxxxxxxxxxxxxxxx'',
''4,0:eax=000011xxxxxxxxxxxxxxxxxxxxxxxxxx'' ]
vcpus=2
cpus=[''4'', ''6''] # pin to 2 different
physical cores

When I boot up the VM, Windows detects just 2 physical cores. Debug
tools like Sysinternals coreinfo, cpu-z also don''t detect 4 logical
cores.

To figure out what is going wrong, I wrote a small win32 tool to check
the result of the cpuid patching. For function 1, it shows that the
correct bits in edx and ebx got updated. The bits of function 4 which
should contain a part of the ACPI ID don''t seem to got patched. In
fact eax/ebx/ecx/edx are all reported as 0. From what I saw in
''xc_cpuid_hvm_policy'' and
''intel_xc_cpuid_policy'' in libxc, Xen
performs filtering on a bunch of cpuid functions. Though all the ones
I patched are preserved by the filtering. From what I saw in past
email threads there were issues with ACPI ID stuff in the past, can
there be new issues here? Does anyone have hyperthreading working?

Thanks,
Roderick Colenbrander

On Wed, 2012-02-29 at 00:28 +0000, Roderick Colenbrander
wrote:
> Hi,
> 
> I have been trying to get Hyperthreading to work in a Windows HVM on
> Xen 4.1.2 as described in ''xmexample.hvm''. I think I have
set it up
> correctly, but I can''t seem to get it to work. There is not much
> documentation on it and most topics are from years ago.
I don''t know the answers to your questions, or whether this particular
type of cpuid modification has ever worked. But might it be that
something on the hypervisor side is also modifying the cpuid results?

I suppose you could start at xen/arch/x86/hvm/emulate.c:hvmemul_cpuid
and trace down to the actual code in question? Looks like hvm_cpuid() is
probably the key function of interest.

Note sure where APIC IDs come from either, xen/arch/x86/hvm/vlapic.c
might be an interesting starting point. Or perhaps this is done by
hvmloader tools/firmware/hvmloader.c?

More questions than answers in the above, sorry about that!

Ian.

On Wed, Feb 29, 2012 at 2:37 PM, Ian Campbell <Ian.Campbell@citrix.com>
wrote:
> On Wed, 2012-02-29 at 00:28 +0000, Roderick Colenbrander wrote:
>> Hi,
>>
>> I have been trying to get Hyperthreading to work in a Windows HVM on
>> Xen 4.1.2 as described in ''xmexample.hvm''. I think I
have set it up
>> correctly, but I can''t seem to get it to work. There is not
much
>> documentation on it and most topics are from years ago.
>
> I don''t know the answers to your questions, or whether this
particular
> type of cpuid modification has ever worked. But might it be that
> something on the hypervisor side is also modifying the cpuid results?
It looks like there is code in xen/arch/x86/hvm/hvm.c which deals with
some more cpuid overrides. Initially it looked like libxc was doing
all the filtering and that ''XEN_DOMCTL_set_cpuid'' just stored
the
info. I''m not sure yet how that hypercall and the hvm.c code interact.
I have to dive more into it.

It also seems that for modern CPUs cpuid function 11 contains
additional topology information which may have to patched as well.
> I suppose you could start at xen/arch/x86/hvm/emulate.c:hvmemul_cpuid
> and trace down to the actual code in question? Looks like hvm_cpuid() is
> probably the key function of interest.
>
> Note sure where APIC IDs come from either, xen/arch/x86/hvm/vlapic.c
> might be an interesting starting point. Or perhaps this is done by
> hvmloader tools/firmware/hvmloader.c?
>
> More questions than answers in the above, sorry about that!
>
> Ian.
>
I''m going to dive further into the code and look at the places you
pointed to. I guess it can be made to work with some patching.
Assuming this would all work, I really wonder whether how this
hyperthreading fixup would interact with Xen itself. Normally Xen
hides hyperthreading for a reason and each guest works with ''virtual
cores'' which get scheduled. My feeling is that the hyperthreading
fixup would have to be combined with CPU pinning to the correct
physical cores. Then to get 4 logical cores in the VM, you need to
assign 2 VCPUs to the VM. Would the guest OS really be able to
schedule threads over 4 logical cores? I really don''t know how this
would work in Xen.

Roderick

On Wed, 2012-02-29 at 18:13 +0000, Roderick Colenbrander
wrote:
> On Wed, Feb 29, 2012 at 2:37 PM, Ian Campbell
<Ian.Campbell@citrix.com> wrote:
> > On Wed, 2012-02-29 at 00:28 +0000, Roderick Colenbrander wrote:
> >> Hi,
> >>
> >> I have been trying to get Hyperthreading to work in a Windows HVM
on
> >> Xen 4.1.2 as described in ''xmexample.hvm''. I
think I have set it up
> >> correctly, but I can''t seem to get it to work. There is
not much
> >> documentation on it and most topics are from years ago.
> >
> > I don''t know the answers to your questions, or whether this
particular
> > type of cpuid modification has ever worked. But might it be that
> > something on the hypervisor side is also modifying the cpuid results?
> 
> It looks like there is code in xen/arch/x86/hvm/hvm.c which deals with
> some more cpuid overrides. Initially it looked like libxc was doing
> all the filtering and that ''XEN_DOMCTL_set_cpuid'' just
stored the
> info. I''m not sure yet how that hypercall and the hvm.c code
interact.
> I have to dive more into it.
> 
> It also seems that for modern CPUs cpuid function 11 contains
> additional topology information which may have to patched as well.
> 
> > I suppose you could start at xen/arch/x86/hvm/emulate.c:hvmemul_cpuid
> > and trace down to the actual code in question? Looks like hvm_cpuid()
is
> > probably the key function of interest.
> >
> > Note sure where APIC IDs come from either, xen/arch/x86/hvm/vlapic.c
> > might be an interesting starting point. Or perhaps this is done by
> > hvmloader tools/firmware/hvmloader.c?
> >
> > More questions than answers in the above, sorry about that!
> >
> > Ian.
> >
> 
> I''m going to dive further into the code and look at the places you
> pointed to. I guess it can be made to work with some patching.
> Assuming this would all work, I really wonder whether how this
> hyperthreading fixup would interact with Xen itself. Normally Xen
> hides hyperthreading for a reason and each guest works with
''virtual
> cores'' which get scheduled. My feeling is that the hyperthreading
> fixup would have to be combined with CPU pinning to the correct
> physical cores. Then to get 4 logical cores in the VM, you need to
> assign 2 VCPUs to the VM. Would the guest OS really be able to
> schedule threads over 4 logical cores? I really don''t know how
this
> would work in Xen.
Normally Xen treats every HT as a CPU (roughly speaking) and will
schedule VCPUs between them I think the scheduler does have some basic
smarts about HT i.e. it won''t schedule 2 VCPUs on sibling HTs while
leaving another pair totally idle.

Ultimately it depends on whether you want to actually expose HT to the
guest or if you just want the guest to think it has HT. In your use case
is having the workload scheduled on two sibling threads actually better
than giving them a core each?

If you want to really expose HT then you will probably have to play
games with pinning as you suggest.

Ian.

On Thu, Mar 1, 2012 at 8:22 AM, Ian Campbell <Ian.Campbell@citrix.com>
wrote:
> On Wed, 2012-02-29 at 18:13 +0000, Roderick Colenbrander wrote:
>> On Wed, Feb 29, 2012 at 2:37 PM, Ian Campbell
<Ian.Campbell@citrix.com> wrote:
>> > On Wed, 2012-02-29 at 00:28 +0000, Roderick Colenbrander wrote:
>> >> Hi,
>> >>
>> >> I have been trying to get Hyperthreading to work in a Windows
HVM on
>> >> Xen 4.1.2 as described in ''xmexample.hvm''. I
think I have set it up
>> >> correctly, but I can''t seem to get it to work. There
is not much
>> >> documentation on it and most topics are from years ago.
>> >
>> > I don''t know the answers to your questions, or whether
this particular
>> > type of cpuid modification has ever worked. But might it be that
>> > something on the hypervisor side is also modifying the cpuid
results?
>>
>> It looks like there is code in xen/arch/x86/hvm/hvm.c which deals with
>> some more cpuid overrides. Initially it looked like libxc was doing
>> all the filtering and that ''XEN_DOMCTL_set_cpuid''
just stored the
>> info. I''m not sure yet how that hypercall and the hvm.c code
interact.
>> I have to dive more into it.
>>
>> It also seems that for modern CPUs cpuid function 11 contains
>> additional topology information which may have to patched as well.
>>
>> > I suppose you could start at
xen/arch/x86/hvm/emulate.c:hvmemul_cpuid
>> > and trace down to the actual code in question? Looks like
hvm_cpuid() is
>> > probably the key function of interest.
>> >
>> > Note sure where APIC IDs come from either,
xen/arch/x86/hvm/vlapic.c
>> > might be an interesting starting point. Or perhaps this is done by
>> > hvmloader tools/firmware/hvmloader.c?
>> >
>> > More questions than answers in the above, sorry about that!
>> >
>> > Ian.
>> >
>>
>> I''m going to dive further into the code and look at the places
you
>> pointed to. I guess it can be made to work with some patching.
>> Assuming this would all work, I really wonder whether how this
>> hyperthreading fixup would interact with Xen itself. Normally Xen
>> hides hyperthreading for a reason and each guest works with
''virtual
>> cores'' which get scheduled. My feeling is that the
hyperthreading
>> fixup would have to be combined with CPU pinning to the correct
>> physical cores. Then to get 4 logical cores in the VM, you need to
>> assign 2 VCPUs to the VM. Would the guest OS really be able to
>> schedule threads over 4 logical cores? I really don''t know how
this
>> would work in Xen.
>
> Normally Xen treats every HT as a CPU (roughly speaking) and will
> schedule VCPUs between them I think the scheduler does have some basic
> smarts about HT i.e. it won''t schedule 2 VCPUs on sibling HTs
while
> leaving another pair totally idle.
>
> Ultimately it depends on whether you want to actually expose HT to the
> guest or if you just want the guest to think it has HT. In your use case
> is having the workload scheduled on two sibling threads actually better
> than giving them a core each?
Let me describe the workload. It consists of closed-source Windows
applications which use multiple threads. Since each thread is not too
stressful (there is blocking I/O and other stuff involved)
hyperthreading helps quite a bit. The problem is that the applications
scale the work load based on the number of logical and physical cores
(through Win32 GetLogicalProcessorInformation and friends). Since on
Xen each VCPU is seen as a physical core, the work load the system can
handle is estimated incorrectly. In order for the workload to utilize
the system at best, hyperthreading would have to be exposed to the VM.
>
> If you want to really expose HT then you will probably have to play
> games with pinning as you suggest.
>
> Ian.
>
I will try to play with the cpuid options. I have written some test
applications based on Intel datasheets, but haven''t had time to
continue with it yet. I will report back when I know more and document
how to get this to work.

Roderick