On Wed, Jul 15, 2020 at 11:24:56AM +0200, Peter Zijlstra wrote:> Can we get some more words -- preferably in actual code comments, on > when exactly #VC happens?Sure, will add this as a comment before the actual runtime VC handler.> Because the only thing I remember is that #VC could happen on any memop, > but I also have vague memories of that being a later extention.Currently it is only raised when something happens that the hypervisor intercepts, for example on a couple of instructions like CPUID, RD/WRMSR, ..., or on MMIO/IOIO accesses. With Secure Nested Paging (SNP), which needs additional enablement, a #VC can happen on any memory access. I wrote the IST handling entry code for #VC with that in mind, but do not actually enable it. This is the reason why the #VC handler just panics the system when it ends up on the fall-back (VC2) stack, with SNP enabled it needs to handle the SNP exit-codes in that path. Regards, Joerg
On Wed, Jul 15, 2020 at 11:34:26AM +0200, Joerg Roedel wrote:> On Wed, Jul 15, 2020 at 11:24:56AM +0200, Peter Zijlstra wrote: > > Can we get some more words -- preferably in actual code comments, on > > when exactly #VC happens? > > Sure, will add this as a comment before the actual runtime VC handler.Thanks!> > Because the only thing I remember is that #VC could happen on any memop, > > but I also have vague memories of that being a later extention. > > Currently it is only raised when something happens that the hypervisor > intercepts, for example on a couple of instructions like CPUID, > RD/WRMSR, ..., or on MMIO/IOIO accesses. > > With Secure Nested Paging (SNP), which needs additional enablement, a #VC can > happen on any memory access. I wrote the IST handling entry code for #VC > with that in mind, but do not actually enable it. This is the reason why > the #VC handler just panics the system when it ends up on the fall-back > (VC2) stack, with SNP enabled it needs to handle the SNP exit-codes in > that path.And recursive #VC was instant death, right? Because there's no way to avoid IST stack corruption in that case.
On Wed, Jul 15, 2020 at 11:55:56AM +0200, Peter Zijlstra wrote:> And recursive #VC was instant death, right? Because there's no way to > avoid IST stack corruption in that case.Right, a #VC exception while still on the IST stack must instantly kill the VM. That needs an additional check which is not implemented yet, as it only becomes necessary with SNP. Regards, Joerg
Hi, On Mon, Jul 20, 2020 at 06:09:19PM -0700, Erdem Aktas wrote:> It looks like there is an expectation that the bootloader will start > from the 64bit entry point in header_64.S. With the current patch > series, it will not boot up if the bootloader jumps to the startup_32 > entry, which might break some default distro images. > What are supported bootloaders and configurations? > I am using grub ( 2.02-2ubuntu8.15) and it fails to boot because of > this reason. I am not a grub expert, so I would appreciate any > pointers on this.This is right, the only supported boot path is via the 64bit EFI entry point. The reason is that SEV-ES requires support in the firmware too, and currently only OVMF is supported in that regard. The firmware needs to setup the AP jump-table, for example. Other boot-paths have not been implemented. Booting via startup_32 would require exception handling in the 32bit-part of the boot-strap code, because verify_cpu is called there. Also an AMD specific MSR can't be accessed there because this would #GP on non-AMD/SEV-ES machines and, as I said, there is no way yet to handle them. How did you get into the startup_32 entry-point, do you have an SEV-ES BIOS supporting this? If it is really needed it could be implemented at a later point. Regards, Joerg
Hi Erdem, On Tue, Jul 21, 2020 at 09:48:51AM -0700, Erdem Aktas wrote:> Yes, I am using OVMF with SEV-ES (sev-es-v12 patches applied). I am > running Ubuntu 18.04 distro. My grub target is x86_64-efi. I also > tried installing the grub-efi-amd64 package. In all cases, the grub is > running in 64bit but enters the startup_32 in 32 bit mode. I think > there should be a 32bit #VC handler just something very similar in the > OVMF patches to handle the cpuid when the CPU is still in 32bit mode. > As it is now, it will be a huge problem to support different distro images. > I wonder if I am the only one having this problem.I havn't heard from anyone else that the startup_32 boot-path is being used for SEV-ES. What OVMF binary do you use for your guest? In general it is not that difficult to support that boot-path too, but I'd like to keep that as a future addition, as the patch-set is already quite large. In the startup_32 path there is already a GDT set up, so whats needed is an IDT and a 32-bit #VC handler using the MRS-based protocol (and hoping that there will only be CPUID intercepts until it reaches long-mode). Regards, Joerg
On Wed, Jul 22, 2020 at 09:54:40AM -0700, Erdem Aktas wrote:> I am using a custom, optimized and stripped down version, OVMF build. > Do you think it is because of the OVMF or grub?Not sure, I havn't looked into how grub decides which entry point to use.> In my case, there are 2 places where the CPUID is called: the first > one is to decide if long mode is supported, along with few other > features like SSE support and the second one is to retrieve the > encryption bit location.Yes, it is basically the verify_cpu() function that is causing the trouble. If you want to work around it you can comment it out in that path for testing purposes. Regards, Joerg