Inter-Processor Interrupts (IPIs) on x86

[dimakuv]

See https://wiki.osdev.org/TLB#Multi-processor_consistency for details.

Page permissions, multicore case: the affected memory region must be "TLB invalidated" on all cores (vCPUs). To this end, inter-processor interrupts (IPIs) are used: a vCPU that wishes to change a page mapping writes the operation to a global struct and sends an interrupt 33 to every other vCPU. Each vCPU receives the interrupt, acknowledges the IPI by writing to a shared variable and invalidates the TLB entries. After all vCPUS acknowledged the IPI, the initiating vCPU finalizes the "invalidate TLB" protocol.

---

From Guest-Host Communication Interface (GHCI) TDX document:

Various TD-VMM-communication scenarios require the TD to request the host generate IPIs to TD vCPUs – for example, synchronizing, guest-TD-kernel-managed-IA-page-table updates. This operation is supported via the TDCALL TDG.VP.VMCALL<Instruction.WRMSR> to the x2APIC ICR MSR.

To perform a cross-VCPU IPI, the guest-TD ILP is designed to request an operation from the host VMM using this TDCALL (TDG.VP.VMCALL<Instruction.WRMSR>). The VMM can then inject an interrupt into the guest TD’s RLPs using the posted-interrupt mechanism. This is an untrusted operation; thus, the TD OS must track its completion and not rely on the host VMM to faithfully deliver IPIs to all the TD vCPUs.

From https://www.sigops.org/s/conferences/sosp/2009/papers/baumann-sosp09.pdf :

In Linux and Windows, inter-processor interrupts (IPIs) are used: a core that wishes to change a page mapping writes the operation to a well-known location and sends an interrupt to every core that might have the mapping in its TLB. Each core takes the trap, acknowledges the IPI by writing to a shared variable, invalidates the TLB entry, and resumes. The initiating core can continue immediately after every IPI has been acknowledged, since by the time each core resumes to user space the TLB entry is guaranteed to be flushed. This has low latency, but can be disruptive as each core immediately takes the cost of a trap (about 800 cycles). The TLB invalidation itself is fast, taking 95-320 cycles on a current x86-64 core.

We have IA32_X2APIC_ICR (Interrupt Command) which is 0x830. Must be accessed via TDG VMCALL.

Intel SDM: read Chapter 11.6.

---

Linux does this:

INVALIDATE_TLB_VECTOR (vector 0xfd) -- Sent to all CPUs but the sender, forcing them to invalidate their Translation Lookaside Buffers. The corresponding handler, named invalidate_interrupt(), flushes some TLB entries of the processor.

• https://learning.oreilly.com/library/view/understanding-the-linux/0596005652/ch04s06.html#understandlk-CHP-4-SECT-6.1
• https://learning.oreilly.com/library/view/understanding-the-linux/0596005652/ch02s05.html#understandlk-CHP-2-SECT-5.7