VMs with >1TiB of memory confuse EDK2

[iximeow]

@askfongjojo saw as disks no longer existing when starting larger instances in oxidecomputer/omicron#7918 (comment) .

the instance had two disks, but from the UEFI shell saw only the cloud-init disk:

UEFI Interactive Shell v2.2
EDK II
UEFI v2.70 (EDK II, 0x00010000)
Mapping table
      FS0: Alias(s):F0:;BLK0:
          PciRoot(0x0)/Pci(0x18,0x0)
Press ESC in 1 seconds to skip startup.nsh or any other key to continue.
Shell> 

debug.out was informative here:

PlatformBootManagerAfterConsole
Found Mass Storage device: PciRoot(0x0)/Pci(0x10,0x0)
NvmExpressDriverBindingStart: start
NvmeControllerInit: the controller doesn't support NVMe command set
NvmExpressDriverBindingStart: end with Unsupported
Found Mass Storage device: PciRoot(0x0)/Pci(0x11,0x0)
NvmExpressDriverBindingStart: start
NvmeControllerInit: the controller doesn't support NVMe command set
NvmExpressDriverBindingStart: end with Unsupported
Found Mass Storage device: PciRoot(0x0)/Pci(0x18,0x0)

which implies that Cap.Css was not 0x01 in NvmeControllerInit.

skipping ahead a bit, it's clear that OVMF's view of Mem64 is bogus:

PciHostBridge: NotifyPhase (AllocateResources)
 RootBridge: PciRoot(0x0)
  Mem64: Base/Length/Alignment = 800000000/100000/FFFFF - Success
  Mem: Base/Length/Alignment = C0000000/100000/FFFFF - Success
  I/O: Base/Length/Alignment = C000/1000/FFF - Success

there should not be MMIO against 0x8_0000_0000, that's 32 GiB and is just... memory.

there's another odd line in OVMF debug output: GetFirstNonAddress: Pci64Base=0x800000000 Pci64Size=0x800000000, Pci64Size is a PCD that defaults to 0x8_0000_0000, how Pci64Base ends up at that same value is a bit awkward. the log comes from GetFirstNonAddress. Pci64Base first starts its life as FirstNonAddress from

FirstNonAddress = BASE_4GB + GetSystemMemorySizeAbove4gb ();

the result here is rounded up twice. first, to 1G, then to the power of two that would contain Pci64Size. since that is a power of two, it's just rounded up to 0x8_0000_0000. to get here, FirstNonAddress could be any value less than 0x8_0000_0000`.

GetSystemMemorySizeAbove4gb is where it all goes off the rails. CMOS[0x5b..0x5d] provide three bytes of the amount of memory in the system above 4G. 1T in units of 64k wraps here:

>>> hex((1024 * (1 << 30)) >> 16)
'0x1000000'

conspicuously above 24 bits. in Propolis we'll dutifully populate the low 24 bits, and miss that high[3] here is non-zero.

[pfmooney]

There is no "even higher memory" field in the CMOS which we can use, so it seems the next mechanism used for such things is exposing an E820 memory map to the OS. The OVMF firmware goes looking for one at the etc/e820 fw_cfg key. So we could generate such a map for the guest.