Ensure the correct EFI boot entries are created after switching from CentOS to Oracle Linux

[metal4lyf]

We are trying to migrate CentOS 8 systems to OL8.

The conversion script reports success, but it renders our systems unbootable: After the BIOS splash, we get several >> Checking media presence ..... messages on the terminal and then the system enters Dell BIOS recovery mode, which performs a memory test and then reports "No bootable devices found! ..."

Boot params are UEFI/Legacy Boot: OFF/Secure Boot: OFF.

I've isolated this issue to OL8 grub. Using a recovery stick, if I re-enable the CentOS BaseOS repo and install the latest version of grub2*, the system will boot to login with expected entries ("Oracle Linux" etc.) in the grub menu.

We're hesitant to proceed with migrations using this workaround because it requires us to continue using a potentially unsupported version of a fundamental component, not to mention we'll have to exclude grub in our dnf config to avoid bricking on dnf upgrades.

We use a stock grub configuration as far as I know.

/etc/default/grub:

GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto resume=/dev/mapper/VolGroup00-swap rd.lvm.lv=VolGroup00/root rd.lvm.lv=VolGroup00/swap rhgb quiet"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true

/boot/grub2/grubenv:

# GRUB Environment Block
kernelopts=root=/dev/mapper/VolGroup00-root ro crashkernel=auto resume=/dev/mapper/VolGroup00-swap rd.lvm.lv=VolGroup00/root rd.lvm.lv=VolGroup00/swap rhgb quiet
boot_success=1
boot_indeterminate=0
######################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################

Kernel: 4.18.0-240.15.1.el8_3.x86_64
Bricking grub2*: 1:2.02-90.0.2.el8_3.1 from ol8_baseos_latest
Working grub2*: <= 1:2.02-90.el8_3.1 from BaseOS

I've yet to see a useful message from grub despite removing rhgb quiet. Please let me know what other info would help here.

[Djelibeybi]

Does 1:2.02-90.0.1.el8 work? If so, that at least will narrow our focus to the fixes in the .0.2 release.

[Djelibeybi]

Also, can you tell us what type of device and controller you're using to boot?

[Djelibeybi]

Could you also try running grub2-install <boot device> prior to rebooting to see if that resolves the issue?

[metal4lyf]

Here's the boot device info. I'll try the grub install now.

$ sudo lshw -class disk
  *-disk
       description: ATA Disk
       product: ST2000DM001-1ER1
       physical id: 0.0.0
       bus info: scsi@0:0.0.0
       logical name: /dev/sda
       version: CC27
       serial: Z4Z703D6
       size: 1863GiB (2TB)
       capacity: 1863GiB (2TB)
       capabilities: 7200rpm gpt-1.00 partitioned partitioned:gpt
       configuration: ansiversion=6 guid=9d1775b4-835a-4c76-9e43-5d544b7ec8fc logicalsectorsize=512 sectorsize=4096

[aburmash]

Boot params are UEFI/Legacy Boot: OFF/Secure Boot: OFF.

To be on the same page, is system in UEFI mode or legacy ?
To check you can check /sys/firmware/efi presence on the booted system.

[metal4lyf]

UEFI

[metal4lyf]

I can't get grub2-install working. It complains about missing modinfo.sh. No directory under /boot contains this file so I'm not sure what to pass it.
Trying 90.0.1 now.

[aburmash]

Yeah, forget about grub2-install. It is for legacy.
Please, just before the reboot do
efibootmgr -v
find /boot |grep redhat
find /boot |grep centos
rpm -qa |grep shim

[metal4lyf]

90.0.1 doesn't boot either. Reinstalled 90.0.2. Here are the results:
efibootmgr -v:

BootCurrent: 0011
Timeout: 1 seconds
BootOrder: 0001,000C,000D,000E,000F,0010,0006,0011,0008,0009,000A,000B
Boot0000* Windows Boot Manager	HD(1,GPT,87d93515-2374-4b87-9701-5a4c527ee83b,0x800,0x145000)/File(\EFI\Microsoft\Boot\bootmgfw.efi)WINDOWS.........x...B.C.D.O.B.J.E.C.T.=.{.9.d.e.a.8.6.2.c.-.5.c.d.d.-.4.e.7.0.-.a.c.c.1.-.f.3.2.b.3.4.4.d.4.7.9.5.}...;.............�...
Boot0001* CentOS Linux	HD(1,GPT,b7460ef9-456e-4086-95f9-7dc69e80ddaa,0x800,0x12c000)/File(\EFI\centos\shimx64.efi)
Boot0006* HDD	NVMe(0x1,01-00-00-00-00-00-00-00)/HD(1,GPT,54969a86-cdfd-4d17-a677-4063a30945af,0x800,0x12c000)
Boot0008* PXE IP4 Intel(R) Ethernet 10G 2P X550-t Adapter	PciRoot(0x2)/Pci(0x0,0x0)/Pci(0x0,0x0)/MAC(b4969130ba1c,1)/IPv4(0.0.0.00.0.0.0,0,0)..BO
Boot0009* PXE IP6 Intel(R) Ethernet 10G 2P X550-t Adapter	PciRoot(0x2)/Pci(0x0,0x0)/Pci(0x0,0x0)/MAC(b4969130ba1c,1)/IPv6([::]:<->[::]:,0,0)..BO
Boot000A* PXE IP4 Intel(R) Ethernet 10G 2P X550-t Adapter	PciRoot(0x2)/Pci(0x0,0x0)/Pci(0x0,0x1)/MAC(b4969130ba1e,1)/IPv4(0.0.0.00.0.0.0,0,0)..BO
Boot000B* PXE IP6 Intel(R) Ethernet 10G 2P X550-t Adapter	PciRoot(0x2)/Pci(0x0,0x0)/Pci(0x0,0x1)/MAC(b4969130ba1e,1)/IPv6([::]:<->[::]:,0,0)..BO
Boot000C* Diskette Drive	BBS(Floppy,Diskette Drive,0x0)..BO
Boot000D* Internal HDD	BBS(HD,Internal HDD,0x0)..BO
Boot000E* USB Storage Device	BBS(USB,SanDisk,0x0)..BO
Boot000F* P7: HL-DT-ST DVD-ROM DH50N	BBS(CDROM,P7: HL-DT-ST DVD-ROM DH50N,0x0)..BO
Boot0010  Onboard NIC	BBS(Network,IBA CL Slot 00FE v0110,0x0)..BO
Boot0011* UEFI: SanDisk	PciRoot(0x0)/Pci(0x14,0x0)/USB(7,0)/USB(1,0)/HD(1,GPT,87182ce7-da3d-414d-9ff3-3182544d7675,0x800,0x1dcf7df)..BO

find /boot | grep redhat:

/boot/efi/EFI/redhat
/boot/efi/EFI/redhat/fonts
/boot/efi/EFI/redhat/grubenv
/boot/efi/EFI/redhat/grubx64.efi

find /boot | grep centos:

/boot/efi/EFI/centos
/boot/efi/EFI/centos/shimx64-centos.efi
/boot/efi/EFI/centos/BOOTX64.CSV
/boot/efi/EFI/centos/mmx64.efi
/boot/efi/EFI/centos/grubenv
/boot/efi/EFI/centos/grub.cfg
/boot/efi/EFI/centos/shimx64.efi

rpm -qa | grep shim:

shim-x64-15-15.el8_2.x86_64

[Djelibeybi]

How are you running centos2ol.sh, i.e. what parameters are you using?

[metal4lyf]

With or without -k, doesn't seem to matter. When we don't pass -k, uek is installed but not enabled.
Shim does upgrade when we downgrade to BaseOS grub.
I've also verified with BaseOS grub that we can boot uek.

[Djelibeybi]

The shim-x64 package should be downgraded as part of the distro-sync that is run by default, i.e. after the switch you should have shim-x64-15-11 installed.

[Djelibeybi]

And if you don't pass -k, the UEK should be installed and enabled, again with the downgrade of shim. Something else is happening here. Can you run the switch and pipe the output to a log file so we can see the entire process? If possible, run the script with no parameters, i.e. bash centos2ol.sh | tee -a centos2ol.log

[aburmash]

@metal4lyf
So you have centos shim and Oracle grub, that explains the problem.
pretty sure if you will do

1. rpm -e shim-x64 ( remove centos shim )
2. yum install shim-x64 ( from Oracle repos )
   and do the reboot
   everything will automagically start working.

if NOT
you will still need to replace centos shim with oracle shim and do
efibootmgr -c -d /dev/sda -p 1 -L "Oracle Linux" -l "\EFI\redhat\shimx64.efi"

Where /dev/sda is the ESP disk
1 is the partition number.
You can do
mount |grep boot
and see what disk is mounted at /boot/efi to determine that.
( please notice, i am writing about ESP disk, not boot disk ).

EDIT: you may need to do
rpm -e shim-x64 --force
but careful(!): 100% install a new shim after removal of old one.

EDIT2: we still need to figure out why in your case shim was not replaced.

[metal4lyf]

Thanks, I'll wipe this system and stage it for another run tomorrow AM. I will update with the logs and then try your suggestions.
(The reason for CentOS shim and Oracle grub is because I downgraded grub to CentOS in recovery mode after the boot failed, which switched to CentOS shim, and thereafter upgraded grub to Oracle, which did not modify shim.)