What to do when grub2 does not honor the default kernel choice

Recently I updated a Fedora 22 system to kernel 4.2.5 (from 4.1).  That broke a few things, because one of my apps needed kernel-specific modules compiled.  Something in this kernel version was too out of whack for me to fix it on the fly, so I decided to boot to the previous kernel until I could find a workaround or patch.

Not so fast.

In the old days (of grub pre-v2), I could do this easily.  But since we’re now on grub2, this became a little more complex.  Even with early versions of grub2 this would have been simpler.  But currently, grub2 uses variables, symlinks, and some black magic to work.

What failed

After doing some reading and understanding that my system is using the “DEFAULT=saved” method, I was (I thought) on my way to making a simple change and rebooting.  In short, I should have been able to do something like this:

grub2-set-default "Fedora (4.1.8-200.fc22.x86_64) 22 (Twenty Two)"
grub2-mkconfig -o /boot/grub2/grub.cfg

But every time I booted, the system went right back to kernel 4.2.5.  After more digging I also checked /boot/grub2/grubenv (which is a symlink to /boot/efi/EFI/fedora/grubenv).  And it showed kernel 4.1.8 should be the default.

But it wasn’t, at least not at boot time.

So I read through Fedora’s wiki page on grub2.  And there I found a clue.  I quote:

The above method [what I tried] fails to work on some F20 systems due to a missing or improperly linked /boot/grub2/grubenv file. The /boot/grub2/grubenv is symbolic linked to /boot/efi/EFI/fedora/grubenv but /boot is not mounted at the time of booting. So grub2 does not have access to the environment variables. To fix this, change /boot/grub2/grubenv to point to ../efi/EFI/fedora/grubenv instead and your chosen default OS will boot without any problems.

So, the fix was this:

cd /boot/grub2
ln -sf ../efi/EFI/fedora/grubenv grubenv

And rebooted, and my default kernel was finally 4.1.8 again.  The reason this rarely manifests itself is that, generally, we want to boot the first kernel.  And when grub2 cannot reference files under /boot (because it isn’t mounted when grub runs), it just picks the first kernel it finds in grub.cfg and goes with it.

In summary, to change your default kernel in Fedora 22 (perhaps 20 and 21 as well), find the label of the kernel you want to change, set it to the default, make a new grub.cfg, and verify where grubenv points to.

# egrep ^menuentry /boot/grub2/grub.cfg  | awk -F\' '{print $2}'
Fedora (4.2.5-201.fc22.x86_64) 22 (Twenty Two)
Fedora (4.1.10-200.fc22.x86_64) 22 (Twenty Two)
Fedora (4.1.8-200.fc22.x86_64) 22 (Twenty Two)
Fedora (0-rescue-f22b46fe9c554c92890b4e9d760ac718) 22 (Twenty Two)
(copy the text of the kernel name to the clipboard)
# grub2-set-default "Fedora (4.1.8-200.fc22.x86_64) 22 (Twenty Two)"

And if needed:

cd /boot/grub2
ln -sf ../efi/EFI/fedora/grubenv grubenv