RHEL, biosdevname, cobbler

I would include more links, etc., but I’m writing this in a bit of a rush.

It is well-documented that Dell and Red Hat decided to modify how Linux names network devices, starting with Fedora 15.  This project is called Consistent Network Device Naming.  It integrates with the biosdevname package to break improve network device naming.  In theory, each time a system boots, CNDN applies a set of rules so that each NIC is named the same each time.  One of the main factors for this is the problem you might have seen in times past when you added a PCI NIC to a system, and suddenly eth0 is now eth1 or some such.

While the motivation was good, CNDN has created problems, as we expect with any project which involves significant change. One of the primary claims, and a legitimate one, is that so many system tools and scripts (especially custom sysadmin scripts) are hard-coded for eth0.  Referring to eth0 has worked for years.  So bringing along a system-level change that now refers to the first NIC as em1 (for an on-board interface) breaks many scripts.

In our shop, using RHEL and cobbler for kickstarting systems, this reared its ugly head when, after installing a new Dell system with RHEL 6.3, networking was broken.  Logging into the console, we ran ifconfig -a and noticed all 4 NICs, but they were named em1 thru em4, rather than the eth0 thru eth3 we are accustomed to.  Moreover, em1 had no IP address. A quick peek in /var/log/messages showed us this gem:

Sep 21 10:16:08 sandmdm kernel: udev: renamed network interface eth0 to em1

That’s nice.  It would have been better had udev (or something else) also modified the network config files so that networking would actually work after udev makes its changes.  But alas, looking in the default location for its config, we noticed (aside from lo) only one NIC had a config file – eth0.  And of course ifconfig knew nothing about eth0.  So, changing the filename from ifcfg-eth0 to ifcfg-em1, and changing the device name inside the file to em1, we restarted networking and all was well.

But we wanted a better fix.  After lots of research, it turns out cobbler is hard-coded to assume eth0 is the first NIC in a system.  Moreover, even after anaconda (the RHEL installer) refers to it as em1 throughout the whole process, cobbler dutifully puts ifcfg-eth0 in /etc/sysconfig/network-scripts.  To us, it’s a show-stopper to have to login to a console to fix networking after a fresh install.  So what were our options?

  1. Deal with it, by making rules in udev
    Not really an option, because we’d have to put them in the post section of the kickstart.  Seems like too much of  a hack.
  2. Deal with it, by changing the interface setting to em1 in cobbler profiles for all RHEL6 systems.
    I didn’t really care for that option. Would have required some testing, because we’re not sure that would even work.
  3. Use biosdevname=0 in /etc/grub.conf to disable this tomfoolery.
    Not a bad idea.  Workable.  We’d have to modify the kernel options in the post section of kickstart.
  4. Remove the biosdevname package altogether.
    That’s the best, cleanest fix.

So, we decided to disable the installation of biosdevname during kickstart.  But it wasn’t that simple.  After lots of testing, we settled on letting biosdevname install, but disabling it in the kernel options (biosdevname=0).  We also had to, in the kickstart profile in RHN Satellite, add the same ‘biosdevname=0’ setting to both Kernel Options and Post Kernel Options.  That resulted in ‘eth0’ being the first interface after installation.

UPDATE – 10/05/2012

After more consideration and testing, we changed our approach.  While we would like to keep existing functionality (i.e. the first NIC is eth0), we realize that’s not a great option long-term.  We wanted to begin adjusting now to this change, rather than hit some wall later, when perhaps we have no choice but to install new systems with em1.

Our solution has been to change the name of the interface in the cobbler profile.  Steps involved:

-add the system to cobbler
-edit the system to add ’em1′ as an interface, make sure it has a DNS name and a MAC
cobbler system edit --name=somebox --interface=em1 --mac=11:22:33:44:55:66 --dns-name="somebox.example.com"
-remove ‘eth0’ from the profile
cobbler system edit --name=somebox --interface=eth0 --delete-interface
-in the kickstart profile, add ‘--device=em1‘ to the ‘network’ line (on the Advanced tab if you use RHN Satellite)

“The Red Hat Enterprise Linux installation tree in that directory does not seem to match your boot media”

In working with Red Hat Satellite Server 5.4.1, we had a problem with Dell PowerEdge R720 servers.  When we kickstarted those systems to RHEL 5.7, using DHCP, the Tigon3 NIC would flap at the point where the installer enables the NIC to get stage2.img, and DHCP would time out.

While troubleshooting this, I updated the kickstart tree to RHEL 5.8 (as a sidepoint to this post, that worked).  But alas, that tree wanted packages that are too new for RHEL 5.7 (I was hoping we could install RHEL 5.7 from the RHEL 5.8 kickstart images, but no).  So I reverted back to the previous kickstart images, and got the above error.

I found the problem by comparing files from /var/satellite/rhn/kickstart/my-installation-tree with /tftpboot/images/my-installation-tree.  I found that the two files in the latter directory (vmlinuz and initrd.img) were the same ones from the base tree for RHEL 5.8 (I did not change them initially, so I assume “cobbler sync” did).  They should have been the ones from the RHEL 5.7 tree.  So, when I copied the correct ones in and re-ran “cobbler sync“, we were good.

Lesson learned: vmlinuz and initrd.img in

and in
must match