VMware API error: “Customization of the guest operating system ‘rhel6_64Guest’ is not supported in this configuration”

While using VMware’s API and pyVmomi to clone templates into a VM, a couple of times I have run into this error:

Cannot deploy template: Customization of the guest operating system ‘rhel6_64Guest’ is not supported in this configuration. Microsoft Vista (TM) and Linux guests with Logical Volume Manager are supported only for recent ESX host and VMware Tools versions. Refer to vCenter documentation for supported configurations.

For me this happened when (A) I was cloning a RHEL 64-bit template (this occurred both with RHEL 6 and RHEL 7 but for entirely different reasons on each), and (B) I was using a customization specification. We had just stood up a new datacenter and I was laying the groundwork for creating masses of virtual machines from a command-line. A Linux command-line, mind you. Not from PowerCLI or PowerShell (which are certainly powerful, but hey, I’m a Linux guy).

In my case, the error had nothing to do with logical volumes (I suspect that’s true for most people who have run across this message). I have a script that has been happily cloning 64-bit LVM-based RHEL 6 templates in multiple vCenters that are on versions between 5.0 and 5.5. Since my customization spec was setting network parameters, I suspected that’s where the root of this problem was. So, before you go chasing that wild goose of LVM incompatibility, or telling vCenter your template is RHEL 5, I have a couple of recommendations.

In most of these cases, you’ll need to convert your template to a VM and power it up, then make the changes, shut it down, and convert it back to a template.

1. If this is happening with RHEL 7, make sure the vCenter is a new enough version to support it

One of our vCenters had not been updated in some time.  When I tried cloning the same RHEL 7 template from an updated vCenter (and had the correct VMware tools installed) all went well.

2. If this is happening with RHEL 6, Make sure your version is  6.4 or later

There were some updates in 6.4 that greatly improved RHEL’s interaction with VMware’s hypervisor, including at the NIC level.

3. Install VMware tools on the template VM

This is the biggest source of problems.  If you’re using  customization specifications during cloning, VMware cannot do the work unless the tools are installed in the image from which you’re cloning.  I like using the “3rd party” tools that you can get from VMware’s own repo. You can clone that repo into your own, or just use theirs. You can even bring it into Satellite or Spacewalk, either as an external repo or a software channel. Or maybe you like doing it the old fashioned way and installing them from vCenter. Bottom line – install those tools!

For RHEL 6 that means “vmware-tools-plugins-deployPkg” has to be present.  It will be if you simply install “vmware-tools-esx-nox” (I also include vmware-tools-esx-kmods).

For RHEL 7, you need open-vm-tools (which comes standard with RHEL 7) and open-vm-tools-deployPkg (which comes from VMware).  Note: If you’re using CentOS 7, open-vm-tools-deployPkg requires open-vm-tools < 9.5.  This created issues for me trying to install in CentOS 7.2, which ships only 9.10.  The solution?  Disable all repos except vmware-tools and C7.1.1503-base from the “vault” when you do the installation.
yum –disablerepo=* –enablerepo=vmware-tools –enablerepo=C7.1.1503-base install open-vm-tools-deployPkg

By the way, the symptoms of this in RHEL 7?  Aside from the customization error, you’ll have network cards named ens32 or ens33 in the template, but ens160 in the clone.  And obviously, networking will be broken.

Also, get rid of the udev stuff that sometimes messes up your NIC settings

This applies more to RHEL 6.  If you clone from a template, VMware will naturally give the new machine a new MAC address for any NICs in the template. In a single-NIC configuration, when udev sees this, it says, “Hey, I this isn’t the MAC for eth0, so let me make a configuration for eth1 using this new MAC and UUID.” You can help yourself here by doing the following (per VMware’s advice):

-remove /etc/udev/rules.d/70-persistent-net.rules

-remove /lib/udev/rules.d/75-persistent-net-generator.rules

You might not need to remove /lib/udev/rules.d/75-persistent-net-generator.rules, but I don’t think it’ll hurt. When I did these things to my RHEL 6 template, I was able to clone just fine using the API again.

How to setup EPEL repos on CentOS

If you’re wanting packages from the EPEL repos that are available to RHEL installations, it’s a simple thing to get them.

You can start at the main Fedora repository for EPEL, and work your way around.  Or if you’re in a hurry, use these instructions.
(Note: these are mirror sites; if they are inaccessible, refer back to the main repo linked abve, and you’ll get redirected to a different mirror in most cases)

For CentOS 5

  1. Install the repo RPM from here.
    # rpm -ivh http://mirror.symnds.com/distributions/fedora-epel/5/x86_64/epel-release-5-4.noarch.rpm
  2. Install the GPG key from here.
    # rpm --import http://mirror.symnds.com/distributions/fedora-epel/RPM-GPG-KEY-EPEL-5

For CentOS 6

  1. Install the repo RPM from here.
    # rpm -ivh http://mirror.symnds.com/distributions/fedora-epel/6/x86_64/epel-release-6-8.noarch.rpm
  2. Install the GPG key from here.
    # rpm --import http://mirror.symnds.com/distributions/fedora-epel/RPM-GPG-KEY-EPEL-6

For either one, if you do not want the repo used by default when you run yum, edit /etc/yum.repos.d/epel.repo, and in the section named [epel] change enabled=1 to enabled=0
Then, to use the repo, add --enabelrepo=epel to your yum commands.  For example:
# yum --enablerepo=epel update

RHEL, biosdevname, cobbler

I would include more links, etc., but I’m writing this in a bit of a rush.

It is well-documented that Dell and Red Hat decided to modify how Linux names network devices, starting with Fedora 15.  This project is called Consistent Network Device Naming.  It integrates with the biosdevname package to break improve network device naming.  In theory, each time a system boots, CNDN applies a set of rules so that each NIC is named the same each time.  One of the main factors for this is the problem you might have seen in times past when you added a PCI NIC to a system, and suddenly eth0 is now eth1 or some such.

While the motivation was good, CNDN has created problems, as we expect with any project which involves significant change. One of the primary claims, and a legitimate one, is that so many system tools and scripts (especially custom sysadmin scripts) are hard-coded for eth0.  Referring to eth0 has worked for years.  So bringing along a system-level change that now refers to the first NIC as em1 (for an on-board interface) breaks many scripts.

In our shop, using RHEL and cobbler for kickstarting systems, this reared its ugly head when, after installing a new Dell system with RHEL 6.3, networking was broken.  Logging into the console, we ran ifconfig -a and noticed all 4 NICs, but they were named em1 thru em4, rather than the eth0 thru eth3 we are accustomed to.  Moreover, em1 had no IP address. A quick peek in /var/log/messages showed us this gem:

Sep 21 10:16:08 sandmdm kernel: udev: renamed network interface eth0 to em1

That’s nice.  It would have been better had udev (or something else) also modified the network config files so that networking would actually work after udev makes its changes.  But alas, looking in the default location for its config, we noticed (aside from lo) only one NIC had a config file – eth0.  And of course ifconfig knew nothing about eth0.  So, changing the filename from ifcfg-eth0 to ifcfg-em1, and changing the device name inside the file to em1, we restarted networking and all was well.

But we wanted a better fix.  After lots of research, it turns out cobbler is hard-coded to assume eth0 is the first NIC in a system.  Moreover, even after anaconda (the RHEL installer) refers to it as em1 throughout the whole process, cobbler dutifully puts ifcfg-eth0 in /etc/sysconfig/network-scripts.  To us, it’s a show-stopper to have to login to a console to fix networking after a fresh install.  So what were our options?

  1. Deal with it, by making rules in udev
    Not really an option, because we’d have to put them in the post section of the kickstart.  Seems like too much of  a hack.
  2. Deal with it, by changing the interface setting to em1 in cobbler profiles for all RHEL6 systems.
    I didn’t really care for that option. Would have required some testing, because we’re not sure that would even work.
  3. Use biosdevname=0 in /etc/grub.conf to disable this tomfoolery.
    Not a bad idea.  Workable.  We’d have to modify the kernel options in the post section of kickstart.
  4. Remove the biosdevname package altogether.
    That’s the best, cleanest fix.

So, we decided to disable the installation of biosdevname during kickstart.  But it wasn’t that simple.  After lots of testing, we settled on letting biosdevname install, but disabling it in the kernel options (biosdevname=0).  We also had to, in the kickstart profile in RHN Satellite, add the same ‘biosdevname=0’ setting to both Kernel Options and Post Kernel Options.  That resulted in ‘eth0’ being the first interface after installation.

UPDATE – 10/05/2012

After more consideration and testing, we changed our approach.  While we would like to keep existing functionality (i.e. the first NIC is eth0), we realize that’s not a great option long-term.  We wanted to begin adjusting now to this change, rather than hit some wall later, when perhaps we have no choice but to install new systems with em1.

Our solution has been to change the name of the interface in the cobbler profile.  Steps involved:

-add the system to cobbler
-edit the system to add ’em1′ as an interface, make sure it has a DNS name and a MAC
cobbler system edit --name=somebox --interface=em1 --mac=11:22:33:44:55:66 --dns-name="somebox.example.com"
-remove ‘eth0’ from the profile
cobbler system edit --name=somebox --interface=eth0 --delete-interface
-in the kickstart profile, add ‘--device=em1‘ to the ‘network’ line (on the Advanced tab if you use RHN Satellite)