Login via SSH is slow

Most of us have, at some time, experienced slow logins when connecting to a server via ssh. If you poke around you’ll discover that there are plenty of recommendations for addressing the issue. For most people the solutions fix their problem. And then there’s the sneaky one that made me say, “Oh, of course.”

1. Set UseDNS to “no” on the server to which you are connecting
This server-side flag tells the ssh daemon to do reverse-IP lookup before allowing the connection. If it struggles finding one or there isn’t one, it will time out and eventually connect. By default UseDNS is set to “yes”. Changing it to “no” and bouncing the ssh daemon will fix that.
For many people, this fixes it right away.
Setting, on server side, in /etc/ssh/sshd_config: UseDNS no
Restart ssh daemon on server after changing.

2. Set GSSAPIAuthentication to “no” on the client
The GSS API serves primarily to allow authentication via Kerberos. If that’s not something you need to use, make sure it’s off.
You can set it in /etc/ssh/ssh_config on most Linux distros. A line containing “GSSAPIAuthentication no” will do the trick. You may also want to verify it’s not set in ~/.ssh/config.
Setting, on client side, in /etc/ssh/ssh_config: GSSAPIAuthentication no
No restarts required

3. Disable the use of IPv6
I remember way back in the late 90s everyone saying we were in danger of running out of IP address space and needed to start planning to move to IPv6. It was not quite as urgent as Y2K, but it seemed pretty important. Things rocked on for a few years and they sounded the warnings again in the early aughts (early 2000s) – if we don’t move everything, and soon, we’ll be in trouble.
Well, here we are approaching 2020 and lots of things still run on IPv4. Still, a lot of folks enable IPv6 in their stuff by default. And that’s fine, but not everything does. And the things that do may also still use IPv4.
So how does that slow down ssh connections? Good question.
When I was troubleshooting this on a MacBook Pro ssh client, I ran “ssh -vvvv some_host” and noticed it was pausing on “resolving host” for several seconds. This was after I had put the destination IP/host combo in /etc/hosts and disabled UseDNS on the server side. I could ping the host by name with no delay in resolution. So why would ssh hang resolving the host name to an IP address? Then, like the baseball that kept getting bigger and bigger, it hit me. Maybe it was resolving IPv6 (after all, it’s on by default in just about every IP stack out there today).
So, what’s the flag to manipulate it? Is it “EnableIPv6“? Nah – too easy. It’s “AddressFamily“, which is set to “any” by default. When I set it to “inet”, my ssh session connected almost instantaneously.
You can also run ssh with the “-4” option on the fly; e.g. “ssh -4 some_host
Setting: on client side, in /etc/ssh/ssh_config: AddressFamily inet
No restarts required.

To wrap this up, let me remind you that “ssh -vvvv” is a very helpful troubleshooting tool. Also, never stop just because the common answers don’t fix your problem. Keep digging til you get it sorted out.

Disabling the crash kernel (and kdump) in CentOS/RHEL 6

Had a system where the user wanted to disable the crash kernel after a previous admin had enabled it.  I found a ton of write-ups showing how to enable and configure kdump, but few describing the opposite.

There are multiple approaches that work here, but for a quick win here’s what we did.

  1. Edit /boot/grub/grub.conf
  2. On the kernel line, find the text crashkernel= and set it to crashkernel=no
  3. Do chkconfig kdump off
  4. Reboot

If you don’t want to reboot, then you could do service kdump stop

VMware API error: “Customization of the guest operating system ‘rhel6_64Guest’ is not supported in this configuration”

While using VMware’s API and pyVmomi to clone templates into a VM, a couple of times I have run into this error:

Cannot deploy template: Customization of the guest operating system ‘rhel6_64Guest’ is not supported in this configuration. Microsoft Vista (TM) and Linux guests with Logical Volume Manager are supported only for recent ESX host and VMware Tools versions. Refer to vCenter documentation for supported configurations.

For me this happened when (A) I was cloning a RHEL 64-bit template (this occurred both with RHEL 6 and RHEL 7 but for entirely different reasons on each), and (B) I was using a customization specification. We had just stood up a new datacenter and I was laying the groundwork for creating masses of virtual machines from a command-line. A Linux command-line, mind you. Not from PowerCLI or PowerShell (which are certainly powerful, but hey, I’m a Linux guy).

In my case, the error had nothing to do with logical volumes (I suspect that’s true for most people who have run across this message). I have a script that has been happily cloning 64-bit LVM-based RHEL 6 templates in multiple vCenters that are on versions between 5.0 and 5.5. Since my customization spec was setting network parameters, I suspected that’s where the root of this problem was. So, before you go chasing that wild goose of LVM incompatibility, or telling vCenter your template is RHEL 5, I have a couple of recommendations.

In most of these cases, you’ll need to convert your template to a VM and power it up, then make the changes, shut it down, and convert it back to a template.

1. If this is happening with RHEL 7, make sure the vCenter is a new enough version to support it

One of our vCenters had not been updated in some time.  When I tried cloning the same RHEL 7 template from an updated vCenter (and had the correct VMware tools installed) all went well.

2. If this is happening with RHEL 6, Make sure your version is  6.4 or later

There were some updates in 6.4 that greatly improved RHEL’s interaction with VMware’s hypervisor, including at the NIC level.

3. Install VMware tools on the template VM

This is the biggest source of problems.  If you’re using  customization specifications during cloning, VMware cannot do the work unless the tools are installed in the image from which you’re cloning.  I like using the “3rd party” tools that you can get from VMware’s own repo. You can clone that repo into your own, or just use theirs. You can even bring it into Satellite or Spacewalk, either as an external repo or a software channel. Or maybe you like doing it the old fashioned way and installing them from vCenter. Bottom line – install those tools!

For RHEL 6 that means “vmware-tools-plugins-deployPkg” has to be present.  It will be if you simply install “vmware-tools-esx-nox” (I also include vmware-tools-esx-kmods).

For RHEL 7, you need open-vm-tools (which comes standard with RHEL 7) and open-vm-tools-deployPkg (which comes from VMware).  Note: If you’re using CentOS 7, open-vm-tools-deployPkg requires open-vm-tools < 9.5.  This created issues for me trying to install in CentOS 7.2, which ships only 9.10.  The solution?  Disable all repos except vmware-tools and C7.1.1503-base from the “vault” when you do the installation.
yum –disablerepo=* –enablerepo=vmware-tools –enablerepo=C7.1.1503-base install open-vm-tools-deployPkg

By the way, the symptoms of this in RHEL 7?  Aside from the customization error, you’ll have network cards named ens32 or ens33 in the template, but ens160 in the clone.  And obviously, networking will be broken.

Also, get rid of the udev stuff that sometimes messes up your NIC settings

This applies more to RHEL 6.  If you clone from a template, VMware will naturally give the new machine a new MAC address for any NICs in the template. In a single-NIC configuration, when udev sees this, it says, “Hey, I this isn’t the MAC for eth0, so let me make a configuration for eth1 using this new MAC and UUID.” You can help yourself here by doing the following (per VMware’s advice):

-remove /etc/udev/rules.d/70-persistent-net.rules

-remove /lib/udev/rules.d/75-persistent-net-generator.rules

You might not need to remove /lib/udev/rules.d/75-persistent-net-generator.rules, but I don’t think it’ll hurt. When I did these things to my RHEL 6 template, I was able to clone just fine using the API again.

How to setup EPEL repos on CentOS

If you’re wanting packages from the EPEL repos that are available to RHEL installations, it’s a simple thing to get them.

You can start at the main Fedora repository for EPEL, and work your way around.  Or if you’re in a hurry, use these instructions.
(Note: these are mirror sites; if they are inaccessible, refer back to the main repo linked abve, and you’ll get redirected to a different mirror in most cases)

For CentOS 5

  1. Install the repo RPM from here.
    # rpm -ivh http://mirror.symnds.com/distributions/fedora-epel/5/x86_64/epel-release-5-4.noarch.rpm
  2. Install the GPG key from here.
    # rpm --import http://mirror.symnds.com/distributions/fedora-epel/RPM-GPG-KEY-EPEL-5

For CentOS 6

  1. Install the repo RPM from here.
    # rpm -ivh http://mirror.symnds.com/distributions/fedora-epel/6/x86_64/epel-release-6-8.noarch.rpm
  2. Install the GPG key from here.
    # rpm --import http://mirror.symnds.com/distributions/fedora-epel/RPM-GPG-KEY-EPEL-6

For either one, if you do not want the repo used by default when you run yum, edit /etc/yum.repos.d/epel.repo, and in the section named [epel] change enabled=1 to enabled=0
Then, to use the repo, add --enabelrepo=epel to your yum commands.  For example:
# yum --enablerepo=epel update

The time I tried compiling an RPM for syslog-ng

Introduction

Had a client for whom I wanted to install syslog-ng.  I’ve never worked with it before, but had read that it can log to a MySQL database.  The project at hand seemed like a great fit for just such a thing, so off I went to get syslog-ng.

(Update on Nov 28, 2012: When I realized that rsyslog comes with RHEL 5 and supports MySQL out of the box, I switched the project to it instead)

Build my own?  No problem.

Having worked with Linx and open source for nearly 2 decades now, I’m accustomed to having a look to find a software package occasionally.  Since most of that time has been spent working with Red Hat, Fedora, and CentOS, I’m also accustomed to having to go out of my way to find RPM versions of packages.  It’s common to find that someone who maintains an application doesn’t by default make an RPM for it.  Sometimes that’s because their experience is in another flavor of Linux or UNIX that doesn’t natively use RPMs (aka Debian, Slackware, *BSD), or because they don’t have time to make an RPM.

Since I didn’t find an RPM readily available for this install (on CentOS 5.8), I started looking around.  My usual sources (rpmforge, etc.) didn’t yield anything.  So I turned to rpmfind.net.  Right away I saw that there is a package available for RHEL 5, thus presumably for CentOS as well.  Unfortunately, though, the CentOs extras repo didn’t have the package.  Bummer.  But the sources for Red Hat’s EPEL packages (http://download.fedoraproject.org/pub/epel/5/SRPMS) had a source RPM.  For me, that’s usually enough to get started.  So, a quick wget and I was on my way:

$ wget http://download.fedoraproject.org/pub/epel/4/SRPMS/syslog-ng-2.1.4-1.el4.src.rpm

From here I usually do either:

rpmbuild --rebuild syslog-ng-2.1.4-1.el4.src.rpm

or I install the src.rpm and do:

rpmbuild -bb /path/to/syslog-ng.spec

But it wasn’t that simple

To make an incredibly long story a bit shorter, I had build errors.  And the reason I did was that this package wanted a devel package that I couldn’t find right away (try finding tcp_wrappers-devel-7.6-40).  What I ended up doing was finding the source RPM for tcp_wrappers and building a new version of it.  Also, I had to build the eventlog RPM that goes along with syslog-ng (same company makes both). Great!  Now all I had to do was build syslog-ng, finally!

Except that it kept croaking, telling me “ld: cannot find -lwrap“.  Now, I thought this thing was looking for libwrap.so, so I made sure it (or some symlink to it) was in all the common lib directories.  Nope, no luck.  Ah, I need to run ldconfig, I thought.  Nope, didn’t help.

Next I popped open the source tarball and look in it.  I saw some references to /usr/local/lib.  I changed those to /usr/lib, re-packed the tarbarll, stuck it in the SOURCES directory in my build environment, and ran it again….nope.  After more digging in Makefiles and configure files, I found that it apparently wanted libwrap.a to be in the lib paths.  But…my tcp_wrappers packages did not install that file.  Odd.  I updated my locate database and did ‘locate libwrap.a‘.  That file existed only in the RPM BUILD directory.  Looking in the spec file, I found that it had been removed around 7.6-42 by the folks at Red Hat.  Perhaps they were purging out old static libraries.  I manipulated the spec file to make it install in /usr/lib and…

Finally

Finally syslog-ng compiled.  So, to get syslog-ng to build, I had to install and/or build these packages:

eventlog-0.2.12-1.x86_64
eventlog-devel-0.2.12-1.x86_64
eventlog-static-0.2.12-1.x86_64
libnet-1.1.5-1.el5.x86_64
libnet-devel-1.1.5-1.el5.x86_64
tcp_wrappers-7.6-59.x86_64
tcp_wrappers-devel-7.6-59.x86_64
tcp_wrappers-libs-7.6-59.x86_64

(the version of tcp_wrappers was 7.6-57 when I got the SRPM, but I bumped the version a couple of times to account for changes I made in the spec file)

Now, this was just on my build server.  On the actual client machine, I installed these RPMs:

 eventlog
eventlog-devel
eventlog-static
syslog-ng
libnet        

The libnet package was already installed, so it was an update.

What’s next?

I know that version 3.2 of syslog-ng comes with RHEL/CentOS 6 EPEL.  I’m going to see if I can get it to build for a CentOS 5 environment, since 2.1 seems rather old by comparison.

Additionally, now I will be working to get the service working with mysql.  But that’s probably another post…  🙂

Moving a Postfix virtual user’s mail to a new address

A user was on too many spam lists and wanted a new mailbox.  He had removed all the spam, and wanted all the existing messages moved over.  The hosting server runs Postfix in a virtual setup.  The script below is what I used.  It makes it easy to move mailboxes in the future for my client.  The script takes two options, and expects the first to be the old email address, the second to be the new email address.  It could stand some error checking, etc., but since I expect it to be run by a clue-possessing administrator, I didn’t put any in here.  This script assumes that the destination mailbox has already been created (in our case, by using PostfixAdmin).

 

#!/bin/bash
mailroot="/var/spool/postfix/virtual"
usersource=$1
userdest=$2
sourcedir="${mailroot}/${usersource}"
destdir="${mailroot}/${userdest}"
echo "Source dir is ${sourcedir}"
echo "Dest dir is ${destdir}"
read -p "Press any key to begin" 
cd ${sourcedir}
for i in *; do
    mv -v $i ${destdir}/
done
cd -