Login via SSH is slow

Most of us have, at some time, experienced slow logins when connecting to a server via ssh. If you poke around you’ll discover that there are plenty of recommendations for addressing the issue. For most people the solutions fix their problem. And then there’s the sneaky one that made me say, “Oh, of course.”

1. Set UseDNS to “no” on the server to which you are connecting
This server-side flag tells the ssh daemon to do reverse-IP lookup before allowing the connection. If it struggles finding one or there isn’t one, it will time out and eventually connect. By default UseDNS is set to “yes”. Changing it to “no” and bouncing the ssh daemon will fix that.
For many people, this fixes it right away.
Setting, on server side, in /etc/ssh/sshd_config: UseDNS no
Restart ssh daemon on server after changing.

2. Set GSSAPIAuthentication to “no” on the client
The GSS API serves primarily to allow authentication via Kerberos. If that’s not something you need to use, make sure it’s off.
You can set it in /etc/ssh/ssh_config on most Linux distros. A line containing “GSSAPIAuthentication no” will do the trick. You may also want to verify it’s not set in ~/.ssh/config.
Setting, on client side, in /etc/ssh/ssh_config: GSSAPIAuthentication no
No restarts required

3. Disable the use of IPv6
I remember way back in the late 90s everyone saying we were in danger of running out of IP address space and needed to start planning to move to IPv6. It was not quite as urgent as Y2K, but it seemed pretty important. Things rocked on for a few years and they sounded the warnings again in the early aughts (early 2000s) – if we don’t move everything, and soon, we’ll be in trouble.
Well, here we are approaching 2020 and lots of things still run on IPv4. Still, a lot of folks enable IPv6 in their stuff by default. And that’s fine, but not everything does. And the things that do may also still use IPv4.
So how does that slow down ssh connections? Good question.
When I was troubleshooting this on a MacBook Pro ssh client, I ran “ssh -vvvv some_host” and noticed it was pausing on “resolving host” for several seconds. This was after I had put the destination IP/host combo in /etc/hosts and disabled UseDNS on the server side. I could ping the host by name with no delay in resolution. So why would ssh hang resolving the host name to an IP address? Then, like the baseball that kept getting bigger and bigger, it hit me. Maybe it was resolving IPv6 (after all, it’s on by default in just about every IP stack out there today).
So, what’s the flag to manipulate it? Is it “EnableIPv6“? Nah – too easy. It’s “AddressFamily“, which is set to “any” by default. When I set it to “inet”, my ssh session connected almost instantaneously.
You can also run ssh with the “-4” option on the fly; e.g. “ssh -4 some_host
Setting: on client side, in /etc/ssh/ssh_config: AddressFamily inet
No restarts required.

To wrap this up, let me remind you that “ssh -vvvv” is a very helpful troubleshooting tool. Also, never stop just because the common answers don’t fix your problem. Keep digging til you get it sorted out.

Disabling the crash kernel (and kdump) in CentOS/RHEL 6

Had a system where the user wanted to disable the crash kernel after a previous admin had enabled it.  I found a ton of write-ups showing how to enable and configure kdump, but few describing the opposite.

There are multiple approaches that work here, but for a quick win here’s what we did.

  1. Edit /boot/grub/grub.conf
  2. On the kernel line, find the text crashkernel= and set it to crashkernel=no
  3. Do chkconfig kdump off
  4. Reboot

If you don’t want to reboot, then you could do service kdump stop

Local Plex Media Server Will Not Allow Login

Went to open my Plex server (running on CentOS 7) and saw that there was only one user listed: me.  Seemed odd since I have several users on that installation.  Clicked the drop-down by my avatar and noticed I appeared to be signed out.  “No problem, I’ll just sign in.”
Except I couldn’t.  Upon clicking to login I was presented with a page stating:

“Plex is not reachable.
Make sure your server has an internet connection and any firewalls or other programs are set to allow access.”

After some brief internet searching, including the Plex forums, found the apparently tried-and-true solution of deleting two bundle directories.
However, these instructions were not accurate in my case.  They list the directory containing these bundles as being:

$PLEX_HOME/Library/Application Support/Plex Media Server/Plug-ins/

Where $PLEX_HOME is /var/lib/plexmediaserver.  In my case, I ended up finding the directories in:

/usr/lib/plexmediaserver/Resources/Plug-ins-4613ce077

So here’s what I did:

systemctl stop plexmediaserver
cd /usr/lib/plexmediaserver/Resources/Plug-ins-4613ce077
rm -rfv System.bundle
rm -rfv Framework.bundle
systemctl start plexmediaserver

And all was well again.

SSH sessions seem to freeze after period of inactivity

I was having an issue with my SSH sessions where, after a period of inactivity, they would freeze. They weren’t disconnecting or timing out; they were just frozen.

To solve this I made the following change. In /etc/ssh/ssh_config (works also for ~/.ssh/config for individual users), I added two lines to the “Host *” section:

ServerAliveInterval 15
ServerAliveCountMax 5

Without going into a lot of detail, you can know that those are somewhat arbitrary – you can adjust those to your needs.
My entire “Host *” section looks like this (this is a RHEL system):

Host *

GSSAPIAuthentication yes
ForwardX11Trusted yes
SendEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY   LC_MESSAGES
SendEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE   LC_MEASUREMENT
SendEnv LC_IDENTIFICATION LC_ALL LANGUAGE
SendEnv XMODIFIERS
ServerAliveInterval 15
ServerAliveCountMax 5

Also, if you want to help out with connections TO your server, a similar directive in the main body of /etc/ssh/sshd_config will work:

ClientAliveInterval 15
ClientAliveCountMax 5

Since this applies to the SSH server, don’t forget to restart the service with
service sshd restart
or
systemctl restart sshd.service

vmware tools install (or vmware-config-tools.pl) hangs after X config

I was updating vmware-tools on some old Linux installations of ours, using a tarball I downloaded from VMware. Once I unzipped the tarball, I ran the installer and told it to take the defaults:
./vmware-install.pl -d

Got to one machine in particular and it hung right after this text:
X is running fine with the new config file

Checked the processes and it seemed “mkinitrd --help” was holding things up. Killed it (just to see what happened) and another “mkinitrd” came up that wanted to build for the installed kernel. But that also hung. Killed the second “mkinitrd” and it warned me about not having a proper initrd and all the horrible things that might happen in life if I rebooted with this botched boot configuration.

So I ran vmware-config-tools.pl manually, then I ran strace on that process, and it occasionally did a call-out to an old AD/LDAP server of ours that is deprecated. What the heck? Also saw it read from /etc/ldap.conf as it did this.

So I killed the config script and moved /etc/ldap.conf to /etc/oldldap.conf. Re-ran the installer, and “whee” it all worked.

Didn’t feel like digging into it further, but I did get the tools installed and configured.

YMMV

Changing WordPress permalinks from default to postname

Had an issue on a WordPress site after I changed the permalinks to “%postname%” from the default.  Immediately after the change, all my wonderful posts gave 404 when I clicked on them.

So I researched it a bit and found some good recommendations (make sure mod_rewrite is present, check permissions on .htaccess).  Initially it looked like a simple fix – the .htaccess file needed to be populated (I didn’t allow the web server daemon write access to that file or the site’s directory).  So I added the recommended code to that file directly (by the way, that file needs to be in DocumentRoot for your site).  However, that still didn’t fix it.

So I poked around some more and a couple of sites recommended checking AllowOverride and similar settings.  In my /etc/httpd/conf/httpd.conf, AllowOverride was set to None.

My sites are in a separate file in /etc/httpd/conf.d so I opened that file and added the following under the VirtualHost section for that site:

<Directory "/site/home">
AllowOverride Options FileInfo
Options FollowSymLinks
RewriteEngine On
</Directory>

(I obfuscated the path above, so in your file, whether /etc/httpd/conf/httpd.conf or a site-specific file in /etc/httpd/conf.d, Directory will certainly not be “/site/home“).  You can set AllowOverride to All, but do so at your own risk.

Then I bounced httpd and ta-da, it worked.

So, in summary:

-Check to make sure your web server loads mod_rewrite
-Set .htaccess (found in your site’s root, DocumentRoot) to writable so WordPress can modify it, OR modify it yourself by pasting in the code that WordPress shows you
-In your site’s config (either /etc/httpd/conf/httpd.conf or your site-specific file in /etc/httpd/conf.d), add:

<Directory "/site/home">
AllowOverride Options FileInfo
Options FollowSymLinks
RewriteEngine On
</Directory>

-Bounce the web service

 

What to do when grub2 does not honor the default kernel choice

Recently I updated a Fedora 22 system to kernel 4.2.5 (from 4.1).  That broke a few things, because one of my apps needed kernel-specific modules compiled.  Something in this kernel version was too out of whack for me to fix it on the fly, so I decided to boot to the previous kernel until I could find a workaround or patch.

Not so fast.

In the old days (of grub pre-v2), I could do this easily.  But since we’re now on grub2, this became a little more complex.  Even with early versions of grub2 this would have been simpler.  But currently, grub2 uses variables, symlinks, and some black magic to work.

What failed

After doing some reading and understanding that my system is using the “DEFAULT=saved” method, I was (I thought) on my way to making a simple change and rebooting.  In short, I should have been able to do something like this:

grub2-set-default "Fedora (4.1.8-200.fc22.x86_64) 22 (Twenty Two)"
grub2-mkconfig -o /boot/grub2/grub.cfg

But every time I booted, the system went right back to kernel 4.2.5.  After more digging I also checked /boot/grub2/grubenv (which is a symlink to /boot/efi/EFI/fedora/grubenv).  And it showed kernel 4.1.8 should be the default.

But it wasn’t, at least not at boot time.

So I read through Fedora’s wiki page on grub2.  And there I found a clue.  I quote:

The above method [what I tried] fails to work on some F20 systems due to a missing or improperly linked /boot/grub2/grubenv file. The /boot/grub2/grubenv is symbolic linked to /boot/efi/EFI/fedora/grubenv but /boot is not mounted at the time of booting. So grub2 does not have access to the environment variables. To fix this, change /boot/grub2/grubenv to point to ../efi/EFI/fedora/grubenv instead and your chosen default OS will boot without any problems.

So, the fix was this:

cd /boot/grub2
ln -sf ../efi/EFI/fedora/grubenv grubenv

And rebooted, and my default kernel was finally 4.1.8 again.  The reason this rarely manifests itself is that, generally, we want to boot the first kernel.  And when grub2 cannot reference files under /boot (because it isn’t mounted when grub runs), it just picks the first kernel it finds in grub.cfg and goes with it.

In summary, to change your default kernel in Fedora 22 (perhaps 20 and 21 as well), find the label of the kernel you want to change, set it to the default, make a new grub.cfg, and verify where grubenv points to.

# egrep ^menuentry /boot/grub2/grub.cfg  | awk -F\' '{print $2}'
Fedora (4.2.5-201.fc22.x86_64) 22 (Twenty Two)
Fedora (4.1.10-200.fc22.x86_64) 22 (Twenty Two)
Fedora (4.1.8-200.fc22.x86_64) 22 (Twenty Two)
Fedora (0-rescue-f22b46fe9c554c92890b4e9d760ac718) 22 (Twenty Two)
(copy the text of the kernel name to the clipboard)
# grub2-set-default "Fedora (4.1.8-200.fc22.x86_64) 22 (Twenty Two)"

And if needed:

cd /boot/grub2
ln -sf ../efi/EFI/fedora/grubenv grubenv

VMware API error: “Customization of the guest operating system ‘rhel6_64Guest’ is not supported in this configuration”

While using VMware’s API and pyVmomi to clone templates into a VM, a couple of times I have run into this error:

Cannot deploy template: Customization of the guest operating system ‘rhel6_64Guest’ is not supported in this configuration. Microsoft Vista (TM) and Linux guests with Logical Volume Manager are supported only for recent ESX host and VMware Tools versions. Refer to vCenter documentation for supported configurations.

For me this happened when (A) I was cloning a RHEL 64-bit template (this occurred both with RHEL 6 and RHEL 7 but for entirely different reasons on each), and (B) I was using a customization specification. We had just stood up a new datacenter and I was laying the groundwork for creating masses of virtual machines from a command-line. A Linux command-line, mind you. Not from PowerCLI or PowerShell (which are certainly powerful, but hey, I’m a Linux guy).

In my case, the error had nothing to do with logical volumes (I suspect that’s true for most people who have run across this message). I have a script that has been happily cloning 64-bit LVM-based RHEL 6 templates in multiple vCenters that are on versions between 5.0 and 5.5. Since my customization spec was setting network parameters, I suspected that’s where the root of this problem was. So, before you go chasing that wild goose of LVM incompatibility, or telling vCenter your template is RHEL 5, I have a couple of recommendations.

In most of these cases, you’ll need to convert your template to a VM and power it up, then make the changes, shut it down, and convert it back to a template.

1. If this is happening with RHEL 7, make sure the vCenter is a new enough version to support it

One of our vCenters had not been updated in some time.  When I tried cloning the same RHEL 7 template from an updated vCenter (and had the correct VMware tools installed) all went well.

2. If this is happening with RHEL 6, Make sure your version is  6.4 or later

There were some updates in 6.4 that greatly improved RHEL’s interaction with VMware’s hypervisor, including at the NIC level.

3. Install VMware tools on the template VM

This is the biggest source of problems.  If you’re using  customization specifications during cloning, VMware cannot do the work unless the tools are installed in the image from which you’re cloning.  I like using the “3rd party” tools that you can get from VMware’s own repo. You can clone that repo into your own, or just use theirs. You can even bring it into Satellite or Spacewalk, either as an external repo or a software channel. Or maybe you like doing it the old fashioned way and installing them from vCenter. Bottom line – install those tools!

For RHEL 6 that means “vmware-tools-plugins-deployPkg” has to be present.  It will be if you simply install “vmware-tools-esx-nox” (I also include vmware-tools-esx-kmods).

For RHEL 7, you need open-vm-tools (which comes standard with RHEL 7) and open-vm-tools-deployPkg (which comes from VMware).  Note: If you’re using CentOS 7, open-vm-tools-deployPkg requires open-vm-tools < 9.5.  This created issues for me trying to install in CentOS 7.2, which ships only 9.10.  The solution?  Disable all repos except vmware-tools and C7.1.1503-base from the “vault” when you do the installation.
yum –disablerepo=* –enablerepo=vmware-tools –enablerepo=C7.1.1503-base install open-vm-tools-deployPkg

By the way, the symptoms of this in RHEL 7?  Aside from the customization error, you’ll have network cards named ens32 or ens33 in the template, but ens160 in the clone.  And obviously, networking will be broken.

Also, get rid of the udev stuff that sometimes messes up your NIC settings

This applies more to RHEL 6.  If you clone from a template, VMware will naturally give the new machine a new MAC address for any NICs in the template. In a single-NIC configuration, when udev sees this, it says, “Hey, I this isn’t the MAC for eth0, so let me make a configuration for eth1 using this new MAC and UUID.” You can help yourself here by doing the following (per VMware’s advice):

-remove /etc/udev/rules.d/70-persistent-net.rules

-remove /lib/udev/rules.d/75-persistent-net-generator.rules

You might not need to remove /lib/udev/rules.d/75-persistent-net-generator.rules, but I don’t think it’ll hurt. When I did these things to my RHEL 6 template, I was able to clone just fine using the API again.

How to disable that pesky auto-comment in vim

Been using a newer release of vim and dislike how, after you type a comment on a line, vim assumes you want the next line commented also? Or, more frustrating, have you pasted something you copied from another file or from the web, only to find vim helpfully commented and indented a bunch of lines? Here’s a quick fix. Place it in your vimrc:

autocmd FileType * setlocal formatoptions-=c formatoptions-=r formatoptions-=o

Source: http://vim.wikia.com/wiki/Disable_automatic_comment_insertion