Login via SSH is slow

Most of us have, at some time, experienced slow logins when connecting to a server via ssh. If you poke around you’ll discover that there are plenty of recommendations for addressing the issue. For most people the solutions fix their problem. And then there’s the sneaky one that made me say, “Oh, of course.”

1. Set UseDNS to “no” on the server to which you are connecting
This server-side flag tells the ssh daemon to do reverse-IP lookup before allowing the connection. If it struggles finding one or there isn’t one, it will time out and eventually connect. By default UseDNS is set to “yes”. Changing it to “no” and bouncing the ssh daemon will fix that.
For many people, this fixes it right away.
Setting, on server side, in /etc/ssh/sshd_config: UseDNS no
Restart ssh daemon on server after changing.

2. Set GSSAPIAuthentication to “no” on the client
The GSS API serves primarily to allow authentication via Kerberos. If that’s not something you need to use, make sure it’s off.
You can set it in /etc/ssh/ssh_config on most Linux distros. A line containing “GSSAPIAuthentication no” will do the trick. You may also want to verify it’s not set in ~/.ssh/config.
Setting, on client side, in /etc/ssh/ssh_config: GSSAPIAuthentication no
No restarts required

3. Disable the use of IPv6
I remember way back in the late 90s everyone saying we were in danger of running out of IP address space and needed to start planning to move to IPv6. It was not quite as urgent as Y2K, but it seemed pretty important. Things rocked on for a few years and they sounded the warnings again in the early aughts (early 2000s) – if we don’t move everything, and soon, we’ll be in trouble.
Well, here we are approaching 2020 and lots of things still run on IPv4. Still, a lot of folks enable IPv6 in their stuff by default. And that’s fine, but not everything does. And the things that do may also still use IPv4.
So how does that slow down ssh connections? Good question.
When I was troubleshooting this on a MacBook Pro ssh client, I ran “ssh -vvvv some_host” and noticed it was pausing on “resolving host” for several seconds. This was after I had put the destination IP/host combo in /etc/hosts and disabled UseDNS on the server side. I could ping the host by name with no delay in resolution. So why would ssh hang resolving the host name to an IP address? Then, like the baseball that kept getting bigger and bigger, it hit me. Maybe it was resolving IPv6 (after all, it’s on by default in just about every IP stack out there today).
So, what’s the flag to manipulate it? Is it “EnableIPv6“? Nah – too easy. It’s “AddressFamily“, which is set to “any” by default. When I set it to “inet”, my ssh session connected almost instantaneously.
You can also run ssh with the “-4” option on the fly; e.g. “ssh -4 some_host
Setting: on client side, in /etc/ssh/ssh_config: AddressFamily inet
No restarts required.

To wrap this up, let me remind you that “ssh -vvvv” is a very helpful troubleshooting tool. Also, never stop just because the common answers don’t fix your problem. Keep digging til you get it sorted out.

Getting systemd services to wait for networking in Raspbian

For isc-dhcp-server:

I tried several things which did not work.  I concluded that this is primarily because isc-dhcp-server doesn’t use a true systemd script.  So I edited the sysV startup script (/etc/init.d/isc-dhcp-server) and inserted a loop that waits until the non-loopback IP is up.  The loop times out after 10 seconds.  At the beginning of the “start” section:

 _IP=$(hostname -I)
 retry_count=0
 while [ ! "$_IP" ] && [ $retry_count -lt 10 ]; do
   _IP=$(hostname -I)
   retry_count=$((retry_count+1))
   sleep 1
 done

This seems to work for DHCP.  It’s really a workaround, but it does the job.

Interestingly, nothing in /etc/rc3.d had a start priority greater than 4.  If you remember the old days, the really critical stuff was symlinked as “S00…” and poor rc.local, the last thing to run, was “S99rc.local”.  But now everything is S01 through S04.

The big takeaway for me is that everything runs so quickly that the DHCP server comes online before the IP is fully configured.  While I love quicker boot times, this part of systemd is frustrating.  And from all the posts I read, many others feel the same way.  Hooray progress!

For bind:

What did not work:

-I edited the systemd script and changed network.target to network-online.target for the Wants= and After= items.

-I also tried setting those to dhcpcd.service, to no avail.

-In /etc/dhcpcd.conf, under the section where I configure the static IP for eth0, I added “waitip 4” to force it to wait for the IP to come up before forking to the background.

 

A workaround that functioned:

In /etc/bind/named.conf.options I set interface-interval to 1 minute.  This is the option that tells named to re-scan for new network interfaces that fall within the limits of listen-on.  Setting it to 1 minute is fine, for the most part.  But frankly I want a solution that works right at boot (yeah, yeah, impatience).

What worked

I installed NetworkManager, which apparently adds “NetworkManager-wait-online.service” that actually works when you set “After=network-online.target”.

 

Disabling the crash kernel (and kdump) in CentOS/RHEL 6

Had a system where the user wanted to disable the crash kernel after a previous admin had enabled it.  I found a ton of write-ups showing how to enable and configure kdump, but few describing the opposite.

There are multiple approaches that work here, but for a quick win here’s what we did.

  1. Edit /boot/grub/grub.conf
  2. On the kernel line, find the text crashkernel= and set it to crashkernel=no
  3. Do chkconfig kdump off
  4. Reboot

If you don’t want to reboot, then you could do service kdump stop

Local Plex Media Server Will Not Allow Login

Went to open my Plex server (running on CentOS 7) and saw that there was only one user listed: me.  Seemed odd since I have several users on that installation.  Clicked the drop-down by my avatar and noticed I appeared to be signed out.  “No problem, I’ll just sign in.”
Except I couldn’t.  Upon clicking to login I was presented with a page stating:

“Plex is not reachable.
Make sure your server has an internet connection and any firewalls or other programs are set to allow access.”

After some brief internet searching, including the Plex forums, found the apparently tried-and-true solution of deleting two bundle directories.
However, these instructions were not accurate in my case.  They list the directory containing these bundles as being:

$PLEX_HOME/Library/Application Support/Plex Media Server/Plug-ins/

Where $PLEX_HOME is /var/lib/plexmediaserver.  In my case, I ended up finding the directories in:

/usr/lib/plexmediaserver/Resources/Plug-ins-4613ce077

So here’s what I did:

systemctl stop plexmediaserver
cd /usr/lib/plexmediaserver/Resources/Plug-ins-4613ce077
rm -rfv System.bundle
rm -rfv Framework.bundle
systemctl start plexmediaserver

And all was well again.

SSH sessions seem to freeze after period of inactivity

I was having an issue with my SSH sessions where, after a period of inactivity, they would freeze. They weren’t disconnecting or timing out; they were just frozen.

To solve this I made the following change. In /etc/ssh/ssh_config (works also for ~/.ssh/config for individual users), I added two lines to the “Host *” section:

ServerAliveInterval 15
ServerAliveCountMax 5

Without going into a lot of detail, you can know that those are somewhat arbitrary – you can adjust those to your needs.
My entire “Host *” section looks like this (this is a RHEL system):

Host *

GSSAPIAuthentication yes
ForwardX11Trusted yes
SendEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY   LC_MESSAGES
SendEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE   LC_MEASUREMENT
SendEnv LC_IDENTIFICATION LC_ALL LANGUAGE
SendEnv XMODIFIERS
ServerAliveInterval 15
ServerAliveCountMax 5

Also, if you want to help out with connections TO your server, a similar directive in the main body of /etc/ssh/sshd_config will work:

ClientAliveInterval 15
ClientAliveCountMax 5

Since this applies to the SSH server, don’t forget to restart the service with
service sshd restart
or
systemctl restart sshd.service

vmware tools install (or vmware-config-tools.pl) hangs after X config

I was updating vmware-tools on some old Linux installations of ours, using a tarball I downloaded from VMware. Once I unzipped the tarball, I ran the installer and told it to take the defaults:
./vmware-install.pl -d

Got to one machine in particular and it hung right after this text:
X is running fine with the new config file

Checked the processes and it seemed “mkinitrd --help” was holding things up. Killed it (just to see what happened) and another “mkinitrd” came up that wanted to build for the installed kernel. But that also hung. Killed the second “mkinitrd” and it warned me about not having a proper initrd and all the horrible things that might happen in life if I rebooted with this botched boot configuration.

So I ran vmware-config-tools.pl manually, then I ran strace on that process, and it occasionally did a call-out to an old AD/LDAP server of ours that is deprecated. What the heck? Also saw it read from /etc/ldap.conf as it did this.

So I killed the config script and moved /etc/ldap.conf to /etc/oldldap.conf. Re-ran the installer, and “whee” it all worked.

Didn’t feel like digging into it further, but I did get the tools installed and configured.

YMMV