systemd job for SSH reverse tunnel

You can “SSH reverse tunnel” to a server to expose a local TCP port to the public internet.

This page describes a way to create a persistent SSH reverse tunnel.

The persistence uses systemd on the local machine. Systemd is Linux-specific and is part of all major Linux distros as of 2015 (RHEL 7, Ubuntu 15.04, Debian Jessie, etc).

Be aware that exposing local TCP ports to the Internet is against some corporate policies. Only do it on a corporate LAN if you think it's okay. Residential ISPs usually have terms of service that nominally ban servers, but don't actually mind unless you are regularly hogging the uplink bandwidth.

As an example purpose, you could expose SSH port 22 on your Raspberry Pi so you can connect to it from another city. To make sure strangers can't SSH into your computer: Before exposing your SSH port to the world, make sure to turn off SSH PasswordAuthentication on your Pi (edit its /etc/ssh/sshd_config and reload its sshd service). Use only RSA keys with 2048+ bits.

Local machine config (part 1)

Choose a Unix user and make sure they have an SSH key (If the user doesn't have a ~/.ssh/id_rsa, create one with ssh-keygen -t rsa -b 4096 -C "$(hostname)-tunnel". The value of -C is just a comment of your choice to help you keep track of which key is which on the server — you can change it.) You will need to copy their ~/.ssh/id_rsa.pub to the server.

Proxy server config

You have to have a server you can SSH to from both your local machine and elsewhere. (It can be Linux, BSD, etc.) Its firewall rules (if any) have to expose at least one otherwise-unused port, and its /etc/ssh/sshd_config has to say GatewayPorts clientspecified (or GatewayPorts yes) (and you have to reload the sshd service after changing this config).

You can ignore the firewall and sshd_config instructions if you only wanted to access your local port from that particular server's localhost. If so, you can delete the colon that immediately follows -R in order to limit it to the server's localhost.

Create or choose a Unix user on the server and add the local machine's designated ~/.ssh/id_rsa.pub as a line to the server user's ~/.ssh/authorized_keys (if it isn't there already). Make sure the ~/.ssh is chmod 700 and the ~/.ssh/authorized_keys is chmod 600 and that they're both owned (chown/chgrp) by the user whose home directory they are in. If they aren't, sshd will refuse to let anyone log in as that user.

Small virtual servers are fine and cost about $5/month (USD) as of 2015. Pick one in a data center near you, to reduce connection latency. (“Near” is not entirely geographical; for example New York City is so central on many networks that it can be faster for a Vermont resident to talk to a server in NYC than to somebody in the next town over.)

Using a friend's server is fine if they are okay with it. Your friend can allow you to create tunnels while preventing you from running commands by prefixing your key's .ssh/authorized_keys line with:

command="/bin/echo this authorized key is only for port forwarding"

followed by a space and then the usual ssh-rsa AAAA....... Unfortunately with reverse tunnels, your friend can't limit which port you open in this manner (permitopen= only works in conjunction with ssh -L), but your friend can add no-pty,no-X11-forwarding to the comma-separated authorized_keys line's prefix to restrict a couple more unnecessary features:

command="/bin/echo this authorized key is only for port forwarding",no-pty,no-X11-forwarding

Out of an excess of caution, I do these even when it is my own server, and I use a Unix user that's not used for anything else.

Local machine config (part 2)

Make sure the TCP port you are exposing is safe to expose to the internet (for example, see PasswordAuthentication commentary above).

Install autossh.

Put the below config in /etc/systemd/system/my-tunnel.service (or anything you want in place of “my-tunnel”). Replace all CONFIG_* with your own values.

[Unit]
Description=Creates a public reverse tunnel from a server to a port of this computer.
After=network.target

[Service]
Type=simple
User=CONFIG_LOCAL_SSH_CLIENT_USER
Environment="AUTOSSH_GATETIME=0"
ExecStart=/bin/sh -c 'exec autossh -M 0 -oStrictHostKeyChecking=no -oServerAliveInterval=60 -oServerAliveCountMax=3 -oExitOnForwardFailure=yes -N -R :CONFIG_REMOTE_PORT_TO_OPEN:localhost:CONFIG_LOCAL_PORT_TO_EXPOSE CONFIG_REMOTE_SERVER_USER@CONFIG_REMOTE_SERVER_IP'

[Install]
WantedBy=multi-user.target

How to use systemd: After editing a file in /etc/systemd/system/, run sudo systemctl daemon-reload so that systemd notices the changes; this does not yet start or restart any services. Start it with sudo systemctl start my-tunnel.service. Enable it so it starts when you boot your computer with sudo systemctl enable my-tunnel.service. Check its status with sudo systemctl status my-tunnel.service. You can reverse start with stop and enable with disable.

Pitfalls

I've seen at least one ISP that somehow prevents connections from working if they start from your home and then return back into your home from the public internet. This can mean your reverse tunnel might work anywhere but the network you set it up on.

I've found this script fairly reliable, but something's went wrong every few months, so:

• not good for use in production
• think about whether someone will be physically there to fix it if something goes wrong

If the proxy server is regenerated, you may have to change something:

• If its ssh server key is different from before, you'll have to delete a line from the ~/.ssh/known_hosts of the user used by the systemd job. This is because SSH is protecting you from an attacker pretending to be that server (which wouldn't be particularly devastating given that you're using strong pubkey authentication for everything, but still).
• If its IP is different and you are using the IP in the systemd file, you will have to change the systemd file (or create a new copy of the systemd service file, start and enable the new one, then stop and disable the old one. If you create the new server before deleting the old server, then you can do this remotely).
• If its IP is different but you are using the DNS name in the systemd file, you may have to restart the systemd job — I'm not sure whether autossh knows to look up DNS again periodically or on failure. (First, you may have to wait up to as long as the server's old DNS TTL.)

Design decision details

Some of this config is inspired by https://wiki.archlinux.org/index.php/Secure_Shell#Run_Autossh_automatically_at_boot_via_systemd

After=network.target is unreliable because there are several steps to getting a net connection — getting on the LAN, setting up DNS, etc. — but it won't hurt, and may help autossh be less confused. See http://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/. AUTOSSH_GATETIME=0 in theory should make this moot. However, even with After=network.target and AUTOSSH_GATETIME=0, I think I've seen occasional failure when using a DNS name for the proxy server so I err towards using the server's IP if it has a stable IP.

-oStrictHostKeyChecking=no means you don't have to figure out how to accept a new server's fingerprint when you set this script up. If the server's fingerprint changes, you'll still have to deal with ~/.ssh/known_hosts (you'll know what went wrong by checking sudo systemctl status name-of-this-service-file.service).

-oServerAliveInterval=60 -oServerAliveCountMax=3 makes sure that if the client-server connection stops working, autossh will notice within three minutes and be able to try restarting ssh. -M 0 disables autossh's inferior built-in method of doing this. The server will probably notice the disconnection too because its TCPKeepAlive defaults to yes, allowing the server to release the open listening port it had made for the defunct connection.

-oExitOnForwardFailure=yes makes autossh able to notice when the SSH connection worked but the forwarding failed, so that it can try again (for example to wait for the server to time-out an old stale port-forwarding connection).