Sometimes you just need to get to things remotely.  Those things might be behind an unbudging firewall with no way to forward proper ports for remote access, or perhaps you just want it setup access to resources this way.  I recently needed to establish a persistent connection to a new Raspberry Pi setup remotely where the ISP did not allow proper port forwards.  Luckily you can use reverse SSH tunneling and AutoSSH (automated tunnel persistence) and carry on.

How it Works
SSH reverse proxies work very much like the following illustration:

• You connect to a remote host via SSH
• Specify the -R option to open up a reverse tunnel on a local port on the remote host
• From the remote host you can access the originating machine via the local port.
• Terminology Below:
  • remoteserver = a remote system you cannot SSH to, but can initiate outbound SSH connections.
  • homeserver = a system you control that can accept inbound SSH connections.

Example in Practice
You connect from remote.example.com to homeserver01 via SSH on port 9991

[user@remoteserver ~]# ssh homeserver01 -p 9991 -R 8081:localhost:9991

On remote machine homeserver01 you can now connect back to the original machine on port 8081 where you have bound a local port.

[user@homeserver01 ~]$  ssh localhost -p 8081

Let’s break down the options above:

• -p 9991 = We connect to SSH listening on the remote host (homeserver01) on TCP/9991
• -R 8081:localhost:9991 = We bind local port 8081 on the remote host, and again specify 9991 as our initial connecting port that where SSH listens.

Automating with AutoSSH
What happens if this disconnects after you leave?  Well, you can no longer access the reverse tunnel you’ve setup.  The good news is there’s a program that will set this up for you and keep it connected.  You can also start it as a service or on reboot.

First, install autossh.  Most distributions will have autossh packaged, if not you’ll need to build it from source but we won’t cover that here.

[root@remoteserver ~]# yum install autossh -y

AutoSSH can take a configuration file or you can pass it parameters when you run it.  We’re going to use the runtime options for illustration here

Make AutoSSH Users
You’ll want a set of dedicated users and SSH public key for AutoSSH.  On the originating host (where you’ll be initiating the tunnel from) you probably want to give it a false shell.

Create the originating host user

[root@remoteserver ~]# useradd -m -s /sbin/nologin autotunnel

Create an SSH key for the user

[root@remoteserver ~]# su - autotunnel -s /bin/bash
[autotunnel@remoteserver ~]$ ssh-keygen -t rsa

You should not set a passphrase here so it can be automated, and go with the other defaults by hitting enter.

[autossh@remoteserver ~]$ ssh-keygen -t dsa

Generating public/private dsa key pair.
Enter file in which to save the key (/home/autossh/.ssh/id_dsa): 
Enter passphrase (empty for no passphrase):

Note: If you’re accessing newer systems with OpenSSH 6.5 or higher (released early 2014) then it’s recommended to use ed25519.  Thanks to previous U.S. President Barack Obama in the comments below for this feedback.

ssh-keygen -t ed25519

Now create a user on the destination server, this time you’ll need to set a valid shell along with a password if you want to remotely copy the SSH public key.

[root@homeserver01 ~]# useradd -m -s /bin/bash autotunnel
[root@homeserver01 ~]# autosshpass=`date | md5sum | cut -c1-10` && echo "$autosshpass" \
| passwd autotunnel --stdin && echo "password is $autosshpass"

Remember the password echoed to the terminal, in this case mine was d53a922e9e.
You’ll need this in the next step.

Note: Ubuntu users have reported that the –stdin option isn’t present on some systems, you can do this instead in that case:

echo -e “$autosshpass\n$autosshpass” | passwd autotunnel

Copy SSH Keys
Back to the remote server (machine you want to establish the reverse connection to) you’ll want to copy the newly created key over using the password you generated.

[autotunnel@remoteserver ~]$ ssh-copy-id homeserver01 -p 9991

Enter the password and you should get a nice message back saying keys are successfully copied.

Test the Connection
At this point you’re ready to test the connection.  Run the following AutoSSH command which should establish your tunnel.  You can use the -vvv flags for extra verbosity to troubleshoot if it doesn’t work.

[root@remoteserver ~]# su - autossh -s /bin/bash
[autotunnel@remoteserver ~]$ autossh -M 0 homeserver01 -p 9991 -N -R 8081:localhost:9991 -vvv

You should see something like the following.

OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 56: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to homeserver01 [x.x.x.x] port 9991.
debug1: Connection established.

At this point you can now test the port on the remote host for an SSH connection back via the reverse proxy.

[user@homeserver01 ~]$ telnet localhost 8081

Connected to localhost.
Escape character is '^]'.
SSH-2.0-OpenSSH_6.6.1

Start on Boot – Cron Method
Great!  If you got this far you’re setup, now let’s make this start on boot and persist with a cron job.  Alternatively you can create a Systemd service if you’re using a distribution that supports it.

Add the following to root’s crontab, adjusting as needed

[root@remoteserver ~]# crontab -e

Add the below lines adjusting to your setup, save and quit.

@reboot /bin/sudo -u autossh bash -c '/usr/local/bin/autossh -M 0 -f autossh@homeserver01 -p 9991 -N -o "ExitOnForwardFailure=yes" -o "ServerAliveInterval 60" -o "ServerAliveCountMax 3" -R 8081:localhost:9991'

Systemd Service (Preferred)
If you’re using Systemd you can create a unit file for this.  Note the binary paths below, they are needed for Systemd unit files when running as non-root users.

Below, the port on the destination host that pins up the other end of the tunnel is 8081/localhost and the SSH listening port on that host where the remote tunnel connects to is 9991.

cat > /etc/systemd/system/autossh-homeserver01.service << EOF

[Unit] 
Description=Keep a tunnel to 'homeserver01' open 
After=network-online.target


[Service]
Type=forking
User=autotunnel
ExecStart=/usr/bin/autossh -f -M 0 -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=false -p 9991 -N autossh@homeserver01 -R 8081:127.0.0.1:22
ExecStop=/usr/bin/pkill -9 -u autotunnel
Restart=always

[Install]
WantedBy=multi-user.target
EOF

Now enable the service.

[root@remoteserver ~]# systemctl enable autossh-homeserver01.service
[root@remoteserver ~]# systemctl start autossh-homeserver01.service

Accessing the Remote AutoSSH Tunnel from the Destination
The remote tunnel is good for all ssh traffic, including other users on the system.
To access your remote tunnel simply ssh to the local port on the destination system it binds to.  If I was accessing it as another user for example I might pass the private key.

ssh -p 8081 -i ~/.ssh/id_ed25519 -o ForwardAgent=yes -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=false myusername@localhost

Closing Old, Stale Tunnels
Sometimes, infrequently tunnels become disconnected and leave stale connections on the host.  The following script will solve this for you by detecting any orphaned SSH connections without actual corresponding tunnels.  You will put this in place on the remoteserver as it’s the one that pins up the autossh reverse tunnel and maintains it.

Copy this to the user account of the system that initiates your tunnel service.

wget https://raw.githubusercontent.com/sadsfae/misc-scripts/master/shell/monitor-autossh-tunnel.sh

If your local autossh account username is different than autotunnel then edit the script.

sed -i 's/autotunnel/mytunnelusername/g' monitor-autossh-tunnel.sh

Place it in the crontab of your autotunnel user via crontab -e

*/5 * * * * /usr/local/bin/tunnel-mon.sh 1>/dev/null 2>&1

That’s it!  Not only will systemd restart the service if it fails but you’ll also have old, orphaned connections cleaned up.

Extending Reverse SSH Tunnels
Reverse tunnels over SSH are useful for more than just SSH connections, you can also use them to access otherwise local-only listening ports.  Another common example might be MySQL (TCP/3306).

Disclaimer
I am in no way endorsing using this for anything nefarious like getting around a corporate VPN or firewall.  This is simply a guide of how to use secure, reverse tunnelling where connectivity would normally not be possible.