Sometimes you just need to get to things remotely. Those things might be behind an unbudging firewall with no way to forward proper ports for remote access, or perhaps you just want it setup access to resources this way. I recently needed to establish a persistent connection to a new Raspberry Pi setup remotely where the ISP did not allow proper port forwards. Luckily you can use reverse SSH tunneling and AutoSSH (automated tunnel persistence) and carry on.
How it Works
SSH reverse proxies work very much like the following illustration:
- You connect to a remote host via SSH
- Specify the -R option to open up a reverse tunnel on a local port on the remote host
- From the remote host you can access the originating machine via the local port.
- Terminology Below:
- remoteserver = a remote system you cannot SSH to, but can initiate outbound SSH connections.
- homeserver = a system you control that can accept inbound SSH connections.
Example in Practice
You connect from remote.example.com to homeserver01 via SSH on port 9991
[user@remoteserver ~]# ssh homeserver01 -p 9991 -R 8081:localhost:9991
On remote machine homeserver01 you can now connect back to the original machine on port 8081 where you have bound a local port.
[user@homeserver01 ~]$ ssh localhost -p 8081
Let’s break down the options above:
- -p 9991 = We connect to SSH listening on the remote host (homeserver01) on TCP/9991
- -R 8081:localhost:9991 = We bind local port 8081 on the remote host, and again specify 9991 as our initial connecting port that where SSH listens.
Automating with AutoSSH
What happens if this disconnects after you leave? Well, you can no longer access the reverse tunnel you’ve setup. The good news is there’s a program that will set this up for you and keep it connected. You can also start it as a service or on reboot.
First, install autossh. Most distributions will have autossh packaged, if not you’ll need to build it from source but we won’t cover that here.
[root@remoteserver ~]# yum install autossh -y
AutoSSH can take a configuration file or you can pass it parameters when you run it. We’re going to use the runtime options for illustration here
Make AutoSSH Users
You’ll want a set of dedicated users and SSH public key for AutoSSH. On the originating host (where you’ll be initiating the tunnel from) you probably want to give it a false shell.
Create the originating host user
[root@remoteserver ~]# useradd -m -s /sbin/nologin autotunnel
Create an SSH key for the user
[root@remoteserver ~]# su - autotunnel -s /bin/bash [autotunnel@remoteserver ~]$ ssh-keygen -t rsa
You should not set a passphrase here so it can be automated, and go with the other defaults by hitting enter.
[autossh@remoteserver ~]$ ssh-keygen -t dsa
Generating public/private dsa key pair. Enter file in which to save the key (/home/autossh/.ssh/id_dsa): Enter passphrase (empty for no passphrase):
Note: If you’re accessing newer systems with OpenSSH 6.5 or higher (released early 2014) then it’s recommended to use ed25519. Thanks to previous U.S. President Barack Obama in the comments below for this feedback.
ssh-keygen -t ed25519
Now create a user on the destination server, this time you’ll need to set a valid shell along with a password if you want to remotely copy the SSH public key.
[root@homeserver01 ~]# useradd -m -s /bin/bash autotunnel [root@homeserver01 ~]# autosshpass=`date | md5sum | cut -c1-10` && echo "$autosshpass" \ | passwd autotunnel --stdin && echo "password is $autosshpass"
Remember the password echoed to the terminal, in this case mine was d53a922e9e.
You’ll need this in the next step.
Note: Ubuntu users have reported that the –stdin option isn’t present on some systems, you can do this instead in that case:
echo -e “$autosshpass\n$autosshpass” | passwd autotunnel
Copy SSH Keys
Back to the remote server (machine you want to establish the reverse connection to) you’ll want to copy the newly created key over using the password you generated.
[autotunnel@remoteserver ~]$ ssh-copy-id homeserver01 -p 9991
Enter the password and you should get a nice message back saying keys are successfully copied.
Test the Connection
At this point you’re ready to test the connection. Run the following AutoSSH command which should establish your tunnel. You can use the -vvv flags for extra verbosity to troubleshoot if it doesn’t work.
[root@remoteserver ~]# su - autossh -s /bin/bash [autotunnel@remoteserver ~]$ autossh -M 0 homeserver01 -p 9991 -N -R 8081:localhost:9991 -vvv
You should see something like the following.
OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 56: Applying options for * debug2: ssh_connect: needpriv 0 debug1: Connecting to homeserver01 [x.x.x.x] port 9991. debug1: Connection established.
At this point you can now test the port on the remote host for an SSH connection back via the reverse proxy.
[user@homeserver01 ~]$ telnet localhost 8081
Connected to localhost. Escape character is '^]'. SSH-2.0-OpenSSH_6.6.1
Start on Boot – Cron Method
Great! If you got this far you’re setup, now let’s make this start on boot and persist with a cron job. Alternatively you can create a Systemd service if you’re using a distribution that supports it.
Add the following to root’s crontab, adjusting as needed
[root@remoteserver ~]# crontab -e
Add the below lines adjusting to your setup, save and quit.
@reboot /bin/sudo -u autossh bash -c '/usr/local/bin/autossh -M 0 -f autossh@homeserver01 -p 9991 -N -o "ExitOnForwardFailure=yes" -o "ServerAliveInterval 60" -o "ServerAliveCountMax 3" -R 8081:localhost:9991'
Systemd Service (Preferred)
If you’re using Systemd you can create a unit file for this. Note the binary paths below, they are needed for Systemd unit files when running as non-root users.
Below, the port on the destination host that pins up the other end of the tunnel is 8081/localhost and the SSH listening port on that host where the remote tunnel connects to is 9991.
cat > /etc/systemd/system/autossh-homeserver01.service << EOF [Unit] Description=Keep a tunnel to 'homeserver01' open After=network-online.target [Service] Type=forking User=autotunnel ExecStart=/usr/bin/autossh -f -M 0 -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=false -p 9991 -N autossh@homeserver01 -R 8081:127.0.0.1:22 ExecStop=/usr/bin/pkill -9 -u autotunnel Restart=always [Install] WantedBy=multi-user.target EOF
Now enable the service.
[root@remoteserver ~]# systemctl enable autossh-homeserver01.service [root@remoteserver ~]# systemctl start autossh-homeserver01.service
Accessing the Remote AutoSSH Tunnel from the Destination
The remote tunnel is good for all ssh traffic, including other users on the system.
To access your remote tunnel simply ssh to the local port on the destination system it binds to. If I was accessing it as another user for example I might pass the private key.
ssh -p 8081 -i ~/.ssh/id_ed25519 -o ForwardAgent=yes -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=false myusername@localhost
Closing Old, Stale Tunnels
Sometimes, infrequently tunnels become disconnected and leave stale connections on the host. The following script will solve this for you by detecting any orphaned SSH connections without actual corresponding tunnels. You will put this in place on the remoteserver as it’s the one that pins up the autossh reverse tunnel and maintains it.
Copy this to the user account of the system that initiates your tunnel service.
If your local autossh account username is different than autotunnel then edit the script.
sed -i 's/autotunnel/mytunnelusername/g' monitor-autossh-tunnel.sh
Place it in the crontab of your autotunnel user via crontab -e
*/5 * * * * /usr/local/bin/tunnel-mon.sh 1>/dev/null 2>&1
That’s it! Not only will systemd restart the service if it fails but you’ll also have old, orphaned connections cleaned up.
Extending Reverse SSH Tunnels
Reverse tunnels over SSH are useful for more than just SSH connections, you can also use them to access otherwise local-only listening ports. Another common example might be MySQL (TCP/3306).
I am in no way endorsing using this for anything nefarious like getting around a corporate VPN or firewall. This is simply a guide of how to use secure, reverse tunnelling where connectivity would normally not be possible.
Thank you for a great article, your advice will be very helpful for me.
Thanks a bunch for very clear step-wise instructions. Everything worked as expected:-)
Amazing, this is so valuable to me! Thank you!
Thanks Kyle, I’m really glad it was useful to you.
btw you need a uppercase m to create a user with no home dir
Hey Joey, I do indeed want to create a homedir so we have a place to keep ssh keys.
This was SUPER helpful. In the “Make AutoSSH Users” section above, my /usr/bin/passwd in Ubuntu 17.10 (Artful Aardvark) seems to be missing the –stdin option for some reason. The line below can be used instead of the –stdin portion for creating the autossh password described above:
echo -e “$autosshpass\n$autosshpass” | passwd autossh
– I used the -t rsa to generate rsa keys. The dsa keys are probably fine but seem to be deprecated in openssh
– I had to add ‘AllowUsers autossh’ to /etc/ssh/sshd_config (then ‘service sshd restart’) on the ssh server to copy keys over
The next step is setting up a systemd service so the Raspberry Pi 3 device will wake up, connect to a mobile hotspot, then open the remote ssh port so login is avaialble over the cellular network. Aside from a custom written tcp tunnel, this is the only way I know how to get login access to a device over a mobile cellular network.
Mosh would be better for a cellular mobile network connection but the ssh remote port forwarding is the only way to get access to an unknown ip address and also through blocked inbound udp ports on the remote device.
Thank you, this was really helpful.
Hey Marc, thanks for the tip rsa vs. dsa as well as the lack of –stdin for Ubuntu (this is present on CentOS/Fedora for me). For ‘AllowUsers’ that’s usually never set in stock configurations that I’ve seen, being an optional setting that someone usually locks down afterwards. I’m glad that you found the guide useful and I appreciate the feedback.
After trying to get this to work for Ubuntu 18 I discovered:
1.When using systemd you must omit the -f from the ExecStart, otherwise when autossh drops into the background systemd thinks the process quit and it just keeps trying to call it again, preventing autossh from functioning.
2. After=network.target should be changed to After=network-online.target so that this only attempts to run if its connected to a network (important for wifi-connected devices).
Hey @cbclement, thanks for the feedback – I’ve adjusted those systemd unit lines.
I find adding this command really helps! ExitOnForwardFailure=yes
Thanks for the suggestion, I’ve added it to the systemd and @reboot cron examples. This is a great improvement for connection awareness.
Bro, DSA keys? Really? Use ed25519
There’s nothing really wrong with RSA, sure ed25519 is newer/better but it’s not supported until OpenSSH 6.5 (released early 2014) which may not be in place on older systems, especially older devices/appliances which seem to ship ancient SSH versions. For this reason I cited usage of RSA for compatibility purposes.
Thanks for the feedback Mr. President – I’ve updated the blog post to add ed25519 keys if you’re using newer systems/devices/OpenSSH.
Love this entry!
For the paranoi^Wcautious, I’d recommend something more like this for generating the password:
autosshpass=`dd if=/dev/urandom count=1 status=none | sha512sum | cut -c1-20` && echo "$autosshpass" | passwd autossh --stdin && echo "password is $autosshpass"
Hi David, I hope you don’t mind I edited your syntax slightly. This is a better option for more entropy from /dev/urandom for sure, great suggestion.
Nice guide. But it seems to be a mix between two trials .. sometimes the username is autotunnel, sometimes it’s autossh (which seems like a bad option, given the command is autossh as well)
also for on RPi buster, I could not get your systemd example working. I then took the command line from your cron example and plugged it in -> works
Thanks for the feedback, I’ll go back through and check those places the instructions might be inconsistent.
Thanks for your great instruction!
I made my Raspberry Pi OS buster work with your systemd service file only after I added ‘-N’ in ExecStart command line, just like the ‘-N’ in your cronjob script that you’ve provided.
Thanks Joe, I’ve adjusted that system unit file to include that also.
The script that cleans the orphaned tunnels — is this run on the client (the one initiating the reverse SSH) or the server?
Hi Y, this would be placed on the remoteserver (the system that initiates the autossh command) as it is the one that pins up and maintains the tunnel persistence. I’ve updated the guide to be a little more specific about that part.