We are going to secure our servers by hiding them behind a private network. Among all servers inside the private network, only one server will be accessible from internet via SSH. And that server will be the "Bastion server", meaning it will allow admins to "SSH jump" to private servers.
The advantage of this approach is, it allows hiding the servers and services (e.g. HTTP Apache server, PostgreSql database, ...) inside a private network. From outside, meaning using a public IP address, it would not be possible to access to those services nor to scan them. This architecture significantly limits the attack surface by only exposing the Bastion server and the external load-balancer to the Internet.
The flow to access to a private server via SSH:
Admin on internet -> via SSH -> Bastion server -> SSH jump to -> private server(s)
The flow to access to a HTTP service (e.g. Nginx) exposed by private server(s):
Client on internet -> via HTTPS -> External load-balancer -> HTTPS -> Nginx on private server(s)
In the flow above, we assumed that the external load-balancer is accessible publicly and has access to the private network. Many hosting companies offer a private network service. For example, OVH offers a "vRack" private network which is accessible via their load-balancers.
Let's set-up this architecture
Let's assume that we have 2 servers: a private server and a public server. They have 2 network interfaces each, as follows:
- interface eth0: public IP, assigned by the hosting company
- interface eth1: private IP, manually assigned with range 192.168.0.0/16
At this stage, both servers are publicly accessible. We would like to make changes so that they have the following characteristics:
- Private server: its network interface eth0 is disabled. It has no public IP and consequently it is not possible to directly access to this server via the internet. It has the network interface eth1 enabled with a static private IP address 192.168.0.2 and a gateway IP 192.168.0.1 .
- Public server: it is a Bastion server and also a Gateway server. Both of its interfaces eth0 and eth1 are enabled. It has a public IP via eth0 and a static private IP 192.168.0.1 via eth1. As a Bastion server it allows admins to access the Private server (192.168.0.2) via a "SSH jump". As a Gateway server with IP 192.168.0.1, it allows the Private server to access to the internet.
Both servers are shipped with a SSHD service listening to port 22.
Configure the Public server as a Bastion server
Check the Bastion server has SSHD running and listening on port 22. And configure bastion to only accept SSH connections as explained here.
Configure the Public server as a Gateway server
We are going to assign the static IP address 192.168.0.1 to the public server. Later, this IP address will be used by the private server as the gateway IP address.
On Ubuntu
Create this file:
sudo vi /etc/netplan/60-static.yaml
Add the followings:
network:
version: 2
ethernets:
eth1:
dhcp4: false
dhcp6: false
addresses:
- 192.168.0.1/16
Save the file and apply the changes:
sudo netplan apply
On Debian
Open this file:
sudo vi /etc/network/interfaces
Add the followings:
auto eth1
iface eth1 inet static
address 192.168.0.1
netmask 255.255.0.0
Save the file and apply the changes:
sudo systemctl restart networking
Test the changes
Check Ip address
ip a
We should see the interface eth1 with the IP 192.168.0.1
Reboot the server to make sure that the IP address remains the same:
sudo reboot
After reboot, connect to the server and check the private IP address works:
ping 192.168.0.1
Enable IP forwarding
Check if it is already enabled
sudo sysctl net.ipv4.ip_forward
If the output is "0" then we need to enable it. Open the file:
sudo vi /etc/sysctl.conf
Uncomment the following line to enable packet forwarding for IPv4:
net.ipv4.ip_forward=1
Save the file and apply the changes:
sudo sysctl -p
Check if it is enabled, the output should be "1":
sudo sysctl net.ipv4.ip_forward
Set-up a NAT rule
Set-up a NAT (network-address-translation) to give private network access to the internet.
If a private IP is statically set, please use SNAT (this is our case):
sudo iptables -t nat -A POSTROUTING ! -d 192.168.0.0/16 -o eth0 -j SNAT --to-source [replace this by the public IP address of eth0]
For example:
sudo iptables -t nat -A POSTROUTING ! -d 192.168.0.0/16 -o eth0 -j SNAT --to-source 145.239.7.56
Enable forwarding:
sudo iptables -A FORWARD -i eth1 -o eth0 -j ACCEPT
sudo iptables -A FORWARD -i eth0 -o eth1 -m state --state ESTABLISHED,RELATED -j ACCEPT
Persist the new IP rules:
su -
apt-get install iptables-persistent
iptables-save > /etc/iptables/rules.v4
After the steps above, it is highly recommended creating firewall rules on Bastion server, using iptables, in order to limit the external connections to SSH port only, as explained here.
Once you applied the firewalls rules for the NAT rules and the Bastion server rules for ssh, the file /etc/iptables/rules.v4 should contain:
# Generated by iptables-save on Fri May 20 20:35:38 2022
*filter
:INPUT ACCEPT [2130:253524]
:FORWARD ACCEPT [210:21141]
:OUTPUT ACCEPT [4469:572757]
-A INPUT -i eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -i eth0 -j DROP
-A FORWARD -i eth1 -o eth0 -j ACCEPT
-A FORWARD -i eth0 -o eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i eth0 -j DROP
COMMIT
# Completed on Fri May 20 20:35:38 2022
# Generated by iptables-save v1.8.7 on Fri May 20 20:35:38 2022
*nat
:PREROUTING ACCEPT [4510:194220]
:INPUT ACCEPT [421:17683]
:OUTPUT ACCEPT [53:3894]
:POSTROUTING ACCEPT [0:0]
-A POSTROUTING ! -d 192.168.0.0/16 -o eth0 -j SNAT --to-source [bastion server's public IP]
COMMIT
# Completed on Fri May 20 20:35:38 2022
Configure the private server
We are going to configure the private server with a static IP address 192.168.0.2 .
On Ubuntu
Create this file:
sudo vi /etc/netplan/60-static.yaml
Add the followings:
network:
version: 2
ethernets:
eth1:
dhcp4: false
dhcp6: false
addresses:
- 192.168.0.2/16
routes:
- to: default
via: 192.168.0.1
nameservers:
addresses: [8.8.8.8]
Save the file and apply the changes:
sudo netplan apply
On Debian
Open this file:
sudo vi /etc/network/interfaces
Add the followings:
auto eth1
iface eth1 inet static
address 192.168.0.2
netmask 255.255.0.0
gateway 192.168.0.1
Save the file and apply the changes:
sudo systemctl restart networking
Connect to the private server via SSH
At this stage, on Ubuntu only you would be disconnected because in IP routes the private IP comes before the public IP.
We are going to ssh jump from the Gateway server to the Private server. From a client computer (a laptop or desktop computer), connect via SSH as follows:
ssh -J [username]@[Bastion server's IP address] [username]@[private server IP]
For example:
ssh -J alex@145.239.7.56 alex@192.168.0.2
If the above works, the Bastion server is ready.
Test the changes
Check Ip address
ip a
We should see the interface eth1 with the IP 192.168.0.2 .
Check that the gateway 192.168.0.1 is present in the routing table:
ip route
We should see something similar to:
default via 192.168.0.1 dev eth1 ...
If we do not see the gateway 192.168.0.1 in the routing table, then add it manually:
ip route add default via 192.168.0.1 dev eth1
Disable the public interface eth0 on private server
Now that we can SSH jump to the Private server, it is time to isolate it from the internet. Let's disable the interface eth0 managing a public IP:
sudo ip link set eth0 down
Test:
ip a
The interface eth0 should not have an IP address. Only the interface eth1 should have one: 192.168.0.2
Check default gateway routing:
ip route
We should not see the interface eth0 in the routing table.
Ping:
ping 192.168.0.1
ping reactive-tech.io
The set-up is completed. To add more servers into the private network, follow the steps in section "Configure the private server" by incrementing the private IP address in 192.168.0.[to increment].
Additional actions and readings
On Bastion, configure a firewall to only allow SSH access
Few notes about SNAT and MASQUERADE.
If a private IP is dynamically set by a DHCP, please use MASQUERADE rather than SNAT:
sudo iptables -t nat -A POSTROUTING ! -d 192.168.0.0/16 -o eth0 -j MASQUERADE
For static IPs, SNAT is recommended by the iptables man page:
“This target is only valid in the nat table, in the POSTROUTING chain. It should only be used with dynamically assigned IP (dialup) connections: if you have a static IP address, you should use the SNAT target. Masquerading is equivalent to specifying a mapping to the IP address of the interface the packet is going out, but also has the effect that connections are forgotten when the interface goes down. This is the correct behavior when the next dialup is unlikely to have the same interface address (and hence any established connections are lost anyway).”
