Secure servers behind a private network
by Alex Arica

We are going to secure our servers by hiding them behind a private network. Among all servers inside the private network, only one server will be accessible from internet via SSH. And that server will be the "Bastion server", meaning it will allow admins to "SSH jump" to private servers.

The advantage of this approach is, it allows hiding the servers and services (e.g. HTTP Apache server, PostgreSql database, ...) inside a private network. From outside, meaning using a public IP address, it would not be possible to access to those services nor to scan them. This architecture significantly limits the attack surface by only exposing the Bastion server and the external load-balancer to the Internet.

The flow to access to a private server via SSH:

Admin on internet -> via SSH -> Bastion server -> SSH jump to -> private server(s)
                    

The flow to access to a HTTP service (e.g. Nginx) exposed by private server(s):

Client on internet -> via HTTPS -> External load-balancer -> HTTPS -> Nginx on private server(s)
                    

In the flow above, we assumed that the external load-balancer is accessible publicly and has access to the private network. Many hosting companies offer a private network service. For example, OVH offers a "vRack" private network which is accessible via their load-balancers.

Let's set-up this architecture

Let's assume that we have 2 servers: a private server and a public server. They have 2 network interfaces each, as follows:

  • interface eth0: public IP, assigned by the hosting company
  • interface eth1: private IP, manually assigned with range 192.168.0.0/16

At this stage, both servers are publicly accessible. We would like to make changes so that they have the following characteristics:

  • Private server: its network interface eth0 is disabled. It has no public IP and consequently it is not possible to directly access to this server via the internet. It has the network interface eth1 enabled with a static private IP address 192.168.0.2 and a gateway IP 192.168.0.1 .
  • Public server: it is a Bastion server and also a Gateway server. Both of its interfaces eth0 and eth1 are enabled. It has a public IP via eth0 and a static private IP 192.168.0.1 via eth1. As a Bastion server it allows admins to access the Private server (192.168.0.2) via a "SSH jump". As a Gateway server with IP 192.168.0.1, it allows the Private server to access to the internet.

Both servers are shipped with a SSHD service listening to port 22.

Configure the Public server as a Bastion server

Check the Bastion server has SSHD running and listening on port 22.

Configure the Public server as a Gateway server

We are going to assign the static IP address 192.168.0.1 to the public server. Later, this IP address will be used by the private server as the gateway IP address.

On Ubuntu

Create this file:

sudo vi /etc/netplan/60-static.yaml
                    

Add the followings:

network:
  version: 2
  ethernets:
    eth1:
       dhcp4: no
       addresses:
          - 192.168.0.1/16
                    

Save the file and apply the changes:

sudo netplan apply
                    

On Debian

Open this file:

sudo vi /etc/network/interfaces
                    

Add the followings:

auto eth1
iface eth1 inet static
address 192.168.0.1
netmask 255.255.0.0
                    

Save the file and apply the changes:

sudo systemctl restart networking
                    

Test the changes

Check Ip address

ip a
                    

We should see the interface eth1 with the IP 192.168.0.1

Reboot the server to make sure that the IP address remains the same:

sudo reboot
                    

After reboot, connect to the server and check the private IP address works:

ping 192.168.0.1
                    

Enable IP forwarding

Check if it is already enabled

sudo sysctl net.ipv4.ip_forward
                    

If the output is "0" then we need to enable it. Open the file:

sudo vi /etc/sysctl.conf
                    

Uncomment the following line to enable packet forwarding for IPv4:

net.ipv4.ip_forward=1
                    

Save the file and apply the changes:

sudo sysctl -p
                    

Check if it is enabled, the output should be "1":

sysctl net.ipv4.ip_forward
                    

Set-up a NAT rule

Set-up a NAT (network-address-translation) to give private network access to the internet.

If a private IP is statically set, please use SNAT (this is our case):

sudo iptables -t nat -A POSTROUTING ! -d 192.168.0.0/16 -o eth0 -j SNAT --to-source [replace this by the public IP address of eth0]
                    

For example:

sudo iptables -t nat -A POSTROUTING ! -d 192.168.0.0/16 -o eth0 -j SNAT --to-source 145.239.7.56
                    

Enable forwarding:

sudo iptables -A FORWARD -i eth1 -o eth0 -j ACCEPT
                    

Persist the new IP rules:

su -
iptables-save > /etc/iptables/rules.v4
                    

Configure the private server

We are going to configure the private server with a static IP address 192.168.0.2 .

On Ubuntu

Create this file:

sudo vi /etc/netplan/60-static.yaml
                    

Add the followings:

network:
  version: 2
  ethernets:
    eth1:
      dhcp4: no
      addresses:
      - 192.168.0.2/16
      gateway4: 192.168.0.1
                    

Save the file and apply the changes:

sudo netplan apply
                    

At this stage, on Ubuntu you would be disconnected from the server because the private IP comes before the public IP in IP routes. You can reconnect to the server by "SSH jump", as follows:

ssh -J alex@[Bastion public IP] alex@192.168.0.2
                    

On Debian

Open this file:

sudo vi /etc/network/interfaces
                    

Add the followings:

auto eth1
iface eth1 inet static
address 192.168.0.2
netmask 255.255.0.0
gateway 192.168.0.1
                    

Save the file and apply the changes:

sudo systemctl restart networking
                    

Test the changes

Check Ip address

ip a
                    

We should see the interface eth1 with the IP 192.168.0.2 .

Check that the gateway 192.168.0.1 is present in the routing table:

ip route
                    

We should see something similar to:

default via 192.168.0.1 dev eth1 ...
                    

If we do not see the gateway 192.168.0.1 in the routing table, then add it manually:

ip route add default via 192.168.0.1 dev eth1
                    

Connect to the private server via SSH

We are going to ssh jump from the Gateway server to the Private server. From a client computer (a laptop or desktop computer), connect via SSH as follows:

ssh -J [username]@[Bastion server's IP address] [username]@[private server IP]
                    

For example:

ssh -J alex@145.239.7.56 alex@192.168.0.2
                    

If the above works, the Bastion server is ready.

Disable the public interface eth0 on private server

Now that we can SSH jump to the Private server, it is time to isolate it from the internet. Let's disable the interface eth0 managing a public IP:

sudo ip link set eth0 down
                    

Test:

ip a
                    

The interface eth0 should not have an IP address. Only the interface eth1 should have one: 192.168.0.2

Check default gateway routing:

ip route
                    

We should not see the interface eth0 in the routing table.

Ping:

ping 192.168.0.1
ping reactive-tech.io
                    

The set-up is completed. To add more servers into the private network, follow the steps in section "Configure the private server" by incrementing the private IP address in 192.168.0.[to increment].

Additional actions and readings

On Bastion, configure a firewall to only allow SSH access

You can create firewall rules on Bastion server, using iptables in order to limit external connections to SSH port only. Please follow the instructions here.

Few notes about SNAT and MASQUERADE.

If a private IP is dynamically set by a DHCP, please use MASQUERADE rather than SNAT:

sudo iptables -t nat -A POSTROUTING ! -d 192.168.0.0/16 -o eth0 -j MASQUERADE
                    

For static IPs, SNAT is recommended by the iptables man page:

“This target is only valid in the nat table, in the POSTROUTING chain. It should only be used with dynamically assigned IP (dialup) connections: if you have a static IP address, you should use the SNAT target. Masquerading is equivalent to specifying a mapping to the IP address of the interface the packet is going out, but also has the effect that connections are forgotten when the interface goes down. This is the correct behavior when the next dialup is unlikely to have the same interface address (and hence any established connections are lost anyway).”

Additional discussions on the Web on this topic.