Over the last few months, I’ve been relocating all my development activites on a remote devserver. The benefits have been numerous:
- When roaming I only need a tiny bit of bandwidth for ssh, and I can do all the
docker pullfrom a machine with a stable and large bandwith
- It’s easier to backup
- It’s easier to collaborate (no need for annoying tunnel solutions to share my work)
- It uses less power on my machine, slighly increasing my laptop battery life
Sadly, many development tools do not accomodate very well yet for this use case, as they very often only expose services to localhost. If it largely makes sense in the general case, it’s a bit annoying when you develop remotely.
One of such tool is
minikube, which doesn’t provide a way to expose the
deployed service externally. I understand they really want to prevent people
from using it to serve production traffic.
In the present article, I’ll share a few notes on how a Wireguard VPN and a few Iptables rules will allow you to expose the minikube host to the client machine, using minikube’s kvm2 driver and libvirt
First step: Wireguard VPN
I’ve elected to use Wireguard to provide me with the tunnel to my devserver network, but it would likely work similarly with any VPN solution (OpenVPN, StrongSWAN’s IPSec, etc). Despite not being extremely well documented, Wireguard is fairly simple and straightforward.
I advise you to go the the Wireguard website to find more.
As I refuse to upgrade my macOS to the latest version (it breaks TotalSpace and I do not support the notarization requirement they introduced), I wasn’t able to use the app they ship via the App Store. So instead I’ve:
- Installed wireguard-tools via homebrew:
brew install wireguard-tools
- Installed macos-menubar-wireguard to control my tunnels
On my devserver, I used the distribution package:
apt-get install wireguard
On both machine, run the following commands to generate the key pairs:
# As root. Use /usr/local/etc if on mac mkdir -p /etc/wireguard/keys && cd /etc/wireguard/keys wg genkey | tee privatekey | wg pubkey > publickey chmod 600 privatekey publickey
Here’s my server
[Interface] PrivateKey = [the content of the devserver private key file] # Update the address and port to your liking Address = 172.27.0.254/32 ListenPort = 999 PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -A FORWARD -o %i -j ACCEPT; iptables -t nat -A POSTROUTING -s 172.27.0.0/16 -j MASQUERADE PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -D FORWARD -o %i -j ACCEPT; iptables -t nat -D POSTROUTING -s -s 172.27.0.0/16 -j MASQUERADE # The client config [Peer] PublicKey = [the content of the client public key file] # Update the client address liking, keep it in the same range as the server ip above AllowedIPs = 172.27.0.1/32
You can then use systemctl to enable it at boot time and start it
systemctl enable firstname.lastname@example.org systemctl start email@example.com
It’s worth noting that you’ll need to enable ip forwarding in sysctl at least for the tunneling device. You can enable it globally with
sudo sysctl net.ipv4.conf.all.forwarding=1
Here’s my client
[Interface] PrivateKey = [replace with the content of your client private key] # Update the address and dns addresses to your liking Address = 172.27.0.1/32 DNS = 126.96.36.199, 188.8.131.52 [Peer] PublicKey = [replace with the content of your devserver public key] # Update according to your setup Endpoint = devserver-hostname-or-public-ip:devserver-wireguard-listen-port # If you're client is behind a NAT, which is likely PersistentKeepalive = 25 # In this context, AllowedIPs is the list of addresses that will be routed # through this tunnel. You can use `0.0.0.0/0` to route all the client traffic # through your devserver. Or only a few networks, like the minikube network # (192.168.42.0/24) AllowedIPs = 0.0.0.0/0 # AllowedIPs = 192.168.42.0/24, 172.27.0.0/16, 172.17.0.0/16
To start it, you can use the wireguard menubar mentionned earlier, or the command line tools:
- Create the tunnel with:
wg-quick up wg0
- Kill it with:
wg-quick down wg0
From your local machine, you should be able to ping the IP you selected for the
devserver side of the tunnel, in my case
$ ping 172.27.0.254 PING 172.27.0.254 (172.27.0.254): 56 data bytes 64 bytes from 172.27.0.254: icmp_seq=0 ttl=64 time=96.572 ms 64 bytes from 172.27.0.254: icmp_seq=1 ttl=64 time=91.428 ms 64 bytes from 172.27.0.254: icmp_seq=2 ttl=64 time=88.648 ms 64 bytes from 172.27.0.254: icmp_seq=3 ttl=64 time=85.733 ms ^C --- 172.27.0.254 ping statistics --- 4 packets transmitted, 4 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 85.733/90.595/96.572/3.995 ms
You can also get some info about the tunnel status using the
$ sudo wg interface: wg0 public key: my_public_key private key: (hidden) listening port: 999 peer: peer_public_key endpoint: 37.165.x.x:18500 allowed ips: 172.27.0.1/32 latest handshake: 35 seconds ago transfer: 272.86 MiB received, 4.07 GiB sent
Here are a few things to check if the ping doesn’t succeed:
- Wireguard logs: do they show any error or warning
- Wireguard routing config: Is the devserver tunnel ip is in the range of the AllowedIPs in the client config ?
- Your firewall rules:
- Can you connect to wireguard listenport on the public IP of your devserver ?
- Does your firewall rules allow forwarded content from the tunnel interface and network ?
Second step: Minikube w/ kvm2
I’m using minikube with the kvm2 driver to have a local development cluster. My goal is to be able to reach my services NodePorts from my client machine.
To do that, I create a dedicated network in libvirt using vcsh, ask minikube to attach the kubernetes host to this network and add a few iptables rules to break the virtual machine isolation, allowing my client to reach it via the VPN tunnel.
Creating the network
The following xml file contains the configuration of the network:
<network connections='1'> <name>vpn-access</name> <forward mode='nat'/> <bridge name='virbr42' stp='on' delay='0'/> <mac address='52:54:00:2c:23:42'/> <!-- This is the IP of the router/bridge in the virtual network --> <ip address='192.168.42.254' netmask='255.255.255.0'> <dhcp> <!-- This network will only have the minikube machine so the dhcp range only contains one IP, this way we don't have to lookup the lease to find the kubernetes IP --> <range start='192.168.42.1' end='192.168.42.1'/> </dhcp> </ip> </network>
The network can be created with the following commands:
sudo virsh net-define /path/to/the/file.xml sudo virsh net-autostart vpn-access sudo virsh net-start vpn-access
We launch these commands as root because minikube use the
When creating minikube, the following options must be added in order for the minikube host to be created with libvirt/kvm and attached to our network:
minikube start \ --cpus=2 --memory='6g' --disk-size='20g' \ --driver=kvm2 --kvn-network=vpn-access
Routing and firewall
We need to allow forwarded traffic to the minikube created virtual machine. So we’ll have to prepend a rules into libvirt’s tables:
sudo iptables -I LIBVIRT_FWI -d 192.168.42.1 -j ACCEPT
It’s worth noting that this also works because in wireguard we’ve adding a masquerading rule for all traffic coming from the the vpn network (
iptables -t nat -A POSTROUTING -s 172.27.0.0/16 -j MASQUERADE)
Testing / Using
The virtual machine should now be accessible from the devserver and the client machine:
$> ping 192.168.42.1 PING 192.168.42.1 (192.168.42.1) 56(84) bytes of data. 64 bytes from 192.168.42.1: icmp_seq=1 ttl=64 time=0.265 ms [...]
You can now access your kubernetes services via their NodePorts. It is a good idea to specify the NodePorts in the service definition to prevent random allocation in the development environment. Still the NodePorts of a service can be retried with kubectl:
$> kubectl get svc -n traefik NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE traefik NodePort 10.98.35.11 <none> 80:32080/TCP,443:32443/TCP 3d18h $> curl http://192.168.42.1:32080/ [...]