Bruno Schaatsbergen website Mastodon PGP Key email A drawing of an astronaut in space The Netherlands

Talking Terraform to Private Kubernetes Clusters

in
blog
date
4/28/2024

Setting up a private Kubernetes cluster is a common practice for organizations that require additional security and control over their infrastructure. However, managing resources in a private Kubernetes cluster can be challenging, especially when using Terraform to provision and manage resources from outside the network that the cluster resides in. This happens when you’re using Terraform Cloud or GitHub Actions to manage your infrastructure, for example.

To proxy from your local machine to a private Kubernetes cluster, you can use a SOCKS5 proxy server. The SOCKS5 proxy protocol is a tunneling protocol that allows you to forward TCP traffic to a destination. This protocol is supported by the Kubernetes and Helm Terraform providers, which the providers use to forward requests to your private Kubernetes cluster.

We’ll use ssh to create a SOCKS5 proxy server, and then configure the Kubernetes Terraform provider to point to the proxy server.

The examples below use Google Cloud, but the same principle applies to other cloud providers, we begin by creating a proxy server.

Create a SOCKS5 proxy server

Using Terraform you can spin up a VM and execute the following cloud-init script during the VM creation process:

#cloud-config

runcmd:
  - [ sed, -i, 's/PermitTunnel no/PermitTunnel yes/g', /etc/ssh/sshd_config ]
  - [ systemctl, restart, sshd ]

This enables SSH tunneling by modifying the SSH daemon configuration and then restarting the SSH service to apply the changes.

If you’re looking for reference material on how to create a SOCKS5 proxy server in Google Cloud, see Proxying your way into GKE.

Create a secure tunnel to the proxy server

To create a secure tunnel to the proxy server in Google Cloud, you can use the gcloud CLI:

CLOUDSDK_PYTHON_SITEPACKAGES=1 gcloud compute ssh <instance-name> \
  --project=<project-name> \
  --zone=<instance-zone> \
  --tunnel-through-iap \
  --ssh-flag="-N -f -D 8888" \

This command creates a secure tunnel to the proxy server on port 8888. The --tunnel-through-iap flag is used to create a secure tunnel through Identity-Aware Proxy (IAP), which is a Google Cloud service that allows you to connect to your VM instances without exposing them to the internet, and the --ssh-flag flag is used to pass additional SSH flags to the gcloud compute ssh command. The flags -N -f -D 8888 are used to create a tunnel without executing a remote command, run in the background, and create a dynamic SOCKS5 proxy on port 8888.

If you want to kill the tunnel, you can use the following command:

kill -9 $(shell lsof 8888 > /dev/null 2> /dev/null || :

To test the tunnel, you can use the following command:

HTTPS_PROXY=socks5://127.0.0.1:8888 kubectl cluster-info

Configure the Kubernetes Terraform provider

To configure the Kubernetes Terraform provider to use the SOCKS5 proxy server, you can set the proxy_url argument in the provider block:

provider "kubernetes" {
  host = "https://${google_container_cluster.example.endpoint}"
  # the `proxy_url` references the locally secured tunnel created by Identity-Aware Proxy.
  proxy_url              = "socks5://127.0.0.1:8888"
  cluster_ca_certificate = base64decode(google_container_cluster.example.master_auth.0.cluster_ca_certificate)

  # we use this for authenticating natively with GKE, rather than relying on tokens and certificates.
  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "gke-gcloud-auth-plugin"
  }
}

Again, ignore the Google Cloud specific parts, the proxy_url argument is the key part here. It references the locally secured tunnel created by Google Identity-Aware Proxy. Configuring the Helm Terraform provider is similar.

/talking-terraform-to-private-kubernetes-clusters