May 15, 2021

Deploy custom Ingress on GKE

The GKE Ingress for HTTP(S) Load Balancing is the managed Ingress controller and implements Ingress resources as Google Cloud load balancers for HTTP(S) workloads in GKE.

This allows you to use the Google Cloud Load Balancing with Container-native load balancing and Cloud CDN, and more, to access the application deployed on GKE.

But, here comes the but, the default Ingress GCE that comes with GKE has some missing functionalities like Cloud CDN is missing multiple features in the BackendConfig as a result you cannot set the option “Use origin headers” or update the TTLs or disable the “Negative caching” option in Cloud CDN. You cannot update the values manually as they will be overwritten by the ingress.

If cannot wait until the official release and fix is available there are ways to install a custom version on your cluster and I will explain how.

Prerequisites

  • GCP project with billing enabled. If you don’t have one then sign-in to Google Cloud Platform Console and create a new project
  • Access to a standard internet browser

Solution

An overview of the solution:

  • get the source code from github.com
  • build a new docker image
  • disable the default GLBC plugin
  • deploy your version of the GLBC

WARNING: The following changes will disrupt traffic for your services. Do not run this unless you fully understand what you are doing and how this may impact your cluster’s state.

This is heavily inspired from docs/deploy/gke/gke-self-managed.sh and most of the commands are copied from this script.

All the commands can be executed from the Cloud Shell console.

Setup

Let’s define our variables first

ZONE=us-central1-a
CLUSTER_NAME=demo-gce-cluster
PROJECT_ID=`gcloud config list --format 'value(core.project)' 2>/dev/null`
REGISTRY="gcr.io/$PROJECT_ID"
GCP_USER=`gcloud config list --format 'value(core.account)' 2>/dev/null`

Clone the repository. For this demo I will use my branch

git clone --branch backendconfig-cdn-config \
    https://github.com/gabihodoroaga/ingress-gce.git

Build the custom image

export REGISTRY=gcr.io/$PROJECT_ID
export VERSION=v1-custom
IMAGE_NAME=$REGISTRY/ingress-gce-glbc-amd64:$VERSION
make -C ingress-gce only-push-glbc

Now we have our own version pushed to the registry, let’s continue

Download the files required to deploy the image to your cluster from this Github Gist


curl https://codeload.github.com/gist/a5644451e1d309ad40b1312eb2369fe6/zip/70d0ec9fd78834ba4662ac7ee1ca46584d3d026c \
    --output deploy-ingress-gce.zip
unzip deploy-ingress-gce.zip
mv a5644451e1d309ad40b1312eb2369fe6-70d0ec9fd78834ba4662ac7ee1ca46584d3d026c \
    deploy-ingress-gce
cd deploy-ingress-gce

First let’s create a cluster, a really small one, so we will not spend too much resources just for a demo

gcloud container clusters \
        create $CLUSTER_NAME \
        --zone $ZONE --machine-type "e2-medium" \
        --enable-ip-alias \
        --num-nodes=2

Interesting fact: The --enable-ip-alias which means “Creating a VPC-native cluster” without this option you cannot use NEG and you will get some weird errors in logs like “Error 400: Invalid value for field ‘resource.ipAddress’: ‘10.0.x.x’”

To get credentials for the local machine to connect to the cluster:

gcloud container clusters get-credentials $CLUSTER_NAME \
    --zone $ZONE

NETWORK_NAME=$(basename $(gcloud container clusters \
    describe $CLUSTER_NAME --project $PROJECT_ID --zone=$ZONE \
    --format='value(networkConfig.network)'))
SUBNETWORK_NAME=$(basename $(gcloud container clusters \
    describe $CLUSTER_NAME --project $PROJECT_ID \
    --zone=$ZONE --format='value(networkConfig.subnetwork)'))
NETWORK_TAGS=$(gcloud compute instances describe \
    $(kubectl get nodes -o jsonpath='{.items[0].metadata.name}') \
    --zone=$ZONE --format="value(tags.items[0])")

sed "s/\[PROJECT\]/$PROJECT_ID/" gce.conf.tpl | \
sed "s/\[NETWORK\]/$NETWORK_NAME/" | \
sed "s/\[SUBNETWORK\]/$SUBNETWORK_NAME/" | \
sed "s/\[CLUSTER_NAME\]/$CLUSTER_NAME/" | \
sed "s/\[NETWORK_TAGS\]/$NETWORK_TAGS/" | \
sed "s/\[ZONE\]/$ZONE/" > gce.conf

Grant permission to current GCP user to create new k8s ClusterRole’s.

kubectl create clusterrolebinding one-binding-to-rule-them-all \
    --clusterrole=cluster-admin --user=${GCP_USER}

Store the nodePort for default-http-backend

NODE_PORT=`kubectl get svc default-http-backend -n kube-system -o yaml \
    | grep "nodePort:" | cut -f2- -d:`
echo $NODE_PORT

Disable the default GLBC from your cluster

gcloud container clusters update ${CLUSTER_NAME} --zone=${ZONE} \
    --update-addons=HttpLoadBalancing=DISABLED

This will take a while as it requires restarting the master node.

Make sure the pod and the service for default-backend do not exists anymore

kubectl get svc -n kube-system | grep default-http-backend
kubectl get pod -n kube-system | grep default-backend

You should not get anything back from the above commands.

Create a new service account for glbc and give it a ClusterRole allowing it access to API objects it needs.

kubectl apply -f rbac.yaml

Generate the default-http-backend.yaml from the template

sed "s/\[NODE_PORT\]/$NODE_PORT/" default-http-backend.yaml.tpl \
    > default-http-backend.yaml

Recreate the default-backend

kubectl create -f default-http-backend.yaml

Create new GCP service account

gcloud iam service-accounts create glbc-service-account \
  --display-name "Service Account for GLBC"

Give the GCP service account the appropriate roles

gcloud projects add-iam-policy-binding ${PROJECT_ID} \
  --member serviceAccount:glbc-service-account@${PROJECT_ID}.iam.gserviceaccount.com \
  --role roles/compute.admin

Create a key for the GCP service account

gcloud iam service-accounts keys create key.json \
  --iam-account glbc-service-account@${PROJECT_ID}.iam.gserviceaccount.com

Store the the service account key as secret

kubectl create secret generic glbc-gcp-key \
    --from-file=key.json -n kube-system

Create the gce.conf ConfigMap. This config map is mounted as a volume in glbc.yaml

kubectl create configmap gce-config --from-file=gce.conf -n kube-system

Generate the glbc.yaml from the template

sed "s~\[IMAGE_URL\]~${IMAGE_NAME}~" glbc.yaml.tpl > glbc.yaml

Deploy the new GLBC controller

kubectl apply -f glbc.yaml -n kube-system

Done! You are now using your version of ingress-gce in your cluster.

Let’s test our new deployed ingress controller and see how it works.

Create a deployment for nginx, add the service, and the BackendConfig

kubectl apply -f app-deployment.yaml

Create the ingress

kubectl apply -f app-ingress.yaml

Creating the ingress takes a lot of time, so be patient.

When is done let’s get the ip of the load balancer

IP_ADDRESS=$(kubectl get ingress nginx-ingress \
             -o jsonpath='{.status.loadBalancer.ingress[0].ip}') \
             && echo $IP_ADDRESS

and test if everything is configured correctly

curl -v http://$IP_ADDRESS

If everything worked well you should be able to check your Cloud CDN configuration and see that the option selected for Cache mode is “Use origin headers”.

You can update the BackendConfig options in app-deployment.yaml.

If you want to see the complete list of the supported options run the following command.

kubectl describe crd backendconfigs.cloud.google.com

Cleaning up

Make sure you still have your variables setup. if not got to Setup.

Delete the test ingress, deployment, service, and all related resources

kubectl delete -f app-ingress.yaml
kubectl delete -f app-deployment.yaml

Delete the glbc deployment and all related resources

kubectl delete -f glbc.yaml
kubectl delete secret glbc-gcp-key -n kube-system
kubectl delete configmap gce-config -n kube-system
kubectl delete -f rbac.yaml
kubectl delete clusterrolebinding one-binding-to-rule-them-all
gcloud projects remove-iam-policy-binding ${PROJECT_ID} \
    --member serviceAccount:glbc-service-account@${PROJECT_ID}.iam.gserviceaccount.com \
    --role roles/compute.admin
gcloud iam service-accounts delete glbc-service-account@${PROJECT_ID}.iam.gserviceaccount.com
kubectl delete -f default-http-backend.yaml

Enable the default GLBC on the GKE master

gcloud container clusters update ${CLUSTER_NAME} --zone=${ZONE} \
    --update-addons=HttpLoadBalancing=ENABLED

Delete the cluster - WARNING!!!

gcloud container clusters delete CLUSTER_NAME

Delete the registry image

gcloud -q container images delete $IMAGE_NAME

Conclusion

Sometimes things in GCP are not perfect. This is it! But the fact that you can disable the default addons in GKE and replace them with your own custom version is very nice, and it allows you to benefit from the new development and bug fixes without waiting for months until the official releases are published. Although, you must assume the risks of more bugs and untested functionalities.