May 3, 2020

How connect securely from Cloud Build to VMs using Identity-Aware Proxy

Sometimes, as part of the build process it is necessary to connect to a compute instance in order to perform different tasks like copying files to the destination VM or run a script to update a database. In order to allow Cloud Build instance to access the VM you need to configure the firewall and expose the required ports to the world like port 22 for instance, but this setup makes your server vulnerable to brute force attacks, even when you setup your sshd to disable password authentication. The ideal solution would be to allow the same functionality without exposing these ports to the world.
By using Google Cloud IAP (Identity-Aware Proxy) it is possible and, in this post, I will show you how to do it.

What is Cloud Build

Cloud Build is a service that executes your builds on Google Cloud Platform infrastructure. Cloud Build can import source code from Google Cloud Storage, Cloud Source Repositories, GitHub, or Bitbucket, execute a build to your specifications, and produce artifacts such as Docker containers or Java archives. You can find out more from the official documentation Cloud Build.

What is IAP (Identity-Aware Proxy)

Identity-Aware Proxy (IAP) lets you establish a central authorization layer for applications accessed by HTTPS, so you can use an application-level access control model instead of relying on network-level firewalls.

IAP’s TCP forwarding feature lets you control who can access administrative services like SSH and RDP on your backends from the public internet. The TCP forwarding feature prevents these services from being openly exposed to the internet. Instead, requests to your services must pass authentication and authorization checks before they get to their target resource. You can find out more from the official documentation Identity-Aware Proxy.

First demo

First, we will demonstrate how to use IAP tunnel to transfer files from Cloud Build container to a VM instance.

Prerequisites:

  • GCP project with billing enabled. If you do not have one, then sign-in to Google Cloud Platform Console and create a new project
  • Access to a standard internet browser

First sign-in to your GCP Console and activate the Cloud Shell.

Enable the required API

# enable IAP
gcloud services enable iap.googleapis.com

Setup the environment variables:

# define the variables
INSTANCE_NAME=iap-demo
ZONE=europe-west3-c

Run the following command to create a new compute instance

# create vm
gcloud compute instances create $INSTANCE_NAME-vm \
    --machine-type f1-micro \
    --tags http-server \
    --zone $ZONE

To allow IAP to connect to your VM instances, create a firewall rule that:

  • applies to all VM instances that you want to be accessible by using IAP.
  • allows ingress traffic from the IP range 35.235.240.0/20. This range contains all IP addresses that IAP uses for TCP forwarding.
  • allows connections to all ports that you want to be accessible by using IAP TCP forwarding, for example, port 22 for SSH.
# add firewall rule
gcloud compute firewall-rules create allow-ssh-ingress-from-iap \
  --direction=INGRESS \
  --action=allow \
  --rules=tcp:22 \
  --source-ranges=35.235.240.0/20

The default-allow-ssh default rule allow SSH connections from all IP addresses, not only from IAP. The best is to disable this rule if you want to prevent direct SSH to your VM instances.

# disable default firewall rule for ssh
gcloud compute firewall-rules update default-allow-ssh --disabled

Now you can ssh into your instance using the IAP tunnel

# ssh into vm
gcloud compute ssh $INSTANCE_NAME-vm --tunnel-through-iap --zone $ZONE

and install nginx, we will use this later

# install nginx, we will use it later
sudo apt-get install nginx
# make the nginx folder writeable
sudo chmod a+w /var/www/html
# exit
exit

If the instance does not have a public IP address, the connection automatically uses IAP TCP tunneling. If the instance does have a public IP address, the connection uses the public IP address instead of IAP TCP tunneling.

You can use the --tunnel-through-iap flag so that gcloud compute ssh always uses IAP TCP tunneling.

Add a firewall rule to allow http access to your nginx instance

# create the firewall rule for http
gcloud compute firewall-rules create default-allow-http \
    --direction=INGRESS \
    --action=ALLOW \
    --rules=tcp:80 \
    --source-ranges=0.0.0.0/0 \
    --target-tags=http-server

and test if you have access to your instance

# get the instance public ip
INSTANCE_ADDRESS=$(gcloud compute instances describe $INSTANCE_NAME-vm \
  --format='get(networkInterfaces[0].accessConfigs[0].natIP)' \
  --zone $ZONE)
# test the access
curl -I http://$INSTANCE_ADDRESS

the output should be like this:

HTTP/1.1 200 OK
Server: nginx/1.14.2

Next, we will use Cloud Build to transfer files to this instance using the gcloud cloud builder. Cloud builders are container images with common languages and tools installed in them. You can configure Cloud Build to run a specific command within the context of these builders. You can find out more from the official documentation Cloud builders.

Cloud Build executes your builds using a service account, a special Google account that executes builds on your behalf. The email for the Cloud Build service account is [PROJECT_NUMBER] @cloudbuild.gserviceaccount.com.

You can view your project’s service accounts via the IAM menu of the Cloud Console or you can run the following script to save it as variable

# get the cloud build service account
PROJECT_ID=$(gcloud config get-value project)
PROJECT_NUMBER=$(gcloud projects list --filter="$PROJECT_ID" --format="value(PROJECT_NUMBER)")
SERVICE_ACCOUNT="$PROJECT_NUMBER@cloudbuild.gserviceaccount.com"
echo $SERVICE_ACCOUNT

Cloud Build service account needs some permissions to be able to connect to the vm and to transfer files.

The Compute Instance Admin (v1) and Service Account User role are required in order for the service account to be able to transfer the ssh keys to the vm instance.

# add permissions to instance
gcloud compute instances add-iam-policy-binding $INSTANCE_NAME-vm \
    --zone=$ZONE \
    --member="serviceAccount:$SERVICE_ACCOUNT" \
    --role="roles/compute.instanceAdmin.v1"

gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member="serviceAccount:$SERVICE_ACCOUNT" \
    --role="roles/compute.viewer"

gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member="serviceAccount:$SERVICE_ACCOUNT" \
    --role="roles/iam.serviceAccountUser"

Compute Instance Admin gives this service account full control of the compute engine instance. The best will be to add a custom role with only the required permissions to perform this action.

IAP-secured Tunnel User roles is also required in order to grant the cloud build service account permission to use IAP§

# add permissions
gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member="serviceAccount:$SERVICE_ACCOUNT" \
    --role="roles/iap.tunnelResourceAccessor"

We will grant this permission at the project level for this demo, but you can also setup IAP tunnel permissions for each vm individually. Follow the instructions from the documentation Using IAP for TCP forwarding

Now let’s create a simple html file and name it test.html

cat <<'EOF' >> test.html
<!DOCTYPE html>
<html>
<head>
<title>Cloud build test</title>
</head>
<body>
<h1>Welcome</h1>
</body>
</html>
EOF

and create the cloudbuild.yaml

cat <<'EOF' >> cloudbuild.yaml
steps:
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
  entrypoint: /bin/sh
  args:
  - '-c'
  - |
    # copy files to destination
    gcloud compute scp --zone ${_ZONE} test.html cloudbuild@${_INSTANCE_NAME}-vm:/var/www/html --tunnel-through-iap
substitutions:
  _ZONE: europe-west3-c # default value
  _INSTANCE_NAME: iap-demo-vm # default value
EOF

You can find out all about cloud build configuration files from the official documentation Cloud Build configuration.

Now let’s create the build

#trigger the build
gcloud builds submit --config cloudbuild.yaml \
    --substitutions=_ZONE="$ZONE",_INSTANCE_NAME="$INSTANCE_NAME" .

You should receive STATUS=SUCCESS somewhere at the end of the output result.

To test if the file was uploaded successfully to the vm run the following script

# get the instance public ip
INSTANCE_ADDRESS=$(gcloud compute instances describe $INSTANCE_NAME-vm \
  --format='get(networkInterfaces[0].accessConfigs[0].natIP)' \
  --zone $ZONE)
# test the access
curl -I http://$INSTANCE_ADDRESS/test.html

the output should be similar to this

HTTP/1.1 200 OK
Server: nginx/1.14.2

Now you can setup any build to connect to any compute instance without exposing the ssh port to the whole world.

Second demo

You can use IAP TCP forwarding for other TCP-based protocols by using gcloud to allocate a local port. The local port tunnels data traffic from the local machine to the remote machine in an HTTPS stream. IAP then receives the data, applies access controls, and forwards the unwrapped data to the remote port. Conversely, any data from the remote port is also wrapped before it’s sent to the local port where it’s then unwrapped.

Connect to the instance and create a simple echo server

# connect to the vm
gcloud compute ssh $INSTANCE_NAME-vm --tunnel-through-iap --zone $ZONE

download and run the echo server script

# download the source file
curl https://gist.githubusercontent.com/gabihodoroaga/a64ed25b69939b54a38a30d36affe2c9/raw \
    --output echo_server.py
# start the echo server
python echo_server.py

on another separate Cloud Shell terminal window

# define the variables
INSTANCE_NAME=iap-demo
ZONE=europe-west3-c

create the firewall rule to allow access from IAP out echo server

# add firewall rule
gcloud compute firewall-rules create allow-echo-ingress-from-iap \
  --direction=INGRESS \
  --action=allow \
  --rules=tcp:5555 \
  --source-ranges=35.235.240.0/20

create a new file and name it cloudbuild-echo.yaml

cat <<'EOF' >> cloudbuild-echo.yaml
steps:
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
  entrypoint: /bin/sh
  args:
  - '-c'
  - |
    gcloud compute start-iap-tunnel ${_INSTANCE_NAME}-vm 5555 \
      --local-host-port=localhost:5555 \
      --zone=${_ZONE} & sleep 5 && python echo_client.py
substitutions:
  _ZONE: europe-west3-c # default value
  _INSTANCE_NAME: iap-demo-vm # default value
EOF

The command in the above step is creating a tcp tunnel from localhost:5555 tp vm on port 5555 waits 5 seconds, this is required to allow the connection to be verified, then executes the script, in our case is python echo_client.py and at the end kills the tcp tunnel.

Next, we need to pull the required echo_client.py script from gist

# get the echo_client.py file
curl https://gist.githubusercontent.com/gabihodoroaga/358af2df0d68d9c555182b688f9563f6/raw \
    --output echo_client.py

and trigger the cloud build task

#trigger the build
gcloud builds submit --config cloudbuild-echo.yaml \
    --substitutions=_ZONE="$ZONE",_INSTANCE_NAME="$INSTANCE_NAME" .

You should receive STATUS=SUCCESS somewhere at the end of the output result and on the other terminal window, the one where the echo sever is running, you should see the echo message.

That’s it. Now you can use IAP TCP forwarding capabilities to connect securely to any backend compute instance, mysql or postgresql and run any script without exposing the server ports to the world.

Clean up

To remove all the resources, you can either delete the project or download and run the clean-up script.

# download the file
curl https://gist.githubusercontent.com/gabihodoroaga/86ed83a591657b3e44b648f8d5924a48/raw \
    --output cleanup.sh
# adjust the values for INSTANCE_NAME and ZONE variables
# make the script executable
chmod +x cleanup.sh
# run the script
./cleanup.sh

Conclusion

By using Identity-Aware Proxy (IAP) TCP forwarding feature you can apply Zero-trust security policy and in the same time to be able to connect to services like ssh, RDP or database.