Jul 28, 2022

HTTP, gRPC, and websocket on Google Cloud Run

When you run services on Google Cloud Run you don’t have the luxury of choosing your port or use multiple ports, and if your requirements are to expose multiple services HTTP, gRPC, and websocket you’re facing a big challenge which may force you choose other solutions and products.

It looks like it is possible to have all three protocols on the same port on Cloud Run with authentication.

TL;DR

You can find the complete implementation and the documentation on github.com/gabihodoroaga/http-grpc-websocket

The solution

The solution to make it work is to write your own HTTP/2 server (using h2c protocol) and instruct Cloud Run to send us all requests (even HTTP/1 ones over HTTP/2).

    addr := resolveAddress()

	listener, err := net.Listen("tcp", addr)
	if err != nil {
		logger.Sugar().Errorf("error listen tcp listen on  %s, %v", addr, err)
		os.Exit(1)
	}

	grpcServer := grpc.NewServer(appgrpc.ServerOptions()...)
	pb.RegisterEchoServer(grpcServer, appgrpc.NewServer())

	mixedHandler := newHTTPAndGRPCMux(r, grpcServer)
	http2Server := &http2.Server{}
	http1Server := &http.Server{Handler: h2c.NewHandler(mixedHandler, http2Server)}

	ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)

	go func(l net.Listener) {
		logger.Sugar().Infof("httpServer: listen on address %s", addr)
		if err := http1Server.Serve(l); err != nil && err != http.ErrServerClosed {
			logger.Sugar().Errorf("http1Server exit with error %v", err)
			os.Exit(1)
		}
	}(listener)

	<-ctx.Done()
	stop()

	fmt.Println("httpServer: shutting down gracefully...")
	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
	defer cancel()
	if err := http1Server.Shutdown(ctx); err != nil {
		logger.Sugar().Errorf("forced to shutdown, error %v", err)
		os.Exit(1)
	}
	fmt.Println("httpServer: server exiting...")

and the implementation for the mixed handler is

func newHTTPAndGRPCMux(httpHand http.Handler, grpcHandler http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		if r.ProtoMajor == 2 && strings.HasPrefix(r.Header.Get("content-type"), "application/grpc") {
			grpcHandler.ServeHTTP(w, r)
			return
		}
		httpHand.ServeHTTP(w, r)
	})
}

For the websocket all you have to do is to handle the http request and accept the connection

func ServeHTTP(w http.ResponseWriter, r *http.Request) {
	zap.L().Sugar().Debug("ws/server: received connect request")
	c, err := websocket.Accept(w, r, &websocket.AcceptOptions{})
	if err != nil {
		zap.L().Error("ws/server: error accept request", zap.Error(err))
		return
	}
	defer c.Close(websocket.StatusNormalClosure, "websocket server exit")
	handleRequest(r.Context(), c)
}

This post does not contain all the source code because is too big to show it here.

If you want to checkout the full source code and to deploy a working example in GCP follow these steps.

Clone the repository.

git clone https://github.com/gabihodoroaga/http-grpc-websocket.git
cd http-grpc-websocket

Set your variables.

PROJECT_ID=$(gcloud config list project --format='value(core.project)')
REGION=us-central1

Create a service account.

gcloud iam service-accounts create demo-grpc-sa \
  --display-name "Demo service account for gRPC on Cloud Run"

Create a new key for this service account.

gcloud iam service-accounts keys create key.json \
  --iam-account demo-grpc-sa@${PROJECT_ID}.iam.gserviceaccount.com

Build the image.

gcloud builds submit \
    --tag gcr.io/$PROJECT_ID/grpcwebapp \
    --project $PROJECT_ID .

Deploy the service.

gcloud run deploy grpcwebapp \
--image gcr.io/$PROJECT_ID/grpcwebapp \
--set-env-vars=AUTH_SERVICE_ACCOUNTS="demo-grpc-sa@${PROJECT_ID}.iam.gserviceaccount.com",AUTH_AUDIENCE=webapp \
--allow-unauthenticated \
--timeout=10m \
--project $PROJECT_ID \
--region $REGION

The timeout option here is important because for gRPC and websocket you need the connection to be always on. Keep in mind that for Cloud Run the maximum timeout is 60 minutes and you need to implement your clients to auto reconnect if the connection drops.

Get the public URL.

SERVICE_URL=$(gcloud run services describe grpcwebapp \
--platform managed --region $REGION --format 'value(status.url)')
echo $SERVICE_URL
SERVICE_HOST=$(echo "$SERVICE_URL" | awk -F/ '{print $3}')
echo $SERVICE_HOST

Test HTTPS.

curl -v $SERVICE_URL/ping

Test gRPC.

go run clients/grpc/main.go --server "$SERVICE_HOST:443" --key key.json --insecure=false

Test Websocket.

go run clients/ws/main.go --server "wss://$SERVICE_HOST/ws" --key key.json

Cleaning up

In order to remove all the resources created and avoid unnecessary charges, run the following commands:

gcloud run services delete grpcwebapp \
    --project $PROJECT_ID \
    --region $REGION

Thanks to / acknowledgments

This example was based on this post Serving gRPC+HTTP/2 from the same Cloud Run container written by Ahmet Alp Balkan (https://github.com/ahmetb).