Dec 14, 2020

Simple SMTP server to forward your emails

If you have many ideas and for each idea you have a domain, like I do, and for each of this domains you want to receive emails, just in case someone wants to rich you, soon you will be in the situation to pay for each of this domains a monthly fee and probably for nothing. As I have all my websites hosted on GCP in containers I thought that it would be cheaper, free actually, to host my own SMTP server to receive all the emails for all the domains, forward them to my gmail account and also to save a copy to Cloud Storage, just to have them there. In this post I will show how this can be done.

TL;DR

Check out this github.com/gabihodoroaga/smtpd-email-forward repository if you want to get started to create your own smtp server.

Prerequisites

golang installed
GCP project with billing enabled. If you don’t have one then sign-in to Google Cloud Platform Console and create a new project

The goals

Let’s start by setting up our goals:

a smtp server to receive the email
a GCP bucket to store the emails
forward the email to another mailbox using Mailgun

The project setup

Create the folder and initialize the golang modules

mkdir smtp-email-forward
cd smtp-email-forward
go mod init github.com/gabihodoroaga/smtpd-email-forward

The SMTP server

For this I used this package github.com/mhale/smtpd. Add smtpd/server.go file with the following content

package smtpd

import (
	"context"
	"net"
	"strings"

	"github.com/gabihodoroaga/smtpd-email-forward/config"
	"github.com/gabihodoroaga/smtpd-email-forward/logger"
	"github.com/gabihodoroaga/smtpd-email-forward/storage"
	"github.com/gabihodoroaga/smtpd-email-forward/forwarder"
	"github.com/mhale/smtpd"
)

// StartServer initialize and start the smtp server on
func StartServer() {
	addr := "0.0.0.0:2525"
	logger.Log.Infof("Start listening on: %s", addr)
	err := listenAndServeTLS(
		addr,
		config.Config.StartTLSCert,
		config.Config.StartTLSKey,
		mailHandler,
		rcptHandler,
		config.Config.AppName,
		config.Config.Hostname)
	if err != nil {
		panic(err)
	}
}

func mailHandler(origin net.Addr, from string, to []string, data []byte) {
	logger.Log.Infof("Received mail from %s for %s", from, strings.Join(to, ";"))
	// save the email to GCP bucket
	if config.Config.GCPBucket != "" {
		ctx := context.Background()
		for _, m := range to {
			filename, err := storage.UploadFileToBucket(ctx, m, data)
			if err != nil {
				logger.Log.Errorf("Error saving file %s to bucket: %s", filename, err)
			} else {
				logger.Log.Infof("Save mail from %s for %s to file %s", from, to, filename)
			}
		}
	}
	// forward the emails
	err := forwarder.ForwardEmail(data)

	if err != nil {
		logger.Log.Errorf("Errors forward email:%s", err)
	}
}

func rcptHandler(remoteAddr net.Addr, from string, to string) bool {
	domain := getDomain(to)
	if domain == "" {
		return false
	}
	for _, b := range config.Config.Domains {
		if b == domain {
			return true
		}
	}
	return false
}

func getDomain(email string) string {
	at := strings.LastIndex(email, "@")
	if at >= 0 {
		return email[at+1:]
	}
	return ""
}

func listenAndServeTLS(addr string, certFile string, keyFile string, handler smtpd.Handler, rcpt smtpd.HandlerRcpt, appname string, hostname string) error {
	srv := &smtpd.Server{
		Addr:        addr,
		Handler:     handler,
		HandlerRcpt: rcpt,
		Appname:     appname,
		Hostname:    hostname,
		TLSRequired: false}
	err := srv.ConfigureTLS(certFile, keyFile)
	if err != nil {
		return err
	}
	return srv.ListenAndServe()
}

What this code does

setup the smtp server and start listening on port 2525: StartServer()
setup our recipient handler to make sure we accept email for the configured domains: rcptHandler()
setup the email handler, the function where we save the emails to a GCP Bucket and we forward de emails: mailHandler()

For logging we will use Uber’s “blazing fast, structured, leveled logging”. Add a new file to the project logger/logger.go

package logger

import (
		"go.uber.org/zap"
)

var Log *zap.SugaredLogger

func InitLogger() {
	logger, err := zap.NewProduction() // or NewProduction, or NewDevelopment
	if err != nil {
		panic("Cannot create the logger")
	}
	defer logger.Sync()
	Log = logger.Sugar()
}

Next let’s add the implementation of the storage.UploadFileToBucket() function, add the file storage/storage.go with the following content

package storage

import (
	"context"
	"fmt"
	"strconv"
	"sync"
	"time"

	"cloud.google.com/go/storage"
	"github.com/gabihodoroaga/smtpd-email-forward/config"
	"google.golang.org/api/option"
)

type storageConnection struct {
	Client *storage.Client
}

var (
	client *storageConnection
	once   sync.Once
)

// UploadFileToBucket uploads the email message to GCP bucket, one folder for each mailbox
func UploadFileToBucket(ctx context.Context, mailbox string, data []byte) (string, error) {
	filename := generateFileNameForGCS(ctx, "email")
	filepath := fmt.Sprintf("%s/%s.eml", mailbox, filename)
	client, err := getGCSClient(ctx)
	if err != nil {
		return "", err
	}
	wc := client.Bucket(config.Config.GCPBucket).Object(filepath).NewWriter(ctx)
	if _, err = wc.Write(data); err != nil {
		return "", err
	}
	if err := wc.Close(); err != nil {
		return "", err
	}
	return filepath, nil
}

func generateFileNameForGCS(ctx context.Context, name string) string {
	time := time.Now().UnixNano()
	var strArr []string
	strArr = append(strArr, name)
	strArr = append(strArr, strconv.Itoa(int(time)))
	var filename string
	for _, str := range strArr {
		filename = filename + str
	}
	return filename
}

func getGCSClient(ctx context.Context) (*storage.Client, error) {
	var clientErr error
	once.Do(func() {
		storageClient, err := storage.NewClient(ctx, option.WithCredentialsFile(config.Config.GCPCredentials))
		if err != nil {
			clientErr = fmt.Errorf("Failed to create GCS client ERROR:%s", err.Error())
		} else {
			client = &storageConnection{
				Client: storageClient,
			}
		}
	})
	return client.Client, clientErr
}

, and the implementation of forwarder.ForwardEmail() function, add the file forwarder/service.go

package forwarder

import (
	"errors"
	"strings"

	"github.com/gabihodoroaga/smtpd-email-forward/config"
)

var forwarders []mailForwarder

type mailForwarder interface {
	ForwardEmail(data []byte, recipient string) error
}

// InitForwarders initialize all the email forwarders
func InitForwarders() {
	initMailGun()
}

// ForwardEmail forwards the email to all the configured forwarders
func ForwardEmail(data []byte) error {
	var errstrings []string
	for _, f := range forwarders {
		if err := f.ForwardEmail(data, config.Config.ForwardTo); err != nil {
			errstrings = append(errstrings, err.Error())
		}
	}
	if len(errstrings) > 0 {
		return errors.New(strings.Join(errstrings, "\n"))
	}
	return nil
}

This is the abstract implementation of the forwarder, for the concrete implementation we will use mailgun. Add this file forwarder/mailgun.go

package forwarder

import (
	"bytes"
	"context"
	"io/ioutil"
	"time"

	"github.com/gabihodoroaga/smtpd-email-forward/config"
	"github.com/gabihodoroaga/smtpd-email-forward/logger"
	"github.com/mailgun/mailgun-go/v4"
)

type mailgunForwarder struct {
	Domain string
	APIKey string
	URL string
}

func initMailGun() {
	if config.Config.MailgunDomain != "" && config.Config.MailgunAPIKey != "" {
		logger.Log.Info("Init Mailgun forwarder")
		forwarders = append(forwarders, &mailgunForwarder{
			Domain: config.Config.MailgunDomain,
			APIKey: config.Config.MailgunAPIKey,
			URL: config.Config.MailgunURL,
		})
	}
}

func (f mailgunForwarder) ForwardEmail(data []byte, recipient string) error {
	// create the client
	mg := mailgun.NewMailgun(f.Domain, f.APIKey)
	mg.SetAPIBase(f.URL)

	// create the message
	message := mg.NewMIMEMessage(ioutil.NopCloser(bytes.NewReader(data)), recipient)
	// create a context
	ctx, cancel := context.WithTimeout(context.Background(), time.Second*10)
	defer cancel()
	// Send the message with a 10 second timeout
	resp, id, err := mg.Send(ctx, message)

	if err != nil {
		return err
	}

	logger.Log.Infof("Email sent successfully to mailgun: ID: %s Resp: %s", id, resp)

	return nil
}

Now, we need to add the configuration package in order to dynamically configure our service. Add the file config/config.go

package config

import (
	"flag"
	"fmt"
	"os"

	"github.com/gabihodoroaga/smtpd-email-forward/logger"
	"github.com/ilyakaznacheev/cleanenv"
)

type args struct {
	ConfigPath string
}

// Configuration struct ...
type Configuration struct {
	AppName        string   `yaml:"appname" env:"APP_NAME"`
	Hostname       string   `yaml:"hostname" env:"HOST_NAME"`
	Domains        []string `yaml:"domains" env:"DOMAINS" env-separator:","`
	StartTLSCert   string   `yaml:"starttlscert" env:"START_TLS_CERT" env-default:"certs/server.crt"`
	StartTLSKey    string   `yaml:"starttlskey" env:"START_TLS_KEY" env-default:"certs/server.key"`
	GCPBucket      string   `yaml:"gcpbucket" env:"GCP_BUCKET"`
	GCPCredentials string   `yaml:"gcpcredentials" env:"GCP_CREDENTIALS" env-default:"certs/gcpServiceAccount.json"`
	ForwardTo      string   `yaml:"forwardto" env:"FORWARD_TO"`
	MailgunDomain  string   `yaml:"mailgundomain" env:"MAILGUN_DOMAIN"`
	MailgunAPIKey  string   `yaml:"mailgunapikey" env:"MAILGUN_API_KEY"`
	MailgunURL     string   `yaml:"mailgunurl" env:"MAILGUN_URL" env-default:"https://api.mailgun.net/v3"`
}

// Config is the application configuration
var Config Configuration

// InitConfig - Initialize the the application configuration
func InitConfig() {
	args := processArgs(&Config)
	// read configuration from the file and environment variables
	if err := cleanenv.ReadConfig(args.ConfigPath, &Config); err != nil {
		fmt.Println(err)
		os.Exit(2)
	}

	// TODO: print configuration values
	logger.Log.Info("Configuration loaded successfully")
}

// ProcessArgs processes and handles CLI arguments
func processArgs(cfg interface{}) args {
	var a args

	f := flag.NewFlagSet("Example server", 1)
	f.StringVar(&a.ConfigPath, "c", "config.yml", "Path to configuration file")

	fu := f.Usage
	f.Usage = func() {
		fu()
		envHelp, _ := cleanenv.GetDescription(cfg, nil)
		fmt.Fprintln(f.Output())
		fmt.Fprintln(f.Output(), envHelp)
	}

	f.Parse(os.Args[1:])
	return a
}

To read the configuration I used an excellent package from Ilya Kaznacheev, because it allows you to read the configuration form the file and from the environment variables combined.

To wrapped up we need to add also the cmd package. Add the file cmd/server.go

package main

import (
	"github.com/gabihodoroaga/smtpd-email-forward/config"
	"github.com/gabihodoroaga/smtpd-email-forward/logger"
	"github.com/gabihodoroaga/smtpd-email-forward/smtpd"
	"github.com/gabihodoroaga/smtpd-email-forward/forwarder"
)

func main() {
	logger.InitLogger()
	logger.Log.Info("smtpd server is starting...")
	config.InitConfig()
	forwarder.InitForwarders()
	smtpd.StartServer()
}

We are almost done, let’s see if we can build

cd cmd
go build

To actually run the server we need create all the external resources

The Mailgun account

navigate to https://signup.mailgun.com/new/signup and create a new account
and then go to Sending/Overview select API and write down the sandbox domain and the API Key
it might be that you must configure an authorized recipient if you used the free account

MAILGUN_DOMAIN=[your_mailgun_sandbox_domain]
MAILGUN_API_KEY=[your_mailgun_api_key]

The GCP bucket

GCP_BUCKET=smtpd-email-forward-123
gsutil mb -c nearline gs://$GCP_BUCKET

if you get the error “ServiceException: 409 Bucket smtpd-email-forward-123 already exists.” just change the name of the bucket, as I already use it when I tested the setup.

The GCP service account

# create the service account
gcloud iam service-accounts create smtpd-email-forward-123 \
    --description="A service account to write to the bucker" \
    --display-name="smtpd-email-forward-123"

# grab the email
GCP_SERVICE_ACCOUNT=$(gcloud iam service-accounts list --format="value(email)" --filter="displayName=smtpd-email-forward-123")

# give the service account the permission to create objects
gsutil iam ch serviceAccount:$GCP_SERVICE_ACCOUNT:objectCreator gs://$GCP_BUCKET

# create and save the service account key
mkdir certs
gcloud iam service-accounts keys create certs/gcpServiceAccount.json \
  --iam-account $GCP_SERVICE_ACCOUNT

The server certificates

openssl req -x509 -newkey rsa:4096 -keyout ./certs/server.key -out ./certs/server.crt -days 365 -nodes

The default configuration file cmd/config.yml

appname: gcp-smtpd
starttlscert: ../certs/server.crt
starttlskey: ../certs/server.key
gcpcredentials: ../certs/gcpServiceAccount.json
forwardto: claire@example.com
domains:
  - example.com

and start up the server

cd cmd
./cmd

to test your server run the following command on a separate terminal

telnet localhost 2525

HELO test.example.com
MAIL FROM:<bob@example.com>
RCPT TO:<alice@example.com>
DATA
From: "Bob Example" <bob@example.com>
To: Alice Example <alice@example.com>
Cc: theboss@example.com
Date: Tue, 15 Jan 2008 16:02:43 -0500
Subject: Test message

Hello Alice.
This is a test message with 5 header fields and 4 lines in the message body.
Your friend,
Bob
.
QUIT

Done. You have you own smtp server.

Deploy to GCP

You should be able to run this server locally without any issue, but in order to deploy to GCP there are a few more actions to do

Create the Dockerfile

FROM alpine:3.7

WORKDIR /app

ADD ./dist/smtpd /app/smtpd
ADD ./cmd/config.yml /app/config.yml

EXPOSE 2525/tcp

CMD ["/app/smtpd"]

Create the cloudbuild.yaml file

steps:
- name: golang:1.13
  entrypoint: /bin/sh
  args:
  - '-c'
  - |
    # build server
    env CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -ldflags="-s" -o dist/smtpd ./cmd    
- name: 'gcr.io/cloud-builders/docker'
  args: [ 'build', '-t', 'gcr.io/$PROJECT_ID/smtp-email-forward:${_VERSION_}', '.' ]
images:
- 'gcr.io/$PROJECT_ID/smtp-email-forward:${_VERSION_}'
substitutions:
  _VERSION_: 1.0.0

Trigger cloud build to push the image to the registry

gcloud builds submit --config cloudbuild.yaml .

Create the compute instance and set it the smtp server image

PROJECT_ID=$(gcloud config list --format 'value(core.project)')
DOCKER_IMAGE=gcr.io/$PROJECT_ID/smtp-email-forward
VM_NAME=smtp-email-forward-vm
ZONE=us-central1-a

gcloud compute instances create-with-container $VM_NAME \
    --zone $ZONE \
    --machine-type=e2-micro \
    --tags $VM_NAME-smtp \
    --container-image $DOCKER_IMAGE

Configure the firewall rules

gcloud compute firewall-rules create $VM_NAME-allow-smtp \
    --allow tcp:2525 \
    --source-ranges 0.0.0.0/0 \
    --target-tags $VM_NAME-http

Test the server

IP_ADDRESS=$(gcloud compute instances describe $VM_NAME \
    --zone=$ZONE \
    --format='get(networkInterfaces[0].accessConfigs[0].natIP)')

telnet $IP_ADDRESS 2525

If you setup the port 25 and is not working make sure that you ISP is not blocking the port 25, most of the residential ISP will block the outgoing port 25.

And the last step is to configure you MX record for your domain. Check this article on Wikipedia

In order to avoid unnecessary chargers you should delete all resources used in this tutorial

gcloud -q compute firewall-rules delete $VM_NAME-allow-smtp
gcloud -q compute instances delete $VM_NAME --zone $ZONE
gcloud -q container images delete $DOCKER_IMAGE:1.0.0
gcloud -q iam service-accounts delete $SQL_SERVICE_ACCOUNT

Conclusion

If you only want to be able to forward your email to another mailbox, you can use this project and configure it for unlimited number of domains and an unlimited number of email addresses and save some money. It will cost around 5$/month on GCP.

You can find all the full source code here github.com/gabihodoroaga/smtpd-email-forward