Nginx - Custom upstream module
Recently I was looking for a solution to configure a reverse proxy that supports NTLM authentication passthrough. I tried nginx because is one of the most widely used web server across all “active” sites and can also be used as a reverse proxy, load balancer, mail proxy and HTTP cache, but the ntlm option for the upstream module is only available with a commercial subscription, since the commercial subscription was not an option for development environment I thought why not to try to create a similar custom module. And I did nginx-ntlm-module but this post is about how to get started if you want to create your own nginx module.
TL;DR
Check out this github.com/gabihodoroaga/nginx-upstream-module repository if you want to get started with a custom nginx upstream module.
Prerequisites
- comfortable with C, and not just “C-syntax”; you should know your way around a struct and not be scared off by pointers and function references
- basic understanding of HTTP
- you should be familiar with Nginx’s configuration file
The module definition
Modules are a great way to extend the default functionality of nginx. In order to add an new module you need static compile the module into nginx using the -add-module=/path/to/module argument to the configure script.
Each standalone nginx module resides in a separate directory that contains at least two files: a config file and the module source code file.
mkdir nginx-upstream-module
The config file contains all information needed for nginx to integrate the module.
So, let’s add the nginx-upstream-module/config file with the following content:
ngx_addon_name=ngx_http_upstream_custom_module
if test -n "$ngx_module_link"; then
ngx_module_type=HTTP
ngx_module_name=ngx_http_upstream_custom_module
ngx_module_srcs="$ngx_addon_dir/ngx_http_upstream_custom_module.c"
. auto/module
else
HTTP_MODULES="$HTTP_MODULES ngx_http_upstream_custom_module"
NGX_ADDON_SRCS="$NGX_ADDON_SRCS $ngx_addon_dir/ngx_http_upstream_custom_module.c"
fi
You can find the complete list of options for the config file from the official nginx development guide.
Next add the source code file nginx-upstream-module/ngx_http_upstream_custom_module.c
First let’s add all the dependencies:
#include "ngx_conf_file.h"
#include <ngx_config.h>
#include <ngx_core.h>
#include <ngx_http.h>
and then define the set of functions that we will use:
static ngx_int_t
ngx_http_upstream_init_custom_peer(ngx_http_request_t *r,
ngx_http_upstream_srv_conf_t *us);
static ngx_int_t
ngx_http_upstream_get_custom_peer(ngx_peer_connection_t *pc,
void *data);
static void
ngx_http_upstream_free_custom_peer(ngx_peer_connection_t *pc,
void *data, ngx_uint_t state);
static void *ngx_http_upstream_custom_create_conf(ngx_conf_t *cf);
static char *ngx_http_upstream_custom(ngx_conf_t *cf, ngx_command_t *cmd,
void *conf);
There will be detailed information about each of this function and how there are used in the sections below.
Next, let’s define configuration structs
typedef struct {
ngx_uint_t max;
ngx_http_upstream_init_pt original_init_upstream;
ngx_http_upstream_init_peer_pt original_init_peer;
} ngx_http_upstream_custom_srv_conf_t;
typedef struct {
ngx_http_upstream_custom_srv_conf_t *conf;
ngx_http_upstream_t *upstream;
void *data;
ngx_connection_t *client_connection;
ngx_event_get_peer_pt original_get_peer;
ngx_event_free_peer_pt original_free_peer;
} ngx_http_upstream_custom_peer_data_t;
Now, the next 3 sections are required in order to properly setup your module.
The module directives are defined as a static array of ngx_command_t.
/* The module directives */
static ngx_command_t ngx_http_upstream_custom_commands[] = {
{ngx_string("custom"), NGX_HTTP_UPS_CONF | NGX_CONF_NOARGS | NGX_CONF_TAKE1,
ngx_http_upstream_custom, NGX_HTTP_SRV_CONF_OFFSET, 0, NULL},
ngx_null_command /* command termination */
};
and this is how it will look on you nginx configuration file:
upstream http_backend {
server 127.0.0.1:8080;
custom 12;
}
The module context is defined as a static ngx_http_module_t struct, which just has a bunch of function references for creating the three configurations and merging them together.
/* The module context */
static ngx_http_module_t ngx_http_upstream_custom_ctx = {
NULL, /* preconfiguration */
NULL, /* postconfiguration */
NULL, /* create main configuration */
NULL, /* init main configuration */
ngx_http_upstream_custom_create_conf, /* create server configuration */
NULL, /* merge server configuration */
NULL, /* create location configuration */
NULL /* merge location configuration */
};
And the last is the actual module definition:
/* The module definition */
ngx_module_t ngx_http_upstream_custom_module = {
NGX_MODULE_V1,
&ngx_http_upstream_custom_ctx, /* module context */
ngx_http_upstream_custom_commands, /* module directives */
NGX_HTTP_MODULE, /* module type */
NULL, /* init master */
NULL, /* init module */
NULL, /* init process */
NULL, /* init thread */
NULL, /* exit thread */
NULL, /* exit process */
NULL, /* exit master */
NGX_MODULE_V1_PADDING};
Next we need to add the implementations of our functions:
The first one id ngx_http_upstream_custom_create_conf, and this function is called when server starts or when the configuration is reloaded. Also here we will setup the initial values of our configuration arguments.
static void *ngx_http_upstream_custom_create_conf(ngx_conf_t *cf) {
ngx_http_upstream_custom_srv_conf_t *conf;
conf = ngx_pcalloc(cf->pool, sizeof(ngx_http_upstream_custom_srv_conf_t));
if (conf == NULL) {
return NULL;
}
conf->max = NGX_CONF_UNSET_UINT;
return conf;
}
Next is ngx_http_upstream_custom and this is main entry point for the module. Nginx lets you hook right into its own mechanisms for dealing with back-end servers (called “upstreams”), so your module can talk to another server without getting in the way of other requests, and the way to do it is to set callbacks that will be invoked when the upstream server is ready to be written to and read from.
// The main entry point of the module
static char *ngx_http_upstream_custom(ngx_conf_t *cf, ngx_command_t *cmd,
void *conf) {
ngx_http_upstream_srv_conf_t *uscf;
ngx_http_upstream_custom_srv_conf_t *hccf = conf;
ngx_int_t n;
ngx_str_t *value;
ngx_log_debug0(NGX_LOG_DEBUG_HTTP, cf->log, 0, "custom init module");
/* read options */
if (cf->args->nelts == 2) {
value = cf->args->elts;
n = ngx_atoi(value[1].data, value[1].len);
if (n == NGX_ERROR || n == 0) {
ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
"invalid value \"%V\" in \"%V\" directive", &value[1],
&cmd->name);
return NGX_CONF_ERROR;
}
hccf->max = n;
}
uscf = ngx_http_conf_get_module_srv_conf(cf, ngx_http_upstream_module);
hccf->original_init_upstream = uscf->peer.init_upstream
? uscf->peer.init_upstream
: ngx_http_upstream_init_round_robin;
uscf->peer.init_upstream = ngx_http_upstream_init_custom;
return NGX_CONF_OK;
}
This function does 3 things:
- read the configuration value for our directive
- save a reference to original init_upstream function
- set the init_upstream function to point to our function instead; ngx_http_upstream_init_custom
Next, let’s write the implementation for our ngx_http_upstream_init_custom function:
static ngx_int_t ngx_http_upstream_init_custom(ngx_conf_t *cf,
ngx_http_upstream_srv_conf_t *us) {
ngx_http_upstream_custom_srv_conf_t *hccf;
ngx_log_debug0(NGX_LOG_DEBUG_HTTP, cf->log, 0, "custom init upstream");
hccf = ngx_http_conf_upstream_srv_conf(us, ngx_http_upstream_custom_module);
ngx_conf_init_uint_value(hccf->max, 100);
if (hccf->original_init_upstream(cf, us) != NGX_OK) {
return NGX_ERROR;
}
hccf->original_init_peer = us->peer.init;
us->peer.init = ngx_http_upstream_init_custom_peer;
return NGX_OK;
}
And this function does:
- get a reference of the configuration using ngx_http_conf_upstream_srv_conf
- sets the initial value for the directive argument (the max argument) if the value was not set ngx_conf_init_uint_value
- calls the original init_upstream function
- saves the reference to the original peer.init callback function
- and then, sets the peer.init to point to our function instead; ngx_http_upstream_init_custom_peer
Now, let’s write the implementation for our ngx_http_upstream_init_custom_peer function. This function is called when a new request is received by the server.
static ngx_int_t
ngx_http_upstream_init_custom_peer(ngx_http_request_t *r,
ngx_http_upstream_srv_conf_t *us) {
ngx_http_upstream_custom_srv_conf_t *hccf;
ngx_http_upstream_custom_peer_data_t *hcpd;
ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, "custom init peer");
hcpd = ngx_palloc(r->pool, sizeof(ngx_http_upstream_custom_peer_data_t));
if (hcpd == NULL) {
return NGX_ERROR;
}
hccf = ngx_http_conf_upstream_srv_conf(us, ngx_http_upstream_custom_module);
if (hccf->original_init_peer(r, us) != NGX_OK) {
return NGX_ERROR;
}
hcpd->conf = hccf;
hcpd->upstream = r->upstream;
hcpd->data = r->upstream->peer.data;
hcpd->client_connection = r->connection;
hcpd->original_get_peer = r->upstream->peer.get;
hcpd->original_free_peer = r->upstream->peer.free;
r->upstream->peer.data = hcpd;
r->upstream->peer.get = ngx_http_upstream_get_custom_peer;
r->upstream->peer.free = ngx_http_upstream_free_custom_peer;
return NGX_OK;
}
and to explain what this function does:
- get a reference to our configuration ngx_http_upstream_custom_srv_conf_t
- call the original_init_peer function
- collect references to all kind of information related to the request: r->upstream, r->upstream->peer.data, r->connection (this is the client connection information) into ngx_http_upstream_custom_peer_data_t. You will use this information to decide which upstream server will handle the request
- also save a reference to the original upstream->peer.get and upstream->peer.free callback functions from the request
- set r->upstream->peer.data to point to our peer data struct
- set new callbacks for r->upstream->peer.get and r->upstream->peer.free to point to our functions instead.
The implementation of ngx_http_upstream_get_ntlm_peer is:
static ngx_int_t ngx_http_upstream_get_custom_peer(ngx_peer_connection_t *pc,
void *data) {
ngx_http_upstream_custom_peer_data_t *hcdp = data;
ngx_int_t rc;
ngx_log_debug2(NGX_LOG_DEBUG_HTTP, pc->log, 0,
"custom get peer, try: %ui, conn: %p", pc->tries,
hcdp->client_connection);
rc = hcdp->original_get_peer(pc, hcdp->data);
if (rc != NGX_OK) {
return rc;
}
/* in this section you can set the upstream server connection */
return NGX_OK;
}
and the details are:
- get the reference of our ngx_http_upstream_custom_peer_data_t
- call the original_get_peer function
- and next, in this function, you have the chance to configure the upstream connection.
The best way to see how this function should be implemented is to check the implementations of the some of the nginx modules like ngx_http_upstream_keepalive_module.c or ngx_http_upstream_ip_hash_module.c from the nginx source code.
The last function that we have to implement is ngx_http_upstream_free_custom_peer. This function si called when the connection to the upstream server finishes and it is a mainly used to update statistics or to save the connection to be reused.
static void ngx_http_upstream_free_custom_peer(ngx_peer_connection_t *pc,
void *data, ngx_uint_t state) {
ngx_http_upstream_custom_peer_data_t *hcdp = data;
ngx_log_debug0(NGX_LOG_DEBUG_HTTP, pc->log, 0, "custom free peer");
hcdp->original_free_peer(pc, hcdp->data, state);
}
How to build
First let’s download all the required dependencies:
# download nginx
curl -OL http://nginx.org/download/nginx-1.19.3.tar.gz
tar -xvzf nginx-1.19.3.tar.gz && rm nginx-1.19.3.tar.gz
# download PCRE library
curl -OL https://ftp.pcre.org/pub/pcre/pcre-8.44.tar.gz
tar -xvzf pcre-8.44.tar.gz && rm pcre-8.44.tar.gz
# download OpenSSL
curl -OL https://www.openssl.org/source/openssl-1.1.1h.tar.gz
tar -xvzf openssl-1.1.1h.tar.gz && rm openssl-1.1.1h.tar.gz
# download zlib
curl -OL https://zlib.net/zlib-1.2.11.tar.gz
tar -xvzf zlib-1.2.11.tar.gz && rm zlib-1.2.11.tar.gz
And then run configure
cd nginx-1.19.3/
./configure --with-debug \
--prefix= \
--conf-path=conf/nginx.conf \
--pid-path=logs/nginx.pid \
--http-log-path=logs/access.log \
--error-log-path=logs/error.log \
--http-client-body-temp-path=temp/client_body_temp \
--http-proxy-temp-path=temp/proxy_temp \
--http-fastcgi-temp-path=temp/fastcgi_temp \
--http-scgi-temp-path=temp/scgi_temp \
--http-uwsgi-temp-path=temp/uwsgi_temp \
--with-pcre=../pcre-8.44 \
--with-zlib=../zlib-1.2.11 \
--with-http_v2_module \
--with-http_realip_module \
--with-http_addition_module \
--with-http_sub_module \
--with-http_dav_module \
--with-http_stub_status_module \
--with-http_flv_module \
--with-http_mp4_module \
--with-http_gunzip_module \
--with-http_gzip_static_module \
--with-http_auth_request_module \
--with-http_random_index_module \
--with-http_secure_link_module \
--with-http_slice_module \
--with-mail \
--with-stream \
--with-openssl=../openssl-1.1.1h \
--with-http_ssl_module \
--with-mail_ssl_module \
--with-stream_ssl_module \
--add-module=../nginx-upstream-module
, of course you can disable some of the module that you don’t need them.
At the end run make to build your version of nginx.
cd nginx-1.19.3/
make
and optionally setup your test environment
cd ..
# create the test folder
mkdir nginx-test
mkdir nginx-test/logs
mkdir nginx-test/conf
mkdir nginx-test/temp
mkdir nginx-test/run
# copy the nginx file
cp nginx-1.19.3/objs/nginx nginx-test/
now we need a configuration file
worker_processes 1;
error_log logs/error.log debug;
pid run/nginx.pid;
events {
worker_connections 1024;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log logs/access.log main;
keepalive_timeout 65;
upstream backend {
server localhost:8081;
custom 12;
}
server {
listen 8080;
server_name _;
root www/;
index index.html;
location / {
proxy_pass http://backend;
proxy_http_version 1.1;
proxy_set_header Connection "";
}
}
}
save this file to nginx-test/conf/nginx.conf and run
cd nginx-test
./nginx
you need an upstream server to listen on port 8081, and for that you can use http-server
http-server -p 8081 .
This is it. You have now your own custom nginx module. Check out logs in logs/error.log you will find the logs from your module.
If you want to implement something more useful the best way is to study the builtin modules included in the nginx source code and of course the official Nginx Development guide. Also an alternative guide with a lot more details is Emiller’s Guide To Nginx Module Development.
Following this guide I created the nginx-ntlm-module and I did inspired/copied a lot of code from the nginx keepalive module (ngx_http_upstream_keepalive_module.c). I will talk more about this module in my next post.