My Nginx Setup Kickstart / Boilerplate

Since the first time I used Nginx in mid-2011, Nginx immediately became my favorite web server. I am slowly starting to leave Apache behind, which was previously the “standard” web server on the Linux operating system.

As time went by, several new web servers began to appear, such as Caddy and Traefik. As a system administrator, of course I have tried to use it, although only to the extent of using personal projects.

However, my heart always seems to return to Nginx. Applications, services, or anything that I can expose via Nginx, I will expose that using Nginx. Maybe because I’m become too comfortable with the configuration and pleasant experience with Nginx. XD

My use case

Because I have very limited IPv4, I mostly use Nginx as a reverse proxy for services that don’t have a public IP (VMs with local / internal networks). This really helps save public IP allocation. Using Nginx as reverse proxy, I played a lot with proxy_cache and http upstream to implement load balancing or failover.

Back then, when I created programs using PHP, I used Nginx and PHP-FPM without Apache (.htaccess) behind it. So I play a lot with Nginx rewrite and fastcgi_cache. When I started making applications using Rust and Go, Nginx always act as reverse proxy while also performing SSL termination.

Besides HTTP reverse proxy, I sometimes use the Nginx stream module for TCP, UDP, and Unix socket data streams.

Regarding traffic monitoring, I always use Nginx VTS module. There are nginx-vts-exporter for [Prometheus](https://prometheus .io/) which is very easy to operate to process data from Nginx VTS module. Meanwhile, for logging, some logs for virtual hosts that I consider crucial are sent in real-time to the remote syslog server.

It’s perfect, all the features I need are met by Nginx. And it’s time for me to start documenting the installation and configuration process.

INFO:

I have an open-source project called nginx-kickstart (boilerplate) to make it easier to install Nginx from the official repository and compile the Nginx VTS module on a FRESH Debian 12 or Ubuntu 22.04 server.

Installing Nginx official repo

This documentation was created for Debian 12 and Ubuntu 22.04, and I used the official repository from Nginx, not the distribution-provided package.

First and foremost, always make sure the system is up-to-date by running sudo aptget update && sudo apt-get dist-upgrade. Then install required packages.

For Debian:

1apt install sudo curl gnupg2 ca-certificates lsb-release debian-archive-keyring

For Ubuntu:

1apt install sudo curl gnupg2 ca-certificates lsb-release ubuntu-keyring

Then, import the official Nginx signing key:

1curl https://nginx.org/keys/nginx_signing.key | gpg --dearmor \
2    | sudo tee /usr/share/keyrings/nginx-archive-keyring.gpg >/dev/null

Set up the apt repository for stable nginx packages:

1echo "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] \
2http://nginx.org/packages/ubuntu `lsb_release -cs` nginx" \
3    | sudo tee /etc/apt/sources.list.d/nginx.list

Set up repository pinning to prefer official packages over distribution-provided ones:

1echo -e "Package: *\nPin: origin nginx.org\nPin: release o=nginx\nPin-Priority: 900\n" \
2    | sudo tee /etc/apt/preferences.d/99nginx

Then, install nginx and nginx-module-geoip:

1sudo apt update && sudo apt install nginx nginx-module-geoip

Load the following http_geoip_module and stream_geoip_module. Put the load_module above event{} block and geoip_country inside http{} block:

 1load_module modules/ngx_http_geoip_module.so;
 2load_module modules/ngx_stream_geoip_module.so;
 3
 4event {
 5    worker_connections 65535; # Nginx default: 1024
 6}
 7
 8http {
 9    geoip_country /usr/share/GeoIP/GeoIP.dat;
10
11    # ...
12}

Preparing the Nginx directory structure

Create the sites-available, sites-enabled, certs, snippets directories inside the /etc/nginx directory:

1sudo mkdir -p /etc/nginx/{sites-available,sites-enabled,certs,snippets}

Create a self-signed certificate (only used as an initial configuration which will later be replaced by certbot):

1sudo openssl req -x509 -newkey rsa:4096 -days 365 -nodes \
2    -keyout /etc/nginx/certs/privkey.pem                 \
3    -out /etc/nginx/certs/fullchain.pem                  \
4    -subj '/CN=example.local/O=My Organization/C=US'

Create DH-param by running:

1sudo openssl dhparam -out /etc/nginx/certs/dhparam.pem 2048

Cloudflare’s IPs trusted proxy

If there is a virtual host behind a Cloudflare reverse proxy, it is highly recommended to add the Cloudflare IP addreses to the trusted proxy in the Nginx configuration.

Create the following executable shell script /etc/nginx/cloudflare-ips.sh:

 1#!/usr/bin/env bash
 2# Nginx setup for cloudflare's IPs.
 3# https://github.com/ditatompel/nginx-kickstart/blob/main/etc/nginx/cloudflare-ips.sh
 4# This is modified version of itsjfx's cloudflare-nginx-ips
 5# Ref of original script:
 6# https://github.com/itsjfx/cloudflare-nginx-ips/blob/master/cloudflare-ips.sh
 7
 8set -e
 9
10[ "$(id -u)" -ne 0 ] && echo "This script must be run as root" && exit 1
11
12CF_REAL_IPS_PATH=/etc/nginx/snippets/cloudflare_real_ips.conf
13CF_WHITELIST_PATH=/etc/nginx/snippets/cloudflare_whitelist.conf
14CF_GEOIP_PROXY_PATH=/etc/nginx/snippets/cloudflare_geoip_proxy.conf
15
16for file in $CF_REAL_IPS_PATH $CF_WHITELIST_PATH $CF_GEOIP_PROXY_PATH; do
17    echo "# https://www.cloudflare.com/ips" > $file
18    echo "# Generated at $(LC_ALL=C date)" >> $file
19done
20
21echo "geo \$realip_remote_addr \$cloudflare_ip {
22    default 0;" >> $CF_WHITELIST_PATH
23
24for type in v4 v6; do
25    for ip in `curl -sL https://www.cloudflare.com/ips-$type`; do
26        echo "set_real_ip_from $ip;" >> $CF_REAL_IPS_PATH;
27        echo "    $ip 1;" >> $CF_WHITELIST_PATH;
28        echo "geoip_proxy $ip;" >> $CF_GEOIP_PROXY_PATH;
29    done
30done
31
32echo "}
33# if your vhost is behind CloudFlare proxy and you want your site only
34# accessible from Cloudflare proxy, add this in your server{} block:
35# if (\$cloudflare_ip != 1) {
36#    return 403;
37# }" >> $CF_WHITELIST_PATH
38
39nginx -t && systemctl reload nginx
40
41# vim: set ts=4 sw=4 et:

The shell script above will fetch Cloudflare’s IP list to be processed and stored in /etc/nginx/snippets/cloudflare_*.conf. Please create a cronjob to run the script periodically (per week / per month).

For the Nginx configuration, add the following configuration to the http{} block in /etc/nginx/nginx.conf:

 1http {
 2    # ...
 3
 4    # Cloudflare IPs
 5    ################
 6    include /etc/nginx/snippets/cloudflare_real_ips.conf;
 7    real_ip_header X-Forwarded-For; # atau CF-Connecting-IP jika menggunakan Cloudflare
 8    # cloudflare map
 9    include /etc/nginx/snippets/cloudflare_whitelist.conf;
10
11    # ...

Logging

The logging feature may slowing down server performance (mainly due to high DISK I/O) on high traffic sites. However, logging is also very important for monitoring and analyzing server activity.

Log format

There are several log formats that are commonly used and can be integrated with 3rd-party applications, for example the (V)COMMON or (V)COMBINED format.

VCOMBINED format

Add the following configuration to the http{} block:

 1http {
 2    # ...
 3
 4    # VCOMBINED log format style
 5    log_format vcombined '$host:$server_port '
 6        '$remote_addr - $remote_user [$time_local] '
 7        '"$request" $status $body_bytes_sent '
 8        '"$http_referer" "$http_user_agent"';
 9
10    # ...

I usually use VCOMBINED format logs which I then integrate with GoAccess.

Custom JSON log format

For some cases, I use Nginx integration in Grafana Cloud which uses custom access log format (JSON):

 1http {
 2    # ...
 3
 4    # JSON style log format
 5    log_format json_analytics escape=json '{'
 6        '"msec": "$msec", ' # request unixtime in seconds with a milliseconds resolution
 7        '"connection": "$connection", ' # connection serial number
 8        '"connection_requests": "$connection_requests", ' # number of requests made in connection
 9        '"pid": "$pid", ' # process pid
10        '"request_id": "$request_id", ' # the unique request id
11        '"request_length": "$request_length", ' # request length (including headers and body)
12        '"remote_addr": "$remote_addr", ' # client IP
13        '"remote_user": "$remote_user", ' # client HTTP username
14        '"remote_port": "$remote_port", ' # client port
15        '"time_local": "$time_local", '
16        '"time_iso8601": "$time_iso8601", ' # local time in the ISO 8601 standard format
17        '"request": "$request", ' # full path no arguments if the request
18        '"request_uri": "$request_uri", ' # full path and arguments if the request
19        '"args": "$args", ' # args
20        '"status": "$status", ' # response status code
21        '"body_bytes_sent": "$body_bytes_sent", ' # the number of body bytes exclude headers sent to a client
22        '"bytes_sent": "$bytes_sent", ' # the number of bytes sent to a client
23        '"http_referer": "$http_referer", ' # HTTP referer
24        '"http_user_agent": "$http_user_agent", ' # user agent
25        '"http_x_forwarded_for": "$http_x_forwarded_for", ' # http_x_forwarded_for
26        '"http_host": "$http_host", ' # the request Host: header
27        '"server_name": "$server_name", ' # the name of the vhost serving the request
28        '"request_time": "$request_time", ' # request processing time in seconds with msec resolution
29        '"upstream": "$upstream_addr", ' # upstream backend server for proxied requests
30        '"upstream_connect_time": "$upstream_connect_time", ' # upstream handshake time incl. TLS
31        '"upstream_header_time": "$upstream_header_time", ' # time spent receiving upstream headers
32        '"upstream_response_time": "$upstream_response_time", ' # time spent receiving upstream body
33        '"upstream_response_length": "$upstream_response_length", ' # upstream response length
34        '"upstream_cache_status": "$upstream_cache_status", ' # cache HIT/MISS where applicable
35        '"ssl_protocol": "$ssl_protocol", ' # TLS protocol
36        '"ssl_cipher": "$ssl_cipher", ' # TLS cipher
37        '"scheme": "$scheme", ' # http or https
38        '"request_method": "$request_method", ' # request method
39        '"server_protocol": "$server_protocol", ' # request protocol, like HTTP/1.1 or HTTP/2.0
40        '"pipe": "$pipe", ' # "p" if request was pipelined, "." otherwise
41        '"gzip_ratio": "$gzip_ratio", '
42        '"geoip_country_code": "$geoip_country_code"'
43        '}';
44
45    # ...
46}

Conditional (dynamic) logging

With map, and if keyword, we can determine what to log and what not to log. For example, I don’t do logging if the URI contains the word “local” or User Agent contains the word “Uptime-Kuma”:

 1http {
 2    # ...
 3    
 4    map $request_uri$http_user_agent $is_loggable {
 5        ~*local          0;
 6        ~*Uptime-Kuma.*  0;
 7        default          1;
 8    }
 9
10    access_log     /var/log/nginx/access-vcombined.log vcombined if=$is_loggable;
11
12    # ...
13}

Remote log UDP (rsyslog)

For me, log centralization really makes my job easier in carrying out server analysis and troubleshooting.

In Nginx, we can easily send logs to remote servers in real-time. For example, we can send logs to a remote rsyslog server** (UDP) with the following example configuration:

1http {
2    # ...
3
4    access_log     syslog:server=192.168.0.7:514,facility=local7,tag=nginx,severity=info vcombined if=$is_loggable;
5    access_log     syslog:server=192.168.0.7:514,facility=local7,tag=nginx_grafana,severity=info json_analytics if=$is_loggable;
6
7    # ...
8}

Compiling Nginx VTS module

Nginx VTS module is not available in the Official Nginx repository, so we cannot install it using apt. To compile the VTS module requires C compiler, git, libpcre, libssl, and zlib. Install the required packages by running this command:

1sudo apt install git build-essential libpcre3-dev zlib1g-dev libssl-dev

This is a very important part, if you want to use a dynamically linked module, the compile module option must be the same as the Nginx binary file that will be used, as well as the version of Nginx used. To find out the information we need, run nginx -V command. Example output:

1nginx version: nginx/1.26.0
2built by gcc 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04)
3built with OpenSSL 3.0.2 15 Mar 2022
4TLS SNI support enabled
5configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-http_v3_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -O2 -ffile-prefix-map=/data/builder/debuild/nginx-1.26.0/debian/debuild-base/nginx-1.26.0=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie'

Download Nginx source with a version that is exactly the same with the one we are using, in this example 1.26.0.

1curl -O https://nginx.org/download/nginx-1.26.0.tar.gz

Extract the Nginx source code archive, then go to it’s directory:

1tar -xvzf nginx-1.26.0.tar.gz
2cd nginx-1.26.0

Clone the vozlt/nginx-module-vts repository and use the latest release tag. When this article was written, the last tag release was v0.2.2, so:

1git clone -b v0.2.2 https://github.com/vozlt/nginx-module-vts.git

Configure with the same arguments from the nginx -V output above and add --add-dynamic-module=./nginx-module-vts/. Examples in this article:

1./configure --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-http_v3_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -O2 -ffile-prefix-map=/data/builder/debuild/nginx-1.26.0/debian/debuild-base/nginx-1.26.0=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie' -add-dynamic-module=./nginx-module-vts/

Build, then copy the VTS module you just compiled to /etc/nginx/modules/:

1make modules -j$(nproc)
2sudo cp objs/ngx_http_vhost_traffic_status_module.so /etc/nginx/modules/

Nginx VTS module configuration

Edit the /etc/nginx/nginx.conf file and load the host_traffic_status_module. Place load_module above event{} block:

1load_module modules/ngx_http_vhost_traffic_status_module.so;

Then inside http{} block, add the following configuration:

1http {
2    # ...
3
4    geoip_country /usr/share/GeoIP/GeoIP.dat;
5    vhost_traffic_status_zone;
6    vhost_traffic_status_filter_by_set_key $geoip_country_code country::*;
7
8    # ...
9}

To display the VTS traffic status page, add the following example configuration to the server{} block (for example in /etc/nginx/conf.d/default.conf):

 1server {
 2    # ...
 3
 4    # example Nginx VTS display page
 5    location /status {
 6        vhost_traffic_status_bypass_limit on;
 7        vhost_traffic_status_bypass_stats on;
 8        vhost_traffic_status_display;
 9        vhost_traffic_status_display_format html;
10        access_log off;
11        # Example restricting VTS access to specific IP
12        allow 127.0.0.1;
13        allow 192.168.0.0/24;
14        deny  all;
15    }
16
17    location / {
18        root   /usr/share/nginx/html;
19        index  index.html index.htm;
20    }
21
22    # ...
23}