RSS

NGINX access_log log the real client IP from X-Forwarded-For/X-Real-IP/cf-connecting-ip instead of the proxy IP

Have a nginx server receive requests from proxy or behind CDN, how to get client user’s real IP instead of proxy/CDN servers IP in access log? This article provide two solutions based on log_format and nginx realip module.

Last Update: September 23, 2023

Problem

Have a nginx server receive requests from proxy or behind CDN, how to get client user’s real IP instead of proxy/CDN servers IP in access log?
Use cloudflare argo tunnel and nginx log show client ip of 127.0.0.1 or ::1

access_log log format

The nginx access_log default use predefined combined format for log format, the log looks like:

91.92.94.95 - - [03/Jan/2022:07:21:59 -0300] "GET /foo HTTP/1.1" 200 13831 "https://duckduckgo.com/" "Mozilla/5.0 (X11; FreeBSD amd64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.82 Safari/537.36"

According ngx_http_log_module log_format The configuration always includes the predefined “combined” format:

log_format combined '$remote_addr - $remote_user [$time_local] '
                    '"$request" $status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent"';

remote_addr defined in ngx_http_core_module :

$remote_addr
  client address

Detail explain of $remote_addr

Detail explain of $remote_addr (client address) with default nginx config:

If user connect nginx server directly (no proxy, no CDN), the client address is user’s real IP.
If user behind a proxy, the client address is proxy server.
If CDN (Content Delivery Network) is used, the client address is CDN server address.

Solution 1: Get client user real IP in nginx access_log

In today’s web, a lot web server use CDN, it is useful to log client user’s real IP instead of CDN server IP. Fortunately, CDN servers send request with X-Forwarded-For header including client user’s real IP. We can use X-Forwarded-For header’s value in log.

To do this, first need define a log format in main nginx config. For example, in debian the nginx main config file is /etc/nginx/nginx.conf. Add a log_format config inside http block:

http {
    ...

    log_format combined_realip '$http_x_forwarded_for - $remote_user [$time_local] '
                    '"$request" $status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent"';

    # For cloudflare argo tunnel, use $http_cf_connecting_ip to get `cf-connecting-ip` header
    log_format combined_realip_cf '$http_cf_connecting_ip $http_x_forwarded_for - $remote_user [$time_local] '
                    '"$request" $status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent"';
}

Then use access_log instruction to config file with our log format combined_realip:

access_log      /var/log/nginx/mysite_access.log combined;
access_log      /var/log/nginx/mysite_access_realip.log combined_realip;
access_log      /var/log/nginx/mysite_access_realip_cf.log combined_realip_cf;
error_log       /var/log/nginx/mysite_error.log;

Notes

log_format have to be in main nginx config file (e.g. /etc/nginx/nginx.conf). It can not in virtual host config, otherwise you will got a error like following:

nginx: [emerg] “log_format” directive is not allowed here in /etc/nginx/sites-enabled/foo.conf:11
Can use multiple access_log to generate multiple access log file with different formats.

X-Real-IP in request header instead of X-Forwarded-For

Some proxy / CDN servers may pass X-Real-IP header instead of X-Forwarded-For, in this case, replace $http_x_forwarded_for with $http_x_real_ip in log_format definition:

    log_format combined_realip '$http_x_real_ip - $remote_user [$time_local] '
                    '"$request" $status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent"';

Solution 2: ngx_http_realip_module with real_ip_header

nginx have a realip module is used to change the client address and optional port to those sent in the specified header field.

With realip, $remote_addr may change to client real IP address even client behind a proxy or request from CDN.

realip module is not built by default, it should be enabled with the --with-http_realip_module configuration parameter.

To check whether current installed nginx have --with-http_realip_module, use nginx -V to show configure options:

$ nginx -V
configure arguments: --with-cc-opt='-g -O2 -fdebug-prefix-map=/build/nginx-KTLRnK/nginx-1.18.0=.
-fstack-protector-strong -Wformat -Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2'
--with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -fPIC' --prefix=/usr/share/nginx
--conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log
--error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid
--modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body
--http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy
--http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug
--with-compat --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module
--with-http_realip_module
--with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module
--with-threads --with-http_addition_module --with-http_flv_module --with-http_geoip_module=dynamic
--with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic
--with-http_mp4_module --with-http_perl_module=dynamic --with-http_random_index_module
--with-http_secure_link_module --with-http_sub_module --with-http_xslt_module=dynamic
--with-mail=dynamic --with-mail_ssl_module --with-stream=dynamic --with-stream_ssl_module
--with-stream_ssl_preread_module
--add-dynamic-module=/build/nginx-KTLRnK/nginx-1.18.0/debian/modules/http-headers-more-filter
--add-dynamic-module=/build/nginx-KTLRnK/nginx-1.18.0/debian/modules/http-auth-pam
--add-dynamic-module=/build/nginx-KTLRnK/nginx-1.18.0/debian/modules/http-cache-purge
--add-dynamic-module=/build/nginx-KTLRnK/nginx-1.18.0/debian/modules/http-dav-ext
--add-dynamic-module=/build/nginx-KTLRnK/nginx-1.18.0/debian/modules/http-ndk
--add-dynamic-module=/build/nginx-KTLRnK/nginx-1.18.0/debian/modules/http-echo
--add-dynamic-module=/build/nginx-KTLRnK/nginx-1.18.0/debian/modules/http-fancyindex
--add-dynamic-module=/build/nginx-KTLRnK/nginx-1.18.0/debian/modules/nchan
--add-dynamic-module=/build/nginx-KTLRnK/nginx-1.18.0/debian/modules/http-lua
--add-dynamic-module=/build/nginx-KTLRnK/nginx-1.18.0/debian/modules/rtmp
--add-dynamic-module=/build/nginx-KTLRnK/nginx-1.18.0/debian/modules/http-uploadprogress
--add-dynamic-module=/build/nginx-KTLRnK/nginx-1.18.0/debian/modules/http-upstream-fair
--add-dynamic-module=/build/nginx-KTLRnK/nginx-1.18.0/debian/modules/http-subs-filter
--add-dynamic-module=/build/nginx-KTLRnK/nginx-1.18.0/debian/modules/http-geoip2

$ nginx -V 2>&1 | tr -- - '\n' | grep realip
http_realip_module

To use realip change $remote_addr to client user’s real IP. need following config in nginx main config:

real_ip_header X-Forwarded-For;
real_ip_recursive on;
set_real_ip_from <your proxy/CDN IP CIDR here>;

set_real_ip_from defines trusted addresses that are known to send correct replacement addresses. If IPv6 is supported, need specific both IPv4 and IPv6:

set_real_ip_from <your proxy/CDN IPv4 CIDR here>;
set_real_ip_from <your proxy/CDN IPv6 CIDR here>;

To trust real_ip_header (in this example X-Forwarded-For) in all IPv4 range, use 0.0.0.0/0:

real_ip_header X-Forwarded-For;
real_ip_recursive on;
set_real_ip_from 0.0.0.0/0;

With this 3 lines of new config, we donot even need define new log format to get client user’s real IP. $remote_addr in combined log format is enough.

Note

In case of real_ip_header X-Forwarded-For, realip module uses the last ip address in the X-Forwarded-For header for replacement.

Summary

There are at least two solutions to log client user real IP address in nginx access log. Use ngx_http_realip_module with real_ip_header without define new log format and have minimal change impact to exist configurations and exist log files.

Take X-Forwarded-For as example for client user real IP, you only need add following three lines in nginx main config (e.g. /etc/nginx/nginx.conf in Debian), inside http block:

#
# File: /etc/nginx/nginx.conf
#

http {
  ...

  real_ip_header X-Forwarded-For;
  real_ip_recursive on;
  set_real_ip_from 0.0.0.0/0;

  ...

}

NGINX config instruction syntax references

The following section list NGINX config instruction syntax for quick reference.

`real_ip_header` syntax reference

Syntax:	real_ip_header field | X-Real-IP | X-Forwarded-For | proxy_protocol;
Default:
real_ip_header X-Real-IP;
Context:	http, server, location

Defines the request header field whose value will be used to replace the client address.

The request header field value that contains an optional port is also used to replace the client port (1.11.0). The address and port should be specified according to RFC 3986.

The proxy_protocol parameter (1.5.12) changes the client address to the one from the PROXY protocol header. The PROXY protocol must be previously enabled by setting the proxy_protocol parameter in the listen directive.

`real_ip_recursive` syntax reference

Syntax:	real_ip_recursive on | off;
Default:
real_ip_recursive off;
Context:	http, server, location
This directive appeared in versions 1.3.0 and 1.2.1.

If recursive search is disabled, the original client address that matches one of
the trusted addresses is replaced by the last address sent in the request header
field defined by the real_ip_header directive. If recursive search is enabled,
the original client address that matches one of the trusted addresses is replaced
by the last non-trusted address sent in the request header field.

`set_real_ip_from` syntax reference

Syntax:	set_real_ip_from address | CIDR | unix:;
Default:	—
Context:	http, server, location

Defines trusted addresses that are known to send correct replacement addresses.
If the special value unix: is specified, all UNIX-domain sockets will be trusted.
Trusted addresses may also be specified using a hostname (1.13.1).

IPv6 addresses are supported starting from versions 1.3.0 and 1.2.1.

`log_format` syntax reference

Syntax:	log_format name [escape=default|json|none] string ...;
Default:
log_format combined "...";
Context:	http

Specifies log format.

The `escape` parameter (1.11.8) allows setting json or default characters escaping in variables,
by default, default escaping is used. The none value (1.13.10) disables escaping.

For default escaping, characters “"”, “\”, and other characters with values less than 32 (0.7.0)
or above 126 (1.1.6) are escaped as “\xXX”. If the variable value is not found, a hyphen (“-”)
will be logged.

For json escaping, all characters not allowed in JSON strings will be escaped: characters “"”
and “\” are escaped as “\"” and “\\”, characters with values less than 32 are escaped as
“\n”, “\r”, “\t”, “\b”, “\f”, or “\u00XX”.

The log format can contain common variables, and variables that exist only at the time of a log write:

$bytes_sent
  the number of bytes sent to a client
$connection
  connection serial number
$connection_requests
  the current number of requests made through a connection (1.1.18)
$msec
  time in seconds with a milliseconds resolution at the time of the log write
$pipe
  “p” if request was pipelined, “.” otherwise
$request_length
  request length (including request line, header, and request body)
$request_time
  request processing time in seconds with a milliseconds resolution; time elapsed between
  the first bytes were read from the client and the log write after the last bytes were
  sent to the client
$status
  response status
$time_iso8601
  local time in the ISO 8601 standard format
$time_local
  local time in the Common Log Format

In the modern nginx versions variables
$status (1.3.2, 1.2.2),
$bytes_sent (1.3.8, 1.2.5),
$connection (1.3.8, 1.2.5),
$connection_requests (1.3.8, 1.2.5),
$msec (1.3.9, 1.2.6),
$request_time (1.3.9, 1.2.6),
$pipe (1.3.12, 1.2.7),
$request_length (1.3.12, 1.2.7),
$time_iso8601 (1.3.12, 1.2.7),
and $time_local (1.3.12, 1.2.7) are also available as common variables.

Header lines sent to a client have the prefix “sent_http_”, for example, $sent_http_content_range.

The configuration always includes the predefined “combined” format:

log_format combined '$remote_addr - $remote_user [$time_local] '
                    '"$request" $status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent"';

References

OmniLock - Block / Hide App on iOS

Block distractive apps from appearing on the Home Screen and App Library, enhance your focus and reduce screen time.

DNS Firewall for iOS and Mac OS

Encrypted your DNS to protect your privacy and firewall to block phishing, malicious domains, block ads in all browsers and apps