NGINX Reverse Proxy Troubleshooting
Test Configuration File
After making change to nginx configuration file nginx.conf
and any included configuration files like site specific configuration,
It is useful to use nginx -t to test configuration syntax and validation before apply it.
$ nginx -t
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
nginx: configuration file /etc/nginx/nginx.conf test failed
You may need sudo:
$ sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
Another useful parameter is -T which will test configuration and dump it to console.
$ sudo nginx -T
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
# configuration file /etc/nginx/nginx.conf:
user www-data;
worker_processes auto;
pid /run/nginx.pid;
events {
worker_connections 768;
# multi_accept on;
}
http {
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
# server_tokens off;
# server_names_hash_bucket_size 64;
# server_name_in_redirect off;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# SSL Settings
##
ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;
...
...
...
Reload NGINX without restart server
Use -s parameter you can send signal to NGINX process.
When you changed configuration file, you can send reload to reload server without restart server.
$ sudo nginx -s reload
All available signals: stop, quit, reopen, reload
-s signal: send signal to a master process: stop, quit, reopen, reload
The difference between a proxy server and a reverse proxy server
A common question is what's the difference between a proxy server and a reverse proxy server?
An ordinary forward proxy is an intermediate server that sits between the client and the origin server. In order to get content from the origin server, the client sends a request to the proxy naming the origin server as the target. The proxy then requests the content from the origin server and returns it to the client. The client must be specially configured to use the forward proxy to access other sites.
A typical usage of a forward proxy is to provide Internet access to internal clients that are otherwise restricted by a firewall. The forward proxy can also use caching to reduce network usage.
A reverse proxy (or gateway), by contrast, appears to the client just like an ordinary web server. No special configuration on the client is necessary. The client makes ordinary requests for content in the namespace of the reverse proxy. The reverse proxy then decides where to send those requests and returns the content as if it were itself the origin.
A typical usage of a reverse proxy is to provide Internet users access to a server that is behind a firewall. Reverse proxies can also be used to balance load among several back-end servers or to provide caching for a slower back-end server. In addition, reverse proxies can be used simply to bring several servers into the same URL space.
Keep-alive not working with proxy_pass
Symptom
proxy_pass does not support keep-alive event server response Connection: keep-alive header like below:
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 31746
Content-Type: text/html; charset=utf-8
Connection: keep-alive
Date: Wed, 05 Feb 2020 21:32:51 GMT
Try to add following Connection: keep-alive in proxy also does not work:
proxy_set_header Connection "keep-alive"
Root Cause
keep-alive should enable in upstream block, not direct proxy_pass.
Keep-alive also require proxy use http version 1.1.
Solution
- Use
upstreaminstead of directproxy_pass. - use http version 1.1
- origin server should have keep-alive enabled.
Here is a configuration sample:
upstream hugo {
server 127.0.0.1:1313;
keepalive 60;
}
location / {
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_pass http://hugo;
}
504 Gateway Timeout
Symptom
Web page takes long time to connect and eventually show 504 Gateway Timeout.
Solution
The original server may slow to response, try to increase timeout:
proxy_connect_timeout 120;
proxy_send_timeout 120;
proxy_read_timeout 120;
send_timeout 120;
502 Bad Gateway - Too large header
Symptom
too big header error in NIGNX error.log:
upstream sent too big header while reading response header from upstream
You can use curl -I to fetch the response headers only, you may see very large header.
For example, it may include lots Cookie caused too large header:
$ curl -I localhost:3000/logout
HTTP/1.1 302 Found
Location: http://localhost:3000/
Content-Type: text/html; charset=utf-8
X-Ua-Compatible: IE=Edge
Cache-Control: no-cache
X-Request-Id: fe41f23970ed792163b49210790a1bc5
X-Runtime: 0.002031
Vary: Origin
X-Rack-Cors: miss; no-origin
Server: WEBrick/1.3.1 (Ruby/2.0.0/2015-12-16)
Date: Thu, 06 Feb 2020 03:19:03 GMT
Content-Length: 88
Connection: Keep-Alive
Set-Cookie: xsrf-token=xfyj%2FGhBGNwbfoet2VPnKTO7BWnkqAS1jY3wr%2BV9cSQ%3D; path=/
Set-Cookie: session=Vom40W3EDxuOWNWrVQ9msOYUvnwM42Eez9AvaMAtj69lbnXPwM7AGCihD40MF3zrVV/58bvj4bIU7R5NtkJPeFIx1JjJ8lI3df9uij8VvuNd4UhvNs8tOM13P4Lsl2KbYz9WY9o4ZIwpkhq+AI641jU4gKGHhHkuiYwwGioYFhLKx3/Fsn+tEEv8cgE4lYoF+G2tFJg3cL05/SVkM52; path=/; HttpOnly
Set-Cookie: lang=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: ae=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: cv=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: es=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: event69=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: mbox=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: promocode=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: s_cc=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: s_fid=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: s_gpid=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: s_nb=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: provider_id=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: provider_name=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: user_id=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: api_token=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: shost=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: app_version=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: user_token=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: dashboard=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: tags=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: option_1=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: option_2=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: option_3=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: option_4=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: option_5=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: option_6=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: memo_4s_1=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: memo_4s_2=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: memo_4s_3=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: memo_4s_4=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: memo_4s_5=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: memo_4s_6=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: memo_4s_7=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: memo_4s_8=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Solution
Increase proxy_buffer_size under server:
proxy_buffer_size 8k;
proxy_buffers 16 8k;
Note: Only update proxy_buffer_size may not work,
when you run nginx -t to test the configuration,
it may report following error:
$ sudo nginx -t
nginx: [emerg] "proxy_busy_buffers_size" must be less than the size of all "proxy_buffers" minus one buffer in /etc/nginx/nginx.conf:68
nginx: configuration file /etc/nginx/nginx.conf test failed
In this case, you need also increase proxy_buffers.
Original server redirect missing port number in URL
Symptom
When original server response a redirect use Location header,
the redirect URL missing proxy server port number.
Root cause
It due to request to original server do not have correct Host header.
For example, Host may wrongly set to $host:
proxy_set_header Host $host
Solution
Set Host header value to $proxy_host to include server name and port number:
proxy_set_header Host $proxy_host
A reference of reverse proxy server configuration
server {
listen 2020;
server_name example.djangocas.dev;
proxy_connect_timeout 120;
proxy_send_timeout 120;
proxy_read_timeout 120;
send_timeout 120;
proxy_buffer_size 8k;
proxy_buffers 16 8k;
proxy_http_version 1.1;
proxy_set_header Host $proxy_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
location /api/ {
proxy_pass https://example.com/demo/api/;
}
location / {
proxy_pass http://localhost:1313;
}
access_log /var/log/nginx/example-site/access.log;
error_log /var/log/nginx/example-site/error.log;
}
NGINX Directives Reference
proxy_pass
Syntax: proxy_pass URL;
Default: —
Context: location, if in location, limit_except
Sets the protocol and address of a proxied server and an optional URI to which a location should be mapped. As a protocol, “http” or “https” can be specified. The address can be specified as a domain name or IP address, and an optional port:
proxy_pass http://localhost:8000/uri/;
or as a UNIX-domain socket path specified after the word “unix” and enclosed in colons:
proxy_pass http://unix:/tmp/backend.socket:/uri/;
If a domain name resolves to several addresses, all of them will be used in a round-robin fashion. In addition, an address can be specified as a server group.
Parameter value can contain variables. In this case, if an address is specified as a domain name, the name is searched among the described server groups, and, if not found, is determined using a resolver.
A request URI is passed to the server as follows:
If the proxy_pass directive is specified with a URI, then when a request is passed to the server, the part of a normalized request URI matching the location is replaced by a URI specified in the directive:
location /name/ {
proxy_pass http://127.0.0.1/remote/;
}
If proxy_pass is specified without a URI, the request URI is passed to the server in the same form as sent by a client when the original request is processed, or the full normalized request URI is passed when processing the changed URI:
location /some/path/ {
proxy_pass http://127.0.0.1;
}
Before version 1.1.12, if proxy_pass is specified without a URI, the original request URI might be passed instead of the changed URI in some cases. In some cases, the part of a request URI to be replaced cannot be determined:
When location is specified using a regular expression, and also inside named locations. In these cases, proxy_pass should be specified without a URI.
When the URI is changed inside a proxied location using the rewrite directive, and this same configuration will be used to process a request (break):
location /name/ {
rewrite /name/([^/]+) /users?name=$1 break;
proxy_pass http://127.0.0.1;
}
In this case, the URI specified in the directive is ignored and the full changed request URI is passed to the server.
When variables are used in proxy_pass:
location /name/ {
proxy_pass http://127.0.0.1$request_uri;
}
In this case, if URI is specified in the directive, it is passed to the server as is, replacing the original request URI. WebSocket proxying requires special configuration and is supported since version 1.3.13.
proxy_buffer_size
Syntax: proxy_buffer_size size;
Default: proxy_buffer_size 4k|8k;
Context: http, server, location
Sets the size of the buffer used for reading the first part of the response received from the proxied server. This part usually contains a small response header. By default, the buffer size is equal to one memory page. This is either 4K or 8K, depending on a platform. It can be made smaller, however.
proxy_buffers
Syntax: proxy_buffers number size;
Default: proxy_buffers 8 4k|8k;
Context: http, server, location
Sets the number and size of the buffers used for reading a response from the proxied server, for a single connection. By default, the buffer size is equal to one memory page. This is either 4K or 8K, depending on a platform.
proxy_busy_buffers_size
Syntax: proxy_busy_buffers_size size;
Default: proxy_busy_buffers_size 8k|16k;
Context: http, server, location
When buffering of responses from the proxied server is enabled, limits the total size of buffers that can be busy sending a response to the client while the response is not yet fully read. In the meantime, the rest of the buffers can be used for reading the response and, if needed, buffering part of the response to a temporary file. By default, size is limited by the size of two buffers set by the proxy_buffer_size and proxy_buffers directives.
proxy_http_version
Syntax: proxy_http_version 1.0 | 1.1;
Default: proxy_http_version 1.0;
Context: http, server, location
This directive appeared in version 1.1.4.
Sets the HTTP protocol version for proxying. By default, version 1.0 is used. Version 1.1 is recommended for use with keepalive connections and NTLM authentication.
proxy_set_header
Syntax: proxy_set_header field value;
Default: proxy_set_header Host $proxy_host;
proxy_set_header Connection close;
Context: http, server, location
Allows redefining or appending fields to the request header passed to the proxied server. The value can contain text, variables, and their combinations. These directives are inherited from the previous level if and only if there are no proxy_set_header directives defined on the current level. By default, only two fields are redefined:
proxy_set_header Host $proxy_host;
proxy_set_header Connection close;
If caching is enabled, the header fields “If-Modified-Since”, “If-Unmodified-Since”, “If-None-Match”, “If-Match”, “Range”, and “If-Range” from the original request are not passed to the proxied server.
An unchanged “Host” request header field can be passed like this:
proxy_set_header Host $http_host;
However, if this field is not present in a client request header then nothing will be passed. In such a case it is better to use the $host variable - its value equals the server name in the “Host” request header field or the primary server name if this field is not present:
proxy_set_header Host $host;
In addition, the server name can be passed together with the port of the proxied server:
proxy_set_header Host $host:$proxy_port;
If the value of a header field is an empty string then this field will not be passed to a proxied server:
proxy_set_header Accept-Encoding "";
NIGNX Embedded Variables
Embedded variables that can be used to compose headers using the proxy_set_header directive.
$host
in this order of precedence: host name from the request line, or host name from the “Host” request header field, or the server name matching a request. It does not include port number.
$remote_addr
client address. e.g. 1.2.3.4.
$remote_port
client port. e.g. 24510
$server_name
name of the server which accepted a request
$server_port
port of the server which accepted a request
$proxy_host
name and port of a proxied server as specified in the proxy_pass directive;
$proxy_port
port of a proxied server as specified in the proxy_pass directive, or the protocol’s default port;
$proxy_add_x_forwarded_for
the “X-Forwarded-For” client request header field with the $remote_addr variable appended to it, separated by a comma. If the “X-Forwarded-For” field is not present in the client request header, the $proxy_add_x_forwarded_for variable is equal to the $remote_addr variable.