Add 503 status code when there are zero upstream endpoints #3406

bjee19 · 2025-05-27T22:23:51Z

Proposed changes

Add 503 status code when there are zero upstream endpoints.

Problem: When using NGINX Plus, the generated state file for an upstream was not correctly generating the 503 server for upstreams with zero endpoints.

Solution: On upstreams with zero endpoints, generate the correct nginx conf in the state file.

Testing: The 503 server socket shows up in the nginx plus API dashboard, both when initially deploying resources, and when scaling the deployment as described in #2090. Sending traffic to the upstream with zero endpoints correctly returns a 503 status code (instead of 502 as previously) and there is no longer an error message outputted in the nginx logs. Verified after deployment/pod gets ready, so the upstream endpoint becomes valid, the state file is updated correctly to have the correct upstream endpoint, and traffic flows correctly.

Closes #2090

Checklist

Before creating a PR, run through this checklist and mark each as complete.

I have read the CONTRIBUTING doc
I have added tests that prove my fix is effective or that my feature works
I have checked that all unit tests pass after adding my changes
I have updated necessary documentation
I have rebased my branch onto main
I will ensure my PR is targeting the main branch and pulling from my branch from my own fork

Release notes

If this PR introduces a change that affects users and needs to be mentioned in the release notes,
please add a brief note that summarizes the change.

internal/mode/static/nginx/agent/agent.go

bjee19 · 2025-05-27T22:26:06Z

nginx conf here:

~/Code/nginx-gateway-fabric/examples/cafe-example main ⇡1281 *6 !2 ❯ kubectl exec -it gateway-nginx-9d9bbc669-spv6b -- nginx -T                                                                     ⎈ kind-kind
Defaulted container "nginx" out of: nginx, init (init)
2025/05/27 22:06:31 [notice] 131#131: js vm init njs: 0000FFFF8D566A00
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
# configuration file /etc/nginx/nginx.conf:
load_module /usr/lib/nginx/modules/ngx_http_js_module.so;
include /etc/nginx/main-includes/*.conf;

worker_processes auto;

pid /var/run/nginx/nginx.pid;

events {
  worker_connections 1024;
}

http {
  include /etc/nginx/conf.d/*.conf;
  include /etc/nginx/mime.types;
  js_import /usr/lib/nginx/modules/njs/httpmatches.js;

  default_type application/octet-stream;

  proxy_headers_hash_bucket_size 512;
  proxy_headers_hash_max_size 1024;
  server_names_hash_bucket_size 256;
  server_names_hash_max_size 1024;
  variables_hash_bucket_size 512;
  variables_hash_max_size 1024;

  sendfile on;
  tcp_nopush on;

  server_tokens off;
}

stream {
  variables_hash_bucket_size 512;
  variables_hash_max_size 1024;

  map_hash_max_size 2048;
  map_hash_bucket_size 256;

  log_format stream-main '$remote_addr [$time_local] '
                         '$protocol $status $bytes_sent $bytes_received '
                         '$session_time "$ssl_preread_server_name"';
  access_log /dev/stdout stream-main;
  include /etc/nginx/stream-conf.d/*.conf;
}

# configuration file /etc/nginx/main-includes/main.conf:

error_log stderr info;


# configuration file /etc/nginx/main-includes/mgmt.conf:

mgmt {
        license_token /etc/nginx/secrets/license.jwt;
        deployment_context /etc/nginx/main-includes/deployment_ctx.json;
}

# configuration file /etc/nginx/conf.d/http.conf:
http2 on;

# Set $gw_api_compliant_host variable to the value of $http_host unless $http_host is empty, then set it to the value
# of $host. We prefer $http_host because it contains the original value of the host header, which is required by the
# Gateway API. However, in an HTTP/1.0 request, it's possible that $http_host can be empty. In this case, we will use
# the value of $host. See http://nginx.org/en/docs/http/ngx_http_core_module.html#var_host.
map $http_host $gw_api_compliant_host {
    '' $host;
    default $http_host;
}

# Set $connection_header variable to upgrade when the $http_upgrade header is set, otherwise, set it to close. This
# allows support for websocket connections. See https://nginx.org/en/docs/http/websocket.html.
map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
}

## Returns just the path from the original request URI.
map $request_uri $request_uri_path {
  "~^(?P<path>[^?]*)(\?.*)?$"  $path;
}


js_preload_object matches from /etc/nginx/conf.d/matches.json;
server {
    listen 80 default_server;
    listen [::]:80 default_server;
    default_type text/html;
    return 404;
}

server {
    listen 80;
    listen [::]:80;

    server_name cafe.example.com;
    status_zone cafe.example.com;

        
    location /coffee/ {
        

        

        proxy_http_version 1.1;
        proxy_set_header Host "$gw_api_compliant_host";
        proxy_set_header X-Forwarded-For "$proxy_add_x_forwarded_for";
        proxy_set_header X-Real-IP "$remote_addr";
        proxy_set_header X-Forwarded-Proto "$scheme";
        proxy_set_header X-Forwarded-Host "$host";
        proxy_set_header X-Forwarded-Port "$server_port";
        proxy_set_header Upgrade "$http_upgrade";
        proxy_set_header Connection "$connection_upgrade";
        proxy_pass http://default_coffee_80$request_uri;
            
            
            
    }
    location = /coffee {
        

        

        proxy_http_version 1.1;
        proxy_set_header Host "$gw_api_compliant_host";
        proxy_set_header X-Forwarded-For "$proxy_add_x_forwarded_for";
        proxy_set_header X-Real-IP "$remote_addr";
        proxy_set_header X-Forwarded-Proto "$scheme";
        proxy_set_header X-Forwarded-Host "$host";
        proxy_set_header X-Forwarded-Port "$server_port";
        proxy_set_header Upgrade "$http_upgrade";
        proxy_set_header Connection "$connection_upgrade";
        proxy_pass http://default_coffee_80$request_uri;
            
            
            
    }
    location = /tea {
        

        

        proxy_http_version 1.1;
        proxy_set_header Host "$gw_api_compliant_host";
        proxy_set_header X-Forwarded-For "$proxy_add_x_forwarded_for";
        proxy_set_header X-Real-IP "$remote_addr";
        proxy_set_header X-Forwarded-Proto "$scheme";
        proxy_set_header X-Forwarded-Host "$host";
        proxy_set_header X-Forwarded-Port "$server_port";
        proxy_set_header Upgrade "$http_upgrade";
        proxy_set_header Connection "$connection_upgrade";
        proxy_pass http://default_tea_80$request_uri;
            
            
            
    }
    location / {
        
        return 404 "";

        

        proxy_http_version 1.1;
    }
}

server {
    listen unix:/var/run/nginx/nginx-503-server.sock;
    access_log off;

    return 503;
}

server {
    listen unix:/var/run/nginx/nginx-500-server.sock;
    access_log off;

    return 500;
}


upstream default_coffee_80 {
    random two least_conn;
    zone default_coffee_80 1m;
    
    state /var/lib/nginx/state/default_coffee_80.conf;
    
    
    
    
}

upstream default_tea_80 {
    random two least_conn;
    zone default_tea_80 1m;
    
    state /var/lib/nginx/state/default_tea_80.conf;
    
    
    
    
}

upstream invalid-backend-ref {
    random two least_conn;
    
        
    server unix:/var/run/nginx/nginx-500-server.sock;
    
    
    
    
}





# configuration file /var/lib/nginx/state/default_coffee_80.conf:
server 10.244.0.7:8080;

# configuration file /var/lib/nginx/state/default_tea_80.conf:
server unix:/var/run/nginx/nginx-503-server.sock;

# configuration file /etc/nginx/conf.d/plus-api.conf:

server {
    listen unix:/var/run/nginx/nginx-plus-api.sock;
    access_log off;

    location /api {
      api write=on;
    }
}

server {
    listen 8765;
    root /usr/share/nginx/html;
    access_log off;
    
    allow 127.0.0.1;
    deny all;

    location = /dashboard.html {}

    location /api {
      api write=off;
    }
}

# configuration file /etc/nginx/mime.types:

types {
    text/html                                        html htm shtml;
    text/css                                         css;
    text/xml                                         xml;
    image/gif                                        gif;
    image/jpeg                                       jpeg jpg;
    application/javascript                           js;
    application/atom+xml                             atom;
    application/rss+xml                              rss;

    text/mathml                                      mml;
    text/plain                                       txt;
    text/vnd.sun.j2me.app-descriptor                 jad;
    text/vnd.wap.wml                                 wml;
    text/x-component                                 htc;

    image/avif                                       avif;
    image/png                                        png;
    image/svg+xml                                    svg svgz;
    image/tiff                                       tif tiff;
    image/vnd.wap.wbmp                               wbmp;
    image/webp                                       webp;
    image/x-icon                                     ico;
    image/x-jng                                      jng;
    image/x-ms-bmp                                   bmp;

    font/woff                                        woff;
    font/woff2                                       woff2;

    application/java-archive                         jar war ear;
    application/json                                 json;
    application/mac-binhex40                         hqx;
    application/msword                               doc;
    application/pdf                                  pdf;
    application/postscript                           ps eps ai;
    application/rtf                                  rtf;
    application/vnd.apple.mpegurl                    m3u8;
    application/vnd.google-earth.kml+xml             kml;
    application/vnd.google-earth.kmz                 kmz;
    application/vnd.ms-excel                         xls;
    application/vnd.ms-fontobject                    eot;
    application/vnd.ms-powerpoint                    ppt;
    application/vnd.oasis.opendocument.graphics      odg;
    application/vnd.oasis.opendocument.presentation  odp;
    application/vnd.oasis.opendocument.spreadsheet   ods;
    application/vnd.oasis.opendocument.text          odt;
    application/vnd.openxmlformats-officedocument.presentationml.presentation
                                                     pptx;
    application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
                                                     xlsx;
    application/vnd.openxmlformats-officedocument.wordprocessingml.document
                                                     docx;
    application/vnd.wap.wmlc                         wmlc;
    application/wasm                                 wasm;
    application/x-7z-compressed                      7z;
    application/x-cocoa                              cco;
    application/x-java-archive-diff                  jardiff;
    application/x-java-jnlp-file                     jnlp;
    application/x-makeself                           run;
    application/x-perl                               pl pm;
    application/x-pilot                              prc pdb;
    application/x-rar-compressed                     rar;
    application/x-redhat-package-manager             rpm;
    application/x-sea                                sea;
    application/x-shockwave-flash                    swf;
    application/x-stuffit                            sit;
    application/x-tcl                                tcl tk;
    application/x-x509-ca-cert                       der pem crt;
    application/x-xpinstall                          xpi;
    application/xhtml+xml                            xhtml;
    application/xspf+xml                             xspf;
    application/zip                                  zip;

    application/octet-stream                         bin exe dll;
    application/octet-stream                         deb;
    application/octet-stream                         dmg;
    application/octet-stream                         iso img;
    application/octet-stream                         msi msp msm;

    audio/midi                                       mid midi kar;
    audio/mpeg                                       mp3;
    audio/ogg                                        ogg;
    audio/x-m4a                                      m4a;
    audio/x-realaudio                                ra;

    video/3gpp                                       3gpp 3gp;
    video/mp2t                                       ts;
    video/mp4                                        mp4;
    video/mpeg                                       mpeg mpg;
    video/quicktime                                  mov;
    video/webm                                       webm;
    video/x-flv                                      flv;
    video/x-m4v                                      m4v;
    video/x-mng                                      mng;
    video/x-ms-asf                                   asx asf;
    video/x-ms-wmv                                   wmv;
    video/x-msvideo                                  avi;
}

# configuration file /etc/nginx/stream-conf.d/stream.conf:


server {
    listen unix:/var/run/nginx/connection-closed-server.sock;
    return "";
}

Showing the correct state file. This is the same for both initial deployment of resources and when scaling the tea deployment.

codecov · 2025-05-27T22:26:26Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 86.77%. Comparing base (209f028) to head (f29f0b4).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3406      +/-   ##
==========================================
- Coverage   86.82%   86.77%   -0.06%     
==========================================
  Files         127      127              
  Lines       15011    15020       +9     
  Branches       62       62              
==========================================
  Hits        13034    13034              
- Misses       1828     1835       +7     
- Partials      149      151       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

bjee19 · 2025-05-27T22:27:05Z

Here's some traffic results showing the correct 503 status code:

~/Code/nginx-gateway-fabric/examples/cafe-example main ⇡1281 *6 !2 ❯ curl --resolve cafe.example.com:8080:127.0.0.1 http://cafe.example.com:8080/tea --insecure 
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx</center>
</body>
</html>

Here is a snippet of the nginx logs showing no errors when returning a 503:

2025/05/27 22:05:55 [info] 66#66: *73 client closed connection while waiting for request, client: unix:, server: unix:/var/run/nginx/nginx-plus-api.sock
2025/05/27 22:05:55 [info] 72#72: *25 client unix: closed keepalive connection
2025/05/27 22:05:55 [info] 66#66: *70 client unix: closed keepalive connection
2025/05/27 22:05:55 [info] 66#66: *33 client unix: closed keepalive connection
127.0.0.1 - - [27/May/2025:22:05:56 +0000] "GET /tea HTTP/1.1" 503 190 "-" "curl/8.7.1"
2025/05/27 22:05:56 [info] 66#66: *64 client 127.0.0.1 closed keepalive connection
2025/05/27 22:05:56 [info] 66#66: *67 client 127.0.0.1 closed keepalive connection
2025/05/27 22:05:56 [info] 66#66: *66 client 127.0.0.1 closed keepalive connection
2025/05/27 22:05:56 [info] 66#66: *65 client 127.0.0.1 closed keepalive connection
2025/05/27 22:05:56 [info] 66#66: *63 client 127.0.0.1 closed keepalive connection
2025/05/27 22:05:56 [info] 66#66: *62 client 127.0.0.1 closed keepalive connection
127.0.0.1 - - [27/May/2025:22:05:56 +0000] "GET /tea HTTP/1.1" 503 190 "-" "curl/8.7.1"
2025/05/27 22:05:57 [info] 67#67: *76 client 127.0.0.1 closed keepalive connection
127.0.0.1 - - [27/May/2025:22:05:57 +0000] "GET /tea HTTP/1.1" 503 190 "-" "curl/8.7.1"
2025/05/27 22:05:57 [info] 68#68: *85 client 127.0.0.1 closed keepalive connection
2025/05/27 22:05:58 [info] 69#69: *88 client 127.0.0.1 closed keepalive connection
2025/05/27 22:06:01 [info] 66#66: *80 client 127.0.0.1 closed keepalive connection
2025/05/27 22:06:01 [info] 67#67: *83 client 127.0.0.1 closed keepalive connection

bjee19 · 2025-05-27T22:30:19Z

This is what the dashboard looks like, both after deploying the resources initially and when scaling the tea deployment. I couldn't screenshot it showing the full name, but the name of the server in the tea upstream is the 503 socket.

salonichf5 · 2025-05-27T22:53:51Z

This is what the dashboard looks like, both after deploying the resources initially and when scaling the tea deployment. I couldn't screenshot it showing the full name, but the name of the server in the tea upstream is the 503 socket.

Looks great 🚀

bjee19 requested a review from a team as a code owner May 27, 2025 22:23

github-project-automation bot added this to NGINX Gateway Fabric May 27, 2025

github-project-automation bot moved this to 🆕 New in NGINX Gateway Fabric May 27, 2025

github-actions bot added the bug Something isn't working label May 27, 2025

sjberman reviewed May 27, 2025

View reviewed changes

internal/mode/static/nginx/agent/agent.go Outdated Show resolved Hide resolved

Add 503 status code when there are zero upstream endpoints

94598cf

bjee19 force-pushed the bug/fix-nginx-plus-state-file-generation branch from 3cbe049 to 94598cf Compare May 27, 2025 22:28

bjee19 added 3 commits May 27, 2025 15:47

Add shared types file for exported 503 constant

46fdb71

Change to types package

a9a6f2d

Remove expected error message

f29f0b4

github-actions bot added the tests Pull requests that update tests label May 27, 2025

salonichf5 approved these changes May 27, 2025

View reviewed changes

sjberman approved these changes May 27, 2025

View reviewed changes

bjee19 merged commit 44472e9 into main May 27, 2025
62 of 63 checks passed

bjee19 deleted the bug/fix-nginx-plus-state-file-generation branch May 27, 2025 23:33

github-project-automation bot moved this from 🆕 New to ✅ Done in NGINX Gateway Fabric May 27, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add 503 status code when there are zero upstream endpoints #3406

Add 503 status code when there are zero upstream endpoints #3406

Uh oh!

bjee19 commented May 27, 2025 •

edited

Loading

Uh oh!

Uh oh!

bjee19 commented May 27, 2025

Uh oh!

codecov bot commented May 27, 2025 •

edited

Loading

Uh oh!

bjee19 commented May 27, 2025

Uh oh!

bjee19 commented May 27, 2025

Uh oh!

salonichf5 commented May 27, 2025

Uh oh!

Uh oh!

Uh oh!

Add 503 status code when there are zero upstream endpoints #3406

Add 503 status code when there are zero upstream endpoints #3406

Uh oh!

Conversation

bjee19 commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Checklist

Release notes

Uh oh!

Uh oh!

bjee19 commented May 27, 2025

Uh oh!

codecov bot commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

bjee19 commented May 27, 2025

Uh oh!

bjee19 commented May 27, 2025

Uh oh!

salonichf5 commented May 27, 2025

Uh oh!

Uh oh!

Uh oh!

bjee19 commented May 27, 2025 •

edited

Loading

codecov bot commented May 27, 2025 •

edited

Loading