Skip to content

Route - OpenResty HTTP Router

Project Specification

Purpose

route is an OpenResty-based HTTP reverse proxy designed for the abcdesktop platform. It provides secure, JWT-authenticated routing to user desktop containers running in Kubernetes.

Architecture Overview

┌─────────────────┐
│   Client        │
│   (Browser)     │
└────────┬────────┘
         │ HTTP Request + JWT
         ▼
┌─────────────────────────────────────────────────────────┐
│                    route (OpenResty)                    │
│  ┌───────────────────────────────────────────────────┐  │
│  │  get.targetmap.lua                                │  │
│  │  - Extract JWT from request                       │  │
│  │  - Verify JWT signature                           │  │
│  │  - RSA Decrypt payload to get target hostname     │  │
│  │  - Cache result in shared dict (10 min)           │  │
│  └───────────────────────────────────────────────────┘  │
└────────┬────────────────────────────────────────────────┘
         │ $target variable resolved
         ▼
┌─────────────────────────────────────────────────────────┐
│     Pods (Kubernetes)                                   │
│     ┌──────────────────────┐                            │
│     │      Desktop Pod     │                            │
│     │  service TCP port    │                            │
│     └──────────────────────┘                            │
└─────────────────────────────────────────────────────────┘

Detailed Analysis: get.targetmap.lua

Role

This Lua script is the core authentication and routing engine. It executes during the rewrite phase for protected endpoints and resolves the $target nginx variable used in proxy_pass directives.

Execution Flow

1. Extract JWT token from request
         │
         ▼
2. Check targetmap cache
   ┌──────────────┐
   │ Cache Hit?   │──Yes──→ Set $target → Continue request
   └──────────────┘
         │ No
         ▼
3. Read public key from env or cache
         │
         ▼
4. Verify JWT signature
         │
         ▼
5. Decrypt payload.hash using dedicated RSA private key
         │
         ▼
6. Cache result in targetmap (TTL: min(exp-now, 600s))
         │
         ▼
7. Set $target = decrypted hostname

Code Walkthrough

1. Token Extraction

local jwt_token = ngx.var.jwt_token

The token is sourced from: - Query parameter: ?jwt_token=xxx - Header: Authorization: Bearer <token> - Header: AbcAuthorization: <token>

The no_bearer() function strips the "Bearer " prefix if present.

2. Response Helper

function ngxexitresponse(status, msg)
    ngx.status = status
    ngx.log(ngx.ERR, msg)
    ngx.say(msg)
    ngx.exit(ngx.HTTP_OK)
end

Returns HTTP error responses with logging. Uses ngx.exit(ngx.HTTP_OK) after setting ngx.status to properly interrupt execution.

3. RSA Decryption

local function decrypt(msg, private_key)
    local rsa = require "resty.rsa"
    local priv, err = rsa:new({ private_key = private_key })
    local crypto = ngx.decode_base64(msg)
    local decrypted, err = priv:decrypt(crypto)
    return decrypted
end

The JWT payload contains a hash field that is: 1. Base64-encoded 2. RSA-encrypted with the server's public key 3. Contains the target pod hostname

4. Cache Lookup

local target = ngx.shared.targetmap:get(jwt_token)

The targetmap shared dictionary (8MB) caches resolved targets to avoid repeated JWT processing.

5. JWT Verification

local jwt = require "resty.jwt"
local jwt_secret = ngx.shared.rsakeymap:get('jwt_desktop_signing_public_key')
  or readfile(os.getenv("JWT_DESKTOP_SIGNING_PUBLIC_KEY"))

local jwt_obj = jwt:verify(jwt_secret, jwt_token)

Verifies the JWT signature using the configured RSA public key.

6. Payload Decryption & Caching

local private_key = ngx.shared.rsakeymap:get('jwt_desktop_payload_private_key')
  or readfile(os.getenv("JWT_DESKTOP_PAYLOAD_PRIVATE_KEY"))

target = decrypt(payload.hash, private_key)

-- Cache with TTL
local expire_value = payload.exp - ngx.time()
if expire_value > 600 then expire_value = 600 end
if expire_value > 1 then
    ngx.shared.targetmap:set(jwt_token, target, expire_value)
end

Cache TTL Logic:

  • Uses JWT exp claim for expiration
  • Capped at 600 seconds (10 minutes) maximum
  • Minimum 1 second required to cache
  • Prevents caching of expired/invalid tokens

7. Target Assignment

ngx.var.target = target

Sets the nginx variable used by proxy_pass http://$target:$port/

Endpoint Routing Table

Service Endpoints

Endpoint Service Variable Default Port Protocol
/spawner $spawner_service_tcp_port 29786 HTTP
/terminals $xterm_tcp_port 29781 WebSocket
/terminals/{id}/size $xterm_tcp_port 29781 HTTP
/filer $file_service_tcp_port 29783 HTTP (8GB max)
/printerfiler $printerfile_service_tcp_port 29782 HTTP
/websockify $ws_tcp_bridge_tcp_port 6081 WebSocket
/signalling $signalling_service_tcp_port 29787 WebSocket
/broadcast $broadcast_tcp_port 29784 WebSocket
/sound $sound_service_tcp_port 29788 WebSocket
/microphone $microphone_service_tcp_port 29789 WebSocket
/gamepad $gamepad_service_tcp_port 29790 WebSocket
/snapshot $snapshot_service_tcp_port 29785 HTTP
/console console upstream - HTTP

pyos Endpoints

Pattern proxy_pass backend: http://$pyos_fqdn:$pyos_service_port:

  • /API/manager/image with client_max_body_size 16m;
  • /API/composer/launchdesktop with proxy_read_timeout 600s;
  • /API

Utility Endpoints

Endpoint Description
/healthz Health check, returns "OK"
/node Returns NODE_NAME environment variable
/.well-known/* Let's Encrypt ACME challenges
/speedtest Raw data throughput testing (no gzip)
/ Default website content

Configuration Files

nginx.conf - Main Configuration

Key directives:

  • worker_processes auto - Scales to available CPUs
  • daemon off - Required for container foreground execution

  • env directives - Exposes environment variables to nginx

  • Lua shared dictionaries:

  • rsakeymap (1MB) - RSA key content cache
  • rsafilenamekeymap (1MB) - Key filename mappings
  • targetmap (8MB) - JWT → target resolution cache

Security settings:

  • server_tokens off - Hides nginx version
  • more_clear_headers Server - Removes Server header
  • TLS 1.2+ with strong cipher suite

sites-enabled/routehttp.conf - Server Block

Initialization:

init_by_lua_block {
  pyos_fqdn = os.getenv("PYOS_FQDN") or "pyos"
  pyos_service_port = os.getenv("PYOS_SERVICE_PORT") or "8000"
}

Service port variables:

set $ws_tcp_bridge_tcp_port             6081;
set $xterm_tcp_port                     29781;
set $file_service_tcp_port              29783;
# ... etc

Listening:

  • Port 80 (HTTP)
  • Port 443 (HTTPS, commented out by default)

proxy.conf - Standard Proxy Headers

proxy_set_header User-Agent            $http_user_agent;
proxy_set_header Origin                $http_origin;
proxy_set_header X-Real-IP             $remote_addr;
proxy_set_header X-Forwarded-For       $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host      $server_name;
proxy_set_header Accept-Language       $http_accept_language;
proxy_set_header Host                  $host;

ws.conf - WebSocket Configuration

sendfile off;
tcp_nopush off;
proxy_buffering off;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";

Docker Images

Default Image (Ubuntu Jammy)

FROM openresty/openresty:jammy

Packages installed:

  • lua-resty-jwt
  • lua-resty-string
  • lua-cjson
  • lua-resty-rsa
  • lua-resty-dns

Alpine Image

FROM openresty/openresty:alpine

Additional tools:

  • certbot (Let's Encrypt)

Build approach:

  • Installs build dependencies, compiles LuaRocks packages, removes build tools

Environment Variables

Variable Required Default Description
NODE_NAME No - Node identifier for /node endpoint
JWT_DESKTOP_PAYLOAD_PRIVATE_KEY Yes - Path to RSA private key for payload decryption
JWT_DESKTOP_SIGNING_PUBLIC_KEY Yes - Path to RSA public key for signature verification
PYOS_SERVICE_PORT No 8000 PyOS backend port
PYOS_FQDN No pyos PyOS backend hostname
NAMESERVER_RESOLVER No From /etc/resolv.conf DNS resolver for nginx

Security Considerations

  1. JWT Validation Chain
  2. Signature verification with RSA public key
  3. Payload decryption with RSA private key

  4. Expiration validation via cache TTL

  5. Information Disclosure Prevention

  6. Server version hidden

  7. Error messages logged but minimal in response

  8. Resource Limits

  9. Upload limits: 8MB (spawner), 8GB (filer)
  10. Cache TTL capped at 10 minutes

  11. Shared dictionary sizes bounded

  12. WebSocket Security

  13. Proper Upgrade/Connection headers
  14. Buffering disabled for real-time communication

Logging

  • Access log: /var/log/nginx/access.log
  • Error log: /var/log/nginx/error.log
  • JWT errors: Logged at ngx.ERR level
  • Cache hits: Logged at ngx.NOTICE level with TTL and target

Rebuild the container images

  • from openresty/openresty alpine
git clone -b 4.4 https://github.com/abcdesktopio/route.git
cd route
REPO=abcdesdesktop
docker build -t $REPO/route:4.4 -f Dockerfile.alpine --build-arg BASE_IMAGE_RELEASE=alpine --build-arg BASE_IMAGE=openresty/openresty .