Working with AI (co-pilot) to uncover hidden haproxy flags for debugging
Combined HAProxy Debug Header Reference
Header Token Description & Why It Matters x-debug-http-response-client-ip%ciIP address of the client making the request. Crucial for tracing source traffic and identifying origin. x-debug-http-response-client-port%cpSource port used by the client. Helps distinguish sessions and trace NAT behavior. x-debug-http-response-request-time%trTimestamp when HAProxy received the request. Useful for correlating logs and measuring latency. x-debug-http-response-frontend%ftName of the frontend that accepted the request. Key for understanding entry point and routing logic. x-debug-http-response-frontend-port%fpPort on which the frontend received the request. Helps validate TLS termination and port-based routing. x-debug-http-response-backend%bBackend name selected by HAProxy. Confirms routing decision based on ACLs or load balancing. x-debug-http-response-backend-port%bpPort used to connect to the backend server. Useful for verifying service exposure and protocol expectations. x-debug-http-response-server%sServer name within the backend. Identifies which specific instance handled the request. x-debug-http-response-timing-tr%TRTime to receive request headers. Indicates client-side latency or slow uploads. x-debug-http-response-timing-tw%TwTime spent in queue. Reveals backend congestion or load imbalance. x-debug-http-response-timing-tc%TcTime to establish TCP connection. Useful for diagnosing network latency or DNS resolution delays. x-debug-http-response-timing-tr%TrTime to receive server response. Helps identify slow applications or backend processing delays. x-debug-http-response-timing-ta%TaTotal time from accept to last byte. Core metric for end-to-end performance. x-debug-http-response-status%STHTTP status code returned by the backend. Essential for error tracking and health checks. x-debug-http-response-bytes%BNumber of bytes read from the server. Indicates payload size and response completeness. x-debug-http-response-conn-count%CCNumber of concurrent connections from the client. Useful for rate limiting or abuse detection. x-debug-http-response-session-count%CSNumber of sessions from the client IP. Helps track stickiness or session reuse. x-debug-http-response-termination%tscTermination state of the session. Reveals clean exits, timeouts, or server failures. x-debug-http-response-conn-stats%ac/%fc/%bc/%sc/%rcConnection counts: active, frontend, backend, server, retries. Vital for load and retry analysis. x-debug-http-response-queue-stats%sq/%bqQueue lengths for server and backend. Indicates overload or slow draining. x-debug-http-response-req-headers%hrRaw request headers from client. Useful for debugging header-based routing or malformed requests. x-debug-http-response-resp-headers%hsRaw response headers from server. Helps validate backend behavior and header rewrites. x-debug-http-response-request-line%{+Q}rFull request line (method, URI, protocol). Crucial for verifying path rewrites and protocol negotiation. x-debug-http-response-ssl-version%sslvSSL/TLS version used. Important for protocol compatibility and handshake debugging. x-debug-http-response-ssl-cipher%sslcCipher used in TLS handshake. Useful for performance tuning and security audits. x-debug-http-after-response-dst%dstDestination IP of the backend server. Confirms routing and DNS resolution. x-debug-http-after-response-dst_port%dst_portDestination port used to reach backend. Validates service exposure and protocol expectations. x-debug-http-after-response-request-line%{+Q}rRepeated for after-response context. Ensures consistency in what was sent to backend.
Leveraging tokens in pfSense/haproxy
http-request add-header X-DEBUG-HTTP-REQUEST-START "====================== DEBUG START Request ==================" http-request set-header Host ab.remote-tech.us http-request add-header X-DEBUG-HTTP-REQUEST-END "========================= DEBUG END Request ===================" http-response add-header X-DEBUG-HTTP-RESPONSE-START "====================== DEBUG START Response ==================" http-response add-header X-DEBUG-HTTP-RESPONSE-CLIENT-IP %ci http-response add-header X-DEBUG-HTTP-RESPONSE-CLIENT-PORT %cp http-response add-header X-DEBUG-HTTP-RESPONSE-REQUEST-TIME [%tr] http-response add-header X-DEBUG-HTTP-RESPONSE-FRONTEND %ft http-response add-header X-DEBUG-HTTP-RESPONSE-FRONTEND-PORT %fp http-response add-header X-DEBUG-HTTP-RESPONSE-BACKEND %b http-response add-header X-DEBUG-HTTP-RESPONSE-BACKEND-PORT %bp http-response add-header X-DEBUG-HTTP-RESPONSE-SERVER %s http-response add-header X-DEBUG-HTTP-RESPONSE-TIMING-TR %TR http-response add-header X-DEBUG-HTTP-RESPONSE-TIMING-Tw %Tw http-response add-header X-DEBUG-HTTP-RESPONSE-TIMING-Tc %Tc http-response add-header X-DEBUG-HTTP-RESPONSE-TIMING-Tr %Tr http-response add-header X-DEBUG-HTTP-RESPONSE-TIMING-Ta %Ta http-response add-header X-DEBUG-HTTP-RESPONSE-STATUS %ST http-response add-header X-DEBUG-HTTP-RESPONSE-BYTES %B http-response add-header X-DEBUG-HTTP-RESPONSE-CONN-COUNT %CC http-response add-header X-DEBUG-HTTP-RESPONSE-SESSION-COUNT %CS http-response add-header X-DEBUG-HTTP-RESPONSE-TERMINATION %tsc http-response add-header X-DEBUG-HTTP-RESPONSE-CONN-STATS %ac/%fc/%bc/%sc/%rc http-response add-header X-DEBUG-HTTP-RESPONSE-QUEUE-STATS %sq/%bq http-response add-header X-DEBUG-HTTP-RESPONSE-REQ-HEADERS %hr http-response add-header X-DEBUG-HTTP-RESPONSE-RESP-HEADERS %hs http-response add-header X-DEBUG-HTTP-RESPONSE-REQUEST-LINE %{+Q}r http-response add-header X-DEBUG-HTTP-RESPONSE-SSL-VERSION %sslv http-response add-header X-DEBUG-HTTP-RESPONSE-SSL-CIPHER %sslc http-response add-header X-DEBUG-HTTP-RESPONSE-END "========================= DEBUG END Response ===================" http-after-response add-header X-DEBUG-HTTP-AFTER-RESPONSE-START "====================== DEBUG START After Response ==================" http-after-response add-header X-DEBUG-HTTP-AFTER-RESPONSE-BACKEND-B %b http-after-response add-header X-DEBUG-HTTP-AFTER-RESPONSE-BYTES %B http-after-response add-header X-DEBUG-HTTP-AFTER-RESPONSE-CLIENT-IP %ci http-after-response add-header X-DEBUG-HTTP-AFTER-RESPONSE-CLIENT-PORT %cp http-after-response add-header X-DEBUG-HTTP-AFTER-RESPONSE-DST %[dst] http-after-response add-header X-DEBUG-HTTP-AFTER-RESPONSE-DST_PORT %[dst_port] http-after-response add-header X-DEBUG-HTTP-AFTER-RESPONSE-FRONTEND-F %f http-after-response add-header X-DEBUG-HTTP-AFTER-RESPONSE-REQUEST-LINE %{+Q}r http-after-response add-header X-DEBUG-HTTP-AFTER-RESPONSE-SERVER-S %s http-after-response add-header X-DEBUG-HTTP-AFTER-RESPONSE-SSL-VERSION %sslv http-after-response add-header X-DEBUG-HTTP-AFTER-RESPONSE-SSL-CIPHER %sslc http-after-response add-header X-DEBUG-HTTP-AFTER-RESPONSE-STATUS %ST http-after-response add-header X-DEBUG-HTTP-AFTER-RESPONSE-TERMINATION %tsc http-after-response add-header X-DEBUG-HTTP-AFTER-RESPONSE-END "========================= DEBUG END After Response ==================="
Testing the rules:
curl -I https://ab.remote-tech.us/site1/phpminiadmin.php HTTP/2 400 server: nginx/1.29.1 date: Sun, 28 Sep 2025 16:42:41 GMT content-type: text/html content-length: 255 x-debug-http-response-start: ====================== DEBUG START Response ================== x-debug-http-response-client-ip: 192.168.50.204 x-debug-http-response-client-port: 55813 x-debug-http-response-request-time: [28/Sep/2025:12:42:41.731] x-debug-http-response-frontend: shared-9443~ x-debug-http-response-frontend-port: 443 x-debug-http-response-backend: ab.remote-tech.us-5006_ipvANY x-debug-http-response-backend-port: 9353 x-debug-http-response-server: site1 x-debug-http-response-timing-tr: 0 x-debug-http-response-timing-tw: 0 x-debug-http-response-timing-tc: 1 x-debug-http-response-timing-tr: 9 x-debug-http-response-timing-ta: -29 x-debug-http-response-status: 400 x-debug-http-response-bytes: 0 x-debug-http-response-conn-count: x-debug-http-response-session-count: x-debug-http-response-termination: ---- x-debug-http-response-conn-stats: 11/11/1/1/0 x-debug-http-response-queue-stats: 0/0 x-debug-http-response-req-headers: x-debug-http-response-resp-headers: x-debug-http-response-request-line: "HEAD https://ab.remote-tech.us/site1/phpminiadmin.php HTTP/2.0" x-debug-http-response-ssl-version: TLSv1.3 x-debug-http-response-ssl-cipher: TLS_AES_256_GCM_SHA384 x-debug-http-response-end: ========================= DEBUG END Response =================== x-debug-http-after-response-start: ====================== DEBUG START After Response ================== x-debug-http-after-response-backend-b: ab.remote-tech.us-5006_ipvANY x-debug-http-after-response-bytes: 0 x-debug-http-after-response-client-ip: 192.168.50.204 x-debug-http-after-response-client-port: 55813 x-debug-http-after-response-dst: 192.168.50.2 x-debug-http-after-response-dst_port: 443 x-debug-http-after-response-frontend-f: shared-9443 x-debug-http-after-response-request-line: "HEAD https://ab.remote-tech.us/site1/phpminiadmin.php HTTP/2.0" x-debug-http-after-response-server-s: site1 x-debug-http-after-response-ssl-version: TLSv1.3 x-debug-http-after-response-ssl-cipher: TLS_AES_256_GCM_SHA384 x-debug-http-after-response-status: 400 x-debug-http-after-response-termination: ---- x-debug-http-after-response-end: ========================= DEBUG END After Response ===================
HAPROXY BACKEND-RESULT
backend ab.remote-tech.us-5006_ipvANY mode http id 158 log global timeout connect 30000 timeout server 30000 retries 3 load-server-state-from-file global http-request add-header X-DEBUG-HTTP-REQUEST-START "====================== DEBUG START Request ==================" http-request set-header Host ab.remote-tech.us http-request add-header X-DEBUG-HTTP-REQUEST-END "========================= DEBUG END Request ===================" http-response add-header X-DEBUG-HTTP-RESPONSE-START "====================== DEBUG START Response ==================" http-response add-header X-DEBUG-HTTP-RESPONSE-CLIENT-IP %ci http-response add-header X-DEBUG-HTTP-RESPONSE-CLIENT-PORT %cp http-response add-header X-DEBUG-HTTP-RESPONSE-REQUEST-TIME [%tr] http-response add-header X-DEBUG-HTTP-RESPONSE-FRONTEND %ft http-response add-header X-DEBUG-HTTP-RESPONSE-FRONTEND-PORT %fp http-response add-header X-DEBUG-HTTP-RESPONSE-BACKEND %b http-response add-header X-DEBUG-HTTP-RESPONSE-BACKEND-PORT %bp http-response add-header X-DEBUG-HTTP-RESPONSE-SERVER %s http-response add-header X-DEBUG-HTTP-RESPONSE-TIMING-TR %TR http-response add-header X-DEBUG-HTTP-RESPONSE-TIMING-Tw %Tw http-response add-header X-DEBUG-HTTP-RESPONSE-TIMING-Tc %Tc http-response add-header X-DEBUG-HTTP-RESPONSE-TIMING-Tr %Tr http-response add-header X-DEBUG-HTTP-RESPONSE-TIMING-Ta %Ta http-response add-header X-DEBUG-HTTP-RESPONSE-STATUS %ST http-response add-header X-DEBUG-HTTP-RESPONSE-BYTES %B http-response add-header X-DEBUG-HTTP-RESPONSE-CONN-COUNT %CC http-response add-header X-DEBUG-HTTP-RESPONSE-SESSION-COUNT %CS http-response add-header X-DEBUG-HTTP-RESPONSE-TERMINATION %tsc http-response add-header X-DEBUG-HTTP-RESPONSE-CONN-STATS %ac/%fc/%bc/%sc/%rc http-response add-header X-DEBUG-HTTP-RESPONSE-QUEUE-STATS %sq/%bq http-response add-header X-DEBUG-HTTP-RESPONSE-REQ-HEADERS %hr http-response add-header X-DEBUG-HTTP-RESPONSE-RESP-HEADERS %hs http-response add-header X-DEBUG-HTTP-RESPONSE-REQUEST-LINE %{+Q}r http-response add-header X-DEBUG-HTTP-RESPONSE-SSL-VERSION %sslv http-response add-header X-DEBUG-HTTP-RESPONSE-SSL-CIPHER %sslc http-response add-header X-DEBUG-HTTP-RESPONSE-END "========================= DEBUG END Response ===================" http-after-response add-header X-DEBUG-HTTP-AFTER-RESPONSE-START "====================== DEBUG START After Response ==================" http-after-response add-header X-DEBUG-HTTP-AFTER-RESPONSE-BACKEND-B %b http-after-response add-header X-DEBUG-HTTP-AFTER-RESPONSE-BYTES %B http-after-response add-header X-DEBUG-HTTP-AFTER-RESPONSE-CLIENT-IP %ci http-after-response add-header X-DEBUG-HTTP-AFTER-RESPONSE-CLIENT-PORT %cp http-after-response add-header X-DEBUG-HTTP-AFTER-RESPONSE-DST %[dst] http-after-response add-header X-DEBUG-HTTP-AFTER-RESPONSE-DST_PORT %[dst_port] http-after-response add-header X-DEBUG-HTTP-AFTER-RESPONSE-FRONTEND-F %f http-after-response add-header X-DEBUG-HTTP-AFTER-RESPONSE-REQUEST-LINE %{+Q}r http-after-response add-header X-DEBUG-HTTP-AFTER-RESPONSE-SERVER-S %s http-after-response add-header X-DEBUG-HTTP-AFTER-RESPONSE-SSL-VERSION %sslv http-after-response add-header X-DEBUG-HTTP-AFTER-RESPONSE-SSL-CIPHER %sslc http-after-response add-header X-DEBUG-HTTP-AFTER-RESPONSE-STATUS %ST http-after-response add-header X-DEBUG-HTTP-AFTER-RESPONSE-TERMINATION %tsc http-after-response add-header X-DEBUG-HTTP-AFTER-RESPONSE-END "========================= DEBUG END After Response ===================" acl site1 var(txn.txnpath) -m beg -i /site1 http-request set-var(txn.txnpath) path http-request set-method HEAD if site1 http-request replace-path site1 %[path] if site1 http-request set-path %[path,regsub(^/site1/,/,g)] if site1 http-request set-uri %[path,regsub(site1,,g)] if site1 use-server site1 if site1 use-server ab.remote-tech.us if !site1 server ab.remote-tech.us 192.168.50.51:5006 id 115 resolvers globalresolvers sni str(ab.remote-tech.us) server site1 192.168.50.204:8080 id 178 resolvers globalresolvers sni str(ab.remote-tech.us)
General
Request URL:https://ab.remote-tech.us/site1/phpminiadmin.php Request Method:GET Status Code:400 Bad Request Remote Address:192.168.50.2:443 Referrer Policy:strict-origin-when-cross-originResponse Headers
content-length:657 content-type:text/html date:Sun, 28 Sep 2025 18:34:00 GMT server:nginx/1.29.1 x-debug-http-after-response-backend-b:ab.remote-tech.us-5006_ipvANY x-debug-http-after-response-bytes:0 x-debug-http-after-response-client-ip:192.168.50.222 x-debug-http-after-response-client-port:51663 x-debug-http-after-response-dst:192.168.50.2 x-debug-http-after-response-dst_port:443 x-debug-http-after-response-end:========================= DEBUG END After Response =================== x-debug-http-after-response-frontend-f:shared-9443 x-debug-http-after-response-request-line:"GET https://ab.remote-tech.us/site1/phpminiadmin.php HTTP/2.0" x-debug-http-after-response-server-s:site1 x-debug-http-after-response-ssl-cipher:TLS_AES_256_GCM_SHA384 x-debug-http-after-response-ssl-version:TLSv1.3 x-debug-http-after-response-start:====================== DEBUG START After Response ================== x-debug-http-after-response-status:400 x-debug-http-after-response-termination:---- x-debug-http-response-acl-check-site1:**SITE1** x-debug-http-response-backend:ab.remote-tech.us-5006_ipvANY x-debug-http-response-backend-port:43382 x-debug-http-response-bytes:0 x-debug-http-response-client-ip:192.168.50.222 x-debug-http-response-client-port:51663 x-debug-http-response-conn-count: x-debug-http-response-conn-stats:18/18/1/1/0 x-debug-http-response-end:========================= DEBUG END Response =================== x-debug-http-response-frontend:shared-9443~ x-debug-http-response-frontend-port:443 x-debug-http-response-queue-stats:0/0 x-debug-http-response-req-headers: x-debug-http-response-request-line:"GET https://ab.remote-tech.us/site1/phpminiadmin.php HTTP/2.0" x-debug-http-response-request-time:[28/Sep/2025:14:34:00.124] x-debug-http-response-resp-headers: x-debug-http-response-server:site1 x-debug-http-response-session-count: x-debug-http-response-ssl-cipher:TLS_AES_256_GCM_SHA384 x-debug-http-response-ssl-version:TLSv1.3 x-debug-http-response-start:====================== DEBUG START Response ================== x-debug-http-response-status:400 x-debug-http-response-termination:---- x-debug-http-response-timing-ta:-2 x-debug-http-response-timing-tc:1 x-debug-http-response-timing-tr:0 x-debug-http-response-timing-tr:11 x-debug-http-response-timing-tw:0Request Headers
:authority: ab.remote-tech.us :method: GET :path: /site1/phpminiadmin.php :scheme: https Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7 accept-encoding:gzip, deflate, br, zstd accept-language:en-US,en;q=0.9 cache-control:max-age=0 sec-ch-ua-platform:"macOS" sec-fetch-dest:document sec-fetch-mode:navigate sec-fetch-site:none sec-fetch-user:?1 upgrade-insecure-requests:1 user-agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36 phpminiadmin.php:1 GET https://ab.remote-tech.us/site1/phpminiadmin.php 400 (Bad Request)