Culvert

command module
v0.0.322 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 20, 2026 License: MIT Imports: 82 Imported by: 0

README

Culvert

Culvert

Enterprise-grade open-source forward proxy
HTTP · HTTPS · SOCKS5 · WebSocket
Single binary · Zero dependencies · Written in Go

CI CodeQL Security Gate Dependency Obituary Post-Quantum Ready Go Report Card


Why Culvert?

Most forward proxies force you to choose: commercial appliance with vendor lock-in, or a minimal open-source tool you have to build around. Culvert is neither. It ships as a single Go binary with everything built in - SSL inspection, identity-aware policy, antivirus scanning, threat feeds, a full admin UI, and enterprise auth (OIDC, SAML, LDAP) - all without plugins, agents, or runtime dependencies.

Deploy it with docker-compose up -d and you get a production-ready proxy with zero-trust default-deny, real-time dashboards, Prometheus metrics, and a 10-check CI security pipeline. Scale it out with the built-in gRPC Control Plane for multi-node deployments. Extend it with the plugin API when you need custom logic.


Features

Proxy & Protocols

  • HTTP / HTTPS forward proxy with full CONNECT tunnel support
  • SSL/TLS inspection - on-the-fly MITM certs (ECDSA P-256), per-host bypass, LRU cert cache (10k entries, 1h TTL)
  • SOCKS5 proxy (RFC 1928/1929) with username/password auth
  • WebSocket tunneling through CONNECT
  • PAC file auto-generation for browser auto-configuration
  • Upstream proxy chaining with round-robin, health checks, and automatic failover
  • Circuit breaker - stops forwarding to hung upstream proxies after consecutive failures

Policy Engine (Zero Trust)

  • Default deny - unmatched traffic is blocked
  • Priority-based rules - first match wins across 8 condition types:
    • Source IP / CIDR
    • Authenticated identity
    • IdP group membership
    • Auth source (OIDC, SAML, LDAP, local)
    • Destination FQDN (exact + wildcard)
    • URL category (Social, Streaming, Gambling, News, Malicious, Adult, ...)
    • Destination country (GeoIP, fail-closed on cache miss)
    • Time schedule (day of week + time window + timezone)
  • Actions: Allow, Drop, Block Page, Redirect
  • Per-rule SSL action: Inspect (full MITM) or Bypass (transparent tunnel)
  • Policy conflict detection - warns on overlapping rules at same priority
  • Per-rule Prometheus metrics with cardinality cap

Authentication & Identity

  • Local auth with bcrypt hashes and first-run setup wizard
  • OIDC Authorization Code + PKCE (Okta, Azure AD, Google, Auth0, Keycloak)
  • SAML 2.0 SP-initiated SSO (Okta, Azure AD, ADFS)
  • LDAP bind + search with group resolution (Active Directory, OpenLDAP, FreeIPA)
  • Multi-IdP - simultaneous providers with email-domain routing
  • TOTP 2FA (RFC 6238, inline stdlib implementation) with backup codes for admin accounts
  • RBAC - admin / operator / viewer roles

Content Security

  • ClamAV antivirus (INSTREAM protocol, concurrency-limited scanning, auto-detection)
  • YARA rules - pure-Go engine (no libyara), runtime reload, ReDoS-safe (5s timeout)
  • Threat feeds - URLhaus + OpenPhish with hourly sync, dynamic domain allowlist for hosting platforms (GitHub, Google Drive, etc.)
  • DPI - regex content scanning on decrypted HTTPS responses
  • File-type blocking - 5 named profiles (Executables, Archives, Documents, Media, Strict)
  • Domain blocklist with wildcard matching and allow-list mode
  • URL category database (UT1) with background sync
  • SHA-256 scan cache with configurable size + TTL
  • Remote scan sidecar - process-isolated ClamAV/YARA scanning via remote microservice
  • Blocklist feed syncer - auto-sync domain blocklists from remote URLs

Admin Web UI

Single-page application with real-time updates:

Panel Description
Dashboard Live stats, timeseries chart, top domains, country traffic map
Live Feed Real-time request log with per-status badges, filtering, CSV/JSON export
Blocklist Domain entries with wildcards; allow-list / deny-list toggle
Policy Visual PBAC rule editor with all 8 condition types
Policy Tester Dry-run evaluation against any host/user/IP
Security IP filter, rate limiting, connection limits, block page editor, scanner status
File Block File-type blocking profile selector
Rewrite Per-host header rewrite rules (request + response)
Upstream Proxies Parent proxy chaining with health checks and circuit breaker
IdP Providers Step-by-step wizard for OIDC, SAML, LDAP
SSL / TLS Root CA viewer, custom TLS upload, SSL bypass patterns
CA Management CA lifecycle, PEM download, cache stats, OCSP toggle, force rotation
Cluster Nodes Multi-node role display, DP node metrics, enrollment command generator
Node Groups Label-based node grouping with geo-aware auto-labeling
Bandwidth / QoS Per-group bandwidth policies with token bucket rate limiting
Config Versions Auto-snapshot history, side-by-side diff, one-click rollback
PAC PAC file generator with custom exclusions
Audit Log Tamper-evident JSONL trail of all admin actions
Diagnostics Operator contract — storage, policy load, root CA, session HMAC, CDR, cluster TLS posture, updater URL, config-version health, risky-but-allowed warnings
Governance Read-only control-plane visibility — route inventory, C2/C2c/C4 counters, derived health, parity-test catalog (admin-only)
Users User management with RBAC role assignment
Settings Session timeout, UI access control, syslog, config export/import

Observability & SIEM

  • Prometheus metrics - requests, blocks, auth failures, AV scans, YARA matches, bytes transferred, latency histogram, per-rule counters
  • Real-time SSE dashboard feed
  • Structured logging - text or JSON with req_id, identity, rule, action fields
  • Rotating log files with configurable size threshold
  • Syslog forwarding (UDP/TCP, RFC 3164 + RFC 5424) for Splunk, Elastic, QRadar
  • JSONL audit trail with actor identity enrichment
  • Webhook alerts - HMAC-SHA256 signed notifications for threats, blocks, lockouts
  • Request tracing - auto-generated X-Request-ID for end-to-end correlation

Self-Update System

  • Docker update sidecar - lightweight Go service with Docker socket access, triggered from the GUI
  • One-click update - pull, recreate, health check, automatic rollback on failure (SSE progress stream)
  • Rollback - previous container preserved for 1 hour (configurable); one-click restore
  • Self-update - updater can update itself via reaper container pattern
  • Air-gapped mode - load images from tarball when no registry access
  • Concurrent operation guard - prevents double-click races on update/rollback

Resilience & Operations

  • Hot config reload - SIGHUP reloads blocklist, policy, rewrite, rate limit, upstream pool
  • Graceful shutdown - lifecycle context + 15s drain window for active tunnels
  • CA auto-rotation - daily expiry check, auto-rotates 30 days before expiry
  • OCSP/CRL checking on upstream TLS certificates (fail-closed)
  • Per-IP connection limiting (configurable, default 1024) with runtime admin API
  • Brute-force lockout - 5 failures triggers 15-min IP + user lock
  • Admin API rate limiting - 60 req/min per IP on mutating endpoints
  • Atomic file writes for CA bundle and config persistence
  • PBKDF2 600k iterations (NIST SP 800-132 2024) for CA key encryption
  • Post-Quantum Cryptography - ML-KEM-768 hybrid key exchange (Go 1.25), protects against "Harvest Now, Decrypt Later" attacks
  • Password complexity - enforces 8+ characters, mixed case, digit requirement
  • Log levels - runtime DEBUG/INFO/WARN/ERROR with admin API control

Distributed Architecture

  • Control Plane / Data Plane - gRPC config sync with mTLS, per-node metrics aggregation
  • Cluster dashboard - connected node list, health, request counts, enrollment wizard
  • Node groups - label-based selectors with auto GeoIP labeling on enrollment
  • Bandwidth / QoS - per-group token bucket rate limiting with admin UI
  • Config versioning - automatic snapshots on every mutation, side-by-side diff, one-click rollback (50 versions)
  • Rolling upgrades - orchestrated cluster updates with drain, canary, HA sync
  • PAC / threat feed / secrets sync - full config snapshot pushed to data plane nodes
  • Bootstrap generator - one-line curl|bash enrollment scripts and docker-compose templates for DP nodes
  • Exponential backoff on connection failures (2s–60s)
  • Client mTLS for upstream proxy authentication
  • See Deployment Guide for single-node, multi-node, and upstream chaining setup

Extensibility

  • Plugin API - Middleware interface for custom request/response inspection
  • HSM/KMS integration - KeyProvider interface (AWS KMS, Azure Key Vault, PKCS#11)

Sizing Guide

Culvert is lightweight by design - a single Go binary with no runtime dependencies. Resource requirements scale with traffic volume, SSL inspection depth, and whether antivirus scanning is enabled.

All benchmarks below are based on x86_64 (AMD64) processors. ARM64 (AWS Graviton, Apple Silicon) typically achieves 10-20% better throughput per vCPU due to Go's efficient ARM code generation. If deploying on ARM, you can conservatively use the same numbers or reduce vCPU count by one tier.

Deployment Profiles

Profile Users Requests/sec Max throughput Concurrent conns SSL Inspection ClamAV YARA
Home Lab 1-5 < 50 ~100 Mbps < 100 Optional Off Off
Small Office 10-50 50-500 ~500 Mbps 100-1000 Recommended Optional Optional
Enterprise (single node) 100-500 500-2000 ~1 Gbps 1000-5000 Yes Yes Yes
Enterprise (cluster) 500-5000+ 2000-10000+ ~10 Gbps (multi-DP) 5000+ Yes Yes Yes

When to cluster: A single node with SSL inspection + ClamAV handles ~500 concurrent users comfortably. Beyond 500 users or 1 Gbps sustained throughput, deploy a Control Plane + Data Plane cluster and add DP nodes horizontally.

Resource Requirements

Resource Home Lab Small Office Enterprise
CPU 1 vCPU 2 vCPU 4+ vCPU
RAM (proxy only) 128 MB 256 MB 512 MB - 1 GB
RAM (with ClamAV) 512 MB 1 GB 2 GB
RAM per connection ~10 KB idle / ~138 KB active same same
Storage (base) 50 MB (binary + config) 50 MB 50 MB
Storage (ClamAV DB) - 300 MB 300 MB
Storage (GeoIP DB) 5 MB 5 MB 5 MB
Storage (UT1 category feed) 150 MB 150 MB 150 MB
Storage (logs) 100 MB - 1 GB 1 - 5 GB 5 - 50 GB
Storage (admin settings) < 10 KB < 10 KB < 10 KB
Disk type Any (HDD OK) SSD recommended SSD required

Connection Memory Breakdown

Each proxied connection consumes memory proportional to its state:

State RAM per connection Notes
Idle (keepalive) ~10 KB Go net/http conn + buffers
Active HTTP forward ~70 KB Request/response buffers (32 KB each)
SSL-inspected tunnel ~138 KB 2x 32 KB relay buffers (pooled) + TLS state
Body scanning (ClamAV/YARA) +64 KB - 4 MB Buffered body up to scan limit, then freed
WebSocket relay ~138 KB Same as tunnel (bidirectional relay)

At 5000 concurrent SSL-inspected connections: ~670 MB for connection buffers alone. Plan RAM accordingly for high-concurrency environments.

Notes

  • ClamAV is the largest resource consumer. It downloads ~300 MB of virus signatures on first boot and keeps them in memory (~500 MB RSS). If you don't need antivirus scanning, disable it to save ~500 MB RAM. ClamAV signature reloads (freshclam updates, typically every 4 hours) cause a brief CPU spike (~2-5 seconds) but do not block proxy traffic - the reload happens in the ClamAV sidecar process, not in Culvert itself. No over-provisioning needed.
  • SSL inspection adds ~1 KB RAM per cached leaf certificate (LRU cache, 10K max = ~10 MB). CPU impact is ~0.5ms per new TLS handshake using ECDSA P-256 signing (Culvert's default). RSA 2048 would be ~5x slower per signing operation, but Culvert exclusively uses ECDSA P-256 for its internal CA - no RSA path exists. Cached certificates (cache hit) have zero signing overhead.
  • Log rotation is automatic. Request logs rotate at 100 MB (configurable via -request-log-max-mb), audit logs at 50 MB, system logs at 50 MB. Storage I/O: log writes are append-only and sequential - HDD is adequate for Home Lab. For Small Office+ with SSL inspection (high request volume), SSD is recommended to avoid I/O wait impacting proxy latency.
  • Threat feeds (URLhaus + OpenPhish) add ~5-20 MB RAM depending on feed size.
  • UT1 category feed (University of Toulouse) downloads a ~50 MB tarball on first sync, stored in BadgerDB at ~150 MB on disk + ~50-100 MB RAM for the index. Provides millions of categorized domains (Adult, Gambling, Malicious, etc.) auto-synced hourly. The SaaS category feed (AI, Marketing, etc.) adds negligible storage (< 100 KB JSON).
  • Prometheus metrics are stateless - scraped externally, no local storage.
  • Post-quantum (ML-KEM-768) adds ~1 KB to initial TLS handshakes. No ongoing RAM/CPU impact.
  • Docker image size: ~30 MB (distroless base + static Go binary).

Minimum Requirements

For a functional deployment with all features enabled:

2 vCPU | 1.5 GB RAM | 2.5 GB disk (SSD recommended)

For proxy-only (no AV, no SSL inspection):

1 vCPU | 128 MB RAM | 100 MB disk (any)

Cluster Deployments (Control Plane / Data Plane)

Component CPU RAM Storage Notes
Control Plane 2 vCPU 512 MB 500 MB No proxy traffic - config sync, enrollment, dashboard only
Control Plane (HA pair) 2 vCPU each 512 MB each 500 MB each Leader + standby with automatic failover
Data Plane node 2 vCPU 1 GB 1 GB Handles proxy traffic, receives config from CP
Data Plane + ClamAV 2 vCPU 2 GB 1.5 GB Add ~1 GB RAM + 300 MB disk for AV

Scale reference:

Setup Nodes Total resources Throughput Handles
Small cluster 1 CP + 2 DP 6 vCPU, 2.5 GB RAM ~2 Gbps ~1000 concurrent users
Medium cluster 1 CP (HA) + 5 DP 12 vCPU, 6 GB RAM ~5 Gbps ~5000 concurrent users
Large cluster 1 CP (HA) + 10 DP 22 vCPU, 11 GB RAM ~10 Gbps ~10000+ concurrent users
  • CP is lightweight - it only serves gRPC config sync, node enrollment, and the admin dashboard. No proxy traffic flows through it.
  • DP nodes are stateless - they receive their entire config from the CP on connect. Lose a DP, spin up a new one, it auto-enrolls and gets the full config in seconds.
  • Bandwidth/QoS policies are enforced per-DP, so rate limits scale linearly with node count.
  • Network: CP to DP communication uses gRPC over mTLS. Typical bandwidth: < 1 KB/s per node (config sync + heartbeat every 30s).
  • Horizontal scaling: each DP node adds ~1 Gbps throughput capacity (with SSL inspection). Scale is linear - no shared state between DP nodes.

Quick Start

Works on Ubuntu, Debian, RHEL, CentOS, Rocky, Alma, Fedora, Amazon Linux, and Arch. Runs anywhere Linux runs: AWS EC2, Azure VM, GCP Compute, DigitalOcean, Hetzner, bare metal. Installs Docker, clones the repo, pulls the pre-built images and starts everything:

curl -fsSL https://raw.githubusercontent.com/KidCarmi/Culvert/main/scripts/install.sh | bash

The script handles all Docker installation quirks (snap removal, compose v2, distro packages) so you don't have to.

Docker (manual)

git clone https://github.com/KidCarmi/Culvert
cd Culvert
docker build -t culvert/proxy:pinned .   # seed the local-only image tag the compose file resolves
docker compose up -d

No configuration required - the setup wizard creates your admin account on first visit.

pull access denied for culvert/proxy? The compose file resolves the local-only tag culvert/proxy:pinned (a security measure — the proxy image is pinned at the sudo boundary, never pulled by name from a registry). Seed it with the docker build line above, or: docker pull ghcr.io/kidcarmi/culvert:latest && docker tag ghcr.io/kidcarmi/culvert:latest culvert/proxy:pinned

Endpoint URL Notes
HTTP/HTTPS Proxy http://localhost:8080 Configure browser/PAC to point here
SOCKS5 Proxy socks5://localhost:1080 Disabled by default, enable via config
PAC File http://localhost:8080/proxy.pac Auto-config for browsers
Admin Web UI https://localhost:9090 Accept the self-signed cert on first visit
Health Check http://localhost:8080/health {"status":"ok",…}
Prometheus Metrics http://localhost:8080/metrics Optional bearer token protection
# Verify it works
curl http://localhost:8080/health
curl -x http://localhost:8080 https://example.com

After it's running

Three things to do once the containers are up:

  1. Setup wizard — open https://<host>:9090 in a browser, accept the self-signed cert, and create the first admin account. This step is required before any other API call.
  2. Verify readiness — confirm Culvert is fit to take traffic:
    curl http://<host>:8080/ready
    
    Returns 200 with {"status":"ready", "checks":{...}} when ready, 503 with "status":"not_ready" when a gating check fails. See docs/OPERATIONS.md for the full checks-map reference.
  3. Open Diagnostics — Admin UI → Infrastructure → Diagnostics. The page surfaces the operator contract: storage path, policy load, root CA, session HMAC, CDR, cluster TLS posture, updater URL, config-version health, and any active risky-but-allowed warnings (cluster-insecure, unauth mode). Resolve any fail rows before exposing the proxy to clients.

Backup & restore. For the supported Docker Compose backup, restore, and cleanup commands, see docs/operator/docker-compose-backup-restore.md.

With custom config
cp config.example.yaml config.yaml   # edit as needed
# Uncomment the config.yaml volume mount in docker-compose.yml, then:
docker compose up -d
With monitoring stack (Prometheus + Grafana)
docker compose -f docker-compose.yml -f docker-compose.monitoring.yml up -d
# Grafana → http://localhost:3000  (admin / culvert)

The Culvert Overview dashboard (12 panels: traffic, latency, security blocks, top policy rules) is auto-provisioned from deploy/grafana/dashboards/culvert-overview.json - no manual import needed.

Binary

# Download the latest release, then:
./culvert                                  # proxy :8080, admin UI :9090
./culvert -port 3128 -socks5-port 1080
./culvert -config config.yaml

Proxy Usage

HTTP / HTTPS

curl -x http://localhost:8080 https://example.com

# With credentials
curl -x http://alice:secret@localhost:8080 https://example.com

# Via environment variable
export http_proxy=http://localhost:8080
export https_proxy=http://localhost:8080

# PAC file (auto-configure browsers)
# Point your browser to: http://localhost:8080/proxy.pac

SOCKS5

curl --proxy socks5://localhost:1080 https://example.com

# SSH tunneling through SOCKS5
ssh -o ProxyCommand="nc -X 5 -x localhost:1080 %h %p" user@remote

SSL Inspection

When SSL inspection is enabled, import the Root CA into your browser/OS trust store:

# Download CA from Admin UI → Certificates, or:
curl -k https://localhost:9090/api/ca-cert > culvert-ca.crt

# Linux
sudo cp culvert-ca.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates

# macOS
sudo security add-trusted-cert -d -r trustRoot \
  -k /Library/Keychains/System.keychain culvert-ca.crt

# Windows (PowerShell as Admin)
Import-Certificate -FilePath culvert-ca.crt -CertStoreLocation Cert:\LocalMachine\Root

Configuration

Culvert works out of the box with sensible defaults. All settings can be managed through the Admin Web UI, a YAML config file, CLI flags, or a combination. See config.example.yaml for full documentation of all 70+ fields.

config.yaml (optional)

proxy:
  port: 8080
  ui_port: 9090
  socks5_port: 1080      # 0 = disabled

default_action: block     # allow | block | drop

auth:
  ldap:
    url: ldaps://ldap.corp.com:636
    bind_dn: "cn=svc-culvert,ou=Services,dc=corp,dc=com"
    base_dn: "ou=Users,dc=corp,dc=com"
    required_group: "proxy-users"

security:
  ip_filter_mode: allow   # allow | block | "" (off)
  ip_list:
    - 192.168.1.0/24
  rate_limit: 60          # requests/min per IP
  max_conns_per_ip: 256

upstream:
  proxies:
    - url: http://parent-proxy:3128
      health_interval: 30s
  circuit_breaker:
    threshold: 5
    timeout: 30s

rewrite:
  - host: "*.internal.example.com"
    req_set:
      X-Forwarded-By: Culvert
    resp_remove:
      - Server
      - X-Powered-By

CLI Flags

Core:
  -port int              Proxy listening port (default 8080)
  -ui-port int           Admin Web UI port (default 9090)
  -socks5-port int       SOCKS5 proxy port (0 = disabled)
  -config string         Path to config.yaml

TLS:
  -ca-path string        Root CA bundle persistence path (/data/ca.bundle)
  -tls-cert string       Custom TLS certificate for Web UI
  -tls-key string        Custom TLS key for Web UI
  -ui-no-tls             Serve Web UI over plain HTTP (not recommended)

Auth & Access:
  -ui-users-file string  Persistent admin user database (/data/ui_users.json)
  -ui-allow-ip string    Comma-separated CIDRs allowed to access the Web UI
  -session-timeout int   Admin session lifetime in hours (default 8)

Rules & Filtering:
  -blocklist string      Domain/IP blocklist file path
  -policy string         Policy rules JSON file path
  -geoip-db string       MaxMind GeoLite2-Country.mmdb path

Security Scanning:
  -clamav-addr string    ClamAV address - tcp:host:port or unix:/path/to/clamd.sock
  -yara-rules-dir string Directory containing .yar / .yara rule files
  -threat-feed-db string Threat feed local database path

Logging:
  -logfile string        Request log file (rotated at -log-max-mb)
  -log-max-mb int        Log rotation threshold in MB (default 50)
  -audit-log string      Persistent JSONL audit log path
  -syslog string         Remote syslog - udp://host:514 or tcp://host:601

Metrics:
  -metrics-token string  Bearer token protecting /metrics (empty = open)
  -rate-limit int        Max requests/min per source IP (0 = off)

Distributed (Control Plane):
  -cp-grpc-addr string   Control Plane gRPC listen address (e.g. :50051)
  -cp-grpc-cert string   Control Plane gRPC TLS certificate
  -cp-grpc-key string    Control Plane gRPC TLS key
  -cp-grpc-ca string     Control Plane gRPC CA for mTLS client validation

Distributed (Data Plane):
  -dp-cp-addr string     ControlPlane gRPC address to connect to (e.g. cp.corp:50051)
  -dp-node-id string     Unique node identifier (default: hostname)
  -dp-cert string        Data Plane gRPC client TLS certificate
  -dp-key string         Data Plane gRPC client TLS key
  -dp-ca string          Data Plane gRPC CA certificate

See docs/deployment-guide.md for complete single-node, multi-node, and upstream chaining examples.

Environment Variables

Variable Description
CULVERT_CA_PASSPHRASE CA private key encryption passphrase (required for SSL inspection)
CULVERT_C2_ENFORCE C2 metadata-driven admin RBAC enforcement mode. Default = enforce (fail-closed). Set to false/0/no/off to revert to shadow (log-only) mode without rebuild. Read once at startup.

Prometheus Metrics

Available at GET http://localhost:8080/metrics:

Metric Type Description
culvert_requests_total counter All proxy requests
culvert_requests_allowed counter Forwarded requests
culvert_requests_blocked counter Blocked requests (all reasons)
culvert_requests_auth_fail counter Authentication failures
culvert_bytes_sent_total counter Total bytes sent to clients
culvert_bytes_recv_total counter Total bytes received from upstream
culvert_request_duration_seconds histogram Request latency (11 buckets, 5ms–10s)
culvert_rule_hits_total{rule="..."} counter Per-rule hit counter
culvert_av_scans_total counter ClamAV scans performed
culvert_av_detections_total counter Malware detections
culvert_yara_matches_total counter YARA rule matches
culvert_threat_feed_blocks_total counter Threat feed blocks
culvert_blocklist_size gauge Blocklist entry count
culvert_policy_rules gauge Active PBAC rule count
culvert_uptime_seconds gauge Proxy uptime

Plugin API

Implement the Middleware interface to add custom request inspection:

package main

import "net/http"

type MyPlugin struct{}

func (p *MyPlugin) Name() string { return "my-plugin" }

func (p *MyPlugin) OnRequest(clientIP, method, host string) Decision {
    if host == "ads.example.com" {
        return DecisionBlock
    }
    return DecisionAllow
}

func (p *MyPlugin) OnResponse(resp *http.Response) {
    resp.Header.Del("Server")
}

func init() { RegisterPlugin(&MyPlugin{}) }

Plugins run before every other check and can short-circuit the chain.


Security

Culvert follows a defence-in-depth approach:

Area Implementation
Zero Trust Default-deny policy engine; unmatched traffic is blocked
SSRF prevention isPrivateHost() resolves DNS and rejects private/loopback IPs before every outbound dial
Log injection (CWE-117) sanitizeLog() strips \n, \r, \t; %q format verb; X-Request-ID sanitized at source
Open redirect isSafeRedirectURL() validates scheme + non-private host
Brute-force IP + user lockout after 5 failures (15 min cooldown)
Admin API rate limiting 60 req/min per IP on mutating endpoints
Slowloris 60s read deadline on SSL-inspected connections
Session security HMAC-SHA256 signed cookies with per-session 128-bit Jti; dynamic Secure flag; fixation prevention; revocation list with disk persistence and gRPC gossip
Admin RBAC defense-in-depth Metadata-driven C2 enforcement layer in addition to per-handler requireRole; report-only audit-completion (C2c) and role-divergence (C4) detectors; governance health surface at /api/governance/control-plane
CA key protection AES-256-GCM + PBKDF2-SHA256 (600k iterations) at rest
OCSP/CRL Upstream certificate revocation checking (fail-closed)
Hop-by-hop RFC 7230 compliant - parses Connection header for dynamic names
GeoIP Fail-closed on cache miss (unknown country = no match)
Header scrubbing Strips private IPs from X-Forwarded-For, removes X-User-Identity
Post-Quantum (PQC) ML-KEM-768 hybrid key exchange via Go 1.25 - quantum-resistant on all TLS connections

Post-Quantum Cryptography (PQC)

Culvert is quantum-resistant by default. Go 1.25's crypto/tls automatically negotiates ML-KEM-768 (formerly Kyber) hybrid key exchange on all TLS connections when the peer supports it. This protects against "Harvest Now, Decrypt Later" attacks where an adversary records encrypted traffic today and decrypts it with a future quantum computer.

Connection PQC Key Exchange Notes
Browser to Admin UI Auto-negotiated Chrome 124+, Firefox 128+, Edge 124+
Proxy to upstream servers Auto-negotiated When upstream supports ML-KEM
Control Plane to Data Plane (gRPC mTLS) Always active Both sides run Go 1.25
SSL-inspected client connections Auto-negotiated When client browser supports ML-KEM

What's quantum-resistant today: All key exchanges (TLS handshakes) use hybrid X25519 + ML-KEM-768, meaning traffic confidentiality is protected even against quantum computers.

What's still classical: Certificate signing uses ECDSA P-256. PQC signature algorithms (ML-DSA / Dilithium) are not yet in Go's standard library. Culvert will adopt PQC signing when Go adds native support (expected Go 1.26+). This is a lower risk - signatures prove identity at connection time and cannot be "harvested" for future decryption.

No configuration required - PQC is enabled automatically. No performance impact - the ML-KEM handshake adds ~1ms to the initial TLS connection.

CI Security Pipeline

Every push runs a 10-check security gate:

  1. gosec - Go security linter
  2. govulncheck - reachable CVE detection
  3. trivy - filesystem + Docker image vulnerability scan
  4. gitleaks - secret scanning on PR diffs
  5. staticcheck - advanced static analysis
  6. hadolint - Dockerfile best practices
  7. Race tests - -race flag on full test suite
  8. Coverage gate - minimum 55% statement coverage
  9. License compliance - no GPL/AGPL/LGPL/CPAL dependencies
  10. SBOM generation - CycloneDX JSON via Syft

Plus: CodeQL semantic SAST, Dependency Obituary dependency health scoring, Cosign keyless signing, and SLSA Level 3 provenance on all releases.


Architecture

main.go            - Entrypoint, CLI flags, graceful shutdown, hot reload (SIGHUP)
proxy.go           - HTTP/HTTPS/WebSocket handler, SSL inspection, structured logging
socks5.go          - SOCKS5 server (RFC 1928/1929)
policy.go          - PBAC engine: rule evaluation, conflict detection, GeoIP fail-closed
session.go         - HMAC-SHA256 signed cookies, revocation, dynamic Secure flag
ui.go              - Admin Web UI bootstrap; routes are registered via register*Routes helpers and tracked in uiRoutes (see CLAUDE.md for the canonical inventory and C2/C2c/C4 governance machinery)
store.go           - Config, blocklist, request log, time-series, audit log
security.go        - IP filter, rate limiter, SSRF guard, DNS cache
security_scan.go   - ClamAV + YARA + threat feed scan coordinator
clam.go            - ClamAV INSTREAM client with connection pooling
yara_scan.go       - Pure-Go YARA engine with ReDoS timeout
threatfeed.go      - URLhaus + OpenPhish sync, domain allowlist
feedsync.go        - UT1 URL category syncer
geoip.go           - MaxMind GeoLite2 with background cache
upstream.go        - Proxy chaining, failover, circuit breaker, health checks
ocsp.go            - OCSP/CRL revocation checking
ca.go              - Root CA, MITM certs, AES-GCM encryption, LRU cache, auto-rotation
auth.go            - Auth provider interface
auth_ldap.go       - LDAP bind + search + group resolution
auth_oidc_flow.go  - OIDC Authorization Code + PKCE
auth_saml.go       - SAML 2.0 SP-initiated SSO
auth_idp.go        - Multi-IdP registry with domain routing
identity.go        - Identity model (Sub, Groups, Source, Provider)
totp.go            - TOTP 2FA (RFC 6238, stdlib HMAC-SHA1)
lockout.go         - Brute-force + API rate limiting
connlimit.go       - Per-IP connection limiter, X-Request-ID
metrics.go         - Prometheus metrics (per-rule, latency, bytes)
logger.go          - Structured text/JSON logging with rotation
syslog.go          - RFC 3164/5424 syslog forwarding
alerts.go          - HMAC-SHA256 signed webhook alerts
events.go          - SSE live dashboard stream
config.go          - YAML config loading + validation (goccy/go-yaml)
rewrite.go         - Per-host header rewrite engine
fileblock.go       - File extension/MIME blocking
pac.go             - PAC file generation
blockpage.go       - Block page HTML template
hashcache.go       - SHA-256 scan cache with TTL
controlplane.go    - gRPC Control Plane / Data Plane
plugin.go          - Middleware plugin chain
catdb.go           - URL category database
update.go          - Self-update system (binary + Docker)
update_cluster.go  - Rolling cluster update orchestrator (canary, drain, HA sync)
scan_remote.go     - Remote scan sidecar client for process-isolated scanning
blocklist_feed.go  - Domain blocklist URL feed syncer
bootstrap.go       - Bootstrap script/compose generators for node enrollment
static/            - Embedded SPA (vanilla JS, Chart.js)
updater/           - Docker update sidecar (Go service with rollback)
scripts/           - Install script (multi-distro), CI runner setup
deploy/            - Prometheus + Grafana stack
yara/              - Starter YARA detection rules

Development

Requires Go 1.25+.

go build -o culvert .                       # build
go test -v -race ./...                      # full suite with race detector
go test -coverprofile=cover.out ./...       # coverage report
go test -fuzz FuzzIsPrivateHost -fuzztime=30s  # fuzz SSRF guard

Fuzz Targets

Target Coverage
FuzzIsPrivateHost SSRF guard (DNS + private IP)
FuzzIsSafeRedirectURL Open redirect prevention
FuzzParseClamResponse ClamAV response parser
FuzzNormaliseFeedURL Threat feed URL normalisation
FuzzMatchDest Policy destination matching
FuzzParseYARALiteralString YARA rule string parser

Docker Build

docker build -t culvert:dev .
docker run -p 8080:8080 -p 9090:9090 culvert:dev

Contributing

  1. Fork the repository
  2. Create a feature branch (git checkout -b feature/my-feature)
  3. Run tests (go test -race ./...)
  4. Commit your changes
  5. Open a Pull Request

All PRs are validated by the full CI pipeline including CodeQL, the security gate, and golangci-lint with 18 linters.


License

MIT

Documentation

Overview

D1.6b: --list-backups one-shot CLI.

The Maintenance Agent's GET /v1/backups endpoint shells out to the cli container (via `docker compose --profile cli run --rm cli --list-backups --backup-dir /backup`) and parses the JSON array this command emits to stdout. Operators can also invoke it manually for debugging — same output format either way.

Encryption detection uses the D1.4 magic prefix `CVRTBK01` at byte 0 (see backup_encrypt.go: backupEncMagic). We peek 8 bytes; entries whose first 8 bytes match are flagged encrypted=true. Anything else is reported as encrypted=false; we do NOT attempt deeper validation (gzip-magic, tar header, etc.) because the agent only needs an operator-facing "encrypted-or-not" hint here, and partial reads are cheap.

Symlinks, directories, and unreadable entries are skipped silently rather than failing the whole listing — the human (or agent) can act on whatever IS readable. Errors at the directory level (e.g. non-existent --backup-dir) DO fail the command.

Culvert — Enterprise-grade open source HTTP/HTTPS proxy https://github.com/KidCarmi/Claude-Test

Release Management API — P1.6d-0 (foundation; NO GUI in this slice).

Read-only catalog/current endpoints plus the async dispatch + resume control endpoints, all backed by the P1.6c DispatchService. The handlers own only HTTP concerns (auth, decode, the async 202 split, and a bounded per-agent status store); planning/execution/verify-by-digest/audit/alert all live in the service. Agent endpoints are resolved by an injected resolver (NOT a config route — config is a later slice), so this slice adds no /api/releases/config, no GUI panel, no auto-update, no scheduling.

Dispatch is asynchronous: the handler starts DispatchService.Dispatch in the background and returns 202 the instant the op_id + resume context are durably recorded (the service's onApplied observer). A planner/preflight refusal that happens BEFORE apply returns a synchronous 4xx/503 instead. The watch then runs to terminal in the background and updates the status store, which the GET status endpoint surfaces.

Release Catalog Runtime — Slice 1 (Catalog Load + Resolve).

A pure, in-process, UNSIGNED, local-source, read-only release catalog. It parses the release index + referenced manifests, validates them fail-closed, verifies each manifest's content hash against its RAW bytes, and builds the forward (channel → release) and reverse (pinned ref → release) indexes the rest of the release layer depends on.

Scope (roadmap/D1.6d-P1.2-release-catalog-runtime-plan.md): Catalog Load + Resolve only. NO signatures, NO network/refresh, NO GUI, NO agent calls, NO upgrade dispatch, NO rollback-candidate computation, NO CP→DP propagation.

Control-Plane-only: nothing here is wired into the proxy/data-plane path by this slice; it returns a DIGEST (`repo@sha256:…`, the agent's native currency) and has no release/channel awareness on the agent side.

Integrity ≠ authenticity: the manifest_sha256 check (§4.9) is a content hash, NOT a signature. It catches corruption/drift but is worthless against a tampering attacker, and the INDEX itself has no integrity protection until a later slice signs it. Do not treat a loaded catalog as authenticated.

Release Catalog Distribution — P1.5 Slice d (air-gap BundleCatalogProvider).

BundleCatalogProvider is a TRANSPORT that stages the CATALOG portion of a signed offline release bundle (a tar / tar.gz archive carried into an air-gapped site) into a fresh temp dir, which the caller then hands to LoadVerifiedCatalog (the P1.3 trust boundary) exactly like any other source. Verification is identical to online — only the transport differs (plan §8).

Unlike the HTTP provider, the bundle is COMPLETE before any parse, so no two-phase gate is needed (plan §5.1: local/bundle providers stage-then-verify unchanged). The provider therefore does NO trust work at all — it only moves bytes, under a hostile-archive discipline:

  • any absolute / traversal ("..") / backslash / NUL entry name rejects the whole bundle (never silently skipped);
  • any symlink or hardlink entry rejects the whole bundle;
  • catalog entries must be regular files within per-file and total bounds;
  • non-catalog entries (the future image blobs, plan §8) are skipped WITHOUT reading their content;
  • index.json is required; index.json.sig is staged when present — the enforce/permissive decision for a missing signature belongs to the P1.3 mode at verify time, not to the transport.

Scope (roadmap/D1.6d-P1.5-catalog-distribution-plan.md — Slice d): catalog extraction + staging + cleanup only. NO image docker-load (the bundle slice that loads images MUST bind loaded-image-digest == authenticated list_digest, P1.3 §7 — recorded there, not here), NO outer bundle signature, NO GUI/API, NO dispatch, NO agent changes, NO HTTP changes.

Release Catalog Distribution — P1.5 Slice a (CatalogHolder + atomic publish).

The CatalogHolder owns the live, atomically-swappable verified *Catalog that the rest of the Control Plane reads. It is the swap component P1.2 deferred ("reload = construct a new *Catalog; the refresh slice will own swap semantics; not here"): this slice loads + verifies from a single local directory, publishes atomically, exposes an explicit no-catalog state, and a manual reload that keeps the current catalog on any failure.

Scope (roadmap/D1.6d-P1.5-catalog-distribution-plan.md — Slice a): holder + atomic publish + local-dir reload only. NO goroutine, NO HTTP provider, NO cache persistence, NO staleness, NO GUI/API, NO agent/dispatch/air-gap. The only path to a published catalog is LoadVerifiedCatalog (the P1.3 trust boundary) — there is deliberately no "publish raw *Catalog" entry point.

Release Catalog Distribution — P1.5 Slice c (HTTP CatalogProvider).

HTTPCatalogProvider is a TRANSPORT that stages a catalog candidate (index.json + index.json.sig + referenced manifests) from an HTTP(S) origin into a fresh temp dir, which the caller then hands to LoadVerifiedCatalog (the P1.3 trust boundary) exactly like a local-dir source.

The §5.1 contract is the heart of this slice: because manifests cannot be fetched without first reading their refs out of the index, the provider runs a TWO-PHASE verify — it verifies the index signature over the RAW index bytes BEFORE parsing the index to enumerate manifest fetches. A forged/unsigned index (in enforce mode) therefore triggers ZERO manifest requests. The final LoadVerifiedCatalog over the staged dir re-verifies everything (defense in depth, incl. the manifest_sha256 hash check that authenticates each fetched manifest).

Scope (roadmap/D1.6d-P1.5-catalog-distribution-plan.md — Slice c): the HTTP transport + two-phase verify + staging + cleanup + timeout + minimal retry/backoff + conditional (ETag/If-Modified-Since) fetch. NO GUI, NO dispatch, NO agent changes, NO air-gap bundle, NO CP→DP propagation. The provider is transport-only apart from the §5.1 index-verify gate.

Release Catalog Distribution — P1.5 Slice b (Refresher + last-good cache + staleness).

The Refresher orchestrates stage → verify → publish on top of the P1.5a CatalogHolder, adds a last-good on-disk cache (restart durability), single- flight, refresh metadata, and staleness. It still uses a LOCAL directory as the source — the HTTP provider and air-gap bundle are later slices.

Scope (roadmap/D1.6d-P1.5-catalog-distribution-plan.md — Slice b): refresher + cache + staleness + single-flight + an OPTIONAL (default-off) interval ticker. NO HTTP provider, NO air-gap bundle, NO GUI/API, NO agent/dispatch, NO metrics/alert wiring. Every publish still goes through LoadVerifiedCatalog (the P1.3 trust boundary); the cache is re-verified on load.

Release Catalog Runtime — Slice 1: query surface (Resolve / Lookup / Current / List). All methods are pure and I/O-free after LoadCatalog.

Release Catalog Authenticity — P1.3 Slice 1 (index signature verification).

Adds a Control-Plane-side authenticity gate on TOP of the P1.2 catalog runtime (release_catalog.go): an ed25519 detached signature over the RAW index.json bytes, verified against a baked/operator TrustStore BEFORE the index is parsed or any manifest_sha256 entry is trusted (plan §5/§5.0). Signing the index transitively authenticates every manifest (the index binds each by sha256), so one signature over one document is the whole authenticity kernel.

Scope (roadmap/D1.6d-P1.3-catalog-authenticity-plan.md — Slice 1): TrustStore + envelope parse + LoadVerifiedCatalog + the three enforcement modes. NO GUI, NO agent/dispatch, NO air-gap bundle, NO network/refresh, NO CP→DP propagation, NO metrics wiring. The verifier is release-agnostic and verifies fully offline.

Release Dispatch — P1.6 Slice a (CP-side dispatch kernel, PURE PLANNING).

The Dispatcher turns an operator target (release_id or channel) into a DispatchPlan: it resolves the release from a PINNED immutable catalog snapshot (P1.5 holder), reconciles the catalog repo against the deployment's proxy_repo (repo equality / one explicit air-gap rewrite, design §4), derives Current/already-current from the agent's running_image.repo_digests (P1.1, design §3/§5), and builds the EXISTING upgrades.apply request object (design §6) — WITHOUT sending it.

It is a PURE, deterministic planner: no I/O, no randomness, no agent contact. The idempotency key is an INPUT (higher orchestration owns op identity); the catalog snapshot is read exactly once at plan start. The agent receives only an image_ref + existing apply flags — no release/channel/version/catalog data crosses to it, and it stays release-agnostic.

Scope (roadmap/D1.6d-P1.6-release-dispatch-plan.md — Slice a): planning + the request object + tests. NO agent POST, NO upgrades.check, NO tags, NO tag updater, NO new agent endpoint, NO agent changes, NO GUI, NO auto-update, NO rollback-candidate computation, NO legacy-updater changes.

Release Dispatch — P1.6 Slice b (CP-side execution wrapper).

DispatchExecutor takes a FROZEN DispatchPlan (P1.6a), generates the CP idempotency key, POSTs the EXISTING upgrades.apply, polls the EXISTING agent op to a terminal state, re-reads running_image.repo_digests, and classifies a terminal DispatchTerminal by VERIFYING THE RUNNING DIGEST itself (never the agent's self-report). It is single-flight per agent and emits audit events via a hook.

The agent is untouched and stays release-agnostic: only image_ref + existing apply flags cross the wire (no upgrades.check, no tags, no fallback). The transport is behind the AgentClient seam so the orchestration is fully testable with a fake; httpAgentClient is the concrete adapter over the existing /v1 endpoints.

Scope (roadmap/D1.6d-P1.6-release-dispatch-plan.md — Slice b): execution + verify-by-digest + terminal classification + audit structs/hooks. NO GUI, NO auto-update, NO rollback-candidate computation, NO legacy-updater change, NO new agent endpoint.

Release Dispatch — P1.6 Slice c (CP-side orchestration wiring).

DispatchService is the CP-side wrapper that ties the PURE planner (P1.6a) to the execution wrapper (P1.6b/c-0) and the rest of the Control Plane: the audit ring (release.dispatch + release.dispatch.outcome), the alert webhook hook, and the real HTTP transport to the EXISTING agent /v1 surface.

It owns an AGENT-KEYED single-flight registry: exactly one DispatchExecutor per agent identity, so the executor's per-instance single-flight becomes a per-AGENT guarantee regardless of how many Dispatch calls arrive. A second dispatch to an agent already mid-op is rejected (errDispatchInFlight) and audited, never queued or duplicated.

The agent stays release-agnostic: only image_ref + existing apply flags cross the wire (no upgrades.check, no tags, no fallback). Verify-by-digest remains the only success gate, and the CP idempotency key is generated ONCE per dispatch op and threaded through the plan so the executor honors it (P1.6c-0).

Scope (roadmap/D1.6d-P1.6-release-dispatch-plan.md — Slice c): service wrapper, agent registry, Resume/re-poll, audit + alert wiring, real transport. NO GUI, NO agent change, NO new agent endpoint, NO legacy-updater work, NO auto-update, NO rollback-candidate logic.

Release Management startup wiring (P1.6d-0.1).

Constructs and publishes the Release Management backend (catalog provider + DispatchService + releaseManager) so the /api/releases* routes are actually usable instead of reporting "not configured". It is deliberately MINIMAL and NON-FATAL: any failure leaves globalReleaseMgr nil and the routes report a clear 503 — never a panic.

Scope: empty catalog holder (a later refresh slice populates it), the default proxy_repo, an optional empty repo_rewrite, and the single CP-LOCAL maintenance agent (key "local"), reached over its unix socket by default or an http(s) URL via CULVERT_MAINT_AGENT_URL. NO mutable config route, NO GUI.

Source Files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL