VCL Deep Dive: Components, Pitfalls, and the Pointers Every Engineer Must Know

Mayank Bansal

14 min read

·

Jun 18, 2026

--

Listen

Share

Press enter or click to view image in full size

Varnish Configuration Language is one of the most powerful tools in web infrastructure. It’s also one of the easiest to get catastrophically wrong.

If you’ve ever worked with Varnish Cache, you’ve encountered VCL — Varnish Configuration Language. On the surface, it looks approachable. A few subroutines, some if statements, a return() call here and there. Engineers write their first VCL in an afternoon and feel confident.

Then something breaks in production. A cache that should never have been cached. A backend that never gets hit. A security header that disappears. A request that loops forever.

VCL is deceptively powerful. It gives you near-complete control over how Varnish handles every request and response — which means every mistake has consequences, and some mistakes are catastrophic.

This article covers what VCL actually consists of, how each component works, and — most importantly — the pointers that experienced Varnish engineers have learned to treat with extreme care.

What Is VCL?

VCL is a domain-specific language that Varnish uses to define its caching behaviour. When you write VCL, you are not writing configuration — you are writing a program that Varnish compiles to C and executes for every request.

This distinction matters. VCL is not a settings file. It is code. It has control flow, variables, subroutines, and the ability to make decisions that fundamentally change how your traffic is handled. It runs in a high-concurrency, low-latency environment where mistakes affect every user simultaneously.

Varnish ships with a default VCL that handles common cases sensibly. Most engineers start by overriding small pieces of the default. The trouble begins when they override the wrong pieces, misunderstand the execution model, or don’t account for edge cases that the default VCL was quietly handling for them.

The Execution Model: The Request Lifecycle

Before diving into components, you need to understand how VCL fits into Varnish’s request lifecycle. Every request flows through a series of states, and VCL subroutines are your hooks into those states.

Client Request
      │
      ▼
  vcl_recv          ← First hook. Decide: hash, pass, pipe, synth, purge?
      │
      ▼
  vcl_hash          ← Build the cache key
      │
      ├──── Cache HIT ────► vcl_hit     ← What to do with a cache hit?
      │                          │
      │                          ▼
      │                      deliver / fetch / restart
      │
      └──── Cache MISS ───► vcl_miss    ← What to do with a cache miss?
                                 │
                                 ▼
                             vcl_backend_fetch   ← Modify request to backend
                                 │
                                 ▼
                             vcl_backend_response ← Inspect/modify backend response
                                 │
                                 ▼
                             vcl_deliver         ← Final hook before sending to client
                                 │
                                 ▼
                            Client Response

Each subroutine is a decision point. Each ends with a return() action that determines what Varnish does next. Understanding which subroutine runs when — and what return actions are valid — is the foundation of writing correct VCL.

The Core Components of VCL

1. Subroutines

Subroutines are the primary building block of VCL. Each subroutine corresponds to a point in the request lifecycle. You override them to inject your logic.

The built-in subroutines are:

Subroutine Runs when Common use vcl_recv Request arrives from client Normalise requests, decide cache/pass vcl_hash Building the cache key Custom cache key construction vcl_hit Object found in cache Decide whether to serve or refresh vcl_miss Object not in cache Decide whether to fetch or synthesise vcl_pass Request bypasses cache Modify pass-through requests vcl_backend_fetch About to fetch from backend Modify outgoing backend request vcl_backend_response Backend response received Inspect, modify, or cache the response vcl_deliver About to send response to client Final modifications, add headers vcl_synth Synthesising a response Build custom error or redirect responses vcl_purge Cache purge requested Handle cache invalidation logic vcl_backend_error Backend returns error Handle backend failures gracefully

You can also write custom subroutines and call them with call:

sub normalise_url {
    # Strip tracking parameters
    set req.url = regsuball(req.url, "\?(utm_[^&]+&?)+", "?");
    set req.url = regsub(req.url, "\?$", "");
}

sub vcl_recv {
    call normalise_url;
    # rest of vcl_recv logic
}

Custom subroutines are one of VCL’s most underused features. They keep your VCL maintainable as it grows.

2. Variables

VCL variables give you access to every meaningful piece of a request and response. They are grouped by prefix:

req.* — The client request (read/write in vcl_recv)

req.url          # Full request URL including query string
req.http.Host    # Host header
req.http.Cookie  # Cookie header
req.method       # HTTP method: GET, POST, etc.
req.is_purge     # Boolean: is this a PURGE request?
req.http.X-Forwarded-For  # Client IP (if set by load balancer)

bereq.* — The backend request (read/write in vcl_backend_fetch)

bereq.url            # URL being fetched from backend
bereq.http.Host      # Host header sent to backend
bereq.retries        # Number of times this request has been retried
bereq.uncacheable    # Boolean: will the response be cached?

beresp.* — The backend response (read/write in vcl_backend_response)

beresp.status        # HTTP status code from backend
beresp.ttl           # How long to cache this object
beresp.grace         # How long to serve stale while revalidating
beresp.http.Cache-Control  # Cache-Control header from backend
beresp.uncacheable   # Set to true to prevent caching

resp.* — The response being sent to the client (read/write in vcl_deliver)

resp.status          # HTTP status code being sent to client
resp.http.X-Cache    # Custom header (e.g. HIT/MISS indicator)
resp.http.Server     # Server header (often stripped for security)

obj.* — The cached object (read-only in vcl_hit)

obj.ttl              # Remaining TTL of cached object
obj.grace            # Grace period of cached object
obj.hits             # How many times this object has been served from cache
obj.http.Content-Type  # Content-Type of cached object

3. ACLs (Access Control Lists)

ACLs let you define sets of IP addresses and use them in conditional logic. They are essential for restricting sensitive operations like cache purging.

acl purge_allowed {
    "localhost";
    "127.0.0.1";
    "10.0.0.0"/8;      # Internal network
    "192.168.1.100";   # Specific admin server
}

acl internal_only {
    "10.0.0.0"/8;
    "172.16.0.0"/12;
    "192.168.0.0"/16;
}sub vcl_recv {
    if (req.method == "PURGE") {
        if (!client.ip ~ purge_allowed) {
            return(synth(403, "PURGE not allowed from this IP"));
        }
        return(purge);
    }
}

4. Backends

Backends define the origin servers Varnish connects to. A basic backend declaration:

backend default {
    .host = "app.internal";
    .port = "8080";
    .connect_timeout = 2s;
    .first_byte_timeout = 30s;
    .between_bytes_timeout = 10s;
    .probe = {
        .url = "/health";
        .interval = 5s;
        .timeout = 2s;
        .threshold = 3;
        .window = 5;
    }
}

Directors let you distribute traffic across multiple backends:

import directors;

backend app1 {
    .host = "app1.internal";
    .port = "8080";
}backend app2 {
    .host = "app2.internal";
    .port = "8080";
}sub vcl_init {
    new cluster = directors.round_robin();
    cluster.add_backend(app1);
    cluster.add_backend(app2);
}sub vcl_recv {
    set req.backend_hint = cluster.backend();
}

5. VMODs (Varnish Modules)

VMODs extend VCL with functionality that the core language doesn’t provide. The most commonly used:

import std;       # Standard utilities: logging, type conversion, real IPs
import directors; # Load balancing: round-robin, random, hash, fallback
import cookie;    # Cookie parsing without manual string manipulation
import header;    # Safe header manipulation (multi-value headers)
import re2;       # Regular expressions (faster than built-in regex)
import uuid;      # Generate unique request IDs

VMODs are where VCL’s real power lives. The cookie VMOD alone eliminates a large class of common VCL bugs caused by manual cookie string parsing.

6. Return Actions

Every subroutine ends with a return() that tells Varnish what to do next. The available return actions differ per subroutine.

In vcl_recv:

return(hash);      # Look up in cache (normal flow)
return(pass);      # Bypass cache, fetch from backend
return(pipe);      # TCP passthrough, Varnish steps back entirely
return(synth(status, "message")); # Generate synthetic response
return(purge);     # Purge matching cache object
return(restart);   # Restart request processing from vcl_recv

In vcl_backend_response:

return(deliver);   # Cache and deliver the response
return(pass);      # Don't cache, pass to client
return(abandon);   # Abort the backend fetch
return(restart);   # Restart request processing

Understanding return actions — and which are valid where — is non-negotiable for writing correct VCL.

The Pointers: What Demands Extra Care

These are the areas where experienced VCL engineers exercise the most caution. Each represents a category of mistakes that are easy to make, hard to debug, and potentially catastrophic in production.

⚠️ Pointer 1: Never Cache POST, PUT, PATCH, or DELETE

This seems obvious. It is violated constantly.

sub vcl_recv {
    # ALWAYS explicitly handle non-GET/HEAD methods
    if (req.method != "GET" && req.method != "HEAD") {
        return(pass);
    }
}

The danger: if your vcl_recv doesn't explicitly pass non-idempotent methods, Varnish's default behaviour may not protect you — especially if you're overriding vcl_recv without calling return(pass) for these methods. A cached POST response is not just wrong — it means state-changing operations are never reaching your backend.

Also critically: never cache responses that set cookies. A response with Set-Cookie is personalised by definition. Varnish's default VCL handles this, but if you're overriding vcl_backend_response, you must handle it explicitly.

sub vcl_backend_response {
    # Never cache responses that set cookies
    if (beresp.http.Set-Cookie) {
        set beresp.uncacheable = true;
        return(deliver);
    }
}

⚠️ Pointer 2: Stripping Cookies Is Not Always Safe

The most common VCL pattern is stripping cookies to improve cache hit rates:

sub vcl_recv {
    unset req.http.Cookie;
}

This dramatically improves cache hit rates — and can completely break your application.

Cookie stripping removes the user’s session from every request. If any backend endpoint depends on a session cookie for authentication, personalisation, or CSRF protection, those requests are now arriving at your backend without context.

The correct approach is surgical cookie stripping:

import cookie;

sub vcl_recv {
    # Parse cookies
    cookie.parse(req.http.Cookie);    # Keep only the cookies your application needs
    cookie.filter_except("session_id,csrf_token,ab_test");    # Rewrite the Cookie header with only the kept cookies
    set req.http.Cookie = cookie.get_string();    # If no relevant cookies remain, remove the header entirely
    if (req.http.Cookie == "") {
        unset req.http.Cookie;
    }
}

Strip broadly only for routes that are genuinely anonymous — static assets, public pages, API endpoints that don’t use sessions. Be explicit about which cookies each route needs.

⚠️ Pointer 3: TTL Overrides Must Account for Grace and Keep

When you override TTLs in VCL, you are setting three separate values — not one:

sub vcl_backend_response {
    # Don't do this — you're only setting TTL
    set beresp.ttl = 5m;

    # Do this — set all three deliberately
    set beresp.ttl = 5m;       # Serve fresh for 5 minutes
    set beresp.grace = 30s;    # Serve stale for 30 seconds while revalidating
    set beresp.keep = 10m;     # Keep in cache for conditional requests for 10 minutes
}

Grace is particularly important. Without grace, a cache miss means every concurrent request for that object hits your backend simultaneously — the “thundering herd” problem. With grace, Varnish serves the stale object to all concurrent requests while a single backend fetch revalidates it.

sub vcl_backend_response {
    # For high-traffic routes, always set generous grace
    if (bereq.url ~ "^/api/products") {
        set beresp.ttl = 2m;
        set beresp.grace = 1m;   # Prevents thundering herd on expiry
    }
}

⚠️ Pointer 4: vcl_hash Determines What Gets Shared

vcl_hash is the most security-critical subroutine in VCL. It determines the cache key — which means it determines whether two different users receive the same cached response.

The default vcl_hash uses the URL and the Host header. This is correct for fully public content. It is catastrophically wrong for any content that varies by user.

sub vcl_hash {
    hash_data(req.url);

    if (req.http.Host) {
        hash_data(req.http.Host);
    } else {
        hash_data(server.ip);
    }    # If your content varies by HTTPS vs HTTP
    if (req.http.X-Forwarded-Proto) {
        hash_data(req.http.X-Forwarded-Proto);
    }    # If your content varies by device type
    if (req.http.X-Device-Type) {
        hash_data(req.http.X-Device-Type);
    }    # If your content varies by country (set from X-Country header upstream)
    if (req.http.X-Country) {
        hash_data(req.http.X-Country);
    }    return(lookup);
}

Every signal that makes a response different for different users must be included in the hash. Miss one, and you risk serving one user’s content to another.

Critical rule: Never hash on req.http.Cookie directly. Cookies are unique per session, which means every user gets their own cache entry — effectively disabling caching. Instead, hash on the specific cookie values that determine content variation:

sub vcl_hash {
    hash_data(req.url);
    hash_data(req.http.Host);

    # Hash on the experiment group, not the full cookie string
    if (req.http.X-Experiment-Group) {
        hash_data(req.http.X-Experiment-Group);
    }    return(lookup);
}

⚠️ Pointer 5: pipe Is Almost Never What You Want

return(pipe) in vcl_recv tells Varnish to step completely out of the request — turning the connection into a raw TCP passthrough between the client and the backend. Varnish applies no logic, no headers, no logging, and no timeouts to piped requests.

Engineers reach for pipe when dealing with WebSockets, long-polling, or other persistent connections. It seems like the right tool. It often isn't.

# Problematic pipe usage
sub vcl_recv {
    if (req.http.Upgrade ~ "websocket") {
        return(pipe);  # Varnish is now completely blind to this connection
    }
}

The problems with pipe:

• Connection reuse breaks — piped connections cannot be reused for subsequent requests
• Logging stops — you have no visibility into what’s happening on piped connections
• Timeouts don’t apply — a piped connection can hold resources indefinitely
• Headers you set in vcl_recv before the return(pipe) are preserved, but nothing after

For WebSockets, consider routing WebSocket traffic to a separate backend or using a proxy that handles upgrades natively, rather than piping through Varnish.

⚠️ Pointer 6: Regex in VCL Is Expensive and Easy to Get Wrong

VCL supports regular expressions for URL matching, header inspection, and string manipulation. They are powerful and frequently misused.

# Expensive — runs a regex on every single request
sub vcl_recv {
    if (req.url ~ "^/(images|css|js|fonts)/") {
        unset req.http.Cookie;
    }
}

Guidelines for VCL regex:

Anchor your patterns. req.url ~ "/api/" matches /anything/api/anything. Use ^ and $ anchors: req.url ~ "^/api/".

Use regsub vs regsuball deliberately. regsub replaces the first match. regsuball replaces all matches. Using the wrong one for URL normalisation produces incorrect cache keys.

# Remove ALL tracking parameters (correct use of regsuball)
set req.url = regsuball(req.url, "&?(utm_[^&]+)", "");

# Remove only the first occurrence of a pattern (regsub)
set req.url = regsub(req.url, "^/v[0-9]+", "");

Compile complex patterns into VMODs. For complex matching logic, the re2 VMOD offers better performance than built-in regex for high-frequency patterns.

⚠️ Pointer 7: The Restart Limit and Infinite Loops

VCL allows return(restart) to reprocess a request from vcl_recv. This is useful for retry logic, authentication redirects, and error recovery. It is also a vector for infinite loops.

# This can loop forever if the condition is never cleared
sub vcl_recv {
    if (req.http.X-Retry == "true") {
        unset req.http.X-Retry;
        return(restart);  # Loops back to vcl_recv
    }
}

Varnish limits restarts to max_restarts (default: 4). When the limit is hit, Varnish returns a synthetic error. Always verify your restart logic has a clear termination condition, and be aware that hitting the restart limit produces a 503 to the client.

sub vcl_recv {
    # Always guard restart logic with a counter check
    if (req.restarts > 0 && req.http.X-Auth-Failed == "true") {
        return(synth(401, "Authentication required"));
    }
}

⚠️ Pointer 8: Headers Are Not Automatically Forwarded or Stripped

Varnish passes most headers through by default — both from client to backend and from backend to client. This has two dangerous implications:

Internal headers leak to clients. If your backend sets X-Internal-Debug, X-Backend-Server, or any other diagnostic header, Varnish will forward it to the client unless you explicitly strip it.

sub vcl_deliver {
    # Strip headers that reveal infrastructure information
    unset resp.http.X-Powered-By;
    unset resp.http.Server;
    unset resp.http.X-Varnish;
    unset resp.http.Via;
    unset resp.http.X-Backend-Server;

    # Add a safe cache status header instead
    if (obj.hits > 0) {
        set resp.http.X-Cache = "HIT";
    } else {
        set resp.http.X-Cache = "MISS";
    }
}

Client headers reach your backend unfiltered. Headers like X-Forwarded-For, X-Real-IP, and custom headers set by the client arrive at your backend unless you normalise them. An attacker can set X-Forwarded-For: 127.0.0.1 and potentially bypass IP-based access controls on your backend.

sub vcl_recv {
    # Trust X-Forwarded-For only from known load balancers
    if (client.ip ~ trusted_proxies) {
        # Append real IP to existing XFF
        set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip;
    } else {
        # Overwrite any client-set XFF header with actual client IP
        set req.http.X-Forwarded-For = client.ip;
    }
}

⚠️ Pointer 9: Health Probes and Backend Sick States

Every backend should have a health probe. Without one, Varnish has no way to detect that a backend is down — it will keep trying to send requests to a dead backend until request timeouts trigger.

backend app {
    .host = "app.internal";
    .port = "8080";
    .probe = {
        .url = "/health";
        .expected_response = 200;
        .interval = 5s;
        .timeout = 2s;
        .window = 5;       # Check last 5 probes
        .threshold = 3;    # Require 3 of last 5 to be healthy
        .initial = 2;      # Start with 2 assumed healthy probes
    }
}

When a backend goes sick, Varnish can serve stale content from cache (if grace is configured) or return an error. Configure grace generously enough to survive short backend outages:

sub vcl_backend_response {
    set beresp.grace = 1h;  # Serve stale for up to 1 hour if backend is sick
}

sub vcl_hit {
    # Serve stale immediately if backend is sick
    if (obj.ttl <= 0s && obj.grace > 0s) {
        if (!std.healthy(req.backend_hint)) {
            return(deliver);  # Serve stale — backend is sick
        }
    }
}

⚠️ Pointer 10: Test VCL Changes in Staging — Always

VCL runs for every request. A bug in VCL is not a bug that affects one request or one user — it is a bug that affects all requests, simultaneously, the moment it is loaded.

A bad regex in vcl_recv can send all traffic to pass mode, eliminating your cache hit rate. A wrong TTL in vcl_backend_response can cache authenticated responses for an hour. An incorrect return(pipe) can break all connections to a backend.

Before deploying VCL changes:

1. Validate syntax: varnishd -C -f /etc/varnish/default.vcl compiles and validates without deploying
2. Test in staging: Your staging environment should have an identical Varnish configuration
3. Use varnishlog: Watch live request processing during testing — varnishlog -g request -q "ReqURL ~ '/path'" shows exactly how VCL is handling specific requests
4. Load VCL with varnishadm: varnishadm vcl.load test_v2 /etc/varnish/default.vcl && varnishadm vcl.use test_v2 lets you load and activate VCL without restarting Varnish

A VCL Review Checklist

Before deploying any VCL change, run through this list:

Security:

• [ ] Non-GET/HEAD methods are explicitly passed or blocked
• [ ] Responses with Set-Cookie are never cached
• [ ] Purge requests are ACL-restricted
• [ ] Internal/diagnostic headers are stripped in vcl_deliver
• [ ] Client-controlled headers (XFF, custom headers) are sanitised before reaching backend

Caching correctness:

• [ ] Cache key in vcl_hash includes every signal that varies the response
• [ ] Cookie stripping is surgical, not blanket
• [ ] TTL, grace, and keep are all set deliberately (not just TTL)
• [ ] Authenticated and personalised routes are explicitly set to uncacheable

Reliability:

• [ ] All backends have health probes configured
• [ ] Grace is set generously enough to survive backend outages
• [ ] Restart logic has a clear termination condition
• [ ] pipe is used only where genuinely required

Observability:

• [ ] vcl_deliver adds an X-Cache header for debugging
• [ ] Logging is sufficient to trace a request through VCL subroutines
• [ ] Syntax has been validated with varnishd -C

The Bottom Line

VCL is one of the most expressive and powerful tools available to engineers building high-performance web infrastructure. It is also one of the most unforgiving — because every line you write runs at traffic speed, for every user, with no safety net.

The engineers who write VCL well are not the ones who memorise every subroutine. They are the ones who understand the execution model, respect the cache key as a security boundary, treat every TTL override as a contract, and test relentlessly before deploying.

Write VCL like it runs for every user simultaneously. Because it does.

On the topic of getting this right in practice — I provide CDN optimisation services for engineering teams who want to get more out of their edge infrastructure without taking on unnecessary risk. This covers VCL audits and rewrites, caching architecture design, cache key strategy, security hardening at the edge (origin lockdown, mTLS, ACL design), and performance tuning across Varnish, Fastly, Cloudflare, Akamai, and CloudFront setups. In a recent engagement, this work took a production CDN from a fragmented, default-heavy configuration to a sustained cache hit rate of over 95%, while also closing several of the exact gaps covered in this article — unscoped cookie stripping, missing grace periods, and an under-specified cache key that was putting personalised content at risk. If your team is dealing with inconsistent cache performance, unexplained production bugs at the edge, or simply hasn’t revisited your VCL or CDN configuration in a while, feel free to connect — happy to take a look and share what I see.

Found this useful? Follow for more deep dives into the infrastructure layers most engineers never look inside.