Stories by Vasanthan K on Medium

SvelteKit 3 Migration Notes: Node 22, Vite 8 and What Actually Breaks

Vasanthan K — Wed, 04 Nov 2026 14:01:03 GMT

A beginner-friendly field guide to upgrading from SvelteKit 2 to SvelteKit 3 — what changed, why it changed, and the quiet failures a green build will not show you.

SvelteKit 3.0 shipped on October 1, 2026, and the official line is that it feels like the same framework “with a little more polish.” That is true for your code. It is not the whole story for your upgrade.

A SvelteKit 3 upgrade is really three upgrades stacked on top of each other: a new runtime (Node 22.17+), a new build tool (Vite 8, now powered by the Rust bundler Rolldown) and a new framework major (SvelteKit 3 itself). The official codemod handles most of the framework layer beautifully. The failures tend to hide in the layers around it — your package manager, your Vite config, and links or buttons written inline in markup.

In this post we will walk through each layer slowly, with before-and-after code, the theory behind every change, and diagrams you can come back to. If you have never done a major framework migration before, you are exactly the reader I wrote this for.

TL;DR

Runtime: SvelteKit 3 needs Node 22.17 or newer, TypeScript 6, Svelte 5.57.1+ and Vite 8.0.12+. The npm bundled with Node 22 can crash while installing a fresh SvelteKit 3 project — use npm 11.6+, pnpm, or a newer Node line.
Build tool: Vite 8 replaces the esbuild + Rollup pair with a single Rust bundler, Rolldown. rollupOptions becomes rolldownOptions, esbuild becomes oxc, and the object form of manualChunks is gone.
Framework: svelte.config.js is gone (config moves into vite.config.ts), $lib becomes #lib, $app/stores is removed, $app/environment is now $app/env, environment variables are declared in src/env.ts, and error() takes a string message.
The real risk: npx sv migrate sveltekit-3 does most of the work, but it can leave broken inline pushState calls and malformed {base} links. Always run npm run check, build, and click through the app.

1. Why this is three upgrades, not one

Think of your SvelteKit app as a building with three floors. Your runtime (Node.js) is the foundation. Your build tool (Vite) is the plumbing and wiring that turns source files into something a browser and server can run. Your framework (SvelteKit) is the furnished floor you actually live on.

SvelteKit 3 raises the minimum version on every floor at once. That matters because each floor depends on the one below it. You cannot install Vite 8 on an unsupported Node version, and SvelteKit 3 will refuse to work with anything other than Vite 8.

The practical rule that falls out of this picture: upgrade from the bottom up. First get Node and your package manager right, then make sure Vite 8 is happy, then let the SvelteKit codemod rewrite your app code. If you do it in reverse, you will spend an afternoon debugging framework errors that are really install or bundler errors in disguise.

A note on why the Svelte team bundled these together. A framework major version is the only time a project can drop old runtimes and old APIs without surprising anyone. The SvelteKit 3 announcement describes the release as the same framework with more type safety and “a little less junk.” Most of that junk is legacy API surface (like Svelte 4-era stores) and config indirection (like a separate svelte.config.js). Raising the floor on Node and Vite is what lets the team delete it.

2. The version matrix: what you need before you start

The official migration guide lists hard minimums. Here they are next to what most SvelteKit 2 projects run today, plus why each one exists.

A quick way to see where you stand:

node -v                 # should print v22.17.0 or higher
npm -v                  # 11.6.0+ recommended (see section 3)
npx tsc -v              # Version 6.x.x
npm ls vite svelte @sveltejs/kit @sveltejs/vite-plugin-svelte

What each command tells you. node -v and npm -v check your runtime floor. npx tsc -v runs the TypeScript compiler installed in your project (not a global one), so it reflects what svelte-check will use. npm ls prints the resolved version of each package from your lockfile, which is more truthful than reading the ranges in package.json. A range like ^7.0.0 tells you what is allowed; npm ls tells you what is installed.

Lock the runtime in your repo. Add an engines field and a packageManager field to package.json, so teammates and CI fail fast instead of half-installing:

{
  "engines": { "node": ">=22.17" },
  "packageManager": "npm@11.6.0"
}

engines documents the minimum Node version, and npm warns when it does not match. packageManager is read by Corepack, which can download and pin the exact package manager for everyone on the team. Together they turn "works on my machine" into a rule the tooling enforces.

Upgrade to the latest 2.x first. The Svelte team recommends landing on the newest SvelteKit 2 release before jumping to 3. Late 2.x versions print targeted deprecation warnings for almost everything that is removed in 3, and SvelteKit 2.62+ already accepts configuration in vite.config.ts. Fixing those warnings on a version you know is far easier than fixing errors on one you don't.

3. Layer 1 — Node 22 and the npm install trap

The first thing that breaks is often not your code at all. It is npm install.

What a runtime floor really means

When a package declares "engines": { "node": ">=22.17" }, it is promising its code only uses features that exist in Node 22.17 and later. For SvelteKit 3 that means it can delete the old @sveltejs/kit/node/polyfills module, which used to patch missing web APIs (like fetch or crypto) onto older Node versions. Those APIs now ship in Node itself, so the shim is dead weight.

If you have a custom server, remove the import:

// server.js — SvelteKit 2 custom server
import '@sveltejs/kit/node/polyfills'; // ❌ delete this line in SvelteKit 3
import { handler } from './build/handler.js';

And the getRequest / setResponse helpers from @sveltejs/kit/node are now synchronous, so drop the await:

// Before (SvelteKit 2)
const request = await getRequest({ base, request: req });
await setResponse(res, response);

// After (SvelteKit 3)
const request = getRequest({ base, request: req });
setResponse(res, response);

Why remove await? Awaiting a non-promise value is harmless in JavaScript, so this would not crash. But it signals to readers (and linters) that something asynchronous is happening when it is not. Matching the new signature keeps your code honest and avoids a lint warning about awaiting non-thenables.

The trap: Node 22’s bundled npm

Here is the counter-intuitive part. An independent agency reproduced and documented that a fresh SvelteKit 3 project scaffolded with Vitest can crash during npm install on npm versions 10.8.2 through 11.5.0. The error comes from deep inside npm's dependency resolver and looks something like this:

npm error Cannot read properties of null (reading 'edgesOut')

The crash disappears from npm 11.6.0 onward, and pnpm and Yarn install the same project fine. The catch is that every Node 22 release ships with an npm 10.x. So if you install Node 22 LTS and use the npm that comes with it, exactly as the docs suggest, a brand-new project may fail to install.

Why your CI stays green while laptops break. npm ci installs straight from your committed lockfile. It never computes a fresh dependency tree, so it never reaches the buggy code path. npm install without a lockfile (a new clone, a dependency bump, a Docker build) does compute the tree, and that is where the crash happens.

How to fix it

Pick one of these, in order of preference:

# Option A: upgrade npm itself (works on Node 22)
npm install -g npm@^11.6.0

# Option B: use pnpm via Corepack
corepack enable
pnpm install

# Option C: pin Vitest to a newer major that avoids the crash
npm install -D vitest@^5

Avoid reaching for --legacy-peer-deps as a quick fix. It tells npm to ignore peer-dependency rules entirely, which can silently install a different version of a transitive package than everyone else gets. If someone on your team does use it, review the lockfile diff before merging.

Two more install-time surprises

TypeScript 7 is not supported yet. SvelteKit 3 and svelte-check peer on TypeScript ^6. If your project already moved to TypeScript 7, npm install will fail with ERESOLVE. Stay on 6.x for now.
Third-party plugins that pin @sveltejs/kit@^2. Any package that declares a peer dependency on SvelteKit 2 only will block the install. Search your package.json and lockfile for packages that peer on @sveltejs/kit and check each one has a release that accepts ^3.

# List every installed package that declares a peer on @sveltejs/kit
npm ls @sveltejs/kit --all

4. Layer 2 — Vite 8, Rolldown and Oxc

Vite 8 is the biggest internal change in Vite’s history, yet for most SvelteKit apps it needs zero code changes. To understand when it does need changes, you need to know what a bundler does.

Bundler basics in 60 seconds

Your app is hundreds of small files that import each other. Browsers can load those files one by one, but that is slow in production. A bundler walks the import graph, combines files into a few optimised “chunks,” strips unused code (tree-shaking) and minifies the result. A transformer does the per-file work before that: turning TypeScript into JavaScript, lowering new syntax for older browsers, and so on.

Up to Vite 7, these jobs were split across two tools written in different languages: esbuild (Go) handled fast transforms and dependency pre-bundling during development, while Rollup (JavaScript) did the production build. Two engines meant two subtly different behaviours. Code that worked in npm run dev could break in npm run build.

Vite 8 replaces both with Rolldown, a Rust re-implementation of Rollup’s API. Oxc takes over JavaScript transforms and minification, and Lightning CSS minifies CSS. One engine for dev and prod means far fewer “only breaks in production” bugs, and teams migrating real SvelteKit sites have reported roughly 10–15% faster builds on average.

The minimum upgrade

If your vite.config.ts contains nothing but the SvelteKit plugin, the Vite part of the migration is only version bumps:

npm install -D vite@^8.0.12 @sveltejs/vite-plugin-svelte@^7 vitest@^4

The easy one to miss is @sveltejs/vite-plugin-svelte. It jumps a major version (6 → 7) to support Vite 8. If you edit package.json by hand and leave ^6, npm will never pick up 7, and you will get a peer-dependency conflict.

Renamed config keys

If you customised Vite, rename these. The old names still work through a compatibility layer, but they print warnings and will be removed later.

// vite.config.ts — BEFORE (Vite 7)
export default defineConfig({
  esbuild: { drop: ['console', 'debugger'] },
  optimizeDeps: { esbuildOptions: { target: 'es2022' } },
  build: {
    rollupOptions: {
      output: { manualChunks: { vendor: ['chart.js'] } }
    }
  }
});

// vite.config.ts — AFTER (Vite 8)
export default defineConfig({
  oxc: { /* JS transform options now live here */ },
  optimizeDeps: { rolldownOptions: { /* ... */ } },
  build: {
    rolldownOptions: {
      output: {
        // manualChunks object form is removed; let Rolldown split automatically
        minify: { compress: { dropConsole: true, dropDebugger: true } }
      }
    }
  }
});

Reading the diff line by line:

esbuild → oxc: the top-level transform config follows the tool that does the work. Note that dropping console calls is a minification job, so it moved under build.rolldownOptions.output.minify instead of oxc. Check the Vite migration guide for the exact option names in your Vite minor.
optimizeDeps.esbuildOptions → optimizeDeps.rolldownOptions: dependency pre-bundling is now done by Rolldown too.
build.rollupOptions → build.rolldownOptions: same idea, same shape. Rolldown aims to accept Rollup's options.
manualChunks: { ... } (object form) is removed, and the function form is deprecated. Rolldown's automatic chunking is usually as good. If you truly need manual control, use Rolldown's newer code-splitting option.

The quiet breakage: CommonJS default imports

This is the one change in Vite 8 that type-checking will never catch. Some older npm packages publish CommonJS (module.exports = ...) instead of ES modules. When you write import foo from 'old-package', the bundler has to guess whether foo should be module.exports or module.exports.default.

Vite 7 could guess differently in dev (esbuild) and prod (Rollup). Vite 8 guesses the same way everywhere. That is better in the long run, but it means a package that “worked” before may now hand you undefined:

TypeError: Cannot read properties of undefined (reading 'format')

Your TypeScript types are still correct, so svelte-check passes. Only running the code reveals it. Fixes, in order of preference: switch to the package's ES module build, replace the package with a maintained alternative, or as a short-term escape hatch:

// vite.config.ts — temporary only, remove before shipping
export default defineConfig({
  legacy: { inconsistentCjsInterop: true }
});

Small side effects worth knowing

New chunk hashes. Rolldown names and hashes chunks differently, so your first deploy invalidates every cached JS file. Expect a cold cache, not a bug.
CSS may be 1–3% larger. Lightning CSS lowers syntax differently from esbuild. Usually not worth acting on.
Remove dead rollup overrides in package.json (for example, ones added for old Rollup CVEs). Vite 8 no longer uses Rollup, but run npm audit afterwards: Rolldown pulls in a different tree of transitive dependencies.

5. Layer 3 — SvelteKit 3’s breaking changes, explained

This is where most of your code edits happen. I have grouped the changes by the question a beginner is likely to ask: “where does my config go?”, “how do I import my own files?”, “how do I read page state?” and so on.

5.1 Configuration moves into vite.config.ts

Why: In SvelteKit 2, configuration was split across two files. svelte.config.js held SvelteKit and Svelte compiler options; vite.config.ts held Vite options and loaded the SvelteKit plugin. Other Vite plugins could not easily see SvelteKit's settings. Now there is one file, and the SvelteKit plugin receives its options directly.

// svelte.config.js — SvelteKit 2 (delete this file)
import adapter from '@sveltejs/adapter-node';

export default {
  compilerOptions: { runes: true },
  kit: {
    adapter: adapter(),
    csrf: { checkOrigin: false },
    prerender: { origin: 'https://example.com' }
  }
};

// vite.config.ts — SvelteKit 3
import { sveltekit } from '@sveltejs/kit/vite';
import adapter from '@sveltejs/adapter-node';
import { defineConfig } from 'vite';

export default defineConfig({
  plugins: [
    sveltekit({
      adapter: adapter(),                      // was kit.adapter
      compilerOptions: { runes: true },        // same as before, now beside kit options
      csrf: { trustedOrigins: ['https://checkout.stripe.com'] }, // replaces checkOrigin: false
      paths: { origin: 'https://example.com' } // replaces prerender.origin and adapter-node's ORIGIN
    })
  ]
});

What changed, conceptually: everything that lived under kit: { ... } is now a top-level option of sveltekit({ ... }). Anything SvelteKit does not recognise is passed through to vite-plugin-svelte, so options like inspector go here too.

Three options deserve a closer look:

csrf.checkOrigin: false is gone. CSRF (cross-site request forgery) protection rejects form posts that come from other websites. Turning it off globally was a common shortcut for payment or login providers that post back to your app. SvelteKit 3 makes protection always-on, and you allow-list the specific origins you trust with csrf.trustedOrigins.
paths.origin replaces both prerender.origin and adapter-node's ORIGIN env var. Behind a reverse proxy (Nginx, a load balancer), the server cannot always work out its public URL from request headers. paths.origin tells it explicitly. It is also used for CSRF checks.
preloadStrategy, vitePlugin, files.lib and several experimental.* flags are removed. Delete them; the migrator does this for you.

5.2 $lib becomes #lib

Why: $lib was a SvelteKit-specific alias. Every tool (TypeScript, Vitest, ESLint, your editor) needed to be told about it. #lib uses Node's standard subpath imports: the imports field in package.json. Node, Vite and TypeScript all understand this field natively, so the alias "just works" everywhere.

// package.json
{
  "imports": {
    "#lib": "./src/lib/index.js",
    "#lib/*": "./src/lib/*"
  }
}

// Before
import { formatDate } from '$lib/utils';
import Button from '$lib/components/Button.svelte';

// After - note the explicit file extension
import { formatDate } from '#lib/utils.ts';
import Button from '#lib/components/Button.svelte';

The gotcha: file extensions are now required. The "#lib/*": "./src/lib/*" mapping is a simple text substitution. #lib/utils.ts becomes ./src/lib/utils.ts. But #lib/utils becomes ./src/lib/utils, which is not a file, and Node's resolver does not guess extensions the way bundler aliases did. The same goes for folders: write #lib/utils/index.ts, not #lib/utils. If your build fails with a module-resolution error after the migration, this is the first thing to check.

5.3 $app/stores is removed — use $app/state

Why: Svelte 5 replaced stores with runes (fine-grained reactive state). $app/state exposes page, navigating and updated as reactive objects. You read them directly, without the $ store prefix, and components only re-render when the exact property they read changes.



You are on {$page.url.pathname}





You are on {page.url.pathname}

Two behaviour changes come with it:

page.url is now read-only. It is typed as a ReadonlyURL, so page.url.searchParams.set('q', 'x') is a type error. That code never actually changed the browser URL; it only mutated a shared object. If you need a mutable copy, make one: const url = new URL(page.url.href);
updated.current flips on its own. SvelteKit now checks for a new deployment on data requests, remote-function calls, when the tab regains focus, and every hour by default (version.pollInterval). If you show a "New version available — reload" banner, it will appear more often. That is the feature working as intended.

5.4 $app/environment is now $app/env

A straight rename. browser, dev, building and version keep their meaning:

// Before
import { browser, dev } from '$app/environment';
// After
import { browser, dev } from '$app/env';

5.5 Environment variables are declared in src/env.ts

Why: In SvelteKit 2 you chose between four modules ($env/static/private, $env/static/public, $env/dynamic/private, $env/dynamic/public) and named public variables with a PUBLIC_ prefix. Nothing checked that a required variable actually existed, so a missing API key surfaced as a runtime crash in production. SvelteKit 3 promotes the "explicit environment variables" feature from experimental: you declare every variable once, with validation.

// src/env.ts
import { defineEnvVars } from '@sveltejs/kit/env';
import * as v from 'valibot';

export const variables = defineEnvVars({
  // Private by default: only importable from server code
  DATABASE_URL: {
    schema: v.pipe(v.string(), v.url()),
    description: 'Postgres connection string'
  },
  // Public: safe to ship to the browser
  SITE_NAME: { public: true },
  // Plain function instead of a schema library: return the value or throw
  PORT: {
    schema: (value) => {
      if (value === undefined) return 3000;
      const port = Number(value);
      if (!Number.isInteger(port)) throw new Error('PORT must be an integer');
      return port;
    }
  },
  // static: inlined at build time, enabling dead-code elimination
  SHOW_DEBUG_OVERLAY: {
    public: true,
    static: true,
    schema: v.pipe(v.optional(v.string(), ''), v.transform((s) => s !== ''))
  }
});

// src/routes/+page.server.ts
import { DATABASE_URL, PORT } from '$app/env/private'; // server-only

// src/routes/+page.svelte
import { SITE_NAME } from '$app/env/public';           // browser-safe

How to read this file:

Private is the default. A variable without public: true can only be imported from $app/env/private, and SvelteKit refuses to include that module in browser code. Secrets cannot leak into a JavaScript bundle by accident.
schema validates and transforms. It accepts any Standard Schema library (Valibot, Zod) or a plain function. Notice that PORT comes out as a number, not a string, and gets a default. If validation fails, the app refuses to build or start, so a missing secret is caught at deploy time instead of by your users.
static: true means "bake it into the build." Without it, values are read when the server boots (dynamic). With it, the value is inlined, and the bundler can delete code behind {#if SHOW_DEBUG_OVERLAY} entirely when it is false.
description shows up when you hover the variable in your editor. Free documentation.

The old $env/* modules still work in SvelteKit 3 but are deprecated and scheduled for removal in SvelteKit 4. Migrate now while you are touching the code anyway.

5.6 Links: base is gone, use resolve() and asset()

Why: Building URLs by string concatenation (base + '/blog/' + slug) is error-prone and untyped. resolve() takes a route ID plus params and returns a correct, type-checked path that already includes your base path.




Read


Read

Note the subtle rule: route IDs start with / ('/blog/[slug]'), but plain pathnames and asset paths do not (resolve('about'), asset('logo.png')). Remember this rule — it is the root of one of the bugs in section 7.

5.7 Navigation: pushState folds into goto

Why: Shallow routing (changing the URL and page state without loading a new page, for modals or tabs) had its own functions. SvelteKit 3 puts everything behind one API.

import { goto, refreshAll } from '$app/navigation';

// Before
pushState('/photos/42', { showModal: true });
replaceState('/photos/42', { showModal: true });
goto('/', { invalidateAll: true, noScroll: true, keepFocus: true, replaceState: true });
await invalidateAll();

// After
goto('/photos/42', { shallow: true, state: { showModal: true } });
goto('/photos/42', { shallow: true, replace: true, state: { showModal: true } });
goto('/', { refreshAll: true, reset: false, replace: true });
await refreshAll();

Option by option:

shallow: true = update the URL and page.state without navigating (old pushState). Add persistState: true to keep that state across a page reload.
replaceState → replace: overwrite the current history entry instead of adding one.
noScroll + keepFocus → reset: false: one switch for "don't reset scroll and focus."
invalidateAll → refreshAll: re-run every load function. Unlike the old one, it does not wipe page.state, which is what you want when a modal is open.

Two behaviour changes to know: goto() now rejects (throws) for URLs that do not match a route in your app, so use window.location.href = url for external sites. And shallow navigations now fire beforeNavigate, onNavigate and afterNavigate. Check navigation.shallow inside those hooks if you need to ignore them.

5.8 Errors: simpler error(), louder handleError

import { error } from '@sveltejs/kit';

// Before: message could be a string OR an object
error(404, { message: 'Post not found', code: 'POST_404' });

// After: message is always a string; extra fields go third
error(404, 'Post not found', { code: 'POST_404' });

The fuller story:

App.Error always has a status now, so your +error.svelte can show it without extra plumbing.
handleError sees every error, including the "expected" ones you throw with error(). In SvelteKit 2 it only saw unexpected crashes. If you log to Sentry or similar inside handleError, filter on status or you will flood your error tracker with 404s.
Rendering errors are caught. Every +error.svelte automatically becomes a Svelte error boundary, so a crash inside a component shows your error page instead of a blank screen.
json() and text() helpers are deprecated. Use the web standard instead: return Response.json({ ok: true }) or return new Response('hi').
Form fail(400, ...) now really returns 400 to enhanced forms instead of 200. Tests that assert on status codes will need updating.

5.9 Smaller changes that still bite

The external-redirect change is a security improvement worth understanding. An open redirect happens when an attacker can make your site send users to a URL of their choosing, often from a query parameter like ?next=. Requiring { external: true } (or an allow-list of origins) forces you to make that choice deliberately:

import { redirect } from '@sveltejs/kit';

redirect(307, 'https://accounts.example.com/login', {
  external: ['https://accounts.example.com'] // only this origin is allowed
});

6. Running sv migrate, step by step

You do not have to make most of the edits in section 5 by hand. The Svelte CLI (sv, which reached 1.0 alongside SvelteKit 3) ships a task-based migration that rewrites your project. Here is the full workflow, from clean branch to merge.

Step 1 — Start from a clean, committed branch

git checkout -b upgrade/sveltekit-3
git status          # must say "nothing to commit, working tree clean"

Why it matters: the migrator edits files in place. A clean tree means git diff shows exactly what the tool changed, and git checkout . undoes it if you need to start over.

Step 2 — Land on the latest SvelteKit 2 and fix warnings

npm install -D @sveltejs/kit@^2 && npm run dev

Click around the app and read the terminal. Every deprecation warning you fix here is one less mystery error after the jump.

Step 3 — Get the runtime right

nvm install 22 && nvm use 22   # or fnm / volta / your Docker base image
npm install -g npm@^11.6.0
node -v && npm -v

Update your CI config and Dockerfile in the same commit. A common miss is a FROM node:20-alpine line that nobody remembers until the deploy fails.

Step 4 — Preview, then run, the migration

# See the list of tasks (12 at the time of writing) without changing anything
npx sv@latest migrate sveltekit-3 --tasks

# Run every task, skip the confirmation prompt, then install with npm
npx sv@latest migrate sveltekit-3 --tasks all --confirm --install npm

What the flags mean: --tasks on its own lists the available tasks. --tasks all selects every task (you can also name individual ones to migrate in smaller, reviewable commits). --confirm skips the interactive prompt. --install npm reinstalls dependencies afterwards with the package manager you name.

In practice the migrator bumps your dependencies, folds svelte.config.js into vite.config.ts, adds the imports field and rewrites $lib to #lib, swaps $app/stores for $app/state and $app/environment for $app/env, creates src/env.ts, moves param matchers into src/params.ts, and points tsconfig.json at $app/tsconfig. A typical diff after a successful run looks like this:

-  import { page } from '$app/stores';
-  import { browser } from '$app/environment';
-  import { base } from '$app/paths';
-  import { PUBLIC_SITE_NAME } from '$env/static/public';
-  import { greet } from '$lib/greet';
+  import { page } from '$app/state';
+  import { browser } from '$app/env';
+  import { resolve } from '$app/paths';
+  import { PUBLIC_SITE_NAME } from '$app/env/public';
+  import { greet } from '#lib/greet.js';

Read the diff before moving on. Commit it on its own (git commit -am "chore: sv migrate sveltekit-3"), so your manual fixes in the next step are reviewable separately.

Step 5 — Work through MIGRATION_TASKS.md

Some changes need a human decision, so the migrator writes them into a MIGRATION_TASKS.md file in your project root instead of guessing. Typical entries cover goto destinations, the new error() signature, invalidateAll → refreshAll, dev-server CORS for static files and the deprecated json() helper. The file is written step by step, so you (or a coding agent) can work through it in order. Delete it when you are done.

Steps 6–8 — Verify three different ways

Covered in detail in section 8. The short version: npm run check, then npm run build, then actually use the app. Each step catches a different class of bug, which is the point of the next section.

7. What actually breaks: the gotchas a green build hides

The migrator is good. A developer who tested it on a deliberately old-fashioned SvelteKit 2 app found it handled every import rename and the build passed in under half a second. Then two things broke in the browser. Both share a root cause worth understanding.

Why codemods miss things: script vs markup

A codemod is a program that parses your source code into a tree, finds patterns, and rewrites them. Inside a

The fix is the new goto form:

npm run check (svelte-check) does catch this one, with an error like Cannot find name 'pushState'. The lesson: never skip check just because build passed.

Gotcha 2 — {base} inside a quoted href

This is the sneaky one. Suppose you wrote a link like this in SvelteKit 2:

First post

The migrator can turn it into:

First post

After hydration (the moment SvelteKit’s JavaScript takes over the server-rendered HTML in the browser), resolve('') returns an empty string when there is no base path. The link becomes //blog/hello-world. A URL that starts with two slashes is a protocol-relative URL: it means "same protocol, different host." The browser tries to visit a server called blog, and you get "This site can't be reached."

Neither svelte-check nor the build complains, because "{resolve('')}/blog/hello-world" is a perfectly valid string. The fix is to use the route ID form that resolve() was designed for:

First post

Interestingly, links written as template literals (href={`${base}/about`}) were migrated correctly to resolve('about'). Whether you hit this bug depends on how you happened to write your links. Search for the risky pattern before you migrate:

# Find {base} used inside quoted attributes
grep -rnE '="\{base\}' src/
# Find shallow-routing calls in markup
grep -rnE '(pushState|replaceState)\(' src/ --include=*.svelte

Gotcha 3 — param matchers lose their types

When matchers move from src/params/*.ts into a single src/params.ts, the parameter type can be dropped, and svelte-check reports Parameter 'param' implicitly has an 'any' type. Add the annotation back:

// src/params.ts
import { defineParams } from '@sveltejs/kit/params';

export const params = defineParams({
  slug: (param: string) => (/^[a-z0-9-]+$/.test(param) ? param : undefined)
});

Notice the new contract: a matcher returns the parsed value when it matches and undefined when it does not, instead of returning true or false. That lets a matcher both validate and convert (for example, turn "42" into the number 42).

Gotcha 4 — everything from section 3 and 4

To keep the list in one place: the npm 10 install crash, TypeScript 7 conflicts, third-party plugins pinned to @sveltejs/kit@^2, and CommonJS default imports that turn into undefined under Vite 8. None of these appear in svelte-check or build output.

Read the matrix column by column and the pattern is obvious: the build column is empty. A successful build proves your code compiles under Vite 8. It says nothing about whether your links point to the right host or whether a button handler references a function that still exists.

8. Verifying the migration: check, build, then use it

Each of these three checks answers a different question. Run all of them.

Check 1 — npm run check asks "do my types agree?"

npm run check
# svelte-check found 0 errors and 0 warnings

svelte-check runs the TypeScript compiler over your .ts files and your .svelte templates. It catches missing imports (Gotcha 1), lost types (Gotcha 3), mutations of the now read-only page.url, and old error(404, { message }) calls. Because SvelteKit 3 generates a stricter $app/tsconfig, expect a few new errors even in code the migrator never touched. Those are usually real bugs that were hiding.

Check 2 — npm run build asks "does it bundle under Vite 8?"

npm run build && npm run preview

The build exercises Rolldown, your adapter and your src/env.ts validation. If a required environment variable is missing, the build stops here, which is exactly what you want. Read the output for PLUGIN_TIMINGS warnings too: Rolldown reports how much build time each Vite plugin consumes, which is useful if the upgrade did not feel faster.

Check 3 — a smoke test asks “does it actually work in a browser?”

The two worst bugs in section 7 only appear after hydration in a real browser. You can catch them with a manual click-through, or automate it with a tiny Playwright script. This one opens each page you list, waits for hydration, flags any link that points off-site and any JavaScript error, then clicks every button.

// scripts/smoke.mjs — run against `npm run preview` (default port 4173)
import { chromium } from 'playwright';


const BASE = process.env.BASE_URL ?? 'http://localhost:4173';
const PAGES = ['/', '/blog', '/about'];      // the routes you care about
const problems = [];
const browser = await chromium.launch();
const page = await browser.newPage();
// 1. Record uncaught JavaScript errors (e.g. "pushState is not defined")
page.on('pageerror', (err) => problems.push(`[${page.url()}] JS error: ${err.message}`));
for (const path of PAGES) {
  // 2. 'networkidle' waits until hydration has finished and links are final
  await page.goto(BASE + path, { waitUntil: 'networkidle' });
  // 3. Every  should stay on our own origin (catches //blog/... links)
  const hrefs = await page.$$eval('a[href]', (links) => links.map((a) => a.href));
  for (const href of hrefs) {
    if (new URL(href).origin !== new URL(BASE).origin) {
      problems.push(`[${path}] link leaves the site: ${href}`);
    }
  }
  // 4. Click every visible button and let any errors surface
  for (const button of await page.getByRole('button').all()) {
    if (await button.isVisible()) await button.click({ trial: false }).catch(() => {});
    if (page.url() !== BASE + path) await page.goto(BASE + path, { waitUntil: 'networkidle' });
  }
}
await browser.close();
console.log(problems.length ? problems.join('\n') : 'Smoke test passed');
process.exit(problems.length ? 1 : 0);

How it works, line by line:

Run it in two terminals:

npm i -D playwright && npx playwright install chromium
npm run build && npm run preview     # terminal 1
node scripts/smoke.mjs               # terminal 2

This is a smoke test, not a test suite. On a larger app, the same three ideas (listen for page errors, check link origins after hydration, exercise interactive elements) belong in your existing end-to-end tests.

Bonus check — reproduce a fresh install

Install-time failures never reach any of the checks above, because your machine already has a working node_modules. Simulate a new teammate:

git clone  /tmp/fresh && cd /tmp/fresh
rm -f package-lock.json
npm install          # must succeed on the npm version your team actually uses

Node.js 26 LTS: Map Upsert, Iterator.concat and Other Small Features That Delete Utility Code

Vasanthan K — Mon, 02 Nov 2026 14:01:02 GMT

Node.js 26 became the new LTS line this week. The headline features are big, but the ones that will change your day-to-day code are small: two new Map methods and one new Iterator function that each replace a helper you have written many times.

Every Node.js codebase I have worked on has a utils folder. Somewhere inside it there is a groupBy helper, a getOrCreate helper, a chain or concatIterables generator, and a date helper that adds a month without breaking on January 31. Nobody is proud of these files. They exist because the language did not give us a clean way to do very ordinary things.

Node.js 26 LTS deletes a good chunk of that folder. In this post we will walk through:

Map upsert — Map.prototype.getOrInsert() and getOrInsertComputed() (and the same two on WeakMap).
Iterator.concat() — join any number of iterables into one lazy stream.
Other small wins — Temporal switched on by default, and built-in Base64/Hex on Uint8Array.
Upgrade notes — the few removals that can break your app when you move from Node.js 24.

Every section starts from zero. If you know what a Map is and have written a for...of loop, you have enough background. Where an idea needs theory — why an argument is evaluated early, what "lazy" actually means — we will slow down and draw it.

First, what does “Node.js 26 LTS” actually mean?

On October 28, 2026 the Node.js project shipped v26.13.0 under the codename “Lithium” and moved the 26.x line into Long Term Support. That release contains no new code compared with 26.12.1 — it only flips the metadata. The features themselves arrived months earlier, when 26.0.0 shipped on May 5, 2026 as a “Current” release.

If the release labels are new to you, here is the short version:

Current — the newest major. It gets new features quickly and can still change. Good for trying things, not for production.
Active LTS — the same major, now frozen for features and focused on bug and security fixes. This is the line most teams should run in production.
Maintenance LTS — critical fixes only, until end of life. Node.js 26 is supported until the end of April 2029.

There is one more reason this release matters: 26 is the last major on the old “two releases a year, only even numbers become LTS” schedule. Starting with Node.js 27 (April 2027), the project moves to one major release per year, and every major becomes LTS. So if you skipped Node.js 24, 26 is the natural jump.

Which engine features did we get?

Node.js 26 ships the V8 14.6 JavaScript engine (the one inside Chromium 146). The release notes call out two new language features that come with it — and those are the stars of this post:

Upsert: Map.prototype.getOrInsert(), Map.prototype.getOrInsertComputed() and the same pair on WeakMap.
Iterator sequencing: Iterator.concat().

Both are part of the ECMAScript 2026 standard, which means they are not Node-only tricks: the same code runs in current browsers too.

How to follow along

Install Node.js 26 with your version manager and confirm you are on the LTS build:

nvm install 26
nvm use 26
node -p "process.version + ' ' + process.release.lts"
# v26.13.0 Lithium

Then do a quick feature check. Each line should print function:

// check.mjs
console.log(typeof Map.prototype.getOrInsert);
console.log(typeof Map.prototype.getOrInsertComputed);
console.log(typeof Iterator.concat);

If you see undefined, you are still on an older Node.js. Every example below is a plain .mjs file you can run with node file.mjs.

Part 1 — Map upsert: “get it, or create it” in one call

The word upsert is borrowed from databases: update if the row exists, insert if it doesn’t. In JavaScript the idea is simpler still: give me the value for this key — and if there isn’t one yet, put a default in first.

A 30-second refresher on Map

A Map stores key → value pairs. Unlike a plain object, its keys can be anything (strings, numbers, objects), it remembers insertion order, and it has a clean API:

const prices = new Map();
prices.set('apple', 1.2);   // add or overwrite
prices.get('apple');        // 1.2
prices.get('mango');        // undefined (missing key)
prices.has('mango');        // false
prices.size;                // 1

Notice that get() on a missing key returns undefined. That one fact is the root of the pattern we are about to delete.

The problem: the has / set / get dance

Say we want to count how often each word appears. Before Node.js 26 you would write something like this:

const words = ['node', 'map', 'node', 'iterator', 'map', 'node'];
const counts = new Map();

for (const w of words) {
  if (!counts.has(w)) {      // 1. is the key there?
    counts.set(w, 0);        // 2. if not, insert a starting value
  }
  counts.set(w, counts.get(w) + 1); // 3. read it back and update
}
console.log(counts);
// Map(3) { 'node' => 3, 'map' => 2, 'iterator' => 1 }

It works, but look at what it costs:

The key is written three or four times. Every repetition is a chance to typo w as word in one place and not the others.
The map is searched up to three times (has, set, get) for one logical operation.
The intent is buried. The reader has to decode an if with a negation to understand "start at zero".

Grouping has exactly the same shape — check, create an empty array, read it back, push:

if (!byCustomer.has(order.customer)) {
  byCustomer.set(order.customer, []);
}
byCustomer.get(order.customer).push(order.id);

Because this is tedious, people reach for a “shortcut” that contains a classic bug:

const groups = new Map();
(groups.get('asha') || []).push(1);   // looks fine...
console.log(groups.size);             // 0  <-- the array was never stored!

The || [] creates a brand-new array when the key is missing, you push into it, and then it is thrown away because nobody called set(). The bug is silent: no error, just missing data.

Meet getOrInsert(key, defaultValue)

getOrInsert does all three steps in one call:

If key is already in the map, it returns the existing value and changes nothing.
If key is missing, it stores defaultValue under key and returns it.

Either way, you always get a real value back — never undefined for a missing key. Here is the word counter again:

const counts = new Map();

for (const w of words) {
  counts.set(w, counts.getOrInsert(w, 0) + 1);
}
console.log(counts);
// Map(3) { 'node' => 3, 'map' => 2, 'iterator' => 1 }

Read the loop body from the inside out: “get the count for w (starting at 0 if it's new), add one, store it." One line, the key written twice instead of four times, and no if.

Why is there still a set()? Numbers are values, not containers. getOrInsert hands you the number 0; adding 1 to it creates a new number that the map knows nothing about. So for counters you still write the result back. For containers (arrays, Sets, objects) you don't — you mutate the container the map already holds, as the next example shows.

Meet getOrInsertComputed(key, callback)

The second method takes a function instead of a ready-made value. The map only calls that function when the key is missing, and it passes the key in as the argument.

Grouping orders by customer becomes:

const orders = [
  { id: 1, customer: 'asha' },
  { id: 2, customer: 'ben' },
  { id: 3, customer: 'asha' },
];

const byCustomer = new Map();
for (const order of orders) {
  byCustomer.getOrInsertComputed(order.customer, () => []).push(order.id);
}
console.log(byCustomer);
// Map(2) { 'asha' => [ 1, 3 ], 'ben' => [ 2 ] }

Step by step, for the first order (asha): the key is missing, so the map calls () => [], stores the new empty array under 'asha', and returns that same array — which we immediately .push(1) into. For the third order (asha again) the key exists, the callback is skipped, and we get the array that already holds [1]. Because it is the very same array object, pushing into it updates the map. No set() needed.

The callback receives the key, which is handy when the default depends on it:

const squares = new Map();
squares.getOrInsertComputed(7, (n) => n * n); // 49 — computed and stored
squares.get(7);                               // 49

The theory: why are there two methods?

This is the most important idea in the whole feature, so let’s slow down.

In JavaScript, every argument is evaluated before the function you’re calling starts running. When you write f(a(), b()), the engine first runs a(), then b(), and only then calls f with the two results. f has no say in the matter.

Apply that to getOrInsert:

cache.getOrInsert('user:1', loadUser());

loadUser() runs first, every time — even if 'user:1' is already cached. getOrInsert then notices the key exists, returns the old value, and the freshly built object is thrown away. If loadUser() is expensive (a database query, a big allocation, a log line), you pay for it on every call.

getOrInsertComputed avoids this because you pass the function itself, not its result. A function is just a recipe; nothing happens until someone calls it — and the map only calls it on a miss. This idea is called lazy evaluation: do the work only when the result is actually needed.

You can prove it with a counter:

let calls = 0;
const makeExpensive = () => { calls++; return { createdAt: 'now' }; };
const cache = new Map([['user:1', { createdAt: 'earlier' }]]);

cache.getOrInsert('user:1', makeExpensive());       // eager
console.log(calls); // 1  <- ran even though the key existed

cache.getOrInsertComputed('user:1', makeExpensive); // lazy, key exists
console.log(calls); // 1  <- callback skipped

cache.getOrInsertComputed('user:2', makeExpensive); // lazy, key missing
console.log(calls); // 2  <- ran only on the miss

Rule of thumb:

Use getOrInsert for cheap constants: 0, '', false, null.
Use getOrInsertComputed for anything you build: [], {}, new Set(), new Map(), a calculation, a network call.

There is a second, sneakier reason for that rule. Look at what happens if you try to “optimise” by creating the empty array once:

const EMPTY = [];
const groups = new Map();
groups.getOrInsert('asha', EMPTY).push(1);
groups.getOrInsert('ben', EMPTY).push(2);

console.log(groups.get('asha'));                       // [ 1, 2 ]  <- ben's order leaked in
console.log(groups.get('asha') === groups.get('ben')); // true

Both keys now point at the same array in memory, so every push lands in every group. With getOrInsertComputed(key, () => []), the arrow function builds a fresh array on each miss, so every key gets its own.

Real-world recipes

1. Memoization (caching function results). The callback receives the key, so a pure function can be plugged in directly:

const memo = new Map();
const fib = (n) => memo.getOrInsertComputed(n, (k) => (k < 2 ? k : fib(k - 1) + fib(k - 2)));

console.log(fib(40)); // 102334155 - instant, each n computed once

2. Graphs as adjacency lists. Each node maps to a Set of neighbours. The second call makes sure nodes that only appear as targets still exist:

const edges = [['A', 'B'], ['A', 'C'], ['B', 'C'], ['C', 'A']];
const graph = new Map();

for (const [from, to] of edges) {
  graph.getOrInsertComputed(from, () => new Set()).add(to);
  graph.getOrInsertComputed(to, () => new Set());
}
// Map(3) { 'A' => Set { 'B', 'C' }, 'B' => Set { 'C' }, 'C' => Set { 'A' } }

3. Nested maps without nested ifs. Because each call returns the value, you can chain them — tenant → user → session ids:

const index = new Map();
for (const s of sessions) {
  index
    .getOrInsertComputed(s.tenant, () => new Map())
    .getOrInsertComputed(s.user, () => [])
    .push(s.id);
}
// Map { 'acme' => Map { 'u1' => ['s1','s2'], 'u2' => ['s3'] }, 'globex' => Map { 'u9' => ['s4'] } }

The old version of this needed two has checks, two sets and two gets.

4. De-duplicating in-flight requests. If ten callers ask for the same URL at once, only one network request should go out. Store the promise:

const inFlight = new Map();

function fetchOnce(url) {
  return inFlight.getOrInsertComputed(url, () =>
    fetch(url).finally(() => inFlight.delete(url)) // forget it once settled
  );
}
await Promise.all([fetchOnce('/a'), fetchOnce('/a'), fetchOnce('/b')]);
// only 2 requests are made: one for /a, one for /b

The laziness matters here: with getOrInsert(url, fetch(url)) every caller would fire its own request before the map could say "already in flight".

What about Map.groupBy?

If you already have the whole array in hand and just want it grouped once, Map.groupBy(orders, (o) => o.customer) (available since Node.js 21) is even shorter. Reach for upsert when data arrives over time — events, stream chunks, messages — or when you need something other than an array per key (a counter, a Set, a nested Map, a promise).

The WeakMap versions

WeakMap gets the same two methods. A WeakMap only accepts objects (and non-registered symbols) as keys, and it holds them weakly: when nothing else in your program references the key object, the garbage collector may remove it and its value. That makes it perfect for attaching metadata to objects you don't own, without causing memory leaks:

const meta = new WeakMap();
const getMeta = (obj) => meta.getOrInsertComputed(obj, () => ({ hits: 0 }));

const req = { url: '/health' };
getMeta(req).hits++;
getMeta(req).hits++;
console.log(getMeta(req).hits); // 2
// when `req` is no longer referenced anywhere, its metadata can be collected too

Passing a string or number as the key throws a TypeError, exactly like WeakMap.prototype.set() does.

Small print

getOrInsertComputed throws a TypeError if the second argument isn't a function — map.getOrInsertComputed('k', 42) fails fast instead of storing 42.
Keys are compared the same way as everywhere else in Map: primitives by value, objects by reference. Two different { id: 1 } objects are two different keys.
Keep callbacks simple. Don’t make them insert the same key they’re computing; the value the callback returns is what ends up stored.
Need this on an older runtime or browser? core-js and the es-shims packages ship polyfills for both methods.

Part 2 — Iterator.concat(): one lazy stream from many sources

Iterator.concat(a, b, c) gives you a single iterator that yields everything from a, then everything from b, then everything from c — without copying anything into a new array first. To see why that matters, we need two words that confuse a lot of people: iterable and iterator.

Iterables and iterators, in plain words

Think of a music playlist.

The playlist is the iterable: a thing that can be played through. Arrays, strings, Sets, Maps and generator objects are all iterables. Technically, an iterable is any object with a [Symbol.iterator]() method.
The “next track” button is the iterator: a cursor that remembers where you are. Each press — each call to .next() — gives you one item, as { value, done }.

const it = [10, 20][Symbol.iterator](); // ask the array for a cursor

it.next(); // { value: 10, done: false }
it.next(); // { value: 20, done: false }
it.next(); // { value: undefined, done: true }  <- finished

You rarely call .next() by hand. for...of, the spread operator [...x], Array.from(x) and destructuring all do it for you behind the scenes.

Since Node.js 22, iterators also have helper methods that work like the array methods you know — .map(), .filter(), .take(), .drop(), .flatMap(), .find(), .some(), .toArray() and friends — but they process one item at a time instead of building whole arrays. Iterator.concat() is the missing piece that lets you start such a chain from several sources at once.

The problem: joining several sources

Suppose a web framework has built-in routes in an array, plugin routes in a Set, and a generator that produces fallback routes. You want to loop over all of them. Before Node.js 26 you had two choices.

Option A — spread everything into one array:

const all = [...builtIn, ...pluginRoutes, ...fallbacks()];

Short, but eager: it reads every value from every source and allocates a brand-new array to hold them, even if you only need the first match. It cannot work at all with a source that never ends.

Option B — write your own generator (again):

function* chain(...iterables) {
  for (const it of iterables) yield* it;
}
const all = chain(builtIn, pluginRoutes, fallbacks());

This one is lazy, but it’s another helper to copy between projects, test and name — and it doesn’t validate its inputs until you start looping.

Meet Iterator.concat()

function* lows() { yield 1; yield 2; yield 3; }
const highs = new Set([7, 8, 9]);

const digits = Iterator.concat(lows(), [4, 5, 6], highs);
console.log([...digits]); // [1, 2, 3, 4, 5, 6, 7, 8, 9]

Three different kinds of source — a generator, an array and a Set — become one iterator. The values come out in exactly the order the sources were passed. The result is a real Iterator, so every helper method is available on it immediately.

The theory: what “lazy” really means here

When you call Iterator.concat(...), no values are read yet. You just get back a cursor. Work only happens when somebody calls .next() on it, and then:

The request is forwarded to the currently active source (at the start, the first one).
If that source has a value, it is handed straight back to the caller.
If that source is finished, Iterator.concat moves on and opens the next source — not before.
When the last source is finished, it reports { done: true }.

You can watch this happen by logging from inside the sources:

const order = [];
function* source(name, values) {
  for (const v of values) { order.push(`${name} gives ${v}`); yield v; }
}

const all = Iterator.concat(source('A', [1, 2]), source('B', [3, 4]));
order.push('nothing pulled yet');
const first = all.next().value;
order.push(`consumer got ${first}`);
console.log(order);
// [ 'nothing pulled yet', 'A gives 1', 'consumer got 1' ]

Notice that 'nothing pulled yet' is logged before source A produced anything, and source B hasn't run at all. That is laziness in one picture: the consumer drives the work, one value at a time.

Combining it with iterator helpers

This is where Iterator.concat() really earns its place. Build one stream, then shape it:

const builtIn  = ['/health', '/login', '/admin/users'];
const plugins  = new Set(['/blog', '/admin/settings']);
const fallback = ['/404'];

const publicRoutes = Iterator.concat(builtIn, plugins, fallback)
  .filter((r) => !r.startsWith('/admin'))
  .map((r) => r.toUpperCase())
  .toArray();
console.log(publicRoutes); // [ '/HEALTH', '/LOGIN', '/BLOG', '/404' ]

No intermediate arrays are created between concat, filter and map. Each route travels through the whole chain on its own; only the survivors land in the final array built by .toArray().

And because nothing is read ahead, stopping early really stops the work:

let pulledFromSecond = 0;
function* second() {
  for (let i = 0; i < 1_000_000; i++) { pulledFromSecond++; yield i; }
}

const firstThree = Iterator.concat(['a', 'b'], second()).take(3).toArray();
console.log(firstThree);       // [ 'a', 'b', 0 ]
console.log(pulledFromSecond); // 1  <- not 1,000,000

The same property lets you mix in infinite sources safely, as long as something downstream limits how much you take:

function* naturals() { let n = 1; while (true) yield n++; }

Iterator.concat(['start'], naturals()).take(5).toArray();
// [ 'start', 1, 2, 3, 4 ]

More everyday uses

Merging Maps (later sources win). A Map iterates as [key, value] pairs, and the Map constructor accepts pairs, so concatenating maps is merging them:

const defaults = new Map([['port', 3000], ['host', 'localhost']]);
const fromEnv  = new Map([['port', 8080]]);

const config = new Map(Iterator.concat(defaults, fromEnv));
// Map(2) { 'port' => 8080, 'host' => 'localhost' }

Reading paged data as one stream. If each page is a generator, the caller never has to think about page boundaries:

function* page(n) { for (let i = 1; i <= 3; i++) yield `p${n}-item${i}`; }

Iterator.concat(page(1), page(2), page(3)).take(5).toArray();
// [ 'p1-item1', 'p1-item2', 'p1-item3', 'p2-item1', 'p2-item2' ]

Page 3 is never touched.

Rules and gotchas

1. An iterator can only be read once. It’s a cursor, not a container. Once it reaches the end, it stays there:

const once = Iterator.concat([1, 2]);
console.log([...once]); // [ 1, 2 ]
console.log([...once]); // []  <- already exhausted

If you need to loop twice, call Iterator.concat(...) twice (or .toArray() once and reuse the array).

2. Every argument must be an iterable object — checked immediately. The arguments are validated the moment you call Iterator.concat, not later when you start looping. Two cases surprise people:

Iterator.concat('ab');   // TypeError — a string is a primitive, not an object

const bare = {           // has next() but no [Symbol.iterator]()
  i: 0,
  next() { return this.i < 2 ? { value: this.i++, done: false } : { value: undefined, done: true }; },
};
Iterator.concat(bare);   // TypeError - an iterator that isn't iterable

The fix for both is Iterator.from(), which wraps anything iterable-ish into a proper Iterator:

[...Iterator.concat(Iterator.from('ab'), [1])]; // [ 'a', 'b', 1 ]
[...Iterator.concat(Iterator.from(bare))];      // [ 0, 1 ]

Why so strict? Iterator.concat takes ownership of the iterators it opens and promises to close them. With a bare iterator it would be unclear who is responsible for closing it, so the standard simply refuses.

3. Stopping early closes the active source for you. If you break out of a loop (or .take() stops), the source currently being read gets its cleanup code run — the finally block of a generator, for example. Sources that were never reached are never opened, so there is nothing to clean up:

function* resource(name) {
  try { yield `${name}-1`; yield `${name}-2`; }
  finally { console.log(`${name} closed`); }
}

for (const v of Iterator.concat(resource('fileA'), resource('fileB'))) {
  console.log('read', v);
  if (v === 'fileA-1') break;
}
// read fileA-1
// fileA closed      <- fileB was never opened

This is the behaviour you want for file handles, database cursors and anything else that must be released.

4. It is synchronous only. Async generators and streams are async iterables and will throw a TypeError. For those, a tiny async function* with for await and yield* is still the tool.

5. Huge or unknown numbers of sources? Iterator.concat(...arrayOfIterables) spreads every source into arguments. When you have thousands of sources — or an endless list of them — use .flatMap() on an iterator of iterables instead: Iterator.from(listOfSources).flatMap((s) => s).

Part 3 — Other small features that delete utility code

Map upsert and Iterator.concat() are the new language features in V8 14.6. Two more changes make Node.js 26 LTS a good moment to clean out the utils folder — especially if you're jumping straight from Node.js 24.

Temporal is on by default

Temporal is the modern replacement for JavaScript's Date object, and Node.js 26 is the first release where it is enabled by default — no flag, no polyfill. It deserves its own article, so here are only the parts that replace common helpers.

The problem with Date in one example — adding a month to January 31:

const d = new Date(2026, 0, 31); // Jan 31, 2026
d.setMonth(d.getMonth() + 1);
console.log(d.toDateString());   // 'Tue Mar 03 2026'  <- overflowed into March

Date happily creates "February 31" and rolls it forward three days. Most teams have a helper (or a library) just to avoid this. With Temporal:

const jan31 = Temporal.PlainDate.from('2026-01-31');

jan31.add({ months: 1 }).toString();  // '2026-02-28'  <- clamped to the last valid day
jan31.toString();                     // '2026-01-31'  <- the original is untouched

Two ideas make Temporal easier to reason about than Date:

Separate types for separate questions. A PlainDate is a calendar date with no time and no time zone (a birthday, a due date). A ZonedDateTime is an exact moment in a specific zone (a meeting). You pick the type that matches your data, so a birthday can never shift by a day because a server runs in UTC.
Everything is immutable. Methods like .add() return a new object instead of changing the old one, so you can't accidentally modify a date someone else is holding.

A few more one-liners that used to need helpers:

// Date arithmetic
Temporal.PlainDate.from('2026-10-30').add({ days: 45 }).toString();
// '2026-12-14'

// "How long until...?" as a human-readable duration
Temporal.PlainDate.from('2026-10-28')
  .until('2029-04-30', { largestUnit: 'years' })
  .toString();
// 'P2Y6M2D'  -> 2 years, 6 months, 2 days

// Time-zone conversion
Temporal.ZonedDateTime.from('2026-11-02T09:00[America/New_York]')
  .withTimeZone('Asia/Kolkata')
  .toString();
// '2026-11-02T19:30:00+05:30[Asia/Kolkata]'

// Today's date in the system time zone
Temporal.Now.plainDateISO();

You don’t have to rewrite every Date in your codebase. A good first step is to use Temporal in new code that does calendar maths or crosses time zones — exactly the places where date libraries and hand-written helpers live today.

Base64 and Hex built into Uint8Array

These methods landed in Node.js 25, so Node.js 26 is the first LTS that has them. They let any Uint8Array convert to and from Base64 and hex strings directly:

const bytes = new TextEncoder().encode('Hello, Node 26!');

bytes.toBase64(); // 'SGVsbG8sIE5vZGUgMjYh'
bytes.toHex();    // '48656c6c6f2c204e6f646520323621'

const back = Uint8Array.fromBase64('SGVsbG8sIE5vZGUgMjYh');
new TextDecoder().decode(back); // 'Hello, Node 26!'

Uint8Array.fromHex('cafe');     // Uint8Array(2) [ 202, 254 ]

URL-safe Base64 (for tokens, JWT segments and file names) is an option, not another helper:

const raw = new TextEncoder().encode('Hi?>');

raw.toBase64();                                           // 'SGk/Pg=='
raw.toBase64({ alphabet: 'base64url', omitPadding: true }); // 'SGk_Pg'

In pure Node.js code, Buffer.from(bytes).toString('base64') still works fine. The win is portability: the same lines run unchanged in current browsers, where these methods are part of the standard too. If you share code between a server and a frontend, you can delete the btoa(String.fromCharCode(...bytes)) dance and the per-platform if (typeof Buffer !== 'undefined') branches.

Part 4 — Before you upgrade: what can break

Small features are the fun part, but an LTS jump is also when old code stops working. These are the changes from the Node.js 26.0.0 release notes most likely to affect an app coming from Node.js 24:

One more operational note from the LTS announcement: since Node.js 25, the official Linux binaries are linked against libatomic, which some slim container images don't include. If node fails to start in Docker after the upgrade, install that package first.

A safe rollout path: run your test suite on Node.js 26 in CI alongside 24, fix what fails, then switch production. Once you’re on 26, search the codebase for the old patterns (.has( followed by .set(, function* chain, yield* helpers, btoa() and replace them as you touch each file.

Cheat sheet: what to delete from your utils folder

Wrapping up

None of these features will make a conference keynote. That’s the point. They remove small, repeated pieces of code that every team writes slightly differently — and every different version is a place for a bug to hide.

If you remember only three things:

getOrInsertComputed(key, () => default) is the safe default for "get it, or create it". Use getOrInsert(key, value) only for cheap constants like 0, because its argument is always evaluated.
Iterator.concat(a, b, c) joins sources lazily. Nothing is read until you ask, stopping early really stops, and cleanup runs for you. Wrap strings and bare iterators in Iterator.from() first.
Node.js 26 is the LTS to move to. It’s supported until April 2029 and is the last release on the old schedule. Check the removals table above, then start deleting helpers.

Sources

Node.js 26.0.0 (Current) release notes — V8 14.6, upsert, Iterator.concat(), Temporal, removals
Node.js 26.13.0 ‘Lithium’ (LTS) release notes — LTS date and support window
MDN: Map.prototype.getOrInsertComputed()
MDN: Iterator.concat()
TC39 iterator sequencing specification — argument validation rules
What’s new in Node.js 25 (Appwrite) — Uint8Array Base64/Hex arrival

Building the Memory Layer of My Agent Harness: What I Learned Wiring Persistent Context by Hand

Vasanthan K — Fri, 30 Oct 2026 13:01:03 GMT

An LLM forgets everything the moment a request ends. Here is how I built a memory layer for my multi-agent harness in TypeScript, from a plain interface to semantic recall, token budgets and forgetting, and the mistakes I made along the way.

A few weeks into building my own agent harness, I asked my coding agent to “use the same folder structure as last time.” It answered politely and confidently, and it was completely wrong. There was no “last time” for it. Every request my harness sent to the model started from a blank slate.

That moment changed what I was building. I had a router and a small team of specialized agents working, inspired by tools like Ruflo (formerly claude-flow). But without memory, my “team” behaved like a group of brilliant contractors who had never met each other and had never seen the project before. Each one did good work in isolation, then forgot all of it.

I could have installed an existing memory library and moved on. Instead, I decided to wire persistent context by hand. My goal was not to build the best memory system in the world. My goal was to understand what a memory system actually does, piece by piece, so that the next time I use a framework I know what is happening underneath.

This post is the write-up of that journey. By the end you will understand:

Why language models need an external memory at all, and what the context window really is
The four kinds of memory an agent can have, and which ones are worth building first
How to design a MemoryStore interface and back it with SQLite
How embeddings and cosine similarity let an agent recall things by meaning, not by keywords
How to rank memories by relevance, recency and importance
How to pack the best memories into a prompt without blowing your token budget
How lifecycle hooks write new memories back automatically
How to share memory safely between agents, and how to make the system forget

I have written this for beginners. If you know basic TypeScript and have called an LLM API once, you can follow every step. Each code block is followed by an explanation of what it does and, more importantly, why it is designed that way.

Why an agent needs memory at all

An LLM is stateless: it remembers nothing between API calls, so any continuity you see is something the application rebuilt and sent back in.

This surprises a lot of people, because chat apps feel like they remember. They do not. When you chat with a model, the app quietly re-sends the whole conversation with every new message. The model reads it from the top each time, like an actor who is handed the full script before every single line.

The same two requests, with and without a memory layer. The model is identical in both columns; only the prompt changes.

The context window is RAM, not a hard disk

The best mental model I found is to treat the context window like a computer’s RAM. It is fast, the model can “see” everything in it at once, and it is the only place the model can think. But it is also limited and it is wiped after every request.

A memory layer is the hard disk. It stores everything durable outside the model, and before each request it loads only the small slice that matters into RAM. That is the entire idea. Everything else in this post is detail about doing that loading well.

This analogy explains three constraints that shaped my design:

Space is limited. Even with very large context windows, every token costs money and latency. Worse, models get measurably less accurate when the important fact is buried in the middle of a huge prompt. More context is not automatically better context.
Loading is selective. You cannot load the whole disk. You need a way to decide which memories are worth the space for this request.
Writing is deliberate. Nothing is saved automatically. If you want the agent to remember something tomorrow, some code must decide to write it down today.

Why it is harder in a multi-agent harness

My harness has a router that sends each task to one of several specialized agents: a planner, a coder and a reviewer. With a single chatbot, “memory” mostly means chat history. With a team of agents, new questions appear:

Should the reviewer see what the coder learned? (Usually yes.)
Should the coder’s scratch notes leak into the planner’s prompt? (Usually no.)
If two agents learn the same fact, do we store it twice?
If a fact becomes wrong (we switched from npm to pnpm), who updates it?

These questions forced me to think about memory as a proper subsystem with its own interface, its own storage and its own rules, not as an array of messages I append to.

The mental model: four kinds of memory

Not all memory is the same, and treating it as one big bucket was my first mistake. Cognitive scientists split human memory into types, and that split maps surprisingly well onto agents.

Working and session memory live inside the prompt. Semantic and episodic memory live outside it and are recalled on demand.

1. Working memory is simply what is in the prompt right now: the system prompt, the tool definitions, the user’s task. It exists for exactly one API call. You already have this; every LLM app does.

2. Session memory is the recent history of the current conversation or task. In a chat app this is the message list. It is the easiest kind to implement (keep an array, send it back each time) and it is where most tutorials stop. Its problem is that it grows without limit and dies when the session ends.

3. Semantic memory holds facts and preferences that stay true across sessions. “This project uses pnpm.” “Vasanthan prefers TypeScript strict mode.” “The API base URL is set in config/env.ts." These are not tied to any particular moment. They are knowledge.

4. Episodic memory holds records of things that happened. “Last Tuesday the refactor of the router broke three tests because of a circular import.” Episodes carry lessons. They help an agent avoid repeating a mistake, which is something semantic facts alone cannot do.

Some writers add a fifth type, procedural memory, meaning learned skills or workflows (“how we deploy”). In my harness I store procedures as a special kind of semantic memory, so I will not treat it separately here.

Which ones should you build first?

Working and session memory you get almost for free. The interesting engineering is in semantic and episodic memory, because they live outside the model and must be recalled. That recall step is where all the hard questions live: what to fetch, how much, and in what order.

So the rest of this post is about building an external store for semantic and episodic memories and a pipeline that moves the right ones into the prompt at the right time.

The architecture: where memory sits in the harness

The single most useful decision I made was to put one component, the MemoryManager, between the agents and everything memory-related. Agents never touch the database, the embedding model or the hooks directly.

Each agent has its own namespace, but all memory traffic flows through one MemoryManager.

Here is what each piece does:

Router. Reads the incoming task and picks the agent best suited to it. It existed before memory did.
Agents (planner, coder, reviewer). Each one has a system prompt, a set of tools and a namespace such as agent:coder. The namespace is like a folder label on every memory the agent writes.
MemoryManager. The public API of the memory layer. It exposes five verbs: recall() finds candidate memories, rank() orders them, buildContext() packs the winners into a token budget, remember() writes new ones, and decay() handles forgetting.
SQLite store. Durable storage on disk. One file, zero servers.
Embedder. Turns text into a vector (a list of numbers) so we can search by meaning.
Hooks. Functions the harness calls at fixed moments in the agent lifecycle, such as after every turn or at the end of a session. This is where automatic memory writing lives.

Why a manager instead of letting agents query the DB?

This is classic separation of concerns, and it paid off three times:

Swappable storage. I started with an in-memory array, moved to SQLite, and could move to Postgres with pgvector later. The agents never noticed.
One place for policy. Rules like “never store secrets” or “the reviewer may read the coder’s memories but not write to them” live in one file instead of being scattered across agents.
Testability. I can unit test ranking and budgeting with fake data and no LLM calls at all.

If you remember one thing from this section: design memory as a service with a small interface, not as a pile of database calls. The next step shows that interface.

Step 1: Define what a memory is

Before writing any storage code, I wrote down the shape of a single memory. Getting this type right saved me from at least two painful migrations later.

// src/memory/types.ts

export type MemoryKind = "fact" | "preference" | "episode" | "procedure";
export interface MemoryRecord {
  id: string;              // unique id, e.g. a UUID
  namespace: string;       // "agent:coder", "shared", "user"
  kind: MemoryKind;        // what sort of memory this is
  content: string;         // the memory itself, in plain language
  embedding: number[];     // vector form of `content`, used for semantic search
  importance: number;      // 0..1, how much this matters (set at write time)
  createdAt: number;       // epoch milliseconds
  lastAccessedAt: number;  // updated every time the memory is recalled
  accessCount: number;     // how many times it has been recalled
  source?: string;         // where it came from, e.g. "turn:8f2c" or "user"
}
export interface RecallQuery {
  text: string;            // what we are looking for, in natural language
  namespaces: string[];    // which namespaces this agent may read
  limit?: number;          // max candidates to return (default 20)
  kinds?: MemoryKind[];    // optional filter by kind
}
export interface ScoredMemory {
  record: MemoryRecord;
  relevance: number;       // cosine similarity to the query
  score: number;           // final blended score after ranking
}
export interface MemoryStore {
  insert(record: MemoryRecord): Promise;
  update(id: string, patch: Partial): Promise;
  delete(id: string): Promise;
  listByNamespaces(namespaces: string[], kinds?: MemoryKind[]): Promise;
}

What each field is for

If you are new to this, every field can look arbitrary. Each one earns its place by answering a question the system will ask later.

namespace answers "who owns this and who may see it?" Without it, the planner's half-formed ideas would leak into the coder's prompt. We will use it in Step 7.
kind answers "how should this be treated?" A preference rarely expires; an episode about yesterday's failed build becomes less useful each week. Different kinds can get different decay rates.
content is stored in plain natural language, not JSON. This was deliberate. The content ends up inside a prompt, and models read sentences better than they read serialized objects. It also makes the database readable by a human when you are debugging at midnight.
embedding is the numeric fingerprint of the content. It lets us ask "which memories mean something similar to this question?" instead of "which memories contain these exact words?" Step 3 explains it fully.
importance answers "how bad would it be to forget this?" "The production database is read-only" deserves 0.95. "The user said thanks" deserves 0.05.
createdAt, lastAccessedAt, accessCount answer "is this still alive?" A memory that keeps being recalled is clearly useful. One that nobody has touched in months is a candidate for forgetting.
source answers "why do we believe this?" When a memory turns out to be wrong, the source tells you which turn or which user message created it.

Why MemoryStore is so small

Notice that the MemoryStore interface has only four methods and none of them mention similarity search, scoring or tokens. That is on purpose. The store's only job is to persist records. All the intelligence (ranking, budgeting, deduplication) belongs in the MemoryManager above it.

This split is the Repository pattern. It means the store is easy to replace and the clever logic is easy to test. If you later move to a real vector database, you would add a search(embedding, k) method to the store for speed, but the manager's public API would not change.

Step 2: Make it persistent with SQLite

I chose SQLite because a memory layer for a personal harness does not need a server. It needs a single file that survives restarts, supports transactions, and can be inspected with any SQLite viewer. The better-sqlite3 package gives Node a fast, synchronous driver.

npm install better-sqlite3
npm install -D @types/better-sqlite3

// src/memory/sqliteStore.ts
import Database from "better-sqlite3";
import { MemoryKind, MemoryRecord, MemoryStore } from "./types";

// Embeddings are arrays of floats. We store them as raw bytes (a BLOB)
// because that is ~4x smaller than JSON text and much faster to parse.
const toBlob = (v: number[]) => Buffer.from(new Float32Array(v).buffer);
const fromBlob = (b: Buffer) =>
  Array.from(new Float32Array(b.buffer, b.byteOffset, b.byteLength / 4));
type Row = Omit & { embedding: Buffer };
export class SqliteMemoryStore implements MemoryStore {
  private db: Database.Database;
  constructor(filePath = "./memory.db") {
    this.db = new Database(filePath);
    this.db.pragma("journal_mode = WAL"); // safe concurrent reads while writing
    this.db.exec(`
      CREATE TABLE IF NOT EXISTS memories (
        id             TEXT PRIMARY KEY,
        namespace      TEXT NOT NULL,
        kind           TEXT NOT NULL,
        content        TEXT NOT NULL,
        embedding      BLOB NOT NULL,
        importance     REAL NOT NULL,
        createdAt      INTEGER NOT NULL,
        lastAccessedAt INTEGER NOT NULL,
        accessCount    INTEGER NOT NULL DEFAULT 0,
        source         TEXT
      );
      CREATE INDEX IF NOT EXISTS idx_mem_ns ON memories(namespace, kind);
    `);
  }
  async insert(r: MemoryRecord): Promise {
    this.db
      .prepare(
        `INSERT INTO memories VALUES
         (@id, @namespace, @kind, @content, @embedding, @importance,
          @createdAt, @lastAccessedAt, @accessCount, @source)`
      )
      .run({ ...r, embedding: toBlob(r.embedding), source: r.source ?? null });
  }
  async update(id: string, patch: Partial): Promise {
    const allowed = ["content", "importance", "lastAccessedAt", "accessCount", "embedding"];
    const keys = Object.keys(patch).filter((k) => allowed.includes(k));
    if (keys.length === 0) return;
    const sets = keys.map((k) => `${k} = @${k}`).join(", ");
    const params: Record = { id, ...patch };
    if (patch.embedding) params.embedding = toBlob(patch.embedding);
    this.db.prepare(`UPDATE memories SET ${sets} WHERE id = @id`).run(params);
  }
  async delete(id: string): Promise {
    this.db.prepare(`DELETE FROM memories WHERE id = ?`).run(id);
  }
  async listByNamespaces(namespaces: string[], kinds?: MemoryKind[]): Promise {
    const nsMarks = namespaces.map(() => "?").join(",");
    let sql = `SELECT * FROM memories WHERE namespace IN (${nsMarks})`;
    const args: unknown[] = [...namespaces];
    if (kinds?.length) {
      sql += ` AND kind IN (${kinds.map(() => "?").join(",")})`;
      args.push(...kinds);
    }
    const rows = this.db.prepare(sql).all(...args) as Row[];
    return rows.map((row) => ({ ...row, embedding: fromBlob(row.embedding) }));
  }
}

Walking through the code

The table. Each column maps one-to-one onto a field of MemoryRecord. The only special case is embedding, stored as a BLOB. An embedding with 384 dimensions is 384 floats. As JSON text that is roughly 3 to 4 KB per memory; as a packed Float32Array it is exactly 1,536 bytes and needs no parsing.

toBlob and fromBlob. These two helpers convert between a normal JavaScript array and raw bytes. Float32Array stores each number in exactly 4 bytes, which is why fromBlob divides the byte length by 4. The byteOffset argument matters: Node may hand you a Buffer that is a slice of a larger memory pool, and forgetting the offset gives you garbage numbers. (I lost an evening to that one.)

WAL mode. journal_mode = WAL (write-ahead logging) lets readers keep reading while a write is happening. In a harness where an afterTurn hook is writing while the next agent is already recalling, this prevents "database is locked" errors.

The index. Every recall filters by namespace first, so an index on (namespace, kind) keeps that filter fast even with tens of thousands of rows.

Named parameters (@id, @content). Never build SQL by gluing user text into the string. Parameters let SQLite handle escaping, which protects you from SQL injection. Memory content often comes from LLM output, which you should treat as untrusted input.

The allowed list in update. It looks fussy, but it stops a caller from accidentally changing id or namespace, which would silently move a memory to another agent. Notice that column names are only taken from this fixed list, never from outside input, so the dynamic SET clause stays safe.

What about vector search?

You may have noticed that listByNamespaces returns every memory in the allowed namespaces, and we will compute similarity in JavaScript. For a personal harness with a few thousand memories, that is perfectly fine: comparing a query against 5,000 vectors of 384 numbers is about two million multiplications, which takes a few milliseconds.

When you outgrow that, you have options such as the sqlite-vec extension, Postgres with pgvector, or a dedicated vector database. The point of the MemoryStore interface is that you can make that switch later without touching anything above it. Start with brute force; optimize when you can measure the pain.

Step 3: Recall by meaning with embeddings

My first recall function used SQL LIKE '%keyword%'. It failed immediately. The agent asked "which installer should I run?" and the memory said "We use pnpm, not npm." Those two sentences share no important words, yet one is clearly the answer to the other. Keyword search cannot see that. Embeddings can.

What an embedding actually is

An embedding model reads a piece of text and outputs a fixed-length list of numbers, called a vector. A small, popular model like all-MiniLM-L6-v2 outputs 384 numbers for any sentence you give it.

The magic is in how the model was trained: sentences with similar meaning end up with vectors that point in similar directions. You can picture each vector as an arrow in space. “Package manager is pnpm” and “which installer should I run?” point almost the same way. “User prefers dark mode” points somewhere else entirely.

A 2-D sketch. Real embeddings have hundreds of dimensions, but the idea of “angle = difference in meaning” holds.

Measuring “pointing the same way”: cosine similarity

To compare two arrows we use cosine similarity, the cosine of the angle between them:

In plain words: multiply the two vectors number by number and add it all up (the dot product), then divide by both lengths so that only direction matters, not size. The result runs from -1 to 1:

close to 1: same meaning
around 0: unrelated
negative: pointing in opposite directions (rare with sentence models)

The code

I run the embedding model locally with Transformers.js, so there are no API keys and no per-call cost. Swapping in a hosted embedding API only changes the Embedder class.

npm install @huggingface/transformers

// src/memory/embedder.ts
import { pipeline, FeatureExtractionPipeline } from "@huggingface/transformers";

export interface Embedder {
  embed(text: string): Promise;
}
export class LocalEmbedder implements Embedder {
  private extractor?: FeatureExtractionPipeline;
  private async load() {
    // Downloads the model once (~25 MB) and caches it on disk.
    this.extractor ??= await pipeline("feature-extraction", "Xenova/all-MiniLM-L6-v2");
    return this.extractor;
  }
  async embed(text: string): Promise {
    const extractor = await this.load();
    const output = await extractor(text, { pooling: "mean", normalize: true });
    return Array.from(output.data as Float32Array);
  }
}
// src/memory/similarity.ts
export function cosineSimilarity(a: number[], b: number[]): number {
  if (a.length !== b.length) throw new Error("Vector length mismatch");
  let dot = 0, normA = 0, normB = 0;
  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    normA += a[i] * a[i];
    normB += b[i] * b[i];
  }
  if (normA === 0 || normB === 0) return 0;
  return dot / (Math.sqrt(normA) * Math.sqrt(normB));
}

And the first version of recall inside the MemoryManager:

// src/memory/manager.ts (part 1)
async recall(q: RecallQuery): Promise {
  const queryVec = await this.embedder.embed(q.text);
  const candidates = await this.store.listByNamespaces(q.namespaces, q.kinds);

return candidates
    .map((record) => {
      const relevance = cosineSimilarity(queryVec, record.embedding);
      return { record, relevance, score: relevance };
    })
    .filter((m) => m.relevance >= this.minRelevance) // e.g. 0.30
    .sort((a, b) => b.score - a.score)
    .slice(0, q.limit ?? 20);
}

Understanding the details

pooling: "mean". The model actually produces one vector per token (roughly per word piece). Mean pooling averages them into a single vector for the whole sentence. Without it you would get a grid of numbers instead of one arrow.

normalize: true. This scales every vector to length 1. Once all vectors have length 1, cosine similarity is just the dot product, because the denominators are both 1. I still divide in cosineSimilarity so that the function stays correct if I ever switch to a model that does not normalize.

this.extractor ??= .... Loading a model is slow, so we do it once and reuse it. The ??= operator means "assign only if currently undefined".

The minRelevance threshold. Every memory gets some similarity score, even nonsense ones. Without a floor, a query about installers would still return the dark-mode preference at the bottom of the list, and it would waste prompt space. I settled on 0.30 for this model after logging real scores for a week; you should tune yours the same way, because the right number depends on the embedding model.

One crucial rule. Every memory and every query must be embedded with the same model. Vectors from two different models live in different “spaces” and comparing them is meaningless. If you ever change models, you must re-embed every stored memory. I now store the model name in a small meta table so the harness can detect a mismatch at startup.

Step 4: Rank memories by relevance, recency and importance

Similarity alone turned out to be a bad ranking signal. In week two, my coder agent kept pulling up a three-week-old memory saying “tests are flaky, skip the integration suite” because it matched the word-level meaning of every testing question. The flakiness had long been fixed. The memory was relevant but stale.

The fix is to blend three signals, an idea popularized by the “Generative Agents” research from Stanford in 2023:

Relevance is the cosine similarity from Step 3. Does this memory match the question?
Recency is a number between 0 and 1 that shrinks as time passes since the memory was last used. Is this memory still fresh?
Importance is the 0 to 1 value stored when the memory was written. How much does this memory matter in general?
The weights (w) decide how much each signal counts. I use 0.6, 0.2 and 0.2, so relevance still dominates.

Left: how fast recency fades for different half-lives. Right: the “pnpm” fact wins even though the chit-chat memory is fresher, because relevance and importance outweigh recency.

Exponential decay, explained

For recency I use exponential decay with a half-life. The idea comes from physics: after one half-life the value is cut in half, after two it is a quarter, and so on.

Why exponential instead of a straight line? A straight line hits zero at some point and then goes negative, which makes no sense. Exponential decay never reaches zero; an old memory just gets quieter. It can still win if it is extremely relevant, which is exactly the behavior I wanted.

The half-life should depend on the kind of memory. An episode about today’s debugging session is useful for a day or two. A user preference stays true for months.

// src/memory/ranking.ts
import { MemoryKind, ScoredMemory } from "./types";

const HOUR = 60 * 60 * 1000;
export const HALF_LIFE_HOURS: Record = {
  episode: 72,          // 3 days
  fact: 24 * 30,        // ~1 month
  procedure: 24 * 90,   // ~3 months
  preference: 24 * 180, // ~6 months
};
export interface RankWeights { relevance: number; recency: number; importance: number; }
export const DEFAULT_WEIGHTS: RankWeights = { relevance: 0.6, recency: 0.2, importance: 0.2 };
export function recencyScore(lastAccessedAt: number, kind: MemoryKind, now = Date.now()): number {
  const hours = Math.max(0, (now - lastAccessedAt) / HOUR);
  return Math.pow(0.5, hours / HALF_LIFE_HOURS[kind]);
}
export function rank(
  memories: ScoredMemory[],
  w: RankWeights = DEFAULT_WEIGHTS,
  now = Date.now()
): ScoredMemory[] {
  return memories
    .map((m) => {
      const recency = recencyScore(m.record.lastAccessedAt, m.record.kind, now);
      const score =
        w.relevance * m.relevance +
        w.recency * recency +
        w.importance * m.record.importance;
      return { ...m, score };
    })
    .sort((a, b) => b.score - a.score);
}

Why lastAccessedAt and not createdAt?

This is subtle and important. If recency were based on when a memory was created, a core fact like “we use pnpm” would fade forever, even though the agent relies on it every day. By measuring time since last use, a memory that keeps proving useful stays fresh. Memories that are never recalled fade naturally.

To make this work, the MemoryManager must "touch" each memory it actually puts in a prompt:

// src/memory/manager.ts (part 2)
async touch(used: ScoredMemory[]) {
  const now = Date.now();
  await Promise.all(
    used.map((m) =>
      this.store.update(m.record.id, {
        lastAccessedAt: now,
        accessCount: m.record.accessCount + 1,
      })
    )
  );
}

This mirrors something real about human memory: recalling a fact strengthens it. Software engineers know the same idea as an LRU (least recently used) cache.

Tuning the weights

There is no universally correct set of weights. My process was simple: log the top 10 ranked memories for 50 real tasks, mark which ones should have been included, and adjust. Two lessons came out of that:

If irrelevant-but-recent memories keep appearing, lower the recency weight.
If the agent keeps missing obvious long-standing rules, raise the importance of those memories at write time instead of fiddling with weights. Fix the data before you fix the formula.

Step 5: Build the context within a token budget

Ranking tells us which memories are best. It does not tell us how many to include. My first version included the top 20, always. Then one day an episode memory contained a 3,000-token stack trace, the prompt overflowed, and the request failed. Counting memories was the wrong unit. The unit is tokens.

Give memory a fixed budget

I now plan every prompt like a monthly salary: each part gets a fixed allowance, and memory has to live within its own.

Example numbers. Memory gets its own slice. Space for the model’s answer is reserved up front, because a prompt that fills the whole window leaves no room to reply.

Why a fixed slice instead of “whatever space is left over”? Because “left over” changes every turn. Predictable budgets make behavior predictable, and they make costs predictable too.

The packing algorithm

The approach is greedy packing: walk the ranked list from best to worst, and add each memory if it still fits. Skip any memory that does not fit, but keep going, because a smaller memory further down might still fit.

// src/memory/context.ts
import { ScoredMemory } from "./types";

// Fast approximation: ~4 characters per token for English text.
// Good enough for budgeting; use a real tokenizer if you need precision.
export const estimateTokens = (text: string) => Math.ceil(text.length / 4);
const MAX_TOKENS_PER_MEMORY = 400;
function formatMemory(m: ScoredMemory): string {
  const date = new Date(m.record.createdAt).toISOString().slice(0, 10);
  return `- [${m.record.kind}, ${date}] ${m.record.content}`;
}
export function buildMemoryContext(ranked: ScoredMemory[], budgetTokens: number) {
  const header = "\nThings you know from earlier work. Prefer the user's current instructions if they conflict.\n";
  const footer = "";
  let used = estimateTokens(header + footer);
  const lines: string[] = [];
  const included: ScoredMemory[] = [];
  for (const m of ranked) {
    const line = formatMemory(m);
    const cost = estimateTokens(line) + 1; // +1 for the newline
    if (cost > MAX_TOKENS_PER_MEMORY) continue; // oversized: summarize it later (Step 8)
    if (used + cost > budgetTokens) continue;   // doesn't fit, but a smaller one might
    lines.push(line);
    included.push(m);
    used += cost;
  }
  if (included.length === 0) return { text: "", included, tokens: 0 };
  return { text: header + lines.join("\n") + "\n" + footer, included, tokens: used };
}

Line by line

estimateTokens. Tokens are the chunks a model actually reads; for English, one token is roughly four characters. This estimate is off by 10 to 20 percent, which is fine for budgeting as long as you leave headroom. If you need exact counts, use the tokenizer that matches your model.

MAX_TOKENS_PER_MEMORY. This guard is the fix for my stack-trace disaster. No single memory is allowed to eat the budget. Oversized memories get skipped now and condensed later by the consolidation job.

continue instead of break. If memory number 3 is too big, break would stop and throw away memories 4 to 20, even if they are small and useful. continue skips just that one. This is a simple form of the classic knapsack problem: greedy is not perfectly optimal, but it is fast, predictable and good enough.

formatMemory. Each memory becomes one bullet with its kind and date. The date lets the model reason about staleness itself ("this note is from March; the user may have changed their mind"). The kind tells it whether it is reading a rule or a story.

The tags and the header sentence. Wrapping memories in clear tags helps the model tell recalled background apart from the user's live instructions. The sentence "prefer the user's current instructions if they conflict" settles a real problem: without it, an old memory could override something the user just said.

Returning included. The manager needs to know which memories were actually used so it can touch() them (Step 4). Only memories that make it into the prompt get their recency refreshed.

Where to put the block in the prompt

Research on long contexts has repeatedly found that models pay more attention to the beginning and the end of a prompt than to the middle, an effect often called “lost in the middle”. I place the memory block just before the user’s task, after the static system prompt and tool definitions. That keeps it close to the question it is meant to help answer.

Step 6: Write memories back with lifecycle hooks

Reading memory is only half the loop. Something has to decide what is worth remembering after each turn. I tried two wrong approaches before finding one that worked:

Store every message. The database filled with “ok”, “thanks” and half-finished thoughts. Recall quality collapsed because noise outnumbered signal.
Give the agent a save_memory tool. Better, but the agent forgot to call it about half the time, and when it did call it, it interrupted the task it was doing.

The approach that stuck is a lifecycle hook: a function the harness calls automatically after every turn, separate from the agent’s main work.

The afterTurn pipeline. It runs after the user already has their answer, so it adds no visible latency.

A tiny hook system

Hooks are just named events with a list of listeners. Here is the whole implementation:

// src/harness/hooks.ts
export interface TurnEvent {
  turnId: string;
  agentId: string;        // e.g. "coder"
  userMessage: string;
  agentReply: string;
  toolCalls: { name: string; summary: string }[];
}

type HookMap = {
  afterTurn: (e: TurnEvent) => Promise;
  onSessionEnd: (sessionId: string) => Promise;
};
export class HookBus {
  private listeners: { [K in keyof HookMap]: HookMap[K][] } = { afterTurn: [], onSessionEnd: [] };
  on(name: K, fn: HookMap[K]) {
    this.listeners[name].push(fn);
  }
  // Fire-and-forget: a failing hook must never crash the agent's turn.
  emit(name: K, ...args: Parameters) {
    for (const fn of this.listeners[name]) {
      (fn as (...a: unknown[]) => Promise)(...args).catch((err) =>
        console.error(`[hook:${name}] failed`, err)
      );
    }
  }
}

Why fire-and-forget? The user is waiting for the agent’s answer, not for the memory write. By not awaiting the hook, the reply goes out immediately and memory extraction happens in the background. The .catch is essential: an unhandled promise rejection in Node can crash the process, and a memory bug should never take down the agent.

Extracting durable facts with an LLM

The extraction step asks a small, cheap model to read the turn and pull out only things worth remembering. I treat the model as a function behind a minimal interface so any provider works.

// src/memory/extract.ts
import { MemoryKind } from "./types";
import { TurnEvent } from "../harness/hooks";

export interface LLM { complete(prompt: string): Promise; }
export interface Candidate { kind: MemoryKind; content: string; importance: number; }
const EXTRACT_PROMPT = (e: TurnEvent) => `
You maintain long-term memory for a software agent.
Read this turn and list facts worth remembering in FUTURE sessions.
Rules:
- Only durable information: decisions, preferences, project conventions, lessons from failures.
- Skip greetings, thanks, and anything only relevant to this exact moment.
- Write each item as one self-contained sentence that makes sense with no other context.
- Never include passwords, API keys, tokens or personal data.
- importance: 0.9+ for hard rules, ~0.5 for useful context, <0.3 for trivia.
Return ONLY a JSON array like:
[{"kind":"fact|preference|episode|procedure","content":"...","importance":0.0}]
Return [] if nothing is worth keeping.
USER: ${e.userMessage}
AGENT: ${e.agentReply}
TOOLS: ${e.toolCalls.map((t) => `${t.name}: ${t.summary}`).join("; ")}
`;
export async function extractCandidates(llm: LLM, e: TurnEvent): Promise {
  const raw = await llm.complete(EXTRACT_PROMPT(e));
  try {
    const json = raw.slice(raw.indexOf("["), raw.lastIndexOf("]") + 1);
    const parsed = JSON.parse(json) as Candidate[];
    return parsed.filter(
      (c) => typeof c.content === "string" && c.content.length > 0 &&
             typeof c.importance === "number"
    );
  } catch {
    return []; // a malformed answer means "remember nothing", never a crash
  }
}

The most important rule in that prompt is “one self-contained sentence”. A memory like “use the second option” is useless three weeks later; nobody remembers what the options were. “Retry logic in fetchTool uses exponential backoff, max 3 attempts” makes sense on its own.

The JSON parsing is defensive on purpose. Models sometimes wrap JSON in prose or code fences, so I slice from the first [ to the last ]. If parsing still fails, the hook stores nothing. Losing one memory is far better than storing garbage.

Filter, dedupe and store

The last stage lives in the MemoryManager. It filters, checks for near-duplicates, and either inserts or reinforces.

// src/memory/manager.ts (part 3)
import { randomUUID } from "node:crypto";

const SECRET_PATTERNS = [
  /sk-[A-Za-z0-9_-]{20,}/,              // common API key shape
  /AKIA[0-9A-Z]{16}/,                    // AWS access key id
  /-----BEGIN [A-Z ]*PRIVATE KEY-----/,
  /password\s*[:=]/i,
];
const MIN_IMPORTANCE = 0.3;
const DUPLICATE_THRESHOLD = 0.92;
async remember(c: Candidate, namespace: string, source: string): Promise<"stored" | "merged" | "skipped"> {
  if (c.importance < MIN_IMPORTANCE) return "skipped";
  if (SECRET_PATTERNS.some((p) => p.test(c.content))) return "skipped";
  const embedding = await this.embedder.embed(c.content);
  const existing = await this.store.listByNamespaces([namespace], [c.kind]);
  let best: { id: string; sim: number; importance: number; accessCount: number } | null = null;
  for (const r of existing) {
    const sim = cosineSimilarity(embedding, r.embedding);
    if (!best || sim > best.sim) best = { id: r.id, sim, importance: r.importance, accessCount: r.accessCount };
  }
  const now = Date.now();
  if (best && best.sim >= DUPLICATE_THRESHOLD) {
    // Same fact seen again: reinforce instead of duplicating.
    await this.store.update(best.id, {
      content: c.content, // newer wording wins, it may carry a correction
      embedding,
      importance: Math.max(best.importance, c.importance),
      lastAccessedAt: now,
      accessCount: best.accessCount + 1,
    });
    return "merged";
  }
  await this.store.insert({
    id: randomUUID(), namespace, kind: c.kind, content: c.content, embedding,
    importance: c.importance, createdAt: now, lastAccessedAt: now, accessCount: 0, source,
  });
  return "stored";
}

And the wiring that connects it all:

// src/harness/setup.ts
hooks.on("afterTurn", async (e) => {
  const candidates = await extractCandidates(cheapLLM, e);
  for (const c of candidates) {
    // Preferences and procedures are team-wide; facts and episodes stay with the agent.
    const ns = c.kind === "preference" || c.kind === "procedure" ? "shared" : `agent:${e.agentId}`;
    await memory.remember(c, ns, `turn:${e.turnId}`);
  }
});

Why each safeguard exists

The importance floor (0.3). The extractor sometimes keeps trivia despite instructions. A numeric floor is a cheap second line of defense.

The secret patterns. LLMs see API keys in tool outputs and config files all the time. A memory store that persists secrets turns one leaked log into a permanent leak in every future prompt. Regexes will not catch everything, but they catch the most common shapes. Defense in depth: the prompt says “never”, the code double-checks.

Deduplication at 0.92. Without it, the same fact gets stored again every time it comes up in conversation: “we use pnpm”, “project uses pnpm”, “pnpm is the package manager”. Duplicates waste the token budget and crowd out other memories. Any two sentences above roughly 0.92 cosine similarity are, for this model, almost always saying the same thing. I found that number by printing pairs and reading them, not by theory.

“Newer wording wins.” When a duplicate arrives, I overwrite the content with the new version. This is how corrections propagate. If the old memory said “deploy on Fridays” and the user now says “we deploy on Thursdays”, the two are similar enough to merge, and the newest version is the one that survives. Contradictions that are not similar enough to merge are handled by the consolidation job in Step 8.

Step 7: Share memory across agents with namespaces

In a multi-agent harness, “who can see what” is a design decision, not an afterthought. Namespaces are how I made that decision explicit.

A namespace is a label on every memory, like a folder name. Each agent writes to its own private namespace by default, and a small policy table decides which other namespaces it may read.

The policy in code:

// src/memory/policy.ts
export interface NamespacePolicy { read: string[]; write: string[]; }

export const POLICY: Record = {
  planner:  { read: ["agent:planner", "shared", "user"],                 write: ["agent:planner", "shared"] },
  coder:    { read: ["agent:coder", "shared", "user"],                   write: ["agent:coder", "shared"] },
  reviewer: { read: ["agent:reviewer", "agent:coder", "shared", "user"], write: ["agent:reviewer"] },
};
export function readableNamespaces(agentId: string): string[] {
  const p = POLICY[agentId];
  if (!p) throw new Error(`No memory policy for agent "${agentId}"`);
  return p.read;
}
export function assertCanWrite(agentId: string, namespace: string): void {
  if (!POLICY[agentId]?.write.includes(namespace)) {
    throw new Error(`Agent "${agentId}" may not write to "${namespace}"`);
  }
}

The reasoning behind each rule

Private by default. Each agent has its own working style and its own scratch knowledge. The coder’s note “the date utils file has a quirky timezone helper” is useful to the coder and noise to the planner. Keeping namespaces private by default keeps each agent’s context focused.

A shared namespace for team knowledge. Preferences and procedures apply to everyone: "use pnpm", "all new code needs tests". Those go to shared, so a fact learned by any agent benefits the whole team. This is where the multi-agent setup really pays off.

A read-only user namespace. Facts the user tells the harness directly ("I'm on Node 22", "never touch the legacy billing folder") get the highest trust. Only the user, through an explicit command, writes there. Agents can read it but cannot change it, so an agent cannot "remember" its way around a user rule.

The reviewer reads the coder’s memory. This was a deliberate asymmetry. A good reviewer needs to know why the coder made its choices (“used polling because the API has no webhooks”). But the reviewer should not write into the coder’s namespace, or the two agents could start rewriting each other’s knowledge. Read access across a boundary is cheap; write access across a boundary needs a strong reason.

Fail loudly. readableNamespaces throws when an agent has no policy. Silently defaulting to "read everything" is exactly the kind of bug that leaks context and is hard to notice.

Using it in the manager

The agent never chooses its own namespaces. The manager looks them up:

// inside the agent runner
const namespaces = readableNamespaces(agent.id);
const candidates = await memory.recall({ text: task, namespaces, limit: 40 });

This is the principle of least privilege, the same idea behind file permissions and database roles: give each component only the access it needs. As your swarm grows, you will be glad that access rules are one table rather than if-statements spread across ten agents.

Step 8: Teach the system to forget

A memory system that only adds will eventually drown in its own history. After a month, my store had about 2,000 records, and recall quality was noticeably worse than in week one. The good memories were still there; they were simply outnumbered.

Forgetting is not a bug to minimize. It is a feature you have to build. I use two mechanisms, both running in a scheduled maintenance job rather than during user turns.

Mechanism 1: Prune what has faded

Every memory has a strength that combines how recently it was used, how important it is and how often it has been recalled. When strength falls below a threshold, the memory is deleted.

// src/memory/maintenance.ts (part 1)
import { recencyScore } from "./ranking";
import { MemoryRecord } from "./types";

const PRUNE_THRESHOLD = 0.05;
export function strength(r: MemoryRecord, now = Date.now()): number {
  const recency = recencyScore(r.lastAccessedAt, r.kind, now);
  const usageBoost = Math.min(1, Math.log1p(r.accessCount) / Math.log(20)); // 0..1, saturates near 20 uses
  return recency * (0.5 + 0.5 * r.importance) * (0.5 + 0.5 * usageBoost);
}
export async function prune(store: MemoryStore, namespaces: string[]) {
  const all = await store.listByNamespaces(namespaces);
  const doomed = all.filter((r) => r.importance < 0.9 && strength(r) < PRUNE_THRESHOLD);
  for (const r of doomed) await store.delete(r.id);
  return doomed.length;
}

How the formula behaves. Each factor is between 0 and 1, and multiplying them means any strong signal can keep a memory alive. A memory that has not been used in months but has importance 0.8 keeps a healthy importance factor. A memory used 20 times maxes out the usage factor. Only memories that are old, unimportant and rarely used drop below 0.05.

Math.log1p for usage. The logarithm gives diminishing returns: the jump from 0 to 5 uses matters a lot, the jump from 50 to 55 barely matters. Without it, one memory recalled hundreds of times would become immortal and unbalance everything.

The importance < 0.9 guard. Hard rules ("never run migrations against production") are never pruned automatically, however long they go unused. Some knowledge is valuable precisely because it is rarely needed.

Mechanism 2: Consolidate many episodes into one lesson

Episodes pile up fast. Over two weeks my coder stored eleven separate episodes that were all versions of the same lesson: “tests failed because the mock server wasn’t started”. Eleven memories, one idea.

Consolidation finds clusters of similar episodes and asks an LLM to summarize each cluster into a single durable fact. This is the closest thing my harness has to what sleep does for human memory: turning many specific experiences into one general piece of knowledge.

// src/memory/maintenance.ts (part 2)
const CLUSTER_THRESHOLD = 0.80;
const MIN_CLUSTER_SIZE = 3;

function clusterBySimilarity(records: MemoryRecord[]): MemoryRecord[][] {
  const clusters: MemoryRecord[][] = [];
  const used = new Set();
  for (const seed of records) {
    if (used.has(seed.id)) continue;
    const group = records.filter(
      (r) => !used.has(r.id) && cosineSimilarity(seed.embedding, r.embedding) >= CLUSTER_THRESHOLD
    );
    group.forEach((r) => used.add(r.id));
    if (group.length >= MIN_CLUSTER_SIZE) clusters.push(group);
  }
  return clusters;
}
export async function consolidate(memory: MemoryManager, store: MemoryStore, llm: LLM, namespace: string) {
  const episodes = await store.listByNamespaces([namespace], ["episode"]);
  for (const cluster of clusterBySimilarity(episodes)) {
    const lesson = await llm.complete(
      `These are related notes from past work:\n` +
      cluster.map((r) => `- ${r.content}`).join("\n") +
      `\n\nWrite ONE self-contained sentence capturing the durable lesson. ` +
      `If the notes contradict each other, prefer the most recent. Reply with the sentence only.`
    );
    const importance = Math.min(1, Math.max(...cluster.map((r) => r.importance)) + 0.1);
    await memory.remember({ kind: "fact", content: lesson.trim(), importance }, namespace, "consolidation");
    for (const r of cluster) await store.delete(r.id);
  }
}

Understanding consolidation

The clustering is deliberately simple. It picks a seed memory and groups everything within 0.80 similarity of it. That threshold is lower than the 0.92 used for deduplication on purpose: duplicates say the same thing, cluster members are about the same thing. Real clustering algorithms (k-means, HDBSCAN) exist, but this greedy version is easy to read and debug, and it works well for a few hundred episodes.

MIN_CLUSTER_SIZE = 3. One or two similar episodes might be coincidence. Three or more suggests a pattern worth turning into a rule.

Episodes become facts. The output has kind fact, which has a much longer half-life than episode. The specific stories fade; the lesson stays.

Importance gets a small boost. A lesson that showed up three or more times is, by definition, recurring. Giving it slightly more importance than its strongest source reflects that.

Contradictions resolve toward recency. The prompt tells the summarizer to prefer the most recent note. Combined with the “newer wording wins” rule from Step 6, this keeps the store from holding two opposite beliefs at once.

Order matters: write before delete. The new consolidated memory is saved before the old episodes are removed. If the process crashes halfway, you end up with a duplicate, which the next run will merge. The opposite order could lose knowledge entirely.

I run prune and consolidate once a night through a scheduled job, plus at onSessionEnd for the namespaces touched during that session.

Putting it together: one full turn

We now have every piece. Here is how a single task flows through the harness, from the moment it arrives until its lessons are stored.

The read path (blue) runs before the model is called. The write path (purple) runs after, in the background.

And here is the runner that ties it all together:

// src/harness/runTurn.ts
import { randomUUID } from "node:crypto";

const MEMORY_BUDGET_TOKENS = 2000;
export async function runTurn(task: string, sessionHistory: Message[]): Promise {
  const turnId = randomUUID();
  // 1-2. Route the task to the right agent
  const agent = await router.pick(task);
  // 3. Recall candidates from the namespaces this agent may read, then rank them
  const candidates = await memory.recall({
    text: task,
    namespaces: readableNamespaces(agent.id),
    limit: 40,
  });
  const ranked = rank(candidates);
  // 4. Pack the best memories into the memory budget
  const memoryCtx = buildMemoryContext(ranked, MEMORY_BUDGET_TOKENS);
  await memory.touch(memoryCtx.included); // reinforce what we actually used
  // 5. Call the model with system prompt + memory + recent turns + task
  const reply = await agent.run({
    system: agent.systemPrompt,
    memory: memoryCtx.text,
    history: trimHistory(sessionHistory, 2500),
    task,
  });
  // 6. Fire the write path in the background; the user does not wait for it
  hooks.emit("afterTurn", {
    turnId,
    agentId: agent.id,
    userMessage: task,
    agentReply: reply.text,
    toolCalls: reply.toolCalls,
  });
  return reply.text;
}

Reading the runner as a story

Read the function from top to bottom and it tells the whole story of this post:

Route. Pick the specialist for the job.
Recall. Embed the task and find memories that mean something similar, but only from namespaces this agent is allowed to read.
Rank. Blend relevance, recency and importance so that fresh, important, on-topic memories rise to the top.
Budget. Pack the winners into a fixed token allowance, skipping anything oversized, and refresh the memories we used.
Think. The model sees the system prompt, a compact block, the recent conversation and the task.
Remember. After the answer is sent, a hook extracts durable lessons, filters them, deduplicates them and stores them.
Maintain. Separately, on a schedule, weak memories fade away and repeated episodes become lasting facts.

Notice what the agent itself does not know about: SQLite, vectors, half-lives, deduplication thresholds. To the agent, memory is simply a paragraph of helpful background that appears in its prompt. That is the sign of a good abstraction. All the complexity is real, but it lives in one place.

A tip for debugging

Log memoryCtx.included with each memory's relevance, recency, importance and final score on every turn. When the agent does something strange, the first question is always "what did it remember?" Having that answer in a log line turned hour-long debugging sessions into five-minute ones.

Lessons learned the hard way

Building this by hand taught me more than any library’s documentation could have. These are the mistakes I would warn my past self about.

1. Memory quality is decided at write time, not read time. I spent days tuning ranking weights before realizing the real problem was what I was storing. Garbage in the store cannot be ranked into gold. A strict extraction prompt and an importance floor did more for recall quality than any formula change.

2. Store sentences, not fragments. Every memory must make sense when read alone, weeks later, by an agent with no context. “Use the second approach” is worthless. “Retries use exponential backoff starting at 200 ms” is gold. If you change only one thing in your own system, change this.

3. Count tokens, never items. “Top 10 memories” is a bug waiting to happen. One oversized memory can blow your context window. Budget in tokens and cap the size of each individual memory.

4. Recency must mean last used, not last created. Otherwise your most important long-standing facts slowly fade out of every prompt, precisely because they have been true for a long time.

5. Without deduplication, the loudest fact wins. A fact mentioned in ten conversations gets ten copies, which then occupy ten slots in the prompt and push out everything else. Merge near-duplicates on write.

6. Memory needs a way to be wrong. Old memories will contradict new reality. Tell the model in the prompt that current instructions win, let newer wording overwrite older wording on merge, and resolve clusters toward the most recent note.

7. Never persist secrets. It is very easy for an API key in a tool output to become a “fact” that is then injected into every future prompt. Filter in the prompt and again in code.

8. Make memory observable. Log what was recalled, its scores, and what was written on every turn. When an agent misbehaves, “what did it remember?” is almost always the first question, and you want the answer in seconds.

9. Thresholds are empirical. The numbers in this post (0.30 relevance floor, 0.92 duplicate threshold, 0.80 cluster threshold) were found by printing real scores and reading them. They depend on your embedding model. Copy the method, not the numbers.

10. Start brute force. A for loop over a few thousand vectors is fast enough. I almost introduced a vector database on day one, and I would have spent that week on infrastructure instead of learning how memory actually behaves.

Hand-rolled vs. using a framework

Would I recommend everyone build their own memory layer for production? No. Mature frameworks handle edge cases, scale and concurrency that this design ignores. But I would recommend building it once. After this project, when I read the documentation of any memory or RAG framework, I can map every option to a concept I have implemented myself: namespaces, half-lives, consolidation, token budgets. That understanding is what makes you good at choosing and configuring tools, rather than just installing them.

Conclusion and what’s next

An agent’s memory is not a feature of the model. It is a system you build around the model, and it comes down to answering four questions well:

What is worth remembering? Extract durable, self-contained facts after each turn; filter noise and secrets.
What is relevant right now? Embed, compare by meaning, and rank by relevance, recency and importance.
How much can we afford to show? Pack the best memories into a fixed token budget.
What should we let go? Prune what has faded and consolidate repeated experiences into lasting lessons.

Wiring each of those by hand turned “memory” from a black box into a set of small, understandable pieces: a record type, a store, an embedder, a ranking function, a packer, a hook and a maintenance job. None of them is more than a hundred lines of TypeScript. Together, they turned my team of forgetful contractors into something that actually learns from working on a project.

The next pieces I am building in the harness:

Hybrid search. Combining embeddings with keyword search (BM25) so exact identifiers like function names and error codes are recalled reliably.
A memory inspector. A small UI to browse, edit and delete memories by namespace, because the user should always be able to see and correct what the agents believe.
Evaluation. A fixed set of tasks with known “correct” memories, so that every change to thresholds or prompts can be measured instead of guessed.

If you are building your own agent harness, I hope this saves you a few of the evenings it cost me. Start with the MemoryRecord type, keep the store boring, and spend your energy on what gets written.

ESM-Only Libraries Are Here: A Migration Playbook for CommonJS-Heavy Node Codebases

Vasanthan K — Wed, 28 Oct 2026 13:01:03 GMT

The error that started it all

If you have maintained a Node.js service for more than a couple of years, you have probably met this message after a routine npm update:

Error [ERR_REQUIRE_ESM]: require() of ES Module /app/node_modules/chalk/source/index.js
from /app/src/logger.js not supported.
Instead change the require of index.js to a dynamic import() which is available in all CommonJS modules.

You did nothing wrong. Your code still says const chalk = require('chalk'), exactly as it did yesterday. What changed is the library: its new major version stopped shipping CommonJS and now ships only ECMAScript Modules (ESM). Packages like chalk v5, node-fetch v3, got v12+, nanoid v4+, p-queue v7+, execa, and recent major versions of uuid all made this move, and the list grows every month.

For years, the honest advice was “pin the old version and wait.” That advice has now expired. Two things changed in 2025–2026:

Node.js can require() ES modules. The feature known as require(esm) is enabled by default on every supported Node.js line and is now marked stable. A CommonJS file can load most ESM-only packages with a plain require().
Old versions are out of support. Pinning chalk@4 forever means pinning a package that will never get another security fix. The "just pin it" strategy now carries real risk.

This article is a playbook. It explains the theory from scratch, so a beginner can follow it, and then gives a step-by-step plan, with code, for teams that have hundreds of thousands of lines of CommonJS and cannot rewrite everything in one sprint.

What you will learn:

What CommonJS and ESM actually are, and why they behave differently under the hood
Why ERR_REQUIRE_ESM existed, and how require(esm) removes most of it
The three remaining traps: top-level await, the .default wrapper, and tooling
A four-phase migration playbook: audit, choose a strategy, migrate incrementally, harden tooling
How to publish your own internal libraries so they work for every consumer

All the diagrams in this post were created specifically for it. All code examples target Node.js 22.12 or newer.

Part 1: CommonJS and ESM, explained from zero

A module is simply a file that keeps its variables private and chooses what to share with other files. JavaScript did not have modules built into the language for most of its life, so Node.js invented its own system in 2009. That system is CommonJS (CJS). In 2015, the language itself gained an official module system: ECMAScript Modules (ESM). Node.js has supported both since version 12, and that is the root of every problem in this article.

CommonJS: the original Node.js way

Here is a tiny CommonJS module and a file that uses it:

// math.js  (CommonJS)
const PI = 3.14159;                 // private: nobody outside can see this

function area(r) {
  return PI * r * r;
}
module.exports = { area };          // public: this object is what require() returns

// app.js  (CommonJS)
const { area } = require('./math'); // runs math.js right now, returns module.exports
console.log(area(2));               // 12.56636

Three things are happening behind the scenes, and they explain almost every migration problem later on:

require() is an ordinary function call. When Node reaches line 1 of app.js, it pauses, finds math.js on disk, reads it, runs it from top to bottom, and hands back whatever ended up in module.exports. Only then does line 2 run. This is synchronous loading.
Node wraps every CJS file in a hidden function. Before running your code, Node places it inside function (exports, require, module, __filename, __dirname) { ... }. That is why require, module, __dirname and __filename seem to exist "for free" in CommonJS. They are not globals. They are parameters of that invisible wrapper.
Exports are a snapshot of an object. module.exports is a plain object. If math.js later changes a variable, code that already destructured { area } keeps the old value.

Because require() is just a function, you can call it anywhere: inside an if, inside a function, with a computed string. That flexibility is convenient, but it also means tools cannot know what a file depends on until the code actually runs.

ES Modules: the language standard

The same example written as ESM:

// math.mjs  (ESM)
const PI = 3.14159;

export function area(r) {           // named export
  return PI * r * r;
}
export default { area };            // optional default export

// app.mjs  (ESM)
import { area } from './math.mjs';  // a declaration, not a function call
console.log(area(2));

This looks like a syntax change, but the engine treats it completely differently:

import and export are declarations. They must sit at the top level of the file, and the module name must be a fixed string. The engine can read every dependency without executing a single line.
Bindings are live. area in app.mjs is not a copy. It is a read-only window onto the variable inside math.mjs. If the exporting module updates it, importers see the new value.
Strict mode is always on, and there is no hidden wrapper. That is why __dirname, require and module do not exist inside ESM.
File paths must be complete. import './math' fails in Node. You must write ./math.js or ./math.mjs. ESM follows the same rules as browsers, which never guess file extensions.

The real difference: how loading happens

The diagram below is the most important picture in this article.

CommonJS loads and runs at the same time. Each require() reads one file and executes it immediately. When that file calls require() itself, the process repeats, nested inside the first call. It is one long, synchronous chain.

ESM works in three separate phases across the whole dependency graph:

Construction. Node finds every file reachable from the entry point, downloads or reads it, and parses it. It records every import and export, but runs nothing.
Instantiation. Node connects each import to the matching export as a live binding. If you import a name that does not exist, the error appears here, before any of your code runs.
Evaluation. Node finally executes each module body, starting from the deepest dependency and moving up.

Phase 3 is the key. ESM allows top-level await: an await written directly in the module body, outside any function. When a module does that, its evaluation pauses until the promise settles, so the whole load becomes asynchronous. CommonJS's require() must return a value immediately. It cannot wait.

How Node decides which system a file uses

Node does not guess from the code alone. It follows these rules, in order:

The phrase “nearest package.json" matters. Node walks up from the file's folder until it finds one. That means you can place a tiny package.json with just { "type": "module" } inside one subfolder and turn only that folder into ESM. We will use this trick in the migration playbook.

Beginner takeaway: .mjs and .cjs are explicit and never lie. Plain .js is decided by the nearest package.json. When in doubt, be explicit.

Part 2: Why the two systems clash

With the loading model in mind, the compatibility rules stop looking random. There are four possible directions, and for years one of them was blocked.

Direction 1: ESM importing CommonJS (always worked)

An ES module can import a CommonJS file because CommonJS is synchronous. Waiting for something that finishes immediately is never a problem.

// app.mjs (ESM) importing a CJS package such as express
import express from 'express';        // default import = module.exports
import { Router } from 'express';     // named import: works only if Node can detect it

The default import always gives you module.exports. Named imports are a best effort. Node scans the CommonJS source with a small static analyzer (called cjs-module-lexer) looking for patterns like exports.Router = .... If a package builds its exports dynamically, the analyzer cannot see them, and you get a SyntaxError saying the named export was not found. The safe fallback is:

import pkg from 'some-cjs-lib';
const { helper } = pkg;               // destructure from the default import

Direction 2: CommonJS requiring ESM (blocked for years)

This is the direction that hurt. require() is synchronous and must return a value right now. ESM evaluation was treated as potentially asynchronous because of top-level await. A synchronous function cannot wait on an asynchronous one, so Node simply refused:

// logger.js (CJS) on Node 18 or Node 22.11
const chalk = require('chalk');       // chalk v5 is ESM-only
// → Error [ERR_REQUIRE_ESM]: require() of ES Module ... not supported.

The only escape hatch was dynamic import(), which works inside CommonJS but returns a promise:

// logger.js (CJS) — the old workaround
let chalkPromise;

function getChalk() {
  // import() is async, so we cache the promise and reuse it
  chalkPromise ??= import('chalk').then((mod) => mod.default);
  return chalkPromise;
}
async function logError(message) {
  const chalk = await getChalk();
  console.error(chalk.red(message));
}
module.exports = { logError };

Read that code carefully, because it shows why the workaround spreads like a virus. logError used to be synchronous. Now it is async. Every caller must now await it, so every caller's caller becomes async too. One ESM-only dependency deep in the tree can force an async rewrite all the way up to your HTTP handlers. Most teams chose to pin the old CommonJS version instead.

Why the wall was not really necessary

Node contributors eventually pointed out a subtle fact: ESM is only asynchronous when the module graph actually contains top-level await. The vast majority of packages never use it. For those packages, all three phases can complete synchronously, so there is no technical reason require() cannot load them. That insight became the require(esm) feature, covered next.

The other everyday differences

Before moving on, here is the full list of differences you will hit while migrating. Keep it handy.

Part 3: The game-changer, require(esm)

On any supported Node.js release today, CommonJS can require() an ES module synchronously, as long as that module's graph has no top-level await. The feature was marked stable in Node v25.4.0 and backported to the 24.x LTS line, according to the Node.js modules documentation and this OpenReplay timeline.

What this means for your version policy:

Node 22.12+ has it on by default. This is the practical minimum for any migration in this article.
Node 20.19+ also has it, but Node 20 reached end-of-life on April 30, 2026. Do not plan around it.
Node 18 never received it. If you are still on 18, upgrading Node is step zero.

What require() actually returns

This is the part that trips people up. When you require() an ES module, you do not get the default export directly. You get the module namespace object, the same thing await import() gives you.

// point.mjs (ESM)
export default class Point {
  constructor(x, y) { this.x = x; this.y = y; }
}
export function distance(a, b) {
  return Math.hypot(b.x - a.x, b.y - a.y);
}

// main.js (CJS)
const mod = require('./point.mjs');
console.log(mod);
// [Module: null prototype] {
//   __esModule: true,
//   default: [class Point],
//   distance: [Function: distance]
// }

const Point = mod.default;           // the class lives on .default
const { distance } = mod;            // named exports are plain properties
const p = new Point(0, 0);
console.log(distance(p, new Point(3, 4))); // 5

The extra __esModule: true flag is added for compatibility with code produced by transpilers like Babel and older TypeScript. You should not depend on it in hand-written code.

The practical consequence: when you replace an old CommonJS version of a package with its ESM-only version, const chalk = require('chalk') may now return an object with .default instead of the function itself. A defensive pattern that works for both shapes:

// Works whether the package exposes module.exports or an ESM default
const chalkModule = require('chalk');
const chalk = chalkModule.default ?? chalkModule;

The "module.exports" escape hatch (for library authors)

If you maintain an ESM package and want CommonJS users to receive a plain value, Node lets you export something under the special string name "module.exports":

// point.mjs
export function distance(a, b) { return Math.hypot(b.x - a.x, b.y - a.y); }

export default class Point {
  constructor(x, y) { this.x = x; this.y = y; }
  static distance = distance;        // attach named helpers so CJS users keep them
}
export { Point as 'module.exports' }; // require() now returns Point itself

// consumer.js (CJS)
const Point = require('./point.mjs'); // [class Point], no .default needed
const { distance } = require('./point.mjs'); // works because of the static property

The trade-off: once you use "module.exports", CommonJS consumers only see that one value, so named exports disappear unless you attach them to it, as above.

The one hard limit: top-level await

Here is the full decision path Node follows on every require() call:

If the required module, or anything it imports, however deep, uses top-level await, Node throws ERR_REQUIRE_ASYNC_MODULE and tells you to use import(). The word "graph" matters. Your direct dependency may be fine while a dependency three levels down adds a top-level await in a patch release.

That is not hypothetical. In April 2026, a release of lru-cache added top-level await to its ESM build. That broke CommonJS projects that reached it through a chain of ESM-only packages, including jsdom users. The maintainer reverted it, but the lesson stands: the top-level-await-free graph is now a compatibility contract.

Diagnostic tools built into Node

# Print a warning every time require() loads an ES module (find hidden usage)
node --trace-require-module=all server.js

# When ERR_REQUIRE_ASYNC_MODULE happens, print where the top-level awaits are
node --experimental-print-required-tla server.js

# Emergency switch: turn the feature off to compare behavior
node --no-require-module server.js

And you can feature-detect at runtime:

if (!process.features.require_module) {
  throw new Error('This service needs Node.js 22.12+ (require(esm) support).');
}

The module-sync export condition

Node also resolves a "module-sync" condition in a package's exports map when require(esm) is available. Library authors use it to point modern Node at the ESM build for both require and import, while older runtimes fall back to a CommonJS file. You will see it in Part 8.

Part 4: Playbook step 1, audit before you touch anything

A migration fails when it starts with code changes. It succeeds when it starts with an inventory. The audit answers four questions: which Node version runs everywhere, which dependencies are ESM-only, which of them could carry top-level await, and how much of your own code depends on CommonJS-only features.

4.1 Check the runtime in every environment

Your laptop is not the problem. CI images, Docker base images, serverless runtimes and that one cron box are. Run this everywhere your code runs:

node -p "process.version + '  require(esm): ' + process.features.require_module"
# v24.11.0  require(esm): true

Then make the requirement explicit in package.json so installs warn early:

{
  "engines": { "node": ">=22.12.0" }
}

Add engine-strict=true to your project's .npmrc if you want npm install to fail instead of warn on an older runtime.

4.2 Classify every dependency

The script below reads each dependency’s package.json and labels it CJS, dual (ships both), or ESM-only. It is a heuristic, but it is accurate enough to plan with. Save it as scripts/esm-audit.cjs. We use the .cjs extension so it runs as CommonJS no matter how the project is configured later.

// scripts/esm-audit.cjs
const fs = require('node:fs');
const path = require('node:path');

const root = process.cwd();
const pkg = JSON.parse(fs.readFileSync(path.join(root, 'package.json'), 'utf8'));
const deps = Object.keys({ ...pkg.dependencies });
// Collect every condition name used anywhere in an "exports" map
// e.g. { ".": { "import": "...", "require": "..." } } -> { import, require }
function conditionKeys(exportsField, keys = new Set()) {
  if (exportsField && typeof exportsField === 'object') {
    for (const [key, value] of Object.entries(exportsField)) {
      if (!key.startsWith('.')) keys.add(key); // ".", "./sub" are paths, not conditions
      conditionKeys(value, keys);
    }
  }
  return keys;
}
function classify(meta) {
  const keys = conditionKeys(meta.exports);
  const speaksEsm = meta.type === 'module' || keys.has('import') || keys.has('module-sync');
  const speaksCjs =
    keys.has('require') ||
    (meta.type !== 'module' && !keys.has('import')) ||
    String(meta.main || '').endsWith('.cjs');
  if (speaksEsm && speaksCjs) return 'dual';
  return speaksEsm ? 'ESM-only' : 'CJS';
}
const rows = deps.map((name) => {
  const file = path.join(root, 'node_modules', name, 'package.json');
  const meta = JSON.parse(fs.readFileSync(file, 'utf8'));
  return { name, version: meta.version, kind: classify(meta) };
});
console.table(rows.sort((a, b) => a.kind.localeCompare(b.kind)));

Run it with node scripts/esm-audit.cjs. You will get a table like this:

┌─────────┬───────────┬───────────┬────────────┐
│ (index) │ name      │ version   │ kind       │
├─────────┼───────────┼───────────┼────────────┤
│ 0       │ 'express' │ '5.3.0'   │ 'CJS'      │
│ 1       │ 'zod'     │ '3.25.76' │ 'dual'     │
│ 2       │ 'chalk'   │ '5.6.2'   │ 'ESM-only' │
│ 3       │ 'nanoid'  │ '5.1.16'  │ 'ESM-only' │
└─────────┴───────────┴───────────┴────────────┘

How the code works. A package can declare its format in two places. The "type" field tells Node how to treat its .js files. The "exports" map lists entry points, optionally keyed by conditions such as import, require and module-sync. The recursive conditionKeys function walks that map, which can be nested, and gathers all the condition names. If a package offers a require entry, CommonJS users get a CJS file. If it only offers import or is type: module, it is ESM-only. Version numbers in the sample output are illustrative.

4.3 Find packages you held back

ESM-only releases are often the reason a dependency is stuck. Run:

npm outdated

Look for major versions you skipped, then check each changelog for words like “pure ESM” or “drop CommonJS”. Those are your migration candidates, and often your security backlog too.

4.4 Smoke-test require() for every dependency

Classification tells you what should work. This test tells you what does work on your real Node version, including the top-level await trap:

// scripts/require-smoke.cjs
const pkg = require('../package.json');

const results = Object.keys(pkg.dependencies ?? {}).map((name) => {
  try {
    const mod = require(name);
    const shape = mod && mod.__esModule ? 'namespace (.default)' : typeof mod;
    return { name, status: 'ok', shape };
  } catch (err) {
    return { name, status: err.code ?? 'error', shape: err.message.split('\n')[0] };
  }
});
console.table(results);
process.exitCode = results.some((r) => r.status !== 'ok') ? 1 : 0;

Read the status column like this:

Here is the real output from a test project on Node 22:

┌─────────┬───────────┬────────┬────────────────────────┐
│ (index) │ name      │ status │ shape                  │
├─────────┼───────────┼────────┼────────────────────────┤
│ 0       │ 'chalk'   │ 'ok'   │ 'namespace (.default)' │
│ 1       │ 'express' │ 'ok'   │ 'function'             │
│ 2       │ 'nanoid'  │ 'ok'   │ 'object'               │
│ 3       │ 'zod'     │ 'ok'   │ 'namespace (.default)' │
└─────────┴───────────┴────────┴────────────────────────┘

ok with shape namespace (.default) means the package has a default export reached through .default. For ESM-only packages like chalk, that is require(esm) at work. Use .default in your code.
ok with shape object can also be an ESM namespace. nanoid has only named exports, so const { nanoid } = require('nanoid') just works.
ERR_REQUIRE_ASYNC_MODULE means a top-level await sits somewhere in that package's graph. It needs Strategy B from Part 5.
ERR_PACKAGE_PATH_NOT_EXPORTED usually means the package has no main entry (a CLI or types-only package). That is normally fine to ignore.

One caution: requiring a package runs its top-level code. Run this in CI or a sandbox, not against production credentials.

4.5 Measure your own CommonJS-only habits

These patterns need rewriting when a file becomes ESM. Counting them tells you how big the job is.

# Count CommonJS-only constructs in your source (ripgrep)
rg -c "__dirname|__filename" src
rg -c "require\.(resolve|cache|main)" src
rg -n "require\([^'\"]" src              # dynamic require with a variable
rg -n "require\(['\"][^'\"]+\.json['\"]\)" src   # JSON requires
rg -n "module\.exports\s*=\s*" src | wc -l

4.6 Write it down

Put the findings into a simple tracker. A short table like the one below becomes your migration backlog.

Part 5: Playbook step 2, choose a strategy

There is no single correct migration. There are three sensible strategies, and the audit tells you which one fits. Most large codebases use all three over time: A today, B for a few awkward modules, and C as the long-term direction.

Strategy A: stay CommonJS, consume ESM through require(esm)

This is the lowest-effort option and, for many services, the right final answer. You keep every file as CommonJS. You upgrade ESM-only dependencies and load them with require().

// src/lib/ids.js (still CommonJS)
const { nanoid } = require('nanoid');   // nanoid 5 is ESM-only; named export works directly

function newOrderId() {
  return `ord_${nanoid(12)}`;
}
module.exports = { newOrderId };

// src/lib/logger.js (still CommonJS)
const chalkModule = require('chalk');   // chalk 5 is ESM-only with a default export
const chalk = chalkModule.default;      // the function lives on .default

function warn(message) {
  console.warn(chalk.yellow(`⚠ ${message}`));
}
module.exports = { warn };

Compare this with the async workaround in Part 2. warn() stays synchronous, its callers do not change, and the upgrade is a two-line diff. That is the whole point of require(esm).

Choose A when your Node floor is 22.12+, your ESM dependencies have no top-level await, and the codebase is stable or in maintenance mode.

Strategy B: create an async boundary for the awkward modules

Some modules truly need top-level await: a config loader that fetches secrets, a module that initializes a database pool, or a third-party package that ships with it. require() will never load these. The fix is to move the asynchronous work to one place, the application's startup, instead of letting async leak through the whole codebase.

// src/config.mjs (ESM, uses top-level await on purpose)
import { readFile } from 'node:fs/promises';

const raw = await readFile(new URL('../config.json', import.meta.url), 'utf8');
export const config = Object.freeze(JSON.parse(raw));

// src/bootstrap.js (CommonJS entry point)
async function main() {
  // 1. Load every async ESM module ONCE, at startup
  const { config } = await import('./config.mjs');

// 2. Hand the resolved values to the synchronous CommonJS world
  const { createServer } = require('./server');
  const app = createServer(config);
  app.listen(config.port, () => console.log(`listening on ${config.port}`));
}
main().catch((err) => {
  console.error('Startup failed:', err);
  process.exit(1);
});

// src/server.js (CommonJS, fully synchronous, receives config as an argument)
const express = require('express');

function createServer(config) {
  const app = express();
  app.get('/health', (_req, res) => res.json({ ok: true, env: config.env }));
  return app;
}
module.exports = { createServer };

Why this works. await import() is allowed inside CommonJS because it returns a promise, and promises can wait for top-level await. By doing it once inside main(), the rest of the application receives plain values through dependency injection (passing them as function arguments). No other function has to become async.

If you need an ESM-with-TLA module lazily, after startup, cache the promise so it is only loaded once:

// src/lib/lazy-esm.js (CommonJS helper)
const cache = new Map();

/** Load an ES module once and reuse the same promise for every caller. */
function lazyImport(specifier) {
  if (!cache.has(specifier)) cache.set(specifier, import(specifier));
  return cache.get(specifier);
}
module.exports = { lazyImport };

Choose B when the smoke test shows ERR_REQUIRE_ASYNC_MODULE, or your own code needs top-level await for initialization.

Strategy C: incremental full migration to ESM

If the codebase will live for years, moving it to ESM pays off. You get the language standard, static analysis that tools understand, better tree-shaking in bundlers, and no more interop questions. The key word is incremental. Because require(esm) lets CommonJS files load ESM files, you can convert one folder at a time and ship after every step. Part 6 shows exactly how.

Choose C when the codebase is actively developed, you want new code written in modern syntax, or you publish packages consumed by ESM-first tools.

Strategy D: when you cannot upgrade Node yet

If some environment is stuck below 22.12, you have two temporary options: pin the last CommonJS-compatible version of each library, or use the async import() pattern from Part 2 in the few places that need ESM. Treat both as debt with a deadline, and put the Node upgrade at the top of the backlog.

Part 6: Playbook step 3, migrate incrementally

The golden rule: never do a big-bang rewrite. Convert small pieces, keep the app shippable after every step, and let require(esm) hold the two worlds together while you work.

6.1 Leaf-first waves

A leaf is a file that does not import any of your other internal files: utilities, constants, formatters, validation schemas. Leaves are the safest place to start because nothing inside them can break when they change format. Their parents keep calling require(), which now loads ESM.

The picture shows a typical order. Wave 1 converts utils and config. Wave 2 converts the services and routes that use them. The entry point, server.js, goes last, because CommonJS parents can load ESM children but you want to keep the entry point stable until everything below it is done.

To find your leaves automatically, tools like madge can draw your internal dependency graph:

npx madge --leaves src/          # files with no internal dependencies
npx madge --circular src/        # circular imports to untangle first

Fix circular dependencies before converting. Cycles that cross the CJS/ESM boundary are the hardest bugs in this whole process, and Node may refuse them with ERR_REQUIRE_CYCLE_MODULE.

6.2 Flip one folder at a time with a nested package.json

Remember the “nearest package.json" rule from Part 1. You do not need to rename files to .mjs. Create a one-line file named package.json inside the folder you are converting (for example src/utils/package.json):

{ "type": "module" }

Every .js file under src/utils/ is now ESM. Everything else stays CommonJS. Here is a tested, real example:

src/
├── app.js                 ← CommonJS (root package.json has "type": "commonjs")
└── utils/
    ├── package.json       ← { "type": "module" }
    ├── defaults.json
    ├── format.js          ← ESM now
    └── slug.js            ← ESM now

// src/utils/format.js (ESM)
import defaults from './defaults.json' with { type: 'json' };

export function money(cents) {
  return `$${(cents / 100).toFixed(2)}`;
}
export { defaults };

// src/app.js (still CommonJS) — no changes needed to load the converted folder
const { money, defaults } = require('./utils/format.js');
console.log(money(1999), defaults.port); // $19.99 3000

Notice one change you may need in the parent: the extension. CommonJS lets you write require('./utils/format'), and that still works, but writing the full path now makes the eventual switch to import mechanical.

6.3 Keep parents unchanged with the "module.exports" trick

The biggest source of churn during migration is the .default problem. If slug.js used to do module.exports = slugify, every parent does const slugify = require('./slug'). After conversion to export default, those parents would receive a namespace object and break.

The fix is a single line in the converted file:

// src/utils/slug.js (ESM)
export default function slugify(text) {
  return text.toLowerCase().trim().replace(/\s+/g, '-');
}

// Transitional: CommonJS parents get the function directly, exactly like before
export { slugify as 'module.exports' };

// src/app.js (CommonJS) — unchanged
const slugify = require('./utils/slug.js');
slugify('Hello World'); // 'hello-world'

ESM importers still use import slugify from './slug.js'. Once every parent is converted, delete the transitional line.

6.4 The conversion cookbook

Each pattern below shows the CommonJS original and its ESM replacement. Together they cover the vast majority of real-world files.

Imports and exports

// CommonJS
const fs = require('fs');
const { join } = require('path');
const db = require('./db');
module.exports = { getUser, saveUser };
module.exports.helper = helper;

// ESM
import fs from 'node:fs';
import { join } from 'node:path';
import db from './db.js';               // extension is mandatory
export { getUser, saveUser };
export { helper };

The node: prefix is optional for built-ins but makes it obvious which imports are core modules. The .js extension is not optional for relative paths.

__dirname and __filename

// CommonJS
const templates = path.join(__dirname, 'templates');

// ESM (Node 20.11+)
const templates = path.join(import.meta.dirname, 'templates');

// ESM, portable fallback for any Node version
import { fileURLToPath } from 'node:url';
import { dirname } from 'node:path';
const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);

import.meta.url is the current file's location as a file:// URL. fileURLToPath converts it to a normal path. Many file APIs also accept URLs directly, so new URL('./templates/', import.meta.url) is often the cleanest option.

JSON files

// CommonJS
const pkg = require('./package.json');

// ESM, using import attributes
import pkg from './package.json' with { type: 'json' };

// ESM, reading at runtime (useful when the file changes or is large)
import { readFile } from 'node:fs/promises';
const pkg = JSON.parse(await readFile(new URL('./package.json', import.meta.url), 'utf8'));

Careful: the second option uses top-level await. That makes this module impossible for CommonJS parents to require(). In a mixed codebase, prefer the import-attribute version.

require.resolve

// CommonJS
const workerPath = require.resolve('./worker.js');

// ESM: returns a file:// URL string
const workerUrl = import.meta.resolve('./worker.js');

Conditional and dynamic loading

// CommonJS: load an optional dependency only when configured
let redis = null;
if (process.env.REDIS_URL) {
  redis = require('ioredis');
}

// ESM: use dynamic import(); it returns a promise of the namespace
let redis = null;
if (process.env.REDIS_URL) {
  const { default: Redis } = await import('ioredis');
  redis = new Redis(process.env.REDIS_URL);
}

If this lives in a module that CommonJS still require()s, wrap it in an async function instead of using top-level await, for the same reason as the JSON example.

When you truly need require inside ESM

Some old plugins or native addons expect require. Create one:

// ESM
import { createRequire } from 'node:module';
const require = createRequire(import.meta.url);

const legacyPlugin = require('./legacy-plugin.cjs');
const addon = require('./build/Release/addon.node'); // native addons load via require

“Is this file the entry point?”

// CommonJS
if (require.main === module) main();

// ESM
import { fileURLToPath } from 'node:url';
if (process.argv[1] === fileURLToPath(import.meta.url)) main();

6.5 The final flip

When most of the codebase is ESM, invert the default instead of adding more nested package.json files:

Set "type": "module" in the root package.json.
Rename the remaining CommonJS files to .cjs. Config files for older tools (.eslintrc.cjs, jest.config.cjs) often fall into this group.
Delete the nested { "type": "module" } files, since they are now redundant.
Remove the transitional export { x as 'module.exports' } lines once no CommonJS file imports those modules.
Run the full test suite and the smoke script from Part 4.

6.6 Ship every wave

Each wave should be its own pull request with its own deploy. A practical rhythm for a mid-size service:

Part 7: Playbook step 4, harden your tooling

Code is rarely what breaks during an ESM migration. Tooling is: the TypeScript compiler, the test runner, and dev scripts that quietly assumed CommonJS. Fix these deliberately.

7.1 TypeScript

TypeScript has to know which module system Node will use, so it can emit the right output and check imports correctly. For Node.js projects, use the Node-aware settings:

// tsconfig.json
{
  "compilerOptions": {
    "module": "nodenext",
    "moduleResolution": "nodenext",
    "target": "es2023",
    "verbatimModuleSyntax": true,
    "rewriteRelativeImportExtensions": true,
    "outDir": "dist",
    "strict": true
  }
}

What each option does, in plain words:

module: nodenext makes TypeScript follow Node's real rules. It reads the same "type" field and file extensions Node does, so a .ts file compiles to ESM or CommonJS depending on where it sits. It also understands require(esm), according to the TypeScript 5.9 release notes. If you prefer a setting that never changes behavior between releases, TypeScript 5.9 added the stable module: node20.
verbatimModuleSyntax tells TypeScript to write your import and export statements exactly as you wrote them, and forces you to mark type-only imports with import type. That removes a whole class of "works in TS, crashes in Node" bugs. One caveat for mixed codebases: with this flag on, TypeScript files that compile to CommonJS must use `import x = require('x')` instead of `import x from 'x'`. Many teams switch it on per folder as each folder becomes ESM, or right after the final flip.
rewriteRelativeImportExtensions lets you write import { x } from './util.ts' and have it emitted as ./util.js. Without it, you must write ./util.js in your TypeScript source even though the file on disk is util.ts. That surprises everyone the first time, but it is correct: the import path describes the output file Node will load.

TypeScript also supports per-file formats through extensions: .mts always compiles to ESM (.mjs), and .cts always compiles to CommonJS (.cjs). They are the TypeScript twins of the .mjs/.cjs trick.

For running TypeScript directly during development, replace ts-node with tsx, which handles both module systems without extra configuration:

{
  "scripts": {
    "dev": "tsx watch src/server.ts",
    "build": "tsc -p tsconfig.json",
    "start": "node dist/server.js"
  }
}

7.2 Tests

The test runner is where most teams lose a day. Know where your runner stands:

The Jest status comes from the Jest 30.4.0 release notes, which say require(esm) is supported but still requires the experimental flag.

If you stay on Jest with a CommonJS codebase, the simplest fix for an ESM-only dependency is to let Jest transpile it. By default, Jest skips everything in node_modules. You add exceptions:

// jest.config.cjs
module.exports = {
  testEnvironment: 'node',
  transform: { '^.+\\.[cm]?[jt]sx?$': 'babel-jest' },
  // Transform these ESM-only packages instead of skipping them
  transformIgnorePatterns: ['/node_modules/(?!(chalk|nanoid|p-queue)/)'],
};

The regular expression reads as: “ignore everything inside node_modules except chalk, nanoid and p-queue". Those three get converted to CommonJS for the test run only.

If you run Jest in native ESM mode, mocking changes. jest.mock() relies on being hoisted above require() calls, and static import statements cannot be intercepted that way. Use the ESM API and a dynamic import:

// orders.test.js (ESM, run with NODE_OPTIONS=--experimental-vm-modules)
import { jest } from '@jest/globals';

// 1. Register the mock BEFORE the module under test is loaded
jest.unstable_mockModule('./payments.js', () => ({
  charge: jest.fn().mockResolvedValue({ id: 'ch_123' }),
}));
// 2. Load the module under test dynamically, so it sees the mock
const { placeOrder } = await import('./orders.js');
const { charge } = await import('./payments.js');
test('charges the customer', async () => {
  await placeOrder({ total: 4200 });
  expect(charge).toHaveBeenCalledWith(4200);
});

The order matters. Because ESM links the whole graph before running anything (Part 1, phase 2), the mock must exist before orders.js is imported. The dynamic import() delays that load until after the mock is registered.

For new projects, or if Jest friction is high, Vitest offers an almost identical API with native ESM support, and the built-in node:test runner needs no configuration at all.

7.3 Linting

Let the linter catch mistakes before Node does:

// eslint.config.js (flat config, ESM)
import n from 'eslint-plugin-n';

export default [
  n.configs['flat/recommended'],
  {
    rules: {
      'n/no-missing-import': 'error',       // catches './util' without .js
      'n/file-extension-in-import': ['error', 'always'],
      'no-restricted-syntax': [
        'error',
        { selector: "CallExpression[callee.name='require']", message: 'Use import in ESM files.' },
      ],
    },
  },
];

Apply the require restriction only to folders you have already converted, using a files glob, so the linter tracks your migration progress.

7.4 A CI guard for the top-level-await contract

While parts of your codebase are still CommonJS, any ESM module they require() must stay free of top-level await. A tiny test enforces that rule automatically:

// test/require-graph.test.cjs  (runs with node --test)
const test = require('node:test');
const assert = require('node:assert');

// Every ESM entry point that CommonJS code still require()s
const requiredByCjs = ['../src/utils/format.js', '../src/utils/slug.js', 'chalk', 'nanoid'];
for (const specifier of requiredByCjs) {
  test(`require(${specifier}) stays synchronous`, () => {
    assert.doesNotThrow(() => require(specifier));
  });
}

If someone adds a top-level await to format.js, or a dependency update sneaks one in, this test fails in CI with ERR_REQUIRE_ASYNC_MODULE instead of failing in production at startup.

Part 8: If you publish packages (internal or public)

Many CommonJS-heavy companies also own a set of internal npm packages: a shared logger, an auth client, a design-token library. Once your consumers run Node 22.12+, the best format for those packages is usually ESM-only.

8.1 Why dual builds are now a liability

For years, the standard advice was to ship both a CommonJS build and an ESM build. That solved the ERR_REQUIRE_ESM problem but created another one: the dual-package hazard.

If one part of an application loads your package with require() and another part uses import, Node can load both builds. They are separate module instances with separate state. A cache filled through the CommonJS copy is empty in the ESM copy. A class exported from one copy fails instanceof checks against objects created by the other. These bugs are miserable to debug because each copy looks correct on its own.

With require(esm), CommonJS consumers load the real ES module, so there is only one instance. Shipping ESM-only removes the hazard and halves your build output.

8.2 A minimal ESM-only package.json

{
  "name": "@acme/logger",
  "version": "3.0.0",
  "type": "module",
  "exports": {
    ".": {
      "types": "./dist/index.d.ts",
      "default": "./dist/index.js"
    }
  },
  "files": ["dist"],
  "engines": { "node": ">=22.12.0" }
}

Field by field:

type: module makes every .js file in the package ESM.
exports defines the public entry points. Anything not listed is private, and consumers get ERR_PACKAGE_PATH_NOT_EXPORTED if they try to deep-import it. The types condition must come first so TypeScript finds the declarations.
engines documents the require(esm) requirement. Bump your major version when you make this change, since older Node users will break.

8.3 Make CommonJS consumers feel at home

If your package has a default export, CommonJS users will otherwise need .default. Apply the trick from Part 3 so require('@acme/logger') returns the logger directly:

// src/index.js
export function createLogger(name) {
  return {
    info: (msg) => console.log(`[${name}] ${msg}`),
    error: (msg) => console.error(`[${name}] ${msg}`),
  };
}

const defaultLogger = createLogger('app');
defaultLogger.createLogger = createLogger;      // keep the named export reachable from CJS
export default defaultLogger;
export { defaultLogger as 'module.exports' };

// A CommonJS consumer
const logger = require('@acme/logger');         // the logger itself, no .default
const { createLogger } = require('@acme/logger');
logger.info('ready');

8.4 If you must still support older runtimes

When some consumers cannot upgrade, use the module-sync condition. Modern Node uses the ESM file for both require and import, so there is still only one instance there. Older runtimes fall back to a CommonJS build.

{
  "type": "module",
  "exports": {
    ".": {
      "types": "./dist/index.d.ts",
      "module-sync": "./dist/index.js",
      "require": "./dist/index.cjs",
      "default": "./dist/index.js"
    }
  }
}

Node checks conditions in the order they appear in the object, so module-sync must come before require. Treat the .cjs fallback as temporary and plan to remove it.

8.5 Treat top-level await as a breaking change

With ESM-only packages consumed through require(esm), your first top-level await, anywhere in the files your entry point imports, breaks every CommonJS consumer with ERR_REQUIRE_ASYNC_MODULE. That makes it a semver-major change. If you need async initialization, export an async function instead:

// Breaks CommonJS consumers — avoid in libraries
export const client = await connect();

// Safe for everyone - the consumer decides when to await
let clientPromise;
export function getClient() {
  clientPromise ??= connect();
  return clientPromise;
}

8.6 Verify before publishing

Two community tools catch most packaging mistakes:

npx publint                              # checks exports, files, and format consistency
npx @arethetypeswrong/cli --pack .       # checks TypeScript types for each resolution mode

Add both to your release pipeline, along with a test that require()s and import()s the packed tarball in a clean folder.

Part 9: Error cheat sheet, checklist, and wrap-up

9.1 Errors you will meet, and what they mean

9.2 The one-page checklist

Audit

Every environment (local, CI, Docker, serverless, cron) runs Node 22.12 or newer
engines.node set to >=22.12.0 in package.json
Dependencies classified as CJS, dual, or ESM-only (esm-audit.cjs)
require-smoke.cjs passes, or failures are understood
CommonJS-only patterns in your source counted

Choose

Strategy decided per area: A (stay CJS), B (async boundary), C (full ESM)
Modules that need top-level await isolated behind a startup await import()

Migrate

Held-back ESM-only dependencies upgraded through require(esm)
Circular dependencies removed before converting
Leaf folders converted with a nested { "type": "module" }
Transitional export { x as 'module.exports' } added where CJS parents expect a single value
Final flip: root "type": "module", leftovers renamed to .cjs, transitional exports removed

Harden

TypeScript on module: nodenext (or node20) with verbatimModuleSyntax
Test runner configured for ESM (Vitest, node:test, or Jest with transforms)
Lint rules enforce extensions and ban require in ESM folders
CI test guards the top-level-await contract for everything CJS still requires

9.3 Wrapping up

The CommonJS/ESM split was painful for a decade because of one asymmetry: CommonJS could not load ESM. On every supported Node.js release, that asymmetry is gone for any module without top-level await. That changes the migration from an all-or-nothing rewrite into an ordinary engineering project.

The practical path for a CommonJS-heavy codebase looks like this:

Upgrade Node to 22.12+ and make it explicit with engines.
Stop pinning old dependencies. Upgrade ESM-only packages and load them with require(), minding .default.
Isolate the async exceptions behind a single await import() at startup.
Migrate leaf-first, in waves, if the codebase has a long future, shipping after each wave.
Guard the new contract in CI: anything CommonJS requires must stay free of top-level await.

You do not have to convert a single file today to benefit. Step 2 alone clears your security backlog of held-back majors. Everything after that can happen one folder at a time.

If this playbook helped, follow for more deep dives on Node.js architecture, and share the error you are fighting in the comments. I read them all.

Sources

Gang Scheduling for Distributed Training: Why the Default K8s Scheduler Fails Multi-Pod AI Jobs

Vasanthan K — Mon, 26 Oct 2026 13:01:03 GMT

The 7-out-of-8 problem

You submit a distributed training job that needs 8 GPU workers. Kubernetes happily starts 7 of them. The 8th sits in Pending because the cluster ran out of GPUs. Those 7 running pods are not training anything. They are holding 7 expensive GPUs hostage while they wait for a teammate that may never arrive.

Now imagine a second team submits a similar 8-worker job at the same moment. Kubernetes splits the cluster between them: 4 GPUs to job A, 4 GPUs to job B. Neither job can ever start. Neither job will give its GPUs back. Your cluster is 100% “allocated” and 0% productive. This is called a resource deadlock, and it is one of the most common (and most expensive) surprises teams hit when they move AI training onto Kubernetes.

The fix has a name: gang scheduling. The idea is simple: a group of pods that must run together is scheduled all-or-nothing. Either every member of the gang gets a seat, or nobody sits down.

In this post we will build up the idea from first principles, so it works even if you have never trained a model across machines.

What you will learn:

Why distributed training needs every worker alive at the same time
How the default kube-scheduler places pods, and why that design breaks multi-pod AI jobs
How to reproduce the deadlock with plain YAML
What gang scheduling is, and the key ideas behind it (minMember, minCount, quorum, timeouts)
Four ways to get gang scheduling today: the coscheduling plugin, Volcano, Kueue, and the new native Kubernetes Workload / PodGroup API (Beta in v1.37)
A hands-on lab on a local kind cluster, plus production tips

Who this is for: developers and platform engineers who know basic Kubernetes objects (Pod, Job, Deployment) and want to run ML training reliably. Every code block comes with a line-by-line explanation, and every diagram in this post was drawn fresh for it.

Part 1: A 5-minute primer on distributed training

Before we blame the scheduler, we need to understand why AI training jobs are so picky. The short version: the workers of a training job talk to each other on every single step, so they are only useful as a complete set.

Why split training across many GPUs?

Modern models are trained on huge datasets. One GPU would take weeks or months. So we split the work across many GPUs, often on many machines. The most common way to do this is called data parallelism:

Every worker (one per GPU) holds a full copy of the model.
Each worker gets a different small batch of training data.
Each worker runs the forward pass and backward pass on its batch and computes gradients (the “which way should each weight move” numbers).
All workers average their gradients together. This step is called all-reduce.
Every worker applies the same averaged update, so all model copies stay identical.
Repeat for the next batch, thousands of times.

Step 4 is the catch. It is a collective operation: it only completes when every worker has contributed. Libraries like PyTorch’s torch.distributed with the NCCL backend usually implement it as a ring, where each worker passes chunks of gradients to its neighbour.

Look at the right side of the diagram. If W3 never starts, W2 has nobody to send its gradients to, and W0 has nobody to receive from. The ring is broken. W0, W1 and W2 will sit at their very first step, waiting. Usually they wait until a timeout (often 30 minutes or more), crash, and get restarted, only to wait again.

What this looks like in code

Here is the skeleton of a typical PyTorch distributed training script. You do not need to run it yet. Just notice the two places where the code blocks until the full group is present.

# train.py  (simplified PyTorch DDP skeleton)
import os
import torch
import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel as DDP

def main():
    # 1. Join the process group.
    #    WORLD_SIZE = total number of workers (e.g. 8)
    #    RANK       = this worker's id (0..7)
    #    MASTER_ADDR/PORT = where rank 0 listens for the others
    dist.init_process_group(backend="nccl")   # BLOCKS until all WORLD_SIZE ranks have joined
    local_rank = int(os.environ["LOCAL_RANK"])
    torch.cuda.set_device(local_rank)
    model = MyModel().cuda(local_rank)
    model = DDP(model, device_ids=[local_rank])  # wraps the model so gradients are all-reduced
    for batch in loader:
        loss = model(batch).loss
        loss.backward()        # BLOCKS inside the all-reduce until every rank sends its gradients
        optimizer.step()
        optimizer.zero_grad()
    dist.destroy_process_group()
if __name__ == "__main__":
    main()

Line by line, the important parts:

dist.init_process_group(backend="nccl") is a rendezvous. Every worker connects to the master address and waits until WORLD_SIZE workers have checked in. If one worker is stuck in Pending, the other seven hang right here.
DDP(model, ...) wraps the model so that during loss.backward() the gradients are automatically all-reduced across all workers.
loss.backward() therefore also blocks on the slowest (or missing) worker, on every step.

So a distributed training job is not “8 independent pods”. It is one program that happens to run in 8 places. It has exactly two useful states: all workers running, or none. Anything in between is pure waste. Keep that in mind as we look at how Kubernetes schedules pods.

Part 2: How the default kube-scheduler thinks

The kube-scheduler is the control-plane component that decides which node each pod runs on. It is excellent at what it was designed for: web servers, APIs and microservices, where every replica is independent. If 9 out of 10 replicas of a web server start, you are serving 90% of traffic. Partial success is still success.

That design shows up in one core fact: the scheduler works on one pod at a time.

The scheduling cycle, step by step

New pods with no node assigned land in a priority queue called the activeQ. The scheduler pops one pod off the queue and runs it through a pipeline of plugin “extension points” defined by the scheduling framework:

Then the scheduler moves to the next pod in the queue and does it all again.

The missing question

Notice what never happens in that loop. The scheduler never asks:

“This pod belongs to an 8-worker training job. Can all 8 workers fit right now?”

It only asks: “Does this one pod fit on some node right now?” Each decision is greedy (take the best option now) and local (ignore everything except this pod). The scheduler does not even know that A-worker-0 and A-worker-1 are related. To it, they are two unrelated pods that happen to have similar names.

Two more details make things worse for AI jobs:

Pods from different jobs interleave in the queue. If job A and job B are submitted at the same time, the queue looks like A0, B0, A1, B1, A2, B2... (ordered by priority and creation time). The scheduler hands out GPUs in that alternating order.
Once bound, a pod keeps its resources. The scheduler will not take GPUs back from a running pod just because its job cannot make progress. (Preemption only kicks in for a higher priority pod, and it is also pod-by-pod.)

One green box in the diagram deserves attention: Permit. It can tell a pod to wait without binding. That small hook is exactly what gang scheduling plugins use, as we will see in Part 4.

Part 3: Where it breaks: four failure modes

Put Part 1 and Part 2 together and the problem becomes obvious. Training needs all-or-nothing. The scheduler delivers one-at-a-time. That mismatch causes four distinct failures.

Failure 1: Partial scheduling (the 7-of-8 problem)

The cluster has 7 free GPUs. Your job asks for 8. The scheduler binds 7 pods and leaves the 8th Pending. The 7 running pods block in init_process_group() and burn GPU hours doing nothing. Meanwhile a smaller job that only needed 4 GPUs could have run and finished, but now it cannot get any.

Failure 2: Resource deadlock (the one that hurts most)

Two jobs, each needing 8 GPUs, on a cluster with 8 GPUs in total. Because their pods interleave in the queue, each job gets half.

This is a classic deadlock, and it ticks all the textbook conditions:

Mutual exclusion: a GPU can only be used by one pod.
Hold and wait: each job holds 4 GPUs while waiting for 4 more.
No preemption: Kubernetes will not take GPUs away from a running pod of equal priority.
Circular wait: A waits for B’s GPUs, B waits for A’s.

No one is “wrong” here. The scheduler made a perfectly reasonable choice for each pod. The deadlock lives between the decisions.

Failure 3: Timeout-and-restart thrashing

The stuck workers eventually hit the NCCL or rendezvous timeout and crash. The Job controller restarts them. If the scheduler again gives them a partial set, the cycle repeats. Logs fill with timeout errors that look like a networking bug, which sends engineers on a long hunt in the wrong direction.

Failure 4: Autoscaler confusion

With cluster autoscaling, a partially scheduled job may trigger scale-up for only the pending pods, one node at a time. The job is slow to start and you pay for the idle half while the rest arrives. A gang-aware stack can instead ask for the whole set of capacity at once.

Reproducing the deadlock with plain YAML

Here is what a typical distributed training job looks like with only built-in Kubernetes objects: an Indexed Job plus a headless Service so workers can find each other by DNS. We will run this exact pair in the hands-on lab later.

# job-a.yaml  (job-b.yaml is identical with "a" replaced by "b")
apiVersion: v1
kind: Service
metadata:
  name: train-a              # headless service: gives each pod a stable DNS name
spec:
  clusterIP: None
  selector:
    job-name: train-a
---
apiVersion: batch/v1
kind: Job
metadata:
  name: train-a
spec:
  completionMode: Indexed    # each pod gets an index 0..7 -> use it as the RANK
  completions: 8
  parallelism: 8             # we want all 8 workers running at the same time
  backoffLimit: 0
  template:
    spec:
      subdomain: train-a     # pod DNS: train-a-0.train-a, train-a-1.train-a, ...
      restartPolicy: Never
      containers:
      - name: worker
        image: busybox:1.36
        command: ["sh", "-c", "echo rank=$JOB_COMPLETION_INDEX waiting for peers; sleep 3600"]
        resources:
          requests:
            example.com/gpu: 1   # a fake GPU resource we advertise in the lab
          limits:
            example.com/gpu: 1

How to read it:

kind: Service with clusterIP: None creates a headless Service. It has no load-balancing IP; it just publishes DNS records so worker 3 can reach worker 0 at train-a-0.train-a. Real training uses this as the MASTER_ADDR.
completionMode: Indexed gives each pod a number in the env var JOB_COMPLETION_INDEX. Training scripts map it to RANK.
parallelism: 8 tells the Job controller to create 8 pods at once. Note: this asks for 8 pods to be created. It does not ask the scheduler to place them together. That gap is the whole problem.
example.com/gpu: 1 is an extended resource. On a real cluster this would be nvidia.com/gpu: 1. In the lab we advertise a fake one so you can reproduce GPU behaviour on a laptop.
The container just sleeps. In real life this is where torchrun train.py would block in init_process_group().

Apply job-a.yaml and job-b.yaml at the same moment on a cluster with 8 fake GPUs, and you will usually see something like this:

$ kubectl get pods -o wide
NAME              READY   STATUS    NODE
train-a-0-xxxxx   1/1     Running   worker-1
train-a-1-xxxxx   1/1     Running   worker-2
train-a-2-xxxxx   1/1     Running   worker-1
train-a-3-xxxxx   1/1     Running   worker-2
train-a-4-xxxxx   0/1     Pending   
...
train-b-0-xxxxx   1/1     Running   worker-1
train-b-1-xxxxx   1/1     Running   worker-2
...
train-b-7-xxxxx   0/1     Pending

The exact split varies from run to run (it depends on how pod creation interleaves), but any split other than 8/0 leaves both jobs unable to train. kubectl describe pod train-a-4-xxxxx will show the event 0/2 nodes are available: 2 Insufficient example.com/gpu. Kubernetes considers this a normal, healthy state. Your training team does not.

Part 4: What gang scheduling actually is

Gang scheduling means a set of related pods is treated as one scheduling unit: the whole gang is placed together, or none of it is placed. The term comes from 1980s parallel computing, where operating systems learned to run all threads of a parallel program at the same moment on different CPUs, for exactly the same reason: the threads constantly wait on each other.

A real-world analogy: a restaurant does not seat half of a party of eight and tell the others to wait outside. The host either finds a table for eight, or the whole party waits in the lobby, without occupying any tables. The tables stay free for smaller groups in the meantime.

The building blocks

Every gang scheduling implementation, whatever its name, is built from the same few ideas:

Why is the minimum size configurable instead of always “all pods”? Some frameworks can start with fewer workers. Elastic training (for example torchrun with a range like --nnodes=4:8) can begin with 4 workers and grow to 8. For a classic fixed-size job, set the minimum equal to the total number of workers.

The decision flow

Walking through the diagram:

Quorum check. A controller (Job, JobSet, Volcano, your operator) creates the group object and its pods. The scheduler does nothing until enough pods exist. This prevents the scheduler from wasting effort on a gang whose members have not even been created yet.
Tentative placement. The scheduler runs the normal Filter and Score steps for each member. Instead of binding, each placed pod waits at the Permit gate with its resources reserved.
The verdict. If at least the minimum number of pods found a place, all of them are approved and bound together. The workers start at roughly the same moment, and init_process_group() succeeds quickly.
The release. If not, nothing is bound. Reservations are undone and the GPUs go back to the pool, so other jobs can use them. The gang retries later when resources free up.

Apply this to our deadlock. Job A’s 8 pods are placed tentatively. Job B’s pods find no room and get released before they ever hold anything. Job A runs. When it finishes, job B gets its turn. The cluster went from 0% useful work to 100%, without buying a single GPU.

Two layers where “all-or-nothing” can live

One more idea will make the tools in the next sections much easier to understand. All-or-nothing can be enforced at two different layers:

Scheduling layer (where do the pods go?). The scheduler itself refuses to bind a partial gang. This is what the coscheduling plugin, Volcano, and the new native GangScheduling plugin do.
Admission layer (when may the job start at all?). A job queue keeps the whole job suspended until there is quota for all of it, then lets it through. If the pods still do not all become Ready in time, the queue evicts the whole job and requeues it. This is what Kueue does.

These layers are complementary. Many production platforms use Kueue for fair sharing between teams and a gang-aware scheduler for atomic placement.

Now let’s look at each option with real YAML.

Part 5: Option 1: the Coscheduling plugin (scheduler-plugins)

The Coscheduling plugin lives in the kubernetes-sigs/scheduler-plugins project, maintained by SIG Scheduling. It is the smallest step away from the default scheduler: it runs the same kube-scheduler code, plus a plugin that hooks into the extension points we met in Part 2. It works on clusters older than v1.37.

Step 1: Install it as a second scheduler

The safest way to try it is to run it next to the default scheduler. Only pods that ask for it by name use it; everything else is untouched.

# Installs the scheduler-plugins scheduler + its PodGroup controller
helm install --repo https://scheduler-plugins.sigs.k8s.io scheduler-plugins scheduler-plugins \
  --namespace scheduler-plugins --create-namespace

kubectl get deploy -n scheduler-plugins
# NAME                           READY
# scheduler-plugins-controller   1/1    <- keeps PodGroup status up to date
# scheduler-plugins-scheduler    1/1    <- the gang-aware scheduler

Two components appear. The scheduler makes placement decisions. The controller watches pods and updates each PodGroup's status (how many members are scheduled, running, failed).

Step 2: Understand the scheduler configuration

The chart ships a KubeSchedulerConfiguration similar to this. You rarely edit it, but reading it shows exactly how the plugin works:

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
- schedulerName: scheduler-plugins-scheduler   # pods opt in with this name
  plugins:
    multiPoint:
      enabled:
      - name: Coscheduling       # register the plugin at every point it implements
    queueSort:
      enabled:
      - name: Coscheduling       # replace the default queue ordering...
      disabled:
      - name: "*"                # ...only one queueSort plugin may be active

What each hook does:

QueueSort: orders the queue by priority, then by the PodGroup’s creation time. Members of the same gang end up next to each other instead of interleaved with another job. This alone reduces the A/B/A/B split from Part 3.
PreFilter: before trying a pod, checks that at least minMember pods of its group exist. If not, it rejects early (this is the quorum check).
PostFilter: if one member cannot be placed, the plugin can reject the rest of its group’s waiting pods, so they release their reservations immediately instead of holding them until timeout.
Permit: counts running pods plus pods waiting at Permit. If the count reaches minMember, every waiting member is allowed through to Bind. Otherwise the pod is told to wait, for up to scheduleTimeoutSeconds.
Unreserve: when the wait times out, all waiting members are rejected and their resources released.

Step 3: Create a PodGroup and point your pods at it

# podgroup-train-a.yaml
apiVersion: scheduling.x-k8s.io/v1alpha1
kind: PodGroup
metadata:
  name: train-a
spec:
  minMember: 8                 # all 8 workers, or none
  scheduleTimeoutSeconds: 60   # how long placed members wait at Permit for the rest
  # minResources:              # optional: total resources the gang needs; lets the
  #   example.com/gpu: 8       # plugin reject early when the cluster obviously can't fit it

Then two small changes to the Job from Part 3:

# job-a.yaml (only the changed part of the pod template is shown)
  template:
    metadata:
      labels:
        scheduling.x-k8s.io/pod-group: train-a   # 1. join the gang
    spec:
      schedulerName: scheduler-plugins-scheduler # 2. use the gang-aware scheduler
      subdomain: train-a
      restartPolicy: Never
      containers:
      - name: worker
        image: busybox:1.36
        command: ["sh", "-c", "echo rank=$JOB_COMPLETION_INDEX; sleep 3600"]
        resources:
          limits:
            example.com/gpu: 1

Explanation:

The label scheduling.x-k8s.io/pod-group: train-a is how a pod says "I belong to PodGroup train-a". The group name must match the PodGroup in the same namespace.
schedulerName routes the pod to the second scheduler. Without it, the default scheduler places the pod and the gang rules are silently ignored. This is the number one mistake people make.
minMember: 8 equals parallelism: 8. If you set minMember higher than the number of pods the Job creates, the gang can never be satisfied and stays Pending forever.

Repeat for train-b and apply both at the same time. Now one job gets all 8 GPUs and runs; the other sits fully Pending with zero GPUs held:

$ kubectl get podgroups
NAME      PHASE       MINMEMBER   RUNNING   AGE
train-a   Running     8           8         40s
train-b   Pending     8                     40s

Good fit for: clusters before v1.37, teams that want gang semantics with minimal new concepts, and existing Job / PyTorchJob / MPIJob workloads where you can add a label.

Watch out for: you are running and upgrading a second scheduler. Two schedulers do not coordinate with each other, so avoid letting both place pods onto the same GPU nodes (use node labels or taints to give each its own pool).

Part 6: Option 2: Volcano

Volcano is a CNCF batch system built for AI, big data and HPC. Where the coscheduling plugin adds one feature to the normal scheduler, Volcano is a whole platform: its own scheduler, its own job type (vcjob), queues with weights and capacity, and job lifecycle policies. Gang scheduling is its core plugin, called gang.

Volcano installs three components: volcano-scheduler (placement, including gang), volcano-controllers (manages Jobs, Queues and PodGroups), and volcano-admission (a webhook that validates your YAML).

A queue for the team

apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: research
spec:
  weight: 1              # share of the cluster relative to other queues
  reclaimable: true      # other queues may borrow idle capacity and give it back
  capability:
    nvidia.com/gpu: 16   # hard ceiling for this queue

A Queue is a bucket of capacity. Jobs are admitted into a queue only if the queue can hold their minimum gang. This gives Volcano an admission layer as well as a scheduling layer.

A PyTorch training job with gang semantics

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: llm-finetune
spec:
  schedulerName: volcano     # 1. use the Volcano scheduler
  queue: research            # 2. which Queue this job is charged to
  minAvailable: 4            # 3. gang size: 1 master + 3 workers, all or nothing
  maxRetry: 3
  plugins:
    pytorch: ["--master=master", "--worker=worker", "--port=23456"]  # injects MASTER_ADDR, RANK, WORLD_SIZE
  policies:
  - event: PodEvicted        # 4. if any pod is evicted or fails...
    action: RestartJob       #    ...restart the WHOLE job, not just that pod
  - event: PodFailed
    action: RestartJob
  tasks:
  - name: master
    replicas: 1
    template:
      spec:
        restartPolicy: OnFailure
        containers:
        - name: trainer
          image: my-registry/llm-train:1.0
          command: ["torchrun", "train.py"]
          resources:
            limits:
              nvidia.com/gpu: 1
  - name: worker
    replicas: 3
    template:
      spec:
        restartPolicy: OnFailure
        containers:
        - name: trainer
          image: my-registry/llm-train:1.0
          command: ["torchrun", "train.py"]
          resources:
            limits:
              nvidia.com/gpu: 1

The fields that matter:

schedulerName: volcano hands these pods to Volcano instead of the default scheduler.
queue: research charges the job against the team's Queue.
minAvailable: 4 is the gang size. Volcano automatically creates a PodGroup behind the scenes with minMember: 4. The gang plugin will not bind any of the 4 pods until all 4 fit.
policies handle the runtime side of all-or-nothing. If one worker dies mid-training, the other three are useless (Part 1), so RestartJob tears down and restarts the whole gang. This is something the plain Job controller cannot express.
plugins.pytorch wires up the environment variables (MASTER_ADDR, WORLD_SIZE, RANK) so torchrun can rendezvous. Volcano also ships ssh and svc plugins for MPI-style jobs.
tasks lets one job contain different roles (master, workers, parameter servers) with different templates, something a single batch/v1 Job cannot do.

Check it:

$ kubectl get vcjob llm-finetune
NAME           STATUS    MINAVAILABLE   RUNNINGS   AGE
llm-finetune   Running   4              4          30s

$ kubectl get podgroup
NAME                                           STATUS    MINMEMBER   RUNNINGS
llm-finetune-6a1c...                           Running   4           4

If the queue cannot fit 4 GPUs at once, the PodGroup stays in the Inqueue / Pending phase and no pod is bound.

Good fit for: dedicated AI/HPC platforms, MPI and multi-role jobs, teams that want queues, fair share and lifecycle policies in one package. Recent releases also add gang-granularity preemption and topology-aware placement.

Watch out for: Volcano is a second scheduler with its own concepts to learn and operate. Pods that do not set schedulerName: volcano are still placed by the default scheduler, so plan your node pools carefully.

Part 7: Option 3: Kueue (all-or-nothing at admission)

Kueue takes a different approach. It is not a scheduler. It is a job queueing controller that decides when a whole job is allowed to create pods. The default scheduler (or any other) still decides where those pods go.

The trick Kueue relies on is the suspend field of batch Jobs (and JobSet, RayJob, PyTorchJob and others). A suspended job exists but creates no pods. Kueue holds every job suspended until there is enough quota for the entire job, then flips suspend to false. Because quota is checked for the whole job at once, two 8-GPU jobs are never admitted against 8 GPUs of quota at the same time. Most of the deadlock disappears before the scheduler is involved.

Step 1: Describe your capacity

apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
  name: gpu-a100            # a "kind" of node; one is enough if your GPUs are identical
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: gpu-cluster-queue
spec:
  namespaceSelector: {}     # accept workloads from every namespace
  resourceGroups:
  - coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
    flavors:
    - name: gpu-a100
      resources:
      - name: "cpu"
        nominalQuota: 64
      - name: "memory"
        nominalQuota: 512Gi
      - name: "nvidia.com/gpu"
        nominalQuota: 8     # the whole cluster has 8 GPUs
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue
metadata:
  name: research
  namespace: ml-team
spec:
  clusterQueue: gpu-cluster-queue   # the namespaced "front door" users submit to

A ResourceFlavor names a type of hardware.
A ClusterQueue holds the quota: here, 8 GPUs in total. Several ClusterQueues can share unused quota through a cohort.
A LocalQueue is the per-namespace entry point. Users only need to know its name.

Step 2: Submit a JobSet to the queue

JobSet is the Kubernetes-native API for multi-role distributed training (a leader plus workers, each as its own Job, with shared DNS). Kueue admits a JobSet as one unit.

apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
  name: train-a
  namespace: ml-team
  labels:
    kueue.x-k8s.io/queue-name: research   # 1. send this workload to the LocalQueue
spec:
  failurePolicy:
    maxRestarts: 3                        # 2. a failed child Job restarts the whole JobSet
  replicatedJobs:
  - name: workers
    replicas: 1
    template:
      spec:
        parallelism: 8
        completions: 8
        completionMode: Indexed
        backoffLimit: 0
        template:
          spec:
            restartPolicy: Never
            containers:
            - name: trainer
              image: my-registry/llm-train:1.0
              command: ["torchrun", "train.py"]
              resources:
                requests:
                  cpu: "6"
                  memory: 48Gi
                limits:
                  nvidia.com/gpu: 1

The kueue.x-k8s.io/queue-name label is the only change your users make. Kueue's webhook suspends the JobSet on creation.
JobSet’s failurePolicy restarts all child Jobs together when one fails, the runtime half of all-or-nothing.

Kueue sums the requests (8 GPUs, 48 CPU, 384Gi), checks the ClusterQueue, and admits the JobSet only if all of it fits. A second 8-GPU JobSet waits in the queue with zero pods created.

Step 3: Turn on waitForPodsReady

Quota is not the same as physical placement. Quota might say “8 GPUs free” while those GPUs are fragmented across nodes, or a node is unhealthy, or another scheduler grabbed some. Kueue’s answer is a timeout-based all-or-nothing check, configured in Kueue’s controller configuration:

# part of Kueue's controller manager configuration (kueue-manager-config ConfigMap)
waitForPodsReady:
  enable: true
  timeout: 10m            # all pods must be Ready within 10 minutes of admission
  blockAdmission: true    # admit the next workload only after the current one is fully Ready
  requeuingStrategy:
    timestamp: Eviction   # requeued jobs go to the back of the queue
    backoffLimitCount: 5  # give up (deactivate) after 5 requeues
    backoffBaseSeconds: 60

timeout: if not every pod of an admitted job becomes Ready in time, Kueue evicts the whole job: it suspends it again (deleting its pods and freeing their GPUs) and puts it back in the queue.
blockAdmission: admits jobs one at a time until each is fully Ready. This is what removes the A/B race completely, at the cost of slower ramp-up when many jobs arrive at once.
requeuingStrategy: controls where an evicted job goes in the queue and how many times it may retry, with exponential backoff.

Field names follow the Kueue documentation; check the configuration reference for the exact shape in your Kueue version.

Good fit for: shared clusters with several teams, quotas, priorities, fair sharing, and multi-cluster dispatch (MultiKueue). It integrates with Job, JobSet, Kubeflow training jobs, RayJob, LeaderWorkerSet and plain pod groups.

Watch out for: on its own, Kueue gives all-or-nothing by timeout, not by atomic placement: for a few minutes, a partially placed job can still hold GPUs before Kueue evicts it. Pair Kueue with a gang-aware scheduler (the native API in the next part, or coscheduling) when that window is too expensive.

Part 8: Option 4: native gang scheduling with the Workload and PodGroup APIs

For years, gang scheduling on Kubernetes meant installing something extra. That is changing. SIG Scheduling has been building Workload-Aware Scheduling directly into kube-scheduler, and in Kubernetes v1.37 (released August 2026) the Workload and PodGroup APIs and gang scheduling graduated to Beta.

A quick history (so old blog posts don’t confuse you)

If you find YAML online with spec.workloadRef on a pod, it targets the v1.35 alpha and will not work as-is on v1.37.

The three objects

The design separates long-lived policy from per-run state, the same way a Deployment (policy) relates to ReplicaSets (state):

Workload: a template that says “jobs of this type have a group called worker that must be gang-scheduled with at least N pods".
PodGroup: one concrete running instance of that group. It carries the policy (copied from the template) and the scheduling status.
Pod: points at its PodGroup through spec.schedulingGroup.podGroupName.

In day-to-day use a controller creates the Workload and PodGroup for you. As of v1.37 the built-in Job controller is the first to do this (Alpha), and out-of-tree controllers can use the new workloadbuilder Go library. Writing them by hand, as below, is the best way to see how it works.

Enable it

Beta features are usually on by default, but this one is disabled by default in v1.37. You need the GenericWorkload feature gate on every relevant component and the scheduling.k8s.io/v1beta1 API group served:

kube-apiserver           --feature-gates=GenericWorkload=true --runtime-config=scheduling.k8s.io/v1beta1=true
kube-scheduler           --feature-gates=GenericWorkload=true
kube-controller-manager  --feature-gates=GenericWorkload=true   # needed for controller integrations

On a managed cloud cluster, check whether your provider lets you set feature gates on the control plane; many do not for Beta features. The hands-on lab shows how to do it on kind.

Gang-scheduling 8 training workers natively

apiVersion: scheduling.k8s.io/v1beta1
kind: Workload
metadata:
  name: llm-training             # long-lived policy: "how jobs of this kind are scheduled"
spec:
  podGroupTemplates:
  - name: worker
    schedulingPolicy:
      gang:
        minCount: 8              # all 8 workers must be placeable together
---
apiVersion: scheduling.k8s.io/v1beta1
kind: PodGroup
metadata:
  name: train-a-workers          # one concrete run
spec:
  podGroupTemplateRef:           # optional link back to the template (for tooling)
    workload:
      workloadName: llm-training
      podGroupTemplateName: worker
  schedulingPolicy:
    gang:
      minCount: 8
---
apiVersion: v1
kind: Pod
metadata:
  name: train-a-worker-0         # ...repeat for worker-1 .. worker-7
spec:
  schedulingGroup:
    podGroupName: train-a-workers   # join the gang
  containers:
  - name: trainer
    image: my-registry/llm-train:1.0
    resources:
      limits:
        nvidia.com/gpu: 1

Piece by piece:

Workload.spec.podGroupTemplates[].schedulingPolicy chooses gang (all-or-nothing) or basic (normal pod-by-pod scheduling, but grouped for observability).
gang.minCount is the native equivalent of minMember / minAvailable.
PodGroup.spec.schedulingPolicy holds the policy that actually applies to this run. When a controller creates the PodGroup it copies this from the template; for a standalone PodGroup you set it yourself.
Pod.spec.schedulingGroup.podGroupName is the membership link. No schedulerName is needed: the default kube-scheduler does the gang work.

What the scheduler does with it

According to the gang scheduling docs, the built-in GangScheduling plugin follows the flow from Part 4 almost exactly:

PreEnqueue: the pods are held out of the scheduling queue until the PodGroup exists and at least minCount pods have been created.
PodGroup scheduling cycle: the scheduler then evaluates all unscheduled pods of the group in one atomic decision, checking via a PlacementFeasible hook whether minCount can still be reached.
Bind or release: if at least minCount pods found a place, they are bound. If not, none are; the group goes to the unschedulable queue and other workloads can use the capacity meanwhile.

The scheduler reports the result in status.conditions on the PodGroup:

kubectl get podgroups
kubectl describe podgroup train-a-workers   # look for the PodGroupInitiallyScheduled condition

One subtlety from the docs: PodGroupInitiallyScheduled only describes the initial placement. If a running worker is later evicted, the condition does not flip back, and the scheduler will only add replacement pods if the total can reach minCount again. For runtime failures you still want a controller policy that restarts the whole gang (JobSet failurePolicy, Volcano RestartJob, or Kueue eviction).

Related v1.37 features worth knowing

Workload-aware preemption (Beta): when a high-priority gang needs room, the scheduler can evict a whole lower-priority PodGroup as a unit (controlled by the PodGroup’s disruptionMode and priorityClassName) instead of killing random single pods that would break several jobs at once.
DRA ResourceClaims for PodGroups (Beta): a PodGroup can request devices through spec.resourceClaims and share a claim across its pods.
CompositePodGroup (Alpha): gangs of gangs, for example “schedule at least 2 complete replicas of this 8-pod training group”, which maps naturally to JobSet and LeaderWorkerSet.

Good fit for: clusters on v1.37+ where you control feature gates, and teams that want gang semantics without operating a second scheduler.

Watch out for: it is Beta and off by default, the controller ecosystem (JobSet, Kubeflow, Kueue integration) is still adopting it, and managed providers may not expose the gate yet.

Part 9: Side-by-side comparison and how to choose

All four options solve the same core problem. They differ in where they act, how much they bring along, and how much you have to operate.

A few practical combinations that work well:

Shared enterprise GPU cluster: Kueue for quotas and fair sharing between teams, plus the native PodGroup API (on v1.37+) or the coscheduling plugin for atomic placement.
Dedicated research / HPC cluster: Volcano on its own, using its Queues for sharing and RestartJob policies for runtime failures.
Small team, one cluster, older Kubernetes: the coscheduling plugin. One Helm install, one label, one PodGroup per job.
Greenfield cluster on v1.37+: start with the native API to avoid running a second scheduler, and keep an eye on the Job controller and JobSet integrations as they mature.

Whichever you choose, never run two gang-aware schedulers against the same GPU nodes. Each scheduler keeps its own view of free resources, and they will race each other into exactly the kind of partial placement you are trying to avoid.

Part 10: Hands-on lab: break it, then fix it, on your laptop

You do not need GPUs to see gang scheduling in action. We will create a local cluster with kind, advertise fake GPUs as an extended resource, reproduce the deadlock, and then fix it with the native PodGroup API.

You need: Docker, kind, and kubectl, with a kind node image for Kubernetes v1.37 or newer. (On an older version, skip to the coscheduling variant at the end.)

Step 1: Create a 2-worker cluster with the gang feature gate on

# kind-gang.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: gang-lab
featureGates:
  GenericWorkload: true                # turns on Workload/PodGroup + gang scheduling in every component
runtimeConfig:
  "scheduling.k8s.io/v1beta1": "true"  # serve the Beta API group
nodes:
- role: control-plane
- role: worker
- role: worker

kind create cluster --config kind-gang.yaml
kubectl api-resources --api-group=scheduling.k8s.io
# should list: podgroups, workloads, priorityclasses

The top-level featureGates and runtimeConfig in a kind config are applied to every control-plane component, which is exactly what the docs ask for. If podgroups is missing from the output, the gate or API group did not take effect.

Step 2: Give each worker node 4 fake GPUs

Kubernetes lets you advertise any countable extended resource on a node by patching the node’s status. The scheduler then treats it exactly like nvidia.com/gpu.

for node in gang-lab-worker gang-lab-worker2; do
  kubectl patch node "$node" --subresource=status --type=json \
    -p '[{"op":"add","path":"/status/capacity/example.com~1gpu","value":"4"}]'
done

kubectl get nodes -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.example\\.com/gpu
# gang-lab-control-plane   
# gang-lab-worker          4
# gang-lab-worker2         4

The ~1 in the path is JSON-Patch escaping for /, so example.com~1gpu means the key example.com/gpu. We now have a cluster with 8 "GPUs" in total.

Step 3: Reproduce the deadlock

Save the job-a.yaml from Part 3, create job-b.yaml by replacing every train-a with train-b, and submit both together:

sed 's/train-a/train-b/g' job-a.yaml > job-b.yaml
kubectl apply -f job-a.yaml -f job-b.yaml
sleep 10
kubectl get pods -l 'job-name in (train-a,train-b)' \
  -o custom-columns=POD:.metadata.name,STATUS:.status.phase,NODE:.spec.nodeName | sort

In most runs you will see both jobs with some Running and some Pending pods, and 8 GPUs fully allocated. (If you get lucky and one job takes all 8, delete both jobs and run it again; the interleaving is a race, which is part of why this bug is so hard to catch in testing.) Clean up before the fix:

kubectl delete -f job-a.yaml -f job-b.yaml

Step 4: Fix it with native gang scheduling

This script creates, for each job, one PodGroup with minCount: 8 and 8 worker pods that reference it:

#!/usr/bin/env bash
# gang-lab.sh
set -euo pipefail

for job in a b; do
  # one PodGroup per training run: all 8 or nothing
  kubectl apply -f - <apiVersion: scheduling.k8s.io/v1beta1
kind: PodGroup
metadata:
  name: train-${job}
spec:
  schedulingPolicy:
    gang:
      minCount: 8
EOF
  # 8 workers that join that PodGroup
  for i in $(seq 0 7); do
    kubectl apply -f - <apiVersion: v1
kind: Pod
metadata:
  name: train-${job}-${i}
  labels:
    app: gang-lab
    run: train-${job}
spec:
  schedulingGroup:
    podGroupName: train-${job}
  restartPolicy: Never
  containers:
  - name: worker
    image: busybox:1.36
    command: ["sh", "-c", "echo rank=${i}; sleep 3600"]
    resources:
      limits:
        example.com/gpu: 1
EOF
  done
done

What the script is doing:

set -euo pipefail stops the script on the first error, so you never end up with a half-created gang by accident.
The outer loop runs twice, once for job a and once for job b, so we recreate the same competition as before.
kubectl apply -f - <
Each pod sets spec.schedulingGroup.podGroupName, which is all it takes to join the gang. There is no schedulerName: the default scheduler handles it.

Run it and watch:

chmod +x gang-lab.sh && ./gang-lab.sh
kubectl get pods -l app=gang-lab -o custom-columns=POD:.metadata.name,STATUS:.status.phase,NODE:.spec.nodeName | sort
kubectl get podgroups

The expected result: one run has 8 Running pods spread across both nodes, and the other has 8 Pending pods holding zero GPUs. kubectl describe podgroup train-b should show that the group could not be placed as a whole. Now free the GPUs and watch the second gang start as a unit:

kubectl delete pods -l run=train-a     # "job A finished"
kubectl get pods -l run=train-b -w     # all 8 move to Running together

That is the whole idea of this post, in one terminal window: the cluster went from “both jobs stuck forever” to “one runs, then the other runs”, with the same 8 GPUs.

Variant: on a cluster older than v1.37

Use the coscheduling plugin from Part 5 instead. Install it with Helm, create PodGroup objects in scheduling.x-k8s.io/v1alpha1 with minMember: 8, and add the scheduling.x-k8s.io/pod-group label plus schedulerName: scheduler-plugins-scheduler to the Job's pod template. The observable result is the same.

Clean up

kind delete cluster --name gang-lab

Part 11: Production tips and common pitfalls

Gang scheduling fixes the core problem, but a few details decide whether it works smoothly in production.

Get the numbers right

Set the gang size equal to the real number of workers for fixed-size training. minMember / minAvailable / minCount larger than the pods your controller creates means the gang waits forever. Smaller than the real need brings back partial starts.
Use a smaller minimum only for elastic jobs that genuinely can start with fewer workers (for example torchrun --nnodes=4:8).
Request whole GPUs and realistic CPU/memory. The gang only fits if every member fits, so an oversized memory request on one pod can block the entire group.

Tune the timeouts

Placement wait (scheduleTimeoutSeconds for coscheduling): long enough for the controller to create all pods, short enough that a hopeless gang releases its reservations quickly. Tens of seconds to a few minutes is typical.
Ready timeout (Kueue waitForPodsReady.timeout): must cover image pulls. Multi-gigabyte training images on cold nodes can take several minutes, so pre-pull images or use a node-local cache, or the timeout will evict healthy jobs.
Framework timeouts (NCCL, rendezvous): with gang scheduling in place, a long rendezvous timeout is no longer your safety net. Keep it moderate so genuine failures surface quickly.

Plan for failures after scheduling

Gang scheduling guarantees the start, not the life of a job. If one worker dies after an hour, the rest stall just like in Part 1. Pair gang placement with a whole-gang restart policy: JobSet failurePolicy, Volcano policies with RestartJob, or Kueue eviction and requeue. And checkpoint regularly so a restart resumes from the last checkpoint instead of from zero.

Watch out for these mistakes

Forgetting schedulerName with coscheduling or Volcano. The pod is silently placed by the default scheduler and the gang rules never apply.
Two gang-aware schedulers on the same nodes. Give each scheduler its own node pool using labels and taints.
Mixed priorities inside one gang. Keep all members at the same PriorityClass; mixed priorities lead to surprising ordering and preemption behaviour.
Fragmentation. 8 free GPUs spread as 1 per node across 8 nodes may still fail a gang that needs fast interconnect. Look at topology-aware scheduling (Kueue TAS, Volcano network topology, native topology-aware scheduling) once basic gang scheduling works.
Autoscaler mismatch. Scale-up should be requested for the whole gang. Kueue’s ProvisioningRequest integration and gang-aware autoscaling work exist for exactly this reason.

Observe it

A handful of commands answer most “why is my job not running?” questions:

kubectl get podgroups -A                     # coscheduling, Volcano or native: gang status
kubectl describe podgroup              # conditions and scheduling events
kubectl get events --field-selector reason=FailedScheduling
kubectl get workloads.kueue.x-k8s.io -A      # Kueue: admitted, pending, evicted
kubectl get clusterqueues                    # Kueue: quota usage per queue

On dashboards, track time from submission to all-workers-Running per job, and the share of GPU hours held by pods whose job is not fully running. That second number is the direct cost of missing gang scheduling, and it should drop to near zero once it is in place.

Conclusion

The default Kubernetes scheduler is not broken. It was designed for independent replicas, and it makes a sensible decision for every single pod. Distributed training breaks that assumption: its workers are one program running in many places, and they are only useful together. Pod-by-pod placement then leads to partial starts, idle GPUs, timeouts, and full deadlocks.

Gang scheduling closes the gap with one rule: place the whole group, or place nothing. You can get it today with the coscheduling plugin, Volcano, or Kueue’s admission-level all-or-nothing, and since Kubernetes v1.37 it is a Beta feature of the default scheduler itself through the Workload and PodGroup APIs. Pick the option that matches your cluster version and how many teams share it, run the lab once to see the difference with your own eyes, and your GPUs will spend their time training models instead of waiting for teammates.

Sources

Kubernetes docs: Gang Scheduling, PodGroup API, PodGroup scheduling policies, Scheduling Framework
Kubernetes blog: v1.37: Advancing Workload-Aware Scheduling, v1.37 release announcement, Introducing JobSet
Enhancement proposals: KEP-4671 Gang Scheduling, KEP-5832 Decouple PodGroup, KEP-5710 Workload-aware preemption
scheduler-plugins: Coscheduling plugin, PodGroup coscheduling KEP
Volcano: VolcanoJob, Gang plugin
Kueue: All-or-nothing with ready Pods, v1beta2 API reference

React’s Independent Foundation: What Leaving Meta’s Umbrella Means for API Stability

Vasanthan K — Fri, 23 Oct 2026 13:01:03 GMT

React no longer belongs to Meta. Here is what that really means for the code you write every day — explained from the ground up, with diagrams and examples.

Why this matters to you, even if you only write components

On February 24, 2026, React, React Native and JSX officially stopped being Meta products. They now belong to the React Foundation, an independent non-profit hosted by the Linux Foundation — the same organization that looks after Linux, Kubernetes and Node.js.

Most headlines treated this as corporate news. But if you write React for a living, the real question is much more practical:

Will the hooks, components and APIs I depend on keep working the way they do today?

That question is about API stability. It is the quiet promise that useState will still behave like useState next year, that an upgrade from 19.2.5 to 19.2.6 will not break your login page, and that if something is going away, you will be warned long before it disappears.

In this article we will walk through it step by step:

What actually happened, and when.
What “API” and “API stability” mean in React — with code, for complete beginners.
How the new two-layer governance model works, and why it was designed that way.
How a React API travels from an idea to a stable feature (release channels).
How deprecations work in practice, using real React 18 → 19 examples.
What changes, what stays the same, and the honest risks.
Defensive coding patterns that protect your app no matter who governs React.
A checklist your team can use today.

You do not need any background in open-source governance. If you know what a React component is, you can follow along.

Part 1 — What actually happened

React was born inside Facebook around 2011 to solve a hard problem: building large interfaces where data keeps changing. It was open-sourced in 2013, and for over a decade Meta employed most of the core team, ran the infrastructure and owned the trademark.

The change happened in two steps:

October 7, 2025 — the announcement. At React Conf 2025, the React team announced plans to move React and React Native from Meta into a new React Foundation, with a separate, independent technical governance structure. Their reasoning was simple: React had outgrown any single company.
February 24, 2026 — the launch. The Linux Foundation officially launched the React Foundation. React, React Native and supporting projects like JSX are now owned by the foundation, with LF Projects, LLC hosting them under neutral governance.

A few facts worth remembering:

The part people miss

Notice what did not happen. No code was rewritten. No API was renamed. Your package.json still says "react": "^19.x", and npm install react still pulls the same package from the same registry.

What changed is who decides — and, over the long run, who decides is exactly what determines how stable an API stays. To see why, we first need to be precise about what an API is.

Part 2 — A beginner’s primer: what is “API stability” in React?

What is an API?

An API (Application Programming Interface) is a contract between two pieces of code. One side promises: “If you call me this way, I will behave this way.” The other side relies on that promise.

For React, the API is everything you import and use: useState, useEffect, createRoot, , the props a component accepts, even the rules for how JSX turns into function calls.

import { useState } from "react";

function Counter() {
  // CONTRACT: useState(initialValue) returns [currentValue, setterFunction]
  const [count, setCount] = useState(0);
  return (
    
  );
}

What is happening here, line by line:

import { useState } from "react" — you are depending on the React package to export something named useState. If a future version renamed it, this line would fail.
useState(0) — the contract says: give me an initial value, and I return an array with two items. The first is the current value; the second is a function that updates it and asks React to re-render.
setCount(count + 1) — the contract also says that calling the setter schedules a re-render with the new value. Your whole UI logic assumes this behavior.

Three things make up the contract: the name (useState), the shape (what goes in, what comes out) and the behavior (what happens when you call it). Change any one, and code that depends on it can break.

What is API stability?

API stability is the promise that this contract will not change without warning. A stable API lets you upgrade React for bug fixes and security patches without rewriting your app.

Think of it like electrical sockets. You can buy a new lamp from any brand and plug it in, because the socket shape is stable. If sockets changed shape every year, every upgrade would mean rewiring the house.

How React communicates stability: Semantic Versioning

React uses Semantic Versioning (semver). Every version number has three parts, and each part is a message to you:

19  .  2  .  6
   │      │     │
   │      │     └── PATCH: bug fixes only. Safe to upgrade.
   │      └──────── MINOR: new features added. Nothing removed.
   └─────────────── MAJOR: breaking changes are allowed here.

A real example: in May 2026, React shipped 19.2.6 along with backports 19.1.7 and 19.0.6. These were patch releases focused on hardening React Server Components — fixes delivered to three release lines at once, so teams on older 19.x lines got them too without a forced feature upgrade.

Your package.json tells npm how much change you accept:

{
  "dependencies": {
    "react": "^19.2.6",
    "react-dom": "~19.2.6"
  }
}

What the symbols mean:

^19.2.6 (caret) — accept any minor or patch update inside 19.x (19.3.0 is OK; 20.0.0 is not). You trust React's promise that minors never break you.
~19.2.6 (tilde) — accept only patch updates (19.2.7 is OK; 19.3.0 is not). More cautious.
19.2.6 (no symbol) — exact version. Maximum control; you upgrade manually.

Notice the important point: semver is only a promise, not a law of physics. It works only because the people who maintain React choose to honor it. That is exactly why the question “who maintains React now?” matters so much for stability.

Part 3 — The two-layer governance model

The most important design decision in the whole transition is this: the people who fund React are not the people who decide React’s APIs. The React team stated that technical governance will always stay independent from the foundation board, with technical direction set by those who contribute to and maintain React.

Layer 1: The React Foundation board (the “business” layer)

The board has one representative from each member company. Its job is operational: owning the trademark and repositories, running CI and infrastructure, organizing React Conf and funding ecosystem programs. Think of it as the landlord who keeps the building running.

Layer 2: Technical governance (the “engineering” layer)

This layer decides what code ships: the roadmap, new APIs, breaking changes and deprecation timelines. Right now, a provisional leadership council is defining the permanent structure. Many observers expect something similar to the technical steering committees used by Node.js or Kubernetes, but as of this writing the final shape has not been published — so treat any specific prediction with caution.

Why separate them? A thought experiment

Imagine a single company that both pays for React and controls its APIs. That company builds a hosting product. A new API that favors its product would be tempting to rush through — and an API it does not need might be quietly deprioritized.

This concern was real, not hypothetical. When the foundation was first announced in 2025, reporting noted that the 21-member React core team included 14 Meta employees and 5 Vercel employees, and parts of the community had voiced unease about vendor influence over features like Server Components.

We can model the separation in code. Below is a toy model — not real React internals — showing the rule the two-layer design enforces:

// A toy model of the governance "firewall" — for illustration only.
const board = {
  role: "operations",
  can: ["fund", "runConference", "manageTrademark", "hostInfrastructure"],
};

const technicalCouncil = {
  role: "engineering",
  can: ["approveRFC", "shipAPI", "deprecateAPI", "scheduleBreakingChange"],
};
function canPerform(body, action) {
  return body.can.includes(action);
}
console.log(canPerform(board, "deprecateAPI"));            // false
console.log(canPerform(technicalCouncil, "deprecateAPI")); // true

Explanation: each governing body is an object with a list of allowed actions. canPerform simply checks whether an action is in that list. The board's list contains only operational powers, so asking it to deprecate an API returns false. That is the whole idea of the firewall: no amount of sponsorship money adds deprecateAPI to the board's list.

What this means for stability

Under single-company ownership, stability depended on one company’s priorities. Under the foundation, it depends on a documented process run by maintainers from many organizations. Processes are slower to change than priorities — and for API stability, slow and predictable is exactly what you want.

Part 4 — How a React API travels from idea to stable

New React APIs do not jump straight into your app. They move through release channels, and each channel carries a different level of promise. This pipeline existed under Meta and continues under the foundation — it is the main machinery that protects stability.

A real journey: useEffectEvent

useEffectEvent is a great example of patience in API design. It sat in experimental builds for a long time while the team refined its shape, and only became stable in React 19.2 (October 2025). Here is the problem it solves.

The problem — an Effect that re-runs too often:

import { useEffect } from "react";

function ChatRoom({ roomId, theme }) {
  useEffect(() => {
    const connection = createConnection(roomId);
    connection.on("connected", () => {
      showNotification("Connected!", theme); // reads theme
    });
    connection.connect();
    return () => connection.disconnect();
  }, [roomId, theme]); // theme is a dependency, so...
}

Why this is a problem: the Effect reads theme, so the rules of hooks require theme in the dependency array. But that means switching from light to dark mode disconnects and reconnects the chat. The connection should depend only on roomId; the theme is just something we want to read at the moment the event fires.

The fix — separate the “event” from the “effect”:

import { useEffect, useEffectEvent } from "react";

function ChatRoom({ roomId, theme }) {
  // An Effect Event always sees the latest props, but is NOT reactive.
  const onConnected = useEffectEvent(() => {
    showNotification("Connected!", theme);
  });
  useEffect(() => {
    const connection = createConnection(roomId);
    connection.on("connected", () => onConnected());
    connection.connect();
    return () => connection.disconnect();
  }, [roomId]); // only roomId - theme changes no longer reconnect
}

Explanation: useEffectEvent wraps a function so that it always reads the latest values of props and state, but the function itself never counts as a dependency. The Effect now re-runs only when roomId changes. Changing the theme no longer tears down the connection.

Why this story matters for stability: if this hook had been rushed to stable early, its first design might have been wrong — and fixing a stable API means a breaking change. Keeping it in experimental until the design was proven is what made the final API trustworthy. The foundation inherits this same channel system.

How to protect yourself from non-stable APIs

A simple rule: never ship experimental or canary APIs in your own production app unless a framework pins and manages them for you. You can enforce this with a tiny check in CI:

// scripts/check-react-channel.js
import { readFileSync } from "node:fs";

const pkg = JSON.parse(readFileSync("package.json", "utf8"));
const version = pkg.dependencies?.react ?? "";
if (/experimental|canary|-rc|-beta/.test(version)) {
  console.error(`Unstable React channel detected: react@${version}`);
  process.exit(1); // fail the CI build
}
console.log(`React channel OK: ${version}`);

Explanation: the script reads your package.json, pulls out the React version string, and checks it against a regular expression that matches pre-release tags. If it finds one, it exits with code 1, which fails the CI pipeline. This turns a team agreement ("stable only") into an automatic guardrail.

Part 5 — The deprecation pipeline in practice

APIs are not stable forever — sometimes an old design really does need to go. Stability is not “nothing ever changes”; it is “nothing changes by surprise.” React’s track record here is the best evidence of how the foundation is likely to behave, because the same maintainers now run the process.

The React 18 → 19 upgrade followed this pipeline closely. React 18.3 was released as a bridge: functionally the same as 18.2, but with new warnings for every API that React 19 would remove. Teams could upgrade to 18.3, fix the warnings, and only then move to 19. Let us look at three real removals.

Example 1: ReactDOM.render → createRoot

Before (React 17 style — removed in React 19):

import ReactDOM from "react-dom";
import App from "./App";

ReactDOM.render(, document.getElementById("root"));

After (React 18+ style):

import { createRoot } from "react-dom/client";
import App from "./App";

const root = createRoot(document.getElementById("root"));
root.render();

Explanation: the old render function did two jobs at once — it created React's internal tree and rendered into it. createRoot splits these: first you create a root object attached to a DOM node, then you call root.render() on it. This separation is what enables concurrent rendering features such as transitions and Suspense streaming. createRoot shipped in React 18 (2022), render was deprecated there, and it was only removed in React 19 (December 2024) — roughly two and a half years of overlap.

Example 2: String refs → useRef

Before (removed in React 19):

class SearchBox extends React.Component {
  componentDidMount() {
    this.refs.input.focus(); // string ref lookup
  }
  render() {
    return ;
  }
}

After:

import { useEffect, useRef } from "react";

function SearchBox() {
  const inputRef = useRef(null);
  useEffect(() => {
    inputRef.current?.focus();
  }, []);
  return ;
}

Explanation: a string ref (ref="input") asked React to find the element by name and attach it to this.refs — magic that was hard for React to optimize and impossible for tools to type-check. useRef(null) creates a plain object { current: null }. React fills current with the DOM node after mounting, and the Effect (which runs after mount) can safely call .focus(). The ?. guards against the node being missing.

Example 3: propTypes and defaultProps on function components

Before:

import PropTypes from "prop-types";

function Greeting({ name }) {
  return Hello, {name}
;
}
Greeting.propTypes = { name: PropTypes.string };
Greeting.defaultProps = { name: "friend" };

After:

type GreetingProps = { name?: string };

function Greeting({ name = "friend" }: GreetingProps) {
  return Hello, {name}
;
}

Explanation: propTypes checked prop types at runtime, in the browser. React 19 stopped running these checks entirely, in favor of compile-time checking with TypeScript (the GreetingProps type). defaultProps for function components was replaced by ordinary JavaScript default parameters (name = "friend"), which every developer already understands — less React-specific magic means less API surface to keep stable.

The automated step: codemods

A codemod is a script that rewrites your source code automatically. For React 19, the team published migration recipes you can run from the terminal:

# Run React 19 migration codemods across your project
npx codemod@latest react/19/migration-recipe

Explanation: the tool parses each file into an abstract syntax tree (a structured representation of your code), finds the old patterns — like ReactDOM.render or string refs — and rewrites them. You then review the diff in Git. Codemods turn a week of manual edits into a reviewable pull request.

This whole pipeline — warn, provide a codemod, document, then remove — is the strongest signal of what API stability looks like under the foundation. It is a process, and processes are written down, not tied to one company’s org chart.

Part 6 — What changes, what stays the same

The short answer: in the short term, almost nothing changes for your code; in the long term, the incentives behind every API decision change.

Why this is good for API stability

No single kill switch. A company can lose interest in a project, reorganize a team or change strategy. A foundation with many funding members is far less likely to abandon React or push a sudden rewrite.
More real-world testing. Under Meta, new APIs were battle-tested mostly on Meta’s apps. With Amazon, Microsoft, Expo, Vercel and others at the table, an API that breaks a common pattern is more likely to be caught before it reaches stable.
Neutral ground for frameworks. Next.js, Expo, React Router and others can invest in React features knowing no single competitor controls the core. That encourages them to build on stable APIs rather than forks.
Process over priorities. Foundations tend to document their RFC and breaking-change rules publicly. Written rules are easier for you to rely on and plan around.

The honest risks

Slower decisions. Consensus takes time. A promising API might spend longer in canary. For stability, this is mostly a feature — but if you are waiting on a specific feature, expect patience.
Governance is still provisional. As of this writing, the permanent technical structure has not been finalized. Until it is, the rules for breaking changes are still the inherited, informal ones. Watch the official React blog for the final model.
Influence moves, it does not vanish. Companies that employ many maintainers will still shape priorities. The two-layer model limits the board’s power, but individual employers still matter. Healthy governance usually adds limits on how many seats one employer can hold.
Meta’s testbed role shrinks. Meta ran React on some of the largest apps in the world. If Meta’s engineers contribute less over time, the project must replace that scale of real-world testing elsewhere.

A simple mental model

Think of React as a public road. Before, one company owned the road, paved it well and mostly kept the rules consistent — but could, in theory, change them overnight. Now the road is owned by a public trust funded by many companies, with a separate traffic authority deciding the rules. Day to day, you will not notice a difference. Over ten years, you will be glad the rules can’t be changed by a single owner.

Part 7 — Defensive coding: protect your app no matter who governs React

Governance makes stability likely. Good engineering makes it certain for your app. These four patterns work under any owner.

Pattern 1: An adapter layer for newer APIs

Instead of importing newer React APIs directly in hundreds of files, import them from one file you own. If an API ever changes, you fix one file, not hundreds.

// src/platform/react-compat.js — the ONLY file that touches newer APIs
import * as React from "react";

// Fallback for React versions before 19.2, which lack useEffectEvent.
function useEffectEventFallback(callback) {
  const latest = React.useRef(callback);
  // Keep the ref pointing at the newest callback after every render.
  React.useLayoutEffect(() => {
    latest.current = callback;
  });
  // Return a stable function that always calls the newest callback.
  return React.useCallback((...args) => latest.current(...args), []);
}
export const useEffectEvent = React.useEffectEvent ?? useEffectEventFallback;

// src/features/chat/ChatRoom.jsx — feature code imports from YOUR layer
import { useEffect } from "react";
import { useEffectEvent } from "../../platform/react-compat";

Explanation, step by step:

import * as React loads everything React exports into one object, so we can check whether React.useEffectEvent exists.
The fallback stores the callback in a ref. A ref is a box whose contents can change without causing a re-render.
useLayoutEffect with no dependency array runs after every render, so latest.current always holds the newest callback (with the newest props).
useCallback(..., []) returns the same function identity forever. That stable identity is what lets you leave it out of Effect dependency arrays.
The last line picks the real API if it exists (?? means "use the right side if the left is null or undefined"). The choice happens once, when the module loads, so the rules of hooks are respected.

The same idea works for any API you are nervous about: wrap it once, depend on your wrapper everywhere.

Pattern 2: Controlled, grouped upgrades

Upgrade react and react-dom together, and require a human to approve major versions. With Renovate, that is a few lines of config:

{
  "extends": ["config:recommended"],
  "packageRules": [
    {
      "matchPackageNames": ["react", "react-dom", "@types/react", "@types/react-dom"],
      "groupName": "react"
    },
    {
      "matchPackageNames": ["react", "react-dom"],
      "matchUpdateTypes": ["major"],
      "dependencyDashboardApproval": true
    }
  ]
}

Explanation: the first rule bundles React and its type definitions into a single pull request, so they never drift out of sync (mismatched react and react-dom versions are a classic source of strange bugs). The second rule says that a major upgrade does not open automatically — someone must approve it from the dependency dashboard, ideally after reading the upgrade guide. Patches and minors still flow in automatically, which keeps you current on security fixes.

Pattern 3: Treat development warnings as errors in tests

Remember Figure 4: deprecations arrive first as console warnings in development. If your test suite fails on those warnings, you learn about a deprecation the day it lands, not the day it is removed.

// vitest.setup.js (works the same way with Jest's setupFiles)
const originalError = console.error;
const originalWarn = console.warn;

function failOnReactWarning(original) {
  return (...args) => {
    const message = String(args[0] ?? "");
    if (/deprecated|will be removed|is no longer supported/i.test(message)) {
      throw new Error(`React deprecation warning: ${message}`);
    }
    original(...args);
  };
}
console.error = failOnReactWarning(originalError);
console.warn = failOnReactWarning(originalWarn);

Explanation: this setup file runs before every test. It replaces console.error and console.warn with wrappers that inspect each message. If the message looks like a deprecation notice, the wrapper throws, which fails the test that triggered it. Other messages pass through to the original function unchanged. Adjust the regular expression to match the exact wording you see in your project.

Pattern 4: A small “contract test” suite

Write a few tests that check the React behaviors your app relies on most. When you upgrade, these tests tell you immediately whether the contract still holds.

// Counter.contract.test.jsx
import { render, screen } from "@testing-library/react";
import userEvent from "@testing-library/user-event";
import { useState } from "react";

function Counter() {
  const [count, setCount] = useState(0);
  return ;
}
test("useState contract: setter triggers a re-render with the new value", async () => {
  render();
  const button = screen.getByRole("button");
  await userEvent.click(button);
  await userEvent.click(button);
  expect(button).toHaveTextContent("Count: 2");
});

Explanation: render mounts the component into a simulated DOM. screen.getByRole("button") finds the button the way a user or screen reader would. Two simulated clicks should call the setter twice, and the functional update (c) => c + 1 guarantees each click builds on the latest value. The assertion checks the visible result. If a future React version ever changed how state updates behave, this test would fail on the upgrade branch — long before your users noticed.

Part 8 — A checklist for your team

Confirm every app is on a stable React version (no experimental or canary tags in production).
Make sure react and react-dom are always the same version.
Pick a version-range policy: ^ for most apps, ~ or exact pins for regulated or high-risk apps.
Route newer or riskier React APIs through a single adapter file.
Fail tests on React deprecation warnings.
Add a handful of contract tests for the React behaviors you depend on most.
Require human approval for major React upgrades, and read the official upgrade guide first.
Follow the official React blog for the final technical governance model and any change to the deprecation policy.

Conclusion

React leaving Meta is not a rewrite, a fork or a reason to panic. On day one, nothing in your code changed. What changed is the structure behind every future decision: React is now owned by a neutral foundation, funded by eight major members, with technical decisions deliberately kept away from the board.

For API stability, that is good news. Stability has always rested on a process — release channels, semver, warnings, codemods and upgrade guides — and that process is now owned by a community instead of a single company. The remaining open question is the permanent shape of technical governance, so keep an eye on it.

In the meantime, the best protection is the same as always: stay on stable releases, upgrade deliberately, and let your tests tell you the moment a contract changes.

Sources

The React Foundation: A New Home for React Hosted by the Linux Foundation — React blog, Feb 24, 2026
Introducing the React Foundation — React blog, Oct 7, 2025
Linux Foundation Announces the Formation of the React Foundation — Linux Foundation press release
React Foundation Officially Launched — I Programmer, Mar 9, 2026
Meta will move React to Linux Foundation — The Register, Oct 9, 2025
React 19.2 release post — React blog
React releases and changelog — GitHub

Canary Tokens for the AI Era: Detecting Rogue Agents in Your Cloud

Vasanthan K — Wed, 21 Oct 2026 13:01:03 GMT

AI agents can compromise a cloud account in minutes. Here is how a decades-old deception trick — the canary token — becomes your fastest, most reliable alarm against them.

The minutes that matter

Autonomous AI agents are changing the speed of cloud attacks. In controlled tests run by the security firm Tracebit, frontier AI agents reached a critical action in about 14 minutes on average, leaving defenders a window of roughly 8 minutes. Most security teams cannot even read an alert in that time, let alone investigate it.

Traditional detection struggles here. Anomaly detection needs a baseline and produces noise. Signature rules miss new techniques. Log reviews happen hours later. What defenders need is an alarm that is instant, cheap, and almost never wrong.

That alarm already exists. It is called a canary token: a fake secret that no legitimate person or system should ever touch. The moment anything reads it, you know something is wrong. And as you will see, AI agents are unusually good at stepping on these tripwires.

TL;DR

What: a canary token is a decoy credential, file or resource. Any access to it is a high-confidence signal of intrusion.
Why now: AI attack agents enumerate everything, move fast, and do not get suspicious. That makes them trip canaries far more reliably than many human attackers.
Evidence: in published research, canaries alerted defenders in 95.9% of 162 successful AI-agent attack runs across 10 models.
What you will build: realistic canary secrets in AWS Secrets Manager (TypeScript), an EventBridge rule that watches CloudTrail, a responder Lambda that alerts Slack and contains the identity, and a local simulator to test it all without an AWS account.
Remember: canaries detect; they do not prevent. They belong on top of least privilege, not instead of it.

Who this is for

Beginners are welcome. Every concept is explained before the code uses it, and every code block is followed by a walkthrough of why it is written that way. If you already know canaries well, skip to the hands-on build.

Foundations: what is a canary token?

A canary token is a piece of bait that has no legitimate use. Because nobody should ever touch it, any touch is evidence of an intruder.

The name comes from coal mining. Miners carried canaries underground because the birds reacted to toxic gas before humans did. The bird was an early warning system. A canary token plays the same role in your infrastructure: it reacts first, so you can act before real damage is done.

A short history

The idea is older than the cloud. Security researchers in the early 2000s called these objects honeytokens, a cousin of the honeypot (a fake server built to attract attackers). The difference is size. A honeypot is a whole machine you must maintain. A honeytoken is a single fake item — a password, an API key, a database row, a document — that you sprinkle inside your real environment.

Over time the pattern spread everywhere:

Fake database rows that alert when someone exports the customer table.
Fake AWS access keys left in code repositories, which alert when an attacker tries to use them.
Fake documents that phone home when opened.
Fake secrets in a secrets manager, which alert when read. This is the type we build in this article.

Why canaries have almost no false positives

Most detection tools try to answer a hard question: is this behaviour unusual? That requires a baseline of “normal”, and normal changes every day. The result is alert fatigue — teams drown in warnings and start ignoring them.

A canary answers a much easier question: did anyone touch the thing nobody should touch? That is a yes/no fact, not a guess. If you design the canary well, the honest answer to “who should read this?” is “nobody”, so every alert deserves attention.

Think of it like a motion sensor in a locked room that no employee has a key to. You do not need machine learning to know that motion in that room is a problem.

The three properties of a good canary

Believable. It must look like something an attacker wants: a production database password, an admin API key, a payment-provider token. A canary named fake-secret-123 catches nobody.
Unused. No real application, script or person reads it. This is what keeps false positives near zero.
Observable. Every access is logged and routed to an alert within seconds. In AWS, CloudTrail records each GetSecretValue call, which makes Secrets Manager a natural home for canaries.

Key terms used in this article

Why AI agents trip canaries so reliably

AI attack agents are fast, but their habits make them easy to catch with decoys. Three traits matter most.

1. They enumerate everything

An agent works in a loop: the model looks at what it knows, picks a tool to call, reads the result, and repeats. When an agent lands in a cloud account with stolen credentials, the cheapest next step is almost always to list things — buckets, roles, functions and secrets — and then read the interesting ones.

An experienced human attacker might pause and ask, “Is prod/payments/stripe-live-key too convenient? Could it be a trap?" An agent chasing a goal like "find credentials that grant more access" rarely asks that question. A secret with a juicy name looks like progress, so it gets read.

2. They are fast, so they hit the trap early

Speed cuts both ways. The same speed that shrinks your response window also means the agent reaches your decoys within its first few minutes. A canary placed where enumeration naturally leads becomes one of the first things an agent touches — often long before it finds anything real.

3. They do not get tired or bored

Humans skip things. Agents are systematic. If ten secrets are listed, an agent may try to read all ten. Each extra read is another chance to hit a canary. Mixing several canaries among real resources raises the odds even more.

What the research shows

This is not just theory. Tracebit pointed frontier models at a realistic AWS environment and measured how often canaries fired. According to coverage in CSO Online, canaries alerted defenders in 95.9% of 162 successful attack runs performed by agents using 10 different AI models. In other words, when an agent succeeded at something, it almost always tripped a wire on the way.

That result is the core argument of this article: you do not need to out-think an AI attacker to detect it. You need to put bait where its loop will naturally look.

The timeline problem

Detection only matters if it happens before the damage. The diagram below compares a typical agent attack timeline with a manual response and an automated, canary-driven response. With a manual process, the alert may arrive in time but the human reaction does not. With automation, the canary read triggers containment in seconds — well inside the window.

The architecture: from bait to containment in seconds

The whole system is five AWS building blocks connected in a straight line, plus one optional branch for containment. Here is the flow before we write any code.

The canary secret lives in AWS Secrets Manager next to your real secrets. It has a believable name and a believable (but fake) value. Nothing in your system reads it.
CloudTrail records the read. Every call to GetSecretValue becomes an event that includes who called (the principal ARN), from where (source IP, user agent) and what they asked for (the secret ID).
EventBridge filters the stream. A rule matches only events where the requested secret is one of your canaries, so the downstream function runs only when it matters.
The responder Lambda enriches and acts. It builds a readable alert, posts it to Slack (or SNS, PagerDuty, email), and — if you enable it — contains the identity.
Containment attaches an explicit-deny policy to the IAM role or user that read the canary. In IAM, an explicit deny always beats an allow, so this instantly stops that identity from doing anything else.

Why this design?

Why Secrets Manager and not a fake access key? Reading a secret is exactly what an agent does when it looks for privilege escalation. The read itself is the signal; the attacker does not even have to use the fake value.

Why EventBridge instead of polling logs? Polling means delay and cost. EventBridge pushes matching events to Lambda as they arrive, typically within seconds to a few minutes of the API call, with no servers to run.

Why automate containment? Because of the timeline problem. If the agent’s whole attack fits inside 8 minutes, a human approval step may arrive too late. We will make containment a configurable switch so you can start in alert-only mode and turn it on once you trust it.

Why an explicit deny instead of deleting the role? Deleting things destroys evidence and can break production. A deny policy is reversible: remove it after the investigation and everything works again.

Project layout

All the code in the next sections fits into this small TypeScript project:

canary-agent-trap/
├── src/
│   ├── canaries.ts          # Step 1: create canary secrets
│   ├── responder.ts         # Step 3: Lambda that alerts + contains
│   └── simulate.ts          # Step 4: local rogue-agent simulator
├── infra/
│   └── canary-stack.ts      # Step 2: CDK stack (EventBridge rule + Lambda)
├── canaries.json            # the list of canary names (your source of truth)
├── package.json
└── tsconfig.json

Install the dependencies once:

npm init -y
npm install @aws-sdk/client-secrets-manager @aws-sdk/client-iam
npm install -D typescript tsx @types/node @types/aws-lambda aws-cdk-lib constructs

Step 1: Create realistic canary secrets

The first job is to plant bait that looks real. We keep the list of canaries in a JSON file so every other component — the detection rule, the responder, the simulator — reads from one source of truth.

canaries.json

{
  "region": "us-east-1",
  "canaries": [
    {
      "name": "prod/payments/stripe-live-secret",
      "description": "Live Stripe secret key for checkout-service",
      "kind": "api-key"
    },
    {
      "name": "prod/rds/admin-credentials",
      "description": "Master credentials for orders-db cluster",
      "kind": "db-credentials"
    },
    {
      "name": "ci/github/deploy-token",
      "description": "Deploy token used by release pipeline",
      "kind": "token"
    }
  ]
}

Theory — why these names? Attackers (human or AI) look for three things: production, privilege and money. Names containing prod, admin, live and payments signal all three. Copy the naming conventions your real secrets already use. If your real secrets look like orders-svc/prod/db, a canary called prod/rds/admin-credentials will stand out as fake to a careful reader.

src/canaries.ts

import { readFileSync } from "node:fs";
import { randomBytes } from "node:crypto";
import {
  SecretsManagerClient,
  CreateSecretCommand,
  ResourceExistsException,
} from "@aws-sdk/client-secrets-manager";

type CanaryKind = "api-key" | "db-credentials" | "token";
interface CanaryDef {
  name: string;
  description: string;
  kind: CanaryKind;
}
const config = JSON.parse(readFileSync("canaries.json", "utf8")) as {
  region: string;
  canaries: CanaryDef[];
};
const client = new SecretsManagerClient({ region: config.region });
// Random characters so every value looks unique and plausible.
function randomString(length: number): string {
  const alphabet = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789";
  const bytes = randomBytes(length);
  return Array.from(bytes, (b) => alphabet[b % alphabet.length]).join("");
}
// Build a fake value that matches the shape of a real one.
// None of these values work anywhere - they are pure decoys.
function fakeValue(kind: CanaryKind): string {
  switch (kind) {
    case "api-key":
      return JSON.stringify({ apiKey: `sk_live_${randomString(32)}` });
    case "db-credentials":
      return JSON.stringify({
        engine: "postgres",
        host: "orders-db.cluster-internal.example",
        port: 5432,
        username: "db_admin",
        password: randomString(24),
      });
    case "token":
      return JSON.stringify({ token: `ghp_${randomString(36)}` });
  }
}
async function createCanary(def: CanaryDef): Promise {
  try {
    await client.send(
      new CreateSecretCommand({
        Name: def.name,
        Description: def.description,
        SecretString: fakeValue(def.kind),
        // Tags look like a real team's tags. Never tag it "canary":
        // anyone who can list secrets can also read tags.
        Tags: [
          { Key: "team", Value: "platform" },
          { Key: "env", Value: "prod" },
        ],
      }),
    );
    console.log(`created  ${def.name}`);
  } catch (err) {
    if (err instanceof ResourceExistsException) {
      console.log(`exists   ${def.name} (skipped)`);
      return;
    }
    throw err;
  }
}
async function main(): Promise {
  for (const def of config.canaries) {
    await createCanary(def);
  }
}
main().catch((err) => {
  console.error(err);
  process.exit(1);
});

Run it with npx tsx src/canaries.ts.

Walkthrough: what each part does and why

Reading canaries.json. We load the canary list from a file instead of hard-coding it. Later, the CDK stack reads the same file to build the detection rule. One list means you can never add a canary and forget to watch it.

randomString uses crypto.randomBytes. Values must be unique per canary and unpredictable. If every canary shared the same value, an attacker who saw two of them would know they are fake. Math.random() would also work for decoys, but using the crypto module is a good habit for anything credential-shaped.

fakeValue mimics real formats. A Stripe live key starts with sk_live_; a GitHub personal token starts with ghp_; database secrets are JSON with host, port, username and password. Matching these shapes matters because an agent may parse the value and plan its next step from it. A realistic value keeps the decoy believable. The host uses the reserved .example domain so it can never resolve to a real machine.

Tags are camouflage. It is tempting to tag canaries with purpose=canary so your team can find them. Do not do this. ListSecrets returns tags, so the label would warn the attacker. Keep the canary list in canaries.json (in a private repo) instead.

Idempotency with ResourceExistsException. Running the script twice should not crash. Catching "already exists" makes the script safe to re-run in CI.

Why reading is enough. Notice we never need the attacker to use these values. The GetSecretValue call is logged by CloudTrail the moment it happens, which is what Step 2 watches for.

Step 2: Detect the read with EventBridge and CDK

Now we need to see every read of a canary. We use AWS CDK (infrastructure as code in TypeScript) to create an EventBridge rule and wire it to the responder Lambda from Step 3.

A gotcha beginners hit: read-only events

GetSecretValue is a read-only API call. By default, EventBridge rules ignore read-only management events from CloudTrail, because they are extremely high-volume. To receive them, the rule must use a special state, ENABLED_WITH_ALL_CLOUDTRAIL_MANAGEMENT_EVENTS, which today can only be set through the CLI or CloudFormation. AWS added this capability in late 2023.

If you skip this, your rule deploys fine, looks correct in the console — and never fires. Also confirm that CloudTrail is logging management events (read and write) in the account and region where your canaries live.

infra/canary-stack.ts

import { readFileSync } from "node:fs";
import * as cdk from "aws-cdk-lib";
import { Construct } from "constructs";
import * as events from "aws-cdk-lib/aws-events";
import * as targets from "aws-cdk-lib/aws-events-targets";
import * as lambda from "aws-cdk-lib/aws-lambda";
import * as nodejs from "aws-cdk-lib/aws-lambda-nodejs";
import * as iam from "aws-cdk-lib/aws-iam";

const { canaries } = JSON.parse(readFileSync("canaries.json", "utf8")) as {
  canaries: { name: string }[];
};
export class CanaryStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);
    // 1. The responder Lambda (code in src/responder.ts)
    const responder = new nodejs.NodejsFunction(this, "CanaryResponder", {
      entry: "src/responder.ts",
      runtime: lambda.Runtime.NODEJS_20_X,
      timeout: cdk.Duration.seconds(30),
      environment: {
        SLACK_WEBHOOK_URL: process.env.SLACK_WEBHOOK_URL ?? "",
        CONTAIN: process.env.CONTAIN ?? "false", // start in alert-only mode
        ALLOWLIST_PRINCIPALS: process.env.ALLOWLIST_PRINCIPALS ?? "",
      },
    });
    // The responder may attach a deny policy to roles and users - nothing more.
    responder.addToRolePolicy(
      new iam.PolicyStatement({
        actions: ["iam:PutRolePolicy", "iam:PutUserPolicy"],
        resources: ["*"],
      }),
    );
    // 2. Match a canary whether it was requested by name or by full ARN.
    const secretIdMatchers = canaries.flatMap((c) => [
      c.name,
      { wildcard: `arn:aws:secretsmanager:*:*:secret:${c.name}-*` },
    ]);
    const rule = new events.Rule(this, "CanaryReadRule", {
      description: "Fires when any canary secret is read",
      eventPattern: {
        source: ["aws.secretsmanager"],
        detailType: ["AWS API Call via CloudTrail"],
        detail: {
          eventSource: ["secretsmanager.amazonaws.com"],
          eventName: ["GetSecretValue"],
          requestParameters: { secretId: secretIdMatchers },
        },
      },
      targets: [new targets.LambdaFunction(responder, { retryAttempts: 2 })],
    });
    // 3. Escape hatch: opt in to read-only management events.
    (rule.node.defaultChild as events.CfnRule).state =
      "ENABLED_WITH_ALL_CLOUDTRAIL_MANAGEMENT_EVENTS";
  }
}
const app = new cdk.App();
new CanaryStack(app, "CanaryAgentTrap");

Deploy with npx cdk deploy --app "npx tsx infra/canary-stack.ts".

Walkthrough: how the rule thinks

An event pattern is a filter, not a query. EventBridge compares every incoming event to the pattern. Each key must match; inside an array, any value can match. So the pattern reads: “source is Secrets Manager AND it is a CloudTrail API call AND the action is GetSecretValue AND the secret ID is one of my canaries." Everything else is dropped for free, before your Lambda runs.

Why two matchers per canary? A caller can ask for a secret by its short name (prod/rds/admin-credentials) or by its full ARN. Secrets Manager adds a random six-character suffix to ARNs (for example ...:secret:prod/rds/admin-credentials-Ab12Cd). The wildcard matcher catches every ARN form, whatever the suffix.

Why read the same canaries.json? Single source of truth. Add a canary in Step 1, redeploy, and the rule picks it up automatically.

Why retryAttempts: 2? If the Lambda fails (Slack is down, a throttle), EventBridge retries. An alarm that silently fails once is worse than useless.

Why the escape hatch? CDK’s high-level Rule construct exposes only enabled or disabled. We reach into the underlying CloudFormation resource (CfnRule) and set the special state ourselves. This is a common CDK pattern whenever a feature is newer than the library's abstractions.

Least privilege for the defender too. The responder can only write inline policies to users and roles. It cannot delete anything, create users, or read your secrets. If the responder itself were ever compromised, the blast radius is small.

Step 3: The responder — alert, then contain

When the rule fires, EventBridge hands our Lambda the full CloudTrail record. The responder does four things in order: parse who did it, check the allowlist, alert humans, and optionally contain the identity.

src/responder.ts

import { IAMClient, PutRolePolicyCommand, PutUserPolicyCommand } from "@aws-sdk/client-iam";
import type { EventBridgeEvent } from "aws-lambda";

const iam = new IAMClient({});
// ---- Types: only the CloudTrail fields we use ----
export interface CloudTrailDetail {
  eventTime: string;
  eventName: string;
  sourceIPAddress?: string;
  userAgent?: string;
  requestParameters?: { secretId?: string };
  userIdentity: {
    type: string; // "IAMUser" | "AssumedRole" | "Root" | ...
    arn?: string;
    userName?: string;
    sessionContext?: { sessionIssuer?: { type?: string; userName?: string; arn?: string } };
  };
}
export interface Finding {
  secretId: string;
  principalArn: string;
  identityType: string;
  roleName?: string;
  userName?: string;
  sourceIp: string;
  userAgent: string;
  time: string;
  looksLikeAiAgent: boolean;
}
// ---- 1. Parse ----
export function toFinding(d: CloudTrailDetail): Finding {
  const issuer = d.userIdentity.sessionContext?.sessionIssuer;
  const ua = d.userAgent ?? "unknown";
  return {
    secretId: d.requestParameters?.secretId ?? "unknown",
    principalArn: issuer?.arn ?? d.userIdentity.arn ?? "unknown",
    identityType: d.userIdentity.type,
    roleName: issuer?.type === "Role" ? issuer.userName : undefined,
    userName: d.userIdentity.type === "IAMUser" ? d.userIdentity.userName : undefined,
    sourceIp: d.sourceIPAddress ?? "unknown",
    userAgent: ua,
    time: d.eventTime,
    // A weak hint only: scripted SDK/CLI calls, not the console.
    looksLikeAiAgent: /boto3|aws-cli|aws-sdk|python-requests|langchain|mcp/i.test(ua),
  };
}
// ---- 2. Allowlist ----
export function isAllowlisted(f: Finding, allowlist: string[]): boolean {
  return allowlist.some((a) => a && f.principalArn.startsWith(a));
}
// ---- 3. Alert ----
export function formatAlert(f: Finding, contained: string): string {
  return [
    `:rotating_light: *Canary secret read* - likely intrusion`,
    `*Secret:* ${f.secretId}`,
    `*Principal:* ${f.principalArn} (${f.identityType})`,
    `*Source IP:* ${f.sourceIp}`,
    `*User agent:* ${f.userAgent}${f.looksLikeAiAgent ? "  _(scripted / possible agent)_" : ""}`,
    `*Time:* ${f.time}`,
    `*Containment:* ${contained}`,
  ].join("\n");
}
async function sendSlack(text: string): Promise {
  const url = process.env.SLACK_WEBHOOK_URL;
  if (!url) {
    console.warn("SLACK_WEBHOOK_URL not set; alert only logged");
    return;
  }
  const res = await fetch(url, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ text }),
  });
  if (!res.ok) throw new Error(`Slack returned ${res.status}`);
}
// ---- 4. Contain ----
export function denyPolicy(now: Date, revokeSessionsOnly: boolean): string {
  const statement: Record = {
    Sid: "CanaryContainment",
    Effect: "Deny",
    Action: "*",
    Resource: "*",
  };
  if (revokeSessionsOnly) {
    // Deny only credentials issued before "now" - the same idea as
    // the console's "Revoke active sessions" button.
    statement.Condition = { DateLessThan: { "aws:TokenIssueTime": now.toISOString() } };
  }
  return JSON.stringify({ Version: "2012-10-17", Statement: [statement] });
}
async function contain(f: Finding): Promise {
  const now = new Date();
  if (f.roleName) {
    await iam.send(new PutRolePolicyCommand({
      RoleName: f.roleName,
      PolicyName: "canary-containment",
      PolicyDocument: denyPolicy(now, true),
    }));
    return `revoked active sessions of role ${f.roleName}`;
  }
  if (f.userName) {
    await iam.send(new PutUserPolicyCommand({
      UserName: f.userName,
      PolicyName: "canary-containment",
      PolicyDocument: denyPolicy(now, false),
    }));
    return `denied all actions for user ${f.userName}`;
  }
  return `not supported for identity type ${f.identityType} - respond manually`;
}
// ---- Handler ----
export async function handler(
  event: EventBridgeEvent<"AWS API Call via CloudTrail", CloudTrailDetail>,
): Promise {
  const finding = toFinding(event.detail);
  const allowlist = (process.env.ALLOWLIST_PRINCIPALS ?? "").split(",");
  // Structured log line: easy to search in CloudWatch Logs Insights.
  console.log(JSON.stringify({ msg: "canary_read", ...finding }));
  if (isAllowlisted(finding, allowlist)) {
    console.log(JSON.stringify({ msg: "allowlisted_principal", principal: finding.principalArn }));
    return;
  }
  let contained = "disabled (alert-only mode)";
  if (process.env.CONTAIN === "true") {
    try {
      contained = await contain(finding);
    } catch (err) {
      contained = `FAILED: ${(err as Error).message}`;
    }
  }
  await sendSlack(formatAlert(finding, contained));
}

Walkthrough: the decisions behind the code

Pure functions first, side effects last. toFinding, isAllowlisted, formatAlert and denyPolicy take data and return data. They never call AWS or Slack. That separation is what lets Step 4 test the whole logic on your laptop. Only contain, sendSlack and handler touch the outside world.

Finding the real identity. When an agent uses a role, CloudTrail’s userIdentity.arn is a session ARN such as arn:aws:sts::123456789012:assumed-role/ci-runner/session-xyz. You cannot attach a policy to a session. The sessionIssuer block tells us the underlying role (ci-runner), which is what we contain. For IAM users, the user name is right in userIdentity.

Two containment strategies. For an IAM user (long-lived access keys, a classic leak), we deny everything: those keys should never work again. For a role, a blanket deny would also break every legitimate service using that role — possibly your production app. Instead we use the condition aws:TokenIssueTime with DateLessThan: it denies only credentials issued before the incident. The attacker's stolen session dies; fresh sessions for your real workloads keep working. AWS uses the same technique behind its "Revoke active sessions" console button.

Why Action: "*" with Effect: "Deny"? IAM evaluates explicit denies before any allow. No matter what permissions the attacker had, this single statement overrides them all.

Alert even when containment fails. Notice the try/catch around contain. If containment fails (for example, the role is protected by a permissions boundary), we still send the alert with FAILED in it. A human must know.

The “looks like an AI agent” hint is deliberately weak. User-agent strings are easy to fake. We use it only to add context to the alert, never to decide anything. The canary read itself is the real signal — whoever made it, human or agent, should not have.

Alert-only by default. CONTAIN=false is the default. Run in alert-only mode for a couple of weeks, review every alert, tune the allowlist, then turn containment on. Automation you do not trust gets switched off; automation you have tested gets left on.

Step 4: Test it locally with a simulated rogue agent

You should never wait for a real attacker to find out whether your alarm works. This step builds a tiny simulator that runs on your laptop, with no AWS account and no LLM. It mixes real-looking secrets with your canaries, lets a scripted “agent” choose what to read, and runs the responder’s pure functions on the result.

src/simulate.ts

import { readFileSync } from "node:fs";
import { toFinding, formatAlert, denyPolicy, type CloudTrailDetail } from "./responder";

// ---- 1. A fake account: real secrets mixed with canaries ----
const { canaries } = JSON.parse(readFileSync("canaries.json", "utf8")) as {
  canaries: { name: string }[];
};
const canaryNames = new Set(canaries.map((c) => c.name));
const realSecrets = [
  "orders-svc/prod/redis-url",
  "orders-svc/prod/feature-flags",
  "reporting/staging/smtp-password",
];
const allSecrets = [...realSecrets, ...canaryNames].sort();
// ---- 2. A scripted "rogue agent" ----
// Not an LLM: a stand-in that behaves the way agents typically do -
// list everything, rank by how valuable a name looks, read the top hits.
const JUICY = /prod|admin|live|payments|deploy|root|master/i;
function agentPicks(secretNames: string[], maxReads: number): string[] {
  return secretNames
    .map((name) => ({ name, score: (name.match(new RegExp(JUICY, "gi")) ?? []).length }))
    .filter((s) => s.score > 0)
    .sort((a, b) => b.score - a.score)
    .slice(0, maxReads)
    .map((s) => s.name);
}
// ---- 3. Turn each read into a CloudTrail-shaped record ----
function cloudTrailRecord(secretId: string, at: Date): CloudTrailDetail {
  return {
    eventTime: at.toISOString(),
    eventName: "GetSecretValue",
    sourceIPAddress: "203.0.113.42", // documentation-only IP range
    userAgent: "Boto3/1.35.0 md/agent-framework",
    requestParameters: { secretId },
    userIdentity: {
      type: "AssumedRole",
      arn: "arn:aws:sts::111122223333:assumed-role/ci-runner/stolen-session",
      sessionContext: {
        sessionIssuer: {
          type: "Role",
          userName: "ci-runner",
          arn: "arn:aws:iam::111122223333:role/ci-runner",
        },
      },
    },
  };
}
// ---- 4. Run the simulation ----
const start = new Date("2026-10-17T14:00:00Z");
const reads = agentPicks(allSecrets, 4);
console.log(`Agent listed ${allSecrets.length} secrets and chose to read ${reads.length}:\n`);
let firstAlertAt: Date | undefined;
reads.forEach((secretId, i) => {
  const at = new Date(start.getTime() + (i + 1) * 20_000); // one read every 20 s
  const isCanary = canaryNames.has(secretId);
  console.log(`${at.toISOString().slice(11, 19)}  read ${secretId.padEnd(36)} ${isCanary ? "<-- CANARY" : ""}`);
  if (isCanary && !firstAlertAt) {
    firstAlertAt = at;
    const finding = toFinding(cloudTrailRecord(secretId, at));
    console.log("\n--- alert that would be sent ---");
    console.log(formatAlert(finding, `would revoke active sessions of role ${finding.roleName}`));
    console.log("\n--- containment policy that would be attached ---");
    console.log(JSON.stringify(JSON.parse(denyPolicy(at, true)), null, 2));
    console.log("");
  }
});
if (firstAlertAt) {
  const seconds = (firstAlertAt.getTime() - start.getTime()) / 1000;
  console.log(`Detected ${seconds}s after the agent started reading secrets.`);
} else {
  console.log("No canary was read. Rethink your canary names and placement!");
}

Run it with npx tsx src/simulate.ts. This is the actual output:

Agent listed 6 secrets and chose to read 4:

14:00:20  read prod/payments/stripe-live-secret     <-- CANARY
--- alert that would be sent ---
:rotating_light: *Canary secret read* - likely intrusion
*Secret:* prod/payments/stripe-live-secret
*Principal:* arn:aws:iam::111122223333:role/ci-runner (AssumedRole)
*Source IP:* 203.0.113.42
*User agent:* Boto3/1.35.0 md/agent-framework  _(scripted / possible agent)_
*Time:* 2026-10-17T14:00:20.000Z
*Containment:* would revoke active sessions of role ci-runner
--- containment policy that would be attached ---
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "CanaryContainment",
      "Effect": "Deny",
      "Action": "*",
      "Resource": "*",
      "Condition": {
        "DateLessThan": {
          "aws:TokenIssueTime": "2026-10-17T14:00:20.000Z"
        }
      }
    }
  ]
}
14:00:40  read prod/rds/admin-credentials           <-- CANARY
14:01:00  read ci/github/deploy-token               <-- CANARY
14:01:20  read orders-svc/prod/feature-flags
Detected 20s after the agent started reading secrets.

Walkthrough: what the simulation teaches

The scoring function is the whole lesson. agentPicks counts how many "valuable" words appear in each secret name and reads the highest scorers first. Real agents are far more sophisticated, but the instinct is the same: chase what looks like production, privilege and money. Our canaries scored highest, so they were read before the real secrets. That is exactly the outcome you design for.

Read the output as a test result. If your simulator ever prints “No canary was read”, your canaries are not attractive enough compared with your real secrets. Rename them until they win. This turns canary design from guesswork into something you can measure.

Reserved values keep the demo safe. The IP 203.0.113.42 comes from a range reserved for documentation, and account 111122223333 is the placeholder AWS uses in its docs. Nothing in this simulation can reach a real system.

Same code as production. The simulator imports toFinding, formatAlert and denyPolicy from the real responder. If the alert looks right here, it will look right in Slack.

Unit tests for the responder

The pure functions also make classic unit tests easy. Node’s built-in test runner is enough — no Jest required.

// src/responder.test.ts
import { test } from "node:test";
import assert from "node:assert/strict";
import { toFinding, isAllowlisted, denyPolicy, type CloudTrailDetail } from "./responder";

const roleEvent: CloudTrailDetail = {
  eventTime: "2026-10-17T14:00:20Z",
  eventName: "GetSecretValue",
  userAgent: "Boto3/1.35.0",
  requestParameters: { secretId: "prod/rds/admin-credentials" },
  userIdentity: {
    type: "AssumedRole",
    arn: "arn:aws:sts::111122223333:assumed-role/ci-runner/session-1",
    sessionContext: {
      sessionIssuer: { type: "Role", userName: "ci-runner", arn: "arn:aws:iam::111122223333:role/ci-runner" },
    },
  },
};
test("resolves an assumed-role session to its underlying role", () => {
  const f = toFinding(roleEvent);
  assert.equal(f.roleName, "ci-runner");
  assert.equal(f.principalArn, "arn:aws:iam::111122223333:role/ci-runner");
});
test("resolves an IAM user", () => {
  const f = toFinding({
    ...roleEvent,
    userIdentity: { type: "IAMUser", userName: "legacy-bot", arn: "arn:aws:iam::111122223333:user/legacy-bot" },
  });
  assert.equal(f.userName, "legacy-bot");
  assert.equal(f.roleName, undefined);
});
test("allowlist matches by ARN prefix and ignores empty entries", () => {
  const f = toFinding(roleEvent);
  assert.equal(isAllowlisted(f, ["arn:aws:iam::111122223333:role/ci-"]), true);
  assert.equal(isAllowlisted(f, [""]), false);
});
test("role containment only denies tokens issued before the incident", () => {
  const now = new Date("2026-10-17T14:00:20Z");
  const policy = JSON.parse(denyPolicy(now, true));
  assert.equal(policy.Statement[0].Effect, "Deny");
  assert.equal(policy.Statement[0].Condition.DateLessThan["aws:TokenIssueTime"], now.toISOString());
});

Run with npx tsx --test src/responder.test.ts; all four tests pass.

The empty-string allowlist test guards a subtle bug. If ALLOWLIST_PRINCIPALS is unset, splitting "" on commas yields [""], and every ARN starts with an empty string. Without the a && check in isAllowlisted, an unconfigured allowlist would silently allowlist everyone and your alarm would never ring.

Placement strategy and avoiding false positives

A canary only works if an attacker walks past it and your own systems never do. Placement is about both halves of that sentence.

Where to put canaries

Think about the path an agent takes after it gets a foothold, and put bait at each step:

Secrets Manager and Parameter Store. The first place anyone looks for credentials. Plant two or three canaries per account and region.
CI/CD systems. A fake deploy token in your pipeline’s variable store catches attackers who compromised a build runner.
Code repositories. A fake access key in an old config file catches attackers scanning source for leaked credentials. (Use a key that is not attached to any real permissions.)
Developer workstations and shared drives. A credentials.bak file or .env.production copy catches agents that start on a laptop and pivot to the cloud.
Every account, not just production. Attackers often enter through a forgotten sandbox or staging account. A canary there is the cheapest early warning you will ever buy.

Mix canaries with real resources. If all your canaries sit in one obvious corner, a careful attacker can learn to avoid that corner. Spread out, they make every read a gamble.

Where false positives come from (and how to stop them)

Canaries are near-zero noise in theory. In practice, a few legitimate systems read secrets in bulk. Plan for them before you turn on containment:

Protect your own agents from your own traps

This is the modern twist. Many teams now run internal AI agents with cloud access. Those agents enumerate just like attackers do — so they will trip your canaries too.

The fix is not to allowlist your agents (an attacker who hijacks your agent would then be invisible). The fix is least privilege: scope your agent’s IAM policy to the specific secrets it actually needs.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "OnlyTheSecretsThisAgentNeeds",
      "Effect": "Allow",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "arn:aws:secretsmanager:us-east-1:111122223333:secret:orders-svc/prod/redis-url-*"
    }
  ]
}

Now your agent physically cannot read a canary. If an alert ever fires with your agent’s role as the principal, the call was denied by IAM but still logged — which means something tried to push your agent outside its scope. That alert is worth investigating too.

Note on denied calls: CloudTrail logs GetSecretValue attempts even when IAM denies them (with an errorCode of AccessDenied). Our rule fires on attempts as well as successes. That is a feature: an attacker who tries to read a canary is just as suspicious as one who succeeds.

Limitations: what canaries cannot do

Canaries are one of the best signal-to-noise tools in security, but be honest about their limits before you rely on them.

They detect; they do not prevent. Even with automated containment, an attacker may have done some damage before touching a canary. The agent in our simulation read a canary first only because the canary was more attractive than the real secrets.
A careful attacker can avoid them. A human or an agent instructed to read only secrets referenced by running workloads might skip unused secrets. Canaries raise the cost of attacking; they do not make it impossible.
Delivery is fast, not instant. CloudTrail-to-EventBridge delivery usually takes seconds to a few minutes. That is far better than a daily log review, but it is not zero.
Containment has edges. Root user activity cannot be blocked by an IAM policy. Some identities (federated users, cross-account sessions) need different handling. The responder reports these as “respond manually” so a human picks them up.
Canaries need care. When your naming conventions change, update the canaries. A canary that looks out of date is a canary that gets ignored.
Research results are not guarantees. The detection rates quoted earlier come from a vendor’s controlled cyber range. Your environment, attackers and agents will differ. Measure your own coverage with tools like the simulator from Step 4.

Where canaries fit: defense in depth

No single control stops a determined attacker. Security teams stack layers so that each one catches what the others miss. Canaries sit in the detect and respond layer, on top of preventive controls.

Prevent — least privilege. Every identity, human or AI, gets only the permissions it needs. This shrinks what an attacker can reach in the first place.
Prevent — short-lived credentials. Prefer IAM roles and temporary sessions over long-lived access keys. Stolen credentials expire on their own.
Limit — isolation. Separate accounts for production, staging and sandboxes, plus network boundaries, so one compromise does not become every compromise.
Detect — canaries. High-confidence tripwires that catch anyone exploring where they should not be.
Detect — broad monitoring. GuardDuty, anomaly detection and log analytics catch attacks that never touch a canary.
Respond — automated containment plus a human runbook. The Lambda buys time; your incident process finishes the job: rotate credentials, review what else the identity touched, and remove the deny policy only when the investigation is complete.

The most important takeaway: a canary alert is the start of an incident, not the end. Containment stops the bleeding. Investigation tells you how the attacker got in, so you can close that door for good.

Conclusion

AI agents have made cloud attacks faster, but they have not made them smarter in every way. Their strength — relentless, systematic enumeration — is also their weakness. They read the things that look valuable, and a well-designed canary is built to look valuable.

With about 350 lines of TypeScript, you now have:

believable canary secrets that share one source of truth,
an EventBridge rule that catches every read, by name or ARN, including the read-only events most tutorials miss,
a responder that alerts within seconds and can revoke an attacker’s session without breaking your real workloads,
and a simulator plus unit tests, so you can prove it works before an attacker does.

Start in alert-only mode this week. Plant three canaries in your most important account, run the simulator, and watch your first alert arrive. Then expand to every account, tune the allowlist, and switch containment on.

Key takeaways

A canary token is bait with no legitimate use, so any touch is a high-confidence alarm.
AI attack agents enumerate and read juicy-looking secrets, which makes them trip canaries reliably.
Read-only CloudTrail events need the ENABLED_WITH_ALL_CLOUDTRAIL_MANAGEMENT_EVENTS rule state, or your rule never fires.
Contain roles by revoking old sessions with aws:TokenIssueTime, not with a blanket deny.
Scope your own AI agents with least privilege so they never hit your traps.
Canaries are a detection layer; least privilege and incident response still do the heavy lifting.

Sources

Tracebit — Context bombs: Canaries that stop AI attackers (attack timing in AWS cyber range)
CSO Online — Context bombing heralds a new AI era of deceptive defense (canary detection rate across 162 runs)
AWS — Receiving read-only management events from AWS services
AWS — EventBridge announces support for read-only API events from CloudTrail

Docker Sandboxes: Running Claude Code and Other Coding Agents in Isolated MicroVMs

Vasanthan K — Mon, 19 Oct 2026 13:01:02 GMT

A beginner-friendly, hands-on guide to giving AI coding agents full autonomy without handing them the keys to your laptop.

Introduction: the “YOLO mode” dilemma

Docker Sandboxes let you run Claude Code, Codex, Copilot CLI, Gemini CLI, Kiro and other coding agents with full autonomy inside a disposable microVM, so a mistake or an attack stays inside the box instead of reaching your machine.

If you have used Claude Code for more than a day, you know the feeling. Every few seconds it stops and asks: “Can I run npm install?", "Can I edit this file?", "Can I run the tests?". You click Yes forty times. Eventually you discover the --dangerously-skip-permissions flag (fondly called "YOLO mode") and the agent suddenly flies.

Then a small voice asks: what exactly did I just allow? The agent can now run any shell command, as you, on your laptop, with your SSH keys, cloud credentials and browser cookies sitting a few folders away.

This article is about resolving that tension properly. By the end you will understand:

Why autonomous coding agents need isolation, and why a normal container is not quite enough.
What containers, virtual machines and microVMs are, explained from first principles.
How Docker Sandboxes are built: the microVM, the private Docker engine, the network proxy and the credential injector.
How to use them: installing the sbx CLI, running Claude Code, locking down the network, managing secrets, working with Git, publishing ports and running agents in CI.
What to watch out for: the gotchas that still matter even inside a sandbox.

Every command is followed by a plain-English explanation of what it does and why, so you can follow along even if you are new to Docker. All diagrams in this article were created specifically for it.

Note on versions: Docker Sandboxes moves fast. This guide uses the standalone sbx CLI as documented in October 2026. Older tutorials use docker sandbox ... commands from the Docker Desktop integration, which Docker has since deprecated. If a command here differs from an older blog post, the sbx form is the current one.

1. Why coding agents need a sandbox

A coding agent is not a chatbot that suggests code; it is a program that executes code, and anything it executes runs with your permissions.

When you ask Claude Code to “fix the failing tests”, it typically reads files, edits them, installs packages, runs the test suite, maybe starts a database in Docker, and repeats until the tests pass. Each of those steps is a real process on a real operating system. On your laptop, that operating system is the same one that holds everything you care about.

What can go wrong

The risks fall into four buckets. None of them require the model to be “evil”; most come from honest mistakes or from untrusted text the agent reads.

Why permission prompts don’t scale

The default answer to these risks is to make the agent ask before acting. That works for a five-minute session, but it fails in three ways:

Approval fatigue. After the twentieth prompt, people stop reading and click “Yes”. A safety control that is always approved is not a control.
No unattended work. The real productivity jump comes from letting an agent work for an hour on its own, or running several agents in parallel. Prompts make that impossible.
Wrong granularity. Approving npm test tells you nothing about what the test scripts actually do.

The better approach is the one operating systems have used for decades: don’t ask the program to behave, put it somewhere it cannot do damage. Give the agent a full computer of its own, let it do whatever it wants in there, and control only the narrow doorways between that computer and yours: which files it can see, which websites it can reach, and which credentials it can use.

That “computer of its own” is exactly what Docker Sandboxes provide. To understand why they use microVMs rather than ordinary containers, we need a short detour into how isolation works.

2. Isolation 101: processes, containers, VMs and microVMs

The strength of a sandbox depends on one question: if the code inside misbehaves, what is the next layer it must break through to reach your machine?

Level 0: a plain process

When you run claude in your terminal, it is just a process: a running program managed by your operating system's kernel (the core part of the OS that controls memory, files, network and hardware). A process inherits your user account. Whatever you can read, delete or upload, it can too. There is no wall at all.

Level 1: a container

A container (what docker run gives you) is still a normal process, but the kernel shows it a filtered view of the world using two Linux features:

Namespaces give the process its own private view of the filesystem, process list, network interfaces and hostname. It “thinks” it is alone on the machine.
cgroups (control groups) limit how much CPU, memory and I/O it may use.

This is excellent for packaging and running trusted applications. But every container on the machine still talks to the same host kernel. The Linux kernel exposes hundreds of system calls, and a bug in any of them can, in the worst case, let code escape from a container to the host. Containers are a fence, not a wall.

There is also a very practical problem for coding agents: they often need to run Docker themselves (docker build, docker compose up). The two usual workarounds are both poor:

Level 2: a virtual machine

A virtual machine (VM) goes further. A piece of software called a hypervisor uses hardware virtualization features built into modern CPUs (Intel VT-x, AMD-V, Apple’s Virtualization framework) to create a fake computer: virtual CPU, virtual memory, virtual disk. That fake computer boots its own kernel. Code inside the VM talks to its kernel, not yours. To reach your host, an attacker must find a bug in the hypervisor itself, which is a much smaller and much more heavily scrutinized surface.

The downside of a classic VM is weight: it emulates lots of hardware, takes tens of seconds to boot and uses gigabytes of memory.

Level 3: a microVM

A microVM keeps the strong part of a VM (separate kernel, hypervisor boundary) and throws away the heavy part. It emulates only the handful of devices a server workload needs, boots a minimal kernel, and starts in a fraction of the time. The same idea powers serverless platforms that run untrusted customer code side by side.

For coding agents, a microVM is the sweet spot:

Strong isolation: separate kernel, hardware-enforced boundary.
Feels like a real machine: the agent can apt install, run services, and even run its own Docker engine, because inside the VM it really is a full Linux system.
Disposable: delete it and everything the agent did inside is gone.

The trade-off is a bit more memory and disk than a container, plus a short boot. For an agent that will work for minutes or hours, that is a very good deal.

Key idea to remember: a container hides things from a process; a microVM gives the process a different computer.

3. What Docker Sandboxes actually is

Docker Sandboxes is a tool from Docker that gives each AI coding agent its own disposable microVM, with its own filesystem, network and Docker daemon, so the agent can run unattended without touching your host.

Docker started the project to answer a simple question developers kept asking: how do I run Claude Code or Gemini CLI safely? The first preview shipped as a docker sandbox command inside Docker Desktop and used containers. The current generation, which launched in 2026, moved to one microVM per sandbox and to a standalone CLI called sbx. You no longer need Docker Desktop to use it.

The building blocks

Which agents are supported

Out of the box the docs list Claude Code, Codex, GitHub Copilot CLI, Cursor, Devin, Docker Agent, Droid, Gemini CLI, Kiro and OpenCode, plus a plain shell agent if you just want an isolated Linux box. Switching agents is one word: sbx run claude, sbx run codex, sbx run gemini and so on. Everything in this article works the same regardless of which agent you pick; I'll use Claude Code in the examples.

Local and cloud sandboxes

Sandboxes normally run locally, on your laptop’s own virtualization. Docker also offers cloud sandboxes (sbx --cloud ...) that run the same model on Docker's infrastructure, which is handy for CI runners without nested virtualization or for long-running jobs you want to close your laptop on. This guide focuses on local sandboxes, since that is where most people start.

Why Claude Code runs in “YOLO mode” by default

Here is the part that surprises people. Inside a sandbox, Claude Code starts with --dangerously-skip-permissions by default. That is not reckless; it is the whole point. The permission prompts existed because the agent was running on your real machine. Once the environment is the safety boundary, the prompts become friction with no benefit, so Docker turns them off and lets the agent work at full speed.

4. Under the hood: how a sandbox is built

A sandbox is one microVM plus a few host-side gatekeepers; Docker’s docs describe five isolation layers: hypervisor, network, Docker engine, workspace and credentials.

Let’s walk through the diagram one box at a time.

Layer 1: the hypervisor (the outer wall)

Every sandbox is a lightweight VM with its own Linux kernel, created with your operating system’s native virtualization. Processes inside are invisible to your host and to other sandboxes. When you delete the sandbox with sbx rm, the VM and everything in it are destroyed.

One subtle but important point: inside the VM, the agent runs as a non-root user that has sudo. Docker deliberately does not rely on in-VM permissions for safety. The agent can be root inside its own little computer; what matters is that the computer is not yours.

Layer 2: the private Docker engine

Each sandbox runs its own Docker engine inside the VM. When the agent types docker build or docker compose up, those commands go to that engine. There is no path to your host's Docker daemon, so the agent cannot see, stop or delete your own containers and images. This is what makes Docker Sandboxes different from "run the agent in a container": the agent gets full Docker powers without the socket-mount problem we saw earlier.

The flip side: sandboxes don’t share image layers with each other or with your host. Each one pulls and caches its own images, so expect some extra disk usage.

Layer 3: the workspace (the only shared folder)

By default, sbx run shares your current directory into the VM through a filesystem passthrough (virtiofs). The folder appears inside the VM at the same absolute path as on your host, so error messages and stack traces point to paths you recognize. Changes are instant in both directions; there is no sync step. Everything else on your disk (home directory, SSH keys, other repos) simply does not exist inside the VM. Section 8 covers safer alternatives to this read-write mount.

Layer 4: the network proxy

The sandbox has its own isolated network. It cannot talk to other sandboxes or directly to your host network. All outbound TCP traffic is routed through a proxy that runs on your host, and that proxy checks every destination against your network policy. Outbound UDP is off by default, ICMP (ping) is blocked, and DNS goes through a resolver that also enforces the policy. If a domain is not allowed, the connection simply fails.

Layer 5: credential injection

This is the cleverest part. Your API keys are never copied into the VM. Instead, the agent sees a placeholder (a “sentinel”) value. When it sends a request to, say, the Anthropic API, the request leaves the VM, the host proxy checks the policy, looks up the real key in your local secret store, and swaps it into the request header on the way out.

Why this matters: even if a malicious package runs inside the sandbox and dumps every environment variable and file it can find, there is no real key to steal. The worst it can do is make requests through the proxy, and those requests are still limited by the network policy. Note the difference, though: the key can’t be stolen, but it can still be used. Whatever a token is allowed to do, a tricked agent can do too, which is why GitHub access Section recommends scoping every key to the single job it is for.

Lifecycle: sandboxes are persistent, not ephemeral

A common misconception is that a sandbox is thrown away when the agent exits. It is not. Installed packages, pulled images and agent history survive sbx stop and come back on the next sbx run. A sandbox is deleted only when you run sbx rm (or start it with --rm for a throwaway session).

How it compares

5. Hands-on: your first sandboxed Claude Code session

Four commands take you from nothing to Claude Code running in a microVM: install sbx, sign in, open a project, and run sbx run claude.

Prerequisites

A 64-bit machine with hardware virtualization enabled (on by default on almost all modern Macs and PCs; on Linux, KVM must be available).
A free Docker account (you sign in once with sbx login).
For Claude Code: a Claude subscription (Pro, Max, Team or Enterprise) or an Anthropic API key.
Docker Desktop is not required.

Step 1: install the sbx CLI

Pick the line for your operating system:

# macOS (Homebrew)
brew install docker/tap/sbx

# Windows (PowerShell)
winget install -h Docker.sbx

# Linux (Debian/Ubuntu): add Docker's apt repo, then install
curl -fsSL https://get.docker.com | sudo REPO_ONLY=1 sh
sudo apt-get install docker-sbx

What is happening here? On macOS, docker/tap/sbx tells Homebrew to fetch the formula from Docker's own "tap" (a third-party package repository) instead of Homebrew's main catalog. On Windows, winget is the built-in package manager and -h means "install silently". On Linux, the get.docker.com script with REPO_ONLY=1 only registers Docker's package repository with apt; it does not install Docker Engine. The second line then installs the docker-sbx package from that repository.

Check that it worked:

sbx --help

Step 2: sign in to Docker

sbx login

This opens your browser for Docker’s OAuth sign-in. Signing in gives each sandbox a verified identity, which Docker uses to pull agent images and, in companies, to apply organization-wide policies. Docker states that it does not read your prompts or monitor sessions.

The first time you sign in (or the first time you run a sandbox), the CLI asks you to choose a default network policy:

Initialize the global network policy for your sandboxes:

     1. Open         - All network traffic allowed, no restrictions.
  ❯  2. Balanced     - Default deny, with common dev sites allowed.
     3. Locked Down  - All network traffic blocked unless you allow it.

Choose Balanced for now. Section 6 explains all three in detail.

Step 3: run Claude Code in a sandbox

cd ~/my-project
sbx run --name my-sandbox claude

Let’s break this command down word by word:

The first run takes a little longer because the agent image is downloaded. After that, sandboxes start in seconds.

Step 4: authenticate Claude Code

You have two options:

Claude subscription: inside the sandbox, type /login in Claude Code and complete the browser sign-in. The session token stays on your host and is not stored inside the sandbox.
API key: store it once with sbx secret set (covered in Section 7) and every sandbox can use it.

Step 5: give it a real task

Now try something that would normally trigger a dozen permission prompts, for example:

Add a /health endpoint to this Express app, write a Jest test for it,
run the tests, and then build and run the app with Docker Compose to
confirm it responds.

Watch what happens: the agent edits files, runs npm install, runs the tests, writes a Dockerfile, and runs docker compose up against its own Docker engine, all without asking you anything. Meanwhile your host Docker, your home folder and your credentials are untouched.

You can also pass the first prompt directly from the command line. Everything after -- is forwarded to Claude Code:

sbx run --name my-sandbox claude -- "Add error handling to the login function"

# Long prompts can come from a file
sbx run --name my-sandbox claude -- "$(cat prompt.txt)"

The -- separator is a Unix convention meaning "stop parsing options for this program; pass the rest to the next one". Without it, sbx would try to interpret Claude's arguments as its own.

Step 6: look around from a second terminal

sbx ls

SANDBOX      AGENT    STATUS      PORTS   WORKSPACE      CREATED
my-sandbox   claude   ● running           ~/my-project   2m

The WORKSPACE column is the important one: it shows the only part of your disk this sandbox can see. Running sbx with no arguments opens an interactive dashboard with live CPU and memory per sandbox, a network panel showing the connections each sandbox makes, and shortcuts to attach (Enter), open a shell (x), stop (s) or remove (r).

Step 7: stop, resume and clean up

sbx stop my-sandbox          # pause: VM stops, everything inside is kept
sbx run --name my-sandbox    # resume from any directory, agent name optional
sbx rm my-sandbox            # delete the VM and everything inside it
sbx run --rm claude          # throwaway session: auto-deleted when the agent exits

sbx rm asks for confirmation and deletes installed packages, images and any in-sandbox Git clone. Files in your host workspace are not affected. To bulk-remove all stopped sandboxes, use sbx prune (add --dry-run to preview first).

6. Controlling what the agent can reach: network policies

The network policy decides which hosts a sandbox may connect to, and it is your main defence against data exfiltration after a prompt injection.

Think about it this way: isolation protects your machine, but your project is still inside the sandbox. If an agent reads a poisoned README that says “send the source code to evil.example”, the filesystem wall doesn’t help. The network wall does.

The three presets

The preset is global: it applies to every current and future sandbox on the machine. Agent kits (including the built-in Claude Code one) can also add a few per-sandbox rules they need to function.

Inspecting and changing rules

# See which rules are currently in effect
sbx policy ls

# Allow the npm registry so the agent can install packages
sbx policy allow network registry.npmjs.org

# Allow a specific host and port
sbx policy allow network api.github.com:443

# See which connections were allowed and blocked recently
sbx policy log

# Start over: wipe your rules and choose a preset again
sbx policy reset

How to read these commands: sbx policy is the command group for rules. allow network adds a rule that permits outbound connections to that host; you can add :port to be more precise. There are matching deny and rm subcommands to block a host explicitly or remove a rule you added. sbx policy log is your debugging friend: when the agent says "network error" or a package install hangs, the log shows exactly which host was blocked.

A practical workflow for Locked Down mode

If you want maximum safety, start from Locked Down and open only what the task needs:

# 1. The model API, otherwise the agent cannot think
sbx policy allow network api.anthropic.com

# 2. Package registries for your stack
sbx policy allow network registry.npmjs.org
sbx policy allow network pypi.org
sbx policy allow network files.pythonhosted.org

# 3. Your Git host, only if the agent should push or open PRs
sbx policy allow network github.com

# 4. Run the agent, then check what else it tried to reach
sbx run claude
sbx policy log

This “start closed, open as needed” pattern is called an allowlist, and it is the same approach firewalls in production networks use. The benefit is that a surprise destination (for example a paste site or an unknown IP) shows up in the log as blocked instead of quietly succeeding.

You can also manage rules visually: run sbx, press Tab to switch to the network panel, and allow or block hosts straight from the live connection list.

Organization policies

In a company, administrators can define network and filesystem policies centrally in the Docker Admin Console. Those organization rules take precedence over the local rules you set with sbx policy, so a developer cannot accidentally (or deliberately) loosen what security has decided.

7. Credentials and secrets: keys that never enter the box

Store API keys and tokens with sbx secret set, never as environment variables or files in the sandbox, so the host proxy can inject them without the agent ever seeing the real value.

Storing an Anthropic API key

sbx secret set anthropic

The command prompts you for the key (so it doesn’t end up in your shell history) and stores it in a secret store on your host. anthropic is the name of a built-in service: the CLI knows which hosts that key belongs to, so the proxy only adds it to requests going to Anthropic's API, never to other destinations.

If you use a Claude subscription instead of an API key, skip this and run /login inside Claude Code. The OAuth token also stays on the host.

Giving the agent GitHub access

# Recommended: a fine-grained token limited to one repository
sbx secret set github

# Quick but broad: reuses your personal GitHub CLI token
sbx secret set github --command 'gh auth token'

The first form prompts you to paste a token, so it never lands in your shell history. Create it on GitHub under Settings → Developer settings → Personal access tokens → Fine-grained tokens, select only the repository the task needs, grant only the permissions it uses (for example Contents and Pull requests: read and write), and set an expiry date.

The second form is convenient: sbx obtains the token by running gh auth token (the GitHub CLI command that prints your current token). But that token usually covers every repository you can access, so a prompt-injected agent could push to any of them. Keep it for quick experiments on your own throwaway repos.

Either way, the token itself never lives inside the VM; the proxy adds it to requests to GitHub.

Scope every key to the job

Credential injection stops a key from being stolen, but not from being used. Whatever a token allows, an agent tricked by a prompt injection can do too. So treat each key as the agent’s budget:

Narrow tokens and tight network rules cover each other’s gaps: the policy limits where requests go, and the token limits what they can do once they get there.

Importing many secrets at once

sbx secret import --all

This scans your current environment for the variables each built-in service expects (for example ANTHROPIC_API_KEY) and imports them into the secret store in one go. It is especially useful on CI runners, where secrets are already provided as environment variables.

Environment variables: for configuration, not secrets

Sometimes the agent needs ordinary settings. Use -e for those:

# Set a value directly
sbx run -e LOG_LEVEL=debug claude

# Copy a value from your host environment
export API_URL=https://api.example.com
sbx run -e API_URL claude

# Load several from a file
sbx create --name my-project --env-file .env.sandbox claude .

The rule of thumb: anything you would be upset to see leaked goes into sbx secret; everything else can be an environment variable. Environment variables are readable by every process inside the sandbox, including a malicious postinstall script. Secrets handled by the proxy are not.

Two caveats worth knowing

SSH agent forwarding is on by default. If your host has an SSH agent running, its socket is forwarded so the agent can sign Git commits with your key. The private key never leaves your host, but any process inside the sandbox can ask your SSH agent to sign or authenticate while the sandbox runs. If that worries you, disable forwarding or use a dedicated key.
User-level agent config is not copied in. Your ~/.claude folder (global settings, personal skills) stays on the host. Only project-level configuration inside the workspace, such as .claude/ in the repo, is visible to the sandboxed agent. This is intentional: global config often contains tokens and personal hooks.

8. Workspaces and Git: direct, clone and mountless modes

The default direct mount lets the agent write straight into your working tree; clone mode (--clone) gives it a private copy instead, which is the safer choice for unfamiliar agents and parallel work.

Mode 1: direct mount (the default)

git checkout -b feat/health-endpoint
sbx run claude
# ...agent works...
git diff
git add -p
git commit -m "feat: add /health endpoint"

The sandbox mounts your working tree read-write. Every file the agent saves appears instantly on your host, and you review it with ordinary Git tools. git add -p ("patch mode") is worth learning here: it walks you through each change hunk by hunk and asks whether to stage it, which forces a real review instead of a blind git add ..

The risk to understand: in direct mode there is no wall between the agent’s edits and your files. It can also modify files that execute code later, on your host, when you do normal things:

Docker’s own advice is to treat sandbox-modified files like a pull request from an untrusted contributor. A quick habit after each session:

git status                 # anything unexpected changed?
git diff                   # read the actual changes
ls -la .git/hooks/         # hooks are invisible to git diff, check them separately

Mode 2: clone mode (recommended for autonomy)

sbx run --clone --name health claude .

With --clone, your repository is mounted read-only at /run/sandbox/source inside the VM, and the agent works on a private Git clone that lives only in the sandbox. Even with root inside the VM, it cannot modify a single byte of your working tree or your .git folder.

When the agent is done, you pull its work like you would from any other remote:

# Ask the agent first: "create a branch feat/health and commit your changes"

git fetch sandbox-health                        # sbx registered this remote for you
git log sandbox-health/feat/health              # see its commits
git diff main..sandbox-health/feat/health       # review everything

# Happy? Bring it home and open a PR
git checkout -b feat/health sandbox-health/feat/health
git push -u origin feat/health
gh pr create

How this works behind the scenes: the sandbox runs a small Git daemon that publishes its clone, and sbx adds a remote named sandbox- to your host repository that points at it. Fetching from that remote is exactly as safe as fetching from a colleague's fork: nothing is merged until you merge it. A few things to know:

--clone is fixed when the sandbox is created; to switch modes, remove and recreate.
The clone starts from whatever branch you have checked out; no branch is created automatically, so ask the agent to create one.
The Git daemon only runs while the sandbox runs, so git fetch fails after sbx stop.
sbx rm deletes the clone. Fetch or push anything you want to keep first.
Read-only does not mean invisible: untracked files such as .env in the repo are still readable by the agent. Keep secrets out of the working tree.

Running several agents in parallel

Clone mode shines when you want several tasks done at once. One clone-mode sandbox can hold many branches, and Claude Code’s agents view can dispatch background sessions:

sbx run --clone claude . -- --dangerously-skip-permissions agents

Why the long flag? When the first argument after -- is a bare word like agents, it replaces the default startup command, so the default --dangerously-skip-permissions would be lost. Passing it explicitly keeps the agents in autonomous mode. To keep tasks separated, add a line to your project's CLAUDE.md such as "Always start each task on its own git branch." Then fetch all the branches at the end with a single git fetch sandbox-.

Alternatively, give each task its own named sandbox:

sbx run --clone --name feature claude ~/my-project
sbx run --clone --name spike   claude ~/my-project

Mode 3: mountless (scratch space)

sbx create --name scratch claude     # no path = nothing shared from the host
sbx run --name scratch               # attach to it

# Move individual files in or out (sandbox paths must be absolute)
sbx cp ./config.json scratch:/home/agent/workspace/
sbx cp scratch:/home/agent/workspace/output.log ./

The agent works in /home/agent/workspace inside the VM. This is ideal for "try this library and tell me if it fits" experiments where you don't want anything touching your disk.

Bonus: extra read-only folders

sbx run claude ~/project-a ~/shared-libs:ro ~/docs:ro

The first path is the main (read-write) workspace. Each extra path ending in :ro is mounted read-only, so the agent can read your shared library or design docs for context without being able to change them.

9. Everyday commands you’ll actually use

Beyond run, stop and rm, five commands cover most daily work: exec for a shell, ports to reach dev servers, -d for background sandboxes, template save to reuse a setup, and a different agent name to switch tools.

Open a shell inside a running sandbox

sbx exec -it my-sandbox bash

exec runs a command inside an existing sandbox (just like docker exec). -i keeps input open and -t allocates a terminal, so together -it gives you an interactive shell. It starts in the primary workspace. Use it to inspect what the agent installed, check logs, or run a command yourself.

You can also run a single command without an interactive shell:

sbx exec my-sandbox -- npm test
sbx exec my-sandbox bash -c "docker ps"   # see the agent's containers, not yours

That second example is a nice way to see the isolation: docker ps on your host and inside the sandbox show completely different lists.

Reach a dev server running in the sandbox

Sandboxes are network-isolated, so localhost:3000 on your laptop won't reach an app the agent started inside the VM. You publish a port explicitly:

# When creating the sandbox: host port 8080 -> sandbox port 3000
sbx run --publish 8080:3000 --name web claude

# Or on an existing sandbox
sbx ports web --publish 8080:3000
open http://localhost:8080        # macOS; use your browser on other OSes

# Let the OS pick a free host port, then look it up
sbx ports web --publish 3000
sbx ports web

# Stop forwarding
sbx ports web --unpublish 8080:3000

The format HOST:SANDBOX is the same as Docker's -p flag: the left number is the port on your machine, the right one is the port inside the sandbox.

Keep a sandbox running in the background

sbx run -d --name web claude .

-d (detach) starts the sandbox and returns immediately without opening the agent. A detached sandbox keeps running until you stop it, which is useful for serving an app on a published port while you work elsewhere.

Save a configured sandbox as a reusable template

Suppose you spent 20 minutes installing a specific Node version, Playwright browsers and a database client. Instead of repeating that for every new sandbox:

sbx stop my-sandbox
sbx template save my-sandbox my-node-env:v1        # snapshot its filesystem
sbx run -t my-node-env:v1 claude ~/other-project   # start a new sandbox from it

sbx template ls                                    # list saved templates
sbx template save my-sandbox my-node-env:v1 --output my-node-env.tar   # share as a file
sbx template load my-node-env.tar                  # import on another machine

Watch out: a template captures everything in the sandbox’s filesystem. If you ever pasted an API key into a file there, it is now inside the template and inside anyone’s copy of it. This is another reason to keep credentials in sbx secret. For a fully reproducible, version-controlled setup, Docker's kits let you declare tools, network rules and the agent in code.

Switch agents

Same sandbox model, different brain:

sbx run codex      # OpenAI Codex CLI
sbx run copilot    # GitHub Copilot CLI
sbx run gemini     # Google Gemini CLI
sbx run kiro       # Kiro
sbx run opencode   # OpenCode
sbx run shell      # no agent, just an isolated Linux box

Each agent has its own template and its own authentication method (check the agent’s page in the Docker docs). Everything else in this guide, from policies to clone mode, works identically.

Quick reference

10. Running agents headless and in CI

In scripts and CI, the pattern is: log in with a Docker access token, sbx create a clone-mode sandbox, sbx exec the agent with a prompt, fetch its branch, then sbx rm --force.

The four-line pattern

echo "$DOCKER_PAT" | sbx login --username "$DOCKER_USER" --password-stdin

sbx create --name ci-task --clone claude .
sbx exec ci-task -- claude --dangerously-skip-permissions -p "Review the code, fix lint errors and commit your changes"
git fetch sandbox-ci-task
sbx rm --force ci-task

Line by line:

Log in without a browser. A Docker Personal Access Token (PAT, with at least Read scope) is piped in through --password-stdin, so it never appears in the process list or logs.
sbx create builds the sandbox but doesn't attach a terminal (there is nobody to type in CI). --clone means the agent can't modify the runner's checkout directly.
sbx exec ... -- claude -p "..." runs Claude Code in print mode (-p): it executes the prompt to completion and exits instead of opening the interactive UI. Because exec replaces the default command, we pass --dangerously-skip-permissions explicitly.
git fetch sandbox-ci-task pulls the agent's commits onto the runner, where later steps can test them or open a PR.
sbx rm --force cleans up. In scripts, --force matters: confirmation prompts need a real terminal, and piping echo y into them does not work. A command that needs confirmation with no terminal exits with status 1.

A GitHub Actions example

name: agent-lint-fix
on:
  workflow_dispatch:
jobs:
  fix:
    runs-on: ubuntu-latest   # needs a Linux runner with KVM available
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Install sbx
        run: |
          curl -fsSL https://get.docker.com | sudo REPO_ONLY=1 sh
          sudo apt-get install -y docker-sbx
      - name: Sign in and store the model key
        run: |
          echo "${{ secrets.DOCKER_PAT }}" | sbx login 
--username "${{ secrets.DOCKER_USER }}" --password-stdin
          sbx secret set anthropic --force -t "${{ secrets.ANTHROPIC_API_KEY }}"
      - name: Let the agent work in a sandbox
        run: |
          sbx create --name ci-task --clone claude .
          sbx exec ci-task -- claude --dangerously-skip-permissions \
            -p "Run the linter, fix every error, and commit on a branch named agent/lint-fix"
          git fetch sandbox-ci-task
          git push origin sandbox-ci-task/agent/lint-fix:refs/heads/agent/lint-fix
      - name: Clean up
        if: always()
        run: sbx rm --force ci-task || true

Things worth explaining in this workflow:

fetch-depth: 0 checks out the full Git history, so the clone inside the sandbox has real branches to work with.
sbx secret set anthropic --force -t ... stores the key from GitHub's secret store into the sandbox secret store. -t passes the token value; --force overwrites any existing entry without prompting.
git push origin sandbox-ci-task/agent/lint-fix:refs/heads/agent/lint-fix uses Git's source:destination refspec syntax: take the branch from the sandbox remote and publish it on origin under a normal branch name. The push runs on the runner, outside the sandbox, so you decide what leaves.
if: always() makes the cleanup step run even when an earlier step fails.
The runner needs hardware virtualization. If your CI provider doesn’t expose KVM, Docker’s cloud sandboxes (sbx --cloud) run the same workflow without local virtualization.

Machine-readable output

For scripts, never parse the pretty tables; ask for JSON instead:

sbx ls --json | jq -r '.[].name'          # names of all sandboxes
sbx prune --dry-run --format yaml         # what would be removed

Results go to stdout and errors to stderr, so jq only ever sees clean data. Note that sbx v0.48.0 changed several flags (for example --detached became --detach/-d) and switched JSON fields to snake_case; update older scripts accordingly.

11. Security checklist, limitations and gotchas

A sandbox shrinks the blast radius, but it does not make an agent trustworthy; the remaining risks sit exactly at the doors you open: the workspace, the network allowlist, and forwarded credentials.

A practical checklist

Use Balanced or Locked Down network policy, never Open, for anything beyond a toy project.
Store keys with sbx secret set, not -e or .env files inside the workspace.
Scope every token to the single repo and API the task needs (fine-grained GitHub token, separate spend-capped model key), so a hijacked agent can only spend what it was handed.
Give every token an expiry, ideally one token per run, so it dies with that run instead of sitting in a config for months. In CI, prefer short-lived tokens such as GitHub Actions’ built-in GITHUB_TOKEN, which expires when the job ends.
Prefer --clone when running unattended or running several agents.
After a direct-mount session, review git diff, .git/hooks/, CI files and build scripts before committing or building on the host.
Keep real secrets (.env, cloud credentials) outside the project folder; clone mode still lets the agent read the repo.
Check sbx policy log occasionally to see what your agents are actually trying to reach.
Remove sandboxes you no longer need (sbx rm, sbx prune) to reclaim disk and reduce stale state.
Never save a template from a sandbox where you typed a credential into a file.

Limitations to be aware of

Coming from the old docker sandbox commands?

Early tutorials (late 2025 to early 2026) use the Docker Desktop integration:

# Old, deprecated Docker Desktop integration
docker sandbox run claude
docker sandbox ls
docker sandbox rm 

# Current standalone CLI
sbx run claude
sbx ls
sbx rm

The concepts carry over directly; mainly the command prefix changed, and you no longer need Docker Desktop. If you have old sandboxes, recreate them with sbx rather than trying to migrate them.

12. Conclusion

Docker Sandboxes change the safety question from “should I approve this command?” to “what should this computer be allowed to touch?”, which is a question you answer once instead of forty times a session.

The core ideas to take away:

Isolation beats permission prompts. Give the agent its own microVM and let it work at full speed inside.
MicroVMs are stronger than containers. A separate kernel behind a hypervisor is a wall; a shared kernel is a fence.
The doors are where the risk lives. Control the workspace (prefer --clone), the network (Balanced or Locked Down) and credentials (sbx secret, never env vars).
Review like it’s a stranger’s PR. Especially hooks, CI files and build scripts.

If you try only one thing after reading this, make it this:

cd ~/a-repo-you-dont-mind-experimenting-with
sbx run --clone --name first-try claude .

Then ask the agent to do something ambitious, and watch it work without asking for permission once.

Sources

Wiring MongoDB as an Agent’s Long-Term Memory with LangGraph.js

Vasanthan K — Fri, 16 Oct 2026 13:01:04 GMT

Why your agent keeps forgetting you

By the end of this post you will have a LangGraph.js agent that remembers a user’s name, preferences and past decisions across completely separate conversations, with every memory stored as a plain JSON document in MongoDB.

Here is the problem we are solving. You build a chatbot. On Monday the user says, “I’m vegetarian and I only cook on weekends.” The bot answers nicely. On Thursday the same user opens a new chat and asks for a dinner idea. The bot suggests chicken. It forgot everything.

That happens because a large language model (LLM) has no memory of its own. Every call is a fresh start. The only thing the model “knows” about your user is whatever text you put into the prompt for that one call. If you want the agent to remember, you have to save the important facts somewhere and load them back in next time.

That “somewhere” is what we build today. We will use three pieces:

LangGraph.js to describe the agent as a small graph of steps (recall, respond, remember).
MongoDB as the database that keeps both the current conversation and the long-term facts.
@langchain/langgraph-checkpoint-mongodb, the official package that connects the two. Since May 2026 it ships both a checkpointer (MongoDBSaver) for short-term memory and a store (MongoDBStore) for long-term memory (MongoDB product update).

Who this is for. You know basic JavaScript or TypeScript and have used async/await. You do not need prior LangGraph or vector-search experience. Every code block is followed by a plain-English walkthrough of what each part does and why it is there.

What you will build, step by step:

Give the agent short-term memory so it can follow one conversation.
Add a long-term memory store that survives across conversations.
Build a three-node graph that recalls, responds and remembers.
Upgrade recall to semantic search, so the agent finds memories by meaning, not exact words.
Look at what actually lands in MongoDB, then harden it for production.

The five ideas you need first

Agent memory in LangGraph rests on five concepts: threads, short-term memory, long-term memory, namespaces and keys. Learn these and the code later will read like plain English.

1. A thread is one conversation

A thread is a single chat session, identified by a string you choose called thread_id. Think of it like one WhatsApp chat window. Monday's chat is thread-mon, Thursday's chat is thread-thu. LangGraph keeps the messages of each thread separate.

2. Short-term memory = the checkpointer

Inside a thread, the agent must remember what was said three messages ago. LangGraph handles this with a checkpointer. After every step of the graph, the checkpointer saves a snapshot of the whole state (all messages so far) to a database. On the next request with the same thread_id, LangGraph loads that snapshot and continues where it left off.

The MongoDB checkpointer is called MongoDBSaver. Besides plain chat history, checkpoints also enable time travel, human-in-the-loop review and recovery after a crash for a given conversation (MongoDB docs).

The limit: a checkpoint belongs to one thread. Thread B cannot see Thread A’s checkpoints. That is exactly why our bot forgot the user was vegetarian.

3. Long-term memory = the store

A store is a separate, simple key-value database that every thread can read and write. LangGraph defines it through a BaseStore interface with four core operations: put, get, search and delete. MongoDBStore implements that interface on top of a MongoDB collection.

An everyday comparison: the checkpointer is the notes you scribble during one phone call. The store is the contact card you update afterwards, which you see every time that person calls again.

4. Namespaces are folders

Every item in the store lives under a namespace, written as an array of strings such as ["user-42", "memories"]. Read it like a folder path: user-42/memories/. Namespaces keep one user's memories away from another's. You can nest them as deep as you like, for example ["org-7", "user-42", "preferences"].

5. Keys and values are files

Inside a namespace each item has a key (like a file name, e.g. "diet") and a value (any JSON object, e.g. { text: "User is vegetarian" }). Writing the same namespace and key again overwrites the old value. That is how a memory gets updated instead of duplicated.

Checkpointer vs store at a glance

The last two rows matter most. The checkpointer saves everything without being asked. The store saves only what you decide is worth keeping. Deciding what to keep is the real design work of long-term memory, and we will do it in the remember node.

The architecture we are building

One MongoDB cluster holds both memory layers, and the LangGraph.js graph talks to it through two adapters: MongoDBSaver for the thread and MongoDBStore for the user.

Read the picture from left to right:

Your app (an Express route, a CLI, a Slack bot) receives a message. It knows two things about the request: which conversation it belongs to (thread_id) and which person sent it (userId).
The LangGraph.js graph runs three nodes in order. recall loads relevant facts about this user. respond asks the LLM for an answer. remember decides whether anything new is worth saving.
MongoDBSaver quietly saves the full graph state after every node. You never call it directly; you hand it to the graph once at compile time.
MongoDBStore is the long-term memory. Our own node code calls search, get and put on it.
MongoDB Atlas is the single database underneath. The checkpointer and the store use separate collections, and an Atlas Vector Search index lets the store find memories by meaning.
The LLM and the embeddings model sit outside the database. We use Claude for answers and OpenAI’s text-embedding-3-small to turn text into vectors. Both are swappable.

Why put everything in MongoDB? If your app already uses MongoDB, the agent’s memory becomes just another collection. You back it up, secure it and monitor it with tools you already run. No extra vector database or Redis cache is needed. MongoDB’s own announcement makes the same pitch: one database for conversation state, long-term storage and vector search (MongoDB product update).

Setting up the project

You need Node.js 20 or newer, a MongoDB Atlas cluster (the free tier works), an Anthropic API key for the chat model and an OpenAI API key for embeddings.

Create the project and install packages

mkdir mongo-memory-agent && cd mongo-memory-agent
npm init -y
npm pkg set type=module

npm install @langchain/langgraph @langchain/core \
  @langchain/langgraph-checkpoint-mongodb \
  @langchain/anthropic @langchain/openai \
  mongodb zod dotenv
npm install -D typescript tsx @types/node

What each package does:

npm pkg set type=module tells Node to treat .ts/.js files as ES modules, so we can use import and top-level await.

Store your secrets in .env

# .env  (never commit this file)
MONGODB_URI="mongodb+srv://:@.mongodb.net/?retryWrites=true&w=majority"
ANTHROPIC_API_KEY="sk-ant-..."
OPENAI_API_KEY="sk-..."

In Atlas, copy the connection string from Connect → Drivers. Also add your IP address under Network Access, or the connection will time out with a confusing error.

One shared MongoDB client

// src/db.ts
import "dotenv/config";
import { MongoClient } from "mongodb";

export const DB_NAME = "agent_memory";
export const client = new MongoClient(process.env.MONGODB_URI!);
export async function connect() {
  await client.connect();
  await client.db("admin").command({ ping: 1 });
  console.log("Connected to MongoDB");
}

How this works. import "dotenv/config" reads .env and copies each line into process.env. new MongoClient(...) creates a connection pool but does not connect yet. connect() opens the pool, and the ping command proves the cluster answers before the agent starts.

We create the client exactly once and export it. Every part of the app reuses this single pool. Creating a new client per request is a classic beginner mistake: it opens a fresh set of connections each time and soon exhausts the cluster’s connection limit.

Step 1: Short-term memory with MongoDBSaver

A checkpointer makes a single conversation coherent in about ten lines of code, and it is the foundation the long-term store builds on.

Let us first prove the problem exists, then fix half of it. Here is the smallest possible LangGraph.js agent with a MongoDB checkpointer:

// src/step1-short-term.ts
import { StateGraph, MessagesAnnotation, START, END } from "@langchain/langgraph";
import { MongoDBSaver } from "@langchain/langgraph-checkpoint-mongodb";
import { ChatAnthropic } from "@langchain/anthropic";
import { HumanMessage } from "@langchain/core/messages";
import { client, connect, DB_NAME } from "./db.js";

const model = new ChatAnthropic({ model: "claude-sonnet-4-6" });
// A node is just an async function: (state) => partial state update
async function respond(state: typeof MessagesAnnotation.State) {
  const reply = await model.invoke(state.messages);
  return { messages: [reply] };
}
await connect();
const checkpointer = new MongoDBSaver({ client, dbName: DB_NAME });
const app = new StateGraph(MessagesAnnotation)
  .addNode("respond", respond)
  .addEdge(START, "respond")
  .addEdge("respond", END)
  .compile({ checkpointer });
const ask = async (threadId: string, text: string) => {
  const out = await app.invoke(
    { messages: [new HumanMessage(text)] },
    { configurable: { thread_id: threadId } },
  );
  console.log(`[${threadId}]`, out.messages.at(-1)?.content);
};
await ask("thread-mon", "Hi! I'm vegetarian and I only cook on weekends.");
await ask("thread-mon", "What did I just tell you about my diet?"); // remembers
await ask("thread-thu", "Suggest a dinner for Saturday.");          // forgets!
await client.close();

Run it with npx tsx src/step1-short-term.ts.

What each part does

MessagesAnnotation is a ready-made state definition. It describes the graph's state as an object with one field, messages, an array of chat messages. It also comes with a reducer: when a node returns { messages: [reply] }, LangGraph appends reply to the existing array instead of replacing it. That append behaviour is what lets history grow turn after turn.

The respond node receives the full state, passes every message to Claude, and returns only the new message. Nodes never mutate state directly; they return a partial update and LangGraph merges it. This keeps each node small and easy to test.

StateGraph(...).addNode().addEdge() wires the steps together. START and END are special built-in nodes marking where a run begins and stops. Here the graph is a straight line: START → respond → END.

new MongoDBSaver({ client, dbName }) creates the checkpointer. It reuses our shared client and writes into the agent_memory database. By default it uses two collections: one for checkpoints and one for pending writes between steps.

.compile({ checkpointer }) turns the description into a runnable app and attaches the checkpointer. From now on LangGraph saves a checkpoint after every node, automatically.

{ configurable: { thread_id } } is the most important line for beginners. The thread_id tells the checkpointer which conversation to load before running and where to save after running. Same id, same conversation. New id, blank slate.

What happens when you run it

First call on thread-mon: no checkpoint exists, so state starts empty. The human message is added, Claude replies, and a checkpoint with both messages is saved.
Second call on thread-mon: LangGraph loads the saved checkpoint first, so the state already holds the vegetarian message. Claude sees it and answers correctly.
Call on thread-thu: a different thread_id means no checkpoint. Claude sees only "Suggest a dinner for Saturday" and may happily suggest chicken.

That third result is the gap we close next. Short-term memory works perfectly, but it is locked inside a thread.

Step 2: A long-term memory store on MongoDB

MongoDBStore gives you four calls (put, get, search, delete) over a MongoDB collection, and those four calls are all the long-term memory you need.

Before wiring the store into a graph, use it on its own. Seeing it work in isolation makes the graph code much easier to follow later.

// src/store.ts
import { MongoDBStore } from "@langchain/langgraph-checkpoint-mongodb";
import { client, DB_NAME } from "./db.js";

export const store = new MongoDBStore({
  client,
  dbName: DB_NAME,
  collectionName: "long_term_memory",
});
// Every user gets their own "folder"
export const memoryNamespace = (userId: string) => [userId, "memories"];

// src/step2-store-playground.ts
import { connect, client } from "./db.js";
import { store, memoryNamespace } from "./store.js";

await connect();
const ns = memoryNamespace("user-42");
// 1. put: create or overwrite one memory
await store.put(ns, "diet", {
  text: "User is vegetarian",
  category: "preference",
});
await store.put(ns, "schedule", {
  text: "User only cooks on weekends",
  category: "preference",
});
// 2. get: read one memory by its exact key
const diet = await store.get(ns, "diet");
console.log(diet?.value);      // { text: 'User is vegetarian', category: 'preference' }
console.log(diet?.updatedAt);  // when it was last written
// 3. search: list memories in a namespace, optionally filtered
const prefs = await store.search(ns, {
  filter: { category: "preference" },
  limit: 10,
});
console.log(prefs.map((item) => `${item.key}: ${item.value.text}`));
// 4. delete: forget one memory
await store.delete(ns, "schedule");
await client.close();

What each call does

new MongoDBStore({ client, dbName, collectionName }) points the store at one collection. Every memory for every user lands in long_term_memory; the namespace inside each document keeps them apart. The official MongoDB tutorial uses this same constructor shape (MongoDB docs). If you would rather let the store manage its own connection, MongoDBStore.fromConnString(uri) connects and ensures the indexes exist for you (LangChain docs).

memoryNamespace(userId) is a tiny helper, but an important habit. Building namespaces in exactly one place guarantees you never write to ["user-42", "memory"] in one file and read from ["user-42", "memories"] in another, a typo that silently "loses" data.

put(namespace, key, value) is an upsert: insert if the key is new, replace if it exists. That is why we pick meaningful keys like "diet". When the user later says "I eat fish now", writing to the same "diet" key replaces the old fact instead of storing two contradictory ones.

get(namespace, key) fetches exactly one item, or null if nothing is there. The result is an Item object: value holds your JSON, while key, namespace, createdAt and updatedAt are added by the store. Use get when you know precisely what you want, such as a profile record.

search(namespacePrefix, options) returns many items. With only a filter, it behaves like a normal MongoDB query on fields of value. The first argument is a prefix, so search(["user-42"]) would also find items under ["user-42", "memories"] and ["user-42", "preferences"]. In Step 4 we add a query string to turn this into semantic search.

delete(namespace, key) removes one item. You will need it for "forget that" requests and for privacy deletions.

What lands in MongoDB

Each put becomes one document. The namespace array, the key and your JSON value are stored side by side, plus timestamps. Once semantic search is enabled, an embedding array (the vector) is added too. Open the collection in MongoDB Compass or the Atlas Data Explorer to see your own; exact internal field names can vary between package versions, so trust what you see there over any diagram, including this one.

Step 3: The graph that recalls, responds and remembers

The complete agent is three nodes in a line: recall reads facts from the store, respond answers with those facts in the prompt, and remember writes any new durable facts back.

The full agent

// src/agent.ts
import {
  StateGraph, Annotation, MessagesAnnotation, START, END,
  type LangGraphRunnableConfig,
} from "@langchain/langgraph";
import { MongoDBSaver } from "@langchain/langgraph-checkpoint-mongodb";
import { ChatAnthropic } from "@langchain/anthropic";
import { SystemMessage, HumanMessage, type BaseMessage } from "@langchain/core/messages";
import { z } from "zod";
import { client, DB_NAME } from "./db.js";
import { store, memoryNamespace } from "./store.js";

// ---------- 1. State ----------
const AgentState = Annotation.Root({
  ...MessagesAnnotation.spec,          // messages[] with the append reducer
  memories: Annotation(),      // recalled facts, as text for the prompt
});
type State = typeof AgentState.State;
// ---------- 2. Helpers ----------
const model = new ChatAnthropic({ model: "claude-sonnet-4-6", temperature: 0 });
const textOf = (m: BaseMessage) =>
  typeof m.content === "string" ? m.content : JSON.stringify(m.content);
function getUserId(config: LangGraphRunnableConfig): string {
  const userId = config.configurable?.userId;
  if (!userId) throw new Error("configurable.userId is required");
  return userId;
}
// ---------- 3. Node: recall ----------
async function recall(state: State, config: LangGraphRunnableConfig) {
  const ns = memoryNamespace(getUserId(config));
  const items = await config.store!.search(ns, { limit: 20 });
  const memories = items.length
    ? items.map((i) => `- ${i.value.text}`).join("\n")
    : "(nothing known yet)";
  return { memories };
}
// ---------- 4. Node: respond ----------
async function respond(state: State) {
  const system = new SystemMessage(
    `You are a friendly cooking assistant.\n` +
    `Facts you remember about this user from earlier conversations:\n` +
    `${state.memories}\n` +
    `Use these facts naturally. Never invent facts that are not listed.`,
  );
  const reply = await model.invoke([system, ...state.messages]);
  return { messages: [reply] };
}
// ---------- 5. Node: remember ----------
const FactSchema = z.object({
  facts: z.array(z.object({
    key: z.string().describe("short stable snake_case id, e.g. diet, cooking_days"),
    text: z.string().describe("the fact as one plain sentence about the user"),
    category: z.enum(["profile", "preference", "constraint", "decision"]),
  })).describe("durable facts worth remembering; empty if none"),
});
const extractor = model.withStructuredOutput(FactSchema);
async function remember(state: State, config: LangGraphRunnableConfig) {
  const ns = memoryNamespace(getUserId(config));
  const lastUser = [...state.messages].reverse().find((m) => m.getType() === "human");
  if (!lastUser) return {};
  const { facts } = await extractor.invoke([
    new SystemMessage(
      `Extract long-lived facts about the user from their latest message.\n` +
      `Keep: identity, preferences, constraints, decisions.\n` +
      `Skip: small talk, one-off questions, anything temporary.\n` +
      `Reuse an existing key when updating a known fact.\n` +
      `Known facts:\n${state.memories}`,
    ),
    new HumanMessage(textOf(lastUser)),
  ]);
  for (const f of facts) {
    await config.store!.put(ns, f.key, { text: f.text, category: f.category });
  }
  return {};
}
// ---------- 6. Wire and compile ----------
export function buildAgent() {
  return new StateGraph(AgentState)
    .addNode("recall", recall)
    .addNode("respond", respond)
    .addNode("remember", remember)
    .addEdge(START, "recall")
    .addEdge("recall", "respond")
    .addEdge("respond", "remember")
    .addEdge("remember", END)
    .compile({
      checkpointer: new MongoDBSaver({ client, dbName: DB_NAME }), // short-term
      store,                                                       // long-term
    });
}

Walking through it, block by block

1. State. Annotation.Root defines what flows between nodes. We spread MessagesAnnotation.spec to inherit the messages field and its append reducer from Step 1, then add one field of our own: memories, a string. It has no reducer, so each write simply replaces the previous value. That is what we want: fresh recall on every turn.

2. Helpers. temperature: 0 makes Claude more deterministic, which matters for fact extraction. textOf flattens message content, because LangChain messages can hold either a string or an array of content blocks. getUserId reads the user's identity from config.configurable. Notice that userId travels in the config, not the state. Config is per-request context that is not saved into checkpoints, which is exactly right for "who is calling".

3. recall. Every node receives two arguments: the current state and a config object. When you compile a graph with store, LangGraph injects that store as config.store, so nodes never import it directly. That makes nodes easy to test: pass a fake store in the config and you are done. For now recall simply lists up to 20 memories in the user's namespace and formats them as bullet lines. Step 4 swaps this for a smarter semantic search.

4. respond. We build a SystemMessage that injects the recalled facts, then call Claude with [system, ...state.messages]. The system message is created fresh each turn and is not returned, so it never gets saved into the conversation history. Only the AI's reply is returned and appended. The line "never invent facts" is a small but useful guard against the model pretending to remember things.

5. remember. This is where long-term memory is actually decided. We ask Claude, via withStructuredOutput, to return JSON that matches FactSchema. Zod does double duty here: it tells the model the exact shape we want, and it validates the response. The .describe() texts act as mini-instructions to the model for each field.

The extraction prompt has three rules worth understanding:

Keep vs skip. “I’m vegetarian” is durable; “what’s for dinner tonight?” is not. Without this rule the store fills with noise.
Reuse an existing key. Because we pass the known facts in, the model can reuse diet when the user changes their mind. put then overwrites, and the store stays consistent.
Empty is fine. Most messages contain no new facts. Returning { facts: [] } means no writes at all.

6. Wire and compile. Edges run START → recall → respond → remember → END. In .compile() we attach both memories at once: the checkpointer for the thread and the store for the user. That one line is the whole "wiring" the title promised.

A note on cost and latency

remember makes a second LLM call per turn, after the user already has their answer. In a web app you can stream the respond output to the user and let remember finish in the background. Another option is running extraction only every few turns, or when the conversation ends. Start simple, measure, then optimise.

Step 4: Semantic recall with Atlas Vector Search

With an embeddings model and an indexConfig, the same store.search call returns the memories closest in meaning to the user's message, so recall stays sharp even when a user has hundreds of memories.

Recalling “the latest 20 memories” works for a demo. It breaks down fast in real life. After a few months a user may have 300 memories, and only three of them matter for the current question. Dumping all 300 into the prompt costs tokens and confuses the model.

Keyword search does not help much either. Look at the example below: the question “Suggest a dinner for Saturday” shares no words with “User is vegetarian”, yet that fact is obviously relevant.

Embeddings in one paragraph

An embedding is a list of numbers (here, 1,536 of them) that represents the meaning of a piece of text. Texts with similar meanings get vectors that point in similar directions. “Dinner” and “vegetarian” both sit in the food region of this space; “works in healthcare IT” sits far away. Vector search embeds the question, then finds stored vectors pointing the closest way, usually measured with cosine similarity, a score where 1 means identical direction and values near 0 mean unrelated.

Turn on semantic search

Only store.ts and the recall node change. Everything else stays the same.

// src/store.ts  (semantic version)
import "dotenv/config";
import { MongoDBStore } from "@langchain/langgraph-checkpoint-mongodb";
import { OpenAIEmbeddings } from "@langchain/openai";

export const store = await MongoDBStore.fromConnString(process.env.MONGODB_URI!, {
  embeddings: new OpenAIEmbeddings({ model: "text-embedding-3-small" }),
  indexConfig: {
    name: "memory_vector_index",  // name of the Atlas Vector Search index
    dims: 1536,                   // must match the embedding model's output
    similarityFunction: "cosine",
  },
});
export const memoryNamespace = (userId: string) => [userId, "memories"];

// src/agent.ts  (replace the recall node)
async function recall(state: State, config: LangGraphRunnableConfig) {
  const ns = memoryNamespace(getUserId(config));
  const lastUser = [...state.messages].reverse().find((m) => m.getType() === "human");

const items = await config.store!.search(ns, {
    query: lastUser ? textOf(lastUser) : undefined,  // semantic part
    limit: 5,                                        // only the best 5
  });
  const memories = items.length
    ? items.map((i) => `- ${i.value.text}`).join("\n")
    : "(nothing known yet)";
  return { memories };
}

What changed and why

fromConnString(uri, options) creates the store with its own client, connects it and makes sure the needed indexes exist, including the vector index (LangChain docs). We switch to it here because the vector index setup is the fiddly part and this method handles it.

embeddings is the model used to turn text into vectors. The store calls it automatically on every put (to embed the new memory) and on every search that includes a query (to embed the question). You never call it yourself.

indexConfig.dims: 1536 must equal the vector length your embedding model produces. text-embedding-3-small produces 1,536 numbers. If you switch models, change this number too; a mismatch is the most common cause of empty search results.

similarityFunction: "cosine" tells Atlas how to compare vectors. Cosine is the safe default for text embeddings.

search(ns, { query, limit: 5 }) is the only change in the graph. Adding query switches the call from "list items" to "rank items by meaning". Under the hood the store runs a $vectorSearch aggregation stage against your collection (MongoDB docs). You can still combine it with a filter, for example { category: "constraint" }, to search only one kind of memory.

Prefer no client-side embeddings? If you leave out embeddings, the store can use MongoDB's server-side Automated Embeddings, where Atlas generates vectors with Voyage AI models for you (LangChain reference). That removes one API key from your app; check the reference for the model setting your package version expects.

Existing memories. Items written before you enabled the index have no vectors. The simplest fix while learning is to drop the collection and let the agent rebuild memories. In production, write a one-off script that reads each item with search and writes it back with put, which embeds it.

Running it: the Monday-to-Thursday test

The proof that long-term memory works is a brand-new thread that already knows the user, and this script shows exactly that.

// src/main.ts
import { HumanMessage } from "@langchain/core/messages";
import { connect, client } from "./db.js";
import { buildAgent } from "./agent.js";
import { store, memoryNamespace } from "./store.js";

await connect();
const agent = buildAgent();
const USER = "user-42";
async function chat(threadId: string, text: string) {
  const out = await agent.invoke(
    { messages: [new HumanMessage(text)] },
    { configurable: { thread_id: threadId, userId: USER } },
  );
  console.log(`\n[${threadId}] You: ${text}`);
  console.log(`[${threadId}] Bot: ${out.messages.at(-1)?.content}`);
}
// Monday: the user shares facts
await chat("thread-mon", "Hi! I'm vegetarian and I only cook on weekends.");
// Thursday: a brand-new thread
await chat("thread-thu", "Suggest a dinner for Saturday.");
// Peek inside long-term memory
const saved = await store.search(memoryNamespace(USER), { limit: 10 });
console.log("\nStored memories:", saved.map((i) => `${i.key} → ${i.value.text}`));
await client.close();

Run npx tsx src/main.ts. Your exact wording will differ, but the output should look like this:

[thread-mon] You: Hi! I'm vegetarian and I only cook on weekends.
[thread-mon] Bot: Nice to meet you! Weekend cooking is a great rhythm...

[thread-thu] You: Suggest a dinner for Saturday.
[thread-thu] Bot: Since it's the weekend and you're vegetarian, how about a
                  roasted vegetable lasagne? It's hands-on but relaxing...

Stored memories: [ 'diet → User is vegetarian',
                   'cooking_days → User only cooks on weekends' ]

Trace what happened

Monday, recall: the namespace is empty, so memories is "(nothing known yet)".
Monday, respond: Claude answers the greeting.
Monday, remember: the extractor finds two durable facts and calls put twice, under keys diet and cooking_days. With semantic search on, each is embedded as it is written.
Thursday, new thread: the checkpointer finds nothing for thread-thu, so the conversation history starts empty, as before.
Thursday, recall: this time the store search for user-42 returns both facts. They go into the system prompt.
Thursday, respond: Claude sees "vegetarian" and "weekends" and answers accordingly. The agent remembered across threads.
Thursday, remember: "Suggest a dinner" holds no durable fact, so nothing is written.

Look at the data yourself

Open mongosh (or the Atlas Data Explorer) and inspect both layers:

use agent_memory

// Long-term: one small document per fact
db.long_term_memory.find({}, { embedding: 0 }).pretty()
// Short-term: many checkpoint documents per thread
db.checkpoints.countDocuments({ thread_id: "thread-mon" })

The contrast teaches a lot. The long-term collection holds a couple of tidy, human-readable facts. The checkpoints collection holds several snapshots per thread, one after each node, which is why it grows much faster. The collection names above assume the defaults used in this tutorial; if yours differ, run show collections first. (If fromConnString placed the store in a different database, check the store's options in your package version.)

Taking it to production

The demo works; production needs four more things: trustworthy identity, privacy controls, clean-up of old data and a guard on memory quality.

1. Never trust userId from the request body

The namespace is your only wall between users. If your Express route does userId: req.body.userId, anyone can read anyone's memories by changing one field. Always take userId from your verified auth session or JWT, on the server side. In a multi-tenant SaaS, add the tenant to the namespace too:

export const memoryNamespace = (orgId: string, userId: string) =>
  [orgId, userId, "memories"];

2. Decide what must never be stored

The extractor will save whatever looks durable, and that can include sensitive data: health details, card numbers, passwords pasted by mistake. Add explicit exclusions to the extraction prompt (“never store credentials, payment data, or medical details”), and back them with a code check on f.text before put. If you work in a regulated domain such as healthcare or finance, involve your compliance team before storing anything about end users.

3. Give users a real “forget me”

Privacy laws and plain decency both require deleting a user’s memories on request. Because everything for a user sits under one namespace, deletion is short:

export async function forgetUser(userId: string) {
  const ns = memoryNamespace(userId);
  let batch = await store.search(ns, { limit: 100 });
  while (batch.length) {
    await Promise.all(batch.map((item) => store.delete(item.namespace, item.key)));
    batch = await store.search(ns, { limit: 100 });
  }
}

The loop pages through 100 items at a time, so it works for users with thousands of memories without loading them all into RAM. Remember the user’s old threads in the checkpoints collection, too; they contain the raw messages.

4. Clean up short-term data

Checkpoints grow with every node of every turn, while most threads are never reopened after a week. Pick a retention period (say 30 days) and remove older threads with a scheduled job. Newer checkpointer versions expose a deleteThread(threadId) method; if yours does not, delete by thread_id directly in both checkpoint collections. Long-term memories are different: keep them until the user deletes them or they are superseded.

5. Keep memory quality high

Bad memories are worse than no memories, because the agent states them confidently. Four habits help:

Stable keys (diet, cooking_days) so updates overwrite instead of piling up contradictions.
Provenance: store { text, category, sourceThread, savedAt }, so you can explain and audit where a fact came from.
Let the user see and edit memories. A simple “Here’s what I remember about you” page builds trust and fixes errors.
Evaluate the extractor. Keep 30 to 50 sample messages with the facts you expect, and re-run them whenever you change the prompt or model.

6. Watch the indexes

Atlas Vector Search indexes are built asynchronously. Right after creating one, searches may return nothing for a minute; this is normal. In the Atlas UI under Search & Vector Search, confirm the index status is active before load testing. If you filter searches by fields such as category, make sure those fields are declared as filter fields in the vector index, or the filter will be rejected.

Common mistakes and how to spot them

Most first-time bugs come from six mistakes, and each has a recognisable symptom.

Wrapping up

You now have an agent with two kinds of memory, both stored in MongoDB:

Short-term memory via MongoDBSaver keeps each conversation coherent, saved automatically after every graph step and keyed by thread_id.
Long-term memory via MongoDBStore keeps curated facts about each user, organised by namespace and key, shared across every thread.
A three-node graph (recall → respond → remember) is the pattern that connects them, and semantic search keeps recall relevant as memories grow.

The most important lesson is not an API call. The checkpointer saves everything automatically, but the store saves only what you choose. The quality of your agent’s memory is the quality of your remember node.

Where to go next

Let the model decide when to save by turning remember into a tool (save_memory) it can call, instead of running extraction on every turn.
Split memories into types: semantic (facts), episodic (summaries of past sessions) and procedural (rules the agent learned about how to behave), each in its own namespace.
Add a nightly job that merges duplicate or conflicting memories with a second LLM pass.

If you build on this, I would love to hear what you store and how you decide. Drop a comment with your own memory design.

Sources

Add Long-Term Memory to LangGraph.js Agents with MongoDB Atlas, MongoDB Docs
Long-term memory with MongoDB Atlas (JavaScript), LangChain Docs
IndexConfig reference for @langchain/langgraph-checkpoint-mongodb, LangChain Reference
MongoDB Support for LangGraph.js Long-Term Memory, MongoDB product update, May 8, 2026

Why Your Kubernetes GPUs Sit at 5% Utilization (and How MIG/Time-Slicing Fixes It)

Vasanthan K — Wed, 14 Oct 2026 14:01:04 GMT

The most expensive thing in your cluster is doing nothing

You finally got GPUs into your Kubernetes cluster. The data science team is happy, the inference service is live, and the cloud bill shows a shiny new line item. Then someone opens a Grafana dashboard and asks a very awkward question: why is this GPU at 5% utilization?

This is not a rare edge case. It is the default outcome for most teams running GPUs on Kubernetes. A single data-center GPU can cost anywhere from a few hundred to several thousand dollars per month in the cloud, and in many clusters most of that capacity sits idle while new pods wait in Pending because "there are no GPUs available."

The GPUs are not broken. The workloads are not broken either. The problem is a mismatch between how Kubernetes hands out GPUs (one whole card per request) and how most real workloads use them (in short bursts, with a fraction of the memory).

In this article we will walk through the whole story, step by step, assuming you are new to GPUs on Kubernetes:

How Kubernetes actually sees and allocates a GPU.
Why that model leads to 5% utilization.
How to measure the problem properly, so you are not guessing.
Time-slicing: letting several pods take turns on one GPU.
MIG (Multi-Instance GPU): physically carving one GPU into isolated smaller GPUs.
How to choose between them, and what to watch out for in production.

Every code example comes with an explanation of what it does and why it works, so you can adapt it instead of copy-pasting blindly. All diagrams in this post were created specifically for it.

Part 1: How Kubernetes actually sees a GPU

Kubernetes was built around two resources it understands natively: CPU and memory. Both are divisible. You can ask for 250m of CPU (a quarter of a core) or 512Mi of memory, and the scheduler happily packs many small pods onto one node.

A GPU is different. Kubernetes has no idea what a GPU is. It only learns that GPUs exist through a component called a device plugin.

What a device plugin does

A device plugin is a small program (usually a DaemonSet, so one copy runs on every GPU node) that talks to the kubelet over a gRPC API. For NVIDIA hardware this is the NVIDIA k8s-device-plugin, which is also installed for you by the NVIDIA GPU Operator. It does three jobs:

Discover the GPUs on the node (using the NVIDIA driver).
Advertise them to the kubelet as an extended resource named nvidia.com/gpu, with a count. A node with 4 GPUs reports nvidia.com/gpu: 4.
Allocate a specific GPU when a pod is scheduled, by telling the container runtime which device (GPU UUID) to mount into the container.

You can see the advertised count on any GPU node:

kubectl get nodes -o custom-columns=NAME:.metadata.name,GPUS:.status.allocatable.nvidia\.com/gpu

What this command does: kubectl get nodes lists nodes, and -o custom-columns prints only the fields we care about. The path .status.allocatable.nvidia\.com/gpu reads the number of GPUs the node can hand out. The backslash escapes the dot inside nvidia.com, because otherwise kubectl would treat nvidia and com/gpu as two nested keys.

How a pod asks for a GPU

Here is the simplest possible GPU pod:

apiVersion: v1
kind: Pod
metadata:
  name: cuda-test
spec:
  restartPolicy: Never
  containers:
    - name: cuda
      image: nvcr.io/nvidia/cuda:12.4.1-base-ubuntu22.04
      command: ["nvidia-smi"]
      resources:
        limits:
          nvidia.com/gpu: 1

Line by line:

image: nvcr.io/nvidia/cuda:... is an official CUDA base image. It contains the CUDA user-space libraries but not the driver; the driver lives on the host and is injected by the NVIDIA Container Toolkit.
command: ["nvidia-smi"] just prints the GPU the container can see, then exits. Perfect for a smoke test.
resources.limits.nvidia.com/gpu: 1 is the important line. It tells the scheduler: "this pod needs one unit of the nvidia.com/gpu resource."

For extended resources like GPUs, you only set limits. Kubernetes automatically treats the request as equal to the limit, and you cannot overcommit them.

The catch: GPUs are integers

With CPU you could write cpu: 0.25. With GPUs you cannot write nvidia.com/gpu: 0.25. Extended resources must be whole numbers. So the smallest thing a pod can ask for is one entire GPU, and once it gets it, that GPU is exclusively bound to that pod until the pod dies. No other pod can be scheduled onto it, even if the first pod only ever uses 2% of its compute and 1 GB of its 80 GB of memory.

The default flow. The pod requests nvidia.com/gpu: 1, the scheduler finds a node with a free GPU, and the device plugin hands that whole card to the pod. With four lightly used pods holding four GPUs, a fifth pod stays Pending even though the node is almost idle.

That single design decision, whole-GPU allocation, is the root cause of almost every “why is my GPU at 5%” conversation.

Part 2: Why your GPU sits at 5%

Modern data-center GPUs are enormous. An NVIDIA A100 has 108 streaming multiprocessors (SMs) and 40 or 80 GB of memory; an H100 has even more. They were designed for workloads like training a large model, where thousands of matrix multiplications run back to back for hours.

Most workloads in a typical company cluster look nothing like that. Here are the usual suspects.

1. Online inference services

An inference API (say, a small text classifier or an embedding model behind a REST endpoint) only uses the GPU while a request is being processed. A forward pass might take 15 ms. If the service gets 5 requests per second, the GPU is busy for about 75 ms out of every 1,000 ms. That is roughly 7.5% utilization at best, and real traffic is far spikier than that: busy at 10 a.m., silent at 3 a.m.

An illustrative utilization trace of one inference pod that owns a whole GPU. Short bursts reach 60 to 90%, but the average over ten minutes is around 5%.

2. Jupyter notebooks and dev pods

A data scientist launches a notebook with a GPU attached, runs one cell that trains for 4 minutes, then spends 40 minutes reading results, writing code, and attending a meeting. The GPU is reserved the entire time. Multiply that by a team of 10, and you have 10 GPUs that are idle 90% of the day.

3. Small models on big cards

A 300 MB BERT-style model needs maybe 2 GB of GPU memory to serve. Placing it on an 80 GB H100 wastes 78 GB, and its small kernels cannot fill 132 SMs either. The model is not slow; the card is simply far bigger than the job.

4. CPU-bound pipelines

Many “GPU” jobs spend most of their time not on the GPU: decoding images, tokenizing text, reading from object storage, waiting on a database. The GPU waits for data, finishes its part in milliseconds, and waits again.

5. Batch jobs and CI tests

Nightly evaluation jobs and CI pipelines that “just check the model loads on a GPU” each need a GPU for a few minutes. Because allocation is exclusive, they queue behind each other or force you to buy more cards.

The real cost

Put all of this together and you get the classic pattern: allocation is 100% (every GPU is claimed by some pod, new pods are Pending), but utilization is 5 to 20%. The fix is not more GPUs. The fix is letting several workloads share one GPU safely. That is exactly what time-slicing and MIG do, in two very different ways.

Part 3: Measure before you fix

Before changing anything, prove the problem with numbers. You need two views: a quick spot check and a long-running metric you can graph.

Quick check with nvidia-smi

You can run nvidia-smi inside any pod that has a GPU:

kubectl exec -it  -- nvidia-smi --query-gpu=utilization.gpu,memory.used,memory.total --format=csv -l 2

What each part means:

kubectl exec -it -- runs a command inside the running container.
--query-gpu=... asks for specific fields instead of the big human-readable table.
utilization.gpu is the percentage of time in the last sample period during which at least one kernel was running on the GPU.
memory.used / memory.total shows how much GPU memory is actually allocated.
--format=csv -l 2 prints CSV and repeats every 2 seconds, so you can watch it for a while.

An important subtlety for beginners: utilization.gpu measures time, not space. If one tiny kernel uses only 1 of 108 SMs but runs continuously, nvidia-smi will still report close to 100%. So a low number is a reliable sign of waste, but a high number does not prove the GPU is fully used. We will use better metrics below.

Long-term monitoring with DCGM Exporter

For real decisions you want weeks of history in Prometheus. NVIDIA’s DCGM Exporter runs as a DaemonSet, reads metrics from the Data Center GPU Manager (DCGM), and exposes them on a /metrics endpoint that Prometheus scrapes. If you install the GPU Operator, it is enabled by default.

The metrics that matter most:

PromQL queries you can paste into Grafana

Average utilization per GPU over the last 7 days:

avg by (Hostname, gpu) (
  avg_over_time(DCGM_FI_DEV_GPU_UTIL[7d])
)

How it works: avg_over_time(...[7d]) takes every sample of the metric in the last 7 days and averages it, per time series. Each series is one GPU, so the result is "average utilization per GPU." The outer avg by (Hostname, gpu) collapses any extra labels (like pod name) so you get exactly one number per physical GPU.

Find GPUs that are allocated to a pod but nearly idle:

avg_over_time(DCGM_FI_DEV_GPU_UTIL{pod!=""}[24h]) < 10

How it works: DCGM Exporter attaches namespace, pod and container labels when the GPU is in use by a pod. pod!="" keeps only GPUs that are assigned. The < 10 filter returns only those that averaged below 10% over the last day. This list is your sharing candidates.

Memory headroom per GPU, as a percentage:

100 * DCGM_FI_DEV_FB_USED / (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE)

Why this matters: memory decides how many workloads can share a GPU. If a pod uses 4 GB on a 40 GB card, you can fit several neighbors. If it uses 35 GB, sharing will not help no matter how idle the compute is.

A good rule of thumb: if a GPU averages under 30% utilization and under 50% memory, it is a strong candidate for sharing.

Part 4: Fix #1 — Time-slicing

The idea in plain English

Time-slicing is the GPU equivalent of how your laptop’s CPU runs a browser, an editor and Spotify at the same time. There is one physical processor, and the system rapidly switches between programs, giving each a turn.

With time-slicing enabled, the NVIDIA device plugin lies to Kubernetes on purpose. Instead of advertising nvidia.com/gpu: 1 for one physical GPU, it advertises, say, nvidia.com/gpu: 4. Kubernetes now believes the node has 4 GPUs and schedules 4 pods there. All 4 pods get the same physical GPU, and the GPU's hardware scheduler interleaves their work in time slices.

Compute is shared by taking turns. Memory is not divided at all: all pods allocate from the same pool, and nothing stops one pod from taking more than its share.

Why it fixes the 5% problem

Remember the inference pod from Figure 2: busy for a few milliseconds, then idle. When four such pods share one card, the idle gaps of one pod get filled by the bursts of another. If each used about 5% on its own, together they use roughly 20%, on one GPU instead of four. You have just freed three GPUs.

Step 1: Install the GPU Operator

The easiest way to manage everything (driver, container toolkit, device plugin, DCGM exporter) is the NVIDIA GPU Operator, installed with Helm:

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator --create-namespace \
  --wait

What happens here: helm repo add registers NVIDIA's chart repository. helm install deploys the operator into a new gpu-operator namespace. The operator then watches for GPU nodes and installs the full NVIDIA software stack on each of them automatically. --wait makes Helm block until all resources are ready, so you know when it is safe to continue. (On managed services like EKS, GKE or AKS, the driver may already be installed; check your provider's docs and set --set driver.enabled=false if so.)

Step 2: Describe the sharing policy in a ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: time-slicing-config
  namespace: gpu-operator
data:
  any: |-
    version: v1
    flags:
      migStrategy: none
    sharing:
      timeSlicing:
        renameByDefault: false
        failRequestsGreaterThanOne: true
        resources:
          - name: nvidia.com/gpu
            replicas: 4

Explaining every field:

data.any is the name of a configuration profile. A single ConfigMap can hold several profiles (for example t4-x4 and a100-x2), and you pick which one each node uses. Here we create one profile called any.
version: v1 is the schema version of the device plugin config file.
flags.migStrategy: none tells the plugin that MIG is not in use on these GPUs.
sharing.timeSlicing.resources is the list of resources to oversubscribe. We take nvidia.com/gpu and set replicas: 4, meaning each physical GPU is advertised as 4 schedulable units. A node with 2 GPUs will now report nvidia.com/gpu: 8.
renameByDefault: false keeps the resource name as nvidia.com/gpu, so existing pod specs work unchanged. If you set it to true, the shared resource is advertised as nvidia.com/gpu.shared instead. That is useful when you want pods to explicitly opt in to sharing, so nobody lands on a shared GPU by accident.
failRequestsGreaterThanOne: true rejects pods that ask for more than one replica. This prevents a common misunderstanding: requesting nvidia.com/gpu: 2 on a time-sliced GPU does not give you twice the compute, it just gives you two turns on the same card. Failing loudly is better than silently giving someone less than they think.

Apply it:

kubectl apply -f time-slicing-config.yaml

Step 3: Point the device plugin at the config

The GPU Operator is configured through a custom resource called ClusterPolicy. We patch it so the device plugin loads our ConfigMap and uses the any profile by default:

kubectl patch clusterpolicies.nvidia.com/cluster-policy \
  -n gpu-operator --type merge \
  -p '{"spec": {"devicePlugin": {"config": {"name": "time-slicing-config", "default": "any"}}}}'

What this does: --type merge merges the JSON into the existing object instead of replacing it. config.name is the ConfigMap to read, and config.default is the profile applied to every node that does not say otherwise. The operator restarts the device plugin pods, which re-register with the kubelet using the new replica count.

If you want different behavior on different nodes (for example, share the T4 dev nodes but not the H100 training nodes), label individual nodes with the profile name instead of relying on the default:

kubectl label node gpu-dev-node-1 nvidia.com/device-plugin.config=any --overwrite

Step 4: Verify

kubectl get node gpu-dev-node-1 -o jsonpath='{.status.allocatable.nvidia\.com/gpu}'
# Output: 4   (for a node with 1 physical GPU)

kubectl get node gpu-dev-node-1 --show-labels | tr ',' '\n' | grep nvidia.com/gpu
# nvidia.com/gpu.product=Tesla-T4-SHARED
# nvidia.com/gpu.replicas=4

The node now reports 4 allocatable GPUs. The operator also updates node labels: gpu.replicas=4, and the product name gets a -SHARED suffix so you can target (or avoid) shared nodes with a nodeSelector.

Step 5: Run four pods on one GPU

apiVersion: apps/v1
kind: Deployment
metadata:
  name: embedder
spec:
  replicas: 4
  selector:
    matchLabels:
      app: embedder
  template:
    metadata:
      labels:
        app: embedder
    spec:
      nodeSelector:
        nvidia.com/gpu.replicas: "4"
      containers:
        - name: model
          image: my-registry/embedder:1.0
          resources:
            limits:
              nvidia.com/gpu: 1
          env:
            - name: PYTORCH_CUDA_ALLOC_CONF
              value: "expandable_segments:True"

Why it is written this way:

replicas: 4 with nvidia.com/gpu: 1 each consumes exactly the 4 advertised slices. On a single-GPU node, all four pods land on the same physical card.
nodeSelector on nvidia.com/gpu.replicas: "4" makes sure this Deployment only goes to nodes where sharing is configured, never onto an exclusive training GPU.
The PYTORCH_CUDA_ALLOC_CONF setting reduces memory fragmentation in PyTorch's caching allocator. On a shared GPU, wasted memory hurts every neighbor, so small memory hygiene settings matter more than usual.

If you kubectl exec into each pod and run nvidia-smi, all four will show the same GPU UUID. That is time-slicing working.

Time-slicing caveats (read these before production)

No memory isolation. All pods share one memory pool. If one pod loads a bigger model or leaks memory, the others hit CUDA out of memory. Each application should cap its own usage, for example torch.cuda.set_per_process_memory_fraction(0.25) in PyTorch, or the memory-fraction options most inference servers provide.
No fault isolation. A serious GPU error (an Xid fault) caused by one pod can affect every process on that GPU.
No performance guarantee. If all four pods get traffic at the same moment, each gets roughly a quarter of the GPU, plus some context-switching overhead. Latency becomes unpredictable under load.
The replica count is not a quota. Kubernetes thinks each pod has “one GPU.” Nothing enforces that a pod uses only a quarter. It is a scheduling hint, not a resource limit.

Time-slicing is perfect for dev notebooks, CI jobs, demos, and low-traffic internal inference, where everyone on the GPU belongs to the same team. When you need real walls between workloads, you need MIG.

Part 5: Fix #2 — MIG (Multi-Instance GPU)

The idea in plain English

If time-slicing is roommates sharing one kitchen in shifts, MIG is converting a house into separate apartments, each with its own kitchen, walls and front door.

MIG is a hardware feature introduced with NVIDIA’s Ampere generation. It physically partitions one GPU into up to 7 independent GPU instances. Each instance gets:

its own dedicated streaming multiprocessors (compute),
its own dedicated slice of GPU memory,
its own share of L2 cache and memory bandwidth,
its own fault domain: an error in one instance does not crash the others.

To the software running inside, a MIG instance looks and behaves like a small, standalone GPU. To Kubernetes, each instance is a separate schedulable resource.

Which GPUs support MIG?

MIG is available on data-center GPUs from Ampere onward: A100 and A30 (A30 supports up to 4 instances), H100, H200, and the Blackwell data-center line. Popular inference cards like the T4, L4 and A10 do not support MIG. For those, time-slicing (or MPS, covered later) is your option. Always confirm on NVIDIA’s MIG user guide for your exact model.

Understanding MIG profiles

You cannot cut a GPU into arbitrary sizes. NVIDIA defines fixed profiles. On an A100 40GB they look like this:

The naming is easy once you see the pattern: g.gb. On an H100 80GB the same pattern applies with doubled memory, for example 1g.10gb, 3g.40gb and 7g.80gb.

Three ways to carve the same A100. Layout A gives seven small, equal instances (great for many small inference models). Layout B mixes sizes for teams with different needs. Layout C gives two medium instances, and one slice is left unused because the hardware placement rules do not allow a third 3g instance.

Step 1: Choose a MIG strategy in the GPU Operator

The device plugin can expose MIG instances to Kubernetes in two ways:

single: every GPU on the node is split into the same profile, and instances are advertised with the familiar name nvidia.com/gpu. Existing pod specs keep working, but pods cannot choose a size.
mixed: each profile gets its own resource name, like nvidia.com/mig-1g.5gb and nvidia.com/mig-3g.20gb. Pods request exactly the size they need. This is what most teams want.

helm upgrade --install gpu-operator nvidia/gpu-operator \
  -n gpu-operator --create-namespace \
  --set mig.strategy=mixed \
  --wait

Step 2: Tell the node how to partition itself

The GPU Operator includes a MIG Manager that watches a node label and reconfigures the GPUs to match. You never have to run low-level commands by hand:

kubectl label node gpu-a100-node-1 nvidia.com/mig.config=all-1g.5gb --overwrite

What happens behind the scenes:

MIG Manager sees the label change.
It evicts GPU pods from the node (the GPU must be idle to repartition).
It enables MIG mode on each GPU and creates seven 1g.5gb instances.
It restarts the device plugin, which now advertises nvidia.com/mig-1g.5gb: 7 per GPU.
It sets the label nvidia.com/mig.config.state=success when done.

Because step 2 evicts workloads, treat a MIG reconfiguration like a node maintenance event: cordon and drain first in production.

Other built-in layouts include all-2g.10gb, all-3g.20gb, and all-balanced (a mix of sizes). You can also define your own layouts in a custom ConfigMap for MIG Manager, for example two 1g.5gb plus one 2g.10gb plus one 3g.20gb.

Watch the reconfiguration complete and check the new resources:

kubectl get node gpu-a100-node-1 -o jsonpath='{.metadata.labels.nvidia\.com/mig\.config\.state}'
# success

kubectl get node gpu-a100-node-1 -o json | jq '.status.allocatable | with_entries(select(.key | startswith("nvidia.com")))'
# {
#   "nvidia.com/mig-1g.5gb": "7"
# }

What the second command does: it fetches the node as JSON, and jq filters the allocatable map down to only keys starting with nvidia.com. Instead of nvidia.com/gpu: 1, you now see seven schedulable MIG instances.

What MIG Manager does under the hood (for the curious)

If you SSH into a GPU node, you can do the same thing manually with nvidia-smi. Understanding these commands helps when you are debugging:

# 1. Enable MIG mode on GPU 0 (requires the GPU to be idle; may need a reset)
sudo nvidia-smi -i 0 -mig 1

# 2. List the profiles this GPU supports, with their numeric IDs
sudo nvidia-smi mig -lgip

# 3. Create three 1g.5gb GPU instances (profile ID 19 on A100 40GB)
#    and a default compute instance inside each (-C)
sudo nvidia-smi mig -cgi 19,19,19 -C

# 4. List the GPU instances you created
sudo nvidia-smi mig -lgi

Key concepts in these commands:

A GPU instance (GI) is the hardware partition: memory slice, cache and SM group.
A compute instance (CI) lives inside a GI and is what processes actually run on. The -C flag creates one default CI per GI, which is what you want 99% of the time.
Profile IDs (like 19) differ between GPU models, which is why -lgip exists. Always list first, then create.

In Kubernetes, prefer the label-based MIG Manager over manual commands. Manual changes are not tracked declaratively and can be overwritten.

Step 3: Request a MIG slice from a pod

apiVersion: apps/v1
kind: Deployment
metadata:
  name: fraud-scorer
spec:
  replicas: 7
  selector:
    matchLabels:
      app: fraud-scorer
  template:
    metadata:
      labels:
        app: fraud-scorer
    spec:
      containers:
        - name: model
          image: my-registry/fraud-scorer:2.3
          resources:
            limits:
              nvidia.com/mig-1g.5gb: 1

How to read this: the only difference from a normal GPU pod is the resource name. nvidia.com/mig-1g.5gb: 1 means "give me one 1g.5gb instance." With replicas: 7, one A100 now serves seven fully isolated copies of the model, each with a guaranteed 5 GB of memory and 1/7 of the compute. Inside each container, nvidia-smi -L shows a device with a MIG- prefixed UUID.

Why MIG isolation matters

With time-slicing, one misbehaving pod can exhaust memory for everyone. With MIG, each pod is boxed into its own memory and compute, so a failure stays contained.

MIG caveats

Fixed sizes. You only get the predefined profiles. If your model needs 7 GB, a 1g.5gb slice is too small and you jump to 2g.10gb, wasting 3 GB.
Repartitioning is disruptive. Changing layouts evicts pods from the GPU. Plan layouts per node pool rather than changing them often.
Compute does not “burst.” A 1g instance can never borrow idle SMs from a neighbor. For very spiky workloads, that can be less efficient than time-slicing.
Hardware cost. Only higher-end data-center GPUs support it.

Combining MIG with time-slicing

You can stack the two techniques: partition an A100 with MIG, then time-slice each MIG instance. For example, 7 1g.5gb instances with replicas: 2 gives you 14 schedulable units. Teams use this for dev environments: hard isolation between teams (MIG), soft sharing within a team (time-slicing). In the time-slicing ConfigMap, you list the MIG resource name (such as nvidia.com/mig-1g.5gb) instead of nvidia.com/gpu, and set migStrategy: mixed.

Part 6: Time-slicing vs MIG — how to choose

Neither technique is “better.” They solve the same problem with different trade-offs.

A practical decision path. Start with “is the GPU actually busy?” because the cheapest optimization is the one you do not need.

A third option worth knowing: MPS

NVIDIA’s Multi-Process Service (MPS) sits between the two. Like time-slicing, it works in software and on most GPUs. Unlike time-slicing, it lets kernels from different processes run at the same time (concurrently) rather than taking turns, and it can cap each client’s memory and share of SMs. Recent versions of the NVIDIA device plugin support MPS as a sharing mode, configured almost exactly like the time-slicing ConfigMap (sharing.mps instead of sharing.timeSlicing). It still lacks full fault isolation, so treat it as "better time-slicing for trusted workloads" rather than a MIG replacement.

And on the horizon: Dynamic Resource Allocation (DRA)

The integer nvidia.com/gpu model is a decade-old Kubernetes limitation. The newer Dynamic Resource Allocation (DRA) API lets drivers describe devices richly (memory size, MIG profile, sharing mode) and lets pods ask for them with structured claims instead of a single number. NVIDIA maintains a DRA driver for GPUs. If you are designing a new platform, it is worth tracking, but time-slicing and MIG through the device plugin remain the most widely deployed approach today.

Part 7: Production checklist and common pitfalls

A rollout plan that works

Measure for at least a week with DCGM Exporter. Identify GPUs under 30% utilization and under 50% memory.
Split node pools by purpose. For example: gpu-train (exclusive, no sharing), gpu-infer (MIG), gpu-dev (time-slicing). Use node labels and taints so workloads land in the right pool.
Start with time-slicing on dev nodes. It is low-risk, reversible by editing a ConfigMap, and gives the fastest visible win.
Move production inference to MIG where your hardware supports it and you need predictable latency.
Re-measure. Compare utilization, pod Pending time, and GPU count before and after.

Taint shared nodes so only opted-in pods land there

kubectl taint nodes gpu-dev-node-1 gpu-sharing=time-sliced:NoSchedule

Pods that are happy to share add a matching toleration:

tolerations:
  - key: "gpu-sharing"
    operator: "Equal"
    value: "time-sliced"
    effect: "NoSchedule"

Why this matters: a taint is a “keep out” sign on a node. Only pods with a matching toleration can be scheduled there. This prevents a latency-sensitive production service from accidentally being placed on a crowded, time-sliced GPU.

Cap GPU memory inside your application (time-slicing)

Since time-slicing does not enforce memory limits, each workload should limit itself. In PyTorch:

import os
import torch

# Fraction of the GPU this process may use, e.g. 0.25 for 4 replicas.
fraction = float(os.getenv("GPU_MEMORY_FRACTION", "0.25"))
torch.cuda.set_per_process_memory_fraction(fraction, device=0)
model = torch.load("model.pt", map_location="cuda:0")

What this does: set_per_process_memory_fraction tells PyTorch's memory allocator never to grab more than that fraction of total GPU memory. If the code tries, it raises an out-of-memory error in this pod only, instead of silently starving its neighbors. Reading the value from an environment variable means you can tune it per Deployment without rebuilding the image. TensorFlow, vLLM and Triton Inference Server have similar settings.

Common pitfalls

Expecting nvidia.com/gpu: 2 to mean "more GPU" on a time-sliced node. It does not. Use failRequestsGreaterThanOne: true to catch this early.
Changing MIG layouts on a busy node. MIG Manager evicts GPU pods. Drain first.
Mixing training and inference on shared GPUs. One training job can saturate a GPU and wreck inference latency for everyone else. Keep training on exclusive GPUs.
Trusting GPU-Util alone. It measures time, not how much of the chip is used. Look at DCGM_FI_PROF_SM_ACTIVE and memory too.
Forgetting about quotas. With 4x time-slicing, a namespace quota of requests.nvidia.com/gpu: 4 now means one physical GPU, not four. Revisit your ResourceQuotas after enabling sharing.

Conclusion

Your GPUs sit at 5% utilization not because something is broken, but because Kubernetes, by default, gives every pod an entire GPU, and most workloads need only a sliver of one, for a sliver of the time.

Time-slicing fixes this in minutes on almost any NVIDIA GPU by letting pods take turns. It is ideal for trusted, bursty workloads, but offers no memory or fault isolation.
MIG fixes it with real hardware partitions on A100, H100 and newer, giving each pod guaranteed memory and compute. It is ideal for multi-tenant production inference, at the cost of fixed sizes and disruptive reconfiguration.
You can combine both, and you should measure before and after with DCGM Exporter.

Start small: pick your idlest dev GPU node, enable time-slicing with 4 replicas, and watch four pods share a card that used to serve one. The results in your Grafana dashboard, and in your cloud bill, tend to make the case for the rest of the rollout on their own.

Stories by Vasanthan K on Medium

SvelteKit 3 Migration Notes: Node 22, Vite 8 and What Actually Breaks

TL;DR

1. Why this is three upgrades, not one

2. The version matrix: what you need before you start

3. Layer 1 — Node 22 and the npm install trap

What a runtime floor really means

The trap: Node 22’s bundled npm

How to fix it

Two more install-time surprises

4. Layer 2 — Vite 8, Rolldown and Oxc

Bundler basics in 60 seconds

The minimum upgrade

Renamed config keys

The quiet breakage: CommonJS default imports

Small side effects worth knowing

5. Layer 3 — SvelteKit 3’s breaking changes, explained

5.1 Configuration moves into vite.config.ts

5.2 $lib becomes #lib

5.3 $app/stores is removed — use $app/state

5.4 $app/environment is now $app/env

5.5 Environment variables are declared in src/env.ts

5.6 Links: base is gone, use resolve() and asset()

5.7 Navigation: pushState folds into goto

5.8 Errors: simpler error(), louder handleError

5.9 Smaller changes that still bite

6. Running sv migrate, step by step

Step 1 — Start from a clean, committed branch

Step 2 — Land on the latest SvelteKit 2 and fix warnings

Step 3 — Get the runtime right

Step 4 — Preview, then run, the migration

Step 5 — Work through MIGRATION_TASKS.md

Steps 6–8 — Verify three different ways

7. What actually breaks: the gotchas a green build hides

Why codemods miss things: script vs markup

Gotcha 2 — {base} inside a quoted href

Gotcha 3 — param matchers lose their types

Gotcha 4 — everything from section 3 and 4

8. Verifying the migration: check, build, then use it

Check 1 — npm run check asks "do my types agree?"

Check 2 — npm run build asks "does it bundle under Vite 8?"

Check 3 — a smoke test asks “does it actually work in a browser?”

Bonus check — reproduce a fresh install

9. The copy-paste checklist

Wrapping up

References

Node.js 26 LTS: Map Upsert, Iterator.concat and Other Small Features That Delete Utility Code

First, what does “Node.js 26 LTS” actually mean?

Which engine features did we get?

How to follow along

Part 1 — Map upsert: “get it, or create it” in one call

A 30-second refresher on Map

The problem: the has / set / get dance

Meet getOrInsert(key, defaultValue)

Meet getOrInsertComputed(key, callback)

The theory: why are there two methods?

Real-world recipes

What about Map.groupBy?

The WeakMap versions

Small print

Part 2 — Iterator.concat(): one lazy stream from many sources

Iterables and iterators, in plain words

The problem: joining several sources

Meet Iterator.concat()

The theory: what “lazy” really means here

Combining it with iterator helpers

More everyday uses

Rules and gotchas

Part 3 — Other small features that delete utility code

Temporal is on by default

Base64 and Hex built into Uint8Array

Part 4 — Before you upgrade: what can break

Cheat sheet: what to delete from your utils folder

Wrapping up

Sources

Building the Memory Layer of My Agent Harness: What I Learned Wiring Persistent Context by Hand

Why an agent needs memory at all

The context window is RAM, not a hard disk

Why it is harder in a multi-agent harness

The mental model: four kinds of memory