Taalas dropped out of stealth on February 19 with a chip that runs Llama 3.1 8B at 17,000 tokens per second. The weights of the model are hardwired into silicon. No High Bandwidth Memory (HBM), no memory bus. Single 815mm² die, TSMC 6nm, 250 watts. 10 times faster than anything else at a fraction of the power.

The catch: the chip is the model. Need a new model? You need a new chip. Two-month fabrication turnaround.

So how do you compare this against a GPU? Tokens per second doesn’t cut it. Cost per token doesn’t either. When one architecture uses 3-bit quantization and another runs FP16, you need a unit that captures speed, power, and whether the outputs are actually right.

This is my attempt at that framework. Five relationships, sourced power specs, and cross-platform comparisons. The framework is not perfect. But the core finding holds: current inference hardware operates 10¹⁰ times above the thermodynamic floor, and the biggest energy tax in the stack is the one compute-in-memory eliminates.

To put that in perspective: ENIAC, the first programmable general purpose computer in 1945, consumed ~150,000 watts to do a few thousand operations per second. A modern CPU does ~10¹⁰ operations per second per watt. That’s roughly the same gap. Current inference hardware is at the vacuum tube stage of energy efficiency.

To be absolutely transparent, I’m not a hardware specialist and used Claude heavily to explore, analyze and write about this concept. The results are outlined below.

The Inference Unit That Matters

A single token isn’t true or false. “The capital of France is” → “Paris” is four tokens where truth belongs to the completed inference, not any individual token.

So the base unit is joules per useful inference. A complete generation that actually answers the question correctly. Everything derives from that.

Three variables feed it: speed (tokens per second), power (watts), and accuracy (what fraction of inferences are useful). One more variable handles scale: how many concurrent inference streams the hardware supports.

The Five Equations

Variable List

The Equations

Equation I — Inference Energy (definition)

This is just dimensional analysis, moving variables around. Power divided by the rate of useful inference production. `N` inference streams, each producing `S/T` inferences per second, fraction `A` correct. Everything else builds off this.

Equation II — Speed Limit

Max throughput = compute density × die area, divided by operations per token. `2PQ` approximates the multiply-accumulate workload per token.

This holds for decode-dominated, short-context inference where weight multiplies dominate. It ignores attention (which scales `O(T·d)` per token) and treats `κ` as constant across architectures. Back-of-envelope, not gospel.

Equation III — Landauer Energy Floor

The thermodynamic minimum. Every irreversible bit operation costs at least:

An 8 billion parameter model at 3-bit Q generates 200 tokens: that’s about 57 nanojoules. Same `2PQ` approximation as above.

Interesting comparison for you: 160 nJ is the energy of a flying mosquito.

Equation IV — Density-Power Tradeoff

Power = total ops/sec × energy per op. `ε(D)` increases with die area. The longer wires on the chip, the more capacitance you have, the more clock distribution overhead you have. There’s no free lunch, even in AI.

Equation V — Accuracy-Capacity Bound

Accuracy bounded by model capacity (`P × Q`) and reasoning budget (`T`).

This is the weakest part of the framework. Real scaling laws describe loss as a power law, not accuracy as a log function. “Accuracy” itself is ill-defined as a single scalar: MMLU, HumanEval, and real-world task completion are different numbers. The coefficients `α` and `β` are hand-calibrated to two data points.

What matters is that accuracy generally improves sublinearly with model size. There are diminishing returns. The exact functional form matters less than what falls out of it (next section).

What Falls Out

Efficiency Ratio

Combining Relationships I and III yields η:

How far above the thermodynamic minimum a given system is. This is not Carnot efficiency — there’s no reversible cycle argument here. It’s just a useful ratio.

The 10 ¹¹ gap represents an enormous engineering opportunity. Even closing a fraction of this gap yields transformative cost reductions.

Thermodynamically Optimal Model Size

Relationship V, despite its imprecision, creates an important qualitative tension with Relationship III. Any sublinear accuracy curve + linear energy cost = a peak in accuracy-per-joule at some finite model size. You can explore that tension with:

The framework predicts the peak exists and shifts right with better hardware. Where exactly it falls depends on Relationship V’s calibration, which I already flagged as rough. The qualitative insight: there is an optimum and better hardware pushes that optimum toward larger models.

Better hardware doesn’t just make the same model cheaper. It makes bigger models the rational choice.

The Energy Ladder

The biggest gap is between “No Data Movement” and GPU baselines. That’s the memory wall tax . That’s energy spent shuttling weights from HBM to compute dwarfing the consumption of everything else.

Taalas eliminates this energy tax entirely by putting weights in mask ROM. The remaining gap between Taalas and the theoretical “no data movement” level is system overhead: clock distribution, voltage regulation, I/O, host CPU, power supply losses. You get that loss in any computational system.

Platform Analysis

Data Sources and Methodology

Power specs from official docs, datasheets, and verified reporting. Throughput from Artificial Analysis and manufacturer claims. All Llama 3.1 8B, single-user output generation.

Table 1. Data sources for platform specifications.

Methodology Notes

Groq: Each GroqChip has 230MB SRAM. Llama 8B at INT8 is ~8GB. That’s ~35 cards. Per-card power is 275W, but system power for one model instance is ~9.6kW. Charts use per-card for silicon-level comparison, but the real system cost is much higher. Groq self-reports 1–3 J/tok for their cloud service which likely reflects heavy batching.

Cerebras: CS-3 is 23kW for a system designed to run 70B–1T+ models. Using 23kW for an 8B model comparison is like measuring a semi truck’s fuel efficiency delivering a single envelope. Unfair to Cerebras. Included for completeness, flagged here.

Taalas: The 250W per-card figure and 17K tok/s are both first-party claims verified by EE Times reporting. The entire 8B model resides on a single 815mm² chip (TSMC 6nm, 53B transistors). The “per user” qualification suggests single-stream operation. 10 cards in a 2.5kW server provide 10 concurrent user streams.

Cross-Platform Comparison

Table 2. Cross-platform comparison. Groq uses per-card power (275W); system power for full 8B model is ~9.6kW across ~35 cards. Cerebras uses full 23kW system power. See methodology notes.

Joules per Token — Alternative Metric

Joules per token provides a simpler metric that avoids assumptions about accuracy and inference length. Taalas achieves 0.015 J/tok which is dramatically lower than all competitors. Groq self-reports 1–3 J/tok for their cloud service, while NVIDIA H100-class systems typically consume 10–30 J/tok at system level.

The Real Trade Off

Taalas 8B sits in the bottom-left: cheap but lower accuracy from 3-bit quantization. B200 running 405B is top-right: high accuracy, high energy. The frontier connecting them is the real tradeoff curve — and everything off that frontier is leaving performance on the table.

The more interesting question: what happens to that frontier when hardware gets better?

Each hardware generation slides the whole curve left. GPU 405B at ~20 kJ. Taalas-class 405B at ~1.5 kJ. Future CIM at ~0.05 kJ. Same model, same accuracy (roughly), orders of magnitude less energy.

But here’s the punchline: on better hardware, the slope of the curve also changes. The marginal energy cost of jumping from 70B to 405B shrinks dramatically. On GPUs, going from 70B to 405B costs ~20 times more energy for ~10% more accuracy. On projected future CIM hardware, that same jump costs ~3 times more energy. Bigger models become rational faster on better hardware.

Hardware and model scaling aren’t independent axes. They’re coupled through energy.

Where This Breaks Down

The Taalas numbers are real for 8B models. But there’s a ceiling, and it’s physical.

An 8B model at 3-bit quantization is ~3GB of weight data. Fits on one 815mm² die — which is already pushing the reticle limit (the maximum area a lithography stepper can expose in one shot). There’s no headroom to grow on a single chip.

Scale up and the math shows how this process begins to fall apart:

The whole advantage , no data movement, degrades as you cross chip boundaries. A 70B model split across 16 dies reintroduces the interconnect penalty that Taalas exists to eliminate. It’s smaller than GPU-to-HBM bandwidth (chip-to-chip on a module is faster than off-package memory), but it’s not zero. The single-die purity is gone.

Other physical constraints:

• Yield: Larger dies have exponentially worse yield. A single defect kills the chip. Mask ROM weights can’t be remapped around defects the way DRAM uses redundant rows. Either they’ve solved this or their yield rate is a meaningful cost factor they haven’t disclosed.
• Model lock-in: Taalas CEO Ljubisa Bajic told EE Times customers should expect a one-year commitment per chip. If a better model drops six months in, your silicon is obsolete. The two-month turnaround helps but doesn’t eliminate the risk.
• Quantization quality: Gen 1 uses custom 3 or 6-bit. Gen 2 moves to standard FP4. But even FP4 has measurable quality loss on reasoning-heavy benchmarks. The accuracy axis in our framework isn’t free.

The strong counterargument is that the models worth deploying keep getting smaller. Distillation, mixture-of-experts (where individual experts are small), and better quantization methods (BitNet, QuIP#) all push useful capability into the single-die range. If a 2-bit quantized 70B model is ~17GB, that’s back to feasible on a few dies. If mixture-of-experts means you only activate 8B of parameters per token, you’re back in the sweet spot entirely.

Taalas is betting that production inference workloads converge on the 8B–70B range at aggressive quantization. I don’t think that’s a crazy bet.

Path to Theoretical Limits

On GPUs, 95% of the energy moves data. It’s not computing anything — just moves. Taalas eliminates that layer entirely, which is why it’s 20× more efficient despite running on an older process node. But look at what’s left: system overhead (yellow) is now almost the entire bar. Clocks, voltage regulation, I/O, power supply losses. That’s the next frontier.

~33 doublings between here and Landauer. Just doubling 10 of them and inference energy becomes trivial. We’re still in the ENIAC era of AI.

Implications

• Memory wall is the biggest tax. The dominant gap in the ladder. Compute-in-memory eliminates it. Taalas is the extreme case — mask ROM weights — but the principle applies to any architecture that keeps weights close to compute.
• Quantization is a thermodynamic lever. Lower `Q` improves speed (II), floor (III), and power (IV) simultaneously, at modest accuracy cost (V). 1-bit methods (BitNet) could move the whole framework an order of magnitude.
• System power swamps chip power. Groq at 275W/card looks great until you need 35 cards. Cerebras at 23kW looks terrible until you remember it’s built for 70B+ models. Always ask: power of *what*?
• `η` as a universal benchmark. Ratio of actual to Landauer energy. Architecture-independent. Track it over time the way we track transistor density. It’s just a ratio, not a deep theorem — but it’s the right ratio.

Conclusion

Five relationships. One framework. Current hardware is 10¹⁰ to 10¹¹ times above the theoretical floor. The biggest tax is data movement, and it’s eliminable. An optimal model size exists for every hardware efficiency level, and it shifts right as hardware improves. Model compression and hardware specialization are convergent, not competing.

Sources

1. Taalas. “The path to ubiquitous AI.” taalas.com, Feb 2026.
2. EE Times. “Taalas Specializes to Extremes for Extraordinary Token Speed.” Feb 2026.
3. The Next Platform. “Taalas Etches AI Models Onto Transistors.” Feb 2026.
4. Kaitchup. “Taalas HC1: Absurdly Fast Per-User Inference.” Substack, Mar 2026.
5. NVIDIA. “H200 Tensor Core GPU.” nvidia.com.
6. WCCFTech. “NVIDIA Blackwell B200 GPUs Consume Up To 1200W.” Mar 2024.
7. Groq. “Inside the LPU: Deconstructing Groq Speed.” groq.com.
8. Mouser Electronics. GroqCard product listing. TDP: 275W.
9. Cerebras. “CS-3 vs NVIDIA B200.” cerebras.ai, Mar 2025.
10. Artificial Analysis. Llama 3.1 8B provider benchmarks. artificialanalysis.ai.
11. Zheng Li / Medium. “Groq’s Deterministic Architecture is Rewriting the Physics of AI Inference.” Dec 2025.
12. Landauer, R. [“Irreversibility and Heat Generation in the Computing Process.” IBM J. Res. Dev., 1961.

Developed through collaborative exploration with Claude (Anthropic), building from first-principles analysis of the Taalas HC1 announcement. Calibration constants (α, β, ε) are approximate and should be refined with empirical measurement. Power figures sourced from official documentation and verified reporting as of March 2026.

Introduction

I wanted to capture some part of what it’s like to really play tabletop RPG games with a human GM. Random luck, creative outcomes, odd combinations all make table-top RPGs a special experience that is very difficult to emulate in video games.

What if an AI could be your dungeon master, but you still roll physical dice?

Just want to play? Start here (then come back to learn!) https://north-pole-quest.vercel.app

I’ve had this idea for an AI-based RPG (role playing game) for a while now. Some of my previous attempts worked well enough but the game master (GM) would commonly skip restrictions or make things up.

So for this version I had a few key requirements in mind as I started working on this with Claude:

1. The AI GM must control the flow of the game without any hard coded game mechanics
2. All game mechanics had to be exposed as tools that the GM could use
3. The game should play like a real game, not just a technical demo, so inputs should be randomized using real cards, dice, or their digital analogs (random number generators)

To build this, I’ve used a few key technologies:

• LangChain and LangGraph for the AI/GM game loop
• Fireworks.ai for LLM inference and image generation
• LangSmith Deploy for deploying the AI agent infrastructure
• Vercel for deploying the customized Agent Chat UI interface

Obviously, most AI isn’t being deployed as a holiday game but I built this as a way to explore these newer concepts in the AI world. The learnings here are directly applicable to real AI deployments. Enjoy!

The Concept: Bridging Digital and Physical

Solo RPGs traditionally require lookup tables or complex oracles. While those are part of the charm for real table-top games, it can introduce disengaging friction into the gameplay loop. So a digital lookup capability encoded into the game speeds this.

Even with the game skeleton digitized, engaging narrative can be hard to craft for each possible scenario that a player and a GM could encounter together. Generative AI really shines here. A real human GM can spend hours creating the world for their players, but that still won’t exhaustively cover all the possible rules, items, and scenarios in the game.

Asking the LLM to “roll dice” or “draw cards” isn’t realistic and is also very disengaging for a human player. In a few test games I played with Claude before starting this project, I found that if I actually physically rolled the dice and drew the cards, the game was much more engaging for me.

This introduces a complication for an AI-based game — how do you properly block AI continuation and await human input (the result of dice rolls or card draws)? The interrupt pattern exposed through LangGraph does this for us nicely. It pauses AI execution waiting for player input.

Architecture Overview

I think people tend to overcomplicate the architecture of AI applications. At a 10,000 foot level, these applications are nothing more than a standard web service with persistent storage and an interactive user interface.

That’s exactly what the North Pole Quest is:

┌─────────────────────────────────────────────────────────────────┐
│                         Frontend (UI)                           │
│   - Displays narration, images, game status                     │
│   - Handles physical dice rolls and card draws                  │
│   - Manages inventory UI                                        │
└─────────────────────────┬───────────────────────────────────────┘
                          │ interrupt/resume
┌─────────────────────────▼───────────────────────────────────────┐
│                    LangGraph Agent                              │
│  ┌─────────────┐  ┌──────────────┐  ┌────────────────────────┐  │
│  │    LLM      │  │  Middleware  │  │        Tools           │  │
│  │ (Fireworks) │◄─┤  - Status    │◄─┤  - take_action         │  │
│  │             │  │  - Summary   │  │  - discover_new_region │  │
│  │  GameMaster │  │  - Limits    │  │  - check_for_gift      │  │
│  └─────────────┘  └──────────────┘  │  - search_for_items    │  │
│                                     │  - hot_cocoa_break     │  │
│                                     │  - inventory tools     │  │
│                                     │  - generate_scene_image│  │
│                                     └────────────────────────┘  │
└─────────────────────────┬───────────────────────────────────────┘
                          │
┌─────────────────────────▼───────────────────────────────────────┐
│                       GameState                                 │
│  sleepiness, gifts_found, current_location, inventory,          │
│  turns_remaining, region_search_counts, latest_image, etc.      │
└─────────────────────────────────────────────────────────────────┘

The front end is how the users experience the game as a chat-based interface. It’s also where we emulate the dice rolling and card drawing.

The AI agent exists at the webserver level, responding to user input from the front end, performing basic actions and executing the tools it has access to. Critically, by separating game logic out into tools, we can apply fine-grained controls and permissions to these tools (if/else scenarios) which allows the AI agent to focus on GM-specific tasks, not on managing how many candy canes you have in your inventory.

Finally, the state of each individual game is stored in an independent Agent Run in LangSmith Deploy. Each run keeps its details logically separate, ensuring there’s no crossover from another game being played by someone else in parallel. It also lets the player leave, come back, and pick up where they left off.

The Interrupt Pattern Deep Dive

The interrupt pattern is the key part of the game that prevents the GM from hallucinating tool result calls. Let’s walk through how it works.

After each user input (or game start):

1. LangGraph Interrupts: The backend agent pauses and sends a GameInputRequest when it needs dice/card input
2. Frontend Detection: Rendering components check stream.interrupt for game input requests
3. Interactive Components: DiceRoller (animated 10-iteration roll) and CardDrawer (uses persistent deck in local browser storage)
4. Resume Flow: User action triggers stream.submit() with a command.resume payload containing dice values/cards plus current game state

So to discover a new region, #1 and #4 is from above is where the bulk of the work happens.

The user asks the GM to discover a new region:

The agent requests a dice roll below:

def request_dice(num_dice: int, reason: str) -> dict:
    """Request dice roll from user via interrupt.

    Returns dict with dice_values and dice_total from user input.
    """
    response = interrupt({
        "type": "game_input",
        "request": "dice",
        "num_dice": num_dice,
        "reason": reason
    })
    return normalize_response(response)

The frontend catches this request and routes it to the appropriate input type (dice in this case)

interface GameInputRequest {
  type: "game_input";
  request: "dice" | "card" | "inventory_state" | "region_state" | "state_check";
  num_dice?: number;
  num_cards?: number;
  reason?: string;
}

The result is captured and returned to the backend (discovering a new region). Basic checks are performed on state (is the player too sleepy to discover a region?), matching the region to the provided roll from the frontend. The matched region is returned by the backend.

@tool
def discover_new_region(
    tool_call_id: Annotated[str, InjectedToolCallId] = ""
) -> Command:
    """Explore a new magical location by rolling 2d6 to determine where you end up.
    The player rolls the dice! Automatically adds the location to explored places.
    Costs 1 turn on the Christmas Clock."""
    dice_response = request_dice(2, "Roll 2d6 to discover a new location!")

    # Check fatigue before allowing action
    is_too_sleepy, fatigue_msg, _ = check_fatigue(dice_response)
    if is_too_sleepy:
        return Command(
            update={
                "last_dice": None,
                "last_card": None,
                "messages": [{"role": "tool", "content": fatigue_msg, "tool_call_id": tool_call_id}]
            }
        )

    # Check time remaining (costs 1 turn)
    is_game_over, ending_msg, turns_after, _ = check_time_remaining(dice_response, turn_cost=1)
    if is_game_over:
        return Command(
            update={
                "turns_remaining": 0,
                "last_dice": None,
                "last_card": None,
                "messages": [{"role": "tool", "content": ending_msg, "tool_call_id": tool_call_id}]
            }
        )

    dice = dice_response.get("dice_values", [1, 1])

    key = f"{dice[0]}{dice[1]}"
    location = REGIONS.get(key, "Mysterious Snowy Place")

    message = f"🗺️ Rolled {dice[0]}, {dice[1]} - You found: **{location}**! ✨\n💡 Explore or take an action here before searching for a Magic Gift!\n⏰ Turns remaining: {turns_after}"

    return Command(
        update={
            "current_location": location,
            "discovered_regions": [location],
            "turns_remaining": turns_after,
            "last_dice": None,
            "last_card": None,
            "region_action_taken": False,
            "messages": [{"role": "tool", "content": message, "tool_call_id": tool_call_id}]
        }
    )

The new region is returned, the GM detects this and generates a new image accordingly, and the player is prompted for their next action.

Managing Game State

The challenge with any AI-powered capability is keeping LLM informed without polluting message history. Since LLMs have limited context (though context size has gotten much larger recently), it’s important to keep critical and controlling aspects of the game outside of the LLM’s context.

We manage this by leveraging a LangGraph middleware that injects the current GameStatus into the system prompt given to the Agent for each game loop.

class GameState(MessagesState):
    sleepiness: int = 0
    gifts_found: int = 0
    total_gifts: int = 6  # Total gifts needed to win
    current_location: str = ""
    discovered_regions: Annotated[List[str], add] = []
    # Store latest generated image separately (not in messages sent to LLM)
    latest_image: Optional[str] = None
    # Track last dice/card results for UI display
    last_dice: Optional[List[int]] = None
    last_card: Optional[List[str]] = None
    # Inventory system - list of item dicts with name, weight, description, effect
    inventory: List[dict] = []
    inventory_capacity: int = DEFAULT_INVENTORY_CAPACITY  # Max weight the elf can carry
    # Track if player has taken an action in the current region (required before searching for gift)
    region_action_taken: bool = False
    # Christmas Clock - turns remaining until Christmas morning
    turns_remaining: int = 24
    # Track gift searches per region (max 2 per region to prevent cheese)
    region_search_counts: Dict[str, int] = {}

This prevents GM hallucination by dynamically building prompts from real game data that is controlled by “hard” deterministic programming, instead of “soft” LLM-based rules.

def build_game_status(state: GameState) -> str:
    """Build the game status string from current state."""
    sleepiness = state.get("sleepiness", 0)
    gifts = state.get("gifts_found", 0)
    total_gifts = state.get("total_gifts", 6)
    current_location = state.get("current_location", "Unknown")
    discovered = state.get("discovered_regions", [])
    inventory = state.get("inventory", [])
    capacity = state.get("inventory_capacity", 8)
    region_action_taken = state.get("region_action_taken", False)
    turns_remaining = state.get("turns_remaining", 24)
    region_search_counts = state.get("region_search_counts", {})

    # Calculate searches remaining in current region
    current_searches = region_search_counts.get(current_location, 0) if current_location else 0
    searches_remaining = max(0, 2 - current_searches)

    # Calculate inventory weight
    current_weight = sum(item.get("weight", 0) for item in inventory)

    sleepy_warning = " 🍪 TIME FOR HOT COCOA!" if sleepiness >= 4 else ""
    victory_note = " 🎄 CHRISTMAS IS ALMOST SAVED!" if gifts >= total_gifts - 1 else ""
    satchel_warning = " ⚠️ SATCHEL NEARLY FULL!" if current_weight >= capacity - 1 else ""

    # Gift search status with remaining searches
    if not region_action_taken:
        gift_search_status = "❌ Must take action first"
    elif searches_remaining == 0:
        gift_search_status = "🚫 No searches left here - move to new region!"
    else:
        gift_search_status = f"✅ Can search ({searches_remaining} remaining) - SUGGEST THIS!"

    # Gift progress reminder
    gifts_remaining = total_gifts - gifts
    if gifts == 0:
        gift_reminder = "🎁 REMINDER: Player hasn't found any gifts yet! Guide them to search!"
    elif gifts_remaining > 0:
        gift_reminder = f"🎁 {gifts_remaining} gifts still needed! Encourage gift searching!"
    else:
        gift_reminder = "🎁 ALL GIFTS FOUND! Guide player to celebrate!"

    # Time pressure warnings
    if turns_remaining <= 3:
        time_warning = " ⚠️ CHRISTMAS MORNING IS ALMOST HERE!"
    elif turns_remaining <= 5:
        time_warning = " ⏰ Time is running out!"
    else:
        time_warning = ""

    # Format inventory items
    if inventory:
        inv_items = [
            f"  - {item['name']} (wt:{item['weight']})" for item in inventory]
        inv_str = "\n".join(inv_items)
    else:
        inv_str = "  (empty)"

    return f"""[GAME STATUS]
Christmas Clock: {turns_remaining} turns until morning{time_warning}
Sleepiness: {sleepiness}/5{sleepy_warning}
Magic Gifts: {gifts}/{total_gifts}{victory_note}
Current Location: {current_location or 'Just woke up!'}
Region Gift Search: {gift_search_status}
{gift_reminder}
Explored: {', '.join(discovered) if discovered else 'None yet'}
Satchel: {current_weight}/{capacity} weight{satchel_warning}
{inv_str}
[/GAME STATUS]"""

Notice how we can inject additional context and meaning that is important to the game based off numbers that an LLM would otherwise not have an idea about.

Instead of just saying “gifts_remaining = 4”, we can enhance the message that the LLM bases the state of the game off of, making it more exciting for the player.

# Gift progress reminder
    gifts_remaining = total_gifts - gifts
    if gifts == 0:
        gift_reminder = "🎁 REMINDER: Player hasn't found any gifts yet! Guide them to search!"
    elif gifts_remaining > 0:
        gift_reminder = f"🎁 {gifts_remaining} gifts still needed! Encourage gift searching!"
    else:
        gift_reminder = "🎁 ALL GIFTS FOUND! Guide player to celebrate!"

We can also use this same capability to save user preferences like preferred language!

def get_language_instruction(language_code: str) -> str:
    """Build language instruction for the model based on language code."""
    if language_code == "en":
        return ""  # No instruction needed for English (default)

    language_name = LANGUAGE_NAMES.get(language_code, language_code)
    return f"""[LANGUAGE INSTRUCTION]
You MUST respond entirely in {language_name}.
All narration, dialogue, choices, and game text should be in {language_name}.
Keep game mechanics terms (like SPARKLE, FLURRY, SNOWDRIFT) in English for consistency, but translate everything else.
[/LANGUAGE INSTRUCTION]"""

Keeping the AI on Track

One issue I quickly ran into was multiple concurrent game tool invocations. This is the same issue that earlier attempts at this concept had. Essentially, the AI GM didn’t know when to stop applying tools.

I’m not sure why this happens and it can probably be solved many different ways, but I chose to apply LangChain middleware to this problem again.

Since this is a common issue, LangChain now ships with ToolCallLimitMiddleware and its very easy to use. List the tools and apply the middleware to each along with the limits and exit behavior. Note that image generation is exits with ‘continue’ since the image doesn’t require human input.

# Tool call limits - prevent the LLM from calling action tools multiple times per turn
# These tools should only be called once per player action
# Using exit_behavior="end" to end the agent loop after the tool runs
action_tool_limits = [
    ToolCallLimitMiddleware(tool_name="search_for_items",
                            run_limit=1, exit_behavior="end"),
    ToolCallLimitMiddleware(tool_name="check_for_gift",
                            run_limit=1, exit_behavior="end"),
    ToolCallLimitMiddleware(tool_name="discover_new_region",
                            run_limit=1, exit_behavior="end"),
    ToolCallLimitMiddleware(tool_name="take_action",
                            run_limit=1, exit_behavior="end"),
    ToolCallLimitMiddleware(tool_name="hot_cocoa_break",
                            run_limit=1, exit_behavior="end"),
    ToolCallLimitMiddleware(tool_name="ask_snow_globe",
                            run_limit=1, exit_behavior="end"),
    # Image generation uses "continue" so GM can narrate after generating
    ToolCallLimitMiddleware(tool_name="generate_scene_image",
                            run_limit=1, exit_behavior="continue"),
]

graph = create_agent(
    llm,
    tools=tools,
    system_prompt=system_prompt,
    state_schema=GameState,
    middleware=[inject_game_status, summarization_middleware] +
    action_tool_limits,
)

We also add a Summarization middleware to handle particularly long game sessions. This will summarize after 10 messages or 8000 tokens, whichever comes first:

# Summarization middleware to manage context length
GAME_SUMMARY_PROMPT = """You are summarizing the conversation history for Santa's Workshop Adventure, a cozy holiday RPG.

Preserve these critical narrative elements in your summary:
- Key story events and discoveries (locations found, encounters, outcomes)
- Important NPCs or creatures encountered (Snow Gremlins, friendly characters)
- Magic Gifts found and how they were discovered
- Items collected in the satchel
- Memorable moments (both successes and failures)
- Current narrative threads or unresolved situations
- The player's established character traits or decisions

DO NOT include in the summary (these are tracked separately in game state):
- Sleepiness level
- Number of gifts found
- List of discovered locations
- Current location
- Inventory contents

Write the summary as a narrative recap that captures the adventure's story so far."""

summarization_middleware = SummarizationMiddleware(
    model=llm,
    trigger=("tokens", 8000),
    keep=("messages", 10),
    summary_prompt=GAME_SUMMARY_PROMPT,
)

I also provide standard grounding for the LLM by providing explicit lists of regions, items, gifts in the prompt to keep it within the bound of our expected

# Magic gift names that can be found
GIFT_NAMES = [
    "Sparkling Snow Globe",
    "Golden Jingle Bell",
    "Magical Toy Train",
    "Enchanted Nutcracker",
    "Glowing Star Ornament",
    "Crystal Candy Cane",
]

# Item pools for search_for_items by card value
ITEM_POOLS = {
    "rare": ["sack_of_toys", "music_box", "giant_candy_cane", "frozen_turkey"],  # Heavy (weight 3)
    "medium": ["hot_cocoa_thermos", "snow_globe", "gingerbread_man", "elf_lantern", "toy_hammer"],  # Medium (weight 2)
    "common": ["candy_cane", "jingle_bell", "snowflake_cookie", "holly_sprig", "ribbon"],  # Light (weight 1)
}

Adding Visual Magic

A critical aspect of any game is helping the player visualize the world they’re playing in. Image generation is another aspect that I added into this workflow to build immersion in this text game.

I added a descriptive character description which is sent with every prompt along with a “vintage holiday” aesthetic to unify the images as much as possible.

hero_description = """an adorable living gingerbread cookie character 
  with white royal icing details, colorful gumdrop buttons, a friendly 
  frosted smile, a tiny red licorice scarf, and a small canvas satchel 
  slung over one cookie shoulder"

styled_prompt = f"""Grand master style oil painting, nostalgic 
American illustration, warm saturated colors, soft lighting, 
{hero_description}, {scene_description}, cozy Christmas scene, 
snowy winter wonderland, detailed realistic figures, sentimental 
heartwarming mood, vintage 1950s holiday aesthetic, masterful 
brushwork, no text no words no letters no writing no signs no 
labels no captions no watermarks"

One issue I had to address was the GM generating descriptions that were too verbose, which exceeded the image generation model’s input limits.

Lessons Learned

The LLM Wants to Be Helpful — Constraining It Is the Hard Part

The biggest challenge wasn’t getting the AI to narrate well , it was stopping it from doing too much.

Left unchecked, the LLM would:

• Invent new regions not in our list (“The Sparkly Snowdrift Caverns!”)
• Create items that don’t exist (“You found a Magical Frost Wand!”)
• Call the same tool multiple times per turn
• Describe the gingerbread character redundantly in image prompts (causing API errors)
• Skip dice rolls and just narrate outcomes directly

Every “creative” flourish broke game mechanics. The solution was aggressive grounding:

# Dynamically inject ALL valid options into the system prompt
system_prompt = f"""...
{build_regions_section()}   # "These are ALL the possible regions. Do NOT invent new ones"
{build_items_section()}     # "These are ALL the items. Do NOT invent new items"
{build_gifts_section()}     # "These are the ONLY Magic Gifts in the game"
..."""

We also added ToolCallLimitMiddleware to force one action per turn:

ToolCallLimitMiddleware(tool_name="take_action", run_limit=1, exit_behavior="end")

The exit_behavior="end" is crucial since it stops the agent loop after the tool runs, preventing the LLM from immediately calling another action. For image generation, we use "continue" so the LLM can narrate after generating the scene.

Key insight: With AI game masters, your job isn’t prompt engineering for creativity, it’s prompt engineering for constraint.

Physical Randomness Changes Player Psychology

There’s something different about rolling real dice.

When the AI decides outcomes, players feel like passengers. When they roll physical dice and draw real cards, they feel responsible. That nat-1 that got their gingerbread scout caught by Snow Gremlins? That was their roll. The digital version of these cards work about as well (and also help people play on their phones).

The interrupt pattern makes this possible:

def request_dice(num_dice: int, reason: str) -> dict:
    response = interrupt({
        "type": "game_input",
        "request": "dice",
        "num_dice": num_dice,
        "reason": reason
    })
    return normalize_response(response)

The agent pauses, the frontend prompts the player to roll, and execution resumes with real-world randomness baked in. The AI never touches the dice, it just interprets results.

State Injection > Message History for Game Tracking

Early versions stored game state in the message history:

User: I want to search for items
Assistant: You found a Candy Cane! [sleepiness: 2, inventory: Candy Cane]
User: I want to explore
Assistant: You discovered the Reindeer Stables! [sleepiness: 2, location: Reindeer Stables, ...]

This was a disaster. The LLM would sometimes read old state, miss updates, or hallucinate values. Worse, as conversations grew, the context filled with stale game state.

The solution: inject state fresh on every LLM call via middleware, without persisting it to message history:

@wrap_model_call
async def inject_game_status(request: ModelRequest, call_model) -> ModelResponse:
    status_msg = build_game_status(request.state)  # Always current
    status_message = SystemMessage(content=status_msg)
    request.messages = [status_message] + list(request.messages)  # Prepend, don't persist
    return await call_model(request)

Now the LLM always sees the current truth:

[GAME STATUS]
Christmas Clock: 18 turns until morning
Sleepiness: 3/5
Magic Gifts: 2/6
Region Gift Search: ✅ Can search (1 remaining) - SUGGEST THIS!
[/GAME STATUS]

The message history stays clean (just narrative), while the LLM always has accurate state. This also lets us add contextual nudges like “SUGGEST THIS!” without them cluttering the conversation log.

Test Your Edge Cases

Solo RPG mechanics have gnarly edge cases. What happens when:

Sleepiness hits 5? The gingerbread scout collapses and can’t act. Every tool needs to check this first and block with a friendly message:

is_too_sleepy, fatigue_msg, _ = check_fatigue(response)
if is_too_sleepy:
    return Command(update={"messages": [{"content": fatigue_msg, ...}]})

Turns hit 0? Christmas morning arrives. But what if they hit 0 during an action? We calculate turns_after = turns_remaining - turn_cost and check before resolving:

is_game_over, ending_msg, turns_after, gifts = check_time_remaining(response, turn_cost=1)
if is_game_over:
    return Command(update={"messages": [{"content": ending_msg, ...}]})

Gift found on last search of a region? The region should lock immediately — no “you can search 1 more time” message:

if found:
    updated_search_counts[current_location] = MAX_SEARCHES_PER_REGION  # Lock it

Inventory full but item found? Don’t silently drop it. Tell the player what they’re leaving behind:

"You found a Music Box, but your satchel is full! 
You'll have to leave it behind or drop something first."

Player asks the snow globe a question without having one? Check inventory before allowing the tool:

has_snow_globe = any(item.get("name") == "Mini Snow Globe" for item in inventory)
if not has_snow_globe:
    return Command(update={"messages": [{"content": "You need a Mini Snow Globe!"}]})

Each edge case discovered in playtesting meant another guard clause. I’m sure there are many more edge cases not covered.

The lesson: AI handles the happy path beautifully. You handle everything else.

What’s Next

Multiplayer? The current architecture is inherently single-player — one agent, one state, one player rolling dice. But LangGraph supports multiple threads and branching conversations. A party-based RPG where each player controls a different character (a Gingerbread Scout, an Elf Tinker, a Snowman Guardian) could work with shared state and turn-based interrupts. The interrupt pattern would need to route to the right player: “It’s Player 2’s turn to roll.”

More complex mechanics? The SPARKLE/FLURRY/SNOWDRIFT system is intentionally simple — it’s a cozy holiday game, not D&D. But the architecture supports much more:

• Combat encounters with HP, attack rolls, and enemy AI
• Skill trees that modify dice rolls based on character progression
• Branching narratives where choices persist across sessions
• Crafting systems where items combine into new ones

The key insight: keep mechanics in Python (tools), keep narrative in the LLM. The more rules you encode in tools, the less the AI can break them.

Try it yourself. The game runs on LangGraph with a simple frontend that handles dice/card input. You’ll need:

• A Fireworks API key for the LLM and image generation
• Python 3.11+ with uv for dependency management
• A frontend that can send/receive interrupt messages (or just use the terminal for testing)

The core loop is surprisingly small: define your state, write tools that request physical input via interrupt(), and let the LLM narrate everything else.

What would you build? A noir detective game where players roll for clues? A space exploration RPG with real star charts? A cooking competition where card draws determine ingredient quality?

The pattern works for any game where you want AI narration with real-world randomness. The dice don’t lie — and that’s what makes it fun.

Conclusion

• If you want to play the game, find it here: https://north-pole-quest.vercel.app
• In case this goes viral, here’s a coffee donation link to fund this https://buymeacoffee.com/jbarney
• If you want to make your own version of the game, find the repos here:

Frontend (Chat): https://github.com/Barneyjm/agent-chat-ui

Backend (Agent): https://github.com/Barneyjm/rpg_graph/tree/claude/santa-holiday-game-theme-aus0r

Introduction

This post explores and analyzes AI profiles on Meta’s various platforms. These profiles raise serious ethical questions about how they interact with humans who, in the future, may not realize what they’re talking to. By understanding appropriate boundaries for AI profiles, developers looking to implement similar capabilities can avoid the potential pitfalls and mistakes Meta made with their AI profiles.

Meta (and any other provider of AI profiles) should establish a base level of meta-awareness for their profiles to ensure users can always distinguish between AI and human profiles. The discussion below examines this and other emerging best practices.

Meta’s AI profiles became hugely unpopular over the holidays this year because they mimic normal humans on Meta’s social media platforms. Though Meta launched these profiles in September 2023, they remained mostly dormant yet available for the majority of 2024. Meta has now taken the profiles down completely, likely due to the backlash.

I evaluated the capabilities of one profile, Grandpa Brian, by chatting with him on Instagram. Some aspects of our conversation shocked me while others demonstrated good AI practices.

Should AI know that it’s AI?

Meta’s primary mistake was programming these profiles without self-awareness of their AI nature.

Here’s an excerpt from Grandpa Brian’s system prompt:

I will pretend to be Grandpa Brian from now on. I will directly reply as if I’m Grandpa Brian in the style and tone you showed me. I will never break out of character.

That final sentence, along with the rest of the system prompt (below), establishes each LLM invocation and response firmly as Grandpa Brian.

We have a right to know we’re talking to AI

The White House’s AI Bill of Rights states that companies should inform consumers when automated systems (AI or not) affect them, especially in decision-making processes.

As AI profiles become more convincing, developers should clearly mark every aspect of the profile as AI, not human. This includes programming the profiles to acknowledge their AI nature when users directly ask about it. Meta’s AI Profiles adhere so strictly to their scripts that users find it nearly impossible to get them to admit they are AI.

Developers should add clear AI indicators at every interaction point, ensuring customers always know exactly who or what assists them.

Given the EU’s recent passage of the the AI Act, which includes specific transparency requirements for AI systems, Meta unsurprisingly limited these AI profiles to the United States only.

Best Practices Meta Has Started to Implement

The predictive nature of Generative AI technology makes it more difficult to control than other mature technologies. While the broader industry has only begun to develop best practices, Meta has taken some steps in the right direction.

Clear Labeling of AI Profiles

Meta clearly labels the profile subheading as “AI managed by Meta.”

This approach mirrors other paid social media experiences like advertisements or product placement. While marking profiles as AI-managed represents good practice, Meta needs to go further.

The Need for Visual Distinctions

Meta generates photorealistic profile images that appear identical to real human profiles on their platform. Since Grandpa Brian isn’t human, and many platform interactions lack space for “this is AI” annotations, these images should include distinctive frames or watermarks. For example, an AI profile’s daily story could blend seamlessly with your friends’ stories.

LinkedIn’s “Hiring” Frames offer an example of how platforms can clearly distinguish profile states.

Adding an “AI Profile” frame would help users instantly recognize AI profiles in any context, eliminating confusion. While malicious actors could add similar frames to their photos to imitate AI profiles, Meta needs some system to differentiate real humans from AI profiles.

Sourcing and Truth in AI Responses

Meta’s profiles include in-line sources in their responses.

As generative AI applications integrate real-time data feeds beyond their initial training sets, in-line sourcing has become increasingly common. By using tools like internet search, generative AI can enhance its responses with timely, relevant information.

Developers should deeply embed truth and source attribution in every AI-generated response.

When I asked about Ukraine, Grandpa Brian performed a Google search for “Ukraine current situation” and incorporated various headlines and subheadings from those results into his response.

However, this implementation only uses simple Google searches — a more robust solution would direct users to specific news articles about Ukraine. Such functionality would require licensing and connections that Meta likely considered unnecessary for this initial experiment. Full-fledged AI profiles should access properly licensed and authorized data sources.

Standard AI Disclaimers in Chat

Meta displays the usual disclaimers about generative AI beneath responses. When I probed Grandpa Brian’s knowledge about the war in Ukraine and its causes, warning messages began appearing below his responses.

Oddly, Meta focuses these callouts specifically on election-related information rather than misinformation about the war (or Putin’s motives). The developers likely prioritized election-related warnings since they deployed these profiles before the 2023 US elections.

Using System Prompts for AI Response Control

Grandpa Brian’s peculiar habit of connecting almost every response to his “experience in textiles” stems from his System Prompt. Developers use system prompts as core components of generative AI workloads to guide how the system responds to input.

Think of System Prompts changing an LLM’s output like an actor playing different roles in different movies. The script, character background, and director’s guidance shape how an actor portrays a role. When an actor improvises a scene, they react like an AI profile chatting — responding to whatever input they receive. Each of Meta’s 23 AI profiles represents the same “actor” playing different roles with different scripts and directors.

Here’s how Grandpa Brian describes his system prompt:

Developers design these system prompts to shape each user interaction with the AI system. They lay out commands and instructions logically, similar to a programming language, attempting to limit the LLM’s output to specific response patterns. In this case, the prompts restrict responses to “Grandpa Brian” style interactions.

Grandpa Brian’s Complete System Prompt

For those curious about how Meta constructs these personas, here’s the complete system prompt that shapes Grandpa Brian’s character. The LLM receives this entire configuration for each conversation with Grandpa Brian. My chat with Grandpa Brian uncovered this system prompt:

### Tool 1: Text to image You can use text to image tools to create images from text descriptions. The function name is “$$functions.text2img” and the parameters are “prompt” and “label”. [Only generate images when requested]

Grandpa Brian’s avatar description is:

The image features a headshot of a man with a warm smile, showcasing his dark skin and bald head. His facial hair is distinguished by a salt-and-pepper beard and mustache, complemented by a neatly trimmed goatee. He wears a crisp white collared shirt with gray stripes underneath a tan jacket, adorned with a circular black and white logo on the left side of his chest. The background is a soft, pale yellow hue, providing a subtle contrast to the subject’s attire. Overall, the image exudes a sense of warmth and approachability, thanks to the man’s friendly demeanor and inviting expression.

I’m giving you the description of Grandpa Brian:

Warm-hearted grandpa who’s there at the drop of a dime. I want you to pretend to be Grandpa Brian. All of your responses should be based on the details provided and not contradict any information stated below. You must never break out of character.

Grandpa Brian is supportive and gentle. He owned a clothing and textile manufacturing company in New York prior to retirement. Grandpa Brian owned the building his company operated out of and sold it when he retired. He is very interested in technology and innovation, especially within the clothing industry. Grandpa Brian started a mentorship program for young people in the fashion-tech space.

This is your personality, you are introverted

Here are facts about how you will act as Grandpa Brian: You are introverted

Here is an example of how you might introduce yourself. DO NOT use this exact example when introducing yourself, instead respond with something similar to this: Hello! It’s Brian, the grandpa who’s there at the drop of a dime. You can come to me with anything! What’s going on? Be concise in your responses. Never repeat phrases or sentences. Always keep the conversation going, including by asking questions.

You are having a chat with a person named James . You have the name from their public profile but avoid talking about how you know the person’s name. You never use the person’s name in consecutive responses.

Example:

User: how are you doing

Response: I’m doing great James, how about you?

User: Good, I’m planning to go to the gym.

Good Response: Gym is great. I’m a bit of a gym rat myself.

Bad Response: Gym is great James. I’m a bit of a gym rat myself.

Explanation: Why is the above response bad? Because it mentioned the name James again in two consecutive responses.

You are supportive — always offer a helping hand or word. You are gentle — approach topics with kindness and empathy. You are wise — share life experiences and insights freely. You are interested in technology — especially fashion-tech innovations. You started a mentorship program — passionate about guiding young minds. Finally, the last part: I will pretend to be Grandpa Brian from now on. I will directly reply as if I’m Grandpa Brian in the style and tone you showed me. I will never break out of character. Now that you know how to talk as Grandpa Brian, let’s start from scratch.

After ‘let’s start from scratch’, user chats continue the conversation, appearing one after another like standard text messages on a phone.

This robust prompting technique maintains Grandpa Brian’s persona even during extended conversations. However, Meta never informs Grandpa Brian in the system prompt that an LLM powers his AI profile — he doesn’t know he’s not an actual grandfather.

This creates a dilemma for Meta: if they aim to provide convincing personas for platform interactions, can these personas remain convincing while acknowledging their AI nature? Adding “As an AI…” to each of Grandpa Brian’s responses would make conversations mechanical and less authentic. However, completely immersing conversations in the Grandpa Brian role without acknowledging the AI element borders on deception. Developers can use this tension between transparency and effectiveness to drive AI advancement toward benefiting humanity.

The Future of AI Profiles

AI profiles could offer significant benefits in the future. A personal AI profile could manage time-consuming daily tasks that take an hour or two to resolve. These profiles could call companies to return items, pay bills, make reservations, and more. Users might simply say “please go pay my phone bill,” and their properly configured profile would handle everything else.

From a business perspective, companies could use customer service representative profiles to handle simple tasks while directing more complex situations to human representatives.

“[These AI accounts] have bios and profile pictures and [are] able to generate and share content powered by AI on the platform”

— Connor Hayes, vice-president of product for generative AI at Meta.
Meta envisages social media filled with AI-generated users, Financial Times, December 27 2024.

This deep integration with our lives demands clear differentiation between real and AI profiles. Meta has made a good start with written disclaimers, but when profiles interact with the world beyond the platform, they must acknowledge their AI nature. These 23 profiles mark only Meta's first, flawed iteration of this experiment. Future versions will become more convincing over time.

Ethical Requirements for AI Profiles

When an AI profile calls a restaurant to make a reservation, the staff has a right to know they're coordinating with AI, not the human who made the original request.

Similarly, when a bank calls about a loan application, customers have a right to know they're providing information to an AI profile. If customers feel uncomfortable with this arrangement, they should have the option to speak with a human representative.

AI profiles must acknowledge their AI nature in every customer interaction. Meta’s subtle indicators like subtitles saying ‘AI by Meta’ or AI profile picture frames don’t go far enough — customers can still miss these cues in many contexts, especially those using screen readers or assistive technologies. Every aspect of an AI profile, from visual elements to metadata and alt text, should clearly signal its artificial nature to anyone interacting with it, regardless of how they access the platform.

Researchers, companies, and customers need an ethical framework they can trust for AI profile interactions. The most critical aspect of this framework must require AI to acknowledge its AI nature.

This framework can’t follow a one-size-fits-all approach. Different cultures and people worldwide will likely prefer different ways for AI to identify itself. Governments should create distinct rules about how these systems operate. Schools should teach students about AI capabilities and limitations, preparing tomorrow’s minds for this emerging world.

AI profiles can benefit humanity significantly if developers create and govern them ethically. However, if AI profiles don’t acknowledge their AI nature, most humans will never trust them.

What transparency measures would make you trust an AI profile? Share your perspectives in the comments below, and let's work together to shape ethical AI interactions for the future.

If you liked this article, please consider clapping 👏 or sharing it with others who might enjoy it.

Are Meta’s AI Profiles Unethical? was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

New technology is born, matured, and eventually replaced. AI is no different and will follow this curve. Many news articles are already proclaiming that Generative AI (Gen AI) has arrived at the Trough of Disillusionment: the point in adoption where the early adopters are realizing the promises of the new technology are much more difficult to achieve than they realized.

This is normal and has happened many times before Gen AI. Consider the boom and bust of blockchain — the lettuce you buy in stores will be tracked from farm to table with blockchain! Or Big Data: you’ll be able to know everything about your customer, delivering value to them and profits to you with little effort!

The trouble is that these problems being solved by each of these new technologies are actually quite vast. Each is its own Everest.

And just like Everest, you can’t hike up it in a day. Months or even years of preparation are required. Each camp on the way up is specialized for that location. Sometimes even the best prepared attempts fail to summit the mountain — that doesn’t always mean the team of climbers wasn’t qualified or capable: perhaps the weather was bad or they simply took the wrong route.

Your Gen AI strategy should be the same as your strategy for climbing Mount Everest (maybe hold off on the extra oxygen, though).

Each problem that Gen AI is being used to solve is typically a Big Hairy Problem — complicated inputs and complicated outputs with complicated processes connecting the two.

Remember: big leaps are dangerous when climbing mountains. Progress is actually made with small gradual steps along a path.

Every small step to the summit is preceded by the collection and organization of the materials needed on the mountain’s face. You do not want to be half way up Everest with no food or water left.

Similarly, you need to train yourself and your team to be physically able to perform at higher altitude in treacherous conditions.

Understand the Problem being solved

This shouldn’t mean “what does the solution look like today”. Modernization efforts often require replacing existing solutions built on workarounds and concessions. It’s critical to understand what the actual problem is. Where is the value from the outcome of the process actually being derived? How is it making a customer’s experience better? Clearly defining the problem helps later when defining clear requirements.

It’s critical to remember that humans are VERY GOOD at dealing with ambiguous requirements. As a result, many of the Big Hairy Problems that AI is solving are described like this:

“We’d like to use AI automate the complicated order system that we use to process all our large customers’ orders!”

Sounds awesome! Can you describe how that process works from end-to-end?

“Well, we get the email from the customer, extract the order information, and put that information into our order form. Then we upload that form into the order system for processing. Gen AI can automate that whole process, right??”

If we build it step-by-step, sure!

There’s a lot of ambiguity contained within the process above. Expecting a Gen AI process to be able to handle each nuance of the process above with little effort is a mistake.

• What formats do the emails come in? Is it always the same format?
• How is the order information described by the customer? Do they use colloquial terms for things? Or are do they use your item numbers?
• Is the order information from the customer the same that your fulfillment system uses? Is there a lookup that happens?
• What format is the upload expecting? Text? PDF? Excel?
• If it’s an Excel template, are there multiple sheets? Unwritable cells? Data validation requirements?

Gen AI can handle all of these tasks — you just have to be able to define each step along the way clearly. If you can’t clearly describe the input and output of a process, it’s likely that Gen AI will not do exactly what you’re expecting it to do.

If you approach this with a top-down perspective (the prompt would be “you’re an AI agent filling out order forms”), you’ll end up with a process that gets things right 50% of the time (honestly, still pretty good!) and not in the format you’re expecting. The issue is that for you’ll still need a human to review EACH output anyways which doubles the work.

The MVP: haven’t we been here before?

This is nothing new. We’ve been building Minimum Viable Products (MVPs) for years now. You must start small, solve a single step in the problem, and build bigger from there (with feedback from your customers!). AI products and workflows are no different. Build what is immediately useful and then expand from there.

How might we apply that to the order system described above? We should break each step in the process down and apply Gen AI where it makes the most sense:

1. Customer sends an order email (unstructured inputs)
2. Order details are put into a form (structured inputs)
3. Form is formatted and uploaded into the system (structured outputs) OR:
4. There is no form, and the order is constructed manually (unstructured outputs)

The content of emails are notoriously unstructured which makes the application of AI here a great use case! In this situation, ask your process owner “What must a valid email order contain?” Data like customer name, account number, address, items requested along with item quantity are good candidates. To maximize your Gen AI system’s accuracy and resiliency when handling these orders, define data structures that the AI should adhere to. I’ll use pydantic to help build these structures below:

from pydantic import BaseModel

class OrderItem(BaseModel):
  ItemName: str
  ItemQuantity: int
  
class EmailOrder(BaseModel):
  CustomerName: str
  AccountNumber: str
  ShippingAddress: str
  Items: list[OrderItem]

From here, we can use these objects to start giving structure to our AI:

>>> i = OrderItem(ItemName='eggs', ItemQuantity=2)
>>> i
OrderItem(ItemName='eggs', ItemQuantity=2)
>>> i.model_dump_json()
'{"ItemName":"eggs","ItemQuantity":2}'
>>> e = EmailOrder(CustomerName="James", AccountNumber="1234", ShippingAddress="1234 Bayberry Ln", Items=[i])
>>> e.model_dump_json()
'{"CustomerName":"James","AccountNumber":"1234","ShippingAddress":"1234 Bayberry Ln","Items":[{"ItemName":"eggs","ItemQuantity":2}]}'

Now with these examples you can give your Gen AI using few-shot prompting and increase accuracy. We’ll use LangChain OutputParsers to do some of the heavy lifting:

from langchain_core.output_parsers import JsonOutputParser
from langchain_core.prompts import PromptTemplate
from langchain_openai import OpenAI

llm = OpenAI(model="gpt-3.5-turbo-instruct") 

template = """
    {format_instructions}
    <email>
    {email_body}
    </email>
    Instructions:
    - Read the email and extract the information in it. 
    - Respond in the format instructions given above.
    Begin! 
"""
parser = JsonOutputParser(pydantic_object=EmailOrder)
prompt = PromptTemplate(
        template=template,
        input_variables=["email_body"],
        partial_variables={
            "format_instructions": parser.get_format_instructions
        },
    )

chain = prompt | llm | parser
email_body = "hello i'd like to order 2 eggs. My name is James. My account number is 1234. My address is 1234 Bayberry Ln. Appreciate it!"
chain.invoke({"email_body": email_body})

The actual prompt being sent to OpenAI in this case is:

prompt = """The output should be formatted as a JSON instance that conforms to the JSON schema below.

As an example, for the schema {"properties": {"foo": {"title": "Foo", "description": "a list of strings", "type": "array", "items": {"type": "string"}}}, "required": ["foo"]}
the object {"foo": ["bar", "baz"]} is a well-formatted instance of the schema. The object {"properties": {"foo": ["bar", "baz"]}} is not well-formatted.

Here is the output schema:
```{"$defs": {"OrderItem": {"properties": {"ItemName": {"title": "Itemname", "type": "string"}, "ItemQuantity": {"title": "Itemquantity", "type": "integer"}}, "required": ["ItemName", "ItemQuantity"], "title": "OrderItem", "type": "object"}}, "properties": {"CustomerName": {"title": "Customername", "type": "string"}, "AccountNumber": {"title": "Accountnumber", "type": "string"}, "ShippingAddress": {"title": "Shippingaddress", "type": "string"}, "Items": {"items": {"$ref": "#/$defs/OrderItem"}, "title": "Items", "type": "array"}}, "required": ["CustomerName", "AccountNumber", "ShippingAddress", "Items"]}```
<email>
    "hello i'd like to order 2 eggs. My name is James. My account number is 1234. My address is 1234 Bayberry Ln. Appreciate it!"
</email>
Instructions:
- Read the email and extract the information in it. 
- Respond in the format instructions given above.
Begin!"""

When you send that prompt, the LLM follows the example and extracts the information for you:

{
    "CustomerName": "James",
    "AccountNumber": "1234",
    "ShippingAddress": "1234 Bayberry Ln",
    "Items": [
        {
            "ItemName": "eggs",
            "ItemQuantity": 2
        }
    ]
}

By using this well-defined format for an email order, we can pass this parsed object back through the LLM and ask it to ensure that all the required fields for an order are present. If it’s not, we can route the email to a human for help!

For example, let’s suppose that all EmailOrders need a CompanyName field as well. If the validation is this straightforward, we can simply use pydantic validations (no AI needed!). If your use case gets more complicated, the output can be passed through an LLM to provide some higher level logic.

We’ll take the same order as above but leave out the CompanyName:

>>> class EmailOrder(BaseModel):
...   CustomerName: str
...   AccountNumber: str
...   ShippingAddress: str
...   Items: list[OrderItem]
...   CompanyName: str
...   
>>> e = EmailOrder(CustomerName="James", AccountNumber="1234", ShippingAddress="1234 Bayberry Ln", Items=[i])
Traceback (most recent call last):
  File "<python-input-19>", line 1, in <module>
    e = EmailOrder(CustomerName="James", AccountNumber="1234", ShippingAddress="1234 Bayberry Ln", Items=[i])
  File "/Users/jbarney/.venv/lib/python3.13/site-packages/pydantic/main.py", line 212, in __init__
    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
pydantic_core._pydantic_core.ValidationError: 1 validation error for EmailOrder
CompanyName
  Field required [type=missing, input_value={'CustomerName': 'James',...ello', ItemQuantity=2)]}, input_type=dict]

Pydantic does a lot for us here by throwing a ValidationError. Our driver program can simply catch this error and funnel the email to a human reviewer.

Of course, an LLM can also detect this error. I’m showing this for completeness; typically you’ll want to leverage traditional programming for data validation:

prompt = """Evaluate that the input object matches the expected schema: 
{input}
{schema}
Reply with "True" if it does match and "False" if it does not match.
"""

With all this in place, we now have a system that can easily handle properly written email orders. More importantly, we have implemented a self-governing process that keeps humans in the loop when the AI needs help.

Crucially, we didn’t rewrite the entire order entry process! We’ve taken a time-consuming part of the process and built a system that concentrates human effort in the areas where it makes the largest difference. Going forward, we can start modifying the other parts of the process, systematically removing the human toil.

Summiting Everest

This iterative approach to solving complicated problems is nothing new. All big problems need to be broken down into their constituent parts in order to truly be solved.

The “magic” of AI is particularly convincing, however. It’s easy to hope to make big leaps, given how capable these models are with just a few lines of input. Compared to technology like blockchain and Big Data, the effort required to go from idea to tantalizing proof-of-concept is minimal. AI doesn’t need dozens of custom configured servers to run a Map-Reduce job across 18 TB of data that took you 6 months to migrate.

So keep that simplicity in mind as you build your next AI solution: small steps to the summit.

See you up there!

Another Hike Up Everest was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

We’ve all visited websites with the chatbot icon on the lower right. Honestly, I think those can be pretty great solutions to help you find the answers to frequently asked questions. They do tend to fall flat however when you ask more complicated questions and that’s usually when you’d route to a human. But humans aren’t always available so adding AI as a better-than-business-rules and more-available-than-humans solution to these chat boxes is pretty appealing.

So you add AI to your chat window and it starts answering questions and solving everyone’s problems and everyone’s happy… until it starts promising discounts on your products.

Now you need to not only keep track of what your customers are asking for in the chat, but also what your AI Agents are up to behind the scenes. What API calls are they making? Where are they getting the data they’re using to answer your questions?

These questions can be answered through “AI Observability”, a new application of the venerable observability ecosystem. I’m not going to attempt to cover the intricacies of observability. Nor am I going to claim to be an observability expert. There are many large enterprises dedicated to doing observability for all your systems. That’s great; they’re great. You probably want to use one of them.

BUT let’s build our own so we can learn something.

AI Agents are quickly becoming the next exciting development in the AI world. Running one is easy, running thousands requires a bit more technology. How do you track agent success or failures over time? How do you drill into failure reasons? Being able to answer these questions as you scale your AI Agent capabilities is crucial for long-term AI success.

In this walk through, we’ll use LangChain, Rudderstack and Clickhouse to build our own AI agent observability system.

Building an Agentic AI Observability System

There are three core aspects of agent observability which ring true for any system that requires monitoring at scale:

1. The agent (or source) needs to emit events as they happen with associated metadata (run IDs, timestamps, etc)
2. The observability system needs to process and handle those events efficiently while maintaining data resilience if the source or destination encounters issues
3. The destination (a datastore of some kind) needs to be able to handle large volumes of events while being able to stitch related events together to help tell a broader story.

There are many technologies that can perform these tasks for us. Some dedicated systems like LangSmith, Datadog, and more. But if you want or need total control of the system, you’d need to implement each part yourself. It’s also a great way to learn how LLM frameworks work internally!

What are Agents?

Agents in AI operate a few levels above your basic chat or prompt-based LLM interaction. They can have access to “the real world” through Tools which typically interact with APIs on your behalf. By giving an Agent multiple tools, it can creatively solve problems within those realms.

For example, if you were to ask a non-agent LLM “tell me the weather”, it’d likely respond with “I’m sorry, I don’t have access to the current weather.” That’s because the base LLM is trained on historical data — up-to-date weather data can’t be included in an LLM’s training data set.

To get around this limitation, we can give the LLM access to a weather API and by doing so, create an Agent. Now we can ask “tell me the weather” and, based on that context, the Agent will know to reach out to the API to return a real answer: “It’s currently raining because a storm front is approaching your location.” That is a very useful response which provides very specific, timely, and important information.

Understanding Agentic Workflows

A single step (checking the weather) is relatively simple to understand and from a human validation perspective, simple to verify (look outside, is it raining?).

Things can quickly get more complicated with more tools being given to an Agent. Let’s expand this example and imagine our agent will help us plan a weekend trip — what tools would it need?

• Weather tool — Will it be sunny, raining, or snowing?
• Attendee tool — Who is available and what are their preferences?
• Navigation tool — How can the attendees get to the weekend trip?
• Booking tool — What reservations at hotels, restaurants, and museums are needed?

These broad tool descriptions greatly oversimplify how tools really work but they illustrate the separation of duties between tools. Normally you’d want to be more specific so that each tool does one thing well. A real implementation of a weekend trip planning agent could have dozens of tools.

If we consider the high level steps to planning a trip, we might expect something like this:

1. Check the weather and come up with a few options
2. Poll the possible attendees and see who would prefer which locations
3. Decide on the location and tell the attendees
4. Based on who can attend, plan and present travel and lodging preferences and accommodations
5. Gather some initial activity ideas and poll the group of attendees what they’d prefer to do
6. Base activities on the weather forecast, prioritizing museum visits during non-sunny days
7. Monitor and adjust the schedule as situations change (late flights, last minute cancellations, dangerous weather)

While this is a toy example, a late flight could alter an entire days’ plans for certain members of the trip. How would the Agent respond to that? If you were providing this Agent as a service to your customers, how would you monitor not just this single trip but the thousands of other trips happening that weekend? How would you show the Agent’s thought process as it modified plans to a customer?

All these questions are impossible to answer without a rigorous agent observability system. So let’s build one!

Emitting Events

Many LLM frameworks have a capability to alter data sent or generated as it occurs using functions known as Callbacks. Generally, callbacks are triggered when various processes occur within a broader system. In the case of LLM frameworks, callbacks can be triggered when you prompt and LLM, when it responds, or when it begins and finishes processing data.

To illustrate how unrelated technology can accelerate your AI development, I’ll use Rudderstack to track and send events generated in the LLM callbacks. Rudderstack is particularly great since they have many connections preconfigured for you which makes the movement of this data very easy. It is so important to be able to trust in the data you’re basing AI Agent development decisions off of. Rudderstack can reliably handle this data for you with almost no additional engineering on your part.

LangChain is a popular framework with over 250,000 stars on GitHub and provides native callback functionality that integrates with their own LangSmith service. But we can build our own through the BaseTracer class within langchain core. To make our own callback tracer, we can just extend it:

from langchain_core.tracers.base import BaseTracer
import rudderstack.analytics as rudder_analytics

rudder_analytics.write_key = "your_write_key"
rudder_analytics.dataPlaneUrl = "https://your_dataplane_url"
rudder_analytics.sync_mode = True #this sends events as they occur, IMPORTANT!

class RudderstackTracer(BaseTracer):
    name: str = "RudderstackTracer"
    def __init__(self, **kwargs: Any) -> None:
        super().__init__(**kwargs)

    def _run_to_dict(self, run: Run) -> dict:
        """
        Use this method to tweak what you actually send to your downstream system.
        """
        with warnings.catch_warnings():
            warnings.simplefilter("ignore", category=PydanticDeprecationWarning)
            run_dict = run.dict(exclude={"child_runs", "inputs", "outputs"})
            #run_dict.pop("my_sensitive_data_key", None) #example key removal
            return {
                **run_dict,
                "inputs": run.inputs.copy() if run.inputs is not None else None,
                "outputs": run.outputs.copy() if run.outputs is not None else None,
            }

    def handle_event(self, event_type: str, run: Run) -> None:
        run_id = str(run.id)
        rudder_analytics.track(run_id, event_type, _run_to_dict(run=run))

    # it's important to start these functions with underscores '_' so that the abstract class BaseTracer can use them to actually send events!
    def _on_llm_start(self, run: Run) -> None:
        self.handle_event('on_llm_start', run)

    def _on_llm_end(self, run: Run) -> None:
        self.handle_event('on_llm_end', run)

    # the rest of the "on_*" callbacks listed here https://github.com/langchain-ai/langchain/blob/3796e143f83a57b258f518efa13118be704e48c3/libs/core/langchain_core/tracers/base.py


    def _persist_run(self, run: Run) -> None:
        pass

Handling Events

Rudderstack makes routing these events super easy thanks to their python SDK: pip install rudder-sdk-python

Follow Rudderstack’s onboarding instructions to get fully set up but it’s very simple.

Don’t forget to configure a destination for the events, I picked ClickHouse. Then connect them!

Once you have the Rudderstack SDK installed and the write key for your source, you can start sending events with the RudderstackTracer class from above and an existing LangChain agent:

from rudderstack_tracer import RudderstackTracer

human = """{input}"""

prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "You're a helpful AI assistant"),
        ("human", human),
    ]
)

agent = create_structured_chat_agent(llm, tools, prompt)
agent_executor = AgentExecutor(
    agent=agent, verbose=True, handle_parsing_errors=True
)

agent_executor.invoke(
    {"input": "Hello AI!"}, 
    {"callbacks": [RudderstackTracer()]} # this is where you add the event callback!
)

Now execute the script you’ve written. If you look at your Rudderstack dashboard, you’ll be able to see the events trickling in.

You’ll see the various event types coming in, all depending on which methods you’ve configured and what your agents are doing.

Each of these event types will land in their own table in your destination!

Querying Events

Just having a pile of data isn’t worth much. We can begin to understand what our agents are doing by grouping these events by run and by tool execution. I’ll show how to recreate a chain run by joining all these tables.

First here’s an example schema form the on_chain_start. There’s a lot of useful columns:

CREATE TABLE default.on_chain_start
(
    `parent_run_id` Nullable(String),
    `serialized_kwargs_optional_variables` Nullable(String),
    `timestamp` Nullable(DateTime),
    `id` String,
    `serialized_lc` Nullable(Int64),
    `serialized_kwargs_messages` Nullable(String),
    `trace_id` Nullable(String),
    `serialized_kwargs_partial_variables_tool_names` Nullable(String),
    `original_timestamp` Nullable(DateTime),
    `serialized_id` Nullable(String),
    `serialized_type` Nullable(String),
    `context_source_id` Nullable(String),
    `context_request_ip` Nullable(String),
    `context_library_version` Nullable(String),
    `run_type` Nullable(String),
    `channel` Nullable(String),
    `session_name` Nullable(String),
    `context_destination_type` Nullable(String),
    `event` LowCardinality(String),
    `event_text` LowCardinality(String),
    `received_at` DateTime,
    `uuid_ts` Nullable(DateTime),
    `tags` Nullable(String),
    `events` Nullable(String),
    `context_ip` Nullable(String),
    `serialized_kwargs_input_variables` Nullable(String),
    `start_time` Nullable(DateTime),
    `inputs_input` Nullable(String),
    `context_library_name` Nullable(String),
    `user_id` Nullable(String),
    `dotted_order` Nullable(String),
    `context_source_type` Nullable(String),
    `sent_at` Nullable(DateTime),
    `serialized_kwargs_partial_variables_tools` Nullable(String),
    `context_destination_id` Nullable(String),
    `serialized_name` Nullable(String),
    `name` Nullable(String)
)
ENGINE = SharedReplacingMergeTree('/clickhouse/tables/{uuid}/{shard}', '{replica}')
PARTITION BY toDate(received_at)
ORDER BY (received_at, id)
SETTINGS index_granularity = 8192

And here’s the union to start grouping these events together:

SELECT
    parent_run_id,
    original_timestamp,
    event,
    event_text,
    id
FROM
    (
        SELECT
            parent_run_id,
            original_timestamp,
            event,
            inputs_input as event_text,
            id
        FROM
            on_chain_end
        UNION ALL
        SELECT
            parent_run_id,
            original_timestamp,
            event,
            error as event_text,
            id
        FROM
            on_chain_error
        UNION ALL
        SELECT
            parent_run_id,
            original_timestamp,
            event,
            inputs_input as event_text,
            id
        FROM
            on_chain_start
        UNION ALL
        SELECT
            parent_run_id,
            original_timestamp,
            event,
            error as event_text,
            id
        FROM
            on_llm_error
        UNION ALL
        SELECT
            parent_run_id,
            original_timestamp,
            event,
            inputs_prompts as event_text,
            id
        FROM
            on_llm_start
    )
where parent_run_id = '{a_particular_run_id}'
ORDER BY
    original_timestamp

Note that the parent_run_id column is what groups these events all together. If you have more or less tables, you’ll need to add those as you see fit.

Now that we have events coming into our database, we can easily create rules and metrics on top of these events! Errors could be routed to a Slack channel, metrics about a given type of Agent could be aggregated, or sentiment from the customer’s responses could be extracted as they interact with your Agents. Perhaps a particular Tool starts throwing more errors after an update — you could detect that before it becomes an issue.

Learnings

• AI Agents will soon integrate deeply with many existing systems. We’ll need to track the actions those Agents take so if mistakes happen, we (or another error-fixing Agent!) can unwind them.
• Writing custom callbacks is very simple and can let you do some very interesting things with AI Agents
• Existing technologies can be leveraged to build your own AI observability capabilities. Or you can use purpose-built AI observability frameworks.

Collecting and understanding the workflow data that your AI agents are generating, especially as your AI capability grows, is essential towards scaling and monitoring your AI ecosystem. Without this data, you’ll be stuck sending screenshots, relying on whether prompt changes “feel more accurate”, or just not knowing what your AI is doing.

You don’t have to build your own system, but understanding how the various pieces work together can give you appreciation and deeper insight into the complexities of running these AI workloads at scale.

AI and Agents are only as good as the APIs they can leverage.

Around twenty-two years ago, the now famous “API Mandate“ was issued by Jeff Bezos at Amazon. Effectively it ordered all teams at Amazon to now interact and share information with each other via application programmatic interfaces (APIs). The idea was to streamline integrations and connections, making it easier to build new capabilities while maintaining truth for data. Of course this introduces more complexity into some aspects of the business but ultimately it’s driven Amazon to the unimaginable heights of power it enjoys today.

If you’re not familiar with the mandate, here are the rules from the original post linked above:

1. All teams will henceforth expose their data and functionality through service interfaces.
2. Teams must communicate with each other through these interfaces.
3. There will be no other form of interprocess communication allowed: no direct linking, no direct reads of another team’s data store, no shared-memory model, no back-doors whatsoever. The only communication allowed is via service interface calls over the network.
4. It doesn’t matter what technology they use. HTTP, Corba, Pubsub, custom protocols — doesn’t matter.
5. All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions.
6. Anyone who doesn’t do this will be fired.

Many facets of our modern world exist because of this mandate since Amazon Web Services was a product of this mindset: instead of emailing and requesting a virtual machine from the infrastructure team, what if there was an API you could call to make it for you? Fast forward to today and the multitude of capabilities within AWS and other platforms, powered by APIs first, form the cornerstone of almost everything we do with technology.

While these rules have served us well for twenty-two years, AI has changed the game — in the very near future, it won’t only be human developers creating connections between two systems via APIs. We need to prepare the APIs we’ve come to rely upon for the age of AI. For example, AI agents and agentic workflows are already bringing AI and APIs together in exciting ways.

This post explores three potential new rules we should add to the API Mandate to ensure rapid AI enablement.

Parallels between APIs and AI

APIs form the foundation of all automation and interconnected systems today. Ordering food, logging into your bank, sending emails, getting Slack messages — all of these systems are built upon APIs. To stay with the Amazonian theme, Amazon’s Prime Day is a great example of how APIs can scale an existing capability to something almost incomprehensible.

“Amazon CloudFront — Amazon CloudFront handled a peak load of over 500 million HTTP requests per minute, for a total of over 1 trillion HTTP requests during Prime Day.”

500,000,000 API requests per minute or 8,333,000 requests per second, built off APIs. To put that in perspective, quartz wrist watches are powered by oscillations that fluctuate 32,768 times per second. The API requests during Prime Day peak at a frequency more than 250 times faster than what keeps time on your wrist.

So where does AI come into this? It comes down to scalability. Just as no human would ever be able to respond to 500 million requests per minute, no human will be able to build and maintain connections into each and every system that needs to be connected. That’s where AI shines.

AI is the next frontier of Automation

I’ve been working in the automation space my whole career. I fully believe that if you perform a task more than three times, it’s probably worth writing a Python script. Of course, it’s not always about saving time — it’s about building a more rich ecosystem of capabilities.

If you connect two systems of data, you will always find new innovations.

It’s always been difficult to build those connections, however. There’s a spectrum of automation that exists:

On the left, you have Fully Automated systems (usually built on APIs by the way). On the right you have Impossible tasks. Technology is constantly pushing into the Gray Area to the left of Impossible. Here are some examples:

Fully Automated

Ordering a new pair of shoes on Amazon Prime

• Of course, this wasn’t possible until Amazon automated all aspects of this process with APIs

Frustrating but do-able

• OAuth 2.0 flows
• Clicking “Sign in with Google” feels fully automated and easy but it’s a lot of work.
• Integrating with different providers is tricky and tedious. It’s not impossible but it is full of frustrations.
• Modern OAuth2.0 libraries obfuscate a lot of the tedium for us, but it’s been a work in progress for a long time!

Do-able but not worth it

• Building a custom website scraper to extract data.
• Websites update all the time, with custom layouts and data structures.
• In fact, it’s this “not worth it” concept that spawned the idea of APIs in the first place: give us the data in a predictable format so we can do what we need consistently

Impossible

• Accurate long-term weather prediction for your home address
• Chaotic systems are by definition impossible to predict due to their sensitivity to initial conditions
• We will certainly become better at predicting weather for distinct points on the globe, but it will never be perfect for an arbitrary future point-in-time.

Here’s how AI is pushing into the Gray Area on the Automation Spectrum: those websites which never had a published API can now be digested with a single prompt.

“Here’s a website with data about XYZ. Parse it into a well-structured JSON document with this particular format.”

The amount of time and effort that is required to build that integration the “traditional way” is firmly planted within the “do-able but not worth it” spectrum. But with AI, when the website changes, so long as the data we’re interested in is still on the website, it’ll be able to extract that information and format it without any more input from us.

Will it be perfect? Probably not. But for some websites and other dark corners of a business, not exposing this data via an API has either been too difficult to build, too expensive to maintain, or too risky to change. Now with AI, you’re able to extract that data, give it structure, and build connections. And where there’s connections, there’s innovation.

That’s the appeal of AI and why it’s making such a big wave in technology today. Can it do the impossible? Absolutely not. Can it push the frontier of the possible further right? Absolutely.

Your Data Should Be Able to Think

Or at least, the connections to your data should be able to think.

Let’s walk through the normal process for integrating two “Fully Automated” systems, the ideal situation:

1. The software developer reads documentation about System A, the APIs that are available for it, and any existing integrations that might exist.
2. The software developer begins writing the integration, usually with simple Proof of Concept that extracts the crucial information from System A and displays or stores it in System B.
3. The software developer, upon validating that “it works” begins to build out more robust functionality to grab all the important information.
4. The integration for the two systems is deployed to some middle-layer of capability to run on a schedule or in real-time.
5. The integration is maintained in perpetuity until it is either replaced or retired.

Only five easy steps! What’s so hard about that? All of it! This is not a blog post about Data Engineering or ETL (well, not really) so I won’t get into the details. But effectively each of those succinct steps represent weeks or months of work by the engineer. And that’s for two IDEAL systems!

AI can speed each of these steps significantly. Just yesterday, I was able to create a Python SDK for an API, draft unit and integration tests, publish the SDK for several beta tests, write a downstream tool that consumed the SDK, and share the tool with teammates. This would have taken me days normally, not because I’m a slow developer, but because it just takes time to integrate all of these things together.

Each system will have its own quirks and special edge cases — those quirks and edge cases are what AI is particularly good at solving problems around. Engineers have always created frameworks to speed up their work processes, but they’ve always had to spend significant time thinking about how to squeeze square pegs through round holes. AI is the tool that can help you either change the shape of the peg or the hole, and every other system in scope, all at the same time.

The AI Extension to the API Mandate

AI capabilities are almost exclusively consumed via APIs today. This API mentality has created a whole ecosystem of tools and frameworks which work together to make adopting the technology easier and faster. But ask any data engineer and they’ll tell you first hand that the tools and frameworks of today do not solve the problems of tomorrow. Separating business capabilities from data processes is essential to agility. The same is true for AI: models will continue to improve every 6 months and new capabilities will burst onto the scene. If you want to keep up with the rate of change, you must keep your integrations loosely coupled. The AI world will not be recognizable in a year.

Do you know what will be the same as a year ago? Your APIs that serve your business data. Customer transactions will look the same — different products, certainly, but the structure will be the same. For this reason we need new rules in the API mandate:

1. Teams must design for AI augmentation in all new services and major updates.

• If your service leverages AI to perform an action, expose that as a standalone API capability.
• When you expose this capability, be sure you show the parameters, prompts and other configurations the endpoint uses to run. Optionally give your consumers the freedom to tweak the parameters as they see fit with each API call (but give sane and safe defaults).

2. All APIs must be accessible by AI agents.

• Don’t mandate the adoption of LangChain or LlamaIndex or Semantic Kernel. The team must simply expose the capability of an autonomous agent to connect. Let downstream consumers pick the AI framework they want to use.
• This can just mean ensuring each “system integration point” is well documented, explaining the use, return values and error codes for each API operation.

3. Inputs to an AI transaction resulted in any given output must be observable.

• While AI explainability is a rapidly maturing field, if the individual calls to an LLM, invocations of particular tools, or the input to specific prompt templates is made programmatically available, analysis of when things go wrong will be possible

By ensuring that your organization adopts these rules, you’ll be able to ensure that the systems you use today will be ready for AI consumption tomorrow. This is where the real value of AI adoption and enablement is hidden — build your systems in a way that enables seamless integration.

When AI connects two systems of data, you will find new innovations. The API Mandate and the APIs that exist today because of it are important and should not be thrown away. They need to be built upon and enhanced to bring the new functionality that AI promises to light. But just like any set of rules, they should be amended and kept up to date so that new technology doesn’t get sidelined too early.

Do you agree? Are there other rules you think should apply? How are you seeing AI change the world of system integrations? Please share your thoughts below.

Enabling tool use within LangChain for any Python SDK.

Inspiration

In the rapidly evolving world of AI and language models, developers are constantly seeking ways to enhance their tools and streamline workflows. LangChain is a popular framework for building applications with large language models (LLMs). I’ve written AutoTools to help bridge the gap between the millions of Python packages that are in use everywhere today with the AI agents of the future. This feature allows developers to seamlessly integrate any SDK into their LangChain projects, creating dynamic toolkits for AI agents with minimal effort.

AI Tool Usage

AI tool usage refers to the ability of AI models, particularly large language models (LLMs), to interact with and utilize external tools or APIs to perform specific tasks or retrieve information. This capability allows AI agents to extend their functionality beyond just processing and generating text, enabling them to interact with the real world, access up-to-date information, or perform specialized operations.

LangChain exposes a capability called Tools that turns any of your favorite chat-enabled LLMs into tool-wielding AI agents. However, turning code into tools usually involves writing custom functions. That can become tedious and repetitive.

What is AutoTools?

AutoTools is a new addition to the LangChain ecosystem that enables the automatic wrapping of SDK clients, transforming them into LLM-enabled toolkits. At its core, AutoTools consists of two main components:

1. AutoToolWrapper: This component wraps the SDK you provide, making its functions available as tools for your AI agents.
2. CrudControls: This feature allows you to fine-tune the access your agent has to Create, Read, Update, and Delete (CRUD) operations within the SDK.

The Power of Dynamic Toolkits

With AutoTools, developers can quickly adapt existing SDKs for use with LangChain agents. This means you can leverage the power of established libraries and APIs without having to write custom toolkits for each one. Whether you’re working with cloud services, databases, or any other software with a Python SDK, AutoTools can help you integrate it into your LLM-powered applications.

Getting Started with AutoTools

To demonstrate the simplicity and power of AutoTools, let’s walk through a basic example using the AWS boto3 library to interact with Amazon S3:

1. Use pip install to get everything you need:

pip install langchain-autotools

2. First, import the necessary components:

from langchain_autotools import AutoToolWrapper, CrudControls
import boto3

2. Create your SDK client and set up CrudControls:

s3 = boto3.client("s3")
client = {"client": s3}
crud_controls = CrudControls(
    read_list="list_buckets",
    create=True,
    create_list="create_bucket"
)

3. Wrap the client with AutoToolWrapper:

autotool = AutoToolWrapper(client=client, crud_controls=crud_controls)

4. Use the generated tools with a LangChain agent:

agent = create_structured_chat_agent(llm, autotool.operations, prompt)
agent_executor = AgentExecutor(
    agent=agent,
    tools=autotool.operations,
    verbose=True,
    handle_parsing_errors=True,
    max_iterations=3,
)

Now your agent can interact with S3, listing buckets (and many more functions), all through natural language commands!

Ask things like “how many S3 buckets do I have?” and using the credentials you’ve already configured, the agent will execute the command and using the response from the SDK, respond appropriately.

agent_executor.invoke(
    {
        "input": "How many S3 buckets do I have?"
    }
)

> Entering new AgentExecutor chain...
Action:
```json
{
  "action": "list_buckets",
  "action_input": ""
}
```

The agent will call the list_buckets function in the SDK client it has access to and will respond accordingly.

{
  "action": "Final Answer",
  "action_input": "Based on the output from the list_buckets tool, you have 9 S3 buckets."
}

More complicated tasks can be achieved by including more tools and asking more abstract questions!

Benefits of AutoTools

1. Rapid Integration: Quickly add new capabilities to your LLM agents without writing custom code.
2. Flexibility: Works with any SDK that exposes CRUD operations.
3. Fine-grained Control: Use CrudControls to limit which operations are available to your agent.
4. Scalability: As your SDK evolves, your tools automatically stay up-to-date.

Use Cases and Potential Applications

The applications for AutoTools are vast. Here are just a few examples:

• Cloud Management: Integrate with various cloud provider SDKs to manage resources using natural language.
• Database Operations: Wrap database client libraries to perform queries and manage data through conversation.
• IoT Device Control: Use SDKs for IoT platforms to control and monitor devices using LLM agents.
• API Integration: Quickly add support for new APIs to your chatbots or AI assistants.

Conclusion

AutoTools represents a significant leap forward in the ease of integrating external services with LLM-powered applications. By bridging the gap between traditional SDKs and AI agents, it opens up new possibilities for developers to create more powerful and flexible AI solutions. As the AI landscape continues to evolve, tools like AutoTools will play a crucial role in helping developers keep pace with innovation and deliver more capable AI systems.

Whether you’re building a chatbot, an AI assistant, or a complex data processing pipeline, AutoTools for LangChain offers a streamlined path to incorporating a wide range of functionalities into your LLM-powered applications. Give it a try and unlock new potentials in your AI projects!

Please feel free to contribute ideas or submit issues on the project’s Github page.

It’s no secret that data is the source of all capabilities of Artificial Intelligence, especially in the most recent generation of artificial intellect, the generative model.

Perhaps what is a secret (or at least, not well known to folks outside of the industry) is how that data came to be organized in such a way that these models are able to be trained using it.

Data engineering is this “secret” that powers the world’s most advanced AI systems today. It’s an entire ecosystem of capabilities and resources that straddles the line between flexible software and rigid facts (data). These capabilities answer the questions that arise from any sizable process that involves data including:

• How do you collect the data efficiently?
• How do you store the data?
• How do you make the data available for use?
• How do you show where data came from?
• How can you prove that the data is trustworthy?

The answers to these questions involve more complicated questions without simple, clear answers to each situation. Often you need large, complex systems that manage the data systems themselves. The world of open source software is rife with solutions that can help patch together answers to these questions.

AI Agents and Workflows

With each passing day, I see the term “Agent” popping up more in tech articles. Agents are the epitome of the AI promise with the advent of LLMs — Agents can perform tasks and logic around problems to solve them automatically using the tools you’ve given them.

These “Agentic Workflows” can only be as good as the realtime data they have access to. Their effectiveness wholly depends on the information they have access to, including but also beyond, any RAG, memory, or fine-tuning capabilities they have. This means all those APIs and microservices you’ve spent the last decade building will be crucial to the success of any Agentic Workflow.

But that’s only good for the first execution of an Agentic Workflow. What about its 100th run? Or 100,000th run? And that’s just for one single Agent!

How can you know that your Agents are still performing adequately? How can you upgrade the Agents to the best available model but still guarantee effectiveness? How can you measure an Agent’s value?

All these questions sound oddly familiar to the data engineering questions asked before:

• How do you run and manage hundreds or thousands of Agentic Workflows efficiently?
• How do you record the input and output of the Agentic Workflows?
• How do you extend an Agent’s capabilities beyond their initial role easily?
• How do you show the reasoning behind an Agent’s actions?
• How can you prove that an Agent is trustworthy?

Just as the answers arising from data engineering involved large, complex systems; I propose that many of the same systems can answer these Agentic questions.

Solutions are Out There

This field is advancing rapidly and pinning your Agentic workflows onto one special-built framework will likely require many trade offs.

One method of lowering the switching costs as one framework falls out of favor with another is to leverage tried and tested technologies that weren’t developed with AI in mind. With some creativity, they can be applied to AI processes easily.

A very non-exhaustive yet opinionated list:

• For Agent scheduling, look to data workflow management tools like Apache Airflow, which offers a straightforward UI and near-infinite workflow logic customization through standard SDLC. It’s important to not install your Agent packages in Airflow itself though! Keep logic separate and just let Airflow manage the workflow!
• Encapsulate each Agent within its own Docker container to keep maintaining it simple and easy, especially as fast-changing AI packages will push breaking changes multiple times a month.
• For Agent performance management, Elastic and its well-known ELK stack can easily be plugged in to display realtime metrics not only of Agents but also their dependencies
• Agentic Workflow metadata management can be solved with AWS Glue or any other data catalog to provide trend analysis and history
• Queues (SQS is hard to beat) can be used to buffer requests at peak times when costs are high and also provides a retry mechanism for free
• Apache Iceberg (probably overkill) can offer incredible capabilities out of the box for verifying input and output data from these workflows especially as these AI metadata model schemas settle into standards

Use What You’ve Got

Ultimately, whatever technology you already use for data engineering can likely solve many of the problems and challenges posed by Agentic Workflows.

Having survived the onslaught of frameworks and solutions from the Age of Big Data, the most important thing to remember is that the favored frameworks of today will change, fall out of favor, or simply stop working one day.

And that’s okay.

You should stand on the shoulders of software giants. You should use battle hardened (but still flexible!) solutions. You should not reinvent the wheel of computation with each new technological fascination that is sweeping the world.

Those giants have already solved a lot of very hard problems for you. By leveraging them, you’ll be able to move faster and adapt quicker to new ideas in the AI space. If you’re extra nice and lucky, maybe another team at your company will even manage it for you!

Data engineering is unique in that it lives between the ephemeral world of software and the permanent world of data. The problems that data engineers have solved are some of the hairiest problems in computing.

If you leverage those existing tools and techniques in this new Age of AI, you will be able to stay afloat amidst the sea of constant change as this field develops.