Building AI Agents in 2026: The Evidence Behind the Stack
We read the source of 100+ agent repos and audited MCP Code Mode. The stack that held up, who should own side effects, and how to test crash recovery.
For a small team building a hosted, multi-user agent: TypeScript, Node, Hono, AI SDK, Zod, Postgres and pg-boss, with a separate worker. The bigger decision is who owns side effects: the model proposes, application code authorizes, records and executes. Judge any stack on a deliberately interrupted task, not on a clean demo.
The short answerLink to this section
We spent the last few weeks reading agent repositories rather than agent launch posts: a 93-profile survey of trending repos, a 26-repo architecture deep dive (107 distinct repos combined), a source audit of MCP Code Mode, a 200-repo landscape over eight layers, and a technical review of 25 repos created between 24 August and 22 November 2026. This guide is what survived that reading. Snapshot date: 22 November 2026.
The target is specific: a small team building a hosted, multi-user web agent that reads business data, calls tools and sometimes takes approved actions. A local assistant, a coding harness or an ML application has different needs and should not copy this stack wholesale. We cover the local case near the end.
For that target we would start with TypeScript, Node LTS, Hono, AI SDK, Zod, PostgreSQL and pg-boss, React/Vite if it needs its own UI, one hosted model, and a separate worker process once a run outlives the HTTP request. One codebase, two processes. One job system, not two.
No single repo uses every piece, and this stack did not win a benchmark. It combines the lean product topology in Agentes, the database-and-worker pattern in Nod, and the typed tool interface in AI SDK.
The decision that matters more than the language is who owns the side effect. Let the model decide what to look up or propose. Let application code own identity, authorization, the approved arguments, durable intent, execution and reconciliation. A persistent chat, a durable queue and a sandbox each solve a different problem. None of them tells you whether a remote write committed before the connection dropped.
If the core work is Python (data processing, model experiments, evals), keep Python: Pydantic AI with FastAPI and Postgres is a sensible typed stack.
How each research pass changed the answerLink to this section
| Pass | What we observed | What it changed |
|---|---|---|
| Trending repo survey | 93 stack profiles; 67 had a positive MCP signal. That counts clients, servers and config files, not only enabled product integrations. | MCP is a normal integration option, not the architecture of the agent. Start with a few direct tools when they are enough. |
| Architecture deep dive | 26 repos, 14 new; 107 distinct combined. Runtime, persistence and tool files show differences a dependency scan cannot. | Separate the product, the loop, the execution environment and the source of truth. A wrapper around an external CLI is not its own loop. |
| Code Mode source audit | 13 of 107 had a source-backed code-mediated MCP path: 6 first-party agents, 4 SDKs/frameworks, 2 gateways, 1 community plugin. 85 had nothing verified in the files we reviewed. | Code Mode is real but not universal. Source code, default state, release status and measured savings are four separate claims. |
| Failure-recovery comparison | Approval pause/resume, workflow replay, interrupted sessions and external effects reviewed as separate capabilities. | Judge a stack on an interrupted task and the downstream state, not on a fluent answer or a reloaded chat. |
| 200-repo landscape | Primary language: Python 88, TypeScript 54. Framework layer 17 Python to 10 TS; product/harness layer 15 Python to 16 TS. | Python-first was fine for a Python-friendly agent, not as a universal default. Counting languages across serving, tooling and UI does not decide a web product stack. |
| Recent-repo review | 299 candidates created 24 Aug to 22 Nov; 25 got a technical review, chosen for architecture evidence over stars. | Recent source and written trade-offs count for more than old popularity in a greenfield decision. |
The cohorts overlap, so the counts do not add up. A primary-language label is not a stack inventory. A README mention is a claim, a dependency is an available component, and a runtime path is a mechanism. None of them tells you deployment frequency, default state or reliability under faults. In particular, this data does not show the ecosystem moving away from Python.
The recent cohort is filtered by GitHub creation date, which is not the same as when the code started. Agentes has decision records dated October in a repo created in November, and Graphicle is a Rust port of Graphify.
Six architectures worth copying fromLink to this section
| Case | What the source shows | Copy this; don't assume that |
|---|---|---|
| Nod: product around a reused loop | Node/Fastify, React, Postgres/pg-boss and the Pi loop. A task ledger coordinates saved sessions; Docker runs workspaces and browsers; QuickJS Code Mode reaches tools through host rules. Architecture · Runtime | Reuse the loop, keep task state in the product. Saved sessions do not mean mid-script crash recovery. |
| Agentes: product owns the loop | Bun/Hono, Vite, Drizzle/Postgres. It replaced a planned Next.js/Redis/BullMQ design. Its outbox leases and fences jobs; run states and approvals live in Postgres. It later dropped a Daytona-specific lifecycle for connected computers. Stack · Outbox · Runner | Good evidence for simpler self-hosting. Not a Redis migration study, a throughput test or a case against managed sandboxes. |
| DevBox: two languages at a real boundary | Bun control plane with Postgres tasks; Python Deep Agents in per-task sandboxes with SQLite checkpoints. Architecture | Split languages when control and execution need different runtimes, not because both have good libraries. |
| Design Agent: Python as the engine | Deep Agents/LangGraph, a sandbox per thread, domain skills and media tooling. Architecture | The work can justify Python. A web chat surface does not force the loop into the frontend's language. |
| Pi Pocket and Talorys: smaller state | Pi Pocket runs on SQLite locally. Talorys puts the agent and its SQLite state in a Cloudflare Durable Object. Pi Pocket · Talorys | Match state to ownership and deployment. Neither is the right sharing model for multi-tenant SaaS. |
| AgentRun: what "resume" means | Its Codex adapter records nested operations and can rebuild an interrupted script's response from them. Code | Ask what happens to each completed and pending operation, not whether a library says "resume". |
The stack, layer by layerLink to this section
| Layer | Why it's the default | Change it when |
|---|---|---|
| TypeScript + Node LTS | One language across API, tool contracts, worker and UI. A team choice, not a speed claim. Bun is a credible alternative (Agentes, DevBox). | The team is Python-first and the work leans on data, ML or evals. |
| Hono + React/Vite | Keep the API thin. The event stream is a view of durable run events, not the run itself. Fastify or Next.js are fine swaps. Hono | A framework already owns the product, or chat lives in Slack. |
| AI SDK + Zod + one model | Streaming and typed tools. Own the task lifecycle outside the SDK. Schema validation, authorization and business rules are three different checks. | Use an existing harness for file or computer tasks. Add a second provider only after testing its tool calls and structured output. |
| Postgres + Drizzle | Run state, approvals, tool operations and job intent behind one transaction. Enforce uniqueness and state transitions in the database. | The state is truly local (use SQLite), or measured contention says split. |
| pg-boss or an outbox | Enqueue work atomically with the run's first state. pg-boss can enqueue inside an existing transaction. | Long waits, timers, workflow versioning and cross-service recovery are routine, not rare. Then look at Temporal. |
| Direct tools first, MCP when reusable | Typed functions cover a few first-party APIs. Keep identity and policy independent of the tool transport. MCP tools spec | The catalog is large: benchmark lazy discovery and Code Mode. |
| Full-text search before vectors | Documents and permissions are canonical, indexes are derived, prompt context is temporary. Start with Postgres full-text search, try pgvector if recall improves tasks. | Measured relevance, latency or corpus size calls for dedicated retrieval. |
Not every workflow needs an agent loop. For an invoice or ticket process with known steps, use plain orchestration and call a model only for the fuzzy parts, such as extraction or drafting. Use one agent with tools when the information gathering varies. Consider multiple agents only when separate permissions or measured parallel speed outweigh the handoff and debugging cost.
Reference design for an agent that can writeLink to this section
Browser / chat client
│ create run / approve operation / read events
▼
API ───── Postgres: runs, events, operations, approvals, queue
│ ▲
▼ │
worker ─── model loop ─── tool policy / dispatcher
│
├── authorized remote API or MCP
└── optional isolated executorThe stream is an interface, not a worker lease. An HTTP disconnect should not lose an accepted run, and a reconnect rereads stored events by cursor. User, tenant, connection and permissions stay in the host, never in model-supplied arguments. If the agent runs scripts or computer tools, put them in an isolated environment that fits the trust model. A Git worktree separates workspaces; it is not a sandbox. Agentes tool gateway
The minimum recordsLink to this section
| Record | Contents | Rule |
|---|---|---|
runs | ID, tenant, principal, task version, model and prompt version, state, budget | A duplicate delivery cannot start a second copy of the same run. |
run_events | Run ID, sequence number, type, bounded payload or artifact ref | Enough to rebuild the UI and debug a run. No credentials, no unrestricted private payloads. |
tool_operations | Stable operation ID, business key, input digest, tool and connection, status, remote receipt | A retry reuses the same identity. Same key with different arguments is an error. |
approvals | Operation ID, action and input digest, approver, expiry, decision | Approval covers one exact operation. Recheck permissions at dispatch, after the wait. |
| Queue or outbox | Job type, run or operation ref, retry budget, claim info | Enqueue with the state change. Workers tolerate redelivery; a late worker cannot overwrite a newer claim. |
These are our proposed invariants, not framework requirements. Agentes shows approvals bound to arguments and queue leases fenced by owner and attempt. Stripe's idempotency docs show why you check each destination's retention rules.
Why "exactly once" is not a queue featureLink to this section
An approved operation creates a support ticket. The host saves its intent, sends the request, the ticket service commits, and the worker dies before saving the receipt. On restart the host knows it meant to act, not what the service did. No queue, transaction, workflow replay or model reasoning resolves that by itself. Temporal describes the same case for activities.
| The destination offers | Recovery policy |
|---|---|
| An idempotency key | Reuse the same key with identical inputs. Use a stable business key, not a new key per worker attempt. |
| A unique external reference | Reconcile by that reference. An eventually consistent read that returns nothing is inconclusive, not "not created". |
| Neither | Mark outcome_unknown and stop replaying the write. Surface it for reconciliation. Don't tell the user it failed, and don't send it again. |
| A recorded success already | Reuse the receipt and move on. Don't repeat a write to rebuild a transcript. |
Idempotency is not reversibility. A compensating action, like closing a wrongly opened ticket, leaves both events in the audit trail and can fail on its own. Define compensation separately from retries.
MCP Code Mode: where it earns its placeLink to this section
| Tool surface | What the model does | Good for |
|---|---|---|
| Eager native tools | Sees every schema, calls tools one by one, sees every result. | A small stable catalog and simple approvals. |
| Lazy discovery | Searches a compact index, loads only the schemas it picks. | Big catalogs where tasks use a few tools. The baseline Code Mode has to beat. |
| Code Mode | Writes a program that calls tools through a host bridge and returns only the useful part. | Loops, fan-out, dependent reads, joins and filtering bulky data before it hits the context. |
Code Mode does not rewrite the MCP server. It moves control flow and intermediate data from model turns into a runtime, while the host still authorizes each nested call. We confirmed three different execution boundaries: Nod uses QuickJS with tool checks, modebox exposes MCP tools inside a WebAssembly sandbox, and Unharnessed offers a Python dispatcher and warns its REPL is not a sandbox.
Feature status varies too. Goose documents an opt-in extension, OpenCode's dev docs describe a default path, and the Codex revision we reviewed had it under development and off by default.
On cost: Anthropic's example shows a drop from 150,000 to 2,000 tokens, and Cloudflare describes a roughly 1,000-token tool surface. Those show why context can shrink, not what your bill will be. To find out, run eager, lazy, and lazy plus Code Mode with the same model, tasks, permissions, cache settings and failure schedule. Divide total spend by successful tasks, so failures and repairs are not free.
Ship Code Mode first for read-heavy work if the gain survives that test. For writes, keep authority in the host and make each nested write its own durable, auditable operation. A script that picks later actions from earlier results is not one approval.
What durable execution buys, and what it doesn'tLink to this section
| Layer | Example | What it proves |
|---|---|---|
| Session durability | Saved sessions in Nod | Transcript survives a restart. Not execution continuation. |
| Approval continuation | Cloudflare's runtime persists approval state and replays recorded calls | An approved operation resumes without repeating recorded calls. Test hibernation, process death and deploys separately. |
| Workflow replay | Pydantic AI with Temporal puts model and tool calls at activity boundaries | Completed activity results are reused. The unacknowledged remote write is still open. |
| Effect durability | A destination with a business key or reconcilable reference; the host stores consent, intent and receipt | One intended effect, even after the response is lost. No framework gives you this alone. |
One outside study backs treating recovery as its own metric. UndoBench reports 83.54% clean-run competence against 46.72% conditional recovery success over 2,880 paired simulated trials. It is not a production benchmark and does not rate the repos above. Its useful idea is the method: break something on purpose, then inspect the final service state.
If your agent runs locallyLink to this section
Most of our readers run models on their own hardware, so a note on the local case. The research points to a smaller topology: SQLite instead of a database service, as Pi Pocket does, and an existing harness if its session model fits. A local model served by Ollama, llama.cpp or vLLM can stand in for the hosted one, but test tool calls and structured output on that exact model and quantization before you rely on it. Context also costs memory: tool schemas and results eat into the window, which our VRAM and KV cache guide explains.
The ownership rules don't change. A local agent that edits files or sends email still needs exact approvals, stable operation IDs and an honest "unknown" state. For ready-made setups, see Open WebUI with MCP tools, n8n with Ollama, a LiteLLM gateway and local agents on Apple Silicon.
Rollout plan and acceptance testsLink to this section
| Stage | Build | Gate |
|---|---|---|
| Read-only slice | One workflow, one model, a few typed tools, versioned prompts, structured events, a repeatable task set with ambiguous and out-of-scope requests. | Correct answers or proper refusals from authorized data. Record latency and cost baselines. |
| First approved write | One operation, durable intent and approval, stable key, receipt, explicit unknown state. Test against a fake integration with its own ledger. | Zero unapproved or duplicate writes in fixture tests. That is not proof of zero production risk. |
| Restarts and deploys | Worker leases, bounded retry, reconciliation, versioned queued work and approvals. | Replayed jobs can't overwrite a newer claim. Old approvals can't run with a changed meaning. |
| More capability | Retrieval, MCP catalog, isolation or orchestration, one layer at a time. Save failures as regression tests. | Keep it only if it improves a measured constraint. |
The fault matrix should cover at least: worker dies before dispatch; the response is lost after the provider commits; worker dies after storing a receipt but before the next step; restart during an approval wait; an old worker returns after its lease expired; the user loses access during the wait; a second write fails after the first succeeds. Check the fake provider's ledger and your own records separately. Report clean success, clean recovery, duplicates, unresolved outcomes, policy violations and p95 recovery time. Count a safe stop separately from a finished task.
The pass/fail questionLink to this section
Buy simplicity where it's reversible, design for correctness where it isn't. Write the agent in your team's natural language, keep the first deploy small, reuse a harness if it fits. Spend early effort on tool contracts, identity, approvals, operation records, reconciliation and fault tests. Those boundaries transfer between stacks; the queue and frontend choices mostly don't.
The one test to run: after the worker dies right after a remote write, does the product end with one intended effect, or stop honestly when it can't know?
Prepared with AI assistance. Repository files were read at the pinned commits linked above; research snapshot 22 November 2026. The repo counts are from our own review; the UndoBench figures are the paper's, not ours.
Frequently asked
Is Python or TypeScript better for building AI agents?
Neither in general. For a hosted web product we default to TypeScript so the API, tool contracts, worker and UI share one language. If the core work is data processing, model experiments or evals, Pydantic AI with FastAPI and Postgres is the better start.
Do I need MCP to build an agent?
No. A few typed direct tools are enough for a small set of first-party APIs. Add MCP when you want to consume or publish reusable connectors. 67 of the 93 stack profiles in our first survey had some MCP signal, but that includes config files and clients, not only enabled integrations.
Does MCP Code Mode always save tokens?
Not automatically. Only 13 of the 107 repos we screened had a source-backed Code Mode path. Published token drops are illustrative examples. Measure cost per successful task against lazy tool discovery with the same model and tasks.
Can a durable queue or workflow engine guarantee exactly-once tool calls?
No. If a remote write commits and the worker dies before saving the receipt, only the destination can tell you what happened. Use the destination's idempotency key or a unique external reference, and mark the outcome unknown when neither exists.
What should a local-only agent use instead?
SQLite instead of a database service, and an existing harness if its session and execution model fits. The ownership rules for writes and approvals stay the same.