He has a newsletter on beehiiv. So do Codie Sanchez, Jay Shetty, David Begnaud, Colin & Samir, and Joanna Stern.

They could publish just about anywhere. They chose beehiiv to build a direct connection with the people who want to hear from them most. And you can do the same, whether you’re world-famous or just at the start of your plan for world domination.

Get started for free.

You're getting this because you subscribed to The AI Hatchery. Not for you anymore? Unsubscribe in one click.

Hi there,

The AI Hatchery is back. The last issue went out on November 23, 2025, and then things went quiet. I had to take a break from The AI Hatchery as I raised funding for Gravitational Ventures and managed the 3x growth of OSS4AI. Now I'm restarting it. If you got my note last week, this is the relaunch I promised.

Here's what changed. The AI Hatchery is now about two things: building AI agents that hold up in production, and growing into the customer-facing AI roles around that work: AI DevRel, forward deployed engineer (FDE) and AI engineering.

There are three sends, and you choose which ones you get:

You can pick in your preferences.

If we haven't met: I'm Yujian Tang. I was a software engineer at Amazon, then worked at startups, then was a developer advocate at Zilliz. Now I run OSS4AI, an open-source AI developer community in Seattle.

Why now? I moderate r/AI_Agents, and I see a lot of posts from people asking how to get into AI roles and how to learn the skills to do it. The AI Hatchery is my answer to those posts.

The 5 places agents break, in one line each:

There are three quick questions at the end. Your answers decide what I write next.

Agents are easy to demo and hard to keep running. The adoption numbers say so, with caveats. Gartner's 2026 Hype Cycle for Agentic AI says "only 17% of organizations have deployed AI agents to date" (the page doesn't give a sample size). And Gartner predicts: "Over 40% of agentic AI projects will be canceled by the end of 2027." That's a forecast, not a count, backed by a poll of webinar attendees.

So where do they break? Benchmarks, engineering write-ups and public incidents point to five places, ranked by strength of evidence. Vendor sources are marked.

A demo shows one good run. Production needs the same good result every time, across many steps and many turns. Each step can fail, and the errors carry forward.

The benchmarks agree on the pattern. In tau-bench (Sierra researchers, 2024), "even state-of-the-art function calling agents (like gpt-4o) succeed on <50% of the tasks, and are quite inconsistent (pass^8 <25% in retail)." Those are older models, and current ones score higher; the gap between one good run and eight in a row is the point. Anthropic's evals guide (vendor) gives the arithmetic: at a 75% per-trial success rate, the chance of passing three trials in a row is "(0.75)³ ≈ 42%."

Long conversations and long contexts make it worse. A paper from Microsoft and Salesforce authors on multi-turn conversation measured "an average drop of 39% across six generation tasks," and found that "when LLMs take a wrong turn in a conversation, they get lost and do not recover."

I've seen tons of agents break at hackathons and demo days. My own agent demo once broke on me in the middle of a tutorial I was giving.

Try this week: stop reporting single-run success. Take your top 20 tasks, run each one 5 times, and track pass^k: the share of tasks that pass every run. As Anthropic's guide puts it, "As k increases, pass^k falls since demanding consistency across more trials is a harder bar to clear."

Your agent reads content it didn't write: issues, emails, web pages. It also holds tools that can act. The model can't reliably tell your instructions apart from an attacker's, and broad permissions turn one bad output into a damaging action.

Simon Willison calls the dangerous combination the lethal trifecta: private data, untrusted content, and a way to send data out. "If your agent combines these three features, an attacker can easily trick it into accessing your private data and sending it to that attacker." OWASP puts prompt injection first on its 2025 Top 10 for LLM applications, and calls the permission side Excessive Agency, with three root causes: "excessive functionality; excessive permissions; excessive autonomy."

Two shipped examples:

Try this week: for every tool your agent holds, note whether it (a) reads private data, (b) ingests untrusted content, (c) can send data out. Wherever all three meet, remove one. Then cut each tool's permissions down to what the task needs. And run the agent on dev or staging credentials only, which is the fix Replit shipped: "separate development and production databases, with the AI Agent restricted to development environments by default."

Agents are non-deterministic, and they change whenever a prompt, model or tool changes. Without a test set you can't see a regression. Without traces you can't see why a run failed.

Anthropic (vendor) hit this with its own research agent: "users would report agents "not finding obvious information," but we couldn't see why." What fixed it: "Adding full production tracing let us diagnose why agents failed and fix issues systematically."

LangChain's survey (1,340 responses, late 2025) shows the gap. Just over half (52.4%) run offline evaluations on test sets, and 22.8% of teams with agents in production don't evaluate at all. Tracing is further along: 89% have some form of observability, and 62% have detailed tracing. Two caveats: LangChain sells an evals and observability product, and respondents chose to answer a framework vendor's survey.

Try this week: build a first eval set from your own failures. Anthropic's guide: "20-50 simple tasks drawn from real failures is a great start." Run it on every prompt, model or tool change. Then turn on step-level tracing of model calls and tool calls. OpenTelemetry's GenAI semantic conventions are the vendor-neutral option.

Demos use clean data and friendly APIs. Production means legacy systems, messy data, ambiguous tool interfaces, and customers who need a person when the agent is wrong.

Gartner: "Integrating agents into legacy systems can be technically complex, often disrupting workflows and requiring costly modifications."

Tool design matters as much as the prompt. Anthropic (vendor): "we actually spent more time optimizing our tools than the overall prompt." After it changed one tool to require absolute file paths, "the model used this method flawlessly."

And you stay responsible for what your bot says. In February 2024, Air Canada argued its chatbot was a "separate legal entity that is responsible for its own actions." The tribunal ordered it to pay "$812.02 (£642.64) in damages and tribunal fees." (A chatbot, not a tool-using agent, but the liability point carries over.)

This is the one I think teams underrate most: connecting to real systems and handing off to a human.

Try this week: rate every tool low, medium or high risk. OpenAI's agent guide (vendor) suggests judging by read-only vs write access, reversibility, required account permissions and financial impact. Put a human approval on the high-risk ones. Add a retry limit that hands off to a person; the guide's trigger is "Exceeding failure thresholds."

An agent calls the model many times per task, each time re-reading a growing context. Retries, sub-agents and loops multiply tokens and wait time.

Anthropic (vendor): "agents typically use about 4× more tokens than chat interactions, and multi-agent systems use about 15× more tokens than chats." Manus (vendor) says "the KV-cache hit rate is the single most important metric for a production-stage AI agent. It directly affects both latency and cost."

How much cost blocks projects isn't settled. LangChain says "cost is less frequently cited as a concern than in previous years," while Gartner names "escalating costs" as one reason it expects those cancellations. LangChain mostly hears from builders at small tech companies; Gartner surveys enterprise executives.

Try this week: add hard stopping conditions, such as a maximum number of iterations, and a per-task token budget. Keep your prompt prefix stable so caching works, and route easy requests to a smaller model (the "Routing" pattern in Anthropic's "Building effective agents").

If you only fix one of the five this month, make it number 4.

Sources for every figure and quote are on the web version.

Here are the first three project submissions in the weekly project display thread's Top view, excluding AutoModerator, as checked November 9, 2026, at 9:27 a.m. Pacific. Reddit's ordering is not a quality endorsement; descriptions come from project documentation, not hands-on testing.

Panda Federation Tools connects citation checks to MCP-equipped agents, including checks for nonexistent, mismatched or retracted scholarly references. Its server is MIT-licensed; documented Apify-backed runs are metered.

Original Reddit comment · Project repository

Platypus is a self-hosted, MIT-licensed platform for agents that can work beyond a chat session. Its documentation lists scheduled runs, MCP tools, Kanban boards and per-workspace sandboxes.

Original Reddit comment · Project repository

Rimplo's website describes agents that connect CRM, billing, support and product data to flag at-risk accounts, expansion signals and stalled deals with supporting evidence. These are vendor-described workflows, not results we've independently tested.

Original Reddit comment · Project website

I promised a survey in the first issue. Three quick questions, so I only send you what's useful:

It takes about 30 seconds.

Yujian

P.S. Which of the five has bitten you? Hit reply and tell me.