Why Meta Won't Let AI Handle Incident Response · Issue #185

Opening

Dear reader, the 2026 incident-response tooling market is full of lines like these: “Cut MTTR¹ by up to 70%.” “The moment an alert fires, we pinpoint the root cause in seconds.” “It’s like hiring a senior engineer who never sleeps and never loses context.”

But there’s a company that has been actually testing this promise for 5 years: Meta. More than 300 teams use its system, which runs 50,000 automated incident investigations a day. Last December, Meta published a paper on it.

The paper reported an average improvement of 20%. And here’s the stranger part: the heart of that system isn’t an AI agent.

Let me give you the conclusion up front. The bottleneck wasn’t the model. The bottleneck was that the organization’s knowledge hadn’t yet become code.

🌙 Every One of the 50,000 Daily Scripts Was Written by a Human

The system Meta published is called DrP. What it does is simple: when an alert fires, it automatically runs a pre-written investigation procedure and attaches the results to the alert page. Instead of an on-call² engineer, woken at 3 a.m., pulling up 5 dashboards and digging through logs, they start the investigation by reading analysis results that are already sitting on the page.

Meta calls this “pre-written investigation procedure” an analyzer. And this is where today’s story starts. An analyzer is code that a human wrote by hand, in Python or PHP — a chain of conditional branches and data queries, in other words, a script for incident investigation.

Let’s look at the scale: more than 2,000 analyzers, more than 300 teams using them, 5 years in operation, 50,000 automated analyses a day. Over 30 days that’s 1.5 million runs — broken down by the second, that means a script is running somewhere every 1.7 seconds.

These scripts replaced three things: wiki pages nobody updates, personal scripts scattered across individual laptops, and tribal knowledge³ that lives only in senior engineers’ heads. In the paper’s own words, their approach was to translate the manual playbooks and tribal knowledge spread across the organization into code.

The effect grew larger the more complex the investigation. When the paper compared 3 scenarios — a simple service error, a container failure, and a feature problem in an AI model — against the manual approach, the number of steps an on-call engineer had to take dropped by 4x to 20x. In the most complex scenario, the engineer’s job was compressed into exactly one task: reading the results on the alert page.

⚠️ One Subheading Buried in Chapter 9 of the Paper

So far, this is a familiar automation success story. But when you turn to Chapter 9, “Lessons Learned,” you find a subheading like this:

“Do not over-index on AI-based systems for diagnosis.”

A company running one of the largest AI infrastructures on Earth wrote this as a lesson in its own incident-investigation paper.

Let me clear up a misunderstanding first: this doesn’t mean Meta doesn’t use AI. DrP’s SDK is packed with statistical and machine-learning libraries for anomaly detection, time-series correlation, and dimensional analysis. There’s even a ranking model that scans through thousands of code and configuration-change events and says, “this is most likely the culprit.”

What Meta doesn’t hand to AI is the skeleton of judgment — what to look at, in what order, and where to go under which condition. Humans write that decision tree. AI is used as a tool that fills in a single cell inside it.

The paper spells out why. Pure machine-learning systems are limited by the quality of training data, the structure of the data actually accessible at investigation time, and the difficulty of tailoring workflows team by team. So the conclusion they reached was a combination: rule-based suggestions grounded in community expertise, with AI layered on top.

But what caught my attention more was the lesson right next to it, titled “Assistance vs. Full Automation.”

Meta’s original goal was full automation. But they changed course, for 3 reasons. First, systems keep changing, so scripts go stale fast. Second, statistics and machine learning produce false positives and false negatives. And third — culturally, engineers and on-call staff don’t always trust a fully automated system.

The first two are technical problems. The third isn’t. After 5 years, 300 teams, and 50,000 runs a day, the wall Meta hit wasn’t accuracy — it was trust.

The paper’s own internal survey backs this up. Asked “How often does DrP reduce your MTTR?”, only 5.9% answered “always.” The most common answer was “sometimes” (47.1%). The paper summarized this as “80%+ reported improvement,” which isn’t false, but the center of gravity in the responses clearly sits on “sometimes.” For reference, working backward from the percentages, the sample size looks to be about 17 respondents. That’s a very small sample for a system used by 300 teams, so this figure should be read as reference only.

📉 What Sits Between 20% and 80%

Now it’s time to look at the numbers — but there’s something to flag first.

Meta’s engineering blog described the outcome this way: “reduced MTTR by 20–80%.” It reads like a range. It’s easy to skim past it assuming the typical result is somewhere around 40–50%.

But the paper’s abstract says something different: an average of 20%, with 80%+ only for some teams. Same numbers, completely different meaning.

Open Chapter 7 of the paper and the reason becomes clear.

Fewer than 5 analyzers → 10–15%
10 or more → 50–80%
Company-wide average → 20%

In other words, the 80% wasn’t a property of the system’s performance — it was a function of how much a team had invested. How much of your own team’s investigation procedure you’d translated into code was what determined the outcome.

The per-team table in Section 7.2.4 of the paper makes this even clearer.

Team 1: 136 analyzers, 82% improvement
Team 2: 92 analyzers, 75%
Team 3: 66 analyzers, 84%
Team 4: 48 analyzers, 69%
Team 5: 39 analyzers, 58%
Team 6: 29 analyzers, 73%
Team 7: 23 analyzers, 56%
Team 8: 12 analyzers, 7%

Look at the last row. Team 8 built 12 analyzers — clearing the paper’s own “10 or more” threshold — yet its improvement was just 7%. The paper’s own claim is disproven by the paper’s own table. The real cliff sits somewhere between 12 and 23 analyzers.

Here’s something I calculated myself. Dividing 2,000 analyzers by 300 teams gives an average of 6 to 7 per team. And the paper itself says fewer than 5 means 10–15%. What does that mean? Even inside Meta, the median team is still in the low-yield zone. This is why the company-wide average stalls at 20%.

Time matters too. According to the paper, a simple analyzer can be built in a day, but fully capturing a team’s investigation workflow takes months. That’s not the time it takes to buy a tool — it’s the time it takes to transfer knowledge.

Finally, let me flag the absolute MTTR figures. Team 1’s went from 771 hours to 139 hours — 32 days became 5.8 days. This isn’t downtime; it’s the full cycle from detection to resolution for incidents that went through post-mortem review. If “20% MTTR reduction” sounds small, you need to look at how large the baseline actually was.

One more thing: recalculating Team 7’s row from its before-and-after values gives 36%, but the paper lists 56.1%. The other seven rows check out to the decimal point, so this is likely a typo. Still, it’s a good reason to run the numbers yourself before copying someone else’s table verbatim.

🔁 So Why “AI-Native” Now

The last paragraph of Meta’s blog post ends like this: they plan to evolve DrP into an AI-native platform going forward.

Doesn’t that seem strange? They write “don’t over-index on AI,” and in the very next paragraph say they’re heading toward AI-native.

It’s not a contradiction. It’s sequencing.

Let’s look again at what Meta actually did over the past 5 years. It took investigation knowledge that was scattered across wiki pages, personal scripts, and senior engineers’ memories, and turned it into this:

Type-defined code (investigation procedures made explicit as branches and conditions)
Structured output (results come out in machine-readable form)
Data with a run history (30 days of past investigation inputs and outputs are retained)

The third point is the decisive one. Every time it modifies an analyzer, Meta replays past investigation records through it to verify the change — a backtesting⁴ system. The original purpose was catching bugs. But as a result, the incident-investigation process itself became a labeled dataset.

Let’s flip the question here. What does an AI agent need in order to investigate an incident?

First, hands — it needs tools to access data. Second, a procedure — it needs a map of what to look at and in what order. Third, an answer key — it needs a record of which past judgments were right. What DrP built over 5 years is exactly these three things.

There’s another lesson in the paper, titled “Data Is Everything.” The quality of observability data, plus structured metadata like service dependencies and data lineage — without these, correlation analysis simply can’t happen. No matter how good a model you attach, there’s nothing for it to attach to.

So here’s the answer to today’s headline. Meta didn’t withhold work from AI because it didn’t trust it. The sequence just hadn’t come up yet. And now, they’ve judged that it has.

For AI to do a job in your place, that job has to become code first. No matter how good the model gets, if an organization’s knowledge lives in wiki pages and someone’s memory, AI has nowhere to attach.

🇰🇷 So Where Do We Stand

At a conference held in Seoul on June 24, the Cloud Team from Samsung Electronics’ MX (Mobile eXperience) division presented its own roadmap. It’s the central SRE⁵ organization responsible for the reliability of more than 50 customer-facing services, including Samsung Pay, Bixby, and the Galaxy Store.

They laid out 4 maturity stages: reactive response, automated response, predictive operations, and autonomous operations. Their own diagnosis put them at stage one, with a target of 2028 for autonomous operations. If this is how one of Korea’s top-tier SRE organizations rates itself, it gives you a rough sense of where everyone else stands.

What’s worth noting is the sentence Group Leader Hyunseong Yoo used to close the presentation: even as the scope of automation expands, final responsibility and judgment remain with humans. That’s exactly the same conclusion as Meta’s paper’s “Assistance vs. Full Automation.”

At the same event, Woocheol Byun, Head of KT’s Palantir Business Division, put it more bluntly: management’s impatience leads companies to bolt on AI before touching the root problem, which is why results don’t show up. The solution he pointed to was organizing data into a structure AI can actually read — the same point as the paper’s “Data Is Everything.”

There’s also a Korean case actually running in production. In April, Yanolja disclosed that 14 people across 6 teams built 6 operations agents in 6 weeks. Among them, the incident-response agent cut the time from incident occurrence to post-mortem report from 2 weeks down to 24 hours. But look under the hood, and here too the agent runs on top of a foundation of searching an internal knowledge base and documents. The sequence is the same.

Oz’s Lens

I went back and counted the six lessons in Chapter 9 of the paper. Not a single one is about model performance. Every one is about adoption.

Let the community build it themselves. Go to where the users already are. Embed it inside the workflow they’re already using. One line in particular caught my eye: building analyzers isn’t most software engineers’ day job. So Meta refined its SDK so anyone could build one in a day, and made results viewable directly inside the code editor.

In more than 20 years of building GTM strategy, I’ve seen the same pattern repeat itself. The first question an organization asks when evaluating a new tool is always “how accurate is it?” But what actually determines adoption is almost always “how annoying is it?” A tool that’s 95% accurate but requires opening three more windows won’t get used. A tool that’s only 70% accurate but surfaces results on a screen you’re already looking at will. Meta pushing results directly into the alert page wasn’t a technical choice — it was a GTM choice.

That’s why I think a lot of organizations are asking the wrong question right now. It’s not “which AI agent should we adopt?” The question to ask first is this: how many of the judgment procedures our team repeats every time already exist as code? If that number is close to zero, it doesn’t matter which model you buy — there’s nowhere for it to attach.

Closing

The heart of the incident-investigation system Meta has run 50,000 times a day for 5 years isn’t an AI agent — it’s 2,000 scripts written by hand, by humans. And what determined the outcome wasn’t the model, but how many scripts had been written. The only reason Meta can talk about “AI-native” now is that it spent 5 years translating tribal knowledge into a form machines can read.

If you want to dig deeper, just open Chapter 9, “Lessons Learned,” of the paper. Three pages hold 5 years of trial and error.

Think of one judgment procedure that repeats in the same order every time on your team. If it still lives only in a wiki or in someone’s head, why hasn’t it become code yet? Is it lack of time, or does the procedure differ from person to person? Tell me the biggest obstacle in the comments, and I’ll fold it into a future issue.

💬 Share your experience with the question above in the comments — I’ll fold it into a future issue. 📨 If you have a colleague weighing operations automation or AI adoption, pass this along.

Every Tuesday, Thursday, and Sunday, I read across technology, economics, and the humanities, and write about what I find at their intersections. Some days, like today, mean digging all the way through a single paper.

References & Further Reading

Primary sources

Shubham Somani et al. (14 authors), “DrP: Meta’s Efficient Investigations Platform at Scale”, arXiv:2512.04250 [cs.SE], December 2025. : Section 7.2 (MTTR evaluation) and Section 9 (Lessons Learned) are the core evidence behind today’s piece. If you’re short on time, the three pages of Section 9 are enough.
Meta Engineering, “DrP: Meta’s Root Cause Analysis Platform at Scale”, December 19, 2025. : This is the official summary of the paper. Note that it compresses MTTR into “20–80%,” which shifts the nuance from the original. If you’re citing figures, I’d recommend going to the paper itself.

Background

Rootly, “What Is an AI SRE Agent? How AI Is Changing Incident Response in 2026”, April 2026. : You can see the tone in which the market promises quoted in today’s Opening are being sold. Read it with vendor-content caveats in mind, but its four-stage autonomy framework is worth placing side by side with Samsung’s four maturity stages.
Google, “Being On-Call”, Site Reliability Engineering. : The original source on why the on-call system exists and why it burns people out. Meta’s paper cites this document in its footnotes too.
ZDNet Korea, “AI That Can’t Read Data Is Useless: KT and Samsung’s Solutions”, June 24, 2026. : Samsung Electronics’ four-stage SRE roadmap and KT’s diagnosis of data structure both appear in one piece. The most useful source for placing the Korean landscape.
Edaily, “Samsung to Tackle Cloud Outages with AI, Targets Autonomous Operations by 2028”, June 2026. : Lays out Samsung Electronics’ numerical targets from the presentation in more detail (90% cut in recovery time, 99% detection rate within 10 minutes, and more).
AWS Tech Blog, “Yanolja’s Case: Building an AIOps Agent with Strands SDK and Bedrock AgentCore”, April 2026. : Lets you look inside the architecture of a real, running operations agent in Korea, sprint by six-week sprint.

📝 Glossary

The author, Kwangseob Ahn, is a professor of business administration at Sejong University and lead consultant at OBF (Oswarld Boutique Consulting Firm). He teaches statistics and data analysis, including business data management and business analytics, while leading GTM and AI strategy consulting in the field, designing the seam between technology and business. He has published academic research on a memory architecture for AI dialogue systems (HEMA) and runs Daily Arxiv, a daily curation of global AI papers. He holds a master's from Korea University's Graduate School of Technology Management and a KMBA. He is the author of Homo Brainless: The People Who Outsource Their Thinking.

Footnotes

MTTR (Mean Time To Resolution): the average time from when an incident is detected to when it’s resolved. Because Meta’s paper only calculated this for incidents that went through post-mortem review, you should read it as the full cycle of root-cause analysis and remediation — not just downtime. ↩
On-call: a rotation system where someone is designated to respond immediately when an incident occurs. If an alert fires at 3 a.m., the person who has to wake up is that day’s on-call. ↩
Tribal Knowledge: knowledge that isn’t written down anywhere and is passed on person to person within an organization — things like “when that server acts up, check there first.” ↩
Backtesting: verifying that changed code produces the same results as before by replaying it against past data. Meta keeps 30 days of investigation records and runs this automatically every time an analyzer is modified. ↩
SRE (Site Reliability Engineering): a discipline and organization that manages service reliability using software-engineering methods. It began at Google in 2003, and on-call rotations and incident response are among its core responsibilities. ↩