We put an AI bot between our alerting and our on-call engineer. It reads the logs, the traffic and the recent deploys, then comments on the incident before anyone has even opened Slack.

The World Cup is upon us, and with games kicking off through the night for our UK team, time to resolution on overnight incidents matters more than it ever has.

Improving incident turnaround seems like a perfect use case for AI. We see it already when we ask Claude Code to fix bugs. It reads our logs and our codebase way faster than we can. Here's what our incident process looked like:

Before

• Alert fires, incident.io opens the incident and a Slack channel.
• Phone buzzes at 2am, maybe a couple of times.
• Engineer wakes up and finds their laptop.
• Opens the error tracker. Reads the stack trace.
• Opens CloudWatch. Works out whether it's still happening, or whether it was a spike five minutes ago that has already passed.
• Opens the analytics. Tries to answer the question everyone is about to ask, which is how many real users hit this.
• Opens GitHub. Checks what shipped in the last 24 hours.
• Starts forming a theory.

Somewhere after all of that, the actual incident response begins.

After

• Alert fires, incident.io opens the incident and a Slack channel. It also pings the webhook URL for our AI bot.
• The bot reads the error, then accesses AWS through MCP to look at the metrics and logs.
• The bot accesses a cached store of previous incidents and resolutions, and looks for similarities.
• The bot reads PostHog for live traffic from the matching user IDs.
• The bot checks what shipped recently.
• It comments on the incident with what it found: the error, whether it has happened before, whether it's still ongoing, how many sessions are affected, and what changed in the last few hours.
• Phone buzzes.
• Engineer wakes up, opens Slack, and reads the bot's analysis.

With this new process, the engineer opens Slack and the information they need to make a decision is already there.

We've taken the data collection away from the sleep-deprived human. The bot doesn't get tired, and it doesn't need to remember which dashboard has the deploy history on it.

We built this for the middle of the night, but it has turned out to matter just as much at 11am on a Tuesday. Every incident starts with the same round of looking things up, whether or not anyone was asleep.

Gathering that information used to take 20 to 30 minutes, and more than one engineer if the incident was severe enough. Now we reach the decision point within a few minutes.

What the engineer still does

Everything that involves a decision. Roll back or fix forward. Escalate or hold. Tell customers now or wait for the next update. The bot has never made one of those calls and I don't want it to.

It stays in the channel afterwards, so the engineer can ask it to check a related service, pull a specific log range, or compare traffic to the same time last week.

If there's a fix worth making, it can raise a ticket and open a PR for a human to review.

Data incidents

Not every incident starts with an error. Running during live events, we've found that a number of ours come from missing or late data instead. So we built an AI bot that watches the games and expects certain events to arrive (there should always be a kick-off, for example). If an event is missing, the bot looks into it, and if our data doesn't match other sources it can raise the incident itself.

A bot worth building

Most AI tooling saves an engineer a few minutes on a good day. Incident triage is different for two reasons. The task is well defined, so the bot is looking for facts that can be checked rather than offering an opinion. And the payoff arrives at the worst possible moment, when your team is tired and stressed.

The tools were always there to answer those questions in seconds. We just needed something quick enough to go and ask them.

How much of your incident process is someone gathering information a bot could have collected already?