Skip to main content
Matt Asbury

The Future of Self-Healing Systems

•7 min read•Written by a human, edited by AI
AIInfrastructureAgentsDevOpsReliability

Recently I was in a conversation with a new Staff-level engineer about what "self-healing" systems actually mean. I was describing the ideal state I'm aiming for in terms of resilience — and then I paused. What does that really look like, beyond the usual picture of Kubernetes, monitors, and auto-cycling a bad pod?

Most people's mental model of self-healing stops there: health checks, restart the pod, maybe scale up. Useful, but it's not healing in the sense of understanding and fixing what went wrong. So I started thinking about what it looks like when you have autonomous agents in the loop — agents that can be notified when something's wrong, capture the issue, open a ticket, find the root cause, implement a fix, test it, and open a PR for a human to review before it ever ships.

Here's the vision I'm working towards.

Self-Healing Today: Where Most Teams Are

Today's "self-healing" is mostly reactive recovery:

  • Health checks — Liveness and readiness probes, auto-scaling on CPU or memory
  • Kubernetes pod recycling — When a pod fails a probe or OOMs, k8s kills it and starts a new one
  • Alert-based runbooks — PagerDuty fires, someone runs a playbook or rolls back a deploy

These systems can restart and scale. They can't fix. A bad deploy, a logic bug, or a misconfiguration still needs a human to roll back, diagnose, and patch. The loop breaks as soon as something requires understanding why it failed.

The Missing Loop

Traditional self-healing is about recovery (get back to a known-good state) rather than repair (address the underlying cause). The full loop I care about looks like this:

Detect → Diagnose → Fix → Test → Review → Deploy → Verify → Close

Today, we might automate some recovery steps — runbooks that roll back or scale — but corrective repair, the kind that requires understanding why something failed and changing code or config, still hands off to a human after Detect. The question is: what happens when agents can carry the loop further?

What Agent-Driven Self-Healing Looks Like

Here's a concrete pass through each stage, with agents in the loop and a human only at the approval gate.

1. Detection and Notification

Agents subscribe to your monitoring and observability stack — Datadog, Sentry, PagerDuty, or whatever you use. When an anomaly, error spike, or failed health check fires, the agent is notified first, not necessarily a human. The agent receives the same context a human would: error type, stack traces, affected services, and metrics.

2. Issue Capture and Triage

The agent captures the full context — stack traces, relevant logs, metrics, affected users — and creates a structured ticket in Jira or GitHub Issues. The ticket isn't a raw paste of logs; it's a summarised incident with severity, impact, and enough detail for both humans and downstream automation. When impact and SLAs are well-defined, the agent can triage severity accordingly.

3. Root Cause Analysis

The agent correlates the failure with recent deployments, config changes, and code diffs. It reads the codebase, follows the failing code paths, and works towards root cause — not just "the service returned 500" but "this null check was missed after refactor X." That might mean pulling in Sentry stack traces, deployment history, and git blame. The output is a clear diagnosis that a human (or the next step) can act on.

4. Fix Generation

The agent proposes a targeted code change that addresses the root cause. Not a blind rollback, but an actual fix: add the null check, fix the config key, correct the query. The change is scoped and described in terms of the diagnosis.

5. Testing

The agent runs the existing test suite and, where it makes sense, adds or extends tests that cover the failure case. It validates that the fix resolves the issue and doesn't regress other behaviour. If tests fail, the agent iterates (or escalates) rather than shipping a broken fix.

6. Pull Request for Human Review

The agent opens a pull request with a clear description: what broke, why, what the fix does, and what tests were added or updated. A human reviews and approves. No deploy happens until that approval. This is the critical handoff: the agent does the legwork; the human provides judgement, context, and accountability.

7. Deployment and Verification

Once the PR is approved and merged, your existing CI/CD pipeline deploys the fix. The agent doesn't trigger the deploy; it monitors the rollout and post-deploy metrics to confirm the issue is resolved — error rate back to normal, health checks green — and can post a short summary back to the ticket.

8. Issue Closure

The agent updates the Jira or GitHub issue with the resolution: what was done, link to the PR, and any follow-up notes. Then it closes the ticket. The loop is closed; the system didn't just "recover," it repaired and left an audit trail.

The Human in the Loop

This isn't about removing humans. It's about changing what they spend time on.

  • Why the approval gate matters — Trust, accountability, and edge cases. The agent can miss business context, subtle race conditions, or trade-offs that only a human knows. The PR review is the place to catch that.
  • How on-call changes — Instead of "wake up, grok the logs, fix it," the human might wake up to a well-documented ticket and a ready-to-review PR. They can approve, request changes, or take over. The cognitive load shifts from discovery and implementation to review and decision.
  • Progressive trust — Start with low-severity, auto-generated fixes and expand as confidence grows. Not everything has to be agent-driven on day one.

What Needs to Be True

Honest take on what has to be in place for this to work:

  • Observability maturity — Agents need structured, rich telemetry. If your monitoring is "we have some logs," the agent has nothing to reason about. Good traces, error grouping, and deployment correlation matter.
  • Test coverage — The agent can only validate fixes if there are tests to run and extend. Without tests, "fix" is hope, not verification.
  • Codebase comprehension — Agents need to navigate and understand the codebase. That means code indexing, context windows, and possibly embeddings or RAG over your repo. The better the agent's view of the system, the better the fixes.
  • Guardrails and blast radius — Limits on what the agent can change (e.g., only certain dirs, no prod config from the agent alone). You want to contain mistakes.
  • Trust and auditability — Full trace of what the agent did: which alerts it reacted to, what it concluded, what it changed, and why. So when something goes wrong, you can debug the agent's behaviour too.

We're Closer Than You Think

The pieces are already emerging:

  • Sentry's Seer and similar tools do automated root cause analysis and suggest likely culprits.
  • GitHub Copilot and Cursor generate code in context; the jump from "suggest a fix" to "open a PR" is being bridged.
  • Kubernetes operators and AWS Auto Scaling already do automated remediation for well-defined failure modes.

The gap between "detect" and "fix" is narrowing. The next step is connecting detection to diagnosis to code change to PR, with a human only at the review step.

Closing the Loop

So back to that conversation with the Staff engineer. The ideal I'm aiming for isn't "restart and hope." It's: the system understands what went wrong and can propose a real fix, with tests and a PR, so a human can review and ship. The human's role shifts from firefighter to reviewer. That's the future of self-healing I want to build towards.

Share: