Aegis: A Root Cause Engine for Distributed Telemetry

Preamble

We operate a distributed system at scale. When it breaks, we know what broke and where — the affected service, the degraded operation, the time it started. This takes minutes.

We do not know why.

The answer exists somewhere in our telemetry — buried in a log line, encoded in a metric curve, hinted at in a trace fragment, or recorded in a change ticket. But these artifacts live in separate systems, speak different languages, and carry no common thread. Bridging them is a human act: an engineer opens half a dozen consoles, triangulates timestamps, forms a mental model of dependencies, and manually reasons backward from symptom to cause. This takes hours. Sometimes days. Every single time.

This is not a tooling gap. It is a reasoning gap. And we intend to close it.

I. The Aegis Thesis

Correlation is not causation, but it is the raw material from which causation can be inferred. If we can fuse change records with telemetry signals across all three pillars — logs, metrics, traces — and if we can construct even a partial map of how our services depend on one another and on their infrastructure, then we can automate the hypothesis-generation step of root cause analysis. We can give the operator a ranked, evidence-backed causal timeline in under five minutes, instead of a blank search bar and a prayer.

Aegis does not aim to replace the engineer. It aims to compress the mechanical, multi-system search-and-correlate phase — the part that burns time without burning insight — so that the engineer starts their investigation already holding a set of plausible explanations, each backed by evidence drawn from the systems they would otherwise query by hand.

II. The Signals We Have

We possess three families of telemetry, each with different character:

Logs

The richest and messiest signal. Some structured, some semi-structured, some nearly free-form text. They carry error messages, latency measurements, status codes, and — where maintained — ownership attribution and caller identity. They are the ground truth of what the application experienced.

Metrics

The cleanest signal. System metrics, application metrics, infrastructure metrics, trace-derived metrics. Well-typed, regularly sampled, pre-aggregated. They tell us that something changed and when, with precision. An existing anomaly pipeline already computes per-metric confidence intervals continuously.

Traces

The connective tissue — where they reach. Distributed tracing gives us span-level visibility into request flow, service-to-service call graphs, and latency propagation. But instrumentation is incomplete: critical infrastructure components, particularly database systems, are excluded from tracing due to agent stability concerns. The dependency graph has holes where it matters most.

Beyond Telemetry

We also possess a rich change record: every database modification, software deployment, and configuration alteration is recorded with precise timestamps through a formal approval pipeline. We know who changed what and when. What we lack is the connection from change to consequence — the blast radius, the dependent services, the expected behavioral signature.

III. The Gaps We Must Bridge

Aegis is defined as much by what is missing as by what is present:

No universal correlation identifier. Some signals carry trace IDs, request IDs, or transaction IDs. Most do not. Cross-signal correlation today relies on temporal proximity and manual search.
Traces amputated at the infrastructure boundary. The most critical dependency — the persistence layer — is invisible to our distributed tracing. When a database degrades, we see the symptoms in the services that call it but cannot see inside.
A fragmented dependency graph. No single source maps which services depend on which other services, databases, or infrastructure. Tracing provides partial edges. Logs encode caller-callee relationships in fields that are being deprecated. The configuration management database aspires to centrality but is not yet complete.
Change events carry no blast radius. A database administrator alters a parameter. The change is recorded. But nothing in that record says "this database serves these twelve services." That knowledge lives in human minds, scattered across teams.
Ownership decays. Service ownership fields, where they exist, are populated at creation time and rarely maintained. A field written five years ago may point to a team that no longer exists.

Aegis must work despite these gaps. It cannot wait for all data to be clean, all instrumentation to be complete, or all dependencies to be declared. It must make defensible inferences from partial information and express its uncertainty honestly.

IV. What Aegis Does

Given a degraded service and a time window, Aegis returns:

A ranked causal timeline. A chronologically ordered sequence connecting change events to anomaly onset to symptom emergence, with each candidate root cause scored by confidence and accompanied by supporting evidence drawn from logs, metrics, and traces.

The operator receives this as a report — pushed to a communication channel and persisted in our log aggregation system for visualization — within five minutes of trigger.

Aegis is triggered in two ways:

Automatically, when the anomaly pipeline detects a metric exceeding a severity threshold.
Manually, when a platform operator initiates an investigation during an incident or post-incident review.

In its first version, Aegis is non-interactive. It generates a complete report. Drill-down, hypothesis challenging, and iterative refinement are deferred to later iterations.

V. How Aegis Reasons

The Engine: Heuristics First

Aegis v1 uses explicit, auditable rules. Each rule encodes a causal pattern: a type of change event, a temporal window, an expected signature in telemetry, and a scoring function. Rules compete against each other for explanatory power. Aegis ranks candidates; it does not pronounce a single verdict.

We begin with rule-based reasoning because it is transparent, debuggable, and requires no training data. It can be deployed immediately. Statistical and machine learning approaches are natural extensions when we have accumulated a labeled corpus of resolved incidents.

Candidate Rule Families

Change-Induced Degradation. A change occurs on an entity. Within a bounded temporal window, dependent services exhibit anomaly onset, error rate increases, and novel error patterns in logs that were absent before the change.

No-Change Degradation. An anomaly with no proximate change event. Suggests gradual resource exhaustion, external dependency failure, or traffic anomaly. Evidence is drawn from metric saturation curves and upstream correlation.

Cascading Failure. Service A degrades. Service B, which depends on A, degrades with a temporal lag consistent with propagation. Service C follows. Evidence is drawn from trace spans and temporal ordering across the dependency graph.

Infrastructure Event. Multiple unrelated services degrade simultaneously. Correlated with a host-level or network-level anomaly. Evidence is drawn from infrastructure metrics and network flow data.

The Dependency Graph: Synthesized, Not Discovered

Since no single source provides a complete dependency graph, Aegis constructs one from fragments:

Trace spans from distributed tracing systems, where instrumented.
Caller-callee relationships inferred from log fields.
Application-to-host mappings from the configuration management database.
Network flow data, mapped to services.
Operator-declared known dependencies, weighted as high-confidence edges.

The graph is constructed lazily, scoped to the investigation at hand. It is never expected to be complete — only useful. Every edge carries a confidence weight, and Aegis propagates uncertainty through its inferences.

Correlation vs. Causation

Aegis employs multiple strategies to avoid conflating coincidence with cause:

Rank, do not decide. All plausible candidates are presented. The operator retains judgment.
Intervention analysis. If a changed entity serves N dependent services and all N degrade, the causal link is strong. If only one degrades, the link is weak.
Temporal precedence. Changes that occurred after symptom onset are excluded as causes.
Counterfactual weakening. If the same or similar change occurred previously without degradation, the hypothesis is discounted.
Signal convergence. Hypotheses supported by multiple independent telemetry pillars gain confidence; those supported by only one are penalized.

VI. What Aegis Does Not Do

Clarity about scope is as important as clarity about function.

Aegis does not:

Replace the on-call engineer. It gives them a head start, not a final answer.
Automate remediation. It identifies probable causes; it does not trigger rollbacks or configuration reversions.
Require perfect data. It is designed to degrade gracefully with partial information, expressing lower confidence when signals are sparse.
Operate continuously in v1. Investigations are triggered on-demand; continuous pre-computation of causal indices is a future optimization.
Provide interactive drill-down in v1. The initial report is a complete artifact; iterative questioning comes later.

VII. The Measure of Success

Aegis v1 is successful if:

Platform operators receive a ranked causal hypothesis list in under five minutes from trigger.
The true root cause appears in the top three candidates in the majority of investigations, as measured against post-incident review outcomes.
Incidents resolved during the event increase, because root cause is identified while the incident is still active — reducing the volume of formal post-incident reviews.
Operators adopt Aegis as their first investigation step, displacing manual multi-system querying as the default workflow.

VIII. The Path Forward

This manifesto describes intent, not implementation. The next steps are concrete:

Ratify the scope. Confirm the heuristic-first approach, trigger modes, output format, and latency target.
Design the initial rule catalog. Convene domain experts — service owners, database administrators, incident commanders — to articulate the causal patterns they recognize today.
Prototype the dependency graph construction. Build a throwaway integration against real data sources and measure coverage against a sample of known service dependencies.
Define data contracts. Formalize the query interfaces and response schemas for each source system.
Establish an evaluation baseline. Instrument current manual investigation workflows to capture time-to-root-cause for a representative set of historical incidents, so we can measure Aegis against reality.

Aegis is not a dashboard. It is not an alerting system. It is a reasoning prosthesis — a machine that does the mechanical correlation work so that the human can do the thinking work. Our operators deserve to start their investigations with a hypothesis, not a blank page.