Hard system-design observability

How would you correlate deploys, alerts, logs and traces into ranked root-cause hypotheses?

The 30 second answer

I collect typed, time-stamped signals per entity and search for ordered paths between them, such as commit, deploy, error, alert, incident. Known causal pairs get a strong prior; other links must earn their place through time proximity and severity inside a window, and weak chains are dropped. Each chain is stored with its evidence and confidence, because it is a hypothesis to check, not a conclusion.

Signals. Netflix's engine reads thirteen signal types: log errors, metric anomalies, slow traces, failed traces, alerts, incidents, deployments, config changes, experiments, autoscaling events, commits, end-to-end bottlenecks and error patterns. It searches for paths among them.

Scoring as Netflix reports it. Some ordered pairs are treated as known causal with a prior weight of 0.9: deployment before log error, config change before metric anomaly, alert before incident, failed trace before incident. Other links are scored as time proximity × 0.6 + severity × 0.4, inside a ten-minute window, and chains below 0.6 confidence are discarded. The example chain is commit "fix retry logic", 14 minutes, deployment v1.4.2, 3 minutes, a ConnectionTimeout log error, 2 minutes, a p99 > 500 ms alert, 5 minutes, incident INC-5377, with confidence 0.87.

# Illustrative chain scoring. Netflix reports score = time_proximity * 0.6 + severity * 0.4,
# a 10-minute window and a 0.6 minimum; how proximity and severity are normalised is not
# published, so the linear proximity below is an assumption.
WINDOW_MIN, MIN_CONF = 10, 0.6
CAUSAL_PRIOR = {("DEPLOYMENT", "LOG_ERROR"), ("CONFIG_CHANGE", "METRIC_ANOMALY"),
                ("ALERT", "INCIDENT"), ("TRACE_FAILED", "INCIDENT")}

def link_score(a, b, severity):              # a, b: (type, minute); severity in [0, 1]
    gap = b[1] - a[1]
    if not 0 <= gap <= WINDOW_MIN:
        return None                          # outside the window, or b happened first
    score = (1 - gap / WINDOW_MIN) * 0.6 + severity * 0.4
    return score if score >= MIN_CONF else None

def chains(events, severity, path=None):     # depth-first search over time-ordered events
    path = path or []
    if len(path) > 1:
        yield path
    for e in events:
        if path and (e in path or link_score(path[-1], e, severity[e[0]]) is None):
            continue
        if not path or e[1] >= path[-1][1]:
            yield from chains(events, severity, path + [e])

ev = [("DEPLOYMENT", 0), ("LOG_ERROR", 3), ("ALERT", 5), ("INCIDENT", 10), ("AUTOSCALE", 40)]
sev = {"DEPLOYMENT": 0.5, "LOG_ERROR": 0.6, "ALERT": 0.8, "INCIDENT": 1.0, "AUTOSCALE": 0.2}
best = max(chains(ev, sev), key=len)
print([t for t, _ in best], [(a[0], b[0]) in CAUSAL_PRIOR for a, b in zip(best, best[1:])])

It prints ['DEPLOYMENT', 'LOG_ERROR', 'ALERT', 'INCIDENT'] [True, False, True]: the unrelated autoscaling event 30 minutes later is excluded, and two of the three links match a known causal pair.

Keep it honest. Time order and proximity show correlation, not causation. Restrict candidates to entities connected in the dependency graph, so a deploy of an unrelated service in the same minute does not qualify. Treat experiments and autoscaling as possible confounders, since they often change at the same time as load. Store each chain with the records it was built from, as Netflix does, so a person or agent can see why a chain exists. Opinion: evaluate the ranking against postmortem root causes, using top-1 and top-3 hit rate and time to correct owner, and recalibrate weights from that, not from intuition.

From memory to rules. Netflix describes a flywheel of observe, enrich, infer and adapt: a chain confirmed in one incident becomes a known pattern, and established patterns become detection rules. Its examples, "api-gateway deploys cause 40% of SEV-2s" and "30% of deployment issues in api-gateway come from Connection Timeouts", show what such an insight looks like; the post does not say whether they are measured. Opinion: before promoting a pattern to a rule, check that it holds on enough incidents and still holds after a quarter.

Alternate interview scenarios

"Your RCA system confidently blamed a deploy, but the real cause was a downstream config change. How do you stop that from happening again?" I would check whether the config change was harvested at all, whether the two services are linked in the dependency graph, and whether the deploy won only because of its 0.9-style prior and its closeness in time. Then I would feed the postmortem back as a labelled example, show several ranked hypotheses with evidence instead of one answer, and require a person to confirm before any automatic rollback.

Sources (1)
  1. Netflix TechBlog: Ontology-Driven Observability: Building an End-to-End Knowledge Graph at Netflix Scale