The 30 second answer
I narrow the output contract instead of trusting the model. Deterministic extraction runs first, then catalog validation, then API lookups, and only then an LLM whose output must be a fixed enum or a schema-checked JSON object. Each fact is validated, typed and stored with its source and confidence. Running enrichers in order removes races, slow sources only degrade a harvest, and humans approve schema changes.
A ladder from cheap and certain to expensive and uncertain. Netflix describes five layers:
- Deterministic extraction with regular expressions: trace IDs, incident references, user mentions. The same input gives the same triples.
- Catalog validation: fuzzy match, then check against the software catalog.
- API enrichment: user IDs become identity, team and on-call through live directory and paging APIs.
- LLM classification: the model may write anything, but its result must fit a constrained set of predicates, such as a category of incident, question, discussion or decision, and a sentiment of positive, neutral, negative or urgent.
- Probabilistic correlation: a search over weighted signals, such as deploy, then error, then alert, then incident.
What the contract guarantees, and what it does not. A schema guarantees that a fact is well formed and uses the vocabulary. It does not guarantee the fact is true, and rerunning a stochastic step can change it. Netflix says the pipeline as a whole becomes deterministic; more precisely, a fixed graph snapshot answers a query the same way every time, while a new harvest may still produce different facts. So I would store the model, prompt version, source message and confidence with each derived fact, keep deterministic facts apart from inferred ones, spot-check inferred facts against labelled samples, and let consumers filter by confidence.
Enricher contract. Each enricher reads noisy input, processes it, possibly stochastically, emits typed triples and reports how many it added. Later enrichers read what earlier ones wrote. Netflix runs them sequentially, loading the graph, adding triples and saving, which removes a class of race conditions at a cost in wall-clock time. External calls are best-effort, so a slow dependency degrades the graph instead of blocking it.
Discussion is data. A large incident channel holds the hypotheses tried, the mitigation that worked and who knew why. Netflix's example is a channel of 200+ messages, 50+ people and 15+ services. Turned into facts such as "deploy 1.4.2 affected api-gateway" and "chain-1 has root cause deploy 1.4.2", that reasoning outlives the channel. The post calls these edges the most valuable in its graph.
Humans approve. Agents may propose fixes, enrichers and ontology changes, but every proposal goes through review.