Hard system-design observability

How do you model entities and resolve identity across tools that name things differently?

The 30 second answer

Identity comes first. I pick one authoritative catalog per entity kind and validate every candidate against it before creating a node, so a scraped string never becomes an application. I avoid vague types like service, give every entity one type and one URI scheme, and keep concepts separate from instances. Popular services get capped dependency edges, so one hub cannot dominate traversals.

One identity per real thing. Names drift by suffix, casing or renames. Resolve them in two steps: deterministic normalisation and fuzzy matching to propose a candidate, then validation against the authoritative catalog. Netflix validates every application against a software catalog of roughly 49,000 entries before creating it, so usernames, team names and dependency names scraped from network topology never become applications. Unresolved names are better kept as unlinked strings or a review queue than turned into new nodes, because a false merge silently joins two systems' histories.

Refuse ambiguous types. Netflix rejected a generic Service type because "service" meant four different things across its tools; every application resolves to exactly one type, ops:Application. Opinion: the same applies to cluster, deployment and environment. If a word means different things in different tools, model each meaning as its own type, with explicit relationships between them.

Invariants. Netflix's enrichers all keep three: every entity carries an rdf:type, every URI follows a single naming scheme, and the catalog gates every application. Deterministic URIs, for example derived from catalog IDs rather than display names, make re-harvests idempotent: the same input produces the same node instead of a duplicate.

Concepts versus instances. Types and relationships change on a human timescale and through review. Instances such as deploys, alerts and incidents change continuously. Keeping them apart lets you evolve the schema without rewriting data, and lets you treat the ontology as code: versioned, reviewed, tested and changed through pull requests.

Hub nodes. One popular service can have thousands of reverse dependencies. Netflix caps dependency edges in both directions so such a node does not dominate the graph; the post does not give the cap. Opinion: I would also limit traversal depth and fan-out per query, and rank neighbours by call volume or error contribution rather than returning all of them.

Sources (2)
  1. Netflix TechBlog: Ontology-Driven Observability: Building an End-to-End Knowledge Graph at Netflix Scale
  2. W3C: RDF 1.1 Concepts and Abstract Syntax (triples, datasets, named graphs)