Hard system-design observability

Why use RDF, OWL, SHACL and SPARQL rather than a property graph, and how do you query and store it?

The 30 second answer

RDF gives a standard fact model, OWL describes the ontology, SHACL validates data against shapes, and SPARQL queries it, including arbitrary-length paths. That makes the model portable across stores. One catch: OWL assumes an open world, so missing facts are unknown, not false, which is why closed validation belongs in SHACL. A property graph is a fine alternative if the team values its tooling more than standards.

What each standard does. RDF 1.1 represents information as subject-predicate-object triples whose elements are IRIs, blank nodes or literals; an RDF dataset is a default graph plus named graphs, which is what a quad store holds. Named graphs are a natural place to record which harvest or source produced a fact. OWL 2 describes classes and properties for reasoning. SHACL validates RDF graphs against a set of conditions written as shapes, and returns a validation report. SPARQL 1.1 matches graph patterns, supports aggregation and negation, and has property paths that match connections of arbitrary length. Turtle is the compact text syntax, which Netflix uses for per-harvest archives because the files are readable and diff well.

The trap. The OWL 2 Primer contrasts databases, where a missing fact is usually considered false (closed world), with OWL, where it may simply be missing but true (open world). So an OWL axiom that an incident affects an application will not reject an incident with no application; a reasoner may just infer that some unknown application exists. If a rule must hold before data is accepted, write it as a SHACL shape.

from rdflib import Graph
from pyshacl import validate

data = """
@prefix ops: <https://example.org/ops#> .
@prefix tl:  <https://example.org/timeline#> .
ops:api-gateway a ops:Application ; ops:ownedBy "Team Bedrock" ;
    ops:deployedIn "us-east-1" ; ops:dependsOn ops:zuul .
ops:zuul a ops:Application ; ops:dependsOn ops:edge-auth .
ops:edge-auth a ops:Application .
ops:INC-5377 a ops:Incident ; ops:affects ops:api-gateway ; ops:severity "SEV 2" .
ops:INC-9001 a ops:Incident ; ops:affects ops:billing ; ops:severity "SEV 3" .
"""
g = Graph().parse(data=data, format="turtle")

q = """PREFIX ops: <https://example.org/ops#>
SELECT ?incident ?sev WHERE {
  ?incident a ops:Incident ; ops:affects ?app ; ops:severity ?sev .
  ?app ops:ownedBy "Team Bedrock" ; ops:deployedIn "us-east-1" .
}"""
print([(str(r.incident).split('#')[1], str(r.sev)) for r in g.query(q)])

# Property path: everything api-gateway depends on, transitively.
q2 = """PREFIX ops: <https://example.org/ops#>
SELECT ?dep WHERE { ops:api-gateway ops:dependsOn+ ?dep }"""
print(sorted(str(r.dep).split('#')[1] for r in g.query(q2)))

# SHACL gate: an incident must point at a typed Application.
shapes = """
@prefix sh:  <http://www.w3.org/ns/shacl#> .
@prefix ops: <https://example.org/ops#> .
ops:IncidentShape a sh:NodeShape ; sh:targetClass ops:Incident ;
  sh:property [ sh:path ops:affects ; sh:minCount 1 ; sh:class ops:Application ] ;
  sh:property [ sh:path ops:severity ; sh:in ("SEV 1" "SEV 2" "SEV 3") ] .
"""
conforms, _, text = validate(g, shacl_graph=Graph().parse(data=shapes, format="turtle"))
print(conforms, "billing" in text)   # False True: ops:billing is not a typed Application

Verified with rdflib 7.6.0 and pySHACL 0.40.2; it prints [('INC-5377', 'SEV 2')], then ['edge-auth', 'zuul'], then False True, because the shape rejects an incident that points at an untyped name.

Property graph alternative. Opinion: a labelled property graph with a query language such as Cypher or ISO GQL stores properties on edges directly and is often easier to adopt. RDF's advantages are standard IRIs for identity, named graphs for provenance, validation through SHACL and portability between stores, which is Netflix's stated reason for choosing it. Either way, identity and validation rules matter more than the storage model.

Storage at scale. Netflix reports a distributed quad store on an embedded key-value engine. The general pattern is index permutations of subject, predicate, object and graph, so any bound position can be looked up by prefix scan. Keep the hot operational window in the graph and summarise old instances. Bound the work a query may do, and cap hub fan-out, so one broad pattern cannot scan the whole store. Netflix does not publish graph size, query latency or index layout.

Sources (9)
  1. Netflix TechBlog: Ontology-Driven Observability: Building an End-to-End Knowledge Graph at Netflix Scale
  2. W3C: RDF 1.1 Concepts and Abstract Syntax (triples, datasets, named graphs)
  3. W3C: OWL 2 Web Ontology Language Primer (open-world assumption)
  4. W3C: SHACL Shapes Constraint Language
  5. W3C: SPARQL 1.1 Query Language (property paths)
  6. W3C: RDF 1.1 Turtle
  7. redb: an embedded key-value database in pure Rust
  8. RDFLib documentation
  9. pySHACL