rete · Reference SPARQL 1.1 conformance

SPARQL 1.1 conformance

How much of SPARQL does rete actually answer correctly? This runs the official W3C SPARQL 1.1 query-evaluation test suite — the canonical w3c/rdf-tests fixtures: each test ships a tiny RDF dataset, a query, and the expected results — against rete and scores every test. It is a correctness suite (the datasets are a handful of triples), so it measures coverage, not speed; performance is benchmarked separately on a real graph (see BENCHMARK).

The harness is scripts/sparql_conformance.py: for each mf:QueryEvaluationTest it builds a .rete from the data, runs the query through the rete CLI, and compares to the expected result — SPARQL Results JSON/XML (SRX/SRJ) for SELECT/ASK as an unordered multiset, and an RDF graph isomorphism for CONSTRUCT/DESCRIBE.

python scripts/sparql_conformance.py \
  --rete target/release/rete --suite <rdf-tests>/sparql/sparql11

Scorecard

312 query-evaluation tests, byte-for-byte against the W3C expected results. Measured 2026-10-09 (after the correlated zero-length path fix) against w3c/rdf-tests 5e5da96 (2026-10-02) — the revision pinned in tests/w3c-rdf-tests.rev (see The pinned suite). "n/s" = errored / not supported.

Categorypassn/snotes
negation12 / 120✅ full
json-res4 / 40✅ full
cast6 / 60✅ full — xsd:integer/decimal/float/double/boolean/string
bind10 / 100✅ full — in-pattern BIND visible to later FILTER/join
grouping4 / 40✅ full
bindings (VALUES)10 / 122
aggregates40 / 422GROUP_CONCAT/SUM/AVG/SAMPLE incl. DISTINCT
property-path30 / 332incl. negated property sets + zero-length on empty data
construct3 / 51graph-isomorphism check
exists4 / 61
project-expression7 / 70✅ full
functions73 / 751nearly full — only NOW() + IRI() base resolution
entailment28 / 704needs build --materialize
subquery7 / 167nested SELECT joins; GRAPH-scoped + RDF/XML data n/a
service1 / 76SERVICE is implemented — most of these tests need a live endpoint, which the offline run does not have
csv-tsv-res0 / 33CSV/TSV result format
TOTAL239 / 312 (76.6%)29

This total is measured in CI, not written by hand. The conformance job runs the harness against the W3C suite on every Rust change and fails if the passing count drops below tests/conformance-baseline.json. It was not always so: the figure here read 232 / 75.1% while the engine actually scored 236 / 76.4%, and nothing would have caught it moving the other way. It then drifted again, to a stale 236 / 309, because only the baseline file is checked, not this page. The suite itself also moved: CI used to clone w3c/rdf-tests at HEAD on every run, so the denominator followed upstream (309 tests in June, 312 by October) and the job could change its verdict with no rete commit. It is pinned now.

The pinned suite

The W3C suite revision lives in one place: tests/w3c-rdf-tests.rev, a single full commit SHA of w3c/rdf-tests — currently 5e5da96bb06c551ae3ac57d30fec8116f7161b9b (2026-10-02, 312 query-evaluation tests). CI checks out exactly that commit (sparse, only sparql/sparql11, cached by SHA) and fails if what it got is anything else; tests/conformance-baseline.json names the file in suite_revision_file, and its counts are only meaningful against that commit. The harness prints the revision it ran against and warns when it is not the pinned one.

Bumping the suite is a deliberate one-line change — replace the SHA in tests/w3c-rdf-tests.rev. Do it in its own PR, and in the same PR:

  1. re-run the harness against the new commit and regenerate the baseline in place (--json tests/conformance-baseline.json keeps _comment and suite_revision_file, and updates only the counts);
  2. update the scorecard above — the date, the SHA and the per-category rows;
  3. explain any change in pass that is the suite's doing (new or changed tests) rather than rete's, since the two now move separately.

Coverage notes

Strong areas. Negation (MINUS / NOT EXISTS), JSON results, XSD casts (with strict lexical validation), and projection expressions are 100%. Property paths are near-full — including negated property sets (!(:p1|…|:pn)) and the zero-length identity solution on */? even against empty data. BIND inside a WHERE pattern is visible to later filters and joins; nested SELECT subqueries are evaluated independently and joined on their projected variables; the aggregate set is complete (GROUP_CONCAT with DISTINCT/SEPARATOR, typed computed numerics, 15-significant-digit decimal round-trips). The built-in function library covers strings (language-tag-preserving REPLACE/CONCAT/SUBSTR/…, STRDT/STRLANG, ENCODE_FOR_URI, LANGMATCHES), the MD5…SHA512 hashes, the xsd:dateTime accessors, and IF/IN/sameTerm — all in pure Rust, so the same coverage holds in the WASM client. CONSTRUCT answers are verified by graph isomorphism. This matches the 24-operator cross-check against Oxigraph in BENCHMARK — which runs in CI as of this change. It had not: the differential oracle lives in the rete-bench crate, and every CI invocation excluded that crate, so the test described in its own header as "a correctness gate" ran nowhere while build reports went on printing "differential oracle green".

Out of scope (counted against the total, but not engine bugs):

  • SERVICE (7) — SERVICE federation is implemented (the block is sent to the remote endpoint and joined — see sparql); these tests need a live endpoint, so they stay out of the offline suite. (Cross-file federation is a different feature — see federation.)
  • entailment (≈49) — RDFS/OWL entailment regimes; rete answers these only when entailments are baked in at build time (rete build --materialize), which this run does not do.
  • subquery (12 n/s) — plain nested SELECT is evaluated (see above); the remaining n/s are GRAPH-scoped subqueries and tests whose data ships as RDF/XML, not the subquery feature itself.
  • csv-tsv-res (3) — the CSV/TSV result serialization isn't implemented.

Remaining gaps in functions (2): NOW() (no wall clock on the wasm32-unknown-unknown target the engine must also compile to) and IRI() relative-base resolution. The non-deterministic builtins RAND, UUID/STRUUID, BNODE work in the browser too (via getrandom's js backend), and an error in an IF condition propagates rather than silently taking the else-branch.

The same answers, lazily and remotely

The conformance run opens each file locally (in memory). rete's three read modes — local, remote-lazy (HTTP range reads via sparql_url / sparql-url), and remote-cached (download once, query in memory) — share the identical evaluator; only the byte source differs. So they return identical results — demonstrated on a real ~12 M-triple graph: the same query yields the same rows whether served locally, lazily from a CDN, or federated across shards. The tiny conformance fixtures make per-mode timing meaningless — that comparison lives in BENCHMARK, on a graph large enough for it to matter.

Reproduce

REV=$(tr -d '[:space:]' < tests/w3c-rdf-tests.rev)     # the pinned suite
git init -q rdf-tests
git -C rdf-tests remote add origin https://github.com/w3c/rdf-tests.git
git -C rdf-tests sparse-checkout set sparql/sparql11
git -C rdf-tests fetch -q --depth 1 --filter=blob:none origin "$REV"
git -C rdf-tests checkout -q FETCH_HEAD
cargo build --release -p rete-cli
python scripts/sparql_conformance.py --rete target/release/rete \
  --suite rdf-tests/sparql/sparql11            # add --relaxed for the value column