rete · Guides Federated queries

Federated queries across several .rete files

rete federate runs one SPARQL query across many .rete sources — local file paths and/or http(s):// URLs, mixed freely — and merges the results. It is the answer to "my data is sharded into several files; query them as one."

rete federate <source…> --query "<SPARQL>" [--json] [--no-route]

This is an honest prototype: it does union federation, not distributed joins. Read Limitations before relying on it. (Federating with a live SPARQL endpoint from inside a query is a different feature — the SPARQL 1.1 SERVICE clause, covered in SPARQL support.)

The same union+routing federation is also available in the browser — the playground turns the SPARQL console into a multi-source one with a "+ Add source" button, each source range-queried lazily.

Why federation works at the term level

How rete federate answers one SPARQL query from several .rete files. The router first reads each file's predicate set from its summary — a few hundred bytes, never the triple index — and skips any file whose predicates cannot match the query. The surviving files each run the query with the ordinary single-file engine and return term-level rows. Those rows are then unioned and deduplicated. The merge has to happen on strings because every file has its own dictionary, so the integer ID 42 in one file has nothing to do with ID 42 in another.
Each file keeps its own dictionary; the router sends each pattern only to files holding its predicate, then results merge at the term level.

Every .rete file carries its own dictionary — the integer IDs that encode terms are local to one file. ID 42 in cites-2021.rete and ID 42 in cites-2024.rete are unrelated. So you cannot merge files by combining their integer indexes.

Federation therefore works at the term (string) level: the query is evaluated independently on each file with the existing single-file engine, and the term-level result rows are merged. This is exactly correct for sharded datasets where each file independently yields complete result rows — the citation data below, sharded by citing year, is precisely this shape: each cites edge (and its date/type) lives wholly within one year-shard, so each shard produces whole, self-contained answer rows.

Merge semantics

Query formMerge across sources
SELECTUnion of solution rows, deduped (identical rows collapse), then ORDER BY, OFFSET and LIMIT applied to the merged rows; aggregates merged as described below. Without ORDER BY: source order, then within-source order.
ASKLogical OR — true if any source matches.
CONSTRUCTUnion of constructed triples, deduped.

Dedup is over the projected variables for SELECT * it is over all bound variables. A row that appears in two shards (e.g. a node that cites the target and shows up in two files) is reported once.

Solution modifiers and aggregates

When more than one source is queried, the query's solution modifiers are applied to the merged rows, not per source:

  • ORDER BY sorts the merged rows, in the same order as a single file (numbers by value, unbound first). Each key must be a variable the query projects; an expression is refused (bind it with (expr AS ?k) and order by ?k).
  • OFFSET / LIMIT cut the merged rows. LIMIT n OFFSET m is also sent to each source as LIMIT n+m (with DISTINCT, since rows are merged as a set), an upper bound: no source can contribute more rows than that to the final answer. OFFSET without LIMIT is applied after the merge only.
  • DISTINCT (and, as before, every merge) removes rows repeated across sources.
  • Aggregates: MIN and MAX merge exactly. COUNT(…) and SUM(…) add each source's total per GROUP BY key, which counts a triple stored in two sources twice, so they are merged only with --disjoint (the sources share no triple) and refused without it. AVG, SAMPLE, GROUP_CONCAT, COUNT(DISTINCT …), HAVING, and an expression over an aggregate are refused with the reason, rather than answered per source. ORDER BY and LIMIT over the merged groups work (the sources return whole groups).
  • A CONSTRUCT with OFFSET or LIMIT is refused (they count solutions, not triples).
  • When routing leaves one source to query, it answers the whole query itself and none of the above applies.
rete federate a.rete b.rete --disjoint --query '
  SELECT ?kind (COUNT(*) AS ?n) WHERE { ?p <http://e/kind> ?kind }
  GROUP BY ?kind ORDER BY DESC(?n) LIMIT 3'

Routing / pruning (default on)

Before evaluating, federate reads each source's predicate set cheaply from its pyramid summary — the (large) triple index is never touched (the same "summary first" path used by rete predicates and rete summary-url). It also extracts the concrete predicate IRIs the query constrains on (from the parsed query's basic graph patterns and property paths). A source whose predicate set is disjoint from the query's predicates cannot contribute a row and is skipped.

  • --no-route disables pruning and queries every source.
  • A query that pins no concrete predicate (every pattern uses a variable predicate, e.g. ?s ?p ?o) cannot be routed — all sources are queried.
  • A source with no summary cannot be inspected, so it is never pruned.

For local files routing reads a few hundred bytes per source; for URLs it issues a handful of HTTP range requests (header + dictionary + summary) — never the whole file.

Per-source diagnostics (rows contributed + time), the queried-vs-pruned tally, and the list of pruned sources are written to stderr the merged results go to stdout.

Limitations

This is a deliberately honest prototype — it is correct for sharded data and clear about what it does not do.

  • Union only, no cross-file joins. A solution that needs a triple from file A joined with a triple from file B (e.g. ?x from one shard joined to ?y from another) will not be found. Each shard is queried in isolation. Cross-file joins need a term-level federated BGP engine that ships intermediate bindings between sources — future work.
  • Aggregates that cannot be folded are refused, not answered per source: see Solution modifiers and aggregates. A sub-SELECT with its own aggregate or LIMIT is still evaluated inside each source, like the rest of the query.
  • Before the fix applied ORDER BY, LIMIT and aggregates per source and concatenated the results (ORDER BY came back unsorted, LIMIT 5 over two shards returned 10 rows, COUNT(*) one count per source).

Real example — OpenCitations, sharded by citing year

The repo ships a real dataset: citations of the AlphaFold paper (<https://doi.org/10.1038/s41586-021-03819-2>), sharded by citing year into data/opencitations/cites-<year>.rete. Each shard uses <http://purl.org/spar/cito/cites>, <http://purl.org/dc/terms/date>, and rdf:type. (Regenerate with python3 scripts/fetch_opencitations.py, then for f in data/opencitations/*.nt; do rete build "$f" -o "${f%.nt}.rete"; done.)

Federated SELECT across two year-shards

rete federate data/opencitations/cites-2021.rete data/opencitations/cites-2024.rete \
  --query 'SELECT ?citing WHERE {
             ?citing <http://purl.org/spar/cito/cites>
                     <https://doi.org/10.1038/s41586-021-03819-2> } LIMIT 5'
  data/opencitations/cites-2021.rete: 5 row(s) in 3.6ms
  data/opencitations/cites-2024.rete: 5 row(s) in 15.6ms
5 solution(s)
federated 2 source(s): 2 queried, 0 pruned (routing on); 5 merged result(s)
?citing=<https://doi.org/10.1001/jama.2021.15728>
?citing=<https://doi.org/10.1002/2211-5463.13301>
?citing=<https://doi.org/10.1002/2211-5463.13316>
?citing=<https://doi.org/10.1002/advs.202102592>
?citing=<https://doi.org/10.1002/advs.202103807>

LIMIT 5 is applied to the merged rows (each shard is asked for at most 5, which is enough), so five rows come back, not ten. Before the fix it printed 10.

Routing in action — a predicate-disjoint shard is pruned

Add a source that uses a different predicate (here a shard whose only predicate is rdfs:label, not cito:cites). The cites query prunes it without ever reading its index:

rete federate data/opencitations/cites-2021.rete other-demo.rete data/opencitations/cites-2024.rete \
  --query 'SELECT ?citing WHERE {
             ?citing <http://purl.org/spar/cito/cites>
                     <https://doi.org/10.1038/s41586-021-03819-2> } LIMIT 3'
  data/opencitations/cites-2021.rete: 3 row(s) in 3.4ms
  data/opencitations/cites-2024.rete: 3 row(s) in 15.8ms
3 solution(s)
federated 3 source(s): 2 queried, 1 pruned (routing on); 3 merged result(s)
  pruned (predicate-disjoint): other-demo.rete
…

--no-route would instead query all three (the label-only shard contributing 0 rows). For year-shards that all share cito:cites, routing keeps every shard — pruning helps when sources are heterogeneous (different predicates per file), which is the common federation case.

ASK federation (logical OR)

rete federate data/opencitations/cites-2017.rete data/opencitations/cites-2024.rete \
  --query 'ASK { ?x <http://purl.org/spar/cito/cites>
                    <https://doi.org/10.1038/s41586-021-03819-2> }'
# → true   (at least one shard has a citation)

URL sources

Sources may be http(s):// URLs of .rete files hosted on R2, S3, or any direct origin that honors HTTP Range requests — mix them with local paths freely:

rete federate \
  https://data.graphplaza.com/worldcup2026/worldcup2026-2026-07-12.rete \
  data/opencitations/cites-2024.rete \
  --query 'SELECT ?citing WHERE { ?citing <http://purl.org/spar/cito/cites>
                                          <https://doi.org/10.1038/s41586-021-03819-2> }'

Each URL is opened with range reads (header → dictionary → index → pyramid), never a full download both routing and evaluation fetch only the bytes they need. The release catalog pins its public R2 objects in web/datasets.lock.json; the catalog audit also verifies that every shard is stable format generation 1 and exposes Content-Range to browser CORS callers.

In the playground

The playground brings the same federation to the browser: federation is a kind of SPARQL, so it lives in the SPARQL console rather than a separate mode. Under the query editor a Sources strip shows the current dataset; a + Add source button federates with another source three ways:

  • From catalog — any registered dataset (a remote one is range-queried lazily; a bundled one is queried in memory);
  • .rete link — a pasted remote URL, range-queried lazily;
  • SPARQL endpoint — a live endpoint (SELECT / ASK) over the SPARQL protocol, subject to the endpoint's CORS policy.

Running with two or more sources fans the same query out to each and merges with the exact CLI semantics above — SELECT union+dedup, ASK OR, CONSTRUCT triple-union — and a per-source banner reports the rows and bytes each source contributed. Each source keeps its own lazy reader, so two remote .rete files are read independently and lazily into one merge; it is orchestrated in the playground's JS over the existing single-source query paths, so no extra engine code runs. An example can pre-load its partner with one click. A live line ticks during the query — N/M sources answered · range requests · MB · elapsed — so a long multi-source fan-out isn't a silent spinner.

Beyond union — cross-source joins and sharded datasets

The playground goes past the CLI's union: it also does a term-level cross-source JOIN — one BGP split across sources and joined on the shared variables — so the "cross-file join is not found" limit above is a CLI limit, not a playground one. Queries that can't be split this way (OPTIONAL / UNION / aggregates / property paths / VALUES) fall back to the union path. So e.g. USTC editions ⋈ the Embassy books they cite, joined on the book IRI, returns real joined rows across the two files. How a BGP is split:

  • Every pattern goes to every source that can contribute, and their rows are unioned. Each .rete or in-memory source is asked (ASK, cached per file for the session) whether it has the pattern's predicate, and — when several do and the pattern has a constant — whether it has the pattern itself, which is what keeps ?b a schema:Book on the one dataset that has the class while rdf:type is in all of them. A live endpoint is never asked (it answers anything): it takes the patterns no other source has, and the patterns whose subject it bound (?wd wdt:P17 ?c . ?c rdfs:label ?l — the labels of its entities).
  • Patterns only one dataset can answer run together on it as one sub-query (exact: every matching triple is in that dataset). Patterns several datasets can answer run one at a time, on each of them.
  • Steps run most-constrained first, each step's join keys VALUES-injected into the next (≤ 250 per hop); a FILTER runs in the first step where all its variables are bound. The 🔍 Plan button shows the routing and the sub-queries without running them; after a run, the banner lists each pattern's sources and the rows, bytes and requests of its step.

This replaced a router that sent each pattern to one source — the one that bound its subject, whenever that source had the predicate at all. One stray triple was enough to break it: crossref-citations.rete holds a single rdfs:label (in its declaration of cx:citedDOIAsWritten), so a title lookup on citing works went there and the Crossref title join returned 0 rows; now the label pattern runs on both files and the titles come from crossref.rete.

A sharded dataset is registered as one logical graph with a shards: [url0, url1, …] list in the catalog; the playground treats url0 as the primary and the rest as intrinsic federation partners, so every query auto-fans across all shards (union) and the rows merge — you query it like one dataset (a "⛓ N shards" chip shows in the Sources strip). This is how a graph too big to build as a single file — e.g. a 600 M- triple Wikidata split into six independent --no-pyramid shards — is served and queried as one. The path to a bigger graph is just more shards. With a partner added, the sharded dataset (on either side: a sharded partner brings all its shards) stays one logical source to the join: a pattern only it can answer runs on every shard that has it, one subject at a time — a subject's triples live in one shard, but a join between two subjects may cross shards, so it is joined across shards like a join between datasets. (Before, every pattern of a sharded dataset with a partner went to its first shard only.) Without a partner, a query still runs whole on each shard and the rows are unioned, so a join between subjects in different shards is not found there.

Worked example — resolve five terms across two ontologies. The bundled chebi-full (the complete ChEBI ontology, 8.83 M triples) and chemotion (a chemistry ELN graph merged with CHMO + RXNO) share CHEBI_* IRIs plus rdfs:label/rdfs:subClassOf, but neither resolves every label. Pick either dataset, open its Federation example (it adds the other as a second source), and run:

PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX obo:  <http://purl.obolibrary.org/obo/>
SELECT ?term ?label WHERE {
  VALUES ?term { obo:CHEBI_15377 obo:CHEBI_27732 obo:CHMO_0000228 obo:BFO_0000015 obo:CHEBI_23367 }
  ?term rdfs:label ?label
}

chebi-full answers water / caffeine / molecular entity; chemotion answers spectroscopy / process; the union resolves all five. The result line shows the per-source split — no single file has all the answers.