Tables, VKG & big builds
Data-engineering companions to the .rete file: lossless tabular exports of
the same graph, the virtual-knowledge-graph alternative, and recipes for
pulling genuinely large real datasets. This page is development-facing — for
building a first file and querying it, start at
Getting started.
Lossless entity tables (the best of both worlds)
scripts/rdf_to_entity_tables.py is the lossless counterpart: it keeps the
readable one-table-per-type shape without dropping anything. Each class
table has the frequent properties as named LIST columns (occupation,
citizenship, date of birth…) plus three things that make it complete: a
types column (all P31 values, so a multi-typed entity lives in exactly one
table, never duplicated), an extra MAP(predicate → objects) column that
catches every other property (rare ones, all the multilingual labels), and an
_untyped residual table for subjects with no type. Object values are stored
as N-Triples term tokens (<iri>, "lit", "lit"@en), so IRIs, literals and
language tags round-trip. Explode types + every column + extra across all
tables and you get back exactly the triples — --verify checks that
(reconstructed == input). It can emit Parquet, a DuckDB file, and a SQLite file
(list/map columns as JSON text) in one run:
uv run python scripts/rdf_to_entity_tables.py --parts 1 --limit 12000000 --props 24 \
--min-entities 50 -o data/ent --duckdb data/ent.duckdb --sqlite data/ent.sqlite --verify
--props only changes how many properties get their own column vs. land in
extra — it never affects losslessness. The _manifest.parquet records each
class's column → predicate map so reconstruction is mechanical, and N-Triples
is the interchange hub (rete export ↔ rete build ↔ these tables).
It works on any RDF, not just the Wikidata Parquet source: pass --nt <file> to read N-Triples directly (objects, language tags and datatypes
round-trip verbatim) and --type-predicate <iri> to group by something other
than Wikidata's P31 — e.g. rdf:type for OBO ontologies. This is how the
chebi-full companions are built from the same chebi.nt as the .rete:
uv run python scripts/rdf_to_entity_tables.py --nt chebi.nt \
--type-predicate "http://www.w3.org/1999/02/22-rdf-syntax-ns#type" \
--props 24 --min-entities 50 -o data/chebi-tables \
--duckdb data/chebi.duckdb --sqlite data/chebi.sqlite --verify
Companion: columnar property tables (Parquet, split by type)
To compare the .rete graph against a columnar layout,
scripts/rdf_to_property_tables.py denormalizes the same Wikidata triples into
one Parquet table per entity type (the classic RDF "property table"): rows
are entities, grouped by wdt:P31 (instance-of); columns are that class's most
common structured properties, multi-valued as LIST(VARCHAR); an English
label column is added and the labelling/description predicates are excluded
so the columns are the real properties. It runs entirely in DuckDB from the
source Parquet:
pip install --break-system-packages duckdb
uv run python scripts/rdf_to_property_tables.py --parts 10 --limit 120000000 -o data/wd-tables
# -> data/wd-tables/Q5.parquet (human), Q16521.parquet (taxon), … + _manifest.parquet
Each class table is independently queryable (SELECT … FROM 'Q5.parquet'), the
_manifest.parquet maps class IRI → label/entity-count/file and each column id
back to its predicate, and a single DuckDB over the set is just views:
CREATE VIEW human AS SELECT * FROM 'data/wd-tables/Q5.parquet'. Match the
.rete slice by passing the same --parts/--limit. This is a property-table
companion for benchmarking storage/query tech against the graph format — not a
lossless graph encoding (sparse properties become NULLs, heterogeneous classes
get wide; that's the point of the comparison).
Alternative approach: a virtual knowledge graph over the companions
The companions above are materialized the rete way: triples → a .rete (the
graph) plus tabular exports you query with SQL. The playground's Explore tab
already queries those exports lazily over httpfs — DuckDB-WASM / SQLite-WASM
fetch only the Parquet row-groups a query touches, the same range-read transport
the .rete uses — so you can compare the same class across the rete engine and
the columnar engines side by side.
A different school of thought skips materialization entirely. A Virtual Knowledge Graph (VKG / OBDA) — e.g. Ontop over DuckDB — keeps the Parquet as the source of truth and answers SPARQL by rewriting it to SQL at query time through declarative R2RML mappings; no RDF file is ever built (tech note). The two are complementary; the trade is materialized-vs-virtual:
| rete (this project) | Virtual KG (Ontop + DuckDB) | |
|---|---|---|
| RDF | materialized into a graph-native .rete | virtual — never materialized |
| Source of truth | the .rete file | the Parquet |
| SPARQL | answered directly over .rete (range reads) | rewritten to SQL over Parquet via mappings |
| Parquet's role | a tabular companion/export | the data |
| Trade-off | a build step; self-contained & graph-native (pyramid, communities, reachability, SHACL, coherence, provenance) | a mapping + a SPARQL→SQL engine at query time; always fresh, no ingestion |
Both lean on the same lazy transport — Parquet's footer + row groups are
range-friendly the way the .rete header + tiles are — so the honest comparison
isn't "lazy vs eager" but a graph-native materialized file vs a virtual
SPARQL view over a columnar source.
And rete's companions are already VKG-ready: the _manifest.parquet beside
each one records the column → predicate map (and class IRI → table) — exactly the
R2RML mapping a VKG needs, generated for free. So because the companions are
range-readable R2 objects, a VKG can ATTACH them in place: a single Ontop/DuckDB
endpoint over every Parquet in the R2 collection — driven by the manifests — is a
virtual knowledge graph over all published companions with no full download and
no .rete: the federated,
virtual mirror of rete's materialized file. (Materializing the same thing is also
mechanical: the entity tables are lossless, so reconstruct each → merge, or build
per-dataset .rete and rete federate across them.)
Benchmark candidate (TODO): rete's range-read SPARQL on a .rete vs an
Ontop-over-DuckDB VKG on the equivalent Parquet — same queries, same HTTP-range
hosting — measuring bytes fetched, latency, and which graph operations (pyramid /
reachability / SHACL / coherence) the VKG can't answer cheaply.
A real-world graph: a Wikidata biology slice
The dataset pipelines on this page (data/scripts/...) live in the private caviri/rete-datasets repository, which a rete checkout clones at data/scripts/;
commands are written from the rete root.
For a genuinely large, real dataset, data/scripts/standalone/fetch_wikidata_bio.py pulls a
life-sciences slice from the Wikidata Query Service:
genes, the proteins they encode, the diseases they associate with, drugs that
treat those diseases, and a disease subclass hierarchy — one connected graph,
every entity labelled in English. It runs a handful of bounded CONSTRUCT
queries (each well under the WDQS timeout) and merges them as N-Triples.
uv run python data/scripts/standalone/fetch_wikidata_bio.py --limit 4000 -o data/wikidata-bio.nt
rete build data/wikidata-bio.nt -o bio.rete
rete stats bio.rete # ~40k triples, ~27k terms, hundreds of communities
A --limit 4000 run is roughly 40,000 triples (≈2,800 genes, ≈4,000 proteins,
≈3,600 diseases) — the community pyramid finds hundreds of organism/disease
clusters, and it exercises every surface: typed-class queries, label joins, the
disease hierarchy via wdt:P279, and HTTP range queries over a real graph.
--taxon Q83310 fetches mouse instead of human; --limit trades size against
WDQS time. Output lands in data/ (git-ignored, like all fetched datasets —
the script is tracked, the bytes are regenerated on demand). Be a good WDQS
citizen: it is rate-limited, so fetch a slice, not a firehose.
Real Wikidata at gigabyte scale (Parquet)
The Query Service is for slices, not bulk. For a genuinely large, real
linked-data graph, data/scripts/standalone/wikidata_parquet_to_nt.py reads the full Wikidata
"truthy" dump from the
piebro/wikidata-extraction
Parquet conversion on Hugging Face (~80 partitions, subject/predicate/object/ language columns) with DuckDB — httpfs streams the remote files, so a bounded
slice needs no full download — and writes N-Triples.
pip install --break-system-packages duckdb
uv run python data/scripts/standalone/wikidata_parquet_to_nt.py --limit 12000000 -o data/wd.nt # ~1 GB
rete build data/wd.nt -o wd.rete
The source Parquet drops literal datatypes, so the converter recovers them
(--datatypes, default auto): it resolves each property's datatype from a
local cache or one WDQS wikibase:propertyType query (~13.5k properties,
cached for reuse) and re-types each literal — dates xsd:dateTime, quantities
xsd:decimal, coordinates geo:wktLiteral; strings stay plain, monolingual
text keeps its language tag, entity values are IRIs. If that map is
unavailable (e.g. WDQS rate-limited), it falls back to an offline heuristic
that types the unambiguous values — ISO timestamps and WKT geometries — leaving
numbers plain (a bare number is indistinguishable from a numeric external-id
without the map). --datatypes heuristic forces the offline path; none emits
plain literals. Once typed, the engine's DATATYPE(?o) / LANG(?o) filters
can select by datatype.
Measured on the dev container, the full --limit 12000000 (~1 GB) run:
converting streams in ~24 s (1.25 GB N-Triples, datatypes recovered) and
builds in ~52 s to a 110 MB .rete — 5 pyramid levels, ~115k
communities — with typed literals intact ("1830-10-04T00:00:00Z"^^xsd:dateTime,
"Point(5.47 49.50)"^^geo:wktLiteral) and a selective lookup answering in
under a second. The slice is a real cross-section of all of Wikidata (people,
places, works, taxa, …); for a curated biology-only graph use the WDQS fetcher
above. --parts N draws from N whole partitions; --local-dir reads partitions
you have already downloaded.
Building a graph that does not fit in memory
rete build holds the whole graph in RAM by default. When it does not fit, or the
container has a hard memory limit, give it a budget:
rete build big.nq.gz -o big.rete --memory-budget-mb 3000 --tmp-dir /scratch
--memory-budget-mb N selects the external builder: it cuts the input into
chunks, spills them to --tmp-dir (default: next to the output), merges the
dictionary and sorts each index on disk, so memory is sized from the budget and
not from the graph. The file has the same content hash as the in-RAM build (only the
build record differs); what changes is the time and the scratch disk. It reads
N-Triples, N-Quads, Turtle and TriG, gzipped or not, directly. The budget is a
ceiling for the build's own buffers, so leave headroom below the container's
limit. Details, measurements and the scratch-disk cost are in
the CLI reference and the
build tutorial.