rete · Project Repository and core

Architecture

Rete turns RDF into an indexed file that clients query locally or through HTTP byte ranges. One Rust engine supplies the behavior behind commands, bindings and browser interfaces. Follow a build and query below to find where your change belongs.

Find the code you need

LocationResponsibility
crates/rete-coreStorage, ingestion, querying, range reads, reasoning and validation
crates/rete-graphPublic Rust package name; re-exports the core rather than implementing another engine
crates/rete-cliCommands, local/remote file handling and terminal output
crates/rete-wasmBrowser bindings around the same core
clients/Python, JavaScript, R, Java, Blender, MCP bundle, relay and desktop integrations
crates/docgen and docs/Documentation renderer, authored Markdown and generated site
crates/bench, tests/ and fuzz/Measurements, cross-system checks, browser tests and fuzzing
web/ and scripts/Editable application sources and build, publishing and verification tools
.github/workflows/CI and release automation

The workspace manifest defines six Rust crates. Several clients and the fuzz harness build separately with their own toolchains. Clients depend on the engine; the engine does not depend on clients. Dataset harvesters and converters live in the separate rete-datasets repository, not in the public product tooling.

For a Rust entry point, start with the format, query, range, card, validation and reasoning namespaces in rete-core/src/lib.rs. They group the supported API without exposing every internal module.

Follow a build into a file

rete build starts in the CLI's commands/build.rs. The core's ingest.rs parses RDF, the dictionary assigns integer IDs to terms, and index.rs sorts triples into permutation indexes. Different sort orders make different bound query patterns efficient. Independently compressed tiles let readers retrieve part of an index.

For larger inputs, --memory-budget-mb selects the external builder in extbuild.rs. It spills chunks and sorted runs to disk, merges dictionaries, remaps IDs and writes the final sections incrementally. The budget sizes the working buffers; it is not an exact process-RSS ceiling, and tile directories still grow with the dataset. This path has different feature limits from an in-memory build; check the CLI reference before adding an option to it.

Current writers use format 0x06: v06/ owns its 128-byte header, root block, front directories and card region. file.rs dispatches reads for both 0x05 and 0x06; header.rs describes the older layout. Besides dictionaries and indexes, files can contain community summaries, schema, label/text indexes and a Dataset Card. Their presence and placement depend on build options and file features. The format specification owns their byte-level contracts. Published readers through 0.4.2 cannot read 0x06; check Compatibility before publishing a new file.

A snapshot stays immutable. Changes to a logical graph can instead use manifest.rs, which combines immutable addition/deletion segments. rete serve can also keep updates in a journal. See Writable graphs for the lifecycle and memory implications.

Follow a query back to an answer

The SPARQL entry points in sparql.rs parse a query, lower its algebra in sparql/lower.rs, and evaluate it through sparql/eval.rs. Integer slot rows carry bindings between operators; dictionary IDs become RDF terms when needed for expressions or output. Selective scans and streaming operators can stop early for LIMIT or ASK, while sorting, aggregation and broad traversals can still require substantial work. results.rs serializes the answer for clients.

Opening strategy matters: Rete::open consumes resident bytes; open_ranged fetches whole relevant sections; open_ranged_lazy defers dictionary chunks and tile payloads until used. reader.rs supplies the RangeReader interface, with caches and an optional native HTTP implementation around it. Native and browser HTTP readers allow a bounded whole-body fallback when a host ignores ranges, normally up to 256 MiB; that download still counts as transferred data. See Hosting.

Exact lazy queries use permutation tiles, sometimes several tiles for one leading ID. Community summaries support overviews and documented summary query shapes; they are not implemented physical community partitions for exact reads. Semantic zoom explains that distinction.

rete why exposes physical routing and byte ranges, rather than a narrative explanation of an answer. Lazy readers must also surface incomplete-fetch errors: missing data must never appear to be a successful empty result.

Extend the engine without breaking its readers

Keep shared behavior in the core. reason.rs handles materialization, sparql/ql.rs supports query-time reasoning, and shacl.rs validates shapes; their supported subsets remain explicit in the reasoning and validation references. Changes to encoded bytes also need format, compatibility and corruption-handling checks.

The default browser engine is single-threaded; native-only dependencies must stay behind their feature/target boundaries. Edit application sources under web/, then use docker compose run --rm wasm, the canonical producer of the WASM and browser artifacts. In a worktree, pass the source revision as described in the contributor guide.

Use Testing and CI to choose checks for the part you changed. Use Benchmarks to compare equivalent workloads and recorded measurements. Check correctness and reading costs before trusting faster timings.