Benchmarks
Use benchmarks to answer a concrete question about a change: does it reduce query time, memory use, file size or network reads? These measures can move in different directions. Start with correct answers, then compare the cost of getting them under the same conditions.
Understand the recorded results
The measurement record keeps the detailed tables and earlier optimization results. It is a collection of measured runs, not a live dashboard or a promise about every graph.
| Measurement | What it includes |
|---|---|
CLI timings from scripts/bench.sh | Process startup, opening the file and executing the command |
rete-bench query timings | Warm, in-process queries; loading/opening is reported separately |
| Memory measurements | The reported metric matters: live/peak heap and process RSS are different |
| Range-read measurements | Bytes and requests, alongside elapsed time and the reader/cache mode |
For example, the checked-in
OpenCitations snapshot records commit 5de5b53,
a run date of 2026-06-15, 32 logical cores and five query repetitions. Those
numbers describe that run, not the current checkout. The
LUBM-style snapshot records its query results but has
less environment metadata. Do not infer a measurement date or revision that
the report does not contain.
The comparison harness reports result-count agreement, with exceptions for
implementation-defined behavior such as DESCRIBE. Equal counts alone do
not prove equal rows; keep the correctness tests and differential checks.
Run a small, reproducible check
After building the development image, start with the deterministic query example. It needs no downloaded dataset:
docker compose run --rm dev cargo run --release -p rete-core --example qbench -- --check
It builds the default 30,000-paper fixture, checks expected row counts and enforces generous per-query median-time ceilings. CI uses those ceilings to catch major regressions; passing them is not evidence of a speedup over another revision or engine. For a generated graph with CLI timings and a local HTTP range query:
docker compose run --rm -e CARGO_TARGET_DIR=/work/target dev bash scripts/bench.sh 20000
The target-directory override matches this script's target/release/rete
lookup. Its elapsed command timings differ from the warm engine measurements
above, and its HTTP test uses a local server rather than a public network.
Treat it as a starting point, not a statistically rigorous comparison.
For deeper work, the
rete-bench harness
provides JSON comparisons, a self-contained LUBM-style workload, and build/query
memory profiling. Reproducing a real-dataset report also requires its exact
input files; ignored data/ paths are not bundled with a public clone.
Publish evidence someone else can compare
Record the commit, dataset identity/hash, hardware, toolchain, build features, execution mode and repetitions with the result. Say whether caches were warm and whether loading, parsing and serialization are included. Run the same workload on the baseline and the change; repeat noisy timings and inspect memory and bytes fetched as well as latency.
Keep the machine-readable report when updating the
measurement record. Its marked OpenCitations and LUBM sections
are generated by
render_benchmark_doc.py;
other sections are authored separately. Regenerating the documentation does
not rerun the experiments. Label older results with their original context
instead of presenting them as newly measured.