rete · Project Benchmarks

Benchmarks

Use benchmarks to answer a concrete question about a change: does it reduce query time, memory use, file size or network reads? These measures can move in different directions. Start with correct answers, then compare the cost of getting them under the same conditions.

Understand the recorded results

The measurement record keeps the detailed tables and earlier optimization results. It is a collection of measured runs, not a live dashboard or a promise about every graph.

MeasurementWhat it includes
CLI timings from scripts/bench.shProcess startup, opening the file and executing the command
rete-bench query timingsWarm, in-process queries; loading/opening is reported separately
Memory measurementsThe reported metric matters: live/peak heap and process RSS are different
Range-read measurementsBytes and requests, alongside elapsed time and the reader/cache mode

For example, the checked-in OpenCitations snapshot records commit 5de5b53, a run date of 2026-06-15, 32 logical cores and five query repetitions. Those numbers describe that run, not the current checkout. The LUBM-style snapshot records its query results but has less environment metadata. Do not infer a measurement date or revision that the report does not contain.

The comparison harness reports result-count agreement, with exceptions for implementation-defined behavior such as DESCRIBE. Equal counts alone do not prove equal rows; keep the correctness tests and differential checks.

Run a small, reproducible check

After building the development image, start with the deterministic query example. It needs no downloaded dataset:

docker compose run --rm dev cargo run --release -p rete-core --example qbench -- --check

It builds the default 30,000-paper fixture, checks expected row counts and enforces generous per-query median-time ceilings. CI uses those ceilings to catch major regressions; passing them is not evidence of a speedup over another revision or engine. For a generated graph with CLI timings and a local HTTP range query:

docker compose run --rm -e CARGO_TARGET_DIR=/work/target dev bash scripts/bench.sh 20000

The target-directory override matches this script's target/release/rete lookup. Its elapsed command timings differ from the warm engine measurements above, and its HTTP test uses a local server rather than a public network. Treat it as a starting point, not a statistically rigorous comparison.

For deeper work, the rete-bench harness provides JSON comparisons, a self-contained LUBM-style workload, and build/query memory profiling. Reproducing a real-dataset report also requires its exact input files; ignored data/ paths are not bundled with a public clone.

Publish evidence someone else can compare

Record the commit, dataset identity/hash, hardware, toolchain, build features, execution mode and repetitions with the result. Say whether caches were warm and whether loading, parsing and serialization are included. Run the same workload on the baseline and the change; repeat noisy timings and inspect memory and bytes fetched as well as latency.

Keep the machine-readable report when updating the measurement record. Its marked OpenCitations and LUBM sections are generated by render_benchmark_doc.py; other sections are authored separately. Regenerating the documentation does not rerun the experiments. Label older results with their original context instead of presenting them as newly measured.