CLI reference
The rete binary (crate rete-cli). Run rete <command> --help for the
authoritative flags. Terms are written as canonical N-Triples tokens —
<http://ex/Alice> for an IRI, "30" or "30"^^<…#integer> for a literal.
Rete-specific --json responses carry "schemaVersion": 1. This covers
Dataset Cards, provenance, search, communities, cost, progressive metadata, and
SHACL reports. Standard SPARQL Results JSON and JSON-LD keep their standard
shapes and do not add that field.
Release archives also contain Bash, Zsh, Fish, and PowerShell completions plus
the rete(1) man page. Maintainers generate these from the exact binary with the
hidden rete generate --output <directory> packaging command; they are never
maintained as hand-written copies of the CLI definition.
Building
rete build <inputs…> -o <out.rete> [--format nt|nq|ttl|trig|rdfxml|jelly|jsonld|yamlld] [--merge-files | --graph-per-file] [--graph-base IRI] [--context IRI=FILE] [--fetch-contexts] [--quoted-triple-syntax rdf12|rdf-star]
Build a file from one or more RDF inputs (files, folders or -), merged under one shared dictionary.
Format is detected by extension (.nt / .nq / .ttl / .trig / .jelly / .jsonld / .yamlld, plus .rdf /
.owl / .rdfxml for RDF/XML — how most OWL ontologies ship); - reads stdin and
defaults to N-Triples; --format forces a format for all inputs (use --format rdfxml for an RDF/XML file with a non-standard extension). N-Quads inputs produce
a dataset with named graphs.
rete build a.nt b.nt -o merged.rete
curl -s https://host/data.nt | rete build - -o data.rete
A folder as input. An input that is a directory is read recursively, and each file becomes a named graph named after it:
rete build papers/ -o papers.rete
rete sparql papers.rete 'SELECT * WHERE { GRAPH <urn:rete:file:2024/march> { ?s ?p ?o } } LIMIT 5'
- Files are read in sorted order of their path relative to the folder, so the same
folder gives the same file on every machine. Hidden files and folders (a leading
.) are skipped, directory symlinks are not followed, and files whose extension is not an RDF one are skipped with one line on stderr naming the first few. A folder with no RDF file is an error.--formatreads every visible file as that format. - The graph is
<base><path>: the path relative to the folder with/separators, the format extension and any.gzremoved, and each segment percent-encoded (every byte but letters, digits and- . _ ~).papers/2024/march.ttl.gzisurn:rete:file:2024/march.--graph-base IRIreplacesurn:rete:file:. Two files that give the same graph (a.ttlanda.nt) are an error naming both. - The triples of a file go to its graph. A quad that names a graph in the file (N-Quads, TriG) keeps it.
--merge-filesreads the folder's files as if they had been listed one by one: no graph per file.--collapse-graphsdoes the same and also folds the files' own graphs into the default graph.--graph-per-filegives a list of files the same graphs, named from the paths as given (relative, without..). It cannot read stdin.- It works with
--memory-budget-mbtoo.
Blank nodes belong to their file, however the files are given (a folder, a list,
merged or not): _:b0 in two files are two nodes, as RDF says, and _:b0 of the
second file is stored as _:f1_b0. A build from one input keeps its labels as
written. --shared-blank-nodes reads the labels of all inputs as one namespace, for
one document cut into several files (a chunked N-Triples dump): there _:b0 in two
files is one node.
Jelly. .jelly and .jelly.gz files are Jelly
RDF streams, a binary format. Both layouts are read: Jelly 1.0 and 1.1 (the row
layout) and Jelly 1.2 (the column layout), triples or quads, with RDF-star quoted
triples and RDF 1.2 triple terms. A file has no magic number, so only the
extension or --format jelly says it is Jelly. It streams one frame at a time, so
a large file is read like an .nq one, including under --memory-budget-mb;
named graphs land in the quad index. A malformed file is refused with the frame
number. Not read: generalized RDF (a literal as subject, predicate or graph),
streams of RDF Messages (the optional 1.2 stream type), lookup tables above
1,048,576 entries, and writing Jelly (rete export has no Jelly format).
JSON-LD and YAML-LD. .jsonld is JSON-LD 1.1 and .yamlld is YAML-LD (its
basic profile: the YAML document is turned into JSON and read as JSON-LD, so the two
give the same quads). .json, .yaml and .yml are not detected, because too many
files share them; name the format: --format jsonld or --format yamlld. Named
graphs (@graph under an @id) become named graphs, and --collapse-graphs folds
them away as for TriG. A blank-node label in a file (_:b0) is renamed per file, so
two files that both use it do not merge their nodes. Relative IRIs are not resolved
against a base, as in Turtle here; JSON-LD drops a relative @id it cannot resolve.
A build is offline: a remote @context is not fetched. A document that needs one
fails with an error naming the context IRI. Either save the context and map it,
--context https://schema.org/=schema.jsonld (repeatable; the file may be JSON or
YAML), or allow the fetch with --fetch-contexts. A JSON-LD document is read whole,
not streamed, so it is not accepted under --memory-budget-mb: convert it to N-Quads
first. The browser engine's build(text, "jsonld") reads inline contexts only.
A long build shows where it is with --progress (a live bar on a terminal by
default; plain lines for logs): see Progress.
--materialize bakes RDFS/OWL-RL entailments into the file at build time (see
Reasoning). --reason runs the same reasoner but instead of
adding triples it stamps the coherence verdict into the Dataset Card (implies
--card) — so a remote reader learns whether the graph is logically coherent from
the index-free card with no compute; unlike --materialize it records
coherent: false honestly rather than aborting (verify later with rete reason --verify-card, combine with --materialize to also bake the inferred triples).
--card (and --card-file / --title / --license / --source /
--description / --created) embeds a Dataset Card —
data-catalog metadata plus an auto-derived profile; the card file's extra
object carries bounded publisher-defined custom fields (no flag — see
Custom fields); a card build also writes a
build record (the build-info section, kept outside the content hash so
identical data still hashes identically): every flag that shaped the file, by
its effective value; the engine (version, git commit, target, profile,
features) and the host's OS and architecture — nothing else about the machine;
each input by file name only with its size, format and a BLAKE3 hashed as
it streamed; the statement counts (read, unique, duplicates dropped, named
graphs); wall time per phase, peak RSS, the memory limit and CPUs seen; the
memory-bounded build's spill; and the measured starter-query costs. See
the build record.
RETE_BUILD_RECORD=reproducible leaves the volatile facts out, so two builds
are byte-identical with it. That same run is what proves each starter query answers:
one measured at zero rows (or returning a row that binds nothing), or one that
does not finish inside the 64 MiB starter byte budget (counted past the open),
is dropped from the
card, loudly and with the reason recorded. The starters themselves are
bounded before they are run: past 2,000,000 statements a template ships a
LIMITed form or, when the card's own profile answers it, nothing — see
Bounded by construction.
--no-card-costs skips the run, and so skips that check as well. --text-index adds a
full-text (word/CONTAINS) index over the literals for rete search --contains
and SPARQL CONTAINS — its scope flags and the memory-bounded build are under
Full-text index below.
--type-predicate <IRI> overrides the predicate that types subjects
with classes for the schema (default rdf:type) —
e.g. Wikidata's --type-predicate http://www.wikidata.org/prop/direct/P31.
--no-pyramid skips the community pyramid entirely (no pyramid section): SPARQL /
SHACL / triple-pattern / reachability queries don't use it, so the file stays
fully queryable and is markedly smaller — only community / summary / progressive
queries need the pyramid. --pyramid-algo louvain|types picks the community
algorithm: louvain (default) detects topological communities; types partitions
by rdf:type instead — deterministic, self-naming communities that stay feasible
on graphs too large for the single-threaded Louvain build (it falls back to
Louvain when the graph is untyped). These are opt-in; without them the output is
a plain build.
Labels and schema have sections of their own, outside the pyramid.
--label-index writes a LABEL_INDEX section (kind 8): the display label of
every labelled subject — rdfs:label, skos:prefLabel/altLabel,
foaf:name, dc/dcterms:title, schema:name, and for people who carry only
given and family names, "family, given" — sorted for
rete search <prefix>.
It has no cap, and a remote prefix search reads a few small ranges of it and no
dictionary. Without the flag the file has no label index at all: the
pyramid-meta no longer carries the 8,192-entry block it used to. Every build also
writes a SCHEMA section (kind 9) — the class hierarchy and level rollups, the
coherence axioms, the planner's per-predicate query_stats and the
characteristic sets — computed without the pyramid, so --no-pyramid and
--memory-budget-mb files have them too; it is a few kilobytes, and --no-schema
leaves it out. Both flags work with --no-pyramid, any --pyramid-algo and
--memory-budget-mb. In format 0x06 both sections are inside the content hash and
have their own entries in the root (SPEC §4.8);
only in a 0x05 file were they outside it, which is why an older rete verify
still passed there (compatibility.md). The publish recipe is
--pyramid-algo types --label-index --card.
rete build data.nt -o data.rete --no-pyramid --label-index # labels + schema, no pyramid
rete build huge.nq.gz -o huge.rete --memory-budget-mb 3000 --label-index
The memory-bounded build (--memory-budget-mb N). It writes format 0x06 directly: the sections stay in the spill, and the one write copies each of them to its final place in the output and deletes it, so the disk holds the spill (its peak is in the sort phases, and all of it when the write starts) plus an output that grows file by file as the spill shrinks: at any moment at most the data of the file being copied is in both (no intermediate file, no convert pass). A failure before the first spill file is deleted keeps the whole spill and prints its path; the output is written under a working name beside the final one and renamed once synced. It reads the input
once and cuts it into chunks of about half the budget, or smaller where the
chunk's dictionary would take its working set past 85% of the budget (0.4.2):
a term-dense stretch, two new terms per statement like the OpenCitations Meta
oc:doiLink tail, used to take the chunk phase a quarter past the budget
(14.99 GB under a 16 GB cap at 12,000 MiB). On a synthetic of 8 M ordinary and
20 M such statements, 1,000 MiB now peaks at 857 MiB instead of 1,348 MiB, and
3,000 MiB at 2,544 MiB in a 4 GiB container, which killed it before; the file
is the same bytes at either budget, and an ordinary graph is cut exactly as
before. Each chunk's dictionary and statements go to --tmp-dir (default: next
to the output). A
k-way merge then builds the one global dictionary, the chunks are rewritten in
global ids one at a time, and external sorts build every index. At any moment
it holds one chunk being sealed, the merge's buffers, or one sort run, and
each of those is sized from the budget, not from the graph. On the
OpenCitations sample (18.9 M statements, --named-graphs quad --text-index --label-index) the peak RSS is 850 MiB at a budget of 1,000 MiB and 2.4 GB at
3,000 MiB, both while the chunks are sealed; eight renamed copies of it (151 M
statements, 186 chunks at 400 MiB) peak at 391 MiB, where 0.4.0 needed
1,084 MiB. --label-index adds nothing to the spill peak, because the label
pass works on ids and reads its strings back from the finished file. Up to
0.4.0 the merge also kept every chunk's id remap table until the last chunk was
rewritten, 4 B per chunk-local term: 15.0 GB for half of OpenCitations Meta,
which is why that graph had to be split.
Its spill is compressed, and holds the statements once. Every spill file
goes through zstd at a fast level, in independent blocks: the statements as
sorted runs of delta-varint ids (1.6 B per statement in a chunk, about 2 B in a
sort run, instead of 12), the chunk term files front-coded (16% of their raw
size), the remap tables, text words and label/schema sorts as compressed byte
blocks. The six permutations are chained: the remap writes the first one's
sorted runs, and each permutation's merge writes the next one's while it
deletes its own, segment by segment, as it reads them. Up to 0.4.1 the
permutation phase held a raw global.tri beside one permutation's raw runs,
24 B per statement before any index byte, and that is what made a one-file
OpenCitations Index (≈ 15.9 B statements) need some 550 GB of scratch. The
finished sections are deleted as they are copied into the output, so the write
does not hold the index twice. The spill now peaks in the last permutations,
where the finished sections — the index itself — are most of it: plan for about
the output's size plus 2–4 B per statement. On the OpenCitations Meta sample
(5.97 M statements, --text-index --label-index, 256 MiB) the build record's
peak fell from 245 MB to 76 MB; on eight renamed copies of it (47.8 M
statements, 512 MiB), from 1.98 GB to 0.51 GB, for 1% more wall time (8% on the
small sample). The files are byte-identical. RETE_SPILL_ZSTD_LEVEL sets the level (default 1; 0 writes
the delta-varint runs and front-coded terms without zstd; negative levels are
zstd's faster modes).
Its sections are byte-identical to the in-RAM --no-pyramid build's of the same
input. With a card (--card, --card-file, a curated flag) it writes the
full card — the derived profile and the starter queries, not counts alone:
once the index sections are on disk, it streams them into the profile and runs
every starter query against the file, exactly as
rete recard
does (one derivation, so the two write the same card for the same file), inside
the same --memory-budget-mb (one byte per subject for the class map, the rest
for the reader's caches). --no-card-costs skips the queries. The card is derived from
the finished sections before the output exists, so it goes into the card region
of the one file the build writes; the starter queries' costs are then measured
again on the file as written (they are a property of its layout). On input
with no duplicate statements and no multi-typed subjects the card, and so the
hash, is the in-RAM build's. The
derivation's time and peak RSS are the card_derive and
starter_queries phases of the build record, and the hash pass its
content_hash. On the OpenCitations sample: see
Maintaining a published card.
The starter queries stay inside the budget. A query's own working rows (a
join's build side, a GROUP BY's groups) are not a cache, so the budget cannot
shrink them. Instead each starter query runs under a ceiling on the process's
resident memory, --memory-budget-mb or what the process holds when the cost
step begins plus 32 MiB, whichever is more, and is stopped when it passes it.
When the process is already past the budget at that point (at scale it is: a
build can hold 13.9 GB on a 9,000 MB budget), the budget says nothing the
process can meet, so the ceiling is the container's memory limit (cgroup v2
memory.max, or v1 memory.limit_in_bytes) less a margin of 10 % or 512 MiB,
whichever is larger, or the process plus 32 MiB if that is more: the guard stops
a query before the container's OOM killer does. It is stopped at its next row,
so the process can pass the ceiling by what one buffer's growth allocates: the
peak stays within the budget plus 128 MiB. The build goes on.
A query the ceiling stops is kept in the card. Whether it trips depends on
where the allocator stands against the ceiling, and the card is inside the
content hash, so memory never decides what the card holds: a bounded build at
any budget has the content hash of the in-RAM build. The query's entry in
query_costs (outside the hash) says status: over_budget (see
the build record),
with the bytes it had read and no rows. Only the content, and the byte budget
(which counts reads and gives the same answer every time), decide which
queries the card keeps.
rete recard and rete merge do the same under their own
--memory-budget-mb. Without a budget (the in-RAM build) nothing changes: the
reader's caches are sized from the free RAM and no query is stopped. The ceiling
reads /proc, so it is a Linux feature; elsewhere no query is stopped.
RETE_COST_MEMORY_HEADROOM_MB replaces the 32 MiB (the tests set it to 0).
The costs are measured twice, and only the second set is recorded. The first
run is against the finished sections, before the output exists, and decides
which queries the card keeps: the card is inside the content hash, so it has to
be final before the file is. The second is against the file as written, because
in 0x06 a query's bytes depend on the card region (the open reads the first
64 KiB). Both are the starter_queries phase, and the record's peak RSS is the
whole process's, the second measurement and the write included.
--strict refuses input carrying an IRI that the N-Triples/N-Quads IRIREF
grammar and RFC 3987 disallow. It is off by default and the build only
counts them — see Invalid IRIs for what that means and why the
default is a warning.
Quoted triples on input (--quoted-triple-syntax). The same flag name and
the same two values rete export takes, pointed the other way: which surface a
<< … >> in a Turtle or TriG input means. rete validate takes it too.
| value | << s p o >> reads as | <<( s p o )>> reads as |
|---|---|---|
rdf-star (default) | a quoted triple — one term, subject or object position | a syntax error |
rdf12 | a reifier: _:r rdf:reifies <<( s p o )>> plus a statement about _:r | a triple term, object position only |
The same file parses into two different graphs under the two values. That is
not a rete quirk — RDF-star and RDF 1.2 genuinely give << s p o >> different
meanings, and nothing in the bytes says which one the author meant, so it cannot
be detected and has to be told.
rdf12 also reads the rest of RDF 1.2 Turtle: {| … |} annotations (which
expand to a reifier carrying the annotation's properties) and "…"@lang--dir
directional literals.
Why the input default is rdf-star while the export default is rdf12.
They answer different questions. Export asks who can read this? — and current
parsers want RDF 1.2. Input asks what did the author mean?, where the two
possible mistakes are not equal:
- An RDF-star file read as
rdf12parses, and gives you a different graph. Silently. That is the failure this flag exists to prevent, and it is the same one #262 found on the write side. - An RDF 1.2 file read as
rdf-staris a hard parse error, because<<(belongs to RDF 1.2 alone. rete detects that case and names the flag in the error rather than complaining about a stray(.
Only one of those is recoverable, so the default is the one that makes the other
mistake loud. The consequence to know: rete export --format trig writes
rdf12 by default, so re-ingesting rete's own dump takes
--quoted-triple-syntax rdf12 — or export with --quoted-triple-syntax rdf-star and it re-ingests with no flag at all.
N-Triples, N-Quads and RDF/XML ignore the flag. Their reader is rete's own,
it has accepted both << s p o >> and <<( s p o )>> since 0.3.2, and RDF 1.2
N-Triples has no reifier syntax for << … >> to be — so there is no ambiguity
to resolve and nothing to choose.
Nothing is stored differently. Either surface lands rete's one canonical
token <<s p o>>, and an RDF 1.2 reifier is ordinary RDF — a blank node, an
IRI and a term — which rete stored long before the flag existed. A file with no
quoted triples builds byte-for-byte identically under either value. See
Triple-store interop,
which also covers using the two flags together to translate between the two
worlds.
--permutations 3|6 (default 6) chooses how many index permutations to
store. 6 writes SPO, POS, OSP, SOP, PSO and OPS; 3 writes only SPO, POS and
OSP — the orders that decide routing. Those three tie the longest bound prefix
on all eight triple-pattern shapes, so a 3-permutation file answers every query
with the same rows, from the same tiles; what it gives up is the sort-merge
join, on the three (bound-set, join-column) shapes SOP/PSO/OPS existed for —
most visibly ?s <p1> ?o1 . ?s <p2> ?o2, two bound predicates sharing a subject.
The three orders are 36.8%–50.5% of a built file
(BENCHMARK.md), and
the file records its own set in the header, so rete info / rete stats and the
card's signals.permutations all report it.
A 3-permutation file is not readable by a Rete older than this one — it
refuses loudly (malformed container: expected 6 permutation sections, exit 1)
rather than answering wrongly, but it does refuse. Keep the default for anything
published; see compatibility.md.
--cluster-subjects-by-class (off by default, format 0x07) lays the default
graph's subject-leading permutations (SPO, and SOP when stored) out in
(class, subject, predicate, object) order instead of subject order
(SPEC.md §6.6).
Ids are not renumbered: only the tile order changes. Up to 32 classes (the ones
with the most subjects, at least 1,000 each) get a contiguous run of tiles, a
subject of several classes goes to its rarest listed class, and every other or
untyped subject to a run of its own. A scan of one class's subjects, or the join
of them with a property (?r a <Software> . ?r <subject> ?t), then reads that
class's few tiles instead of one tile in every few thousand; a lookup of one
subject costs more, since it must look in every run (one batch of leaves and one
of tiles, up to 33 tiles). The answers are the same either way; the order of an
unsorted result may differ. The writer needs one byte of RAM per subject id
(951 MB for OpenAIRE's 951 M subjects). rete info lists the runs and rete verify checks that every row is in the run of its subject's class;
rete convert with the flag clusters a 0x06 or 0x07 file, without it undoes
a clustering. It does nothing to the default graph when it has no subject of a
listed class.
The flag also clusters the quad index's SPOG, by its own classes: the
rdf:type rows of any named graph, at most 10 of them (a lookup of one subject
reads one leaf in every run, and a dump with named graphs is large, so the runs
are fewer than the default graph's 32). GRAPH ?g { ?s a <C> . ?s ?p ?o } then
probes SPOG for the subjects of C, which are one run of tiles, instead of
walking every graph that holds a C. The planner takes the probe unless it
estimates more reading than the walk would do, and keeps the walk under a small
LIMIT; an unclustered file takes the probe too, which already saves requests
(RETE_CLASS_ROUTE=probe|walk forces one). On the 2018 nanopub dump the flag
takes 75 s to 80 s and shrinks the file by 1.3% (SPOG by 8%), and a class of
653,639 subjects goes from 214 s to 59 s over a 120 ms, 4 MB/s link; see SPEC
§6.6 for the rest.
rete info lists the SPOG runs, rete verify checks them.
Derived IRIs (format 0x08, on by default): the builder looks for IRIs the file
itself spells out, and does not store them
(SPEC.md §5.3). A subject IRI that
is its own objects' texts put together, byte for byte, is one: in the OpenCitations
Index the citation …/ci/<citing>-<cited> is the objects of its hasCitingEntity
and hasCitedEntity triples. There is nothing to write and nothing dataset-specific:
the writer samples the subject-only dictionary section, finds the template and the
predicates by itself, rebuilds every term of the section from the index and compares it
byte for byte, and drops the stored text of each chunk whose terms all match (a chunk
with a term that does not match stays as it is). It takes a family only when the bytes
it removes are at least 5% of the file (and 256 KiB): a derived name costs a lookup of
one name two more requests and rete export about twice the time, so a small saving is
not worth it. Ids, order and answers are the same. On a sample of the OpenCitations
Index the file is 10.5% smaller and the list of works citing one work reads half the
bytes. It needs the PSO permutation (the default) or an unclustered SPO, and the file
says so in its header: a Rete that does not know the flag refuses it by name.
--no-derive-iris keeps every IRI as text. rete info and the build record
(derive_iris) say what was found and why a family was taken or left; rete verify
rebuilds the dropped chunks. Rete::query_ranged (the one-request single-pattern query)
refuses such a file; sparql-url and the rest read it.
--named-graphs quad (the default, and since format 0x06 the only layout)
stores the named graphs of an N-Quads/TriG input in the quad index. The
0x05 alternative, per-graph (one permutation index per graph, the
NAMED_GRAPHS section), is refused; rete convert re-encodes a per-graph file as
a quad index. The quad index is (SPEC.md §6.5):
every named-graph quad stored four times, as GSPO (the graph first, in tiles
that many graphs share, behind a sorted graph directory) and as SPOG, POSG and
OSPG (the graph last), with graph names as dictionary terms. The three graph-last
orders make GRAPH ?g { <s> ?p ?o }, GRAPH ?g { ?s <p> ?o } and
GRAPH ?g { ?s ?p <o> } a seek, not a pass over the file: on the 3.9 GB nanopub
dump a bound subject across graphs went from 28,179 requests and 333 MB to 18
requests and 0.29 MB, and the file got smaller (3.88 to 2.51 GB). --permutations
is the default graph's: the quad index has these four orders whatever it says. The rows a query returns
are the same either way. What changes is the cost of many small graphs — on the
OpenCitations sample (1.5 M per-entity provenance graphs of ~9 quads) the
named-graph section drops from 974 MB to 102 MB, the build from 33 min to 77 s,
and a remote GRAPH <g> query from 414 MB in 81 requests to 3.7 MB in 26
(BENCHMARK.md).
The layout applies to every build path, --memory-budget-mb included;
repyramid and merge keep the layout of their input.
A quad file is refused by every Rete before 0.4.0 — loudly, with the same
container section-count error as above, on every query, export and remote open
(info, verify and card-url still work). Keep per-graph for a file older
engines must read.
The flag exists on build only. The two commands that re-assemble an existing
file do not re-decide how it was built: repyramid
preserves its input's set (and says so on stderr when that set is not the
default six), and merge
writes the union of its inputs' sets — one full shard is enough to keep the
merge-join orders its queries relied on. To change a file's set, rebuild it from
its RDF source.
Full-text index: --text-index and its scope
--text-index adds a TEXT_INDEX section: every word of every literal object in
the index's scope maps to the subjects that carry it. rete search --contains answers from it, and SPARQL uses it to prune a FILTER(CONTAINS(…))
before re-checking every row. Three flags set the scope; each needs
--text-index, and repyramid takes them too:
| flag | default | what it covers |
|---|---|---|
--text-index-graphs default|all | default | default: the default graph's literals — the scope of every index built before 0.4.0. all: those and every named graph's. |
--text-index-include <IRI> | every predicate | only the literals of this predicate; repeat the flag for several |
--text-index-exclude <IRI> | none left out | every predicate's literals but this one's; repeat for several |
The IRI is a full one, bare or in <…>; a prefixed name is refused, because a
build has no prefix map to expand it with. Include and exclude do not combine: an
include list already says exactly what is indexed. A listed predicate that no
literal in scope carries is reported on stderr, since a misspelt exclude would
otherwise exclude nothing.
rete build data.nq -o data.rete --text-index
# OpenCitations-style: index the provenance graphs too, but not the generated
# change notes or the stored SPARQL update text
rete build oc.nq -o oc.rete --text-index --text-index-graphs all \
--text-index-exclude http://purl.org/dc/terms/description \
--text-index-exclude https://w3id.org/oc/ontology/hasUpdateQuery
The scope is written into the section (SPEC §6.3), so a reader acts on what the file covers rather than on how it was built:
rete search --containsfinds only words of literals in scope.- The SPARQL
CONTAINSpushdown runs only where the index covers the query: the active graph must be the default graph (or, underall, any graph), and the pattern's predicate must be indexed — a variable predicate only when every predicate is, or when the filter itself confines it to indexed predicates:FILTER(?p NOT IN (…))naming every predicate--text-index-excludeleft out (or?p != <…>for each),?p IN (…)/?p = <…>naming only indexed ones. The filter is pushed through aUNIONbranch by branch, intoGRAPH <g>(when it has noEXISTS), and, underallon a quad-index file (--named-graphs quad), intoGRAPH ?g, which then visits only the graphs holding a candidate subject. The index holds literals only, andSTR()makes an IRI's or a blank node's text searchable too, soCONTAINS(STR(?o), …)takes the index only where?ois provably a literal:isLiteral(?o), a comparison onLANG(?o)orDATATYPE(?o), or a constant predicate whose objects are all literals (read off its tiles' synopses, in the default graph or aGRAPH <g>). Anywhere else the filter scans and returns the same rows, just without the shortcut. (Before 0.4.0 the pushdown also ran insideGRAPHagainst a default-graph index, where it returned no rows at all; from 0.3.0 to 0.4.1 it dropped the rows whoseSTR(?o)matched an IRI.) - An include/exclude index is stored as section kind 11, not 6. Readers
older than 0.4.0 do not know kind 11 and see no text index:
rete search --containssays the file has none, andCONTAINSscans. That is on purpose. Those readers pruneCONTAINSwith whatever text index they find, as if it covered every literal; with a predicate-restricted one they would silently drop the rows of the predicates it left out. A--text-index-graphs allindex over every predicate stays kind 6, because it covers more than they assume, never less.
A carded build records the flags in its build-info (text_index_graphs,
text_index_include, text_index_exclude), next to text_index.
What a scope saves, measured on the OpenCitations Meta sample (M=1200, 18,869,242 statements) laid out with its provenance in the default graph, memory-bounded at 3,000 MiB. The file without a text index is 166,562,617 B.
--text-index scope | tokens | section | token table | file |
|---|---|---|---|---|
| every predicate | 2,178,475 | 68,256,530 B | 9,996,927 B | 234,819,147 B |
all but oco:hasUpdateQuery | 2,164,679 | 60,352,482 B (−12%) | 9,770,745 B | 226,915,099 B |
all but dcterms:description | 831,343 | 36,789,772 B (−46%) | 4,667,222 B | 203,352,389 B |
| all but both | 534,449 | 23,825,664 B (−65%) | 3,177,349 B | 190,388,281 B |
all but both and prov:generatedAtTime, prov:invalidatedAtTime | 533,705 | 10,199,845 B (−85%) | 3,156,034 B | 176,762,462 B |
The last row is the data alone: its postings are byte for byte those of the same sample built with provenance in named graphs and the default scope, and it finds the same subjects for every data word. The provenance prose ("The entity … has been modified.") and the stored SPARQL update text are 65% of the full index, the timestamps another 20%. The token table, which a first remote search fetches whole, shrinks from 10.0 MB to 3.2 MB.
The other way round: with the provenance in named graphs (the layout
OpenCitations ships), the default scope indexes the data alone (10.2 MB), and
--text-index-graphs all indexes the same literals as the first row above —
the same 2,178,475 tokens and the same postings, 68.3 MB, +5.4% on the 1.07 GB
file — so the provenance becomes searchable without moving it out of its
graphs.
In the memory-bounded build. --text-index, with any scope, works with
--memory-budget-mb, and the TEXT_INDEX section is byte-identical to the
in-RAM build's for the same input. Each chunk writes its literal words with
chunk-local subject ids to --tmp-dir while it is sealed. The ids are remapped
with the tables the dictionary merge already produces, sorted externally by
(word, subject) in runs of at most a quarter of the budget, and merged into
the section. The token table is the one part a reader must hold whole, so the
build streams it to disk and compresses it from there; the in-RAM build
compresses it with the same streaming encoder, which keeps the two identical
past zstd's 4 MiB window (0.4.0 did not: its memory-bounded token table could
come out a few KB larger). extbuild: text index … reports the token and
record counts.
Measured on the same sample with provenance in the default graph (50.5 M word records, 2.18 M tokens): at 1,500 MiB the build took 83 s instead of 59 s, peak RSS 1.19 GB against 1.17 GB without the index, and the peak spill grew from 0.91 GB to 1.38 GB. The peak is set by the chunk phase, not by the text sort, so it does not move with the index. With provenance in named graphs and the default scope (3.3 M records) the text phase takes under a second of a build whose time goes to the 1.49 M named graphs.
Compression knobs (--tile-kib, --dict-chunk-kib, --zstd-level,
--frequent-objects). How the index and dictionary are cut and compressed.
None of them changes the file format: every combination opens in every reader of
format 0x05, released ones included (see
compatibility.md),
and every one changes the bytes but never an answer. The measurements behind the
defaults are in BENCHMARK.md.
| flag | default | what it trades |
|---|---|---|
--tile-kib N (4–4096) | 64 | Uncompressed size of an index tile, the unit a reader fetches and decompresses. 128 makes the file about 1% smaller and each touched tile larger; above 128 the file grows again (zstd's small-input parameters stop applying) and remote queries fetch more bytes (+9% at 256, +24% at 512). |
--dict-chunk-kib N (4–4096) | 64 | The same for dictionary chunks. |
--zstd-level N (1–22) | 9 | Level of every tile and chunk. 19 makes a file 4–9% smaller than 9 for about 2.7× the build CPU, and decodes 8–18% slower: a choice for a file built once and read for a long time. Levels between 9 and 19 gain little or nothing. |
--frequent-objects N | 0 (off) | Moves the N most frequent object-only terms (classes, controlled vocabularies, status literals) into the shared dictionary section, which gives them small ids where they sort ahead of the shared entities — the size effect of declaring the vocabulary in a TBox, without one. 64 made the OpenCitations samples 4.8–6.5% smaller, against 2.7–5.5% for their 499-statement TBox, and two other datasets 2.3–3.2% smaller, at no measurable build cost. With --memory-budget-mb too (refused up to 0.4.1): its chunks count their objects, one more pass over the chunks' terms picks the same N (most statements, ties by term order), and the file is the in-RAM build's, byte for byte — on the OpenCitations sample 5.75% smaller (data only, 10 chunks) and 3.80% (18.9 M statements with provenance, quad index), for about 1% more build time. |
Each flag falls back to an environment variable when left out
(RETE_TILE_KIB, RETE_DICT_CHUNK_KIB, RETE_ZSTD_LEVEL,
RETE_FREQUENT_OBJECTS), which is also how merge and repyramid take them
(merge runs the external builder, so it ignores RETE_FREQUENT_OBJECTS). A card
build records the settings in its build info (params.write).
Since 0.4 every tile and chunk is a single-segment zstd frame that records its
size, which lets zstd use its small-input parameters: at the same level the
four measured datasets came out 0.7–7.0% smaller, and the build used less CPU. RETE_ZSTD_FRAME=stream writes
the streaming frames of 0.3 instead, byte for byte.
rete recard <file> (-o <out.rete> | --in-place) [--card-file F | --keep-curated] [--title …] [--no-card-costs] [--memory-budget-mb N] [--verify]
Give a finished .rete a new Dataset Card without rebuilding it — the
re-card path for a file of any size.
The card is derived by streaming the file's own sorted sections: the
rdf:type range of POS gives every typed subject its class, SPO gives each
subject's out-degree and namespace the moment the subject changes, and OSP each
object's in-degree and every literal object, decoded once with the multiplicity
of each predicate that uses it. Every tally is kept per vocabulary term
(predicates, classes, datatypes, languages, namespaces), never per statement;
the one per-term structure is a class code per subject id — one byte per
subject below 255 classes, two above. Memory is that plus the reader's caches
(--memory-budget-mb, default 1024), not the statement count. The
memory-bounded rete build runs this same derivation for its own card, so a
re-card of its output writes the card it already has.
Then the starter queries are generated, each is run against the file inside the
starter byte budget (64 MiB past the open, see Dataset Cards)
and inside --memory-budget-mb (the process may not pass it by more than
128 MiB; see "The starter queries stay inside the budget" under rete build).
The ones that do not answer within the byte budget are dropped and recorded; one
the memory ceiling stops stays in the card and is marked over_budget in
query_costs. With --in-place the costs are measured again on the file as
written. With
-o, the file is rewritten with only the card (metadata section) and the
build record (build-info section) replaced: every other section — dictionary,
index, pyramid, named graphs, text index, any section kind this binary does not
know — is copied byte for byte, in its place in the file order, the header's
offsets are fixed, and the content hash is recomputed while the bytes go past
(the card is inside it). One read of the file to derive, one to copy;
--verify adds a third to check the written hash before the file is moved into
place. The output must not exist: a re-card writes a new file and never over
its input.
--in-place skips the copy. On a format 0x06 file the new card and
build record are written into the free part of the card region, beside the
old pair; the file is synced; then the 128-byte header is rewritten to point at
them, with the new content hash, and synced again. No data byte is read or
moved: the content hash covers the card and the root's section digests, not the
data itself (SPEC §4.4). When the new pair
does not fit the region's free space, the file is copied once instead (a fresh
card region, the data byte for byte) and renamed over the input; the summary
line says which happened. The writer sizes the region at twice the card and
record plus 8 KiB, so a re-card of a similar card fits.
On a 0x05 file the new card and build record are written
after the file's last section, followed by a new footer; the file is
synced; only then is its 1 KiB header rewritten to point at them, with the new
content hash, and synced again. That costs one read of the hashed sections (the
card is hashed first, so the hash can only be folded once it exists) and a few
KB of writes, instead of a second copy of the file. Nothing the old header
points at is written over, so a crash before the header write leaves the old
file exactly as it was, readable by every reader, plus bytes past its end that
the next run overwrites; a crash after it leaves the new one. The old card,
build record and footer stay in the file as dead bytes — a few KB, more on a
card with large curated fields — that no section points at; every reader
ignores them, and a re-card with -o drops them. --verify re-reads the file
after the header write. A file re-carded in place, or built memory-bounded with
a card, keeps its card after the data; every reader
since the format froze finds it there.
The curated half comes from --card-file and the curated flags exactly as
rete build --card-file takes them, or with --keep-curated from the input's
card, verbatim — description, example queries and extra bag included. With
neither, an input card that carries curated fields is refused rather than
silently stripped (the defect of a bare repyramid --card).
The build record keeps the original build's timestamp, builder and parameters —
the sections they describe were copied, not rebuilt — and gains a recarded
entry (at, by, from_content_hash, and the re-card's own engine, host,
input by file name, params — with in_place — and run); its
query_costs and dropped_queries are the re-card's own measurement. In place,
the record is sealed before the final write, so its run covers everything but
those few KB.
Two differences from the card an in-RAM build derives, both because a finished file is read rather than its input: counts are over the deduplicated statements the file holds, and a subject typed more than once is classified by its class with the highest dictionary id (index order). On input with neither, the two cards are identical, byte for byte.
rete recard old.rete -o new.rete --keep-curated --verify
rete recard old.rete -o new.rete --card-file curated.json # curated fields from a document
rete recard big.rete --in-place --keep-curated # no copy: card region, header last
bash scripts/recard/recard.sh --source /work/old.rete --out /work/new.rete # with both proofs
Measured on an OpenCitations Meta part (26.5 GB, 2,904,375,666 statements, 978,932,751 terms): see Maintaining a published card.
rete repyramid <file> -o <out.rete> [--type-predicate <IRI>] [--pyramid-algo louvain|types] [--text-index] [--label-index] [--card …]
Rebuild a file's pyramid in place, reading the triples straight from the
existing .rete — no export | build N-Quads round-trip. Use it to add a schema
(or a --text-index, a --label-index / a Dataset Card) to a file built before
those existed, or to re-derive the schema pyramid under a different --type-predicate
or --pyramid-algo (same semantics as rete build). The card flags match
rete build (--card-file / --title / --license / …) — and take the
curated half only from those flags, so a bare --card drops the
publisher's title/license/source/description. --pyramid-algo likewise
defaults to louvain rather than to whatever the file was built with.
It loads and materializes every quad, so its RAM tracks the statement count
(~350–700 B each), not the file size — roughly 80 M statements on a 48 GB
machine. To change only the card, use rete recard,
which copies every other section instead of rebuilding it and has no such
ceiling; repyramid is for when the pyramid or the text index must change too.
See Maintaining a published card.
rete repyramid old.rete -o new.rete --type-predicate http://www.wikidata.org/prop/direct/P31
rete repyramid old.rete -o new.rete --text-index # add full-text search to an existing file
# re-card AND re-pyramid a published file, carrying its curated fields across
# (to change only the card, `rete recard old.rete -o new.rete --keep-curated`)
rete card old.rete --json > old-card.json
python3 scripts/recard/card_tools.py curated old-card.json -o curated.json
rete repyramid old.rete -o new.rete --card --card-file curated.json --pyramid-algo types
repyramid preserves its input's permutation set: a 3-permutation file stays
3-permutation, a 6 stays 6, and it prints the preserved set on stderr when it is
not the default six. It re-assembles an existing file rather than re-deciding how
it was built, so it has no --permutations of its own — rebuild from source to
change the set. It keeps a LABEL_INDEX the input carries (--label-index adds
one), and writes the SCHEMA section and the community-only pyramid-meta rete build
writes. The named-graph layout is kept the same way: a quad-index input
(--named-graphs quad) comes out as a quad-index file, and merge writes one
when any of its inputs is one.
rete convert <in.rete> -o <out.rete> [--tmp-dir dir] [--cluster-subjects-by-class] [--no-derive-iris] [--no-label-index] [--no-card-costs] [--memory-budget-mb N]
Turn a format 0x05, 0x06 or 0x07 file into format 0x08
(SPEC §4.11) without going back to the
RDF source. Format 0x08 keeps its dictionary in the SPARQL order
(SPEC §5.2), so every term gets a
new id and every structure that holds ids has to be written again: a conversion is a
re-encode, not a copy of the payload. The statements are walked through the source
(never as text) and written by the memory-bounded builder, rete merge of one file:
the terms are sorted and merged, every index and quad order is sorted and tiled again,
and the label index, the text index (with its scope), the SCHEMA section and, with
--cluster-subjects-by-class, the class clusters are rebuilt for the new ids. The
card, its annex and the build record are kept. The input may be a path or an http(s)
URL.
Derived IRIs are found by convert as by rete build (on by default; see there):
the conversion's writer looks for IRIs the file spells out and drops their stored text
when that is at least 5% of the file. --no-derive-iris keeps them as text. A file that
already has derived chunks keeps them as they are; there is no flag to store them again.
--memory-budget-mb (default 2048) is the builder's working-set target and the
ceiling of the card-cost measurement; the spill (sorted runs, a few bytes per statement
and order) goes to --tmp-dir, by default beside the output. The 2018 nanopub dump
(378.7 M quads, 2.48 GB) is re-encoded in 28 minutes at a peak of 2.8 GB under
docker --memory 4g with --memory-budget-mb 2800, the 202 MB OpenCitations sample in
110 s at 2.0 GB; with --cluster-subjects-by-class the dump was converted with
RETE_QUAD_RUN_ROWS=50000000 (the default sort run is 100 M rows; not measured under 4 GB). See the
CHANGELOG for the figures.
The card is carried as it is (its format_version says 8). The build record is carried
too, with a conversions entry added that names the converting engine, the time, the
command and flags, and the source's format, content hash and size (a file
without a record gets one holding just that). The output has the content hash
rete build gives the same data and flags — not the 0x05 file's, since the
format is part of the identity — and, record apart, the same bytes; the record
is outside the hash. RETE_BUILD_RECORD=reproducible makes two conversions of
one file byte-identical.
The card's starter queries are run again, cold, on the converted file, and their
costs are recorded as query_costs in the new conversions entry — the source
record's own costs were measured on the old layout and are kept as they were,
but no longer describe this file. rete card, card-audit and
build_record() show the conversion's. --no-card-costs skips the run;
--memory-budget-mb bounds it as it does the build's.
A source with a community pyramid or with per-graph named graphs goes through the bounded builder like any other:
- A community pyramid is written as it is: the same communities, superedges and
descriptors, with the superedges under the predicate ids of the new dictionary. It is
not computed again, so
rete summaryprints the same thing before and after, whatever--pyramid-algobuilt the source. Its schema, statistics and labels are sections of their own in0x08; a file whose only labels are the pyramid's legacy block gets a label index (--no-label-indexleaves it without one). The summary is held in memory while it is written, outside--memory-budget-mb: about 64 bytes per superedge at the peak (2 GB for the 32 M superedges of a Louvain pyramid over 51 M statements). - The schema's type predicate is the source's. A
0x06or0x07source names it in its SCHEMA section. A0x05pyramid does not, so it is taken from the data: of the predicates that reach the least frequent class of the source's finest rollup, the one with as many statements as that class has instances (rdfs:subClassOf,rdfs:rangeand the like are not considered). A file built with--type-predicate, or one withoutrdf:typewhose typing predicate the build found, keeps its classes; the lineconvertprints saysschema typed by <…>when it is notrdf:type. - Per-graph named graphs (the
0x05NAMED_GRAPHS section) are walked graph by graph and written in the quad index.
Named graphs come out with the four 0x07 quad orders (GSPO, SPOG, POSG, OSPG, SPEC
§6.5); the graph-leading G+POS and the others of a 0x06 input are not written.
--quad-clustering is accepted for 0x06 scripts and ignored: 0x07 and 0x08 are
order-major, always.
What a converted file is. The same statements under the same card: a query returns the
same rows from the input and the output (the tests compare the sorted rows of every query
and the exact order of every ORDER BY). It is the file rete build writes for the same
data and flags, record apart, byte for byte (the content hash is the build's), unless
the source has a community pyramid: the converted file has the source's, a build computes
its own. Converting
twice, or converting a 0x08 file, gives the same bytes; a 0x08 input is only written
again under the current directory-leaf policy, payload copied (about two seconds for the
2.4 GB nanopub dump). rete verify of the output checks the dictionary's order.
rete convert davidrumsey.rete -o davidrumsey-08.rete
rete convert nanopubs.rete -o nanopubs-08.rete --memory-budget-mb 2800 --tmp-dir /scratch
rete verify davidrumsey-08.rete
rete merge <inputs…> -o <out.rete> [--memory-budget-mb N] [--tmp-dir dir] [--card …] [--no-card-costs]
Fold several .rete files into one without going back through text — the way to
consolidate a sharded dataset when the original RDF is gone or expensive to
re-emit. It reads the shards (dictionary-encoded and compressed, roughly a
quarter of the RDF bytes), so it skips the conversion and the parse.
merge writes format 0x06 directly, the way the memory-bounded build does: the
spill plus one output on disk, no intermediate file. Its output is the direct
build of the union of the inputs, byte for byte.
It does not skip the sorting. The dictionary is HDT-style (shared /
subject-only / object-only) and IDs are assigned per section, so a term that is
subject-only in one shard and object-only in another becomes shared in the
merge and changes ID section: the shards' orderings do not survive the remap and
every permutation is rebuilt. Memory is bounded at both ends, by the one
--memory-budget-mb (default 4096): each input is opened by range and
streamed, never read whole, with its dictionary, tile and block caches capped
to a quarter of the budget (64 MiB to 1 GiB) and each named graph's index
dropped once the graph is walked; the quads feed the same memory-bounded
external builder rete build --memory-budget-mb uses, whose chunks take the
rest of the budget while the inputs are read, and which spills into
--tmp-dir. Shards carrying named graphs fold like any other: the
builder puts the graph in its sort key, so each graph comes back out of the
merge as one contiguous run.
The merged file carries the union of its inputs' permutation sets — an
all-lean merge stays lean, and one six-permutation input is enough to keep the
merge-join orders. Like repyramid, it has no --permutations flag: it
consolidates shards, it does not re-decide how they were built.
The card. Card flags match rete build (--card, --card-file, the
curated flags, --no-card-costs). With one, the merged file gets the full
card — the derived profile and the starter queries — exactly as the
memory-bounded rete build writes it: once the merged file's index sections are
on disk they are streamed into the profile and every starter query is run
against the file, by the derivation
rete recard
runs (so a re-card of the merged file writes the same card, byte for byte),
inside this command's --memory-budget-mb, the queries' memory included (see
"The starter queries stay inside the budget" under rete build); the card and the
build record then go into the card region, as for the memory-bounded build. The data sections are
the same bytes as
without a card, and the same as rete build --memory-budget-mb writes for the
union of the shards' statements. Up to 0.4.0 a merged card carried counts only
and the full one took a rete recard afterwards; on two halves of the
OpenCitations sample the one pass costs what the merge alone did (see
Maintaining a published card).
The budget covers the merge's walk of its inputs too (0.4.2). Up to 0.4.1 it
bounded the builder alone: a shard small enough was read whole, a larger one
opened with uncapped caches, and on that sample the merge peaked about 1 GiB
over --memory-budget-mb (3,025 MiB at 2,000) and was killed in a 4 GiB
container at 3,000. Now it peaks at 1,551 MiB at 2,000 and 2,174 MiB at 3,000
(1,568 and 2,126 MiB with the full card), the same merged file, for 15–19%
more wall time at 2,000, where the shards' caches are smallest; two shards of
a term-dense synthetic (14 M statements) merge at 2,047 MiB instead of
3,204 MiB at 3,000, in the same time.
The build record names what was merged: each input's file name, size and
BLAKE3 (one sequential read of each shard before the merge, the record's read
phase — a shard is otherwise read by range, in index order, and cannot be
hashed on the way), the effective flags (command: merge, the permutation set
and named-graph layout it chose, the budget, the --card-file digest), the
counts — statements_read counts a statement once for every shard that holds
it, so duplicates_dropped is the shards' overlap — the spill, and the time of
every phase, the card's own (card_derive, starter_queries,
content_hash) with their own peak RSS.
The inputs' own cards are not carried. The merged card's curated half comes
from --card-file and the curated flags alone: a shard's title, licence,
description, example queries and coherence verdict describe that shard, and
neither one of them nor a blend of them describes the merge. When an input's
card carries curated fields, merge says so on stderr; without any card flag
the merged file has no card at all, whatever its inputs carry. To point at the
shards as published files, list them in the card file's derived_from.
rete merge shard-*.rete -o all.rete --memory-budget-mb 8192 --tmp-dir /spill \
--card-file curated.json
rete estimate <inputs…> [--memory-budget-mb N] [--label-index] [--text-index] [--card] [--no-card-costs] [--no-pyramid] [--sample-mb N] [--sample-statements N] [--json]
What a build would cost, before you run it: the size of the .rete, the wall
time, the peak spill and the memory. Pass the flags you will pass to
rete build, because the label index, the text index and the card each add to
the file and to the time.
rete estimate solar-big.nt --memory-budget-mb 512 --label-index --card --no-pyramid
It reads the whole input once (--sample-mb N reads only the first N MiB and
extrapolates), counting statements and distinct terms. It keeps a sample of
--sample-statements statements (default 1,000,000), cut from windows spread
over the input, and builds them in memory with the writer rete build uses. It
measures each section of that file (dictionary, index, label index, text index,
directories) and scales it up: the index, labels and text by the statement
count, the dictionary by the bytes of its distinct terms. An input no bigger
than the sample is built whole, so its size is exact. Nothing is written to
disk. The sample costs 2 to 5 seconds and about 0.4 GB, whatever the size of the
input; the rest of the time is the pass over it.
The spill and the time are projected from those sections and from the parse and
the sample's build timed on this machine, with constants fitted to measured
builds. Every figure is a range. The starter-query measurement of a card (about
6 to 15 seconds, whatever the graph) is a separate line of the time, and
--no-card-costs removes it. A build with --memory-budget-mb writes no
pyramid; without it a pyramid is projected at 18% of dictionary and index,
which is the one figure the sample does not measure. --json prints the same
projection for a script.
Against real builds of two synthetic graphs, from 0.8 to 29.9 million statements, at budgets of 512 and 1500 MiB (the numbers are in the build tutorial and the pull request), every projected range held the real value. The file size was exact to 1% when the sample was the whole input and within 8% of the middle of the range when it was scaled (the 30 M scholarly graph came out 8% over: its dictionary 17% and its index 13% under, because ids and long terms cost more as a graph grows, which a sample cannot show). The spill was within 18% of the middle and the time within 21%. Memory is given as a fraction of the budget (0.2 of it up to 1.15 of it plus 128 MiB): a small graph does not use a large budget, and the 30 M scholarly graph went 27% past a budget of 512.
Progress
rete build (in RAM and with --memory-budget-mb), rete merge and
rete recard take --progress auto|bar|plain|json|off, or the
RETE_PROGRESS variable when the flag is not given. It shows the step and how
many remain, how far through it, the speed, the statements, terms and chunks so
far, memory against the container's limit, the spill on disk, the time elapsed,
an ETA and the finish time.
| Mode | Prints |
|---|---|
auto (default) | the bar when stderr is a terminal, nothing otherwise — logs do not change unless asked |
bar | one line redrawn in place; the build's own lines (extbuild: …) print above it |
plain | a line every --progress-interval seconds (default 30) and one when a step begins, for Docker and CI logs |
json | the same events, one JSON object per line, for scripts |
off | nothing |
All of it goes to stderr. Stdout keeps its one result line, and the file is the same bytes with or without progress (the test suite holds every mode to it). The builders only bump counters; a separate thread samples them, so a build with progress is no slower: on the OpenCitations M=1200 sample (6 M statements) the median of five or six runs per mode stayed within 3% of a build without the feature, either way, inside the run-to-run spread.
progress: build in 10 steps: parse+chunk, dictionary, encode, index_permutations, named_graphs, write, labels+schema, card_derive, starter_queries, content_hash
progress: build 1/10 parse+chunk 61.7% 53.3 MB/s, 714 k st/s | 3.57 M st, 1 chunk | RAM 330 MiB, peak 757 MiB of 4.0 GiB | spill 11.5 MiB | 7s elapsed, ETA 9s ≈ 12:04 31 Oct UTC
progress: build 4/10 index_permutations 29.5% | 5.97 M st, 2.35 M terms, 3 chunks | RAM 191 MiB, peak 757 MiB of 4.0 GiB | spill 27.3 MiB | 16s elapsed, ETA 4s ≈ 12:04 31 Oct UTC (after dictionary 1s, encode 0s)
progress: build done in 20s: 5,971,936 statements, peak RAM 757 MiB, spill peak 56.7 MiB, output 63.7 MB (out.rete)
The bar, at the same point:
[4/10] index permutations ▕██████████ ▏ 50.0% 14.44 M rec/s RAM 186 MiB / 4.0 GiB 0:16 ETA 3s ≈ 12:05 31 Oct UTC
Steps carry the build record's phase names (Phases).
A step the run turns out not to take — named_graphs for an .nq input that
has none — leaves the count when a later one begins. The percentage counts
input bytes as stored for the parse (exact for gzip too), chunk-local terms
for the dictionary merge, statements for the remap, merged records for the
permutations, triples for the card's passes, starter queries, and bytes for
the content hash and a re-card's copy. A step with nothing to count shows a
spinner and its counts, not a percentage.
The ETA is the current step's measured rate plus the steps still to come,
each priced as a multiple of the parse step's duration: the rule of thumb that
a build takes 3–5× its parse, split per step from recorded builds,
so it follows the speed the parse actually reaches. It is marked an estimate (~ on the bar, (estimate) in
plain, eta_estimate in json) until the parse is 5% done and, in the
memory-bounded build and merge, until the first chunk is sealed — the seals
stall the parse, and a rate measured before the first one has none of them —
and again when a step without a count runs past its estimate. On that
sample (builds of 20–27 s, two runs each) the first ETA came in 33–43% short
in RAM, where the card took longer than the model gives it, and the first firm
one 15–17% short memory-bounded, where three chunk seals are a large part of
the parse. The finish time is in UTC.
JSON. One object per line, { first; the other stderr lines are the
build's own messages. The fields are stable:
| Field | Meaning |
|---|---|
event | start (first line), phase (a step began), progress (every interval), then done or failed (last line) |
command, t | build, merge or recard; seconds since the start |
phases (start) | the planned steps, in order; steps their count |
step, steps, phase | step k of n and its name |
ended (phase) | the steps that ended since the last line: {"phase", "seconds"} |
done, total, unit, percent | progress through the step; total and percent are null without a total; unit is bytes, statements, terms, records, queries or none |
rate, mb_per_s, statements_per_s | smoothed speeds: units per second, input MB/s while parsing, statements per second while they are counted |
statements, terms, chunks | running totals |
rss_bytes, peak_rss_bytes, memory_limit_bytes, spill_bytes | memory now and at peak, the cgroup limit (null when there is none), the spill directory's size (null when nothing spills) |
elapsed_s, eta_s, eta_estimate, finish_at | time so far, time to go (null until it can be estimated), whether it is still an estimate, and the finish as RFC 3339 UTC |
output, output_bytes, spill_peak_bytes (done) | the file written; done also has statements and peak_rss_bytes |
On Unix the bar reads stderr through a pipe so that other lines land above it; elsewhere they may land on the bar's line.
Validating
rete validate <inputs…> [--format nt|nq|ttl] [--strict] [--quoted-triple-syntax rdf12|rdf-star]
Parse RDF input(s) without building, to check they are well-formed N-Triples/N-Quads/Turtle. Reports statement and named-graph counts, or exits non-zero with a precise parse error (file, line, column).
rete validate data.ttl
curl -s https://host/data.nt | rete validate - --format nt
It also reports invalid IRIs, because "it parses" and "it is valid RDF" are
different claims and only the second one predicts whether a dump will load
anywhere else. --strict turns the report into a non-zero exit.
--quoted-triple-syntax means exactly what it means on
build, which makes validate the cheap way
to ask which surface is this file actually in? — the counts differ, because
the graphs do:
rete validate claims.ttl # 2 statement(s) — << >> is a quoted triple
rete validate claims.ttl --quoted-triple-syntax rdf12 # 4 statement(s) — << >> is a reifier
Invalid IRIs
rete's N-Triples/N-Quads reader is deliberately tolerant: it stores whatever
sits between < and the next >, so the file is a faithful container of what
it was given. That tolerance is why every published dataset built — and it is
also why rete export --format nq could emit something no strict parser
accepts. build and validate now count what they ingested:
warning: 5 statement(s) carry an invalid IRI (5 IRI occurrence(s)).
They are stored verbatim, so `rete export --format nq` emits N-Quads that a
strict parser (Oxigraph, Jena, rapper) rejects — and a bulk loader rejects the
whole chunk, not the line.
2 '[' or ']' outside an IP-literal host
e.g. <http://example.org/a[b]>
1 more than one '#'
e.g. <http://example.org/c#d#e>
1 '%' not followed by two hex digits
e.g. <http://example.org/%x-template-artifact>
1 a character the IRIREF grammar excludes (space, control, or one of <>"{}|^`\)
e.g. <http://example.org/a b>
Validity is decided by a parser, not by a list
Two documents apply: the IRIREF production
('<' ([^#x00-#x20<>"{}|^] | UCHAR)* '>'), which is a character set, and RFC 3987, which the same grammar requires the content to satisfy as an **absolute** IRI. The first is settled by a scan. The second is settled by **oxiri`** — the
crate Oxigraph's own N-Triples reader validates with — so rete's answer to "will
a strict loader take this" is the loader's answer, not an imitation of it.
The classes below do not decide validity. They answer the other question: can this be repaired, and how.
| class | example | repairable by escaping |
|---|---|---|
| no scheme (relative IRI) | <noscheme/path> | no |
| an excluded character | <http://ex/a b> | yes |
[ / ] outside an IP-literal host | <http://ex/a[b]> | yes |
more than one # | <http://ex/c#d#e> | yes |
% not followed by two hex digits | <http://ex/%x> | yes |
| rejected by RFC 3987, no class recognises it | <https://::1> | no |
That last row is the important one. An IRI the parser rejects that matches none
of the repair shapes is still counted and still blocks — it is never
silently passed. <https://::1> is an IPv6 literal in the authority without the
brackets RFC 3987 requires; it has a scheme, no excluded character, no bracket,
one # and no %, so the five repair classes are all blind to it. A dump
carrying one was published and Oxigraph refused to load it. Since the parser
decides, a future shape nobody has thought of fails the same way this one now
does: as an unrepairable defect that stops the export, not as a zero.
What is not judged: scheme semantics (<nonsense://x> is well-formed and
accepted), whether the IRI resolves to anything, and normalisation — rete never
case-folds a scheme or removes a dot segment, because that would change the
dictionary key. Non-ASCII is judged now, by the parser: RFC 3987 ucschar
is legal, so <http://ex/café> is valid and is never touched, while the narrow
sub-ranges 3987 excludes are caught because the loader catches them too.
UCHAR escapes are resolved before the parser sees the IRI — <http://ex/café>
denotes http://ex/café, and the backslash is N-Triples syntax, not IRI content.
Skipping that step would flag every escaped dump, and flagging something valid
means percent-encoding an IRI that was fine, which is the one failure mode a
sanitizer must not have.
Three flags act on this, and they are separate on purpose:
rete build --strict/rete validate --strict— refuse the input, naming the IRI and the rule. Not the default: datasets that build today, including published ones, must keep building.rete export --sanitize-iris— percent-encode the repairable classes on the way out. Also not the default, because it changes the data.
An IRI with no scheme is reported and never rewritten. Escaping cannot
repair it: resolving it needs a base IRI the .rete never recorded. This is the
class that cost the most in the field — one such line made a bulk loader drop an
entire ~102,000-line chunk — and the only honest thing a tool can do with it is
say where it is. The same holds for an unrecognised defect: rete will not invent
a repair for something it cannot explain, because <https://::1> percent-encoded
to <https://%3A%3A1> would name a different host entirely. Fix those at the
source.
Automating on the report. --sanitize-iris prints one line meant to be
parsed:
--sanitize-iris: totals invalid=11 repairable=11 unrepairable=0 unclassified=0
A publication gate should key on unrepairable > 0, never on one named
class. unrepairable is the count of occurrences that are still invalid after
everything the sanitizer could do, and it includes the unclassified bucket by
construction — which is precisely why a gate written against it does not need
updating when a new defect shape turns up.
Inspecting
rete info <file>
Print what the header says, as a short summary: the format, size and content
hash, the flags, the front of the file (root, card region, data offset), the
counts (quad_count, term_count, graph_count), the codecs, the pyramid
levels and every section with its range (and, on 0x06, its digest) — plus
the Dataset Card catalog when the file carries one.
--raw also prints the header's fields as Rust Debug output, for a bug
report. Reads only the header, the root and the card region, so a 50 GB file
answers in about a second.
rete stats <file>
Human-friendly overview: file size, statement counts, distinct terms, pyramid levels, and top predicates. Notes the label-index and text-index sizes when the file carries them.
The header counts every statement in the file, so stats prints that as
statements and splits it: default-graph triples, named graphs (the graph
count) and, when there are any, named-graph quads. The split costs no read of
the named graphs: with none, the header says it all; otherwise it comes from
the Dataset Card's build-time counts when the card agrees
with the header, else from counting the default graph's index tiles. The line
says which.
opencitations-2026-06-small-full-labels.rete — 1073354201 bytes
statements : 18869242 (all graphs)
default-graph triples : 5971936
named graphs : 1494934
named-graph quads : 12897306 (split from the dataset card)
Before 0.4.0 the first line read default-graph triples : 18869242, which
on this file is the whole dataset, 1.5 M provenance graphs included.
rete verify <file> [--section <name>]
Check the file's integrity. On a format 0x06 file: the header's bounds, the
root and card digests, every section against its digest in the root, and the
content hash recomputed from them; a failure names the section
(FAILED — content hash mismatch: the digest of index does not match).
--section <name> checks one section: dict, index, quad, labels,
text, pyramid, schema, or a directory (dict-leaves, index-leaves,
quad-leaves, label-leaves, text-leaves). On a 0x05 file: recompute the
blake3 content hash and compare to the header, and check the LABEL_INDEX /
SCHEMA sections against their own digests (they are outside its content hash).
Exits non-zero on corruption/truncation. Sections are streamed in 8 MiB reads,
so verifying a 49 GB file needs megabytes, not 49 GB.
rete search <file> [<prefix>] [--contains <word>…] [--contains-prefix P] [--limit N] [--json]
Two modes.
Label prefix (default) — the subjects whose label starts with prefix
(case-insensitive), printed as label<TAB><iri> (or
{"schemaVersion":1,"matches":[{"label":…,"subject":…}]} with --json).
An empty prefix returns the first --limit labels (default 20).
Answered from the file's LABEL_INDEX section (rete build --label-index) —
no literal scan, every labelled subject, results in label order — so it is the
fast path for autocomplete. Labels come from rdfs:label, skos:prefLabel/altLabel,
foaf:name, dc(terms):title and schema:name, and for people with only
given/family names "family, given" (so svobodová, j finds the author). A file
built before the section existed answers from the bounded block its pyramid-meta
carried instead (the 8,192 most-connected labelled subjects, direct labels only).
A file with neither says so; rebuild with --label-index.
Full-text (--contains <word>…) — the subjects whose literals contain
every given word (whole-word, case-insensitive — AND), printed one IRI per
line (or the same versioned matches envelope with {subject} entries under
--json). --contains-prefix einst additionally
requires a literal word starting with einst. Answered from the opt-in
TEXT_INDEX section (rete build --text-index); on a remote file only the
queried words' posting lists are fetched, not the whole index. A file built
without --text-index reports that it has no text index. Only words of the
literals in the index's scope are
found — by default the default graph's, every predicate. To search a file you
have not downloaded, use rete search-url (below).
rete search data.rete gluc # label prefix (autocomplete)
rete search data.rete --contains glucose # literals containing "glucose"
rete search data.rete --contains glucose phosphate # both words (AND)
rete search data.rete --contains-prefix einst # a word starting with "einst…"
rete card <file> [--json] [--full] [--format jsonld|croissant] [--sha256 <hex>]
Print the embedded Dataset Card — curated metadata
(title/license/source/creators/publisher/…) plus the derived profile (counts, top
predicates and classes, vocabularies), the content-hash checksum, and (when the
file carries one) the build record as a build: block: when it was built,
by which rete (version, commit, target), on what OS, the command line that
reproduces it, its inputs (name, format, size, BLAKE3), the statement counts,
the wall time per phase and the peak RAM, and the starter queries' measured
costs. Costs two small range
reads (header + one coalesced card/build-info range), never the dictionary or
index — instant on any size. A card with an annex (format 0x07: extra or description over
8 KiB, moved to a cold section) shows a card annex line; --full reads the
annex too (one more range) and puts the fields back, in the text view and in
--json. rete card-url --full is the same over HTTP. rete card --json without
--full has an annex marker (fields, bytes) in place of the moved fields. --json emits the card object plus
schemaVersion: 1 and a build block (the whole record). --format jsonld
projects the card to JSON-LD (VoID + schema.org + PROV-O — already RDF when
lifted out; the build is a prov:Activity with prov:startedAtTime,
prov:endedAtTime, prov:used for each input and prov:wasAssociatedWith
the engine, the rest under rete: as JSON literals);
--format croissant emits the honestly-mappable Croissant subset (no
recordSet: a graph is not a table). --sha256 supplies the whole-file
sha256 the Croissant projection's FileObject requires — a file cannot carry
its own, so compute it outside (sha256sum file.rete); with it the document
is validator-clean. For a file built without one it prints
(no dataset card — …), still naming what the header alone decides: whether
the file carries a full-text index (signals.text_index, measured from the
section directory rather than read out of the card — see
Dataset Cards).
rete info shows the same catalog beneath the header when a card is present.
rete graphs <file>
List the named-graph IRIs in a dataset (the default graph is unnamed).
rete export <file> [--format nq|ttl|trig|jsonld|hdt] [--graph G] [--subject S] [--predicate P] [--object O] [--sanitize-iris] [--no-prefixes] [--quoted-triple-syntax rdf12|rdf-star] [--compress none|zstd|gzip] [--compress-level N] [--in-memory]
Serialize the dataset, or a slice of it.
Two formats keep every graph and are lossless round-trips:
nq(the default) — N-Quads, one statement per line.trig— TriG: Turtle syntax plusGRAPH <g> { … }blocks. The same data, in fewer bytes, because a subject and its namespaces are written once rather than once per statement.
Two carry no graph term and therefore serialize a single graph:
ttl— Turtle.jsonld— expanded JSON-LD.
And one is binary:
hdt— HDT, a compact format that stays queryable without being decompressed. See below; it is triples-only and it has a size ceiling the text formats do not.
nq, ttl and trig stream: peak memory follows --memory-budget-mb and the
output is byte-identical at every budget. jsonld builds the whole document in
memory, so it is not the format for a file that does not fit in RAM.
rete export data.rete # N-Quads (default, lossless)
rete export data.rete --format trig # TriG — lossless and smaller
rete export data.rete --format ttl # Turtle, one graph
rete export data.rete --format jsonld # expanded JSON-LD, one graph
Which graph a single-graph format writes. --graph if given, otherwise the
default graph. The full ladder, which ttl and jsonld share and which always
reports its choice on stderr:
| situation | what is exported |
|---|---|
--graph <iri> | that graph; an error naming the real graphs if it does not exist |
--graph '' | the default graph, explicitly |
no --graph, default graph has content | the default graph, noting that named graphs were left out |
no --graph, default graph empty, one named graph | that graph |
no --graph, default graph empty, several named graphs | an error listing them, pointing at --graph and --format trig |
Graphs are never silently merged, and a non-empty selection is never silently dropped.
Quoted triples: which surface (--quoted-triple-syntax). A quoted triple —
a statement standing inside another statement — has two spellings in the wild,
and nq, ttl and trig can write either. Both are supported; rete re-ingests
either losslessly. They differ in who else can read the dump.
| value | surface | read by |
|---|---|---|
rdf12 (default) | <<( s p o )>>, the ratified RDF 1.2 triple term | oxttl 0.2 and everything on it, incl. the oxigraph CLI; Jena 5.x, GraphDB 11 |
rdf-star | <<s p o>>, the RDF-star community-group surface — and the token rete stores | oxrdf 0.2 / oxttl 0.1 (the versions rete itself links), and RDF-star mode in older Jena/GraphDB |
The default is RDF 1.2 because the alternative is a dump current tooling will
not take: oxigraph convert --from-format nq rejects <<s p o>> outright,
and a load is atomic, so one such line costs the whole file. In Turtle and TriG
the failure is quieter and worse — an RDF 1.2 parser reads << s p o >> as a
reifier, so one statement silently becomes two with a blank node where the
triple term was. Pick the surface your consumer speaks; if you do not know, the
default is the one that fails loudly rather than quietly.
A file with no quoted triples in it is unaffected. The header records in one bit whether the file holds any, so the check is skipped entirely and the dump is byte-for-byte the same under either value.
Two edges worth knowing:
- RDF 1.2 puts a triple term in object position only (
ttSubject ::= iri | BlankNode). A graph with a quoted triple in subject position — legal RDF-star, and legal rete — has no RDF 1.2 spelling at all, so the export refuses it by name and points at--quoted-triple-syntax rdf-star, rather than writing a dump no parser will accept. - Reading a dump back takes the matching input surface.
rete buildandrete validateaccept the same--quoted-triple-syntaxflag with the same two values (see above), but their default isrdf-star, notrdf12— so a default--format trigor--format ttldump containing triple terms needsrete build --quoted-triple-syntax rdf12. The export says so on stderr when it writes one.--format nqround-trips with no flag in either surface: that reader is rete's own tokenizer and takes both spellings unconditionally.
--format jsonld and --format hdt have no term kind for a quoted triple at
all and refuse a file that contains one, naming --format trig. They used
to write it as though it were an IRI — a file that loads cleanly and means
something else.
The browser engine's CONSTRUCT writers (format "ttl" / "jsonld", see
browser), the desktop explorer's Turtle, and the file
explorer's N-Triples extract all follow the same rule. RDF 1.2 is the
default, rdf-star is opt-in ("ttl;quotedTripleSyntax=rdf-star", or
extract(…, { quotedTripleSyntax: "rdf-star" })), and a subject-position
quoted triple or JSON-LD is refused rather than mis-written.
HDT (--format hdt). A binary serialization whose point is that a reader
memory-maps it and answers triple patterns against the mapped bytes. Opening the
1.39 GB reference HDT costs 50.7 MB of RSS and 0.31 s, whatever the file's size —
that is what a compressed text dump cannot do.
rete export data.rete --format hdt > data.hdt
Four limitations, all enforced rather than merely documented:
- Triples only. HDT has no graph term, so it takes one graph by the same ladder Turtle uses. The quad extension (HDTQ) is niche and poorly supported and is not implemented.
- It cannot stream, and it has a size ceiling. rete stores IRIs as
<http://…>and HDT stores them bare, which changes the dictionary's sort order, so every term id has to be remapped and the triples re-sorted in the new order. That needs the graph in memory.rete exportestimates the cost from the file header and refuses before doing any work if it exceeds--memory-budget-mb, naming both numbers. Measured, the estimate is about 512 bytes per distinct term plus 16 per triple; an 88M-triple, 39.7M-term graph needs roughly 14 GB. - No quoted triples. HDT predates RDF-star and RDF 1.2 and has no term kind
for a triple term. It did not notice: a quoted triple was interned into HDT's
dictionary as an IRI, brackets stripped by the same rule that strips a real
one's, so a consumer read back an "IRI" with spaces and angle brackets in it.
rete exportnow refuses, from the header flag, before any work. - A second, independent cap: 2^32 object ids. The reference implementation
truncates object dictionary ids to 32 bits while building its query index, so a
larger file would be written successfully and then answer queries incorrectly
on whoever's machine read it. Rather than document that trap,
rete exportrefuses to produce such a file at all.
Which limit binds depends on the graph's shape: a sparse graph with a huge object vocabulary hits the id cap, a dense one hits memory.
HDT is not the format to archive or ship in, and the numbers say so.
Measured on a 1.5M-triple graph: HDT 14,217,647 bytes against trig.zst's
4,296,486 — HDT is 3.3x larger, and larger than the source .rete
(11,810,101) too. On the 88M-triple DBLP graph the ratio is the same: HDT
1,378,298,174 against trig.zst's 462,563,780, 3.0x larger.
So pick by what happens to the file next:
| you want to… | use |
|---|---|
| store or ship it | --format trig --compress zstd — smallest, lossless, no ceiling |
| feed a line-oriented pipeline | --format nq |
| query it in place | --format hdt |
HDT earns its place on the last row only, and that advantage is real and
measured: a reader memory-maps it and answers patterns in ~50 MB of RSS without
decompressing anything, where trig.zst has to be decompressed and parsed in
full before it can answer at all.
Above either limit, use --format trig --compress zstd — lossless, compact, and
with no ceiling.
--compress is rejected with --format hdt: wrapping an in-place-queryable
format in a codec removes the only property it has over a compressed text dump.
The first object-bound query a reader runs will build and cache a .index.v1-1
next to the file. That is normal, and safe for any file rete produces, because
the object-id cap is enforced at export time.
Prefix compression (ttl and trig). IRIs are abbreviated to QNames —
rdfs:label rather than <http://www.w3.org/2000/01/rdf-schema#label> — which
is where most of the size advantage over N-Quads comes from. Well-known
vocabularies keep their conventional prefix names; on top of that the exporter
reads a bounded sample of the data (a hundred thousand statements, whatever the
file's size) and names the namespaces that are frequent in this file, from
their own last path segment. An IRI is abbreviated only when the part after the
namespace is a legal Turtle PN_LOCAL needing no backslash escapes — otherwise
it is written in full, because a smaller file that does not parse is not a
smaller file. --no-prefixes writes every IRI in full.
Compression (--compress). zstd or gzip, applied as the dump is
written. The codec sits in the writer chain, so --memory-budget-mb still
bounds peak memory and a dump far larger than RAM still works — nothing is
buffered up to be compressed afterwards.
rete export data.rete --format trig --compress zstd > dump.trig.zst
rete export data.rete --compress zstd --compress-level 19 > archive.nq.zst
Prefer zstd. Measured on a 609 MB N-Quads dump (see the table below), zstd at its default level produces a file 21% smaller than gzip's default while compressing 2.4x faster and decompressing 1.5x faster — which is the trade that matters for a dump written once and read repeatedly.
| codec | bytes | of plain | compress | decompress |
|---|---|---|---|---|
| plain | 609,574,054 | 100% | — | — |
| zstd -1 | 31,411,730 | 5.15% | 3.19s | 3.36s |
| zstd -3 | 30,417,765 | 4.99% | 3.12s | 3.27s |
| zstd -6 (default) | 27,926,890 | 4.58% | 3.27s | 3.26s |
| zstd -9 | 26,718,710 | 4.38% | 3.24s | 3.41s |
| zstd -19 | 23,760,373 | 3.90% | 195.30s | 3.33s |
| gzip -1 | 42,476,408 | 6.97% | 6.51s | 4.86s |
| gzip -6 | 35,249,561 | 5.78% | 7.92s | 4.86s |
| gzip -9 | 32,656,726 | 5.36% | 13.75s | 5.18s |
--compress-level defaults to 6 for both codecs. For zstd that is
deliberately not the library's own default of 3: the 3 -> 6 step is the largest
remaining gain on the ratio curve (-8.2%), and everything after it returns less
(-4.4% to level 9, -0.9% to level 12) until level 19, which costs 60x the time
for 10% more. Negative zstd levels are accepted and trade ratio for speed.
stdout becomes binary when a codec is selected. Every note and report rete export prints already goes to stderr, so redirecting stdout to a file is safe —
but do not pipe a compressed dump into something expecting text.
Compression applies to the text formats. Decompress with the standard tools; the output is ordinary zstd and gzip, verified against both CLIs.
Export streams from disk by default. The file is opened through the lazy
ranged reader — header, section directories and index up front, then dictionary
chunks and tiles faulted in as the scan reaches them — so the raw file image is
never held resident. The old behaviour read the whole file into memory and
decoded every dictionary chunk and all six index permutations first, so export
RSS scaled ~3 GB per GB of file (6 GB on a 1.53 GB graph; a 52 GB dump would
need ~150 GB and OOM). --in-memory forces that whole-file load back — faster
for small files that comfortably fit in RAM, but it needs RAM ≈ the file size.
The dump is byte-identical either way: the reader choice changes only how
the bytes are fetched, never the order or content.
What --memory-budget-mb N bounds (default 4096; 0 = unlimited). The
budget is split between three evicting caches — decoded dictionary chunks, the
scanned permutation's index tiles (N/16, at most 256 MiB) and raw file blocks
(N/8, at most 256 MiB) — and a reserve of N/32 (at most 128 MiB) for what no
cache owns: one window's term strings and in-flight reads. Before the
dictionary gets its share, the open's fixed directories are taken out of it:
one descriptor per dictionary chunk and per index tile, held for the whole
export (88 MB on the 14.75 GiB openaire-2021-relation, 224 MB on the 1.5 GB
epfl-infoscience, whose pre-#198 directory stores whole boundary terms). Caches
are charged what the allocator really hands out for each body, not its length,
and the CLI uses jemalloc, which returns freed memory to the OS; with the
system allocator a long export used to keep up to 75% more than its budget in
freed-but-resident memory. Set RETE_MEM_TRACE=<seconds> to print RSS, cache
residency and the fixed directories periodically on stderr.
Measured on a full nq export (peak RSS from /usr/bin/time -v; 0.3.4 is the
release before this change, on glibc malloc):
| file | budget | peak RSS | 0.3.4 | export time |
|---|---|---|---|---|
| openaire-2021-relation (14.75 GiB, 1.8 G quads) | 1024 MB | 1422 MB (1.39×) | 19 431 s | |
| 4096 MB | 4352 MB (1.06×) | 7256 MB (1.77×), 7 811 s | 10 345 s | |
| 8192 MB | 8452 MB (1.03×) | 2 067 s | ||
| datacite (48.6 GiB, 9.8 G quads) | 8192 MB | 8248 MB (1.01×) | 16.7 GB (2.04×), 5 h 16 min¹ | 6 986 s |
| epfl-infoscience (1.53 GiB, 9.8 M quads) | 256 MB | 605 MB (floor) | 790 MB, 50 s | 90 s |
| 1024 MB | 1234 MB (1.21×) | 1836 MB, 35 s | 46 s | |
| 4096 MB | 3877 MB (0.95×) | 4279 MB, 23 s | 22 s |
¹ The scholar audit's run (with --sanitize-iris), not measured side by side.
So peak RSS stays within N plus ~0.4 GB, whatever the file size: steady state is N plus 0.05–0.26 GB (jemalloc's not-yet-purged pages and thread caches, the decode threads, the binary), and the worst transient seen was +0.4 GB, once, at 1024 MB on relation. The output is byte-identical at every budget.
There is one floor: the dictionary share never drops below 64 MiB, so a budget smaller than the fixed directories plus ~350 MB is not honoured. On epfl-infoscience that is anything under ~600 MB; there the open itself briefly holds the raw chunk directory beside the parsed one (about twice the directory). Rebuilding a pre-#198 file shrinks its directory to a few bytes per chunk.
A smaller budget costs time, not correctness: the dictionary cache re-decodes chunks it had to evict, so a file whose decoded dictionary is far larger than the budget exports more slowly the smaller the budget is — openaire-relation takes 34 min at 8192 MB, 2.9 h at 4096 and 5.4 h at 1024. A budget also buys less cache than it did in 0.3.4, which spent memory it did not count: at the same 4096 MB this build is ~30% slower on that file, and at the same peak RSS (8192 MB here vs 4096 MB in 0.3.4, both ~7–8 GB) it is 3.8× faster.
--sanitize-iris percent-encodes IRIs that are outside the N-Triples/N-Quads
grammar (see Invalid IRIs), so the dump loads into a strict
store instead of being rejected — and a bulk loader rejects the whole chunk,
not the line, so one bad IRI is worth a hundred thousand good statements.
It is off by default because it changes the data. <http://ex/a[b]> is
written as <http://ex/a%5Bb%5D>, which is a different IRI: the dump no longer
joins against the graph it came from, rete → store → rete stops being the
identity, and if the graph happened to contain both spellings they collapse into
one. Deciding to accept that is the exporter's call, which is why it is a flag.
What it did goes to stderr (stdout is the dump), per class:
--sanitize-iris: totals invalid=5 repairable=5 unrepairable=0 unclassified=0
--sanitize-iris: percent-encoded 5 IRI occurrence(s). The dump's IRIs are NOT the
file's IRIs: it no longer joins against the source graph, and
rete → store → rete is no longer the identity.
2 '[' or ']' outside an IP-literal host
1 more than one '#'
…
An IRI escaping cannot repair — no scheme, or a defect no class recognises — is
counted, named, and written verbatim. The dump is then still not valid
N-Quads and the summary says exactly that rather than implying a fix it did not
make; the totals line's unrepairable field is the machine-readable form of
that statement. The triple-store interop page has the measured
Oxigraph round-trip for all three cases.
Filters prune the file; they are not | grep. --graph /--subject /
--predicate / --object become a triple pattern the engine routes: it picks
the index permutation that sorts on the bound components, binary-searches its
tile directory down to the tiles that can match, and drops the rest by their
recorded synopsis without fetching them. On a lazily-opened file that is the
difference between exporting a slice and reading the graph.
# one predicate of one named graph: 16 MB read instead of 376 MB, 0.4 s
# instead of 12.8 s, 183 MB peak RSS instead of 2.1 GB (cordis.rete, 801 MB)
rete export cordis.rete --format nq \
--graph http://data.europa.eu/s66/graph/results \
--predicate http://data.europa.eu/s66#doi
rete export data.rete --format nq --graph '' # the default graph alone
rete export data.rete --format nq --subject http://ex/a # everything about one subject
rete export data.rete --format nq --object '"text"@en' # an exact literal
Terms are bare IRIs or N-Triples tokens (<iri>, "lit"@en, "lit"^^<dt>,
_:b) — quote literals for your shell. A term the file's dictionary does not
contain matches nothing, which is an empty export, not an error. --graph ''
selects the default graph alone; omitting --graph keeps the current behaviour
(default graph, then every named graph).
Two caveats worth knowing before you measure it:
- A filtered dump prunes the index, not the dictionary. Resolving the rows it keeps still faults the dictionary chunks their terms live in, so on a graph whose payload is long literals the saving is much smaller: on the same file, a predicate whose objects are abstracts went 261 MB → 213 MB, not 23×.
- Rows arrive in the routed permutation's order. Unfiltered (and
subject-bound) that is
(s, p, o)as before; a bound predicate streams(p, o, s), a bound object(o, s, p). The set is identical; N-Quads does not care, butdiffdoes — sort both sides.
Preview any of this before running it with rete cost --dump.
Querying
rete query <file> [--subject S] [--predicate P] [--object O]
Match a single triple pattern; omitted positions are wildcards.
rete query data.rete --predicate '<http://ex/knows>'
rete why <file> [--subject S] [--predicate P] [--object O] [--json]
Explain the provenance of each triple-pattern result. The command reports the
matched terms, dictionary IDs, graph scope, the chosen permutation (one of the
six), and the file byte ranges for the dictionary, full index container, selected
permutation payload, and pyramid metadata. With --json, the same data is
emitted as stable machine-readable JSON with schemaVersion: 1.
rete why data.rete --predicate '<http://ex/knows>'
rete why data.rete --subject '<http://ex/Alice>' --json
rete why-url <url> [--subject S] [--predicate P] [--object O] [--json]
The remote counterpart of rete why: explain a triple-pattern result over a
.rete served on HTTP, range-fetching only the routed tiles — the same
provenance (permutation, section, byte ranges, the physical tile) plus the bytes
fetched and range-request count. The CLI version of the browser's why_url.
rete why-url https://host/data.rete --predicate '<http://ex/knows>'
# … provenance …
# (fetched 4096 bytes in 1 range request(s); file is 1048576 bytes)
Provenance is honest about the physical layout: it identifies the index
container, the selected permutation payload, and — for tiled files —
the physical tile holding each match (PERM/index) with its compressed byte
range.
rete bgp <file> "<pattern> . <pattern> …"
Evaluate a Basic Graph Pattern. Patterns are separated by ., terms by spaces;
?name is a variable (terms may not contain spaces).
rete bgp data.rete "?x <http://ex/knows> ?y . ?y <http://ex/knows> ?z"
rete sparql <file> ("<query>" | --query-file <path>) [--json] [--entail] [--union-default-graph] [--memory-budget-mb N]
Run SPARQL: SELECT / ASK / CONSTRUCT / DESCRIBE. With --json, emit
standard SPARQL Results JSON (for SELECT/ASK). See SPARQL support.
--entail turns on OWL 2 QL reasoning: the query is rewritten so the answer
includes ontology-entailed solutions (rdfs:subClassOf / subPropertyOf /
domain / range / owl:inverseOf / someValuesFrom), computed over the raw
data with no materialization — off by default, so a plain query is unchanged. Same
flag on rete sparql-url reasons over a remote file, fetching only what the
rewritten query touches. See Reasoning by query rewriting.
Warnings on stderr. When a function raises a SPARQL type error, a FILTER
treats it as false and a BIND leaves the variable unbound, so the query can
come back empty with no error. Examples: CONTAINS(?s, "x") on an IRI,
STRSTARTS(?year, "15") on an xsd:integer, and a language-tag mismatch.
rete sparql (and rete sparql-url) keep those results exactly as they are
and print one warning: line per kind of error to stderr. Each line gives
the function, the argument, a sample value and a fix such as "wrap it in
STR()". stdout stays the result. With --json, the result object also gets a
warnings array. An empty result can also get a hint: line saying that
string matching is case-sensitive. See
Type errors and query warnings.
$ rete sparql maps.rete 'SELECT ?s WHERE { ?s rdfs:label ?l FILTER(CONTAINS(?s, "geneva")) }'
0 solution(s)
warning: CONTAINS received an IRI as argument 1 in 1 row: a SPARQL type error, which makes a FILTER drop the row, even under ! (BIND leaves the variable unbound); e.g. <http://ex.org/map/geneva-1572>. Hint: wrap it in STR() to match the IRI's text.
--union-default-graph turns on the opt-in, non-standard union default
graph: a pattern outside GRAPH matches the default graph and every named
graph, a triple stored in several graphs counting once; a FROM in the query
still wins, and GRAPH ?g is unchanged. It is for files whose data lives
entirely in named graphs, where a plain ?s ?p ?o answers nothing. On a file
with a quad index (--named-graphs quad) the union is read in place, so a
selective query reads only the tiles it needs; on any other layout every graph
is merged in memory first, and a materialized-union hint on stderr says so.
The same flag is on rete sparql-url, rete federate (each source) and
rete serve (every query the endpoint answers). See
Union default graph.
rete sparql data.rete "PREFIX e: <http://ex/> SELECT ?p (COUNT(?f) AS ?n) WHERE { ?p e:knows ?f } GROUP BY ?p"
rete sparql data.rete "SELECT ?o WHERE { ?o a <…/Aves> }" --entail # ontology-aware
Long queries: --query-file <path>. The operating system caps a command
line: one argument may be at most 128 KiB on Linux, and the whole line 32,767
characters on Windows. A query with a VALUES block of a few thousand IRIs
crosses either and fails before rete starts (Argument list too long, or a
command line too long). Put the query in a file instead, or pipe it in with
-:
rete sparql data.rete --query-file lookup.rq --json
generate_query | rete sparql data.rete --query-file - --json
Every command that takes a query takes --query-file too: sparql,
sparql-url, federate (in place of --query), cost, progressive,
cypher and manifest query. It is an alternative to the query argument,
never an addition: giving both, or neither, is a usage error (exit 2). The
file must be UTF-8 (a leading byte-order mark is ignored); a file that cannot
be read, or holds only whitespace, is an error (exit 1).
Memory & I/O. The query commands (sparql, query, bgp, why,
cypher, search, stats, graphs, schema, shacl) open a local file
through the same lazy range reader the -url commands use (block size:
RETE_BLOCK_KB), and read it whole only when it is at most 1 MiB. On the
OpenCitations sample (223 MB) reading it whole cost 0.9–1.2 s and 1.2 GB on
every query; lazily the same queries took 32–305 ms and 21–165 MiB, and the
lazy open also won every full scan (81 local jobs: 28.9 s against 4.8 s). The
commands that walk the whole graph (communities, reach, reason, serve)
keep the rule below. For those, reading whole holds the file image, its decoded
dictionary and every index permutation, none of it evictable: 6.8× the file
on the OpenCitations sample. So a file is read whole only when it is at most
a sixteenth of the memory available to the process, never past 1 GiB (and
for sparql, within its 1 MiB, also at most a quarter of
--memory-budget-mb). Available memory
is the smaller of the kernel's MemAvailable and the headroom under the
process's cgroup limit (v2 memory.max or v1 memory.limit_in_bytes, page
cache not counted as used), so inside a container the container's limit is
what counts. Where neither can be read (macOS, Windows) the threshold is
256 MiB. RETE_LOCAL_LAZY_ABOVE_MB overrides the threshold outright (0 =
always lazy); RETE_OPEN_DEBUG=1 prints which way each open went and why.
Measured in a 4 GiB container on the OpenCitations sample: under the fixed
1 GiB threshold of earlier releases, stats, sparql, search and serve
on the 1.07 GB file each read it whole, passed 4 GiB and were killed after
24–49 s. Now the same file opens lazily: a one-row sparql answers in
0.035 s at 15 MB peak, search --contains in 0.11 s at 59 MB, stats in
0.53 s at 421 MB (it reads the pyramid's statistics blocks). Its 0.24 GB
sibling was still read whole by them (3.1 s, 1.6 GB) until the query commands
went lazy above 1 MiB. Aggregation
streams through per-group accumulators. Preview a query's byte cost with
rete cost, and see the exact ranges a query read with rete why.
--memory-budget-mb N (0 = unlimited) bounds what the
lazy reader keeps resident. Omitted, it is sized from the memory the process
can still use — half of it, at least 256 MiB and at most 4096 MiB, read the
same way as the open threshold above (MemAvailable and the cgroup limit) —
so a 4 GiB container gets about 1.9 GiB rather than a default it cannot hold;
4096 where the memory cannot be read. RETE_QUERY_BUDGET_MB overrides that
default (0 = unlimited), and the flag overrides both. It bounds the decoded dictionary chunks, the decoded index
tiles with their group directories, and the raw file blocks, split as for
rete export (see "What --memory-budget-mb N bounds" below). Least recently
used entries are evicted and re-read on demand, so the budget changes memory
and speed, never the answer. Two per-query memos (decoded terms, parsed
numbers) are capped at 64 MiB each on top, and COUNT(DISTINCT …) keeps at
most an eighth of the budget in memory, spilling sorted runs to TMPDIR past
it (exact either way). What the budget does not cover is the rest of the
query's own working state: hash-join build sides, GROUP BY groups,
SELECT DISTINCT sets and an unbounded ORDER BY, which grow with the data
the query asks for (see Memory budget). Measured on the 49 GB
DataCite file (9.53 B triples): GROUP BY ?p over the whole graph went from an
out-of-memory kill at 16 GB to 876 s at 1.1 GB peak RSS, and a CONTAINS count
over every title from 15.1 GB to 4.0 GB (the default budget, filled by the
title dictionary chunks).
rete serve <file> [--bind addr] [--token t] [--journal path] [--union-default-graph]
Serve one .rete — or a manifest of segments (.json), whose
visible fold becomes the served state — as a live SPARQL 1.1 Protocol
endpoint — queries and SPARQL Update. The base file is never mutated:
updates append to a plain-text journal next to it (<file>.changes, one
+/--prefixed N-Quads line per change) and the merged state answers the
very next query; a restart replays the journal. GET /snapshot.rete downloads the current state as a
fresh .rete — the update cycle's publishable artifact (upload the snapshot,
delete the journal). Any rete client — including the browser playground — can
federate against the endpoint with SERVICE <http://host:port/sparql>.
GET/POST /sparqlwithquery=→ results (application/sparql-results+json; CONSTRUCT as N-Triples). Queries may themselves containSERVICEblocks.POST /sparql(or/update) withupdate=or anapplication/sparql-updatebody →INSERT DATA/DELETE DATA/DELETE/INSERT … WHERE/CLEAR/DROP(LOADandUSINGare rejected).- A query whose expressions raised SPARQL type errors (a FILTER drops those
rows silently; see query warnings) gets two response
headers:
Rete-Warnings, the warnings as a JSON array (ASCII-escaped), andRete-Warning-Count. Headers, not a body member: the result document stays standard SPARQL JSON or N-Triples, which any client parses unchanged. A query with nothing to report gets neither header. - Binds loopback by default. Expose deliberately (
--bind 0.0.0.0:7878) and set--token: updates then requireAuthorization: Bearer <t>(reads stay open). CORS is enabled, so browsers can query it directly. --union-default-graphanswers every query against the union default graph, the way Oxigraph's server can be asked to: a pattern outsideGRAPHmatches the default graph and every named graph, each triple once. A server-wide switch, off by default; a query's ownFROMwins.- Scale: until the first update the endpoint answers straight from the
file, opened like any local command opens it (read whole only when it is a
small fraction of the free memory, lazily otherwise; see
rete sparql's Memory & I/O), andGET /snapshot.retestreams the file itself. So serving a multi-GB catalog file read-only costs what its queries touch. Before 0.4.0 the server loaded every quad at startup: in a 4 GiB container it was killed before answering on both the 1.07 GB and the 0.24 GB OpenCitations files (18.9 M quads each); now it answers its first query in 0.06 s at 16 MB, and in 3.0 s at 1.6 GB on the smaller file, which it reads whole. The first update (or a journal with changes to replay at startup) loads every quad into memory and rebuilds after each write, which is built for living small/medium datasets (annotation, curation; up to a few million triples). An update whose load would not fit is refused with503and a message, and the endpoint keeps answering queries. The estimate is 1 KiB per quad against the available memory (measured: 646 MiB at peak for 1 M OpenCitations quads), so a graph of very long literals can still need more than it predicts.
rete serve notes.rete
curl -s --data-urlencode 'query=SELECT * WHERE { ?s ?p ?o } LIMIT 5' http://127.0.0.1:7878/sparql
curl -s --data-urlencode 'update=INSERT DATA { <http://ex/n1> <http://ex/note> "hello" }' http://127.0.0.1:7878/sparql
curl -sO http://127.0.0.1:7878/snapshot.rete # the updated companion .rete
rete cost <file-or-url> ("<query>" | --query-file <path>) [--json] [--explain]
Preview the byte/range-request cost of a SPARQL query without evaluating it. The report parses the query, lists the concrete predicates that can drive summary-based routing, and compares three access paths:
- summary overview — header + dictionary + pyramid summary, skipping the triple index.
- routed pattern open — for a single default-graph triple pattern, header + dictionary + the one selected permutation payload (the best of the six).
- full query open — the eager open, which reads the dictionary + index (+ pyramid, text-index and named-graph sections when present) before evaluation. Its figure is computed from the header, not by performing the open: previewing a large file no longer reads it whole (a 26 GB file used to be killed at 2 GB).
rete cost data.rete "PREFIX e: <http://ex/> SELECT ?y WHERE { e:Alice e:knows ?y }"
rete cost https://host/data.rete "ASK { ?s <http://ex/knows> ?o }" --json
rete cost <file-or-url> --dump [--graph G] [--subject S] [--predicate P] [--object O] [--json]
The same preview for a dump — what rete export (or a client's streaming
dump) will fetch for that filter, before it starts. Same report shape as above.
The index figure is computed, not sampled: the tile directories say which tiles the filter's routed scan can touch and how big each is, so no tile is fetched to produce it.
$ rete cost cordis.rete --dump \
--graph http://data.europa.eu/s66/graph/results \
--predicate http://data.europa.eu/s66#doi
file bytes: 801016143
lazy dump open: 43156179 bytes in 66 range request(s) · reads index
graphs selected: 1
<…/graph/results>: POS · 31 of 528 tile(s) admitted · 1217273 bytes (section 21227100)
index tiles: 31 of 528 admitted · 1217273 bytes of 21227100 · computed from the
tile directories, no tile fetched
dictionary ceiling: 417246556 bytes
estimated dump cost: 44373452 – 461620008 bytes
Read the range honestly: the floor (open + admitted tiles) is exact; the ceiling adds the whole dictionary, which only a dump touching every term pays. Term resolution faults chunks the tile directories cannot name, so the true cost sits between — near the floor when the slice is small and its terms are short, near the ceiling on a literal-heavy file however well the index prunes. The dump previewed above actually read 59,672,911 bytes.
For the exact summary-only shapes SELECT (COUNT(*) AS ?n) WHERE { ?s <p> ?o },
SELECT (COUNT(*) AS ?n) WHERE { ?s ?p ?o },
SELECT ?p (COUNT(*) AS ?n) WHERE { ?s ?p ?o } GROUP BY ?p,
SELECT DISTINCT ?p WHERE { ?s ?p ?o },
SELECT (COUNT(DISTINCT ?p) AS ?n) WHERE { ?s ?p ?o }, ASK { ?s <p> ?o },
and ASK { ?s ?p ?o }, the JSON output includes summary_answer with the
exact count/boolean value read from the pyramid summary. Predicate-specific
shapes also include the predicate; predicate totals return all predicate/count
pairs, predicate lists return all predicates present in the summary, and
predicate distinct counts return the number of predicates. More complex shapes
are marked requires_index.
Add --explain to include a planner explanation. In JSON, this adds an
explain object with the classified query_shape, whether the answer is
summary_exact, the planned access path (summary-only, routed-pattern, or
full-index), and whether the current engine path still reads the index.
For HTTP(S), the host must honor Range requests, just like query-url and
sparql-url. Treat this as a deployment/debugging preview: it reports the
current SPARQL engine's range budget, the summary budget, and the exact routed
single-pattern budget when the query shape allows it.
rete progressive <file-or-url> ("<query>" | --query-file <path>) [--json]
Run the first summary-only progressive query path. This command answers only the exact shapes that can be proven from the pyramid summary without opening the triple index:
SELECT (COUNT(*) AS ?n) WHERE { ?s <p> ?o }SELECT (COUNT(*) AS ?n) WHERE { ?s ?p ?o }SELECT ?p (COUNT(*) AS ?n) WHERE { ?s ?p ?o } GROUP BY ?pSELECT DISTINCT ?p WHERE { ?s ?p ?o }SELECT (COUNT(DISTINCT ?p) AS ?n) WHERE { ?s ?p ?o }ASK { ?s ?p ?o }ASK { ?s <p> ?o }
rete progressive data.rete "PREFIX e: <http://ex/> SELECT (COUNT(*) AS ?n) WHERE { ?s e:knows ?o }" --json
The JSON output is SPARQL Results JSON with schemaVersion: 1 and an added progressive object
describing the summary stage, exactness, predicate when one is fixed, bytes
fetched, request count, and reads_index: false. Other query shapes fail
clearly; use rete sparql for full-index evaluation or rete cost --explain to
inspect why a query is not summary-answerable yet.
rete cypher <file> ("<query>" | --query-file <path>) [--base <iri>] [--json]
Run a read-only Cypher subset (a prototype). The query is translated to an
equivalent SPARQL SELECT and evaluated by the same engine — no second query
engine. Supports MATCH … [WHERE …] RETURN … [LIMIT n]: node patterns
((a), (a:Label)), forward/reverse relationships ((a)-[:REL]->(b),
(a)<-[:REL]-(b)), variable-length relationships (-[:REL*]-> → the SPARQL
property path REL+, one-or-more), simple WHERE comparisons on a property
(a.age > 30) or identity (a = <iri>) joined by AND/OR, and RETURN of
variables and/or properties. Writes, OPTIONAL MATCH, WITH, aggregations,
relationship variables/properties, and multiple labels are rejected with a clear
error. See compatibility for the full subset and the
name→IRI convention. With --json, emit standard SPARQL Results JSON.
A bare label/relationship/property name X maps to <BASE + X>, where BASE
defaults to http://ex/ and is overridable with --base.
rete cypher deps.rete "MATCH (a:Application) RETURN a"
rete cypher deps.rete "MATCH (a)-[:dependsOn*]->(b) WHERE b = <http://ex/log4x> RETURN a"
Reasoning
rete reason [<file>] [--url <url>] [--materialize] [--check] [--verify-card] [--format nq|ttl]
Run the prototype OWL RL / RDFS reasoner: materialize RDFS/OWL entailments to
a fixpoint and report any logical inconsistencies ("incoherent points", e.g. a
disjoint-class violation). Prints the count of newly entailed triples and each
inconsistency (kind + detail). Exits non-zero if any inconsistency is found
(zero if coherent), so it works as a coherence gate in CI. With --materialize,
also serialize the base + inferred graph (nq default, or ttl). This is a
documented subset, not full OWL DL — see Reasoning & coherence.
--url <url>reads a remote.reteover HTTP range requests instead of a local file (omit the positional<file>).--checkis coherence-gate mode: print one verdict line and exit non-zero on any incoherent point (suppresses--materializeoutput) — the minimal CI gate.--verify-cardchecks the file's baked coherence card (fromrete build --reason) against a fresh reasoning run, guarding against drift or a stale ruleset; it exits non-zero if the stored verdict disagrees with recomputation.
rete reason data.rete
rete reason data.rete --materialize --format ttl
rete reason data.rete --check # one-line CI verdict
rete reason --url https://host/data.rete # reason over a remote file
rete reason data.rete --verify-card # baked card vs fresh run
Shape validation
rete shacl <file> --shapes <shapes.ttl> [--graph <iri>] [--format text|json|ttl]
Validate a .rete graph against SHACL Core shapes read from Turtle. The default
graph is validated unless --graph names one dataset graph. The command exits
zero when the report conforms and non-zero when it finds validation results, so
it can be used as a CI data-quality gate. See SHACL validation for
the supported components and current limits.
rete shacl data.rete --shapes shapes.ttl
rete shacl data.rete --shapes shapes.ttl --format json
rete shacl data.rete --shapes shapes.ttl --graph '<http://ex/snapshot>'
rete shacl-url <url> --shapes <shapes.ttl> [--format text|json|ttl]
Validate a remote .rete over HTTP, range-reading only what the shapes
target. The file is opened lazily and each focus node's values are fetched as
routed range reads, so a targeted shape (sh:targetClass / targetNode /
targetSubjectsOf / targetObjectsOf) never downloads the whole graph — it
faults only the tiles holding the target nodes and their property values. Reports
the bytes fetched and the range-request count. Validates the default graph.
rete shacl-url https://host/data.rete --shapes shapes.ttl
# (fetched 38912 bytes in 7 range request(s); file is 1048576 bytes)
Coarse graphs (no index read)
rete summary <file> [--level k]
Print the structural coarse graph: the Louvain community quotient graph
(community → community relations with counts), plus — for v2 files — the schema
pyramid: a leveled rdf:type histogram where abstract classes describe coarse
levels and leaf classes resolve as you zoom in (e.g. Agent → Person → Scientist → Astronomer). --level k prints just level k's type histogram (0 =
coarsest / most abstract). Everything here reads index-free from the
pyramid-meta — summary-url shows the same over HTTP without fetching the index.
rete schema <file>
Print the semantic coarse graph: classes (by rdf:type) with instance
counts, and the class→predicate→class relations between them — the dataset's
effective schema.
rete communities <file> [--json] [--profile] [--predicate IRI] [--round N] [--min-size N]
Recompute the Louvain communities and expose, per community, its member
subjects and the literal text of its triples — the per-community text
corpus for downstream topic modeling. --profile adds a no-ML "topic" profile
per community (top literal words, rdf:type classes, and predicates).
--predicate <iri> detects communities using only that relation's edges — a
criterion-specific partition (see multi-criteria splitting).
--json emits {schemaVersion:1, communities:[{community, size, members:[<iri>…], text:[lexical…]}]} (plus a profile object on each record
when --profile is set); --round N cuts the dendrogram at a
specific round (default: the round chosen for the tile budget); --min-size N
drops communities with fewer than N members. See the
topic modeling tutorial.
rete communities papers.rete --profile # no-ML topic labels
rete communities researchers.rete --predicate '<http://ex/cites>' # one criterion
rete communities papers.rete --json | python3 scripts/lda_topics.py --topics 3
rete predicates <file>
Exact per-predicate triple totals, summed from the pyramid summary alone — the triple index is never read.
Graph traversal
rete reach <file> --predicate <iri> --seed <iri>… [--seeds-file F] [--reverse] [--parallel] [--count]
Multi-source transitive reachability over one relation: for each seed, the set of
nodes it transitively reaches. --reverse answers "who reaches the seed?"
(impact analysis); --seeds-file reads one seed IRI per line; --count prints
only sizes; --parallel runs one rayon task per
seed (the batch-reachability workload that benchmarks ~14–15× on many cores).
# Who (transitively) depends on the vulnerable package? (impact analysis)
rete reach deps.rete --predicate '<http://ex/dependsOn>' --seed '<http://ex/log4x>' --reverse
# Many seeds at once, in parallel:
rete reach g.rete --predicate '<http://ex/knows>' --seeds-file seeds.txt --parallel --count
Over HTTP
Both URL commands work against http:// and https:// hosts that honor Range
requests (S3, GitHub, any CDN).
How the ranges are fetched. On a remote file the time goes to round trips,
not bytes, so the HTTP reader behind every -url command (and cost,
progressive, card-audit, reason --url, shacl-url and federate on a
URL):
- keeps its connections alive: one client with a connection pool, where each
range used to open (and on
https://TLS-handshake) a connection of its own; - fetches the engine's batches in parallel, up to 16 requests in flight — the tiles a scan or a join step is about to read, the dictionary chunks a result decodes (all four dictionary sections in one batch), the subjects a full-text search returns — so a batch costs one round trip instead of one per range; before 0.4.0 the CLI passed every batch on one range at a time;
- walks the dictionary and the index containers at the same time while opening a file: each is a chain of small dependent reads (a container's sections are found through their length prefixes), and the header says where both start;
- merges ranges of one batch that lie at most 32 KiB apart into one request (never past 8 MiB), and slices them apart again;
- reads the tiles of a probe join window (scattered by construction) as exact
ranges: the block cache does not round each one out to its 64 or 128 KiB
block, two of them are merged only when at most 32 KiB apart
(
RETE_HTTP_SPARSE_GAP_KB), and up to twiceRETE_HTTP_CONCURRENCY(at most 64) requests are in flight, since they are many and small; - opens with one
bytes=0-131071GET, which is also the length probe (the total in itsContent-Range): noHEAD. A file of at most 8 MiB (RETE_HTTP_WHOLE_MB,0= never) is then fetched whole in one more request, and no query on it reads the network again: 2 round trips per query on a 7 MB catalog file, where a ranged reader took 11–33; - holds every
206to itsContent-Range: the start, the end and the total must be the ones asked for, so a file replaced under the same URL is refused by name instead of being read as two versions spliced together, and a body longer or shorter than the range is an error, never cut to fit; - retries a transient failure (a dropped connection, a short body,
408,429, a5xx) twice with backoff, and fails at once on a permanent one: aContent-Rangethat does not match, a404. One range failing fails the batch, and a query never returns a partial answer as if it were whole; - copes with a host that ignores
Range(a plain static server, a proxy that strips the header:200and the whole body for a range request). Since 0.4.2 that body is kept when it is the file (the length theHEADreported) and at most 256 MiB, the browser readers' cap, and every later read is a slice of it, with no further request; a larger body is refused by name (the host ignored the Range header … not kept in memory) without being read. Before 0.4.2 every such host was refused. A host that omitsContent-LengthonHEAD, or refusesHEAD(a GET-only presigned URL), is measured with abytes=0-0GET instead.
The Python and R clients read remote files through this same reader (it lives
in rete-core, behind its http feature), so all of the above, and the knobs
below, apply to rete_graph.open(url) and rete_open(url) too.
Knobs: RETE_HTTP_CONCURRENCY (requests in flight, 1–64, default 16; 1
fetches one range at a time, the behaviour before 0.4.0),
RETE_HTTP_COALESCE_KB (default 32; 0 merges only touching ranges),
RETE_HTTP_RETRIES (default 2) and RETE_HTTP_WHOLE_MB (default 8; 0 never
fetches a file whole). The fetched … in N range request(s) line
counts the ranges handed to the HTTP reader (after the block cache has merged
adjacent blocks); when the reader merges two nearby ones into one request the
server sees one fewer, and retries are not counted.
Measured through a latency-injecting proxy on the OpenCitations sample: see the CHANGELOG entry for 0.4.0 for the per-query numbers at 20 and 80 ms RTT.
rete card-url <url> [--json] [--full] [--format jsonld|croissant] [--sha256 <hex>]
Fetch only the embedded Dataset Card. On a format 0x06
file that is one first read of the first 128 KiB, which holds the header, the
root, the card region (card and build record) and the statistics
(SPEC §4.6), and at most one follow-up, sized
from the header, when the card (and the build record, which every format but Croissant prints) ends past that; it stops there and does not read the slack that follows them. A small remote file is not
fetched whole, as the other URL commands do. A 0x05 file has its card after
the 1 KiB header, so it takes the first read and one range for the card and
the build record. The dictionary, index, and pyramid are never fetched: this is the index-free CARD
tier, the cold-start self-description (counts, vocabulary, class graph,
signals, starter queries, build record) a client reads before it knows what to
query. Reports the bytes fetched and the range requests that went over the wire.
The --format projections (and
--sha256 for Croissant) match rete card.
rete card-url https://host/data.rete --json
rete card-audit <path|url> [--json] [--measure] [--only IDS] [--max-mb N] [--write-costs] [--allow-empty]
Do the starter queries a card already ships still answer on the file that
carries them? A published .rete cannot be re-carded for free, so this decides
each query's fate from the card's own profile — the CARD tier again, so a
multi-GB file costs tens of KB to check.
Each query gets one verdict, and the two that matter are kept apart: empty is
the card refuting a shipped query (a property path through a predicate the
file does not have, a VALUES list disjoint from the dataset's link predicates,
a class joined to a label predicate the class-link quotient accounts for and
never pairs with it), while undecidable and suspect are the honest middle —
run the query to settle those. answers is the card proving a row comes back,
and revision says what a re-card would do to the body (current,
superseded, dropped).
The judgement is the query generator's own (Cap::joint_with, NonEmpty,
provably_empty), not a second copy of it. Input is a .rete (local path or
URL) or a card JSON document from rete card --json / rete card-url --json.
The curated example_queries are audited too, and reported apart — under
example_findings in the JSON and in a section of their own in the text, with
the ids example-1, example-2, … and revision: "curated". A curated query
makes no claim, so the card decides only what it can show: invalid (it does
not parse), empty (it reads only the default graph of a file whose default
graph the card records as empty), suspect (it names a predicate the card's
complete predicate list does not contain — not empty, since the predicate may
sit in an OPTIONAL or UNION branch), and otherwise undecidable.
--measure runs them like the starters (and --only example-2 picks one);
their costs are reported beside, never stored by --write-costs, whose cost
list is the starters'.
rete card-audit https://host/data.rete
rete card-audit card.json --json | jq '.findings[] | select(.verdict=="empty")'
rete card-audit data.rete --json | jq '.example_findings[] | select(.verdict!="undecidable")'
--measure: run them instead of reasoning about them
The static pass has a hard ceiling. Some templates are undecidable by
construction — nothing in a card ties a subject to a predicate, so top-reach
cannot be settled from one, and a card does not record which objects are also
subjects, so top-dangling cannot either. No better card-only reasoning closes
that; a run does, and records what the answer cost.
--measure runs each starter query cold — a fresh lazy open, logical range
reads, no block cache — and reports rows, bytes, requests and a reference
timing. It is the same measure_query a rete build uses to fill in its
query_costs, so when the file already carries a build record the command
checks itself against it and says whether the two agree (= build record /
!= build record). On switzerland-fedlex.rete — the one published file that
has a record — all ten queries reproduce it exactly.
Each row shows the card's verdict and the run's outcome side by side; they are never merged, because one is what a card can prove and the other is what the file did.
card says run says query rows bytes req ms
answers answers ov-triples 1 row(s) 2800822 B 74 req 294 ms
suspect empty lb-labels 0 row(s) 228126 B 68 req 12 ms
undecidable answers top-dangling 100 row(s) 4468038 B 118 req 592 ms
Local or remote. Point it at a path or at an http(s):// URL. bytes and
requests are the same quantity either way — no block cache is in the stack, so
the range sequence is a function of layout and query, not of transport — but
only the remote run actually pays for them. The output always names which it
was, in the text header and in the JSON measurement.transport, because a cost
figure without its transport is not a cost figure.
It is a download. --only ov-triples,lb-labels measures a subset;
--max-mb 8 abandons a query once it has asked for more than that, and reports
the abandonment with the bytes it spent — "costs more than 8 MB" is itself an
answer. Both matter: eight of switzerland-fedlex's ten starter queries read
~1.02 GB each, so measuring that card remotely without a leash is an 8 GB
download.
--write-costs: make the measurement durable
Records the run in the file's build-info section, so the next reader gets the
figures from the CARD tier (one first read, see card-url) instead of re-measuring. Local
files only.
The record sits outside the blake3 content hash — that is why two builds of
identical data hash equal — so the file keeps its identity: same checksum, same
rete verify, byte-identical N-Quads. On a format 0x06 file the record goes
into the card region's free space and only the 128-byte header changes; when it
does not fit, the file is copied once. On a 0x05 file it sits near the
front, right after the card, so making room for it shifts everything behind
it and the file is rewritten end to end (streamed through a 4 MiB buffer,
committed by rename: one full pass of I/O and, for a published file, a full
re-upload).
Where that is worth it: when the queries answer and only the record is
missing. When they do not, re-card instead — rete recard
is the same single rewrite with a new card in it, at any size.
The RAM goes into the measurement, not the rewrite: the engine evaluates
eagerly, so a starter query with a big result materializes it. A card written
by an older build can carry unbounded starters — switzerland-fedlex (1.04 GB,
56.3 M quads) peaked at 14.2 GiB for --measure --write-costs, 3.2 GiB of
it ng-list's 497,905 rows — so budget for those queries (or --max-mb them),
not for the file. A card generated since 0.4.0 carries only bounded starters
(Dataset Cards).
So: if the audit says the queries are stale or broken, re-card — same rewrite,
more value. If the queries answer and the only gap is the missing record, write
the costs. The command enforces that split: it refuses when a query measured
zero rows (--allow-empty overrides, for top-dangling on a fully-described
graph), when a run did not finish, and when --only measured a subset that
would be stored as if it were the whole card.
rete card-audit data.rete --measure --json | jq '.findings[] | select(.observed.outcome=="empty")'
rete card-audit https://host/data.rete --measure --only ov-triples --max-mb 8
rete card-audit data.rete --measure --write-costs
rete search-url <url> [<prefix>] [--contains <word>…] [--contains-prefix P] [--limit N] [--json]
rete search over HTTP — the same two modes and the same output, without
downloading the file.
The open is deliberately narrower than any other remote command's. It reads the
header and the subject halves of the dictionary (the shared and subject-only
sections), and stops: no permutation tile directories, no pyramid, no index.
--contains then faults the TEXT_INDEX token table once and one range request
per posting list. The bare prefix mode reads the LABEL_INDEX section — its head
(preamble and root, one range of at most 64 KiB), one directory node per level and
the data blocks it needs — and on such a file the dictionary is not opened at
all, because the section stores each subject's term. On an older file it faults
the pyramid instead, where the legacy label block lives.
That narrowness is the point, because on a literal-heavy graph the dictionary is
most of the file, and on a file built before 2026-08 its object-only chunk
directory — which carried every chunk's first term verbatim — is most of what a
normal open costs. (Newer builds key that directory by the shortest separator
instead, a few bytes per chunk: on epfl-infoscience that is 234 MB of routing
keys against ~600 KB. It is a write-side change, so a published file keeps its
large directory until it is rebuilt — the figures below are the published one.)
Searching epfl-infoscience.rete (1.64 GB; a 1.35 GB dictionary, a 186 MB text
index over titles and 132k abstracts) for photosynthesis:
| bytes fetched | requests | time | |
|---|---|---|---|
sparql-url with FILTER(CONTAINS(…)) | 334 MB | 83 | 28.6 s |
search-url --contains | 29.5 MB | 5 | 4.8 s |
Same five hits. The 29.5 MB is the token table itself; label-prefix mode over the
same file costs 1.4 MB in 4 requests. A file built without --text-index says so
instead of silently scanning.
rete search-url https://host/data.rete --contains glucose phosphate
rete search-url https://host/data.rete --contains graphs --contains-prefix existen --json
rete search-url https://host/data.rete "Photosynth" --limit 3 # label prefix
rete summary-url <url>
Fetch only the header + dictionary + pyramid summary and print the coarse graph. The (large) index is never fetched — the "overview first" path.
rete query-url <url> [--subject S] [--predicate P] [--object O]
Match a triple pattern over HTTP, fetching only the byte ranges the query needs and reporting how many ranges/bytes were pulled. Bound terms are resolved from the dictionary first; if they exist, the reader fetches only the selected permutation payload rather than the whole index container. If a bound term is unknown, the index is skipped entirely and the result is empty.
rete query-url https://host/data.rete --object '<http://ex/Dave>'
rete sparql-url <url> ("<query>" | --query-file <path>) [--json] [--entail] [--union-default-graph] [--memory-budget-mb N]
Run a full SPARQL query over HTTP with lazy tile faulting (tiled
files): the open fetches the header, dictionary, pyramid, and the index's
small tile directories; index tiles are then range-fetched only when the
query's scans and probes touch them, so a selective query reads O(touched
tiles) rather than the whole index. A range failure mid-query is reported as
an error, never as silently fewer rows. --memory-budget-mb bounds the
block, dictionary and tile caches exactly as for rete sparql.
--union-default-graph is as for rete sparql; on a quad-index file it fetches
only the tiles the query routes to.
rete sparql-url https://host/data.rete \
"PREFIX e: <http://ex/> SELECT ?d WHERE { ?d e:dependsOn+ e:log4x }"
rete sparql-url https://host/data.rete --query-file lookup.rq # a long query
Federation
rete federate <sources…> (--query "<SPARQL>" | --query-file <path>) [--json] [--no-route] [--disjoint] [--union-default-graph]
Run one SPARQL query across several .rete sources — local file paths and/or
http(s):// URLs, mixed — and merge the results at the term (string) level:
SELECT rows are unioned + deduped, ASK is OR'd across sources, CONSTRUCT
triples are unioned + deduped. Routing (default on) reads each source's
predicate set from its summary (index never touched) and prunes sources whose
predicates are disjoint from the query's; --no-route queries every source.
Per-source diagnostics and the queried/pruned tally go to stderr.
Each source opens the way a single-source query opens it: a URL lazily, as
sparql-url does (dictionary chunks and index tiles fetched as the query
touches them, under the default memory budget, with the bytes and range
requests reported per source), and a local path as rete sparql does. A range
request that fails while a source is read fails the federation instead of
silently dropping that source's rows. URL sources used to be read eagerly
(the whole dictionary and index directory first, unbounded): three 55–63 GB
published files in one federated query ran a 4 GiB container out of memory.
--union-default-graph evaluates the query against each source's union default
graph, as rete sparql --union-default-graph
does for one file.
This is union federation — correct for sharded data where each file
independently yields complete rows. It does not do cross-file joins. With
several sources queried, ORDER BY, OFFSET and LIMIT apply to the merged rows
(LIMIT is also sent to each source as an upper bound), and MIN, MAX and, with
--disjoint (no triple is stored in two sources), COUNT and SUM merge across
sources, per GROUP BY key; AVG, GROUP_CONCAT, HAVING and the like are refused
with the reason instead of being answered per source. See
Federated queries for the full model, limitations, and a real
OpenCitations multi-shard example.
rete federate data/opencitations/cites-2021.rete data/opencitations/cites-2024.rete \
--query 'SELECT ?citing WHERE { ?citing <http://purl.org/spar/cito/cites>
<https://doi.org/10.1038/s41586-021-03819-2> } LIMIT 5'
rete federate shard-*.rete --query-file lookup.rq --json # the query from a file
rete manifest <init|add|status|query|seal|compact>
Manage a writable logical graph as an ordered log of immutable .rete
segments plus tombstone (deletion) segments, described by a small JSON
manifest — grow a dataset across sessions, delete/update quads, query it all
as one graph (joins across segments resolve, unlike federate's UNION),
and fold it back into a single .rete. rete serve accepts a manifest, and
manifest seal checkpoints the server's journal into fresh segments. See
Writable graphs — manifest & WAL.
rete manifest init g.rete-manifest.json base.rete
rete manifest add g.rete-manifest.json --adds delta.rete [--dels tomb.rete]
rete manifest query g.rete-manifest.json "SELECT … WHERE { … }" # or --query-file q.rq
rete manifest status g.rete-manifest.json --count
rete manifest seal g.rete-manifest.json # journal → segments (stop serve first)
rete manifest compact g.rete-manifest.json # the whole log → ONE fresh .rete
Exit codes
0— success.1— runtime, data, or network failure: malformed RDF, a missing/corrupt or unsupported.rete, HTTP failure, ignoredRange, or unsupported query input.2— command-line usage error reported by Clap.3— the command completed its requested check, but the graph failed it: SHACL non-conformance or reasoning incoherence.