Compiled dataset
The compiled matrix dataset is the runtime contract of WS4 (see notes/adr/0002 and
notes/adr/0006). Ingestion writes it; the simulator, the optimizer, and the validator read
it. Nothing downstream reads the raw recipe or inventory.
Layout
A dataset is a directory with a manifest.json and one Parquet file per table.
Manifest
manifest.json validates against ws4.model.types.CanonicalDatasetManifest:
schema_versionThe schema version. The current value is
0.1.tablesA map from logical table name to file name.
sourcesThe source identities, each with a path and a hash.
optionsThe compile options.
metadataFree-form metadata.
Tables
Every table below is required, and its listed columns are required.
Table |
Columns |
Meaning |
|---|---|---|
|
|
The decision units. |
|
|
The runtime aggregate view. |
|
|
Succession pathways. |
|
|
Development strata over offset age. |
|
|
Available operations. |
|
|
Growth and yield curves. |
|
|
Per-track feature curves. |
|
|
Product definitions. |
|
|
Account definitions. |
|
|
Block grouping. |
|
|
Compile diagnostics. |
Invariants
The validator also checks:
area_hais non-negative infragmentsandblocks.ageis non-negative infragments.Every
fragments.block_idexists inblocks.Every
blocks.track_idexists intracks.Each curve’s
xvalues are strictly increasing within acurve_id.
Validation
Run:
ws4 validate --dataset build/compiled
The command prints a table of findings and exits non-zero when the dataset is invalid. Each finding has a stable code:
DATASET-MISSINGThe dataset directory does not exist.
DATASET-MISSING-MANIFESTThe manifest file is missing.
DATASET-BAD-MANIFESTThe manifest does not parse.
DATASET-SCHEMA-VERSIONThe manifest schema version differs from the supported version.
DATASET-UNLISTED-TABLEA required table is not listed in the manifest.
DATASET-MISSING-TABLEA listed table file is missing.
DATASET-MISSING-COLUMNA required column is missing.
DATASET-COLUMN-KINDA column has the wrong kind.
DATASET-NEGATIVE-AREAA negative area.
DATASET-NEGATIVE-AGEA negative age.
DATASET-UNKNOWN-BLOCKA fragment references an unknown block.
DATASET-UNKNOWN-TRACKA block references an unknown track.
DATASET-CURVE-DOMAINA curve’s domain is not strictly increasing.
API
- class ws4.model.types.CanonicalDatasetManifest(*, schema_version: str = '0.1', tables: dict[str, str]=<factory>, sources: list[dict[str, str]]=<factory>, options: dict[str, object]=<factory>, metadata: dict[str, object]=<factory>)[source]
Manifest for a compiled dataset: schema version, tables, and sources.
- schema_version
The dataset schema version.
- Type:
str
- tables
The table name to file name mapping.
- Type:
dict of str to str
- sources
The provenance records of the inputs the dataset was built from.
- Type:
list of dict
- options
The build options that affect the tables.
- Type:
dict
- metadata
Further free-form metadata.
- Type:
dict
- model_config = {}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].