Compiled corridor dataset

The compiled dataset is the boundary between ingestion and execution. The simulator and the planner read the dataset and nothing else from the raw sources. A dataset directory contains a manifest.json and one file per table.

Schema version: brushline.dataset.v1.

Directory layout

dataset/
  manifest.json
  segments.parquet

The first release requires the segments table. P1.2 and P1.3 add the corridor, state, treatment, observation, access, and structure tables. P1.4 adds the plan and run contracts, which live in the run directory rather than the compiled dataset.

Manifest fields

Field

Meaning

schema_version

brushline.dataset.v1.

dataset_id

Stable dataset identifier.

name, description

Human-readable identity.

created_utc

Timezone-aware creation time.

crs

Coordinate reference system, for example EPSG:3005.

units

Named units, such as {"length": "m", "area": "ha"}.

tables

One TableSpec per table.

provenance

Source, version, licence, creator, configuration, and model cards.

content_digest

SHA-256 of the canonical content fields.

Table fields

Field

Meaning

name

Logical table name. segments is required.

path

Dataset-relative file path. Absolute paths and .. are rejected.

format

parquet or csv.

primary_key

One or more columns that must be present, non-null, and unique.

columns

Column contracts.

row_count

Expected row count.

sha256

SHA-256 of the table file.

Column fields

Field

Meaning

name

Column name.

dtype

Logical type: string, int64, float64, bool, datetime, or date.

required

Whether the column must be present. Defaults to true.

unit

Physical unit, when the column has one.

description

Short human description.

The segments table

The P1.1 contract requires four columns:

Column

Type

Meaning

segment_id

string

Primary key.

corridor_id

string

Corridor that owns the segment.

length_m

float64

Segment length in metres.

geometry_wkt

string

Segment geometry as WKT.

P1.2 adds the span, treatment-unit, site, access, structure, and service-standard fields.

Content digest

The content digest covers the schema version, the dataset id, the CRS, and the table contracts: name, path, format, primary key, row count, checksum, and column names, types, required flags, and units. It does not cover created_utc, name, or description, so a rebuild with the same content keeps the same digest.

Validation

Run:

brushline validate --dataset build/compiled
brushline validate --dataset build/compiled --json

The validator checks:

  • the manifest exists and is valid JSON;

  • the schema version is brushline.dataset.v1;

  • every table path is relative and stays inside the dataset;

  • every table file exists and can be read;

  • every required column is present and has the declared logical type;

  • the primary key is present, non-null, and unique;

  • the row count matches;

  • the SHA-256 matches;

  • the content digest matches;

  • the required segments table is present.

Errors use stable prefixes: DATASET-MANIFEST, DATASET-SCHEMA, DATASET-PATH, DATASET-TABLE, DATASET-COLUMN, DATASET-DTYPE, DATASET-KEY, DATASET-ROW-COUNT, DATASET-CHECKSUM, and DATASET-DIGEST. An undeclared extra column is a warning (DATASET-EXTRA-COLUMN), not an error.

Example

{
  "schema_version": "brushline.dataset.v1",
  "dataset_id": "synthetic-corridor-v1",
  "name": "Synthetic corridor",
  "description": "A two-corridor contract fixture.",
  "created_utc": "2026-11-14T00:00:00Z",
  "crs": "EPSG:3005",
  "units": {"length": "m", "area": "ha"},
  "tables": [
    {
      "name": "segments",
      "path": "segments.parquet",
      "format": "parquet",
      "primary_key": ["segment_id"],
      "columns": [
        {"name": "segment_id", "dtype": "string", "required": true},
        {"name": "corridor_id", "dtype": "string", "required": true},
        {"name": "length_m", "dtype": "float64", "required": true, "unit": "m"},
        {"name": "geometry_wkt", "dtype": "string", "required": true}
      ],
      "row_count": 2,
      "sha256": "0000000000000000000000000000000000000000000000000000000000000000"
    }
  ],
  "provenance": {
    "source": "synthetic",
    "source_version": "1.0",
    "licence": "CC0-1.0",
    "created_by": "brushline"
  },
  "content_digest": "0000000000000000000000000000000000000000000000000000000000000000"
}

The example digests are placeholders. A real manifest carries the actual file checksum and the computed content digest.