Held-out validation
P4.6 validates the growth and state model against held-out observations and benchmarks at least one candidate model. The validation is deterministic, offline, and separate from the calibration set.
The committed fixture is notes/reference/vegetation-validation.v1.json.
Dataset
A ValidationRecord carries:
segment id, spatial block, observation date, and prediction year;
functional group;
observed and predicted height, cover, and optional density;
predicted height and cover standard deviations;
observation coverage and domain status;
treatment history, source, and source version;
optional candidate predictions.
Density requires both observed and predicted values. Candidate names must be unique per record. Out-of-domain records need a reason.
Holdout and eligibility
The config names one or more holdout blocks. A record is eligible when:
it is in a holdout block;
coverage meets the minimum;
it is in domain, unless out-of-domain records are explicitly included;
the absolute prediction lag is within the maximum.
Excluded records are counted by reason: coverage, out_of_domain, and time_lag. A holdout
with no eligible records is an error.
Metrics
For each functional group and variable, the report records:
record count;
observed and predicted means;
RMSE, MAE, and bias;
interval coverage: the share of records inside the predicted interval.
Height and cover are always reported when present; density is reported only when both observed and predicted values exist.
Time-lag sensitivity
Records are grouped by absolute prediction lag. Each lag reports height and cover RMSE and bias. This shows whether a model degrades as the prediction moves away from the observation year.
from datetime import UTC, datetime
from pathlib import Path
from brushline.vegetation.validation import (
ValidationConfig,
ValidationDataset,
validate_vegetation,
validate_vegetation_validation,
)
payload = __import__("json").loads(
Path("notes/reference/vegetation-validation.v1.json").read_text()
)
dataset = ValidationDataset.model_validate(
{"schema_version": payload["schema_version"], "records": payload["records"]}
)
report = validate_vegetation(
dataset,
ValidationConfig.model_validate(payload["config"]),
created_utc=datetime(2026, 11, 14, tzinfo=UTC),
)
assert validate_vegetation_validation(report).is_valid
Candidate benchmark
Each candidate prediction contributes a CandidateSummary with height/cover RMSE and bias, plus the
RMSE difference from Brushline. A difference above the disagreement tolerance raises a reviewer
question. The disagreement is recorded; it is not hidden or automatically corrected.
Output
VegetationValidationReport carries:
group/variable metrics;
lag summaries;
candidate summaries and differences;
exclusion counts;
errors, warnings, and open questions;
counts.
Named errors
Prefix |
Meaning |
|---|---|
|
A holdout block is not in the dataset. |
|
No records fall in the holdout blocks. |
|
No eligible records, or a metric/candidate has no records. |
|
The report time is not timezone-aware. |
|
Duplicate metrics or candidate names. |
|
An unknown validation variable. |
|
Interval coverage is outside 0 to 1. |
|
A negative time lag. |
|
No candidate benchmark (warning). |
Limits
The fixture is synthetic; it is not a field validation.
No live MEDFATE, FVS/FFE, or Botequim run is performed.
The validation does not fit parameters; P4.4 calibrates treatment parameters.
A candidate difference is a finding, not a verdict on which model is correct.
The report shows error and interval coverage; it does not prove operational fitness.