Stand density management and the AU Explorer

WS4 is adding a stand density management diagram (SDMD) layer. It is a diagnostic and constraint layer, not a growth model. It places AU yield evidence, observed sample plots, and compiled FVS trajectories on a species/region diagram, compares silviculture regimes as paths, and turns the reviewed result into solver constraints. The decision is ADR 0013.

Note

The layer is being built in Phase 87. This page documents the stable schema that ingestion, the API, the web tab, and the solver read. Ingestion, digitisation, the API, the web tab, and solver filtering arrive in later tasks.

The six tables

The contract lives in ws4.sdmd.schema. A row always carries a source and a review_status of unreviewed, provisional, reviewed, or rejected.

plots

One sample plot visit: region, BEC, TSA, AU, year, stand age, site index, slope, aspect, elevation, plot area, sources, and review status. The primary key is (plot_id, visit_year).

trees

One measured tree: species, DBH, height, crown ratio, expansion factor, live flag, damage, and source. The primary key is (plot_id, tree_id).

stands

One stand summary per plot visit: age, TPH, basal area, QMD, top and mean height, volume, crown cover, source, and review status.

sdmd_points

One AU-level point on the diagram: site index, age, top height, QMD, TPH, basal area, mean tree volume, relative density, SDI, maximum SDI, zone, source, and review status.

regimes

One silviculture regime as an ordered state machine: entry age and type, residual basal area, removal fraction, cycle length, repeat count, post- treatment track, eligibility and operability expressions, and review status.

au_constraints

One allowed or denied regime for one AU: entry-age window, maximum entries, minimum interval, maximum removal fraction, required SDMD zone, source, and review status. The primary key is (au_id, regime_id).

Validation

coerce_to_schema types a frame with nullable pandas dtypes. validate_frame checks the structure of one table: required columns, value kinds, non-null required values, non-empty required text, and a unique primary key. validate_dataset adds value domains and cross-table references.

Findings use stable codes:

SDMD-MISSING-COLUMN, SDMD-COLUMN-KIND, SDMD-NULL-VALUE

Structure. A table without a required column or value cannot pass.

SDMD-EMPTY-VALUE

A required text field is blank.

SDMD-DUPLICATE-KEY

A primary key repeats.

SDMD-MISSING-TABLE, SDMD-UNKNOWN-TABLE

A required table is absent, or a name is not part of the contract.

SDMD-RANGE

A numeric value is outside its domain.

SDMD-ENTRY-WINDOW

A constraint’s minimum entry age exceeds its maximum.

SDMD-ENUM

A review status, SDMD zone, or regime entry type is unknown.

SDMD-UNKNOWN-PLOT, SDMD-UNKNOWN-REGIME, SDMD-UNKNOWN-AU

A reference points at a row that does not exist.

SDMD-REGIME-AU

A constraint names a regime that belongs to another AU.

The fixture in tests/fixtures/sdmd/ has two AUs and ten plots and validates cleanly. It is the contract that later tasks reuse.

Ingestion

The adapters in ws4.sdmd.ingest take a DataFrame and return a typed contract table. They do not read a proprietary file format and do not fetch data; the caller is responsible for obtaining the public source under its licence.

from ws4.sdmd.ingest import (
    label_synthetic,
    leading_species,
    match_plots_to_aus,
    read_lidar_evidence,
    read_psp_plots,
    read_psp_trees,
    read_vri_aus,
)

plots = read_psp_plots(raw_plots)
trees = read_psp_trees(raw_trees, plots=plots)
aus = read_vri_aus(raw_vri)
species = leading_species(trees)
result = match_plots_to_aus(plots, aus, leading_species=species)
matched, unmatched = result.matched, result.unmatched
lidar = read_lidar_evidence(raw_lidar)

read_psp_plots and read_psp_trees accept the canonical column names and a small set of short aliases, coerce the types, and refuse a schema or domain finding. read_vri_aus requires one row per AU with a BEC/TSA key, a species group, and optional age and site-index windows.

match_plots_to_aus fills a blank au_id only when exactly one AU matches on region, BEC, TSA, species group, age, and site index. No match and more than one match go to unmatched with a match_reason; neither is guessed. A preassigned au_id is kept only when the VRI table contains it.

label_synthetic prefixes the source with synthetic: and sets the review status to provisional. read_psp_plots refuses a row that names nemora without that label, so a gap-fill cannot enter as an observation.

read_lidar_evidence accepts piece size, clump fraction, canopy cover, and a total TPH. A row whose tph_basis is canopy_only may carry structure, but a TPH on such a row is refused: canopy-derived density is not total stand density. The basis is a closed set, and every row must carry at least one metric.

Diagram and AU report

A diagram is a JSON file with ordered vector isopleths, never a raster. The x axis is trees per hectare and the y axis is mean tree volume; interpolation is linear in log10 space, which is how the classic diagram is read. load_diagram validates the identity, the source, the review record, and every curve before the diagram is used.

Each diagram needs at least two top_height curves, at least two qmd curves, and exactly one each of crown_closure, competition, and self_thinning. An optional cai_band is a closed display-only polygon for the source’s grey “Density Range for Maximizing CAI” region. It has no numeric level, never feeds placement or constraints, and is not a substitute for a reviewed curve. A reviewed diagram must name a reviewer and a review time. The maintainer (G. Paradis) is the reviewer of record for this pilot; an RPF can be named later without a schema change. An unreviewed or provisional diagram cannot enter an AU report unless the caller passes allow_unreviewed=True for method work, and the report records the gap.

from ws4.sdmd import (
    Trajectory,
    assess_trajectory,
    build_au_report,
    load_diagram,
    observed_envelope,
    place_point,
)

diagram = load_diagram("ws4/sdmd/data/sdmd/idf-dk3-fdi-illustrative.json")
placement = place_point(diagram, tph=1000.0, mean_tree_volume_m3=1.2)
envelope = observed_envelope([(980.0, 0.757), (2200.0, 0.128), (1700.0, 0.543)])
assessment = assess_trajectory(
    Trajectory(label="fvs", points=((1700.0, 0.543), (1350.0, 0.793))),
    envelope,
)
report = build_au_report(
    au_id="AU-IDFdk3-FDI",
    diagram=diagram,
    observed=points,
    trajectories=(Trajectory(label="ct60", points=ct60_points),),
    initial_volume_m3_ha=500.0,
    initial_volume_limit_m3_ha=800.0,
    allow_unreviewed=True,
)

place_point names the zone: understocked below the crown-closure line, density-management between closure and competition, competition below the self-thinning line, and outside above it or beyond the digitised axis domain. It also interpolates top height and QMD, and flags a point outside the isopleth range.

observed_envelope builds the convex hull of an AU’s observed points in log space. validate_trajectory refuses a path where density and mean tree volume both increase between points: a stand cannot gain trees and stem volume at once. assess_trajectory returns inside or extrapolated and names the first point outside. build_au_report combines the observed evidence, the trajectories, and the initial volume into one status: supported, extrapolated, rejected, or unsupported. The initial-volume check rejects a volume above the supplied evidence limit; an upper-tail trajectory that leaves the observed envelope is flagged as extrapolation even when the volume appears in a few old plots.

Regimes and constraints

A regime is a repeated entry, not a one-off schedule. load_regime_library reads the regimes table, validates the sequence, and expands cycle_length and repeat_count into ordered entry ages. Regime.entry_ages() returns the sequence, entry_count() returns the number of entries, and is_sdmd_valid is false for a selection regime. The untreated baseline has no entries and is exempt from the entry window.

load_constraints reads the au_constraints table. check_regime returns findings instead of raising, so a caller fails closed. It checks the AU/regime pair, the allow flag, the entry-age window, the entry cap, the minimum interval, the maximum removal fraction, and whether a required SDMD zone can be met by a selection regime. allowed_regimes returns only the regimes that pass every check, and option_counts reports the regime and entry counts before and after.

from ws4.sdmd import (
    allowed_regimes,
    check_regime,
    load_constraints,
    load_regime_library,
    option_counts,
)

regimes = load_regime_library(regime_frame)
constraints = load_constraints(constraint_frame)
allowed = allowed_regimes(regimes, constraints, "AU-IDFdk3-FDI")
counts = option_counts(
    tuple(item for item in regimes if item.au_id == "AU-IDFdk3-FDI"),
    allowed,
)

A forbidden entry never reaches the model: the caller turns a non-empty finding list into a refusal. Solver filtering itself is P87.7.

Stand density equations

ws4.sdmd.equations is an independent Python implementation of the density equations used by the BC stand density management diagrams. It does not call, wrap, or vendor the R application or the standview package. The equation forms and coefficients come from the published sources; the unit handling follows the BC app so the two implementations can be compared.

Sources and ideas:

  • The BC Government SDMD Shiny application, bcgov/SDMD (Apache-2.0), for the equation-set selection, the metric/English conversion path, and the maximum-SDI defaults and ranges.

  • The standview R package by M. W. Ritchie (USFS), mwritchie/standview, for the diagram calculation logic that the BC app uses.

  • The published equations: Long and Shaw (2005) ponderosa pine; Jang, Ritchie, and Zhang (2021) ponderosa pine; Long and Shaw (2012) Sierra Nevada mixed-conifer; Drew and Flewelling (1979) Douglas-fir; McCarter and Long (1986) lodgepole pine; and Reineke (1933) for the stand density index.

  • The static/dynamic/structural framing and the review of SDMD variants: 10.1139/cjfr-2020-0289.

  • The BC stand density management page, which publishes the diagram set: Stand density management.

Five equation sets have a complete volume and top-height calculation: 2, 3, 6, 7, and 9. Each set declares whether it has volume, top height, crown cover, or biomass, and its maximum-SDI default and range. The public functions take metric inputs and return metric outputs.

from ws4.sdmd import (
    basal_area_m2_ha,
    relative_density,
    stand_density_index,
    top_height_m,
    volume_m3_ha,
    volume_isoline,
)

volume = volume_m3_ha(3, tph=1000.0, qmd_cm=25.0)
height = top_height_m(3, tph=1000.0, qmd_cm=25.0)
sdi = stand_density_index(1000.0, 25.0)
rd = relative_density(1000.0, 25.0, max_sdi=988.0)
isoline = volume_isoline(3, target_m3_ha=300.0, tph_values=(500.0, 1000.0))

stand_density_index defaults to a 25.4 cm reference, the 10-inch convention the BC app uses for its metric maximum-SDI values. Pass reference_cm=25.0 for the common metric convention. volume_isoline inverts a volume equation by bisection on QMD; densities that cannot reach the target are omitted rather than clipped.

The module’s docstrings and the equation registry carry the per-equation citations. No R code or third-party text is copied into this repository.

AU Explorer API

ws4.sdmd.api provides a FastAPI router over a directory of the six tables, an optional diagram.json, an optional initial_volume.json, and optional trajectories/<au_id>/<regime_id>.json files. The initial-volume file records the measured value, its evidence limit, the source, and the review status; the gate uses it when the command line does not supply an override. The main web app mounts it when ws4 serve receives --sdmd-dir:

ws4 serve --dataset build/compiled --runs runs \
          --sdmd-dir data/sdmd

The read endpoints are:

  • GET /api/sdmd/regions

  • GET /api/sdmd/aus?region=...

  • GET /api/sdmd/au/{au_id}

  • GET /api/sdmd/au/{au_id}/plots

  • GET /api/sdmd/au/{au_id}/yield

  • GET /api/sdmd/au/{au_id}/sdmd

  • GET /api/sdmd/au/{au_id}/regimes

  • GET /api/sdmd/au/{au_id}/trajectory?regime_id=...

  • GET /api/sdmd/au/{au_id}/constraint-preview

POST /api/sdmd/au/{au_id}/constraints writes a complete constraint set. The validator requires one entry per regime for the AU, a matching au_id, valid field types, the schema limits, and a passing check_regime() for every entry. A deliberate allowed: false is valid; any other finding refuses the write with 422 and leaves the stored set unchanged. A successful write is atomic and is stored at constraints/<au_id>.json; the next store load reads it back.

GET /api/sdmd/au/{au_id}/trajectory reads a stored trajectory and assesses it against the AU’s observed envelope. The response carries the stored source, scenario, and review_status beside the points. When the file is absent, the response is status: "unavailable" with a named data gap rather than an invented path.

AU Explorer tab

The browser shell gains an optional SDMD workspace when the service is started with --sdmd-dir. It is a reader over the API above, with no build step.

  • The left rail chooses a region and AU, filters the yield rows by source, and shows the AU’s plot count, age support, species, and diagram state.

  • The main canvas draws the SDMD isopleths, the observed points, the convex envelope, and the selected regime path on logarithmic axes. A legend and a hover readout name the curves and the zone. The yield chart has a metric selector (volume, density, basal area, QMD, top height) and its own hover readout.

  • An interpretation panel states the observed support, envelope range, diagram review state, the selected regime’s verdict and reasons, the option reduction, and the data gaps. It says explicitly when the diagram is an illustrative fixture and when a regime has no stored trajectory.

  • The constraint editor has one row per regime: allow/deny, entry-age window, maximum entries, minimum interval, and maximum removal fraction. Saving posts the complete set to the validated endpoint and refreshes the option-count preview. A refused save leaves the stored set unchanged.

  • Use in scenario records the AU in the URL for the solver handoff.

  • The selected AU and workspace are URL state, so a link opens the same view.

  • The trajectory menu lists only regimes with a stored path; the other regimes are disabled and labelled “no stored path”.

Solver integration

The optimizer reads the compiled matrix, so the safe integration is to remove forbidden treatment rows before it builds candidate chains. ws4.sdmd.solver.filter_dataset reads tracks.parquet and treatments.parquet, maps each track to an AU from the AU(...) field in the track name or a sidecar map, matches each treatment label to a regime, and drops every treatment row for a track that fails the constraint checks.

ws4 sdmd filter-dataset \
  --dataset build/compiled \
  --out build/compiled-sdmd \
  --regimes regimes.csv \
  --constraints au_constraints.csv \
  --map sdmd-map.yaml

The filter enforces the per-track entry-age window, entry cap, minimum interval, and removal fraction. It writes a dataset copy with every other table unchanged and reports tracks_before, tracks_after, treatments_before, and treatments_after, plus one record per dropped track with its reasons. Unmapped tracks are kept by default; --strict drops them. A sidecar map can override the track AU and treatment aliases:

tracks:
  16: AU-IDFdk3-FDI
treatments:
  CC: REG-A-CLEARCUT

The filter can also enforce the required SDMD zone: pass a diagram and a trajectory directory, and a track is kept only when its stored trajectory reaches the constraint’s required_sdmd_zone. A missing trajectory fails closed.

SdmdScheduleGuard checks a whole schedule for the per-AU entry cap and minimum interval. It maps each block to its AU, counts the selected entries, and returns SDMD-SCHEDULE-ENTRIES or SDMD-SCHEDULE-INTERVAL findings. The optimizer wires it into every move when –sdmd-dir names a store with regimes.csv and au_constraints.csv; an infeasible seed raises rather than being repaired. The run summary records constraint_stats.sdmd_guard and the number of rejected moves as constraint_stats.sdmd_rejected, which is the enforcement benchmark.

Validation gates

ws4.sdmd.validation runs eight gates for one AU:

  1. Plot support — at least three observed plots and at least three PSP-backed points. Each point must have a reviewed PSP plot at its age and a tree list; synthetic points are labelled and excluded from the observed envelope.

  2. Density plausibility — reported basal area agrees with TPH and QMD.

  3. SDMD fit — the diagram exists, is reviewed, and the points are inside it.

  4. Trajectory plausibility — every stored path is physical and assessed against the observed envelope.

  5. Initial volume — the value is supplied and inside the evidence.

  6. Regime completeness — every regime has a constraint with no unexplained finding.

  7. Solver option reduction — the constraints reduce the option space without removing every regime.

  8. UI payload contract — the detail, yield, SDMD, regime, and preview payloads have the keys the tab reads.

ws4 sdmd validate --sdmd-dir data/sdmd --au AU-IDFdk3-FDI
ws4 sdmd validate --sdmd-dir data/sdmd --au AU-IDFdk3-FDI \
  --initial-volume 863 --initial-volume-limit 800

The command exits 1 on any failed gate. The fixture warns rather than fails: its diagram is provisional, it supplies no initial volume, and one demo trajectory is an extrapolation. The initial-volume tests reject the 863 m3/ha value against an 800 m3/ha limit and flag 1,173 m3/ha as above the observed maximum.

Public sources

The BC Ministry of Forests publishes the complete public set as 14 one-page PDFs: seven species, each in a natural and a planted version. The species are coastal Douglas-fir, interior Douglas-fir, lodgepole pine, Sitka spruce, western hemlock, western redcedar, and white spruce. The machine-readable catalog is ws4/sdmd/data/sdmd/source_catalog.json; the BEC zone/subzone/variant coverage is ws4/sdmd/data/sdmd/bec_variants.csv (229 rows). The full note is notes/reference/bc-sdmd-sources.md.

The diagrams are species- and origin-based, not BEC-variant-based. The variant CSV expands the zone-level species occurrence from the BC Tree Species Compendium, so it is an inference that a forester must confirm against the AU’s leading species and stand origin. For the pilot AU IDFdk3 the diagram is fdin (natural) or fdip (planted) for Interior Douglas-fir, with lodgepole pine and western redcedar as main associates.

The catalog records the index-page label and the PDF title separately: the BC page labels swn/swp as “White pine”, but the PDFs are white spruce, and the catalog keeps “White pine” as an alias so the page label still resolves.

The Farnden 1996 method report (Pacific Forestry Centre Information Report BC-X-360) is public and is recorded in the catalog with its URL and checksum. The PDFs are referenced, not copied into the repository. The natural and planted Interior Douglas-fir PDFs are digitised into idf-dk3-fdi-natural.json and idf-dk3-fdi-planted.json. The natural lodgepole pine PDF (pln, version 1.20) is digitised into idf-dk3-pli-natural.json with QMD 10-40 cm, top height 6-34 m, the competition/self-thinning boundaries, and the 60% crown-closure curve. The planted PDF (plp, version 1.20) is digitised into idf-dk3-pli-planted.json with top height 6-30 m. The real PLI pilot store is ws4/sdmd/data/pilot-tsa23-idfdk3-pli/. The method, curve assignments, and review checklist are in notes/reference/bc-sdmd-digitisation.md. The review packet with overlay figures for all four diagrams is notes/review/p95-curve-review.md.

The digitisation pipeline uses the public figrecover package for PDF rendering, overlay QA, and review manifests, and freshforge for the workflow graph. The optional extra installs both:

python -m pip install -e '.[digitise]'
freshforge plan workflows/sdmd-digitise.yaml

The vector-path extractor in ws4.sdmd.digitise stays responsible for isopleth precision; figrecover owns the rendered page, overlay, quality metrics, and review record. A diagram spec names the PDF axis ticks, the drawing path indices, and the printed isopleth values, so the workflow does not guess curve identities.

For a line-type pass, ws4.sdmd.import-lines reads the printed value unit labels, finds the paths with the requested dash pattern and width, requires a unique close match with vertical order, and refuses ambiguous, distant, duplicate, or unordered assignments. The confirmed fdcp top-height pattern is locked in workflows/sdmd-line-import.yaml; it renders the source with figrecover and writes a ws4.sdmd.line-import.v1 artifact with the path index and label distance for every curve. The ten diagrams remain provisional until the maintainer reviews the curves; see notes/review/p137-overlay-findings.md. The synthetic idf-dk3-fdi-illustrative.json remains the test fixture, not the published diagram. Four method trajectories from fresh-fvsbc are in ws4/sdmd/data/trajectories/; they are method runs over synthetic tree lists, not real AU projections. See notes/reference/sdmd-pilot-trajectories.md.

The CCISS project (Apache-2.0) supplies complementary species evidence: 37 BC tree species codes, silvics traits for the six SDMD species, IDFdk3 stocking targets, and the CCISS label table (bec_crosswalk.csv). The derived CCISS files are ws4/sdmd/data/sdmd/tree_species.csv, species_traits.csv, idfdk3_stocking.csv, idfdk3_suitability.csv, and idfdk3_sibec.csv; the method and limits are in notes/reference/cciss-data-mining.md.

The BEC authority tables come from BC BECdb v13, not CCISS: bec_official.csv, bec_retired.csv, and bec_sources.json. P121 derives bec_official_crosswalk.csv and bec_crosswalk_review.csv from those tables; the counts and limits are in notes/reports/p121-bec-crosswalk.md. CCISS does not contain SDMD isopleths, so it cannot replace the missing lodgepole pine diagram.

All 14 public diagrams are now stored: seven species in natural and planted versions. Four Interior Douglas-fir/lodgepole pine diagrams are reviewed; the other ten are provisional. Their crown closure is a straight log-log line fitted to the source, and the grey CAI band is stored as a display-only polygon. The atlas status and BEC counts are in notes/reports/p120-atlas-coverage.md.

The first real pilot store is ws4/sdmd/data/pilot-tsa23-idfdk3/. It holds 11 reviewed public PSP points for TSA 23 / IDFdk3 / FDI and two FVS paths from the real tree list of one cluster. Its SOURCE.md names the BC Data Catalogue record, the Open Government Licence, the selection rule, the field derivations, the FVS version, and the limits. The diagram is still provisional, no initial volume is supplied, and both FVS paths leave the observed envelope; the store is evidence for the pipeline, not a yield forecast.

BC-wide coverage

bec_coverage.csv is the 219-row coverage registry. Each current BEC label names its zone, subzone, variant, applicable Atlas diagrams, species/origin, review counts, and separate AU evidence. The Atlas diagrams are species/origin curves with zone-level applicability from the public source catalogue, so the status is explicit about the match:

  • 1 reviewed_variant: IDFdk3, the pilot variant

  • 5 reviewed_zone labels with only reviewed diagrams

  • 209 mixed_zone labels where reviewed and provisional diagrams both apply

  • 4 no_diagram labels in BAFA, CMA, and IMA

match_level records variant, zone, or zone_rare. A zone-level match is not a variant-specific claim; verify against the AU’s leading species, origin, and BEC. GET /api/sdmd/coverage returns the registry and filters by zone or status; GET /api/sdmd/coverage/{bec} returns one label. The SDMD workspace exposes a BC coverage selector. bec_review_queue.csv names the 10 provisional diagrams and the 210 BEC labels they affect. The registry never substitutes an unrelated AU.

Evidence roles and limits

PSP supplies total TPH, basal area, QMD, height, and age support. VRI supplies AU identity and species composition. LiDAR supplies piece size, crown structure, and clump evidence. Canopy-only LiDAR is never total TPH, and nemora is a labelled gap-fill rather than an observed source.

The classic diagram assumes even-aged, healthy, untreated development. Selection regimes are flagged as outside the diagram, not plotted as valid paths. The provincial atlas, legal-compliance determination, a new growth model, and an AFRF calibration without enough plots are out of scope.