Data Contract Guide

This guide summarises the structured inputs FHOPS expects when authoring scenarios. It builds on the examples/tiny7 and tests/fixtures/regression assets and reflects recent extensions (mobilisation, geo metadata, crew assignments).

Reference dataset generator

All bundled scenarios (examples/{tiny7,small21,med42,large84}) now originate from a single deterministic generator: scripts/rebuild_reference_datasets.py. The script samples Lahrsen-style stand attributes, derives ADV6N7 skidding distances from block geometry, and emits the full CSV + YAML bundle (blocks, machines, calendar, production rates, landings, mobilisation distances, README). Run:

python scripts/rebuild_reference_datasets.py tiny7        # or small21 / med42 / large84

Use --seed to reproduce specific variants. Regenerate the dataset before updating docs/fixtures so the CSVs, README stats, and MILP bundles stay in sync across the ladder.

Core Tables (CSV)

Each scenario references a set of CSV files. Required columns and notes:

  • schema_version: Contract version tag (currently 1.0.0).

Table

Required Columns

Notes

blocks.csv

id, landing_id, work_required

Optional: earliest_start/latest_finish (defaults 1 / num_days); stand metrics (avg_stem_size_m3, volume_per_ha_m3, stem_density_per_ha, ground_slope_percent) plus optional uncertainty columns (volume_per_ha_m3_sigma, stem_density_per_ha_sigma) are supported and default to Lahrsen (2025) BC ranges in synthetic bundles. Blocks may also specify harvest_system_id (must match a registry entry) and salvage_processing_mode (standard_mill | portable_mill | in_woods_chipping) so salvage presets and telemetry keep the ADV1N5 portable-mill vs. in-woods-chipping warnings aligned with the scenario data.

machines.csv

id

Optional: role, crew; numeric fields must be non-negative. daily_hours defaults to 24 — we assume machines are available around the clock unless explicitly constrained by the calendar or shift-level availability. operating_cost is the machine rental rate in $/SMH (scheduled machine hour), i.e., SMH = scheduled hours and PMH = productive hours; Lahrsen (2025) productivity coefficients are calibrated in m³/PMH15 (PMH with short delays ≤15 min included). Convert to $/m³ post‑hoc using playback outputs or the costing helper. When operating_cost is omitted or set to 0 and a role is provided, FHOPS looks up the default machine-rate entry (owning + operating + optional repair) and auto-fills the value during scenario validation. Supply repair_usage_hours (nearest 5 000 h bucket) to pick a different FPInnovations usage class when applying the repair/maintenance allowance; omit the field to stick with the default 10 000 h bucket. The fhops dataset inspect-machine CLI warns when a machine advertises non-24 h availability so you can catch accidental edits before shipping datasets.

landings.csv

id

daily_capacity defaults to 2, must be ≥ 0

calendar.csv

machine_id, day

available ∈ {0,1}; days must lie within num_days

production_rates.csv

machine_id, block_id, rate

rate ≥ 0; IDs must exist in machines/blocks

Cross References & Validators

The Pydantic models enforce consistency:

  • Blocks reference known landings; harvest-system IDs must exist (see Harvest System Registry for defaults).

  • Calendar and production rates must reference defined machines/blocks and lie within the scenario horizon.

  • Mobilisation distances must reference known blocks; mobilisation parameters must reference known machines.

  • Crew assignments (optional) require unique crew IDs and valid machine IDs.

Optional Extras

Recent helpers enable richer metadata:

  • MobilisationConfig — per-machine mobilisation costs and block distance matrices.

  • GeoMetadata — optional GeoJSON paths and CRS tags for blocks/landings.

  • CrewAssignment — map crew identifiers to machines/roles for downstream planners.

  • TimelineConfig — shift definitions and blackout windows controlling daily availability.

  • ScheduleLock — pre-assign specific machine/block/day combinations (enforced in MIP & SA).

  • ObjectiveWeights — tweak solver objective weighting (production, mobilisation penalties, transition counts, optional landing-cap slack penalties).

  • Round-the-clock operations — unless you override machines.csv or the calendar, FHOPS expects continuous availability (24 h/day) and only pauses work when you add blackouts, downtime sampling, or calendar gaps.

ObjectiveWeights fields are optional; omit any you do not need. For example:

objective_weights:
  production: 1.0
  mobilisation: 0.5
  transitions: 2.0
  landing_surplus: 3.0

This configuration maximises production while penalising moves between blocks and soft landing capacity violations. Setting a weight to 0 reverts to the default hard behaviour.

Reference tests/test_contract_validations.py for examples that exercise these validators.

Timeline Example

Add a top-level timeline block in your scenario YAML to describe shifts and blackouts:

timeline:
  shifts:
    - name: day
      hours: 10
      shifts_per_day: 1
  blackouts:
    - start_day: 5
      end_day: 7
      reason: wildfire risk
  days_per_week: 5

The loader converts this into a TimelineConfig instance available via scenario.timeline.

Shift Calendars & Defaults

When you need per-machine shift availability (for example, night crews that only run certain machines), add a shift_calendar CSV and reference it under data.shift_calendar:

machine_id,day,shift_id,available
H1,1,day,1
H1,1,night,0
H2,1,day,1
H2,1,night,1

shift_id values must match names declared in the timeline’s shifts section; the loader rejects unknown labels or days outside num_days. Supplying multiple rows for the same machine/day lets you model overlapping shifts with different availability flags, and you can leave individual rows at available=0 to block a specific crew without editing the base timeline. When a shift_calendar is omitted entirely, FHOPS synthesises a single shift named S1 for every day so legacy day-indexed scenarios continue to work without edits.

Run fhops validate to confirm the calendar files and shift labels line up before solving:

fhops validate examples/tiny7/scenario.yaml

The CLI surfaces precise validation errors (missing columns, bad shift IDs, horizon overruns) so you can correct the dataset without spelunking through Pydantic stack traces.

Schedule Locking

Lock a machine to a block on a given day by adding locked_assignments:

locked_assignments:
  - machine_id: YARDER1
    block_id: B12
    day: 5

Any attempt to reassign that machine/day is blocked in both the MIP builder and the SA heuristic.

GeoJSON Ingestion & Distances

When supplying block or landing geometries:

  • Provide GeoJSON files with a FeatureCollection containing polygon features.

  • Each feature must include an id or properties.id matching the block/landing IDs defined in the CSV tables.

  • Use a projected CRS suitable for distance calculations; FHOPS defaults to EPSG:3005 (BC Albers) but any metre-based CRS is acceptable when specified via geo.crs.

  • Store relative paths in GeoMetadata.block_geojson / landing_geojson so the CLI tooling can locate the files.

To generate mobilisation distances from geometries, run:

fhops geo distances --blocks blocks.geojson --out mobilisation_distances.csv

The command computes centroid-to-centroid distances (in metres) respecting the CRS. The resulting CSV aligns with the MobilisationConfig distance format and can be referenced under scenario.data.mobilisation_distances.

Machine-Rate Defaults & Costing Helper

FHOPS includes a BC-focused rental-rate catalogue under data/machine_rates.json to keep costing consistent across scenarios:

  • Each entry records machine_name, role, owning/operating costs ($/SMH), default utilisation, move-in allowance, citation, and notes. Baseline values combine Dodson et al. (2015) Montana machine-rate data (converted at ~1.33 CAD/USD with diesel ≈ CAD 1.80/L) and Hartley & Han (2007) coastal grapple-yarders to match BC market conditions.

  • Repair and maintenance allowances come from FPInnovations Advantage Vol. 4 No. 23 (2003). The report’s 2002 CAD regression averages are escalated to 2024 CAD using the Statistics Canada Machinery & Equipment CPI (Table 18-10-0005-01), giving a cumulative multiplier of ≈1.56. FPInnovations reports the allowances at cumulative usage classes of 0–5k, 0–10k, 0–15k, 0–20k, and 0–25k operating hours; we treat the 10 000 h bucket as the default because every machine class has data at that point and it best represents a “typical” fleet age. Newer machines (≤5 000 h) and end-of-life machines (≥15 000 h) sit in the neighbouring buckets, so supplying repair_usage_hours lets you scale the allowance up or down using the published FPInnovations ratios when a scenario deviates from the baseline. These allowances remain optional per role.

  • The CLI exposes the defaults via fhops dataset estimate-cost:

    # Deterministic Lahrsen productivity with table defaults
fhops dataset estimate-cost --machine-role grapple_yarder \
  --avg-stem-size 0.45 --volume-per-ha 320 --stem-density 750 --ground-slope 35

    The command prints the chosen role, source, owning/operating/repair split, utilisation, productivity (m³/PMH15), and $/m³. Add --include-repair/--exclude-repair to toggle the FPInnovations allowance or --usage-hours to select a different usage-class multiplier (nearest 5 000 h bucket from Table 2). Supplying --dataset <scenario.yaml> --machine <ID> auto-loads the machine’s role, operating_cost (when already populated), and repair_usage_hours so recurring reports always respect the dataset defaults. Override components directly via:

    fhops dataset estimate-cost --machine-role feller_buncher \
  --owning-rate 95 --operating-rate 120 --repair-rate 40 \
  --productivity 30 --utilisation 0.8

  • When you also need a road/subgrade allowance, append the TR-28 helper directly to the same command:

    fhops dataset estimate-cost --machine-role grapple_yarder \
  --productivity 22 --utilisation 0.85 \
  --road-machine caterpillar_235_hydraulic_backhoe --road-length-m 150

    Supplying --road-machine (slug or name from fhops dataset tr28-subgrade) together with --road-length-m prints a CPI-adjusted “TR-28 Road Cost Estimate” table after the machine-cost summary. Use --road-exclude-mobilisation when movement is covered elsewhere—the CLI warns in either case and cites the soil-protection guidance from FNRB3 (Cat D7H vs. D7G trial) and ADV4N7 (compaction thresholds). When the scenario already lists road jobs (see below), fhops.dataset estimate-cost --dataset … auto-selects the only row or lets you pick via --road-job-id; attach additional soil warnings by referencing data/reference/soil_protection_profiles.json through --road-soil-profile (or via the CSV).

Road Construction Tables & Validation

To keep road-building metadata alongside the rest of the scenario, add an optional road_construction table:

id,machine_slug,road_length_m,include_mobilisation,soil_profile_ids
RC1,caterpillar_235_hydraulic_backhoe,150,True,fnrb3_d7h|adv4n7_compaction

Reference the file under data.road_construction in scenario.yaml. Each row requires a unique id, a TR-28 machine slug (see fhops dataset tr28-subgrade), the road/subgrade length in metres, and whether the published mobilisation charge should be included by default. soil_profile_ids (pipe- or comma-separated) tie into data/reference/soil_protection_profiles.json so the CLI can print structured reminders (ground-pressure multipliers, compaction thresholds, recommended mitigation). When a scenario contains exactly one row, estimate-cost --dataset ... pulls that entry automatically; specify --road-job-id RC1 if multiple road jobs exist or pass --road-machine / --road-length-m to override everything from the command line.

Harvest-system templates now infer these rows from the skyline/cable presets they assign: any block that references a preset with defined road defaults (see fhops.scheduling.systems.SYSTEM_ROAD_DEFAULTS) gains the matching TR-28 slug/length automatically, and the synthetic dataset generator writes data/road_construction.csv alongside the other tables so downstream tooling picks up the entries without manual edits. Run fhops validate after editing the table to catch duplicate IDs or malformed slugs before passing the scenario to solvers or costing helpers.

  • Supplying --rental-rate bypasses the lookup for bespoke studies, but machines.csv rows should normally use the curated rates (or CLI recomputed totals) so costing/evaluation tools stay aligned.

  • fhops dataset inspect-machine prints the machine metadata and the same default owning/operating/repair breakdown (honouring repair_usage_hours) so you can audit scenario inputs without running a full cost estimate. You can also inspect the rental table directly via --machine-role (e.g., fhops dataset inspect-machine --machine-role loader_barko450 dumps the TN-46 Barko 450 live-heel loader rate). Historical Appendix II presets such as loader_cat966c_tr45, skidder_tr45, and bulldozer_tr45 are exposed the same way so you can compare CPI-normalized 1979 CAD rates against modern entries. When you pair the command with a harvest system such as ground_fb_loader_liveheel, the loader job automatically swaps its cost role to loader_barko450 so budgeting outputs stay aligned with the TN-46 preset. Add --json-out machine.json to capture the same payload for automated QA pipelines. Solver telemetry (solve-heur, solve-ils, solve-tabu, eval-playback) automatically embeds this machine_costs bundle whenever --telemetry-log is used, so KPI histories and dashboards can trace the assumed repair buckets.

Overriding the defaults follows two common paths:

  1. Quick sensitivity checks – keep the built-in role but pass overrides to the CLI, e.g.

    fhops dataset estimate-cost --machine-role grapple_yarder \
  --owning-rate 185 --operating-rate 260 --repair-rate 50 \
  --avg-stem-size 0.5 --volume-per-ha 280 --stem-density 700 --ground-slope 40

    This keeps the source/role metadata but swaps in your costs for the report/run only.

  2. Persistent dataset changes – clone data/machine_rates.json or edit your scenario’s machines.csv. For the JSON table, add a new entry (e.g., "role": "custom_grapple") and point machines.csv.role at that slug; the loader will backfill operating_cost from your entry. Alternatively, write the full $/SMH into machines.csv.operating_cost for the specific machine if the rate is project-specific. Both approaches play nicely with the CLI helpers and solver defaults.

When authoring datasets set machines.csv.operating_cost to the all-in rental rate ($/SMH) for each machine/system. Use the CLI helper (deterministic or distribution-based) to turn Lahrsen stand descriptors and machine rates into comparable $/m³ for QA, reporting, or solver inputs.

Skyline Stand Profiles

Arnvik (2024) Appendix 5 stand metadata is bundled under data/reference/arnvik/appendix5_stands.json and exposed via the CLI:

  • fhops dataset appendix5-stands lists authors/species and the parsed slope/ground descriptors (filter with --author). The parser converts ranges (e.g., 23 (5-55)%) and qualitative labels (Level - Erasmus) into approximate slope percentages.

  • fhops dataset estimate-cable-skidding evaluates the Ünver-Okan (2020) uphill skidding regressions (spruce stands in north-east Turkey). Supply either --slope-percent or a stand --profile (author name) to pull slope defaults directly from Appendix 5, and treat the results as non-BC placeholders unless you calibrate them.

  • fhops dataset estimate-skyline-productivity wraps the Lee et al. (2018), TR-125, and TR-127 regressions, with optional --tr119-treatment multipliers (strip cut / retention levels) so partial-cut scenarios automatically apply the published BC productivity and cost offsets. Lee et al. (2018) regressions are small-scale tethered yarder studies from South Korea (HAM300), while TR-127 block-specific models (tr127-block1 … block6) use the Appendix VII coefficients from the FPInnovations case studies in northwestern BC. TR-112 only publishes descriptive productivity tables, so no regression helper is available for that report yet.

  • Programmatically, fhops.reference.arnvik_appendix5 exposes dataclasses and helper functions so skyline helpers and costing workflows can bind stand descriptors to the new cable productivity models.

Authoring Checklist

  1. Populate required CSV tables with consistent IDs and non-negative numeric values.

  2. Supply timeline and mobilisation sections when shift scheduling or mobilisation costs matter.

  3. Use crew_assignments and geo only when you have supporting data.

  4. Run fhops validate <scenario.yaml> to confirm the scenario satisfies the contract.