Recovery and External Runtime Boundaries

Purpose

This page is the compact source of truth for restart paths, recovery workflow, and the runtime assumptions FEMIC makes about external tools.

External Runtime Boundaries

Runtime seam

Contract

BatchTIPSY

Default unattended BTC runtime boundary. FEMIC writes the canonical 03_input-*.csv handoff, runs TIPSYbtc.exe /TSR on Windows, and resumes from returned 04_output-*.csv / 04_error-*.csv files.

Patchworks

Proprietary runtime boundary. FEMIC can export packages, run preflight, and launch commands, but users must supply the local Patchworks install, license wiring, and host-ready runtime. The only known-good default operator path is a Windows workstation with Patchworks already installed and the real SPS_LICENSE_SERVER value already present in system environment configuration. The current proving-ground headless seam is now real on native Windows: FEMIC can launch a .pin without classic_GUI(control), wait one unattended iteration, save a stage, and return control cleanly.

FAN$IER

Windows-only proprietary runtime boundary. FEMIC now owns a tracked unattended batch seam around Fansier.exe: it can launch a clean session, load one .rgm plus optional .dis, run Batch mode, harvest deterministic report outputs, and parse those reports through FEMIC-owned reporting/workflow surfaces.

ArcRasterRescue

Treat as an explicit external executable; if auto-discovery fails, set FEMIC_ARC_RASTER_RESCUE_EXE to the compiled path.

ArcGIS Pro fallback

Windows-only fallback path for SiteProd geoprocessing when canonical SiteProd artifacts are unavailable.

Critical BTC /TSR Runtime Note

The unattended BTC seam has one especially important hidden rule:

  • plain installed TIPSYbtc.exe /TSR consults the per-user overlay report under the current user’s Windows Documents folder: - <Documents>\BatchTIPSY Composer\TimberSupply.rpt before falling back to the stock installed report under C:\Program Files\TIPSY 4.7\BTC

For unattended FEMIC BTC /TSR work, this live user-overlay path is the only known-valid runtime seam.

When a Windows FEMIC environment is missing TIPSYbtc.exe, the standard recovery path is to install TIPSY 4.7 from the BC Government distribution package:

  • https://www2.gov.bc.ca/assets/gov/farming-natural-resources-and-industry/forestry/stewardship/forest-analysis-inventory/software/tipsy47.msi

After a normal install, FEMIC expects BTC at:

  • C:\\Program Files\\TIPSY 4.7\\BTC\\TIPSYbtc.exe

If FEMIC does not auto-discover BTC after install, set either FEMIC_BATCHTIPSY_EXE or pass --btc-exe explicitly.

Operational consequences:

  • a broken user-overlay TimberSupply.rpt can make apparently normal stock /TSR runs fail

  • moving the overlay out of the way restores stock fallback behavior

  • the safest unattended customization path is to preserve the stock TSR report structure and extend it conservatively through that overlay seam

  • copied-install-local TimberSupply.rpt overrides and stock-report-only /TSR probes are useful clue-gathering at best; they are not equivalent validation of the live unattended FEMIC seam

  • do not assume that replacing TimberSupply.rpt wholesale with a clean-room generated template is equivalent to the stock report contract

  • FEMIC should resolve the overlay path from the current user’s Windows Documents directory, not from any machine-specific OneDrive naming pattern

This is now a critical FEMIC development invariant for BTC reverse-engineering and unattended report-template probing.

Recovery Workflows

When a run stops at a known boundary, prefer the narrow restart path instead of rerunning the entire pipeline.

After Stage 01a / before BTC:

  1. confirm 03_input-*.csv exists and is the intended handoff payload

  2. run femic tsa btc-post-tipsy ... to launch unattended BTC and resume

  3. inspect 04_output-*.csv / 04_error-*.csv if the run fails

The command group and flag still use the legacy tsa naming seam for compatibility. Read them generically as the selected FMU/code target.

After BTC output refresh:

$env:FEMIC_EXTERNAL_DATA_ROOT="$PWD\external\femic-public-data\data"
python -m femic tsa btc-post-tipsy --instance-root external/femic-k3z-instance --run-config config/run_profile.k3z.yaml --tsa k3z --run-id k3z_resume

Before Patchworks runtime launch:

  1. run femic patchworks preflight ...

  2. confirm Java or Wine + Java is available for the host mode

  3. confirm patchworks.jar and SPSHOME resolve, and on Windows confirm the workstation already exposes the intended system-level SPS_LICENSE_SERVER value

  4. confirm the compiled Patchworks package is materially present

    Matrix-Builder-ready minimum:

    • forestmodel.xml

    • the full fragments shapefile sidecar set (fragments.{shp,dbf,shx,prj,cpg})

    • and, if you are resuming from an already-built model surface, the expected compiled tracks/*.csv tables

    • this tier is sufficient for preflight / Matrix Builder work, not for a fully shipped standalone launch surface by itself

    Standalone launch-ready published minimum:

    • compiled tracks/*.csv tables

    • blocks/blocks.shp plus required sidecars

    • the topology CSV used by the shipped analysis surface

    • the analysis/PIN launch surfaces used to open the model directly

  5. launch build-blocks or matrix-build only after preflight is clean

Before FAN$IER unattended extraction:

  1. confirm you are on a Windows host with Fansier.exe available

  2. confirm the target .rgm exists

  3. optionally confirm a .dis file exists if you want to load discount assumptions instead of creating/selecting them in-session

  4. choose the intended output lane: - lean ingest: short txt, product columns on, activity columns off - archive/discovery: long txt, broad products/ages

Patchworks Headless Runtime Note

The first successful FEMIC-controlled no-GUI Patchworks seam now has one critical scheduler rule:

  • in the proving-ground BeanShell helper, let Control.waitForIterations(...) own scheduler startup

  • do not call control.resume() immediately before the wait in this headless path

In the current native-Windows proving ground, the explicit resume() caused the old java.lang.IllegalStateException: Not suspended failure. Removing that pre-resume step allows the headless helper to:

  1. load the proving-ground .pin

  2. reach PatchWorks_Init completion

  3. wait one unattended iteration

  4. suspend after the wait

  5. call saveStage(...)

  6. return control cleanly while FEMIC tears down the Patchworks Java tree

FEMIC now also supervises these Windows headless runs directly:

  • success and failure are detected from explicit headless trace/log markers

  • failed runs no longer leave dead console shells for the human to close

  • successful runs are also terminated cleanly after the success marker and saved-stage verification

The proving-ground seam has now advanced one step further:

  • a minimal headless scenario mode can activate product.Yield.managed.Total with a modest annual minimum before the bounded wait/save cycle;

  • the saved proving-ground stage now records that target as active in scenario/targetStatus.csv;

  • scenario/targetSummary.csv contains non-zero managed-yield currents and derived flow.even.product.Yield.managed.Total values; and

  • scenario/schedule.csv is non-empty and contains real managed treatments.

One useful reverse-engineering nuance is now established too:

  • directly activating flow.even.product.Yield.managed.Total changed target state but still left the saved schedule empty;

  • activating the underlying product.Yield.managed.Total target produced the first useful saved headless schedule on the K3Z proving ground.

The next proving-ground refinement is now also established:

  • a real flow.even.* headless smoke works when FEMIC treats it as a two-phase scheduler problem instead of a one-shot target toggle;

  • the helper must first seed the underlying product.Yield.managed.Total target so there is harvest pressure in the final period, then suspend, then activate the companion flow.even.product.Yield.managed.Total target for the second wait phase;

  • proving-ground smoke p49_smoke_20260328q saved a stage where both the underlying harvest target and the even-flow companion were active in scenario/targetStatus.csv, both had non-zero currents in scenario/targetSummary.csv, and scenario/schedule.csv remained non-empty with real managed treatments.

  • the normal CLI/default-target path now proves the same seam too: p49_smoke_20260328r omitted an explicit scenario target and relied on FEMIC’s default product.Yield.managed.Total resolution; the saved stage still recorded both targets as active, and scenario/schedule.csv remained non-empty.

The current closeout-level proving-ground contract is now anchored on the real base K3Z variant:

  • FEMIC’s max-even-flow-smoke mode now defaults to a useful K3Z recipe: default target product.Yield.managed.Total, default iteration budget 100000, seed harvest first on the underlying target, force that base target into linear penalty mode, set its maximum to 200000 in every period at default weight, seed its minimum to 10000 per period, then activate flow.even.product.Yield.managed.Total with minimum = maximum = 0 and minimum = maximum weight = 100 across periods.

  • proving-ground smoke p49_base_closeout_20260328b ran against analysis/base.pin and saved a stage where both the underlying harvest target and the even-flow companion were active, the base target stabilized at roughly 122200 per period inside the 100000..200000 band, the even-flow summary values stayed tightly clustered near zero, and scenario/schedule.csv remained non-empty with real treatments.

Patchworks Registry Operator Note

The current preferred operator surface for shipped Patchworks examples is now registry-backed, not raw-path-first:

  • inspect with instances list / variants list / variants show;

  • inspect grouped download/materialization work with variants materialization-plan;

  • launch with run-variant, run-scenario, or the scenario-set helpers.

Use raw .pin paths only when you are intentionally bypassing the FEMIC registry/operator layer.

Patchworks Track Overlay Note

For FEMIC/Patchworks debugging, do not assume every file under a tracks/ directory is part of the same compile contract.

  • The main track package (for example curves.csv, features.csv, products.csv, treatments.csv, and protoaccounts.csv) is generated by export + Matrix Builder.

  • groups.csv may be a user-edited post-build overlay.

Operational rule:

  • if the task is specifically to swap or edit groups.csv on an already built model surface, treat that as a runtime/user-overlay investigation first;

  • do not assume a rebuild is necessary;

  • do not try to “repair” the situation with ad hoc BeanShell group-expression calls unless the instance documentation explicitly says that is how the overlay is activated.

Host Assumptions

  • Windows is the authoritative host for native Patchworks launch, native VDYP, and ArcGIS Pro fallback workflows.

  • Linux is a supported development host and can run the non-proprietary FEMIC path plus Wine-based Patchworks runtime where configured.

  • patchworks.use_xvfb: true requires xvfb-run on non-Windows hosts.

  • A successful FEMIC preflight validates config and environment shape, not the entire proprietary runtime behavior of third-party tools.

If Something Looks Wrong

  • Wrong files or configs resolved: check Instance and Data Roots.

  • BTC / BatchTIPSY resume blocked unexpectedly: check Stage Boundaries and Canonical Artifacts.

  • Patchworks launch fails after a correct export: check Patchworks runtime prerequisites and host mode before changing export logic.

  • Public-data fallback missing: confirm datalad get completed and FEMIC_EXTERNAL_DATA_ROOT points at real payloads.

See Also