Stage 01b: Post-TIPSY Integration and Comparison ================================================ Scope ----- Stage 01b ingests BTC/BatchTIPSY output, aligns managed-track curves with VDYP untreated-track curves, and writes downstream bundle artifacts. Required Input -------------- - ``03_input-.csv`` generated by Stage 01a (canonical BTC input). - ``tipsy_params_tsa.xlsx`` generated by Stage 01a (human-readable mirror). - ``04_output-.csv`` and ``04_error-.csv`` generated by unattended BTC. - Corresponding Stage 01a outputs (AUs, VDYP curves, handoff metadata). Legacy compatibility seam: - ``02_input-.dat`` and ``04_output-.out`` remain supported only for older manual BatchTIPSY workflows. They are no longer the default FEMIC contract. Freshness guard --------------- Stage 01b treats ``03_input-.csv`` as canonical on the default BTC seam and uses input-content fingerprints for stale detection: - If the current input hash differs from the hash previously paired with ``04_output-.csv``, Stage 01b fails fast. - If no hash sidecar exists yet, Stage 01b falls back to input-vs-output mtime and then performs an input/output coherence check (AU/table coverage): - coherent pairs warn-and-continue by default (development-friendly mode), - incoherent pairs still fail fast. - Set ``FEMIC_STRICT_TIPSY_TIMESTAMP_MISMATCH=1`` to escalate coherent timestamp mismatch back to a hard error. - ``tipsy_params_tsa.xlsx`` is only a human-readable mirror and is not authoritative for freshness. - Use ``FEMIC_ALLOW_STALE_TIPSY_OUTPUT=1`` only for explicit debugging on the legacy DAT/OUT seam. Legacy DAT/OUT seam note: - When resuming from ``02_input-.dat`` + ``04_output-.out``, Stage 01b keeps the older DAT-first freshness behavior for compatibility. Managed-curve mode note: when ``managed_curve_mode != tipsy`` (for example ``vdyp_transform``), Stage 01b skips the BatchTIPSY freshness guard because the managed curve path does not depend on refreshed BatchTIPSY output. Core Responsibilities --------------------- - Parse TIPSY output tables into model-ready curve points. - Align/compare managed and untreated curves by AU. - Generate per-AU comparison plots for QA and tuning. - Publish updated bundle tables for export stages. - Apply any configured post-TIPSY yield assumptions before writing the final bundle tables. For TSA29, this includes the later section ``7.1.5`` rule that removes broadleaf volume from conifer-leading untreated curves while leaving THLB step ``015`` as the separate broadleaf-leading area exclusion. Optional TSR Yield Assumptions ------------------------------ Stage 01b post-TIPSY bundling can apply a narrow instance-local TSR ``yield_assumptions.yaml`` file before writing ``data/model_input_bundle``. - CLI seam: - ``femic tsa post-tipsy --yield-assumptions-path `` - ``femic tsa btc-post-tipsy --yield-assumptions-path `` - Run-profile seam: - ``modes.yield_assumptions_path: config/tsr/yield_assumptions.yaml`` - Default behavior: - if ``config/tsr/yield_assumptions.yaml`` exists under the instance root, the post-TIPSY workflow uses it automatically; - otherwise no later yield-assumption adjustment is applied. TSA29 broadleaf rule split: - THLB step ``015`` still removes **broadleaf-leading stands** from THLB area. - TSA29 section ``7.1.5`` is a later bundle/yield assumption: - for **conifer-leading untreated** AUs only, - remove the broadleaf share from the untreated total curve, - zero untreated broadleaf species-proportion sidecars, and - renormalize the remaining untreated conifer sidecars to sum to ``1.0``. Treated curves are left unchanged. Interpretation Guide -------------------- - Expect coherent relative behavior between untreated and managed trajectories according to scenario assumptions. - Identify AUs where managed curves are implausibly weak/strong and route them to parameter tuning or managed-curve transform workflows. Exit Criteria ------------- - Non-empty managed and untreated curve sets for all intended AUs. - QA plots generated without parse/fit failures. - Bundle tables ready for Patchworks/Woodstock export. Primary Legacy Notebook Coverage -------------------------------- Derived from ``01b_run-tsa.ipynb`` plus parent orchestration notes in ``00_data-prep.ipynb``. Linux Source-Checkout Prerequisites ----------------------------------- Before running Stage 01b from a fresh Linux source checkout, ensure: .. code-block:: bash python -m pip install -r requirements-dev.txt git submodule update --init --recursive git -C external/femic-public-data annex enableremote arbutus-s3 datalad get -r external/femic-public-data/data export FEMIC_EXTERNAL_DATA_ROOT=$PWD/external/femic-public-data/data Known-Good Windows K3Z Resume Path ---------------------------------- Once unattended BTC has refreshed the canonical output files: - ``external/femic-k3z-instance/data/04_output-tsak3z.csv`` - ``external/femic-k3z-instance/data/04_error-tsak3z.csv`` resume **only** the downstream path: .. code-block:: powershell $env:FEMIC_EXTERNAL_DATA_ROOT="$PWD\external\femic-public-data\data" python -m femic tsa btc-post-tipsy --instance-root external/femic-k3z-instance --run-config config/run_profile.k3z.yaml --tsa k3z --run-id k3z_windows_cleanstart The command group and flag still use the legacy ``tsa`` naming seam for compatibility. Read them generically as the selected FMU/code target. Then continue into Patchworks export/build as needed, for example: .. code-block:: powershell python -m femic patchworks build-blocks --instance-root external/femic-k3z-instance --config config/patchworks.runtime.windows.yaml python -m femic patchworks matrix-build --instance-root external/femic-k3z-instance --config config/patchworks.runtime.windows.yaml --run-id k3z_windows_cleanstart This is the intended Windows clean-start/restart boundary for K3Z: upstream Stage 01a writes the BTC handoff, unattended ``/TSR`` runs under FEMIC, then ``tsa btc-post-tipsy`` resumes downstream without rerunning the expensive GIS/VDYP work.