Troubleshooting and Recovery Cookbook
=====================================

Common Issues
-------------

BatchTIPSY input parsing errors
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Likely causes:

- Fixed-width DAT misalignment
- Column-wizard mismatch versus previously validated settings
- Unsupported species/FIZ pairings

Recovery:

1. Keep wizard column mapping constant across runs.
2. Regenerate DAT from FEMIC with unchanged schema.
3. Apply species-code overrides in case/FMU TIPSY YAML config where needed.

Sparse/unstable VDYP fits
^^^^^^^^^^^^^^^^^^^^^^^^^

Likely causes:

- Over-fragmented strata/SI bins
- Too-few stands in fit bins
- Outlier points driving NLLS behavior
- Sampled VDYP polygon/layer temp files drifting out of feature alignment

Recovery:

1. Reduce stratification complexity for small areas.
2. Increase SI-bin collapse aggressiveness or merge bins.
3. Inspect the sampled VDYP temp CSVs and confirm polygon rows and layer rows
   still share the same ``FEATURE_ID`` set before VDYP launch.
4. Apply targeted fit overrides and compare diagnostics only after the batch
   handoff itself looks sane.

VDYP fit-policy config surface
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

- FEMIC-level default per-case/per-FMU smoothing exceptions live in
  ``config/vdyp_fit_policy.yaml``.
- Case-specific overrides can live beside the instance in
  ``<instance_root>/config/vdyp_fit_policy.yaml``.
- Normal precedence is:
  explicit runtime override map -> instance-local YAML -> FEMIC default YAML ->
  code fallback for missing/malformed shared defaults.
- Use the instance-local overlay only for bounded, reviewable exceptions such
  as accepted K3Z curve-specific tail handling. Do not treat it as a shortcut
  for broad global smoothing-policy experiments.

Unexpected cache/resume behavior
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Likely causes:

- Resume with stale checkpoints under changed run profile
- Debug-row mode interacting with cached artifacts

Recovery:

1. Use explicit run IDs per experiment.
2. Disable/clear relevant caches when run semantics changed.
3. Confirm manifest provenance and runtime parameters.

Docs/Pages visibility confusion
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Likely causes:

- GitHub Pages source set to branch/Jekyll instead of Actions artifact
- Deploy job skipped by workflow ``if`` guard

Recovery:

1. Set Pages source to **GitHub Actions**.
2. Ensure deploy guard matches intended trigger (push/manual).
3. Re-run workflow and validate guide URLs directly.

Total Managed OK, Species-wise Managed Empty
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Failure signature:

- ``product.Yield.managed.Total`` reports nonzero behavior in Patchworks.
- Species-wise managed accounts are empty/near-zero unexpectedly.

Deterministic troubleshooting flow:

1. Run account-surface diagnostics and capture JSON evidence:

   .. code-block:: bash

      python -m femic instance account-surface \
        --config config/patchworks.runtime.windows.yaml \
        --output runtime/logs/account_surface-<run_id>.json \
        --instance-root <instance-root>

2. If diagnostics prints ``total OK, species-wise empty``:

   - Inspect ``tracks/products.csv`` and ``tracks/curves.csv`` for missing or
     zero-signal species labels.
   - Inspect matrix manifest ``accounts_sync.excluded_patterns`` for
     over-broad regex exclusions.
3. Re-run deterministic rebuild with Patchworks:

   .. code-block:: bash

      python -m femic instance rebuild \
        --spec config/rebuild.spec.yaml \
        --with-patchworks \
        --instance-root <instance-root>

4. Confirm fatal species policy invariants pass:
   ``required_present``, ``expected_absent``, ``required_nonzero``,
   ``expected_zero``.
5. If still failing, compare against baseline/allowlist diff output in
   ``instance_rebuild_report-<run_id>.json`` and only allowlist intentional
   deltas.

VDYP Gate Rescue Picks Worse Tail Curve
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Failure signature:

- ``tail_blend_selection`` already logs ``tail_blend_rejected`` for a stratum.
- A later ``fit_quality_gate`` event used to rescue back to ``tail_blend``
  anyway because that curve cleared ``early_overshoot_exceeds_gate``.
- Final ``fallback_policy`` then reported ``selected_path = tail_blend`` even
  though the rejected tail fit had materially worse RMSE or tail RMSE.

Deterministic troubleshooting flow:

1. Inspect the stratum's Stage 01a curve events in
   ``vdyp_curve_events-tsaXX-<run_id>.jsonl``.
2. If ``tail_blend_selection`` already rejected the candidate, confirm the
   later rescue path only considers previously accepted candidates
   (``reparameterized_nlls``, ``censored_refit``, accepted ``tail_blend``,
   accepted ``merchantable_floor``, plus ``primary_nlls``).
3. If the selected fit still fails the gate and no accepted candidate resolves
   it, expect ``selected_curve_gate_unresolved`` plus
   ``fallback_policy.selected_path = primary_nlls`` rather than a revived raw
   tail blend.
4. Treat any remaining unresolved gate as a separate fitting-policy problem,
   not as evidence that the rejected tail-blend candidate should be revived.