How to Interpret Rebuild Reports and Regressions

Report Location

Each rebuild run writes a machine-readable report:

  • runtime/logs/instance_rebuild_report-<run_id>.json

Start with the run summary at top-level keys:

  • run_id

  • failed

  • outcomes (step execution details)

  • artifact_references

  • metrics

  • invariant_results

  • baseline

  • regression_gate

Step Outcomes

Use outcomes first to confirm command execution order and status.

For each step, review:

  • step_id

  • status (ok or failed)

  • duration_seconds

  • error (if present)

If any required step failed, treat the run as invalid and resolve that before interpreting downstream metrics.

Invariant Results

invariant_results evaluates spec-defined checks against measured metrics.

Each result includes:

  • invariant_id

  • severity (fatal or warn)

  • metric

  • comparator

  • target

  • measured

  • status (pass, warn, fail)

  • remediation

Interpretation rule:

  • Any fatal invariant with status=fail is a hard regression.

  • warn should be reviewed and either remediated or explicitly accepted.

Baseline and Allowlist Diffs

baseline captures structural drift from snapshot comparisons.

Focus fields:

  • status

  • diff (track/XML structural differences)

  • allowlist

  • allowlist_result

Key metrics:

  • metrics.baseline_diff_count

  • metrics.baseline_unexpected_diff_count

  • metrics.baseline_allowlist_match

If unexpected diffs are intentional, update config/rebuild.allowlist.yaml and re-run.

Regression Gate

regression_gate is the final pass/fail policy summary.

Critical fields:

  • step_failure

  • fatal_invariant_failure

  • unexpected_diff_regression

  • baseline_unexpected_diff_threshold

  • baseline_unexpected_diff_count

The run is blocked when any of these are true:

  • step_failure

  • fatal_invariant_failure

  • unexpected_diff_regression

Evidence Trend Drift Across Releases

When evidence is promoted with femic instance promote-evidence (or via femic instance refresh-reference-evidence), the normalized artifact includes trend_drift for release-over-release interpretation.

Key fields:

  • trend_drift.previous_summary

  • trend_drift.warn_increase

  • trend_drift.baseline_diff_increase

  • trend_drift.thresholds.max_warn_increase

  • trend_drift.thresholds.max_baseline_diff_increase

  • trend_drift.warnings

Interpretation rule:

  • Positive warn_increase means more warning-level invariant results than the previously promoted evidence.

  • Positive baseline_diff_increase means additional structural drift versus the previously promoted evidence.

  • Non-empty trend_drift.warnings means configured thresholds were exceeded and should trigger explicit maintainer review before release.

Recommended release workflow:

  1. Refresh evidence with thresholds: femic instance refresh-reference-evidence --reference-root . --max-warn-increase 0 --max-baseline-diff-increase 0

  2. Inspect trend_drift deltas and warning messages.

  3. If drift is intentional, document rationale in roadmap/changelog and update baseline/allowlist artifacts as needed.

  4. Re-run rebuild and evidence refresh until drift warnings are either cleared or explicitly accepted with documented rationale.

Triage Workflow

  1. Confirm step execution succeeded in outcomes.

  2. Resolve any fatal invariant failures.

  3. Review baseline diff scope in baseline.diff.

  4. Distinguish intentional vs unintentional drift: update allowlist only for intentional changes.

  5. Re-run rebuild and confirm regression_gate is clear.

Common Patterns

  • Block/topology join regressions: non-zero block_join_mismatch_count usually indicates model-side join/key inconsistency between blocks.shp and tracks/blocks.csv.

  • Seral-account regressions: low/zero seral_account_count typically indicates XML export or matrix build drift.

  • Baseline drift regressions: unexpected_diff_regression=true means structural changes exceeded accepted allowlist thresholds.