Compact Validation Evidence

Modelwright can package compact validation evidence from generated-model workflow artifacts that already exist on disk. This is useful when a downstream package or CI job needs a small, sanitized summary of a generated-model run without copying raw generated source, raw generated output values, source workbook contents, or full validation reports.

The packaging step is extraction-only. It does not infer a contract, generate Python, execute the generated model, evaluate a workbook, or rerun a benchmark.

Expected Artifacts

By default, Modelwright looks under tmp/generated-model for:

  • inference-result.json from modelwright model infer-contract;

  • generation-result.json from modelwright model generate;

  • generated-values.json from generated-model execution;

  • validation-scenario.json describing declared validation outputs;

  • evaluation-report.json from modelwright validation evaluate.

The default output directory is tmp/validation-evidence/generated-model. The command writes:

  • summary.json for machine-readable automation;

  • summary.md for lightweight human review.

CLI Usage

modelwright validation evidence \
  --evidence-id generated-model \
  --artifact-dir tmp/generated-model \
  --output-dir tmp/validation-evidence/generated-model \
  --json

Use --scenario when the validation scenario JSON lives outside the artifact directory:

modelwright validation evidence \
  --artifact-dir tmp/generated-model \
  --scenario tmp/scenarios/baseline.json \
  --output-dir tmp/validation-evidence/baseline

Missing artifacts are skipped by default so optional evidence workflows can run cleanly in environments where ignored local artifacts have not been restored. Add --require-artifacts when the command should fail instead:

modelwright validation evidence \
  --artifact-dir tmp/generated-model \
  --require-artifacts

Matrix Evidence

FreshForge matrix runs can produce one generated-model workflow per case. Modelwright can aggregate those saved matrix records into a compact matrix-level evidence package:

modelwright validation matrix-evidence \
  --evidence-id generated-model-matrix \
  --matrix-run tmp/matrix-run.json \
  --artifact-root tmp/generated-model-matrix \
  --output-dir tmp/validation-evidence/generated-model-matrix \
  --json

The matrix command is also extraction-only. It does not run FreshForge, generate models, execute generated code, or validate workbooks. It reads a saved FreshForge matrix run or matrix summary and then looks for per-case generated-model artifacts under --artifact-root. For each case, Modelwright checks <artifact-root>/<case-id> and then <artifact-root>/<namespace> when the FreshForge namespace is relative.

The output is again compact:

  • matrix-level summary.json and summary.md;

  • one sanitized row per matrix case;

  • evidence/equivalence status per case;

  • comparable, match, and mismatch counts when available.

Use --require-evidence when every matrix case must have generated-model evidence:

modelwright validation matrix-evidence \
  --matrix-summary tmp/matrix-summary.json \
  --artifact-root tmp/generated-model-matrix \
  --require-evidence

Status Rules

evidence_status is conservative:

  • skipped means one or more expected artifacts are missing and missing artifacts were allowed;

  • incomplete means artifacts exist but explicit comparison counts are unavailable;

  • complete means comparison counts are available and support a pass/fail equivalence status.

equivalence_status is even stricter:

  • pass only when comparable output count equals match count and mismatch count is zero;

  • fail when explicit counts show mismatches or missing matches;

  • incomplete when the comparison counts are unavailable.

A successful generated-model execution by itself is not treated as zero-mismatch evidence. The summary must include explicit comparable-output, match, and mismatch counts from validation evidence.

Python API

from modelwright import (
    extract_matrix_evidence,
    extract_validation_evidence,
    matrix_evidence_paths,
    validation_evidence_paths,
    write_matrix_evidence,
    write_validation_evidence,
)

paths = validation_evidence_paths(
    evidence_id="synthetic-baseline",
    artifact_dir="tmp/generated-model",
    output_dir="tmp/validation-evidence/synthetic-baseline",
)
summary = extract_validation_evidence(paths)
write_validation_evidence(summary, paths)

matrix_paths = matrix_evidence_paths(
    evidence_id="strategy-matrix",
    matrix_run_path="tmp/matrix-run.json",
    artifact_root="tmp/generated-model-matrix",
    output_dir="tmp/validation-evidence/strategy-matrix",
)
matrix_summary = extract_matrix_evidence(matrix_paths)
write_matrix_evidence(matrix_summary, matrix_paths)

Boundary

Compact evidence is meant for publication, CI summaries, and downstream orchestration. It is not a replacement for raw local validation artifacts while debugging. Keep raw workbooks, generated source, generated values, and full validation reports under ignored working directories unless a later roadmap phase explicitly approves a sanitized tracked artifact.