Thesis Evaluation Workflow (Chapter 2)

This guide translates Rosalia Jaffray’s MASc proposal (Chapter 2: “Does FHOPS work, and does it close the gaps we identified in Chapter 1?”) into a repeatable workflow. Use it when preparing the thesis case-study experiments: assemble operational datasets, run FHOPS solvers, evaluate KPIs, and document trade-offs for Chapter 2 narratives.

Context & Goals

• Chapter 1 (literature review) catalogues BC operational-planning gaps (data accessibility, solver transparency, mobilisation awareness, shift-level sequencing).

• Chapter 2 must show that FHOPS can replicate and improve existing planning efforts via reproducible case studies on small-scale operations.

• Deliverables for each case:

  1. Validated FHOPS scenario (data contract-compliant, shift-aware, mobilisation-enabled).

  2. Baseline solver runs (MIP + heuristics) with traceable KPIs.

  3. Trade-off discussion (production vs mobilisation vs utilisation).

  4. Documentation package (commands, telemetry, plots) that can be cited in the thesis.

Pipeline Overview

1. Curate the Case Dataset

   • Start from examples/med42 or examples/large84 as a template.

   • Replace data/*.csv with the case-study inventory (blocks, machines, landings, calendars, production rates, optional road_construction).

   • If the case uses known harvest systems, add a harvest_system_id column per block using IDs from Harvest System Registry.

   • Record provenance notes (tenure, timeline, data sources) in README.md for the case folder.

2. Validate the Scenario

   fhops validate case_study/scenario.yaml
   

   • Fix reported errors (missing references, shift IDs, schema version) before running solvers.

   • Use docs/howto/data_contract.rst if new columns/optional extras are required.

   See also

   fhops.cli.main.validate() – command reference for the scenario validator (includes all CLI options and schema checks).

3. Run Baseline Solvers

   • MIP (reference solution):

     fhops solve-mip case_study/scenario.yaml \
       --out case_study/out/mip_solution.csv \
       --driver auto --time-limit 1800
     

   • Simulated Annealing (fast heuristic):

     fhops solve-heur case_study/scenario.yaml \
       --out case_study/out/sa_solution.csv \
       --iters 12000 --seed 42 \
       --operator-preset explore \
       --telemetry-log case_study/out/sa_telemetry.jsonl \
       --show-operator-stats
     

   • Optional: add fhops solve-ils and fhops solve-tabu for comparative analysis.

   • Document runtime, objective values, and solver settings (Chapter 2 must highlight reproducibility).

   See also

   fhops.cli.main.solve_mip_cmd(), fhops.cli.main.solve_heur_cmd(), fhops.cli.main.solve_ils_cmd(), and fhops.cli.main.solve_tabu_cmd() – each CLI entrypoint documents the complete option set, telemetry hooks, and solver-specific notes.

4. Evaluate KPIs & Mobilisation Spend

   fhops eval-playback case_study/scenario.yaml \
     --assignments case_study/out/mip_solution.csv \
     --shift-out case_study/out/mip_shift.csv \
     --day-out case_study/out/mip_day.csv \
     --summary-md case_study/out/mip_summary.md
   

   • Collect:

     • total_production and completed_blocks.

     • mobilisation_cost and mobilisation_cost_by_machine (Chapter 1 gap: no cost audit).

     • utilisation_ratio (shift/day), makespan.

     • sequencing_violation_* counts (showing constraints hold).

   • Repeat for heuristic schedules. Compare KPI deltas in a table (include convergence rationale).

   See also

   fhops.cli.main.evaluate() for KPI-only summaries and fhops.cli.main.eval_playback() for shift/day playback exports (deterministic or stochastic).

5. Benchmark Trade-offs

   • Run the benchmark harness to quantify solver differences and generate plots:

     fhops bench suite --scenario case_study/scenario.yaml \
       --include-ils --include-tabu --out-dir case_study/bench \
       --time-limit 900 --sa-iters 12000 --tabu-iters 8000 --ils-iters 400
     python scripts/render_benchmark_plots.py case_study/bench/summary.csv \
       --out-dir case_study/bench/plots
     

   • Use the summary CSV/JSON to extract:

     • Objective gap vs best heuristic (evidence of improved solution quality).

     • Runtime ratios (feasibility for small-scale operators).

     • Operator telemetry (link back to mobilization-aware operators when discussing Chapter 1 needs).

   See also

   fhops.cli.benchmarks.bench_suite() – benchmark CLI helper that powers fhops bench suite.

6. Synthesize Chapter 2 Materials

   • Insert KPI tables, mobilisation spend charts, and sequencing status into the chapter draft.

   • Reference appendix artefacts: telemetry logs, shift/day CSVs, benchmark plots.

   • Describe how FHOPS addresses Chapter 1 gaps (e.g., distance-aware mobilisation, shift calendars, open-source reproducibility).

Worked Example (med42 Template)

1. Copy examples/med42 to case_study/; replace data/ with the case inventory.

2. Run validation + solvers as above. Capture commands and seeds for thesis appendices.

3. Highlight insights:

   • kpi_mobilisation_cost decreased by X % when switching from default to mobilisation-focused operator preset.

   • Sequencing violations remained zero, confirming registry accuracy for the target system.

   • Runtime remained under N minutes on lab hardware (relevant for small operations).

4. Discuss trade-offs (production vs mobilisation vs utilisation) with references to Chapter 1 gaps.

Tips & References

• Maintain a case_study/log.md file capturing every command, seed, and data edit (supports Chapter 2 audit trail).

• Include small screenshots or plots (generated from case_study/bench/plots) to visualise KPI movement.

• Cite the proposal folders under tmp/jaffray-rosalia-masc-* when referencing scope and motivation.

• Keep FHOPS docs updated (see notes/sphinx-documentation.md) whenever new thesis-driven workflows appear.