Telemetry Ops Runbook

This runbook codifies the “always-on” telemetry workflow for FHOPS: how we capture tuning runs, keep the weekly analytics notebooks current, maintain the telemetry store, and publish artefacts (CI + GitHub Pages). Use it as the operational checklist for Phase 4.

Weekly Notebook Pipeline

  1. Schedule

    • CI: primary workflows run on every push; the “full analytics” notebook suite should run at least weekly via a scheduled GitHub Actions workflow (e.g., Sunday 03:00 UTC) to keep stochastic KPIs fresh.

    • Local smoke: when large changes land, run notebooks (scripts/run_analytics_notebooks.py –light) before opening a PR.

  2. Commands

    # Light CI run (already wired in workflows)
python scripts/run_analytics_notebooks.py --out docs/examples/analytics --light

# Full local run before the weekly upload (no --light)
python scripts/run_analytics_notebooks.py --out tmp/analytics_full

    • Ensure FHOPS_ANALYTICS_LIGHT=1 is unset for the weekly full run so stochastic ensembles execute at full sample counts. Record duration in the workflow logs.

  3. Artefact handling

    • CI stores analytics-notebooks (rendered HTML/MD) and telemetry-report (CSV/MD/HTML) as artifacts.

    • Weekly workflow should download the latest telemetry-report artifact, append it to docs/examples/analytics/data/tuner_reports/, and re-run the history script (see below).

    • If GitHub Pages is enabled, publish the refreshed HTML files (history_summary.html, history_delta.html, analytics notebooks) to the Pages branch.

Telemetry Store Maintenance

  • Location – default telemetry directory: telemetry/ (JSONL, SQLite, history snapshots).

  • Rotation policy

    • Keep the latest 4 weeks of JSONL/SQLite telemetry in the repo (rename older files to .archive/ or move to long-term storage to prevent bloat).

    • Archive large step logs (telemetry/steps/*.jsonl) after generating summary reports.

    See also

    fhops.cli.telemetry.prune() – CLI helper for trimming run logs and deleting aligned step files.

  • Vacuum & integrity

    sqlite3 telemetry/runs.sqlite "PRAGMA optimize; VACUUM;"
sqlite3 telemetry/runs.sqlite "PRAGMA quick_check;"

    Run weekly (after the full notebook workflow) to keep the SQLite store compact and healthy.

  • Locks/contention

    • Avoid simultaneous writes to the same JSONL/SQLite file (especially when running multiple tuning commands concurrently). Use separate --telemetry-log paths per developer or configure the CI job to serialize writes.

    • If a workflow crashes mid-write, delete the partially written row from the JSONL file and re-run. SQLite should remain consistent thanks to WAL journaling, but PRAGMA quick_check will confirm.

Automation Checklist

  1. CI (per push)

    • fhops tune-grid / fhops tune-random on smoke scenarios (tiny7, med42).

    • fhops telemetry report → telemetry-report artifact with tuner_report.{csv,md}, history/.

    • scripts/analyze_tuner_reports.py –history-dir to regenerate history_summary.* and history_delta.*.

  2. Weekly full run

    • Execute notebooks without –light.

    • Run the longer tuning bundles (e.g., fhops tune-bayes –bundle synthetic –trials 6).

    • Rebuild history charts and push to GitHub Pages.

    • Commit updated history CSV/MD files if the repo tracks canonical snapshots.

  3. Manual interventions

    • When telemetry schema fields change, update scripts/analyze_tuner_reports.py and rerun the history command so older snapshots stay compatible (document the migration in CHANGE_LOG.md).

    • If GitHub Pages fails to update, inspect the Pages workflow logs and check for large HTML files (GitHub rejects >100 MB). Trim sample counts or compress images as needed.

See also

fhops.cli.main.solve_heur_cmd(), fhops.cli.main.solve_ils_cmd(), and fhops.cli.main.solve_tabu_cmd() – solver entrypoints that emit --telemetry-log metadata used by this runbook. fhops.cli.main.tune_random_cli(), fhops.cli.main.tune_grid_cli(), and fhops.cli.main.tune_bayes_cli() – tuning orchestrators responsible for the weekly telemetry bundles. fhops.cli.benchmarks.bench_suite() – benchmarking command that feeds KPI comparisons into the telemetry store. fhops.cli.telemetry.report() – generates CSV/Markdown summaries from telemetry/runs.sqlite for publication.

Runbook Template (per week)

Week of YYYY-MM-DD
-------------------
[ ] Run full notebooks (duration: __)
[ ] Run extended tuning bundles (commands + seeds)
[ ] Generate telemetry report / history / delta
[ ] Upload analytics + telemetry artefacts (CI + Pages)
[ ] Vacuum telemetry/runs.sqlite
[ ] Archive >4-week-old telemetry logs
[ ] Update README badges if history chart moved

Keep a copy of this checklist in docs/howto/telemetry_ops.rst (or a shared issue template) and tick each box when the weekly workflow completes.

References

  • Telemetry-Driven Tuning – CLI usage, report interpretation, dashboard links.

  • .github/workflows/*telemetry*.yml – scheduled workflow definitions (tune, notebooks, Pages).

  • scripts/run_analytics_notebooks.py – executed commands for notebooks.

  • scripts/analyze_tuner_reports.py – history/delta generation.