Telemetry Report Comparison
This notebook demonstrates how to load multiple telemetry tuning reports, merge them, and visualize objective improvements using the helper analyze_tuner_reports.py.
Setup
We’ll reuse the CLI helpers that generated the latest demo_tuner_report.csv under docs/examples/analytics/data/tuner_reports/.
[1]:
from pathlib import Path
import pandas as pd
try:
import altair as alt
except ImportError: # pragma: no cover - optional for CI
alt = None
if alt is not None:
alt.renderers.enable("mimetype")
CANDIDATE_DIRS = [
Path("data/tuner_reports"),
Path("../data/tuner_reports"),
Path("examples/analytics/data/tuner_reports"),
Path("docs/examples/analytics/data/tuner_reports"),
Path.cwd() / "data/tuner_reports",
Path.cwd() / "examples/analytics/data/tuner_reports",
]
DATA_DIR = None
for candidate in CANDIDATE_DIRS:
candidate = candidate.resolve() if not candidate.is_absolute() else candidate
if (candidate / "demo_tuner_report.csv").exists():
DATA_DIR = candidate
break
if DATA_DIR is None:
print("Telemetry reports not found. Generating synthetic sample data instead.")
BASELINE = None
EXPERIMENT = None
else:
BASELINE = DATA_DIR / "demo_tuner_report.csv"
EXPERIMENT = DATA_DIR / "demo_tuner_report.csv" # replace with new report as needed
Load Reports
We can either call the helper script via subprocess or load the CSVs directly for ad-hoc comparisons.
CI executes with a lightweight telemetry sweep, so the required CSV files are generated automatically.
[2]:
if BASELINE is None or EXPERIMENT is None or not BASELINE.exists() or not EXPERIMENT.exists():
print("Telemetry reports not found. Generating synthetic sample data instead.")
sample = pd.DataFrame(
[
{
"algorithm": "random",
"scenario": "SampleScenario",
"best_objective": 6.2,
"mean_objective": 5.9,
"runs": 2,
"label": "baseline",
"machine_costs_summary": "feller_buncher: own=90, op=110, rep=30, usage=10,000h",
"repair_usage_alert": "",
},
{
"algorithm": "random",
"scenario": "SampleScenario",
"best_objective": 7.1,
"mean_objective": 6.6,
"runs": 2,
"label": "experiment",
"machine_costs_summary": "feller_buncher: own=90, op=110, rep=30, usage=10,000h",
"repair_usage_alert": "",
},
{
"algorithm": "grid",
"scenario": "SampleScenario",
"best_objective": 6.8,
"mean_objective": 6.3,
"runs": 2,
"label": "baseline",
"machine_costs_summary": "grapple_skidder: own=70, op=95, rep=32, usage=5,000h",
"repair_usage_alert": "M-GRID",
},
{
"algorithm": "grid",
"scenario": "SampleScenario",
"best_objective": 7.4,
"mean_objective": 6.9,
"runs": 2,
"label": "experiment",
"machine_costs_summary": "grapple_skidder: own=70, op=95, rep=32, usage=15,000h",
"repair_usage_alert": "M-GRID",
},
{
"algorithm": "bayes",
"scenario": "SampleScenario",
"best_objective": 7.0,
"mean_objective": 6.7,
"runs": 1,
"label": "baseline",
"machine_costs_summary": "swing_yarder: own=150, op=230, rep=68, usage=10,000h",
"repair_usage_alert": "",
},
{
"algorithm": "bayes",
"scenario": "SampleScenario",
"best_objective": 7.6,
"mean_objective": 7.2,
"runs": 1,
"label": "experiment",
"machine_costs_summary": "swing_yarder: own=150, op=230, rep=68, usage=20,000h",
"repair_usage_alert": "M-BAYES",
},
]
)
baseline = sample[sample["label"] == "baseline"].drop(columns="label").reset_index(drop=True)
experiment = (
sample[sample["label"] == "experiment"].drop(columns="label").reset_index(drop=True)
)
else:
baseline = pd.read_csv(BASELINE)
experiment = pd.read_csv(EXPERIMENT)
merged = pd.concat(
[baseline.assign(label="baseline"), experiment.assign(label="experiment")], ignore_index=True
)
display(merged)
| algorithm | scenario | runs | best_objective | mean_objective | best_run_id | best_started_at | best_config | summary_best | summary_configurations | summary_updated_at | label | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | bayes | FHOPS MiniToy | 1 | 3.0 | 3.0 | 73a84eb549714c6da52b92ffda8752b6 | 2025-11-11T21:15:55+00:00 | iters=50; operators=(block_insertion:1.9858747... | NaN | NaN | NaN | baseline |
| 1 | grid | FHOPS MiniToy | 1 | 9.0 | 9.0 | f590aa00bb3543debcaebd6bdf288e6b | 2025-11-11T21:15:54+00:00 | iters=50; operators=(block_insertion:0.0, cros... | NaN | NaN | NaN | baseline |
| 2 | random | FHOPS MiniToy | 1 | -6.0 | -6.0 | ad611c7346f84950be678527ae2de434 | 2025-11-11T21:15:53+00:00 | batch_size=2; iters=50; operators=(block_inser... | NaN | NaN | NaN | baseline |
| 3 | bayes | FHOPS MiniToy | 1 | 3.0 | 3.0 | 73a84eb549714c6da52b92ffda8752b6 | 2025-11-11T21:15:55+00:00 | iters=50; operators=(block_insertion:1.9858747... | NaN | NaN | NaN | experiment |
| 4 | grid | FHOPS MiniToy | 1 | 9.0 | 9.0 | f590aa00bb3543debcaebd6bdf288e6b | 2025-11-11T21:15:54+00:00 | iters=50; operators=(block_insertion:0.0, cros... | NaN | NaN | NaN | experiment |
| 5 | random | FHOPS MiniToy | 1 | -6.0 | -6.0 | ad611c7346f84950be678527ae2de434 | 2025-11-11T21:15:53+00:00 | batch_size=2; iters=50; operators=(block_inser... | NaN | NaN | NaN | experiment |
Visualize Best Objectives
Plot the best objective per algorithm to see improvements/deltas.
[3]:
if alt is None or merged.empty:
display(
"Altair not installed or no data available; install `altair` or supply reports to render the chart."
)
else:
chart = (
alt.Chart(merged)
.mark_line(point=True)
.encode(x="label:N", y="best_objective:Q", color="algorithm:N", column="scenario:N")
)
chart
from IPython.display import display
display(chart)
<VegaLite 6 object>
If you see this message, it means the renderer has not been properly enabled
for the frontend that you are using. For more information, see
https://altair-viz.github.io/user_guide/display_frontends.html#troubleshooting
Delta Table
Join baseline and experiment to compute deltas using pandas.
[4]:
comparison = baseline.merge(
experiment, on=["algorithm", "scenario"], suffixes=("_baseline", "_experiment")
)
comparison["best_delta"] = (
comparison["best_objective_experiment"] - comparison["best_objective_baseline"]
)
comparison
[4]:
| algorithm | scenario | runs_baseline | best_objective_baseline | mean_objective_baseline | best_run_id_baseline | best_started_at_baseline | best_config_baseline | summary_best_baseline | summary_configurations_baseline | ... | runs_experiment | best_objective_experiment | mean_objective_experiment | best_run_id_experiment | best_started_at_experiment | best_config_experiment | summary_best_experiment | summary_configurations_experiment | summary_updated_at_experiment | best_delta | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | bayes | FHOPS MiniToy | 1 | 3.0 | 3.0 | 73a84eb549714c6da52b92ffda8752b6 | 2025-11-11T21:15:55+00:00 | iters=50; operators=(block_insertion:1.9858747... | NaN | NaN | ... | 1 | 3.0 | 3.0 | 73a84eb549714c6da52b92ffda8752b6 | 2025-11-11T21:15:55+00:00 | iters=50; operators=(block_insertion:1.9858747... | NaN | NaN | NaN | 0.0 |
| 1 | grid | FHOPS MiniToy | 1 | 9.0 | 9.0 | f590aa00bb3543debcaebd6bdf288e6b | 2025-11-11T21:15:54+00:00 | iters=50; operators=(block_insertion:0.0, cros... | NaN | NaN | ... | 1 | 9.0 | 9.0 | f590aa00bb3543debcaebd6bdf288e6b | 2025-11-11T21:15:54+00:00 | iters=50; operators=(block_insertion:0.0, cros... | NaN | NaN | NaN | 0.0 |
| 2 | random | FHOPS MiniToy | 1 | -6.0 | -6.0 | ad611c7346f84950be678527ae2de434 | 2025-11-11T21:15:53+00:00 | batch_size=2; iters=50; operators=(block_inser... | NaN | NaN | ... | 1 | -6.0 | -6.0 | ad611c7346f84950be678527ae2de434 | 2025-11-11T21:15:53+00:00 | batch_size=2; iters=50; operators=(block_inser... | NaN | NaN | NaN | 0.0 |
3 rows × 21 columns
Machine-Cost Summary Pivot
Telemetry reports now carry a machine_costs_summary column summarizing the owning/operating/repair allowance (and usage bucket) applied during the best run. The snippet below groups the merged report by scenario, label, and machine-cost summary so you can spot assumption drift over time.
[5]:
if "machine_costs_summary" not in merged.columns or merged["machine_costs_summary"].isna().all():
display("`machine_costs_summary` column not present in the loaded reports.")
else:
machine_costs_pivot = (
merged.dropna(subset=["machine_costs_summary"])
.groupby(["scenario", "label", "machine_costs_summary"], dropna=False)["best_objective"]
.agg(["max", "mean", "count"])
.reset_index()
.rename(columns={"max": "best_objective", "mean": "mean_objective", "count": "runs"})
)
display(machine_costs_pivot)
'`machine_costs_summary` column not present in the loaded reports.'
Machine-Cost Trend Chart
Use the grouped chart below to see how machine-cost buckets shift between reports (baseline vs experiment or historical labels).
[ ]:
if alt is None or merged.empty:
display(
"Altair not installed or no data available; install `altair` or supply reports to render the chart."
)
elif "machine_costs_summary" not in merged.columns or merged["machine_costs_summary"].isna().all():
display("`machine_costs_summary` column not present in the loaded reports.")
else:
trend_df = (
merged.dropna(subset=["machine_costs_summary"])
.groupby(["label", "machine_costs_summary"], dropna=False)["algorithm"]
.count()
.reset_index(name="runs")
)
chart = (
alt.Chart(trend_df)
.mark_bar()
.encode(
x="label:N",
y="runs:Q",
color="machine_costs_summary:N",
column="machine_costs_summary:N",
)
)
chart
Use this notebook as a starting point for richer analytics (e.g., trendlines across multiple nightly reports).
Compare Against Historical Summary
If you have downloaded history artefacts, you can re-use the helper script to compare the latest report against prior snapshots. Update HISTORY_DIR to point to your archive.
This notebook focuses on objectives. See tuner_history_analysis.ipynb for multi-metric (KPI) trends.
[6]:
from pathlib import Path
HISTORY_DIR = Path("docs/examples/analytics/data/tuner_reports")
if HISTORY_DIR.exists():
!python scripts/analyze_tuner_reports.py --report latest={BASELINE} --history-dir {HISTORY_DIR} --out-history-markdown tmp/notebook_history.md
display(Path("tmp/notebook_history.md").read_text())
else:
print("History directory not found. Run analyze_tuner_reports.py manually.")
History directory not found. Run analyze_tuner_reports.py manually.
Repair-Usage Alerts
When the KPI layer emits repair_usage_alert, the table below highlights which scenarios/labels deviated from the baseline 10 000 h FPInnovations bucket.
[ ]:
if "repair_usage_alert" not in merged.columns or merged["repair_usage_alert"].isna().all():
display("`repair_usage_alert` column not present in the loaded reports.")
else:
alert_summary = (
merged.dropna(subset=["repair_usage_alert"])
.groupby(["scenario", "label", "repair_usage_alert"], dropna=False)["best_objective"]
.agg(["max", "mean", "count"])
.reset_index()
.rename(columns={"max": "best_objective", "mean": "mean_objective", "count": "runs"})
)
display(alert_summary)