Telemetry History Analysis

This notebook demonstrates how to consume the history_summary.* artefacts produced by continuous integration and extend the analysis locally.

Load History Snapshot

The telemetry-report artefact includes history_summary.csv and history_summary.html. We’ll load the CSV and pivot the data for quick analysis.

[1]:

from pathlib import Path

import pandas as pd

try:
    import altair as alt
except ImportError:
    alt = None

HISTORY_CSV = Path("docs/examples/analytics/data/tuner_reports/history_summary.csv")
if HISTORY_CSV.exists():
    history = pd.read_csv(HISTORY_CSV)
else:
    # Fallback synthetic history if CI artefact not present locally
    history = pd.DataFrame(
        [
            {
                "snapshot": "2024-11-10T000000Z",
                "algorithm": "random",
                "scenario": "SampleScenario",
                "best_objective": 6.2,
                "mean_objective": 6.0,
                "runs": 2,
            },
            {
                "snapshot": "2024-11-11T000000Z",
                "algorithm": "random",
                "scenario": "SampleScenario",
                "best_objective": 6.8,
                "mean_objective": 6.4,
                "runs": 2,
            },
            {
                "snapshot": "2024-11-12T000000Z",
                "algorithm": "grid",
                "scenario": "SampleScenario",
                "best_objective": 7.1,
                "mean_objective": 6.8,
                "runs": 2,
            },
            {
                "snapshot": "2024-11-13T000000Z",
                "algorithm": "grid",
                "scenario": "SampleScenario",
                "best_objective": 7.5,
                "mean_objective": 7.1,
                "runs": 2,
            },
        ]
    )
history

[1]:

	snapshot	algorithm	scenario	best_objective	mean_objective	runs
0	2024-11-10T000000Z	random	SampleScenario	6.2	6.0	2
1	2024-11-11T000000Z	random	SampleScenario	6.8	6.4	2
2	2024-11-12T000000Z	grid	SampleScenario	7.1	6.8	2
3	2024-11-13T000000Z	grid	SampleScenario	7.5	7.1	2

Compute Deltas

We can compute day-over-day improvements for each algorithm/scenario pair.

[2]:

history = history.sort_values(["scenario", "algorithm", "snapshot"]).reset_index(drop=True)
history["best_delta"] = history.groupby(["scenario", "algorithm"])["best_objective"].diff()
history

[2]:

	snapshot	algorithm	scenario	best_objective	mean_objective	runs	best_delta
0	2024-11-12T000000Z	grid	SampleScenario	7.1	6.8	2	NaN
1	2024-11-13T000000Z	grid	SampleScenario	7.5	7.1	2	0.4
2	2024-11-10T000000Z	random	SampleScenario	6.2	6.0	2	NaN
3	2024-11-11T000000Z	random	SampleScenario	6.8	6.4	2	0.6

Visualize Trends

Use Altair to plot the best objective trend per scenario.

[3]:

if alt is None or history.empty:
    display(
        "Altair not installed or no data available; install `altair` or supply reports to render the chart."
    )
else:
    value_columns = ["best_objective"]
    value_columns += [
        c
        for c in [
            "best_total_production",
            "best_mobilisation_cost",
            "best_utilisation_ratio_shift",
            "best_utilisation_ratio_day",
            "best_downtime_hours",
            "best_weather_severity",
        ]
        if c in history.columns
    ]
    chart_df = history.melt(
        id_vars=["snapshot", "algorithm", "scenario"],
        value_vars=value_columns,
        var_name="metric",
        value_name="value",
    )
    metric_labels = {
        "best_objective": "Best Objective",
        "best_total_production": "Total Production",
        "best_mobilisation_cost": "Mobilisation Cost",
        "best_utilisation_ratio_shift": "Utilisation (Shift)",
        "best_utilisation_ratio_day": "Utilisation (Day)",
        "best_downtime_hours": "Downtime Hours",
        "best_weather_severity": "Weather Severity",
    }
    chart_df["metric_label"] = chart_df["metric"].map(metric_labels).fillna(chart_df["metric"])
    chart = (
        alt.Chart(chart_df)
        .mark_line(point=True)
        .encode(
            x="snapshot:N",
            y="value:Q",
            color="algorithm:N",
            column=alt.Column("metric_label:N", title="Metric"),
            row="scenario:N",
        )
    )
    chart

Next Steps

Merge med42 or regression scenario reports into the history before comparing.
Use run-level telemetry (from runs.sqlite) to inspect KPI deltas for each snapshot.