check_notebooks

The notebook drift check. The examples gallery renders each notebook’s stored outputs, and the docs build never executes them, so an example can show output the current code no longer produces and the build will publish it anyway. This script re-runs every example in memory and compares what it produces, cell by cell, with what is committed.

Run it with python scripts/check_notebooks.py. It exits non-zero when anything has drifted, naming the notebook and the cell, and writes nothing. The remedy is --refresh, which executes the notebooks and writes their new outputs back, so the commit that follows is the one the check asks for.

What it cannot compare it says out loud: the environment’s own version banner, a duration or a throughput, and a path are normalised away and counted, and a cell whose output is a live render can be tagged no-drift-check in the notebook itself. It takes several minutes, so it is its own CI job rather than part of scripts/dashboard.py.

Execute the example notebooks and report output that has drifted from what is committed.

The gallery renders the notebooks’ stored outputs and the docs build never executes them (nbsphinx_execute = "never"), so an example can show output the current code no longer produces and the docs will publish it anyway. This script closes that gap: it re-runs each notebook in memory and compares what it produces, cell by cell, with what is committed — reporting the differences and leaving the repository untouched.

What it cannot compare, and what it does about it:

  • The environment’s own banner. Python, ws4 and the optional dependencies report their versions, and a path reports the machine’s layout. Those are normalised away, and the number of times each normalisation fired is printed, so nothing is smoothed silently.

  • Time and rate. A duration or a throughput depends on the host, not on the code.

  • Terminal rendering. ANSI escapes are stripped, and so is the layout: box-drawing characters and column padding are removed, because a table is drawn to the width of its values. A cell whose output is an environment report rather than a result can be tagged no-drift-check in the notebook itself, which is visible in the source rather than hidden here. The count of skipped cells, and of every normalisation applied, is reported.

Everything else is compared exactly. If the check reports drift, either the notebook is stale (re-run it and commit the new outputs) or something changed that the example should have caught.

When the check reports drift, the remedy is to run it with --refresh, which executes the notebooks and writes their new outputs back, so the commit that follows is the one the check would have asked for. Nothing is refreshed silently: the default reports and exits non-zero.

Command-line usage:

python scripts/check_notebooks.py
python scripts/check_notebooks.py --only 070_library_path
python scripts/check_notebooks.py --refresh
check_notebooks.SKIP_TAG = 'no-drift-check'

its output is a live render or an environment report.

Type:

A cell tagged with this is not compared

class check_notebooks.CellOutcome(index: int, skipped: bool, stored: str, fresh: str)[source]

Bases: object

One code cell’s comparison.

index

The cell’s position in the notebook.

Type:

int

skipped

True when the cell is tagged as not comparable.

Type:

bool

stored

The committed output’s normalised text.

Type:

str

fresh

The re-run output’s normalised text.

Type:

str

property drifted: bool

Whether the cell’s output changed, ignoring cells that are not compared.

Returns:

bool – True when the cell is compared and its outputs differ.

check_notebooks.output_text(cell: dict[str, Any]) → str[source]

Return a cell’s combined output as text.

Parameters:

cell (dict) – A notebook cell.

Returns:

str – The stream and text/plain outputs concatenated, or an empty string.

check_notebooks.normalise(text: str, counts: dict[str, int]) → str[source]

Strip rendering and mask what the environment produces, counting every mask.

Parameters:
  • text (str) – The output text.

  • counts (dict of str to int) – Incremented in place per normalisation applied.

Returns:

str – The normalised text: rendering stripped, layout flattened, unstable values masked.

check_notebooks.first_difference(stored: str, fresh: str) → str[source]

Return the first line where two outputs differ, with both sides.

Parameters:
  • stored (str) – The committed text.

  • fresh (str) – The re-run text.

Returns:

str – A one-line description of the first difference.

check_notebooks.check_notebook(path: Path, timeout: int, counts: dict[str, int], refresh: bool = False) → tuple[list[CellOutcome], bool][source]

Re-run one notebook and compare its outputs with the committed ones.

Parameters:
  • path (Path) – The notebook to check.

  • timeout (int) – Seconds to allow for each cell.

  • counts (dict of str to int) – Incremented in place per normalisation applied.

  • refresh (bool) – Whether to write the executed notebook back over the committed one.

Returns:

tuple of (list of CellOutcome, bool) – One outcome per code cell, and whether the notebook executed at all.

check_notebooks.main(argv: list[str] | None = None) → int[source]

Execute the notebooks and report any output that drifted.

Parameters:

argv (list of str, optional) – Command-line arguments.

Returns:

int – Zero when every compared cell matches, one otherwise.