Fetch plan, materialization, and coverage

P2.5 is plan-only. It reads the catalogue, estimates size and time, and records the approval and coverage state. It does not download, order, read a private row, or call a secret store.

The four contracts are:

  • brushline.fetch-plan.v1: the plan-only preview.

  • brushline.materialization.v1: checksums, sizes, licences, and pointers.

  • brushline.coverage.v1: observed, stale, partial, missing, out-of-domain, and excluded states.

  • brushline.gap-report.v1: the next action and the stop/continue decision.

Fetch plan

A FetchPlan has a plan id, a timezone-aware creation time, an AOI reference, mode = plan_only, and one item per requested source or connector. A FetchItem records:

  • kind: source or connector;

  • the source or connector id;

  • the query or approved read scope;

  • the licence or data-sharing record;

  • the access status and classification;

  • the estimated size in megabytes and time in seconds;

  • the estimate basis;

  • the bulk flag and optional approval;

  • the materialization target and pointer;

  • whether the item is required.

The plan copies metadata from the catalogue. This is a snapshot: if the catalogue licence or classification changes later, validate_fetch_plan reports FETCH-DRIFT.

Estimates

The planner uses deterministic planning placeholders when the request does not supply an estimate. They are not measurements.

Access method

Size (MB)

Time (seconds)

Basis

bulk_download

5000

3600

planning default

stac

200

120

planning default

wfs, api, rest

50

60

planning default

wms

10

30

planning default

manual

0

0

manual

unknown

0

0

unknown

A request can override an estimate with FetchEstimate(size_mb=..., seconds=..., basis="user"). The plan total is the sum of its items.

from datetime import UTC, datetime
from brushline.integrations.catalogue import DataCatalogue
from brushline.integrations.fetch import FetchRequest, plan_fetch, validate_fetch_plan

catalogue = DataCatalogue.model_validate_json(
    open("notes/reference/data-catalogue.v1.json", encoding="utf-8").read()
)
plan = plan_fetch(
    catalogue,
    FetchRequest(aoi="aoi.geojson", source_ids=["lidarbc", "vri"]),
    plan_id="plan-1",
    created_utc=datetime(2026, 11, 14, tzinfo=UTC),
)
report = validate_fetch_plan(plan, catalogue)
if not report.is_valid:
    raise SystemExit("\n".join(report.errors))

A bulk request requires an approval with scope="bulk". A missing approval, a single-scope approval, or an incomplete approval fails. A non-bulk public item without an approval is a warning because the plan is still a preview. An unapproved private connector is a FETCH-CONNECTOR error; the owner must approve the catalogue connector first.

Materialization manifest

A MaterializationManifest records what was retrieved, not the payload. A MaterializedItem has an id, source id, status, instance-relative path, SHA-256, size in bytes, timezone-aware retrieval time, licence, pointer, and reason.

Pointers are one of:

  • data_lad:... for a DataLad or annex pointer;

  • annex:... for an annex pointer;

  • public_mirror:... for a public mirror;

  • owner_store:... for a private owner store.

The pointer is metadata. The repository does not commit a payload.

from pathlib import Path
from brushline.integrations.fetch import (
    MaterializationManifest,
    validate_materialization,
    materialization_decision,
)

report = validate_materialization(manifest, Path("."))
if materialization_decision(manifest, Path(".")) == "stop":
    raise SystemExit("\n".join(report.errors))

A materialized item requires a path, checksum, size, licence, retrieval time, and pointer. When a root is supplied, the validator also checks that the path is relative, stays inside the root, exists, and matches the checksum and size. A mismatch fails with MATERIALIZATION-CHECKSUM, MATERIALIZATION-SIZE, or MATERIALIZATION-PATH. The decision is stop for any error.

A missing, partial, or failed item does not need a checksum, but it must have a reason. That is how a gap is recorded without inventing a file.

Coverage states

A CoverageReport has one entry per source and target, where the target is a segment, span, or aoi. The six states are:

State

Rule

observed

Positive coverage and an evidence reference.

stale

Evidence and a vintage are required.

partial

Coverage strictly between 0 and 100, with evidence.

missing

Zero coverage and a reason.

out_of_domain

A reason the source does not apply.

excluded

A reason the source was deliberately excluded.

The report does not impute coverage. An unknown or unavailable source is recorded as missing or out-of-domain, with the evidence and the next action. coverage_counts returns one count per state plus total.

Gap report

build_gap_report turns every non-observed coverage entry into a GapRecord with a next action.

State

Recommended action

missing

Request or acquire the source, or mark the affected segments unknown.

stale

Refresh the source or treat the observation as historical.

partial

Fill the gap with another source, or mark the affected segments partial.

out_of_domain

Exclude the source or use a local calibration.

excluded

Confirm the exclusion and record the decision.

A required missing or out-of-domain source sets decision = stop. A stale, partial, excluded, or optional gap sets continue_with_caveat. A report with no gaps is continue and uses next_action = none. The first gap action becomes next_action. The decision and the action are validated together; a stop with no stopping gap, or a continue with a gap, fails.

Named errors

Prefix

Meaning

FETCH-ID, FETCH-TIME

Duplicate items and a non-timezone-aware plan.

FETCH-KIND, FETCH-LICENCE, FETCH-CLASSIFICATION, FETCH-ACCESS

Item shape and access metadata.

FETCH-ESTIMATE, FETCH-MATERIALIZATION

Negative estimates and pointer/target problems.

FETCH-APPROVAL

Missing, single-scope, incomplete, or unknown approval.

FETCH-SOURCE, FETCH-CONNECTOR, FETCH-DRIFT

Unknown references and catalogue drift.

FETCH-EMPTY

An empty plan (warning).

MATERIALIZATION-*

Status, path, checksum, size, licence, pointer, retrieval time, reason, or empty manifest.

COVERAGE-*

State, evidence, partial range, missing coverage, reason, duplicate, time, or empty report.

GAP-*

Decision, state, action, duplicate, time, or caveat.

What P2.5 does not do

  • It does not download or order data. P2.5 has no network call.

  • It does not create a second downloader. A later stage uses FEMIC’s catalogue and materialization paths.

  • It does not approve a private connector. The owner approval lives on the catalogue connector.

  • It does not place a payload in Git. Pointers and checksums are the record.

  • It does not turn a missing or partial source into a value.