Fetch plan, materialization, and coverage
P2.5 is plan-only. It reads the catalogue, estimates size and time, and records the approval and coverage state. It does not download, order, read a private row, or call a secret store.
The four contracts are:
brushline.fetch-plan.v1: the plan-only preview.brushline.materialization.v1: checksums, sizes, licences, and pointers.brushline.coverage.v1: observed, stale, partial, missing, out-of-domain, and excluded states.brushline.gap-report.v1: the next action and the stop/continue decision.
Fetch plan
A FetchPlan has a plan id, a timezone-aware creation time, an AOI reference, mode =
plan_only, and one item per requested source or connector. A FetchItem records:
kind:sourceorconnector;the source or connector id;
the query or approved read scope;
the licence or data-sharing record;
the access status and classification;
the estimated size in megabytes and time in seconds;
the estimate basis;
the bulk flag and optional approval;
the materialization target and pointer;
whether the item is required.
The plan copies metadata from the catalogue. This is a snapshot: if the catalogue licence or
classification changes later, validate_fetch_plan reports FETCH-DRIFT.
Estimates
The planner uses deterministic planning placeholders when the request does not supply an estimate. They are not measurements.
Access method |
Size (MB) |
Time (seconds) |
Basis |
|---|---|---|---|
|
5000 |
3600 |
planning default |
|
200 |
120 |
planning default |
|
50 |
60 |
planning default |
|
10 |
30 |
planning default |
|
0 |
0 |
manual |
|
0 |
0 |
unknown |
A request can override an estimate with FetchEstimate(size_mb=..., seconds=..., basis="user").
The plan total is the sum of its items.
from datetime import UTC, datetime
from brushline.integrations.catalogue import DataCatalogue
from brushline.integrations.fetch import FetchRequest, plan_fetch, validate_fetch_plan
catalogue = DataCatalogue.model_validate_json(
open("notes/reference/data-catalogue.v1.json", encoding="utf-8").read()
)
plan = plan_fetch(
catalogue,
FetchRequest(aoi="aoi.geojson", source_ids=["lidarbc", "vri"]),
plan_id="plan-1",
created_utc=datetime(2026, 11, 14, tzinfo=UTC),
)
report = validate_fetch_plan(plan, catalogue)
if not report.is_valid:
raise SystemExit("\n".join(report.errors))
A bulk request requires an approval with scope="bulk". A missing approval, a single-scope approval,
or an incomplete approval fails. A non-bulk public item without an approval is a warning because the
plan is still a preview. An unapproved private connector is a FETCH-CONNECTOR error; the owner must
approve the catalogue connector first.
Materialization manifest
A MaterializationManifest records what was retrieved, not the payload. A MaterializedItem has an
id, source id, status, instance-relative path, SHA-256, size in bytes, timezone-aware retrieval time,
licence, pointer, and reason.
Pointers are one of:
data_lad:...for a DataLad or annex pointer;annex:...for an annex pointer;public_mirror:...for a public mirror;owner_store:...for a private owner store.
The pointer is metadata. The repository does not commit a payload.
from pathlib import Path
from brushline.integrations.fetch import (
MaterializationManifest,
validate_materialization,
materialization_decision,
)
report = validate_materialization(manifest, Path("."))
if materialization_decision(manifest, Path(".")) == "stop":
raise SystemExit("\n".join(report.errors))
A materialized item requires a path, checksum, size, licence, retrieval time, and pointer. When a root
is supplied, the validator also checks that the path is relative, stays inside the root, exists, and
matches the checksum and size. A mismatch fails with MATERIALIZATION-CHECKSUM,
MATERIALIZATION-SIZE, or MATERIALIZATION-PATH. The decision is stop for any error.
A missing, partial, or failed item does not need a checksum, but it must have a reason. That is how a gap is recorded without inventing a file.
Coverage states
A CoverageReport has one entry per source and target, where the target is a segment, span, or
aoi. The six states are:
State |
Rule |
|---|---|
|
Positive coverage and an evidence reference. |
|
Evidence and a vintage are required. |
|
Coverage strictly between 0 and 100, with evidence. |
|
Zero coverage and a reason. |
|
A reason the source does not apply. |
|
A reason the source was deliberately excluded. |
The report does not impute coverage. An unknown or unavailable source is recorded as missing or
out-of-domain, with the evidence and the next action. coverage_counts returns one count per state
plus total.
Gap report
build_gap_report turns every non-observed coverage entry into a GapRecord with a next action.
State |
Recommended action |
|---|---|
|
Request or acquire the source, or mark the affected segments unknown. |
|
Refresh the source or treat the observation as historical. |
|
Fill the gap with another source, or mark the affected segments partial. |
|
Exclude the source or use a local calibration. |
|
Confirm the exclusion and record the decision. |
A required missing or out-of-domain source sets decision = stop. A stale, partial, excluded, or
optional gap sets continue_with_caveat. A report with no gaps is continue and uses
next_action = none. The first gap action becomes next_action. The decision and the action are
validated together; a stop with no stopping gap, or a continue with a gap, fails.
Named errors
Prefix |
Meaning |
|---|---|
|
Duplicate items and a non-timezone-aware plan. |
|
Item shape and access metadata. |
|
Negative estimates and pointer/target problems. |
|
Missing, single-scope, incomplete, or unknown approval. |
|
Unknown references and catalogue drift. |
|
An empty plan (warning). |
|
Status, path, checksum, size, licence, pointer, retrieval time, reason, or empty manifest. |
|
State, evidence, partial range, missing coverage, reason, duplicate, time, or empty report. |
|
Decision, state, action, duplicate, time, or caveat. |
What P2.5 does not do
It does not download or order data. P2.5 has no network call.
It does not create a second downloader. A later stage uses FEMIC’s catalogue and materialization paths.
It does not approve a private connector. The owner approval lives on the catalogue connector.
It does not place a payload in Git. Pointers and checksums are the record.
It does not turn a missing or partial source into a value.