brushline.integrations.fetch
The plan-only fetch preview, materialization manifest, coverage states, and gap report: models, the planner, checksum and path checks, the coverage validator, and the stop/continue decision.
Plan-only fetch, materialization, coverage, and gap contracts.
The planner reads the catalogue and writes a plan; it never fetches. The materialization manifest records pointers and checksums, not payloads. The coverage report preserves observed, stale, partial, missing, out-of-domain, and excluded states instead of imputing a value. The gap report turns a non-observed state into a next action and a stop/continue decision.
DataLad, annex, public-mirror, and owner-store pointers are metadata strings. This module does not call a downloader, a remote service, or a secret store.
- class brushline.integrations.fetch.CoverageEntry(*, source_id: Annotated[str, MinLen(min_length=1)], target_id: Annotated[str, MinLen(min_length=1)] = 'aoi', state: str = 'missing', coverage_pct: Annotated[float, Ge(ge=0), Le(le=100)] = 0, vintage: str = '', evidence: str = '', reason: str = '', required: bool = True)[source]
Bases:
BaseModelOne source coverage state for one target.
- Parameters:
source_id (str) – Source or connector id.
target_id (str, optional) – Segment, span, or
aoitarget. Defaults toaoi.state (str, optional) –
observed,stale,partial,missing,out_of_domain, orexcluded.coverage_pct (float, optional) – Coverage percentage from 0 to 100.
vintage (str, optional) – Source vintage for the observation.
evidence (str, optional) – Evidence reference supporting the state.
reason (str, optional) – Reason for a non-observed state.
required (bool, optional) – Whether the gap stops the build. Defaults to
True.
- model_config = {'extra': 'forbid', 'frozen': True}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- class brushline.integrations.fetch.CoverageReport(*, schema_version: ~typing.Literal['brushline.coverage.v1'] = 'brushline.coverage.v1', plan_id: ~typing.Annotated[str, ~annotated_types.MinLen(min_length=1)], created_utc: ~datetime.datetime, aoi: ~typing.Annotated[str, ~annotated_types.MinLen(min_length=1)], entries: list[~brushline.integrations.fetch.CoverageEntry] = <factory>, notes: str = '')[source]
Bases:
BaseModelThe coverage states for one fetch plan.
- Parameters:
schema_version ({"brushline.coverage.v1"}) – Report schema version.
plan_id (str) – Fetch plan identifier.
created_utc (datetime) – Timezone-aware creation time.
aoi (str) – Area-of-interest reference.
entries (list of CoverageEntry) – Coverage entries.
notes (str, optional) – Human notes.
- model_config = {'extra': 'forbid'}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- class brushline.integrations.fetch.FetchApproval(*, approved_by: str = '', approved_utc: datetime, scope: str = 'single', record: str = '')[source]
Bases:
BaseModelA user approval recorded on a fetch plan.
- Parameters:
approved_by (str, optional) – Person or role that approved the fetch.
approved_utc (datetime) – Timezone-aware approval time.
scope (str, optional) –
singleorbulk.record (str, optional) – Reference to the approval record.
- model_config = {'extra': 'forbid', 'frozen': True}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- class brushline.integrations.fetch.FetchEstimate(*, size_mb: Annotated[float, Ge(ge=0)] = 0, seconds: Annotated[float, Ge(ge=0)] = 0, basis: str = 'planning_default')[source]
Bases:
BaseModelA planning estimate for one source or connector.
- Parameters:
size_mb (float, optional) – Estimated transfer size in megabytes. Defaults to
0.seconds (float, optional) – Estimated transfer time in seconds. Defaults to
0.basis (str, optional) –
planning_default,user,manual,unknown, orowner_side.
- model_config = {'extra': 'forbid', 'frozen': True}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- class brushline.integrations.fetch.FetchItem(*, kind: str, reference_id: Annotated[str, MinLen(min_length=1)], query: str = '', licence: str = '', access_status: str = 'unknown', classification: str = 'unknown', estimated_size_mb: Annotated[float, Ge(ge=0)] = 0, estimated_seconds: Annotated[float, Ge(ge=0)] = 0, estimate_basis: str = 'unknown', bulk: bool = False, approval: FetchApproval | None = None, materialization: str = 'not_materialized', materialization_pointer: str = '', required: bool = True)[source]
Bases:
BaseModelOne source or connector in a fetch plan.
- Parameters:
kind (str) –
sourceorconnector.reference_id (str) – Source or connector id.
query (str) – Query or read scope.
licence (str) – Licence or data-sharing record.
access_status (str) – Catalogue access status, or
blockedfor an unapproved connector.classification (str) – Source or connector classification.
estimated_size_mb (float, optional) – Estimated transfer size in megabytes.
estimated_seconds (float, optional) – Estimated transfer time in seconds.
estimate_basis (str, optional) – Where the estimate came from.
bulk (bool, optional) – Whether the item is part of a bulk request.
approval (FetchApproval or None, optional) – Recorded approval for the item.
materialization (str, optional) –
data_lad,annex,public_mirror,owner_store, ornot_materialized.materialization_pointer (str, optional) – Metadata pointer, never a payload path.
required (bool, optional) – Whether a missing source stops the build. Defaults to
True.
- model_config = {'extra': 'forbid', 'frozen': True}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- class brushline.integrations.fetch.FetchPlan(*, schema_version: ~typing.Literal['brushline.fetch-plan.v1'] = 'brushline.fetch-plan.v1', plan_id: ~typing.Annotated[str, ~annotated_types.MinLen(min_length=1)], created_utc: ~datetime.datetime, aoi: ~typing.Annotated[str, ~annotated_types.MinLen(min_length=1)], mode: ~typing.Literal['plan_only'] = 'plan_only', items: list[~brushline.integrations.fetch.FetchItem] = <factory>, notes: str = '')[source]
Bases:
BaseModelA plan-only fetch preview.
- Parameters:
schema_version ({"brushline.fetch-plan.v1"}) – Plan schema version.
plan_id (str) – Stable plan identifier.
created_utc (datetime) – Timezone-aware creation time.
aoi (str) – Area-of-interest reference.
mode ({"plan_only"}) – The planner never downloads.
items (list of FetchItem) – Sources and connectors in plan order.
notes (str, optional) – Human notes.
- model_config = {'extra': 'forbid'}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- property total_size_mb: float
Return the estimated total transfer size.
- Returns:
float – Sum of item size estimates in megabytes.
- property total_seconds: float
Return the estimated total transfer time.
- Returns:
float – Sum of item time estimates in seconds.
- property network_items: int
Return the number of items with a materialization pointer.
- Returns:
int – Count of items that are not
not_materialized.
- class brushline.integrations.fetch.FetchRequest(*, aoi: ~typing.Annotated[str, ~annotated_types.MinLen(min_length=1)], source_ids: list[str] = <factory>, connector_ids: list[str] = <factory>, queries: dict[str, str] = <factory>, estimates: dict[str, ~brushline.integrations.fetch.FetchEstimate] = <factory>, optional_ids: list[str] = <factory>, bulk: bool = False, approvals: dict[str, ~brushline.integrations.fetch.FetchApproval] = <factory>)[source]
Bases:
BaseModelA request to plan a fetch without executing it.
- Parameters:
aoi (str) – Area of interest reference, such as an instance-relative AOI path.
source_ids (list of str, optional) – Public source ids to plan.
connector_ids (list of str, optional) – Private connector ids to plan.
queries (dict of str to str, optional) – Query overrides keyed by source or connector id.
estimates (dict of str to FetchEstimate, optional) – User estimates keyed by source or connector id.
optional_ids (list of str, optional) – Ids that may be missing without stopping the build.
bulk (bool, optional) – Whether the request is a bulk fetch. Defaults to
False.approvals (dict of str to FetchApproval, optional) – Approvals keyed by source or connector id.
- model_config = {'extra': 'forbid', 'frozen': True}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- class brushline.integrations.fetch.GapRecord(*, gap_id: Annotated[str, MinLen(min_length=1)], source_id: Annotated[str, MinLen(min_length=1)], target_id: Annotated[str, MinLen(min_length=1)], state: str, decision: str, recommended_action: Annotated[str, MinLen(min_length=1)], reason: str = '')[source]
Bases:
BaseModelOne non-observed coverage gap and its next action.
- Parameters:
gap_id (str) – Stable gap identifier.
source_id (str) – Source or connector id.
target_id (str) – Segment, span, or
aoitarget.state (str) – Coverage state that created the gap.
decision (str) –
stoporcontinue_with_caveat.recommended_action (str) – Next action.
reason (str) – Why the gap exists.
- model_config = {'extra': 'forbid', 'frozen': True}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- class brushline.integrations.fetch.GapReport(*, schema_version: ~typing.Literal['brushline.gap-report.v1'] = 'brushline.gap-report.v1', plan_id: ~typing.Annotated[str, ~annotated_types.MinLen(min_length=1)], created_utc: ~datetime.datetime, decision: str = 'continue', next_action: str = 'none', gaps: list[~brushline.integrations.fetch.GapRecord] = <factory>)[source]
Bases:
BaseModelThe gap summary and stop/continue decision.
- Parameters:
schema_version ({"brushline.gap-report.v1"}) – Report schema version.
plan_id (str) – Fetch plan identifier.
created_utc (datetime) – Timezone-aware creation time.
decision (str) –
stop,continue, orcontinue_with_caveat.next_action (str) – The first recommended action, or
none.gaps (list of GapRecord) – Non-observed coverage entries.
- model_config = {'extra': 'forbid'}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- class brushline.integrations.fetch.MaterializationManifest(*, schema_version: ~typing.Literal['brushline.materialization.v1'] = 'brushline.materialization.v1', plan_id: ~typing.Annotated[str, ~annotated_types.MinLen(min_length=1)], created_utc: ~datetime.datetime, items: list[~brushline.integrations.fetch.MaterializedItem] = <factory>, notes: str = '')[source]
Bases:
BaseModelThe materialization record for one fetch plan.
- Parameters:
schema_version ({"brushline.materialization.v1"}) – Manifest schema version.
plan_id (str) – Fetch plan identifier.
created_utc (datetime) – Timezone-aware creation time.
items (list of MaterializedItem) – Materialized and missing items.
notes (str, optional) – Human notes.
- model_config = {'extra': 'forbid'}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- class brushline.integrations.fetch.MaterializedItem(*, id: Annotated[str, MinLen(min_length=1)], source_id: Annotated[str, MinLen(min_length=1)], status: str = 'materialized', path: str = '', sha256: str = '', size_bytes: Annotated[int, Ge(ge=0)] = 0, retrieved_utc: datetime | None = None, licence: str = '', pointer: str = '', reason: str = '')[source]
Bases:
BaseModelOne materialized or missing source.
- Parameters:
id (str) – Stable item identifier.
source_id (str) – Source or connector id.
status (str, optional) –
materialized,missing,partial, orfailed.path (str, optional) – Instance-relative path. Empty for a missing item.
sha256 (str, optional) – Expected SHA-256 of the file.
size_bytes (int, optional) – Expected file size in bytes.
retrieved_utc (datetime or None, optional) – Timezone-aware retrieval time.
licence (str, optional) – Licence or data-sharing record.
pointer (str, optional) –
data_lad:,annex:,public_mirror:, orowner_store:pointer.reason (str, optional) – Why the item is missing, partial, or failed.
- model_config = {'extra': 'forbid', 'frozen': True}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- brushline.integrations.fetch.build_gap_report(coverage: CoverageReport) GapReport[source]
Turn non-observed coverage entries into gaps and a decision.
A required missing or out-of-domain source stops the build. A stale, partial, excluded, or optional gap continues with a caveat. The gap report never imputes coverage.
- Parameters:
coverage (CoverageReport) – Coverage report.
- Returns:
GapReport – Gaps, the overall decision, and the next action.
- brushline.integrations.fetch.coverage_counts(report: CoverageReport) dict[str, int | float][source]
Count coverage states in a report.
- Parameters:
report (CoverageReport) – Coverage report.
- Returns:
dict of str to int or float – Counts for each state plus
total.
- brushline.integrations.fetch.default_estimate(access_method: str) FetchEstimate[source]
Return the deterministic planning estimate for an access method.
These are planning placeholders, not measurements. A user estimate overrides them in a
FetchRequest.- Parameters:
access_method (str) – Catalogue access method.
- Returns:
FetchEstimate – Size, time, and basis.
- brushline.integrations.fetch.file_sha256(path: Path) str[source]
Return the SHA-256 digest of a file.
- Parameters:
path (Path) – File to hash.
- Returns:
str – Lowercase hexadecimal SHA-256 digest.
- brushline.integrations.fetch.materialization_decision(manifest: MaterializationManifest, root: Path | None = None) str[source]
Return the materialization decision.
- Parameters:
manifest (MaterializationManifest) – Manifest to validate.
root (Path, optional) – Workspace root used for file checks.
- Returns:
str –
continuewhen the manifest is valid, otherwisestop.
- brushline.integrations.fetch.plan_fetch(catalogue: DataCatalogue, request: FetchRequest, *, plan_id: str = 'fetch-plan', created_utc: datetime | None = None) FetchPlan[source]
Build a plan-only fetch preview from the catalogue.
The function resolves source and connector metadata and copies it into the plan. It does not fetch, test a connection, or read a secret. Unknown ids raise
brushline.integrations.catalogue.UnknownCatalogueReferenceError.- Parameters:
catalogue (DataCatalogue) – Catalogue and connector registry.
request (FetchRequest) – Requested sources, connectors, estimates, and approvals.
plan_id (str, optional) – Plan identifier. Defaults to
fetch-plan.created_utc (datetime, optional) – Creation time. Defaults to the current UTC time.
- Returns:
FetchPlan – Plan-only preview with size and time estimates.
- Raises:
UnknownCatalogueReferenceError – When a requested source or connector is not registered.
- brushline.integrations.fetch.validate_coverage(report: CoverageReport) ContractValidation[source]
Validate coverage states and their evidence.
- Parameters:
report (CoverageReport) – Coverage report to validate.
- Returns:
ContractValidation – Errors, warnings, and counts.
- brushline.integrations.fetch.validate_fetch_plan(plan: FetchPlan, catalogue: DataCatalogue | None = None) ContractValidation[source]
Validate a fetch plan without fetching.
- Parameters:
plan (FetchPlan) – Plan to validate.
catalogue (DataCatalogue, optional) – Catalogue used to check source, connector, and approval references.
- Returns:
ContractValidation – Errors, warnings, and counts.
- brushline.integrations.fetch.validate_gap_report(report: GapReport) ContractValidation[source]
Validate gap records, the stop/continue decision, and the next action.
- Parameters:
report (GapReport) – Gap report to validate.
- Returns:
ContractValidation – Errors, warnings, and counts.
- brushline.integrations.fetch.validate_materialization(manifest: MaterializationManifest, root: Path | None = None) ContractValidation[source]
Validate checksums, sizes, licences, pointers, and path containment.
A materialized item with a checksum, size, or containment mismatch fails. The caller should stop materialization when
is_validis false.- Parameters:
manifest (MaterializationManifest) – Manifest to validate.
root (Path, optional) – Workspace root used to resolve relative paths and verify files.
- Returns:
ContractValidation – Errors, warnings, and counts.