brushline.integrations.fetch

The plan-only fetch preview, materialization manifest, coverage states, and gap report: models, the planner, checksum and path checks, the coverage validator, and the stop/continue decision.

Plan-only fetch, materialization, coverage, and gap contracts.

The planner reads the catalogue and writes a plan; it never fetches. The materialization manifest records pointers and checksums, not payloads. The coverage report preserves observed, stale, partial, missing, out-of-domain, and excluded states instead of imputing a value. The gap report turns a non-observed state into a next action and a stop/continue decision.

DataLad, annex, public-mirror, and owner-store pointers are metadata strings. This module does not call a downloader, a remote service, or a secret store.

class brushline.integrations.fetch.CoverageEntry(*, source_id: Annotated[str, MinLen(min_length=1)], target_id: Annotated[str, MinLen(min_length=1)] = 'aoi', state: str = 'missing', coverage_pct: Annotated[float, Ge(ge=0), Le(le=100)] = 0, vintage: str = '', evidence: str = '', reason: str = '', required: bool = True)[source]

Bases: BaseModel

One source coverage state for one target.

Parameters:
  • source_id (str) – Source or connector id.

  • target_id (str, optional) – Segment, span, or aoi target. Defaults to aoi.

  • state (str, optional) – observed, stale, partial, missing, out_of_domain, or excluded.

  • coverage_pct (float, optional) – Coverage percentage from 0 to 100.

  • vintage (str, optional) – Source vintage for the observation.

  • evidence (str, optional) – Evidence reference supporting the state.

  • reason (str, optional) – Reason for a non-observed state.

  • required (bool, optional) – Whether the gap stops the build. Defaults to True.

model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class brushline.integrations.fetch.CoverageReport(*, schema_version: ~typing.Literal['brushline.coverage.v1'] = 'brushline.coverage.v1', plan_id: ~typing.Annotated[str, ~annotated_types.MinLen(min_length=1)], created_utc: ~datetime.datetime, aoi: ~typing.Annotated[str, ~annotated_types.MinLen(min_length=1)], entries: list[~brushline.integrations.fetch.CoverageEntry] = <factory>, notes: str = '')[source]

Bases: BaseModel

The coverage states for one fetch plan.

Parameters:
  • schema_version ({"brushline.coverage.v1"}) – Report schema version.

  • plan_id (str) – Fetch plan identifier.

  • created_utc (datetime) – Timezone-aware creation time.

  • aoi (str) – Area-of-interest reference.

  • entries (list of CoverageEntry) – Coverage entries.

  • notes (str, optional) – Human notes.

model_config = {'extra': 'forbid'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class brushline.integrations.fetch.FetchApproval(*, approved_by: str = '', approved_utc: datetime, scope: str = 'single', record: str = '')[source]

Bases: BaseModel

A user approval recorded on a fetch plan.

Parameters:
  • approved_by (str, optional) – Person or role that approved the fetch.

  • approved_utc (datetime) – Timezone-aware approval time.

  • scope (str, optional) – single or bulk.

  • record (str, optional) – Reference to the approval record.

model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class brushline.integrations.fetch.FetchEstimate(*, size_mb: Annotated[float, Ge(ge=0)] = 0, seconds: Annotated[float, Ge(ge=0)] = 0, basis: str = 'planning_default')[source]

Bases: BaseModel

A planning estimate for one source or connector.

Parameters:
  • size_mb (float, optional) – Estimated transfer size in megabytes. Defaults to 0.

  • seconds (float, optional) – Estimated transfer time in seconds. Defaults to 0.

  • basis (str, optional) – planning_default, user, manual, unknown, or owner_side.

model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class brushline.integrations.fetch.FetchItem(*, kind: str, reference_id: Annotated[str, MinLen(min_length=1)], query: str = '', licence: str = '', access_status: str = 'unknown', classification: str = 'unknown', estimated_size_mb: Annotated[float, Ge(ge=0)] = 0, estimated_seconds: Annotated[float, Ge(ge=0)] = 0, estimate_basis: str = 'unknown', bulk: bool = False, approval: FetchApproval | None = None, materialization: str = 'not_materialized', materialization_pointer: str = '', required: bool = True)[source]

Bases: BaseModel

One source or connector in a fetch plan.

Parameters:
  • kind (str) – source or connector.

  • reference_id (str) – Source or connector id.

  • query (str) – Query or read scope.

  • licence (str) – Licence or data-sharing record.

  • access_status (str) – Catalogue access status, or blocked for an unapproved connector.

  • classification (str) – Source or connector classification.

  • estimated_size_mb (float, optional) – Estimated transfer size in megabytes.

  • estimated_seconds (float, optional) – Estimated transfer time in seconds.

  • estimate_basis (str, optional) – Where the estimate came from.

  • bulk (bool, optional) – Whether the item is part of a bulk request.

  • approval (FetchApproval or None, optional) – Recorded approval for the item.

  • materialization (str, optional) – data_lad, annex, public_mirror, owner_store, or not_materialized.

  • materialization_pointer (str, optional) – Metadata pointer, never a payload path.

  • required (bool, optional) – Whether a missing source stops the build. Defaults to True.

model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class brushline.integrations.fetch.FetchPlan(*, schema_version: ~typing.Literal['brushline.fetch-plan.v1'] = 'brushline.fetch-plan.v1', plan_id: ~typing.Annotated[str, ~annotated_types.MinLen(min_length=1)], created_utc: ~datetime.datetime, aoi: ~typing.Annotated[str, ~annotated_types.MinLen(min_length=1)], mode: ~typing.Literal['plan_only'] = 'plan_only', items: list[~brushline.integrations.fetch.FetchItem] = <factory>, notes: str = '')[source]

Bases: BaseModel

A plan-only fetch preview.

Parameters:
  • schema_version ({"brushline.fetch-plan.v1"}) – Plan schema version.

  • plan_id (str) – Stable plan identifier.

  • created_utc (datetime) – Timezone-aware creation time.

  • aoi (str) – Area-of-interest reference.

  • mode ({"plan_only"}) – The planner never downloads.

  • items (list of FetchItem) – Sources and connectors in plan order.

  • notes (str, optional) – Human notes.

model_config = {'extra': 'forbid'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

property total_size_mb: float

Return the estimated total transfer size.

Returns:

float – Sum of item size estimates in megabytes.

property total_seconds: float

Return the estimated total transfer time.

Returns:

float – Sum of item time estimates in seconds.

property network_items: int

Return the number of items with a materialization pointer.

Returns:

int – Count of items that are not not_materialized.

class brushline.integrations.fetch.FetchRequest(*, aoi: ~typing.Annotated[str, ~annotated_types.MinLen(min_length=1)], source_ids: list[str] = <factory>, connector_ids: list[str] = <factory>, queries: dict[str, str] = <factory>, estimates: dict[str, ~brushline.integrations.fetch.FetchEstimate] = <factory>, optional_ids: list[str] = <factory>, bulk: bool = False, approvals: dict[str, ~brushline.integrations.fetch.FetchApproval] = <factory>)[source]

Bases: BaseModel

A request to plan a fetch without executing it.

Parameters:
  • aoi (str) – Area of interest reference, such as an instance-relative AOI path.

  • source_ids (list of str, optional) – Public source ids to plan.

  • connector_ids (list of str, optional) – Private connector ids to plan.

  • queries (dict of str to str, optional) – Query overrides keyed by source or connector id.

  • estimates (dict of str to FetchEstimate, optional) – User estimates keyed by source or connector id.

  • optional_ids (list of str, optional) – Ids that may be missing without stopping the build.

  • bulk (bool, optional) – Whether the request is a bulk fetch. Defaults to False.

  • approvals (dict of str to FetchApproval, optional) – Approvals keyed by source or connector id.

model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class brushline.integrations.fetch.GapRecord(*, gap_id: Annotated[str, MinLen(min_length=1)], source_id: Annotated[str, MinLen(min_length=1)], target_id: Annotated[str, MinLen(min_length=1)], state: str, decision: str, recommended_action: Annotated[str, MinLen(min_length=1)], reason: str = '')[source]

Bases: BaseModel

One non-observed coverage gap and its next action.

Parameters:
  • gap_id (str) – Stable gap identifier.

  • source_id (str) – Source or connector id.

  • target_id (str) – Segment, span, or aoi target.

  • state (str) – Coverage state that created the gap.

  • decision (str) – stop or continue_with_caveat.

  • recommended_action (str) – Next action.

  • reason (str) – Why the gap exists.

model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class brushline.integrations.fetch.GapReport(*, schema_version: ~typing.Literal['brushline.gap-report.v1'] = 'brushline.gap-report.v1', plan_id: ~typing.Annotated[str, ~annotated_types.MinLen(min_length=1)], created_utc: ~datetime.datetime, decision: str = 'continue', next_action: str = 'none', gaps: list[~brushline.integrations.fetch.GapRecord] = <factory>)[source]

Bases: BaseModel

The gap summary and stop/continue decision.

Parameters:
  • schema_version ({"brushline.gap-report.v1"}) – Report schema version.

  • plan_id (str) – Fetch plan identifier.

  • created_utc (datetime) – Timezone-aware creation time.

  • decision (str) – stop, continue, or continue_with_caveat.

  • next_action (str) – The first recommended action, or none.

  • gaps (list of GapRecord) – Non-observed coverage entries.

model_config = {'extra': 'forbid'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class brushline.integrations.fetch.MaterializationManifest(*, schema_version: ~typing.Literal['brushline.materialization.v1'] = 'brushline.materialization.v1', plan_id: ~typing.Annotated[str, ~annotated_types.MinLen(min_length=1)], created_utc: ~datetime.datetime, items: list[~brushline.integrations.fetch.MaterializedItem] = <factory>, notes: str = '')[source]

Bases: BaseModel

The materialization record for one fetch plan.

Parameters:
  • schema_version ({"brushline.materialization.v1"}) – Manifest schema version.

  • plan_id (str) – Fetch plan identifier.

  • created_utc (datetime) – Timezone-aware creation time.

  • items (list of MaterializedItem) – Materialized and missing items.

  • notes (str, optional) – Human notes.

model_config = {'extra': 'forbid'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class brushline.integrations.fetch.MaterializedItem(*, id: Annotated[str, MinLen(min_length=1)], source_id: Annotated[str, MinLen(min_length=1)], status: str = 'materialized', path: str = '', sha256: str = '', size_bytes: Annotated[int, Ge(ge=0)] = 0, retrieved_utc: datetime | None = None, licence: str = '', pointer: str = '', reason: str = '')[source]

Bases: BaseModel

One materialized or missing source.

Parameters:
  • id (str) – Stable item identifier.

  • source_id (str) – Source or connector id.

  • status (str, optional) – materialized, missing, partial, or failed.

  • path (str, optional) – Instance-relative path. Empty for a missing item.

  • sha256 (str, optional) – Expected SHA-256 of the file.

  • size_bytes (int, optional) – Expected file size in bytes.

  • retrieved_utc (datetime or None, optional) – Timezone-aware retrieval time.

  • licence (str, optional) – Licence or data-sharing record.

  • pointer (str, optional) – data_lad:, annex:, public_mirror:, or owner_store: pointer.

  • reason (str, optional) – Why the item is missing, partial, or failed.

model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

brushline.integrations.fetch.build_gap_report(coverage: CoverageReport) → GapReport[source]

Turn non-observed coverage entries into gaps and a decision.

A required missing or out-of-domain source stops the build. A stale, partial, excluded, or optional gap continues with a caveat. The gap report never imputes coverage.

Parameters:

coverage (CoverageReport) – Coverage report.

Returns:

GapReport – Gaps, the overall decision, and the next action.

brushline.integrations.fetch.coverage_counts(report: CoverageReport) → dict[str, int | float][source]

Count coverage states in a report.

Parameters:

report (CoverageReport) – Coverage report.

Returns:

dict of str to int or float – Counts for each state plus total.

brushline.integrations.fetch.default_estimate(access_method: str) → FetchEstimate[source]

Return the deterministic planning estimate for an access method.

These are planning placeholders, not measurements. A user estimate overrides them in a FetchRequest.

Parameters:

access_method (str) – Catalogue access method.

Returns:

FetchEstimate – Size, time, and basis.

brushline.integrations.fetch.file_sha256(path: Path) → str[source]

Return the SHA-256 digest of a file.

Parameters:

path (Path) – File to hash.

Returns:

str – Lowercase hexadecimal SHA-256 digest.

brushline.integrations.fetch.materialization_decision(manifest: MaterializationManifest, root: Path | None = None) → str[source]

Return the materialization decision.

Parameters:
  • manifest (MaterializationManifest) – Manifest to validate.

  • root (Path, optional) – Workspace root used for file checks.

Returns:

str – continue when the manifest is valid, otherwise stop.

brushline.integrations.fetch.plan_fetch(catalogue: DataCatalogue, request: FetchRequest, *, plan_id: str = 'fetch-plan', created_utc: datetime | None = None) → FetchPlan[source]

Build a plan-only fetch preview from the catalogue.

The function resolves source and connector metadata and copies it into the plan. It does not fetch, test a connection, or read a secret. Unknown ids raise brushline.integrations.catalogue.UnknownCatalogueReferenceError.

Parameters:
  • catalogue (DataCatalogue) – Catalogue and connector registry.

  • request (FetchRequest) – Requested sources, connectors, estimates, and approvals.

  • plan_id (str, optional) – Plan identifier. Defaults to fetch-plan.

  • created_utc (datetime, optional) – Creation time. Defaults to the current UTC time.

Returns:

FetchPlan – Plan-only preview with size and time estimates.

Raises:

UnknownCatalogueReferenceError – When a requested source or connector is not registered.

brushline.integrations.fetch.validate_coverage(report: CoverageReport) → ContractValidation[source]

Validate coverage states and their evidence.

Parameters:

report (CoverageReport) – Coverage report to validate.

Returns:

ContractValidation – Errors, warnings, and counts.

brushline.integrations.fetch.validate_fetch_plan(plan: FetchPlan, catalogue: DataCatalogue | None = None) → ContractValidation[source]

Validate a fetch plan without fetching.

Parameters:
  • plan (FetchPlan) – Plan to validate.

  • catalogue (DataCatalogue, optional) – Catalogue used to check source, connector, and approval references.

Returns:

ContractValidation – Errors, warnings, and counts.

brushline.integrations.fetch.validate_gap_report(report: GapReport) → ContractValidation[source]

Validate gap records, the stop/continue decision, and the next action.

Parameters:

report (GapReport) – Gap report to validate.

Returns:

ContractValidation – Errors, warnings, and counts.

brushline.integrations.fetch.validate_materialization(manifest: MaterializationManifest, root: Path | None = None) → ContractValidation[source]

Validate checksums, sizes, licences, pointers, and path containment.

A materialized item with a checksum, size, or containment mismatch fails. The caller should stop materialization when is_valid is false.

Parameters:
  • manifest (MaterializationManifest) – Manifest to validate.

  • root (Path, optional) – Workspace root used to resolve relative paths and verify files.

Returns:

ContractValidation – Errors, warnings, and counts.