brushline.integrations.catalogue
The public source catalogue and private connector registry: metadata models, licence and classification validation, the secret scan, the agent view, and the approval gate.
Data catalogue and connector registry for public and private sources.
The catalogue is metadata-only. It records what a source is, which licence governs it, where it covers, and how it may be read. Private connectors add an approval gate: the agent sees connector metadata, never an authentication value or private rows, until an owner approval exists.
No field stores a credential. An authentication reference names an environment variable, a secret-store item, or a managed identity. The validator rejects a raw secret-shaped value, and the text scanner is a second line of defence for descriptions and sample metadata.
- class brushline.integrations.catalogue.AuditRule(*, log_access: bool = True, retention_days: Annotated[int, Ge(ge=1)] = 365, review_required: bool = True, record_location: str = '')[source]
Bases:
BaseModelThe audit and retention rule for a private connector.
- Parameters:
log_access (bool, optional) – Whether every read is logged. Defaults to
True.retention_days (int, optional) – Audit-log retention in days. Defaults to
365.review_required (bool, optional) – Whether a derived output needs review before release. Defaults to
True.record_location (str, optional) – Where the audit record lives. Required by the validator.
- model_config = {'extra': 'forbid', 'frozen': True}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- class brushline.integrations.catalogue.ConnectionTest(*, method: str = 'environment', reference: str = '', sample_metadata: dict[str, object]=<factory>)[source]
Bases:
BaseModelA non-secret connection test and its expected metadata response.
- Parameters:
method (str, optional) –
environment,secret_store, ormanaged_identity.reference (str, optional) – Non-secret reference used by the test.
sample_metadata (dict of str to object, optional) – Expected metadata keys, never rows or credentials.
- model_config = {'extra': 'forbid', 'frozen': True}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- class brushline.integrations.catalogue.ConnectionTestResult(connector_id: str, status: str, message: str, sample_metadata: dict[str, object])[source]
Bases:
objectThe outcome of a connector test.
- Parameters:
connector_id (str) – Connector that was tested.
status (str) –
blocked,ok,failed, ornot_configured.message (str) – Human-readable explanation.
sample_metadata (dict of str to object) – Expected metadata response; empty when blocked.
- property ok: bool
Return whether the test completed successfully.
- Returns:
bool –
Trueonly for theokstatus.
- class brushline.integrations.catalogue.ConnectorApproval(*, approved_by: str = '', approved_utc: datetime, data_sharing_record: str = '')[source]
Bases:
BaseModelAn owner approval for one private connector.
- Parameters:
approved_by (str) – Owner or role that approved the connector.
approved_utc (datetime) – Timezone-aware approval time.
data_sharing_record (str) – Reference to the data-sharing or access record.
- model_config = {'extra': 'forbid', 'frozen': True}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- class brushline.integrations.catalogue.ConnectorRecord(*, id: ~typing.Annotated[str, _PydanticGeneralMetadata(pattern='^[a-z0-9][a-z0-9._-]{1,63}$')], name: ~typing.Annotated[str, ~annotated_types.MinLen(min_length=1)], owner: ~typing.Annotated[str, ~annotated_types.MinLen(min_length=1)], description: ~typing.Annotated[str, ~annotated_types.MinLen(min_length=1)], classification: str = 'private', linked_source_ids: list[str] = <factory>, authentication_method: str = 'environment', authentication_reference: str = '', read_scope: ~brushline.integrations.catalogue.ReadScope = <factory>, allowed_outputs: list[str] = <factory>, audit: ~brushline.integrations.catalogue.AuditRule = <factory>, test_connection: ~brushline.integrations.catalogue.ConnectionTest = <factory>, approval: ~brushline.integrations.catalogue.ConnectorApproval | None = None)[source]
Bases:
BaseModelOne private or restricted data connector.
- Parameters:
id (str) – Stable connector identifier.
name (str) – Human-readable connector name.
owner (str) – Data owner or responsible role.
description (str) – What the connector reads and why.
classification (str) –
privateorrestricted.linked_source_ids (list of str, optional) – Public catalogue sources that provide context for this connector.
authentication_method (str, optional) –
environment,secret_store, ormanaged_identity.authentication_reference (str, optional) – Non-secret reference such as
env:ASSET_REGISTRY_TOKEN.read_scope (ReadScope, optional) – Tables, columns, filters, and row limit.
allowed_outputs (list of str, optional) –
raw,derived, and/oraggregate.audit (AuditRule, optional) – Audit and retention rule.
test_connection (ConnectionTest, optional) – Offline or owner-run connection test and sample metadata.
approval (ConnectorApproval or None, optional) – Owner approval. The agent cannot read the connector without it.
- model_config = {'extra': 'forbid', 'frozen': True}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- property approved: bool
Return whether the owner has approved the connector.
- Returns:
bool –
Truewhen an approval record exists.
- class brushline.integrations.catalogue.DataCatalogue(*, schema_version: Literal['brushline.catalogue.v1'] = 'brushline.catalogue.v1', name: Annotated[str, ~annotated_types.MinLen(min_length=1)] = 'Brushline data catalogue', created_utc: datetime, sources: list[SourceRecord] = <factory>, connectors: list[ConnectorRecord] = <factory>)[source]
Bases:
BaseModelThe versioned public catalogue and private connector registry.
- Parameters:
schema_version ({"brushline.catalogue.v1"}) – Catalogue schema version.
name (str) – Catalogue name.
created_utc (datetime) – Timezone-aware creation time.
sources (list of SourceRecord) – Public and synthetic sources.
connectors (list of ConnectorRecord) – Private and restricted connectors.
- model_config = {'extra': 'forbid'}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- class brushline.integrations.catalogue.ReadScope(*, tables: list[str] = <factory>, columns: dict[str, list[str]]=<factory>, row_filters: list[str] = <factory>, max_rows: Annotated[int, ~annotated_types.Ge(ge=0)] = 1000)[source]
Bases:
BaseModelThe row and column scope of a private connector.
- Parameters:
tables (list of str) – Tables or views the connector may read.
columns (dict of str to list of str, optional) – Per-table column allowlist.
row_filters (list of str, optional) – Required row filters, expressed as policy text.
max_rows (int, optional) – Maximum rows returned in one call. Defaults to
1000.
- model_config = {'extra': 'forbid', 'frozen': True}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- class brushline.integrations.catalogue.SourceRecord(*, id: Annotated[str, _PydanticGeneralMetadata(pattern='^[a-z0-9][a-z0-9._-]{1,63}$')], name: Annotated[str, MinLen(min_length=1)], provider: Annotated[str, MinLen(min_length=1)], dataset: Annotated[str, MinLen(min_length=1)], query: str = '', vintage: str = '', crs: str = '', licence: str = '', classification: str = 'public', coverage: str = '', access_method: str = 'unknown', access_status: str = 'unknown', url: str = '', description: str = '')[source]
Bases:
BaseModelOne public or synthetic source in the catalogue.
- Parameters:
id (str) – Stable source identifier.
name (str) – Human-readable source name.
provider (str) – Data provider or agency.
dataset (str) – Dataset or product name.
query (str) – Query or selection rule used to read the source.
vintage (str) – Data vintage or version.
crs (str) – Native coordinate reference system.
licence (str) – Licence identifier. Empty and unknown values are rejected.
classification (str) –
publicorsyntheticfor the public catalogue.coverage (str) – Geographic and thematic coverage note.
access_method (str) – Access method, such as
wfs,stac,bulk_download, ormanual.access_status (str) –
available,request,unknown, orunavailable.url (str, optional) – Public landing page or service URL.
description (str, optional) – Short description and caveats.
- model_config = {'extra': 'forbid', 'frozen': True}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- exception brushline.integrations.catalogue.UnknownCatalogueReferenceError(kind: str, reference: str)[source]
Bases:
KeyErrorRaised when a catalogue source or connector id is not registered.
- brushline.integrations.catalogue.agent_can_read(connector: ConnectorRecord) bool[source]
Return whether the agent may read a connector.
- Parameters:
connector (ConnectorRecord) – Connector to check.
- Returns:
bool –
Trueonly when an approval record exists.
- brushline.integrations.catalogue.agent_metadata(catalogue: DataCatalogue) dict[str, object][source]
Return the catalogue view that the agent may see.
Public sources remain visible. Private connectors are reduced to safe metadata, and no authentication reference or private row is included.
- Parameters:
catalogue (DataCatalogue) – Catalogue to expose.
- Returns:
dict of str to object – Public source metadata and safe connector metadata.
- brushline.integrations.catalogue.approved_connector_ids(catalogue: DataCatalogue) tuple[str, ...][source]
Return the ids of approved connectors.
- Parameters:
catalogue (DataCatalogue) – Catalogue to search.
- Returns:
tuple of str – Approved connector ids in catalogue order.
- brushline.integrations.catalogue.connector_agent_metadata(connector: ConnectorRecord) dict[str, object][source]
Return the connector metadata that the agent may see.
The authentication reference and test reference are omitted. The approval flag is included so the agent can ask for approval without inventing access.
- Parameters:
connector (ConnectorRecord) – Connector to expose.
- Returns:
dict of str to object – Safe connector metadata.
- brushline.integrations.catalogue.resolve_connector(catalogue: DataCatalogue, connector_id: str) ConnectorRecord[source]
Return one connector by id.
- Parameters:
catalogue (DataCatalogue) – Catalogue to search.
connector_id (str) – Connector identifier.
- Returns:
ConnectorRecord – The matching connector.
- Raises:
UnknownCatalogueReferenceError – When the connector is not registered.
- brushline.integrations.catalogue.resolve_source(catalogue: DataCatalogue, source_id: str) SourceRecord[source]
Return one source by id.
- Parameters:
catalogue (DataCatalogue) – Catalogue to search.
source_id (str) – Source identifier.
- Returns:
SourceRecord – The matching source.
- Raises:
UnknownCatalogueReferenceError – When the source is not registered.
- brushline.integrations.catalogue.run_connection_test(catalogue: DataCatalogue, connector_id: str, *, probe: Callable[[str], bool] | None = None, environ: Mapping[str, str] | None = None) ConnectionTestResult[source]
Test one connector without returning a secret.
An unapproved connector is always blocked. With a probe, the probe receives the non-secret authentication reference. Without a probe, an
environmenttest checks whether the named environment variable is present; other methods arenot_configuredbecause this process cannot reach the owner’s secret store or managed identity.- Parameters:
catalogue (DataCatalogue) – Catalogue that owns the connector.
connector_id (str) – Connector to test.
probe (callable, optional) – Test hook that receives the authentication reference and returns success.
environ (mapping of str to str, optional) – Environment source. Defaults to
os.environ.
- Returns:
ConnectionTestResult – Status, message, and sample metadata.
- Raises:
UnknownCatalogueReferenceError – When the connector is not registered.
- brushline.integrations.catalogue.scan_for_secrets(text: str) tuple[str, ...][source]
Scan catalogue text for secret-shaped values.
- Parameters:
text (str) – Serialized catalogue, connector, or free text.
- Returns:
tuple of str – Stable
CATALOGUE-SECRETfindings. The secret value is never echoed.
- brushline.integrations.catalogue.validate_catalogue(catalogue: DataCatalogue) ContractValidation[source]
Validate the public sources, private connectors, and approval metadata.
- Parameters:
catalogue (DataCatalogue) – Catalogue to validate.
- Returns:
ContractValidation – Errors, warnings, and counts.