Provider Ingestion Pipeline

This page documents the complete three-stage provider ingestion workflow that turns external supplier catalog data into trusted entries in the Vancouver PoC database. If you want to import species from a source provider, follow these three stages in order.

Tip

The full set of detailed docs lives under provider-sandbox, provider-approvals, and provider-review-workflow. This page gives the big picture; linked sections have step-by-step commands.

Stage 1: Scrape & Build Review Bundle

Goal: Pull raw data from a provider’s website into a sandbox directory, then generate an expert review app from that sandbox.

Steps:

  1. Install the workflow extra (one-time setup):

    python -m pip install -e .[workflow]
    
  2. Run the provider source-review workflow for your chosen provider (PROV-NWM, PROV-WCS, PROV-OAKSUMMIT, or PROV-SATIN):

    .\scripts\build-provider-source-review.cmd PROV-XXXX -OpenReview
    

    Or generate a FreshForge workflow YAML file first if you need custom parameters:

    bc-nppd generate-provider-source-workflow PROV-XXXX \
      --catalog-url https://example.com/seeds \
      --out-path examples/workflows/providers/PROV-XXXX.yaml \
      --force
    
  3. The workflow fetches catalog pages, populates sandbox artifacts (candidate_species.csv, supplier_availability.csv, etc.), and opens a static review app at outputs/provider_approval_review/PROV-XXXX/index.html.

Output of Stage 1: A sandbox directory tree in outputs/provider_sandbox_source_sweep/PROV-XXXX/ and a review app with an approval_manifest.csv download button.

See also: Provider Review Workflow for full command examples, and Provider Sandbox for the sandbox contract schema.

Stage 2: Review & Approve

Goal: A human expert inspects candidate rows in the review app and exports an approval manifest containing only the species/attributes to accept.

Steps:

  1. Open the review app generated by Stage 1 (usually auto-opens with -OpenReview). The app groups candidates by botanical name and shows:

    • whether the candidate matches an existing Vancouver PoC species;

    • supplier availability rows;

    • candidate attribute rows;

    • provisional mowability rows;

    • provider URLs and taxonomy flags.

  2. Inspect, search, filter, and batch-decide each candidate species:

    • approved — ready to enter the PoC;

    • rejected — should stay out;

    • deferred — not ready for decision;

    • needs_source_review — provenance insufficient;

    • needs_taxonomy_review — name review needed.

  3. Download or copy approval_manifest.csv from the review app.

  4. Validate the manifest before applying:

    bc-nppd validate-provider-approvals path/to/approval_manifest.csv --json
    
  5. (Optional) Preview dry-run: Apply to ignored outputs before committing:

    .\scripts\apply-downloaded-provider-approval.cmd -OpenPreview
    

    Or for cumulative multi-provider preview:

    .\scripts\apply-downloaded-provider-approval.cmd `
      -ManifestPaths \
        outputs/provider_approval_review/PROV-NWM/approval_manifest.csv,\
        outputs/provider_approval_review/PROV-WCS/approval_manifest.csv \
      -OpenPreview
    

Output of Stage 2: A validated approval_manifest.csv with statuses like approved, rejected, and deferred. Only approved rows will be imported.

See also: Provider Approvals for manifest columns and validation, and Provider Review Workflow for the review app interface.

Automated Sandbox Import Preview

For smoke tests, fixture-backed provider runs, or intentionally unsupervised preview builds, bc-nppd can combine the three stages into one ignored output. This command scrapes or reads provider pages into sandbox tables, auto-approves every non-excluded candidate species, includes linked supplier availability, candidate attribute, and mowability rows, and applies the provider sandboxes cumulatively:

bc-nppd auto-import-provider-sandboxes \
  PROV-NWM PROV-WCS PROV-PREMIER \
  --input-dir tests/fixtures/providers \
  --poc-dir data/poc/vancouver \
  --out-dir outputs/provider_auto_import_vancouver \
  --sandbox-root outputs/provider_sandbox_auto_import \
  --json

Use --live-fetch only when provider robots policy permits the configured fetch. If robots policy blocks a provider URL, the command stops with diagnostics rather than bypassing the review contract. Generated sandboxes and preview products belong in ignored outputs/ or local/ paths unless the maintainer explicitly promotes reviewed artifacts. The generated provider_data/ directory preserves provider-side candidate species, candidate attributes, supplier availability, and mowability tables alongside the auto-approved manifest.

Stage 3: Import into Vancouver PoC

Goal: Apply approved rows to update existing species or add new ones in the Vancouver PoC database, regenerating downstream artifacts (evidence, usability, pollinator).

Steps:

  1. Dry-run preview (recommended): Apply approved manifest into ignored outputs:

    bc-nppd apply-provider-approval-sequence path/to/approval_manifest.csv \
      --poc-dir data/poc/vancouver \
      --out-dir outputs/provider_approved_vancouver \
      --json
    

    Inspect outputs/provider_approved_vancouver/ to verify changes look correct.

  2. Promote the manifest (optional, for audit trail):

    Copy the approved manifest into a tracked reviewed path:

    mkdir -p data/poc/vancouver/provider_data/reviewed_manifests/
    cp path/to/approval_manifest.csv \
       data/poc/vancouver/provider_data/reviewed_manifests/
    
  3. Apply to tracked product data (final step, after human verification):

    bc-nppd apply-provider-approvals path/to/approval_manifest.csv \
      --poc-dir data/poc/vancouver \
      --out-dir data/poc/vancouver/provider_data \
      --json
    

    This command updates plant lists, source attribution tables, and regenerates evidence hardening, usability layer, and pollinator module artifacts.

  4. Verify the import:

    bc-nppd validate-vancouver-poc-list data/poc/vancouver --json
    bc-nppd generate-vancouver-usability data/poc/vancouver \
      --out-dir outputs/provider_approved_vancouver --json
    

Output of Stage 3: Updated Vancouver PoC with new or revised species entries, new source attribution records, and refreshed downstream artifacts.

See also: Provider Approvals for the apply command reference, and Vancouver Plant List PoC for the PoC schema.

Complete Three-Stage Workflow Example (PROV-NWM)

Here’s the full pipeline for one provider in sequence:

# Stage 1 — Scrape & review app
.\scripts\build-provider-source-review.cmd PROV-NWM -OpenReview

# Stage 2 — Review in the web app, then download approval_manifest.csv

# Stage 2b — Validate manifest
bc-nppd validate-provider-approvals ~/Downloads/approval_manifest.csv --json

# Stage 3a — Dry-run preview
.\scripts\apply-downloaded-provider-approval.cmd -OpenPreview

# Stage 3b — Final apply (after verification)
bc-nppd apply-provider-approval-sequence ~/Downloads/approval_manifest.csv \
  --poc-dir data/poc/vancouver \
  --out-dir outputs/provider_approved_vancouver \
  --json

Available Source Providers

Five default providers are preconfigured:

Provider ID

Provider Name

Status in Vancouver PoC

PROV-SATIN

Satinflower Nurseries

Imported in tracked PoC

PROV-NWM

Northwest Meadowscapes

Imported in tracked PoC

PROV-WCS

West Coast Seeds

Imported in tracked PoC

PROV-OAKSUMMIT

Oak Summit Nursery

Imported in tracked PoC

PROV-PREMIER

Premier Pacific Seeds

Imported in tracked PoC

The tracked Vancouver PoC was promoted on 2026-07-07 from the all-provider automated provider pathway. It validates at 493 plant rows with provider data from Satinflower, Northwest Meadowscapes, West Coast Seeds, Premier Pacific Seeds, and Oak Summit Nursery. Provider-derived observations remain candidate/pending review data.

Notes and Caveats

  • Provider observations enter the PoC as candidates, not facts. They must pass through Stage 2 review before being imported.

  • Mowability from providers is a provisional 0–5 candidate signal; it does not make ecological scores ready until independently verified.

  • West Coast Seeds vegetable rows are excluded automatically by Vancouver eligibility rules (see Provider Sandbox).

  • The review app is static and dependency-free — no server, database, or login required. It runs from a single HTML file.