Provider Ingestion Pipeline
This page documents the complete three-stage provider ingestion workflow that turns external supplier catalog data into trusted entries in the Vancouver PoC database. If you want to import species from a source provider, follow these three stages in order.
Tip
The full set of detailed docs lives under provider-sandbox,
provider-approvals, and provider-review-workflow.
This page gives the big picture; linked sections have step-by-step commands.
Stage 1: Scrape & Build Review Bundle
Goal: Pull raw data from a provider’s website into a sandbox directory, then generate an expert review app from that sandbox.
Steps:
Install the workflow extra (one-time setup):
python -m pip install -e .[workflow]Run the provider source-review workflow for your chosen provider (
PROV-NWM,PROV-WCS,PROV-OAKSUMMIT, orPROV-SATIN):.\scripts\build-provider-source-review.cmd PROV-XXXX -OpenReviewOr generate a FreshForge workflow YAML file first if you need custom parameters:
bc-nppd generate-provider-source-workflow PROV-XXXX \ --catalog-url https://example.com/seeds \ --out-path examples/workflows/providers/PROV-XXXX.yaml \ --force
The workflow fetches catalog pages, populates sandbox artifacts (
candidate_species.csv,supplier_availability.csv, etc.), and opens a static review app atoutputs/provider_approval_review/PROV-XXXX/index.html.
Output of Stage 1: A sandbox directory tree in outputs/provider_sandbox_source_sweep/PROV-XXXX/ and a review app with an approval_manifest.csv download button.
See also: Provider Review Workflow for full command examples, and Provider Sandbox for the sandbox contract schema.
Stage 2: Review & Approve
Goal: A human expert inspects candidate rows in the review app and exports an approval manifest containing only the species/attributes to accept.
Steps:
Open the review app generated by Stage 1 (usually auto-opens with
-OpenReview). The app groups candidates by botanical name and shows:whether the candidate matches an existing Vancouver PoC species;
supplier availability rows;
candidate attribute rows;
provisional mowability rows;
provider URLs and taxonomy flags.
Inspect, search, filter, and batch-decide each candidate species:
approved— ready to enter the PoC;rejected— should stay out;deferred— not ready for decision;needs_source_review— provenance insufficient;needs_taxonomy_review— name review needed.
Download or copy
approval_manifest.csvfrom the review app.Validate the manifest before applying:
bc-nppd validate-provider-approvals path/to/approval_manifest.csv --json(Optional) Preview dry-run: Apply to ignored outputs before committing:
.\scripts\apply-downloaded-provider-approval.cmd -OpenPreviewOr for cumulative multi-provider preview:
.\scripts\apply-downloaded-provider-approval.cmd ` -ManifestPaths \ outputs/provider_approval_review/PROV-NWM/approval_manifest.csv,\ outputs/provider_approval_review/PROV-WCS/approval_manifest.csv \ -OpenPreview
Output of Stage 2: A validated approval_manifest.csv with statuses like
approved, rejected, and deferred. Only approved rows will be
imported.
See also: Provider Approvals for manifest columns and validation, and Provider Review Workflow for the review app interface.
Automated Sandbox Import Preview
For smoke tests, fixture-backed provider runs, or intentionally unsupervised
preview builds, bc-nppd can combine the three stages into one ignored
output. This command scrapes or reads provider pages into sandbox tables,
auto-approves every non-excluded candidate species, includes linked supplier
availability, candidate attribute, and mowability rows, and applies the
provider sandboxes cumulatively:
bc-nppd auto-import-provider-sandboxes \
PROV-NWM PROV-WCS PROV-PREMIER \
--input-dir tests/fixtures/providers \
--poc-dir data/poc/vancouver \
--out-dir outputs/provider_auto_import_vancouver \
--sandbox-root outputs/provider_sandbox_auto_import \
--json
Use --live-fetch only when provider robots policy permits the configured
fetch. If robots policy blocks a provider URL, the command stops with
diagnostics rather than bypassing the review contract. Generated sandboxes and
preview products belong in ignored outputs/ or local/ paths unless the
maintainer explicitly promotes reviewed artifacts. The generated
provider_data/ directory preserves provider-side candidate species,
candidate attributes, supplier availability, and mowability tables alongside
the auto-approved manifest.
Stage 3: Import into Vancouver PoC
Goal: Apply approved rows to update existing species or add new ones in the Vancouver PoC database, regenerating downstream artifacts (evidence, usability, pollinator).
Steps:
Dry-run preview (recommended): Apply approved manifest into ignored outputs:
bc-nppd apply-provider-approval-sequence path/to/approval_manifest.csv \ --poc-dir data/poc/vancouver \ --out-dir outputs/provider_approved_vancouver \ --json
Inspect
outputs/provider_approved_vancouver/to verify changes look correct.Promote the manifest (optional, for audit trail):
Copy the approved manifest into a tracked reviewed path:
mkdir -p data/poc/vancouver/provider_data/reviewed_manifests/ cp path/to/approval_manifest.csv \ data/poc/vancouver/provider_data/reviewed_manifests/
Apply to tracked product data (final step, after human verification):
bc-nppd apply-provider-approvals path/to/approval_manifest.csv \ --poc-dir data/poc/vancouver \ --out-dir data/poc/vancouver/provider_data \ --json
This command updates plant lists, source attribution tables, and regenerates evidence hardening, usability layer, and pollinator module artifacts.
Verify the import:
bc-nppd validate-vancouver-poc-list data/poc/vancouver --json bc-nppd generate-vancouver-usability data/poc/vancouver \ --out-dir outputs/provider_approved_vancouver --json
Output of Stage 3: Updated Vancouver PoC with new or revised species entries, new source attribution records, and refreshed downstream artifacts.
See also: Provider Approvals for the apply command reference, and Vancouver Plant List PoC for the PoC schema.
Complete Three-Stage Workflow Example (PROV-NWM)
Here’s the full pipeline for one provider in sequence:
# Stage 1 — Scrape & review app
.\scripts\build-provider-source-review.cmd PROV-NWM -OpenReview
# Stage 2 — Review in the web app, then download approval_manifest.csv
# Stage 2b — Validate manifest
bc-nppd validate-provider-approvals ~/Downloads/approval_manifest.csv --json
# Stage 3a — Dry-run preview
.\scripts\apply-downloaded-provider-approval.cmd -OpenPreview
# Stage 3b — Final apply (after verification)
bc-nppd apply-provider-approval-sequence ~/Downloads/approval_manifest.csv \
--poc-dir data/poc/vancouver \
--out-dir outputs/provider_approved_vancouver \
--json
Available Source Providers
Five default providers are preconfigured:
Provider ID |
Provider Name |
Status in Vancouver PoC |
|---|---|---|
|
Satinflower Nurseries |
Imported in tracked PoC |
|
Northwest Meadowscapes |
Imported in tracked PoC |
|
West Coast Seeds |
Imported in tracked PoC |
|
Oak Summit Nursery |
Imported in tracked PoC |
|
Premier Pacific Seeds |
Imported in tracked PoC |
The tracked Vancouver PoC was promoted on 2026-07-07 from the all-provider automated provider pathway. It validates at 493 plant rows with provider data from Satinflower, Northwest Meadowscapes, West Coast Seeds, Premier Pacific Seeds, and Oak Summit Nursery. Provider-derived observations remain candidate/pending review data.
Notes and Caveats
Provider observations enter the PoC as candidates, not facts. They must pass through Stage 2 review before being imported.
Mowability from providers is a provisional 0–5 candidate signal; it does not make ecological scores ready until independently verified.
West Coast Seeds vegetable rows are excluded automatically by Vancouver eligibility rules (see Provider Sandbox).
The review app is static and dependency-free — no server, database, or login required. It runs from a single HTML file.