Provider Ingestion Pipeline ============================ This page documents the complete **three-stage provider ingestion workflow** that turns external supplier catalog data into trusted entries in the Vancouver PoC database. If you want to import species from a source provider, follow these three stages in order. .. tip:: The full set of detailed docs lives under ``provider-sandbox``, ``provider-approvals``, and ``provider-review-workflow``. This page gives the big picture; linked sections have step-by-step commands. Stage 1: Scrape & Build Review Bundle ------------------------------------- **Goal:** Pull raw data from a provider's website into a sandbox directory, then generate an expert review app from that sandbox. Steps: 1. **Install the workflow extra (one-time setup):** .. code-block:: console python -m pip install -e .[workflow] 2. **Run the provider source-review workflow** for your chosen provider (``PROV-NWM``, ``PROV-WCS``, ``PROV-OAKSUMMIT``, or ``PROV-SATIN``): .. code-block:: console .\scripts\build-provider-source-review.cmd PROV-XXXX -OpenReview Or generate a FreshForge workflow YAML file first if you need custom parameters: .. code-block:: console bc-nppd generate-provider-source-workflow PROV-XXXX \ --catalog-url https://example.com/seeds \ --out-path examples/workflows/providers/PROV-XXXX.yaml \ --force 3. The workflow fetches catalog pages, populates sandbox artifacts (``candidate_species.csv``, ``supplier_availability.csv``, etc.), and opens a static review app at ``outputs/provider_approval_review/PROV-XXXX/index.html``. **Output of Stage 1:** A sandbox directory tree in ``outputs/provider_sandbox_source_sweep/PROV-XXXX/`` and a review app with an ``approval_manifest.csv`` download button. **See also:** :doc:`provider-review-workflow` for full command examples, and :doc:`provider-sandbox` for the sandbox contract schema. Stage 2: Review & Approve ------------------------- **Goal:** A human expert inspects candidate rows in the review app and exports an approval manifest containing only the species/attributes to accept. Steps: 1. **Open the review app** generated by Stage 1 (usually auto-opens with ``-OpenReview``). The app groups candidates by botanical name and shows: * whether the candidate matches an existing Vancouver PoC species; * supplier availability rows; * candidate attribute rows; * provisional mowability rows; * provider URLs and taxonomy flags. 2. **Inspect, search, filter, and batch-decide** each candidate species: * ``approved`` — ready to enter the PoC; * ``rejected`` — should stay out; * ``deferred`` — not ready for decision; * ``needs_source_review`` — provenance insufficient; * ``needs_taxonomy_review`` — name review needed. 3. **Download or copy** ``approval_manifest.csv`` from the review app. 4. **Validate** the manifest before applying: .. code-block:: console bc-nppd validate-provider-approvals path/to/approval_manifest.csv --json 5. **(Optional) Preview dry-run:** Apply to ignored outputs before committing: .. code-block:: console .\scripts\apply-downloaded-provider-approval.cmd -OpenPreview Or for cumulative multi-provider preview: .. code-block:: console .\scripts\apply-downloaded-provider-approval.cmd ` -ManifestPaths \ outputs/provider_approval_review/PROV-NWM/approval_manifest.csv,\ outputs/provider_approval_review/PROV-WCS/approval_manifest.csv \ -OpenPreview **Output of Stage 2:** A validated ``approval_manifest.csv`` with statuses like ``approved``, ``rejected``, and ``deferred``. Only ``approved`` rows will be imported. **See also:** :doc:`provider-approvals` for manifest columns and validation, and :doc:`provider-review-workflow` for the review app interface. Automated Sandbox Import Preview -------------------------------- For smoke tests, fixture-backed provider runs, or intentionally unsupervised preview builds, ``bc-nppd`` can combine the three stages into one ignored output. This command scrapes or reads provider pages into sandbox tables, auto-approves every non-excluded candidate species, includes linked supplier availability, candidate attribute, and mowability rows, and applies the provider sandboxes cumulatively: .. code-block:: console bc-nppd auto-import-provider-sandboxes \ PROV-NWM PROV-WCS PROV-PREMIER \ --input-dir tests/fixtures/providers \ --poc-dir data/poc/vancouver \ --out-dir outputs/provider_auto_import_vancouver \ --sandbox-root outputs/provider_sandbox_auto_import \ --json Use ``--live-fetch`` only when provider robots policy permits the configured fetch. If robots policy blocks a provider URL, the command stops with diagnostics rather than bypassing the review contract. Generated sandboxes and preview products belong in ignored ``outputs/`` or ``local/`` paths unless the maintainer explicitly promotes reviewed artifacts. The generated ``provider_data/`` directory preserves provider-side candidate species, candidate attributes, supplier availability, and mowability tables alongside the auto-approved manifest. Stage 3: Import into Vancouver PoC ---------------------------------- **Goal:** Apply approved rows to update existing species or add new ones in the Vancouver PoC database, regenerating downstream artifacts (evidence, usability, pollinator). Steps: 1. **Dry-run preview (recommended):** Apply approved manifest into ignored outputs: .. code-block:: console bc-nppd apply-provider-approval-sequence path/to/approval_manifest.csv \ --poc-dir data/poc/vancouver \ --out-dir outputs/provider_approved_vancouver \ --json Inspect ``outputs/provider_approved_vancouver/`` to verify changes look correct. 2. **Promote the manifest** (optional, for audit trail): Copy the approved manifest into a tracked reviewed path: .. code-block:: console mkdir -p data/poc/vancouver/provider_data/reviewed_manifests/ cp path/to/approval_manifest.csv \ data/poc/vancouver/provider_data/reviewed_manifests/ 3. **Apply to tracked product data** (final step, after human verification): .. code-block:: console bc-nppd apply-provider-approvals path/to/approval_manifest.csv \ --poc-dir data/poc/vancouver \ --out-dir data/poc/vancouver/provider_data \ --json This command updates plant lists, source attribution tables, and regenerates evidence hardening, usability layer, and pollinator module artifacts. 4. **Verify the import:** .. code-block:: console bc-nppd validate-vancouver-poc-list data/poc/vancouver --json bc-nppd generate-vancouver-usability data/poc/vancouver \ --out-dir outputs/provider_approved_vancouver --json **Output of Stage 3:** Updated Vancouver PoC with new or revised species entries, new source attribution records, and refreshed downstream artifacts. **See also:** :doc:`provider-approvals` for the apply command reference, and :doc:`vancouver-poc-list` for the PoC schema. Complete Three-Stage Workflow Example (PROV-NWM) ------------------------------------------------ Here's the full pipeline for one provider in sequence: .. code-block:: console # Stage 1 — Scrape & review app .\scripts\build-provider-source-review.cmd PROV-NWM -OpenReview # Stage 2 — Review in the web app, then download approval_manifest.csv # Stage 2b — Validate manifest bc-nppd validate-provider-approvals ~/Downloads/approval_manifest.csv --json # Stage 3a — Dry-run preview .\scripts\apply-downloaded-provider-approval.cmd -OpenPreview # Stage 3b — Final apply (after verification) bc-nppd apply-provider-approval-sequence ~/Downloads/approval_manifest.csv \ --poc-dir data/poc/vancouver \ --out-dir outputs/provider_approved_vancouver \ --json Available Source Providers -------------------------- Five default providers are preconfigured: .. list-table:: :header-rows: 1 :widths: auto * - Provider ID - Provider Name - Status in Vancouver PoC * - ``PROV-SATIN`` - Satinflower Nurseries - Imported in tracked PoC * - ``PROV-NWM`` - Northwest Meadowscapes - Imported in tracked PoC * - ``PROV-WCS`` - West Coast Seeds - Imported in tracked PoC * - ``PROV-OAKSUMMIT`` - Oak Summit Nursery - Imported in tracked PoC * - ``PROV-PREMIER`` - Premier Pacific Seeds - Imported in tracked PoC The tracked Vancouver PoC was promoted on 2026-07-07 from the all-provider automated provider pathway. It validates at 493 plant rows with provider data from Satinflower, Northwest Meadowscapes, West Coast Seeds, Premier Pacific Seeds, and Oak Summit Nursery. Provider-derived observations remain candidate/pending review data. Notes and Caveats ----------------- * Provider observations enter the PoC as **candidates**, not facts. They must pass through Stage 2 review before being imported. * Mowability from providers is a provisional 0–5 candidate signal; it does not make ecological scores ready until independently verified. * West Coast Seeds vegetable rows are excluded automatically by Vancouver eligibility rules (see :doc:`provider-sandbox`). * The review app is static and dependency-free — no server, database, or login required. It runs from a single HTML file.