INV-01 TSA 23 Full Run

This guide reproduces the TSA 23 point-cloud feature run used for the INV-01 pilot. It is a research runbook, not an operational scheduler.

Architecture

Sockeye compute nodes have no public internet. Public LidarBC point clouds are downloaded to Sockeye scratch, processed there, and archived to Chinook through Globus. The compute phase reads scratch, not Chinook S3.

LidarBC HTTPS
  -> Sockeye scratch raw/laz + raw/dem
  -> per-batch Parquet parts
  -> local reduce + model cards
  -> GeoTIFF / inventory table

Chinook (Globus):
  inv01/raw/laz
  inv01/raw/dem
  inv01/parts2
  inv01/products

Runbook

The executable examples are in examples/inv01_tsa23_full:

  • build_manifest.py pairs LidarBC point-cloud and 1 m DEM tiles.

  • split_manifest.py creates download shards and 64-tile batches.

  • download_tiles.py downloads and resumes tiles.

  • process_batch.py derives first-return metrics on the global grid.

  • compute_batches.sbatch runs the batch array.

  • reduce_full.py clips to the TSA boundary, applies the IDF cards, and optionally writes the 25 m GeoTIFF.

Build the manifest:

python examples/inv01_tsa23_full/build_manifest.py \
  --selection lidarbc_selection_full.json \
  --boundary tsa23_boundary.gpkg \
  --output tile_manifest_full.csv

Split it:

python examples/inv01_tsa23_full/split_manifest.py \
  --manifest tile_manifest_full.csv \
  --download-shards 2 \
  --batch-size 64 \
  --output-dir "$INV01_ROOT/manifests"

Download:

python examples/inv01_tsa23_full/download_tiles.py \
  "$INV01_ROOT/manifests/download_a.csv" \
  "$INV01_ROOT/raw/laz" "$INV01_ROOT/raw/dem" 8

Process batches after setting INV01_ROOT, PYTHON, FEMIC_SRC, and RUNBOOK_DIR:

export INV01_ROOT=/scratch/st-gep-1/gep/inv01
export PYTHON=/arc/project/st-gep-1/gep/venv-inv01-312/bin/python
export FEMIC_SRC=/arc/project/st-gep-1/gep/inv01/femic-src/src
export RUNBOOK_DIR=$PWD/examples/inv01_tsa23_full
cd "$INV01_ROOT"
sbatch "$RUNBOOK_DIR/compute_batches.sbatch"

The global 25 m grid used for this run is in UTM zone 10N:

  • origin (549275, 5784675)

  • cell size 25 m

  • minimum 10 above-2 m first returns

  • minimum cover 10%

Reduce locally:

python examples/inv01_tsa23_full/reduce_full.py \
  --parts /srv/shared-data/gep/ws4/tmp/inv01-data/full-output/parts \
  --boundary tsa23_boundary.gpkg \
  --models config/inventory/models \
  --out-dir /srv/shared-data/gep/ws4/tmp/inv01-data/full-output \
  --raster

Results and limits

The 2026-10-30 run produced:

  • 5,899 point-cloud/DEM tile pairs

  • 93 batches

  • 9,523,476 TSA-clipped cells

  • 6,573,196 height cells

  • 2,942,755 BA/volume cells

  • mean height 16.09 m, BA 23.87 m2/ha, volume 139.33 m3/ha

Coverage was 64% of TSA 23 by tile footprint and 48% by modelled cell area. 907 point-cloud tiles had no matching 1 m DEM; 24 tiles had no usable first returns. The model cards are the 14-plot IDF pilot cards, not an operational inventory. No independent P48/P49 validation was available.

The detailed evidence, timings, hashes, and provenance are in notes/reports/inv01-tsa23-full-mosaic.md.

See Also