GitHub + DataLad + Arbutus Pattern

This guide explains the split-repo pattern FEMIC currently uses for large datasets:

  • the main FEMIC GitHub repo holds code, docs, config, and lightweight metadata;

  • large GIS/model inputs and outputs live in a separate DataLad + git-annex dataset repo; and

  • Arbutus S3 holds the heavy annexed payload objects.

That split keeps the codebase reviewable on GitHub while still giving FEMIC a reproducible way to publish and materialize large datasets.

Short Version

FEMIC does not try to hide DataLad, git-annex, and Arbutus behind fake magic.

Instead it splits responsibilities cleanly:

  • repo creation and special-remote publishing stay in a documented maintainer runbook;

  • clone, install, path-resolution, and preflight checks are wrapped in code and CLI guardrails; and

  • normal runtime commands resolve external data roots explicitly instead of hard-coding workstation paths.

What FEMIC Wraps Today

Registered repo install / clone

FEMIC has a generic catalog loader for known support repos and example instance repos. Catalog entries come from installed instance packages or explicit catalog YAML, not from hardcoded example metadata in FEMIC core:

  • src/femic/builtin_instances.py

  • installed providers exposed through femic.instance_catalogs

This layer wraps practical repo-install behavior:

  • clone known support repos and instance repos with normal Git;

  • install them into a managed external root; and

  • let FEMIC resolve those locations consistently later.

Important boundary:

  • this layer does not automatically run datalad get;

  • FEMIC intentionally separates repo install from payload materialization.

User-scoped managed roots

FEMIC keeps machine-specific install roots in user config instead of scattering hard-coded paths through the codebase:

  • src/femic/user_config.py

Key ideas there:

  • managed_external_root

  • user_instance_root

That lets FEMIC install or resolve external support repos in stable locations without baking a personal workstation path into project logic.

Instance-root and data-root resolution

FEMIC separates instance root from external data root.

Relevant code:

  • src/femic/instance_context.py

  • src/femic/pipeline/io.py

This is the bridge from:

  • “we have a separate DataLad dataset repo somewhere”

to:

  • “normal FEMIC runtime commands can find and trust the data they need.”

The most important operational seam is the explicit external-data root:

  • FEMIC_EXTERNAL_DATA_ROOT

CLI guardrails and preflight

The operational wrapper layer lives mostly in:

  • src/femic/cli/main.py

This is where FEMIC starts behaving like a guarded front-end instead of just a pile of instructions:

  • checking that git-annex is present;

  • checking that DataLad is usable;

  • checking Arbutus environment setup on Windows; and

  • warning early when the runtime environment is obviously not ready.

The practical wrapper commands contributors are expected to use include:

  • git submodule update --init --recursive

  • git -C external/femic-public-data annex enableremote arbutus-s3

  • datalad get -r external/femic-public-data/data

  • export/set FEMIC_EXTERNAL_DATA_ROOT

  • femic prep validate-case

  • femic prep geospatial-preflight

What Stays in Runbooks on Purpose

The repo-creation / publish side is still intentionally documented as a maintainer workflow, not an all-in-one FEMIC CLI feature.

That includes:

  • creating a new DataLad dataset from scratch;

  • initializing the Arbutus S3 special remote from scratch;

  • wiring GitHub sibling publication dependency from scratch; and

  • doing the first publish of main and git-annex.

That flow is documented here:

Why FEMIC stops there:

  • these steps are stateful;

  • they depend on account/project/bucket authority outside the repo; and

  • they are exactly the kind of bootstrap/admin seam where fake automation tends to create harder-to-debug failures later.

How Collaborators Normally Consume the Data

For ordinary developers or students, FEMIC does not expect them to recreate the mirror.

The normal collaborator path is:

  1. clone FEMIC with submodules;

  2. enable the Arbutus special remote on the linked public-data submodule;

  3. materialize only the data they need with datalad get; and

  4. point FEMIC at that materialized data root.

The best docs for that workflow are:

How the Coding Agent Fits In

The safe pattern is not “let the agent improvise raw DataLad and S3 commands from memory.”

The useful FEMIC pattern is:

  • write the dangerous/bootstrap pieces down as a runbook;

  • put stable clone/install/path-resolution/preflight behavior behind FEMIC wrappers and contracts; and

  • let the coding agent drive those wrappers and those docs.

In plain language, the agent is useful for:

  • reading the contracts before doing work;

  • invoking things like femic instance catalog install, femic prep validate-case, and femic prep geospatial-preflight;

  • checking whether the public-data repo is installed and materialized; and

  • steering the human back to the documented runbook when the task crosses into higher-risk DataLad / Arbutus bootstrap territory.

So the useful pattern is:

  • agent pilots the safe wrapper layer;

  • human supervises the risky bootstrap/publish layer.

How to Copy This Pattern into Another Project

If you want to reuse this pattern in a non-FEMIC modelling project, copy these design choices:

1. Keep code and heavy data in separate repos

Use:

  • one normal code repo; and

  • one separate DataLad dataset repo for large input/output artifacts.

2. Use git-annex for payloads, not GitHub blobs

Use GitHub for:

  • lightweight metadata;

  • issue tracking; and

  • normal source control.

Use git-annex + object storage for:

  • heavy rasters;

  • large model inputs;

  • scenario outputs; and

  • archived benchmark/reference bundles.

3. Write down the repo-creation/publish flow before over-automating it

FEMIC’s current runbook-first approach is a good pattern because the bootstrap path is fragile and platform-sensitive.

4. Wrap the stable seams in code

The stable seams worth wrapping are:

  • registered repo catalog;

  • install/clone helpers;

  • user-configured managed roots;

  • instance-root resolution;

  • external-data-root resolution; and

  • preflight checks.

5. Use one explicit external-data environment variable

FEMIC uses:

  • FEMIC_EXTERNAL_DATA_ROOT

A similar project should likely use something like:

  • PROJECT_EXTERNAL_DATA_ROOT

That gives the runtime a single explicit place to look for materialized heavy data.

6. Let the coding agent drive wrappers, not raw infrastructure bootstrapping

The agent should mostly be calling:

  • project CLI commands;

  • preflight checks; and

  • documented runbooks.

It should not be inventing new git annex / S3 / GitHub sibling bootstrap rituals on the fly.

If You Only Read 6 Files

If you want the fastest orientation, start here:

  1. Public Data Mirror Runbook

  2. Deployment Instance Setup

  3. Instance and Data Roots

  4. src/femic/builtin_instances.py

  5. src/femic/pipeline/io.py

  6. src/femic/cli/main.py

That set gives you:

  • the publication model;

  • the install/deployment model;

  • the path-resolution contract; and

  • the main wrapper code.

Bottom Line

The simplest honest description is:

  • FEMIC does not try to hide DataLad/GitHub/Arbutus behind fake magic;

  • it treats dataset publication as a documented, reviewable operator workflow;

  • it wraps the stable downstream seams in code so normal runtime commands can work cleanly; and

  • it uses the coding agent mainly to pilot those wrappers and guardrails, not to improvise infrastructure behavior.

That is the most reusable part of the design for another large-data modelling project.