Skip to content

Process

The end to end process for building a new version is covered by the pipeline_cli module.

This script can be invoked with poetry run pipeline.

pipeline_cli

A CLI to run the sequential pipeline steps and manage versions.

General

  • All the time-consuming steps all accept a site_limit argument. This is used to run the steps using a small chunk of the data - useful for sanity checking. Use the same value (e.g 50) for each step.
  • the pipeline_cli script can be run using poetry run pipeline.
  • check command --help for detailed advice.

Pipeline commands:

  • 01-initialise: creates new version folders in resources & reports folders.
  • 02-hazard: get NSHM hazard curves.
  • 03-tables: build sat & D_M tables and save as *-combo.json for both named and gridded sites.
  • 04-geometry: build geojson artefacts:
    • from CFM get major faults geojson.
    • from nicks file resources\pipeline\v2\polygons_locations.geojson.
    • internally from create_sites_df.
  • 05-report: create reports in PDF & CSV format.
  • 06-publish: seal the version, recording details in resources\version_list.json.
  • 07-deliverables: compile the version in the deliverables folder as requested by Standards New Zealand.

Version commands:

  • ls: list the published versions
  • info: get metadata about a given version

pipeline

A CLI to run the sequential pipeline steps and manage TS1170.5 versions

Usage:

pipeline [OPTIONS] COMMAND [ARGS]...

Options:

  --help  Show this message and exit.

pipeline 01-init

Create the target resource folders for a new version

This should be used before pipeline build steps are run.

Usage:

pipeline 01-init [OPTIONS] VERSION_ID

Options:

  -V, --verbose
  --help         Show this message and exit.

pipeline 02-hazard

Import the NSHM hazard curves from a given model version

Usage: requires env configuration

NZSHM22_HAZARD_STORE_LOCAL_CACHE=~/.cache/toshi_hazard_store
NZSHM22_HAZARD_STORE_REGION=ap-southeast-2
NZSHM22_HAZARD_STORE_STAGE=PROD
AWS_PROFILE=toshi_batch_devops

Usage:

pipeline 02-hazard [OPTIONS] NZSHM_MODEL

Options:

  -V, --verbose
  --site-limit INTEGER
  --help                Show this message and exit.

pipeline 03-tables

Build the resource/v{n}/json tables from a given model version.

Usage:

pipeline 03-tables [OPTIONS] VERSION_ID NZSHM_MODEL

Options:

  -V, --verbose
  --no-cache
  --site-limit INTEGER
  --help                Show this message and exit.

pipeline 04-geometry

Build the geojson artefacts.

Usage:

pipeline 04-geometry [OPTIONS] VERSION_ID

Options:

  -V, --verbose
  --help         Show this message and exit.

pipeline 05-report

Build PDF reports and csv files from json tables.

Usage:

pipeline 05-report [OPTIONS] VERSION_ID

Options:

  --final                      Final version has no DRAFT watermark
  -V, --verbose
  --site-limit INTEGER         For use when testing, uses input json with the
                               same `site_limit`.
  --report-limit INTEGER       Limit the number of locations in the report
  -f, --table [named|gridded]  Choose the tables to build
  --help                       Show this message and exit.

pipeline 06-publish

Write version info to the version list

Usage:

pipeline 06-publish [OPTIONS] VERSION_ID NZSHM_MODEL DESCRIPTION

Options:

  -V, --verbose
  -U, --update
  --help         Show this message and exit.

pipeline 07-deliverables

Build the deliverable artifacts.

Usage:

pipeline 07-deliverables [OPTIONS] VERSION_ID

Options:

  -V, --verbose
  --help         Show this message and exit.

pipeline info

Get detailed info for a given version_id

Usage:

pipeline info [OPTIONS] VERSION_ID

Options:

  -V, --verbose
  --help         Show this message and exit.

pipeline ls

List the available versions

NB any unpublished versions will not be listed.

Usage:

pipeline ls [OPTIONS]

Options:

  -V, --verbose
  --help         Show this message and exit.