Skip to content

Amazon AWS

NSHM project uses Amazon Web Services (AWS) for a our 24/7 services (e.g. Public web sites, APIs and datatsets), and also for compute tasks like modelling.

We employ a serverless architecture to ensure that fixed operating costs are minimised.

Here're the key services we use, with some context:

AWS S3 (SimpleStorageService)

This the preferred storage service for NSHM, for it's cost/performance and flexibility. It's used for:

  • NSHM dataset storage (notably the NSHM Hazard datasets were migrated here from DynamoDB),
  • large object storage for the ToshiAPI service,
  • general object storage for graphql API services,
  • Serverless deployments for graphql API services,
  • CI/CD deployments for the Public Websites,
  • Static website deployments e.g. nzshm22-static-reports

We have an NSHM_Storage Dashboard (ap-southeast-2) configured in the AWS Console.

We have experimented with Other Bucket types, but to date we use only General Purpose buckets and primarily the Standard storage class.

Amazon OpenSearch Service (formerly ElasticSearch)

This search engine is used to index the contents of Toshi API, and supports the Search Feature of the Weka Application.

This service is the main outlier in our primarily serverless project, using a pay-per-minute model for a 24/7 service. Here We run a single PROD instance, and a couple of test/dev instances, using the smallest possbile instance configuration to keep costs down.

See AWS OpenSearch Domains (ap-southeast-2)

DynamoDB

This is a big-table style NoSQL database service with a serverless pay-for-usage model. We use it where necessary to provide performant queries on well structured data, but we are migrating more static data away where possible to keep costs down and improve science community access e.g for our Hazard Data Migration project.

The main usage in NSHM is for the metadata storage layer of the ToshiAPI service, where four tables contain > 15 million objects.

See AWS DynamoDB Dashboard (ap-southeast-2)

Lambda

All NSHM web APIs are serverless, meaning that their compute is handled by Lambda functions, without "always on" web server infrastructure. In NSHM Web Stacks there's a diagram showing how these are configured.

NSHM lambda functions use Python3, except for the Graphql API Gateways which use Node. This is because graphql stitching features needed in the API Gateway services are only available from the Apollo Graphql who use the Node ecosystsem with javascript/typescript.

API Gateway

API gateway provides the interface between HTTP(S) web requests and the serverless lambda functions for NSHM web APIs. It is configured via the serverless configuration file for each service i.e. not manually. But in the AWS console one can see the accompanying dashboard, configuration and also the API keys associated with each service e.g. the dashboard of the Kororaa APIGW service.

CloudWatch

CloudWatch provides monitoring facilities for other AWS services. Logging is used frequently to diagnose any service issues.

See CloudWatch Log Groups.

A few App dashboards have been manually configured to help assess the overall performanbce of NSHM services and the user experience.

See CloudWatch Dashboards

a CloudFront Distribution and will be cached on the AWS CloudFront ContentDistributionNetwork

See AWS CloudFront Distributions

CloudFormation

Cloudformation is used indirectly via the serverless configuration files. It can sometimes be useful to inspect the event log for a given stack if there are problems with serverless deployments.

See AWS CloudFormation Stacks

CloudFront

Cloudfront is used to attach SSL certificates to our public HTTPS resources. The Koroaa and Weka web applications, and a couple of static web sites. Each of these is a CloudFront Distribution and will be cached on the AWS CloudFront ContentDistributionNetwork

See AWS CloudFront Distributions

Certificate Manager

Certificates Manager is used to produce SSL certificates. It produces auto-renewing certificates that are easily integrated with the wider AWS ecosystem e.g Cloudfront.

See Setup a Cloudfront Distribution

See AWS Certificates Manager

Backup

cloud/aws_backup.md TODO

Batch

cloud/aws_batch.md TODO

EC2

cloud/aws_ecs2.md TODO

IAM

cloud/aws_iam.md TODO