Uptime & Data Collection Monitoring

How GitHub Actions check Rates API uptime every 15 minutes, monitor the hourly collection of New Zealand rates, and open issues for failures.

Rates API uses GitHub Actions workflows to monitor the service. The workflows monitor uptime, data collection, and code quality.

Uptime Checks

The API Uptime Monitoring workflow starts every 15 minutes. It does these steps:

It installs the dependencies with Bun.
It runs apps/api/bin/uptime-check.ts. This script sends requests to the production API at https://www.ratesapi.nz/api/.
If one or more requests fail, the workflow fails.
If the workflow fails, it makes a GitHub issue with the failure details.

If the API returns HTTP 500, 502, 503, or 504, the script sends the request again. It does this a maximum of 3 times.

Monitored Endpoints

The health endpoint
The list endpoint and one detail endpoint for each category
The time-series endpoint for each category, with no filter and with an institution or issuer filter
The mortgage time series, with a date, with a date range, and with the termInMonths filter

The script does not check /openapi or /openapi/json.

Alerts

When a request fails, the workflow makes a GitHub issue with the title 🚨 API Endpoint(s) Down. The issue has the labels incident and high-priority. It mentions the owner of the repository, and it contains the output of the failed requests.

Necessary Permissions

permissions:
  issues: write
  contents: read

If your organization limits permissions, make sure that GitHub Actions can make issues. Also make sure that the incident and high-priority labels exist.

Data Collection

The Hourly Scraping and Database Updates workflow starts each hour. It also starts on each push to main. It does these steps:

It runs bun run check.
It makes sure that it can get access to the remote D1 database.
It runs the four scraper scripts.
It shows the saved data in the workflow log.

A scraper saves a dataset only when the data changed.

Code Quality

These workflows start for pull requests and for pushes to main:

Workflow	Checks
`CI`	`bun run check` (Oxlint, Oxfmt, TypeScript through `tsgo`, docs type generation, and tests) and `bun run build` (the API dry run, the docs OpenNext build, and the web build)
`Fallow`	Static analysis of the code
`React Doctor`	React diagnostics for the changed files

GitHub code scanning also runs CodeQL analysis through its default setup. This analysis does not have a workflow file in the repository.

The Deploy to Cloudflare workflow starts on each push to main. It runs bun run check before it deploys.

Add Checks

To monitor a new endpoint, add it to apps/api/bin/uptime-check.ts.
If a docs route becomes important, add a check for it.
If GitHub issues are not enough, add a notification channel.
If old data is more important than endpoint uptime, add a limit for data age.

Monitoring