Tobserver/docs/bitcoin-pulse-operations.md

Bitcoin Pulse production operations

This runbook covers routine Bitcoin Pulse deployment and maintenance on Tobserver. The declarative configuration lives in modules/bitcoin-pulse.nix; PostgreSQL details are in bitcoin-pulse-postgresql.md, and public ingress checks are in bitcoin-pulse-ingress.md.

Run production commands from the Tobserver checkout unless noted otherwise. Never place plaintext credentials in Git, the Nix store, command arguments, shell history, logs, or issue comments.

Standard deployment

Prepare and review every change on a workstation before touching the server:

nix flake check --no-build
nix build .#nixosConfigurations.tobserver.config.system.build.toplevel

The second command can run only on a compatible Linux builder. It is optional on a workstation without one because the server build below performs the same full configuration build before activation.

Commit and push the reviewed change. On Tobserver, update the clean checkout and build without changing the running system:

cd ~/tobserver
test -z "$(git status --porcelain)" || {
  echo "Refusing to deploy from a dirty checkout" >&2
  exit 1
}
git pull --ff-only
git rev-parse HEAD
nix flake check --no-build
sudo nixos-rebuild build --flake .#tobserver

The cleanliness check stops the procedure before pulling when the checkout has local changes. Compare the printed commit with the approved deployment commit. Review build failures before continuing. A successful build creates the ignored result symlink but does not restart any service.

Activate the already-reviewed configuration:

sudo nixos-rebuild switch --flake .#tobserver

Do not use a live full-host NixOS generation rollback as a Bitcoin Pulse test or routine recovery action. Tobserver runs unrelated production services that a full generation switch can also change.

Post-deployment verification

Verify service state, migration completion, loopback-only listeners, and local responses:

sudo systemctl is-active \
  podman-bitcoin-pulse-postgres.service \
  bitcoin-pulse.service \
  nginx.service

sudo systemctl show bitcoin-pulse-migrate.service \
  --property=ActiveState \
  --property=Result \
  --property=ExecMainStatus

sudo ss -ltnp | grep -E ':(3000|5432)\b'

curl --fail --show-error --silent http://127.0.0.1:3000/health \
  | jq -e '.status == "OK"'
curl --fail --show-error --silent http://127.0.0.1:3000/block-height \
  | jq -e .
curl --fail --show-error --silent http://127.0.0.1:3000/difficulty-adjustment \
  | jq -e .
curl --fail --show-error --silent http://127.0.0.1:3000/price \
  | jq -e .

All three long-running services must be active. The migration unit is a completed one-shot, so ActiveState=inactive, Result=success, and ExecMainStatus=0 are expected. PostgreSQL and the backend must listen only on 127.0.0.1:5432 and 127.0.0.1:3000 respectively.

From a machine outside Tobserver, run the public checks in bitcoin-pulse-ingress.md. Inspect relevant logs without copying credentials:

sudo journalctl \
  -u podman-bitcoin-pulse-postgres.service \
  -u bitcoin-pulse-migrate.service \
  -u bitcoin-pulse.service \
  -u nginx.service \
  --since "-15 minutes" \
  --no-pager

Updating the Bitcoin Pulse backend

Update only the pinned backend input from a workstation:

cd ~/Code/Tobserver
nix flake update bitcoin-pulse
git diff -- flake.lock
nix flake check --no-build

Confirm that flake.lock points to the intended reviewed BitcoinPulseAPI revision. Commit and push the lock-file update, follow the standard deployment, and perform all post-deployment checks. The switch runs the packaged version's pending migrations before starting that backend version.

Every database migration must remain compatible with the previously deployed application while it is still a possible recovery version. Do not run bitcoin-pulse-migrate rollback automatically.

Backups

The daily logical backup, retention policy, credentials, and initial verification procedure are documented in bitcoin-pulse-postgresql.md.

Before a database image change or other maintenance with data risk, create and verify a fresh snapshot:

sudo systemctl start restic-backups-bitcoin-pulse.service
sudo systemctl show restic-backups-bitcoin-pulse.service \
  --property=Result \
  --property=ExecMainStatus
sudo restic-bitcoin-pulse snapshots --latest 1
sudo restic-bitcoin-pulse ls latest | grep bitcoin-pulse.dump
sudo find /var/backup/bitcoin-pulse -maxdepth 1 -type f -ls

Expect Result=success, ExecMainStatus=0, a snapshot containing bitcoin-pulse.dump, and no local staging file after completion. Alerting and routine restore tests are separate backlog work.

PostgreSQL image updates within the current major version

Automatic image updates are disabled. For a PostgreSQL 18 patch or minor image update:

1. Read the official PostgreSQL image release notes and confirm the data directory remains compatible.
2. Create and verify a fresh Bitcoin Pulse backup.
3. Test the candidate image with the backend package and migrations outside production where practical.
4. Verify that the candidate digest identifies the intended PostgreSQL 18 Bookworm patch release, then replace the immutable digest in modules/bitcoin-pulse.nix.
5. Update the adjacent exact-version and tested-date comment and the declared version in bitcoin-pulse-postgresql.md in the same change.
6. Run nixpkgs-fmt --check modules/bitcoin-pulse.nix and nix flake check --no-build.
7. Review, commit, push, and follow the standard deployment procedure.
8. Repeat all post-deployment and backup checks.

Never enable Podman automatic updates or deploy a floating image tag.

PostgreSQL major upgrades

A major PostgreSQL upgrade is a separate maintenance project, not a routine image-digest change. A new major version must never start against the existing bitcoin-pulse-db volume because PostgreSQL data directories are not major-version compatible.

Before approving a major upgrade, prepare a version-specific plan that:

1. Defines a maintenance window and stops the Bitcoin Pulse backend from accepting writes during the final export.
2. Creates and verifies a fresh logical backup with the old PostgreSQL version.
3. Provisions a new, separately named Podman volume for the new major version.
4. Initializes the administrator, migration, and application roles in the new cluster without changing or deleting the old volume.
5. Restores the logical dump into the new cluster before normal backend startup.
6. Runs Migratus, starts the pinned backend, and repeats database, endpoint, collector, ingress, and backup verification.
7. Keeps the old volume untouched until the new cluster has been explicitly accepted and a new backup has completed.
8. Documents a version-specific recovery path before removing any old data.

Do not improvise this procedure directly on production. The exact export, restore, and compatibility steps depend on the source and target PostgreSQL versions and require a separately reviewed implementation change.

Secret rotation

PostgreSQL role rotation is documented in the credential-rotation section of bitcoin-pulse-postgresql.md. Rotate one role at a time and synchronize the database role password with its SOPS value during a maintenance window.

The shared backblaze/tobcloud-backup-env credential affects both Bitcoin Pulse and Nextcloud backups. Coordinate its rotation across both services, deploy the new SOPS value once, and verify read, write, and retention access by running both backup jobs during a maintenance window:

sudo systemctl start restic-backups-bitcoin-pulse.service
sudo systemctl start restic-backups-nextcloud.service

sudo systemctl show \
  restic-backups-bitcoin-pulse.service \
  restic-backups-nextcloud.service \
  --property=Id \
  --property=Result \
  --property=ExecMainStatus

sudo restic-bitcoin-pulse snapshots --latest 1
sudo restic-nextcloud snapshots --latest 1

Both units must report Result=success and ExecMainStatus=0, and both repositories must contain a fresh snapshot.

A Restic repository password is independent of the Backblaze application key. Retain each repository password in the approved password manager; losing one makes that repository unreadable.

First-response diagnostics

For an unhealthy deployment, gather state before restarting services:

sudo systemctl status \
  podman-bitcoin-pulse-postgres.service \
  bitcoin-pulse-migrate.service \
  bitcoin-pulse.service \
  nginx.service \
  --no-pager

sudo journalctl \
  -u podman-bitcoin-pulse-postgres.service \
  -u bitcoin-pulse-migrate.service \
  -u bitcoin-pulse.service \
  -u nginx.service \
  -b \
  -n 300 \
  --no-pager

A migration failure intentionally prevents backend startup. Correct the database readiness, migration, or credential error and then restart bitcoin-pulse.service; its dependency starts the idempotent migration unit again. Do not delete the PostgreSQL volume, bypass migrations, expose ports 3000 or 5432 publicly, or paste unredacted logs into an issue.