250 lines
9 KiB
Markdown
250 lines
9 KiB
Markdown
# Bitcoin Pulse production operations
|
|
|
|
This runbook covers routine Bitcoin Pulse deployment and maintenance on
|
|
Tobserver. The declarative configuration lives in `modules/bitcoin-pulse.nix`;
|
|
PostgreSQL details are in `bitcoin-pulse-postgresql.md`, and public ingress
|
|
checks are in `bitcoin-pulse-ingress.md`.
|
|
|
|
Run production commands from the Tobserver checkout unless noted otherwise.
|
|
Never place plaintext credentials in Git, the Nix store, command arguments,
|
|
shell history, logs, or issue comments.
|
|
|
|
## Standard deployment
|
|
|
|
Prepare and review every change on a workstation before touching the server:
|
|
|
|
```console
|
|
nix flake check --no-build
|
|
nix build .#nixosConfigurations.tobserver.config.system.build.toplevel
|
|
```
|
|
|
|
The second command can run only on a compatible Linux builder. It is optional
|
|
on a workstation without one because the server build below performs the same
|
|
full configuration build before activation.
|
|
|
|
Commit and push the reviewed change. On Tobserver, update the clean checkout and
|
|
build without changing the running system:
|
|
|
|
```console
|
|
cd ~/tobserver
|
|
test -z "$(git status --porcelain)" || {
|
|
echo "Refusing to deploy from a dirty checkout" >&2
|
|
exit 1
|
|
}
|
|
git pull --ff-only
|
|
git rev-parse HEAD
|
|
nix flake check --no-build
|
|
sudo nixos-rebuild build --flake .#tobserver
|
|
```
|
|
|
|
The cleanliness check stops the procedure before pulling when the checkout has
|
|
local changes. Compare the printed commit with the approved deployment commit.
|
|
Review build failures before continuing. A successful `build` creates the
|
|
ignored `result` symlink but does not restart any service.
|
|
|
|
Activate the already-reviewed configuration:
|
|
|
|
```console
|
|
sudo nixos-rebuild switch --flake .#tobserver
|
|
```
|
|
|
|
Do not use a live full-host NixOS generation rollback as a Bitcoin Pulse test or
|
|
routine recovery action. Tobserver runs unrelated production services that a
|
|
full generation switch can also change.
|
|
|
|
## Post-deployment verification
|
|
|
|
Verify service state, migration completion, loopback-only listeners, and local
|
|
responses:
|
|
|
|
```console
|
|
sudo systemctl is-active \
|
|
podman-bitcoin-pulse-postgres.service \
|
|
bitcoin-pulse.service \
|
|
nginx.service
|
|
|
|
sudo systemctl show bitcoin-pulse-migrate.service \
|
|
--property=ActiveState \
|
|
--property=Result \
|
|
--property=ExecMainStatus
|
|
|
|
sudo ss -ltnp | grep -E ':(3000|5432)\b'
|
|
|
|
curl --fail --show-error --silent http://127.0.0.1:3000/health \
|
|
| jq -e '.status == "OK"'
|
|
curl --fail --show-error --silent http://127.0.0.1:3000/block-height \
|
|
| jq -e .
|
|
curl --fail --show-error --silent http://127.0.0.1:3000/difficulty-adjustment \
|
|
| jq -e .
|
|
curl --fail --show-error --silent http://127.0.0.1:3000/price \
|
|
| jq -e .
|
|
```
|
|
|
|
All three long-running services must be active. The migration unit is a
|
|
completed one-shot, so `ActiveState=inactive`, `Result=success`, and
|
|
`ExecMainStatus=0` are expected. PostgreSQL and the backend must listen only on
|
|
`127.0.0.1:5432` and `127.0.0.1:3000` respectively.
|
|
|
|
From a machine outside Tobserver, run the public checks in
|
|
`bitcoin-pulse-ingress.md`. Inspect relevant logs without copying credentials:
|
|
|
|
```console
|
|
sudo journalctl \
|
|
-u podman-bitcoin-pulse-postgres.service \
|
|
-u bitcoin-pulse-migrate.service \
|
|
-u bitcoin-pulse.service \
|
|
-u nginx.service \
|
|
--since "-15 minutes" \
|
|
--no-pager
|
|
```
|
|
|
|
## Updating the Bitcoin Pulse backend
|
|
|
|
Update only the pinned backend input from a workstation:
|
|
|
|
```console
|
|
cd ~/Code/Tobserver
|
|
nix flake update bitcoin-pulse
|
|
git diff -- flake.lock
|
|
nix flake check --no-build
|
|
```
|
|
|
|
Confirm that `flake.lock` points to the intended reviewed BitcoinPulse
|
|
revision. Commit and push the lock-file update, follow the standard deployment,
|
|
and perform all post-deployment checks. The switch runs the packaged version's
|
|
pending migrations before starting that backend version.
|
|
|
|
Every database migration must remain compatible with the previously deployed
|
|
application while it is still a possible recovery version. Do not run
|
|
`bitcoin-pulse-migrate rollback` automatically.
|
|
|
|
## Backups
|
|
|
|
The daily logical backup, retention policy, credentials, and initial
|
|
verification procedure are documented in `bitcoin-pulse-postgresql.md`.
|
|
|
|
Before a database image change or other maintenance with data risk, create and
|
|
verify a fresh snapshot:
|
|
|
|
```console
|
|
sudo systemctl start restic-backups-bitcoin-pulse.service
|
|
sudo systemctl show restic-backups-bitcoin-pulse.service \
|
|
--property=Result \
|
|
--property=ExecMainStatus
|
|
sudo restic-bitcoin-pulse snapshots --latest 1
|
|
sudo restic-bitcoin-pulse ls latest | grep bitcoin-pulse.dump
|
|
sudo find /var/backup/bitcoin-pulse -maxdepth 1 -type f -ls
|
|
```
|
|
|
|
Expect `Result=success`, `ExecMainStatus=0`, a snapshot containing
|
|
`bitcoin-pulse.dump`, and no local staging file after completion. Alerting and
|
|
routine restore tests are separate backlog work.
|
|
|
|
## PostgreSQL image updates within the current major version
|
|
|
|
Automatic image updates are disabled. For a PostgreSQL 18 patch or minor image
|
|
update:
|
|
|
|
1. Read the official PostgreSQL image release notes and confirm the data
|
|
directory remains compatible.
|
|
2. Create and verify a fresh Bitcoin Pulse backup.
|
|
3. Test the candidate image with the backend package and migrations outside
|
|
production where practical.
|
|
4. Verify that the candidate digest identifies the intended PostgreSQL 18
|
|
Bookworm patch release, then replace the immutable digest in
|
|
`modules/bitcoin-pulse.nix`.
|
|
5. Update the adjacent exact-version and tested-date comment and the declared
|
|
version in `bitcoin-pulse-postgresql.md` in the same change.
|
|
6. Run `nixpkgs-fmt --check modules/bitcoin-pulse.nix` and
|
|
`nix flake check --no-build`.
|
|
7. Review, commit, push, and follow the standard deployment procedure.
|
|
8. Repeat all post-deployment and backup checks.
|
|
|
|
Never enable Podman automatic updates or deploy a floating image tag.
|
|
|
|
## PostgreSQL major upgrades
|
|
|
|
A major PostgreSQL upgrade is a separate maintenance project, not a routine
|
|
image-digest change. A new major version must never start against the existing
|
|
`bitcoin-pulse-db` volume because PostgreSQL data directories are not
|
|
major-version compatible.
|
|
|
|
Before approving a major upgrade, prepare a version-specific plan that:
|
|
|
|
1. Defines a maintenance window and stops the Bitcoin Pulse backend from
|
|
accepting writes during the final export.
|
|
2. Creates and verifies a fresh logical backup with the old PostgreSQL version.
|
|
3. Provisions a new, separately named Podman volume for the new major version.
|
|
4. Initializes the administrator, migration, and application roles in the new
|
|
cluster without changing or deleting the old volume.
|
|
5. Restores the logical dump into the new cluster before normal backend startup.
|
|
6. Runs Migratus, starts the pinned backend, and repeats database, endpoint,
|
|
collector, ingress, and backup verification.
|
|
7. Keeps the old volume untouched until the new cluster has been explicitly
|
|
accepted and a new backup has completed.
|
|
8. Documents a version-specific recovery path before removing any old data.
|
|
|
|
Do not improvise this procedure directly on production. The exact export,
|
|
restore, and compatibility steps depend on the source and target PostgreSQL
|
|
versions and require a separately reviewed implementation change.
|
|
|
|
## Secret rotation
|
|
|
|
PostgreSQL role rotation is documented in the credential-rotation section of
|
|
`bitcoin-pulse-postgresql.md`. Rotate one role at a time and synchronize the
|
|
database role password with its SOPS value during a maintenance window.
|
|
|
|
The shared `backblaze/tobcloud-backup-env` credential affects both Bitcoin Pulse
|
|
and Nextcloud backups. Coordinate its rotation across both services, deploy the
|
|
new SOPS value once, and verify read, write, and retention access by running
|
|
both backup jobs during a maintenance window:
|
|
|
|
```console
|
|
sudo systemctl start restic-backups-bitcoin-pulse.service
|
|
sudo systemctl start restic-backups-nextcloud.service
|
|
|
|
sudo systemctl show \
|
|
restic-backups-bitcoin-pulse.service \
|
|
restic-backups-nextcloud.service \
|
|
--property=Id \
|
|
--property=Result \
|
|
--property=ExecMainStatus
|
|
|
|
sudo restic-bitcoin-pulse snapshots --latest 1
|
|
sudo restic-nextcloud snapshots --latest 1
|
|
```
|
|
|
|
Both units must report `Result=success` and `ExecMainStatus=0`, and both
|
|
repositories must contain a fresh snapshot.
|
|
|
|
A Restic repository password is independent of the Backblaze application key.
|
|
Retain each repository password in the approved password manager; losing one
|
|
makes that repository unreadable.
|
|
|
|
## First-response diagnostics
|
|
|
|
For an unhealthy deployment, gather state before restarting services:
|
|
|
|
```console
|
|
sudo systemctl status \
|
|
podman-bitcoin-pulse-postgres.service \
|
|
bitcoin-pulse-migrate.service \
|
|
bitcoin-pulse.service \
|
|
nginx.service \
|
|
--no-pager
|
|
|
|
sudo journalctl \
|
|
-u podman-bitcoin-pulse-postgres.service \
|
|
-u bitcoin-pulse-migrate.service \
|
|
-u bitcoin-pulse.service \
|
|
-u nginx.service \
|
|
-b \
|
|
-n 300 \
|
|
--no-pager
|
|
```
|
|
|
|
A migration failure intentionally prevents backend startup. Correct the
|
|
database readiness, migration, or credential error and then restart
|
|
`bitcoin-pulse.service`; its dependency starts the idempotent migration unit
|
|
again. Do not delete the PostgreSQL volume, bypass migrations, expose ports 3000
|
|
or 5432 publicly, or paste unredacted logs into an issue.
|