oibot/Tobserver
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

Add Bitcoin Pulse operations runbook

Browse source
This commit is contained in:
Tobias Ostner 2026-07-30 13:45:51 +07:00
parent 81fea16376
commit 1bdf25bf06
3 changed files with 271 additions and 16 deletions
Download patch file Download diff file Expand all files Collapse all files

16
docs/bitcoin-pulse-ingress.md
View file

@ -38,9 +38,9 @@ generation and verify locally on Tobserver:
```console ```console
systemctl is-active nginx.service bitcoin-pulse.service systemctl is-active nginx.service bitcoin-pulse.service
nginx -t
ss -ltn '( sport = :3000 )' ss -ltn '( sport = :3000 )'
curl --fail --silent http://127.0.0.1:3000/health | jq -e '.status == "OK"' curl --fail --show-error --silent http://127.0.0.1:3000/health \
| jq -e '.status == "OK"'
``` ```
The socket listing must show only `127.0.0.1:3000`, not `0.0.0.0:3000` or the The socket listing must show only `127.0.0.1:3000`, not `0.0.0.0:3000` or the
@ -49,11 +49,15 @@ endpoint, and each functional endpoint:
```console ```console
dig +short A bitcoin-pulse.tobiasostner.de dig +short A bitcoin-pulse.tobiasostner.de
curl --fail --silent https://bitcoin-pulse.tobiasostner.de/health \ curl --fail --show-error --silent \
https://bitcoin-pulse.tobiasostner.de/health \
| jq -e '.status == "OK"' | jq -e '.status == "OK"'
curl --fail --silent https://bitcoin-pulse.tobiasostner.de/block-height | jq curl --fail --show-error --silent \
curl --fail --silent https://bitcoin-pulse.tobiasostner.de/difficulty-adjustment | jq https://bitcoin-pulse.tobiasostner.de/block-height | jq -e .
curl --fail --silent https://bitcoin-pulse.tobiasostner.de/price | jq curl --fail --show-error --silent \
https://bitcoin-pulse.tobiasostner.de/difficulty-adjustment | jq -e .
curl --fail --show-error --silent \
https://bitcoin-pulse.tobiasostner.de/price | jq -e .
``` ```
The A lookup must return `162.55.103.218`. The HTTPS certificate must be valid The A lookup must return `162.55.103.218`. The HTTPS certificate must be valid

250
docs/bitcoin-pulse-operations.md Normal file
View file

@ -0,0 +1,250 @@
# Bitcoin Pulse production operations
This runbook covers routine Bitcoin Pulse deployment and maintenance on
Tobserver. The declarative configuration lives in `modules/bitcoin-pulse.nix`;
PostgreSQL details are in `bitcoin-pulse-postgresql.md`, and public ingress
checks are in `bitcoin-pulse-ingress.md`.
Run production commands from the Tobserver checkout unless noted otherwise.
Never place plaintext credentials in Git, the Nix store, command arguments,
shell history, logs, or issue comments.
## Standard deployment
Prepare and review every change on a workstation before touching the server:
```console
nix flake check --no-build
nix build .#nixosConfigurations.tobserver.config.system.build.toplevel
```
The second command can run only on a compatible Linux builder. It is optional
on a workstation without one because the server build below performs the same
full configuration build before activation.
Commit and push the reviewed change. On Tobserver, update the clean checkout and
build without changing the running system:
```console
cd ~/tobserver
test -z "$(git status --porcelain)" || {
echo "Refusing to deploy from a dirty checkout" >&2
exit 1
}
git pull --ff-only
git rev-parse HEAD
nix flake check --no-build
sudo nixos-rebuild build --flake .#tobserver
```
The cleanliness check stops the procedure before pulling when the checkout has
local changes. Compare the printed commit with the approved deployment commit.
Review build failures before continuing. A successful `build` creates the
ignored `result` symlink but does not restart any service.
Activate the already-reviewed configuration:
```console
sudo nixos-rebuild switch --flake .#tobserver
```
Do not use a live full-host NixOS generation rollback as a Bitcoin Pulse test or
routine recovery action. Tobserver runs unrelated production services that a
full generation switch can also change.
## Post-deployment verification
Verify service state, migration completion, loopback-only listeners, and local
responses:
```console
sudo systemctl is-active \
podman-bitcoin-pulse-postgres.service \
bitcoin-pulse.service \
nginx.service
sudo systemctl show bitcoin-pulse-migrate.service \
--property=ActiveState \
--property=Result \
--property=ExecMainStatus
sudo ss -ltnp | grep -E ':(3000|5432)\b'
curl --fail --show-error --silent http://127.0.0.1:3000/health \
| jq -e '.status == "OK"'
curl --fail --show-error --silent http://127.0.0.1:3000/block-height \
| jq -e .
curl --fail --show-error --silent http://127.0.0.1:3000/difficulty-adjustment \
| jq -e .
curl --fail --show-error --silent http://127.0.0.1:3000/price \
| jq -e .
```
All three long-running services must be active. The migration unit is a
completed one-shot, so `ActiveState=inactive`, `Result=success`, and
`ExecMainStatus=0` are expected. PostgreSQL and the backend must listen only on
`127.0.0.1:5432` and `127.0.0.1:3000` respectively.
From a machine outside Tobserver, run the public checks in
`bitcoin-pulse-ingress.md`. Inspect relevant logs without copying credentials:
```console
sudo journalctl \
-u podman-bitcoin-pulse-postgres.service \
-u bitcoin-pulse-migrate.service \
-u bitcoin-pulse.service \
-u nginx.service \
--since "-15 minutes" \
--no-pager
```
## Updating the Bitcoin Pulse backend
Update only the pinned backend input from a workstation:
```console
cd ~/Code/Tobserver
nix flake update bitcoin-pulse
git diff -- flake.lock
nix flake check --no-build
```
Confirm that `flake.lock` points to the intended reviewed BitcoinPulseAPI
revision. Commit and push the lock-file update, follow the standard deployment,
and perform all post-deployment checks. The switch runs the packaged version's
pending migrations before starting that backend version.
Every database migration must remain compatible with the previously deployed
application while it is still a possible recovery version. Do not run
`bitcoin-pulse-migrate rollback` automatically.
## Backups
The daily logical backup, retention policy, credentials, and initial
verification procedure are documented in `bitcoin-pulse-postgresql.md`.
Before a database image change or other maintenance with data risk, create and
verify a fresh snapshot:
```console
sudo systemctl start restic-backups-bitcoin-pulse.service
sudo systemctl show restic-backups-bitcoin-pulse.service \
--property=Result \
--property=ExecMainStatus
sudo restic-bitcoin-pulse snapshots --latest 1
sudo restic-bitcoin-pulse ls latest | grep bitcoin-pulse.dump
sudo find /var/backup/bitcoin-pulse -maxdepth 1 -type f -ls
```
Expect `Result=success`, `ExecMainStatus=0`, a snapshot containing
`bitcoin-pulse.dump`, and no local staging file after completion. Alerting and
routine restore tests are separate backlog work.
## PostgreSQL image updates within the current major version
Automatic image updates are disabled. For a PostgreSQL 18 patch or minor image
update:
1. Read the official PostgreSQL image release notes and confirm the data
directory remains compatible.
2. Create and verify a fresh Bitcoin Pulse backup.
3. Test the candidate image with the backend package and migrations outside
production where practical.
4. Verify that the candidate digest identifies the intended PostgreSQL 18
Bookworm patch release, then replace the immutable digest in
`modules/bitcoin-pulse.nix`.
5. Update the adjacent exact-version and tested-date comment and the declared
version in `bitcoin-pulse-postgresql.md` in the same change.
6. Run `nixpkgs-fmt --check modules/bitcoin-pulse.nix` and
`nix flake check --no-build`.
7. Review, commit, push, and follow the standard deployment procedure.
8. Repeat all post-deployment and backup checks.
Never enable Podman automatic updates or deploy a floating image tag.
## PostgreSQL major upgrades
A major PostgreSQL upgrade is a separate maintenance project, not a routine
image-digest change. A new major version must never start against the existing
`bitcoin-pulse-db` volume because PostgreSQL data directories are not
major-version compatible.
Before approving a major upgrade, prepare a version-specific plan that:
1. Defines a maintenance window and stops the Bitcoin Pulse backend from
accepting writes during the final export.
2. Creates and verifies a fresh logical backup with the old PostgreSQL version.
3. Provisions a new, separately named Podman volume for the new major version.
4. Initializes the administrator, migration, and application roles in the new
cluster without changing or deleting the old volume.
5. Restores the logical dump into the new cluster before normal backend startup.
6. Runs Migratus, starts the pinned backend, and repeats database, endpoint,
collector, ingress, and backup verification.
7. Keeps the old volume untouched until the new cluster has been explicitly
accepted and a new backup has completed.
8. Documents a version-specific recovery path before removing any old data.
Do not improvise this procedure directly on production. The exact export,
restore, and compatibility steps depend on the source and target PostgreSQL
versions and require a separately reviewed implementation change.
## Secret rotation
PostgreSQL role rotation is documented in the credential-rotation section of
`bitcoin-pulse-postgresql.md`. Rotate one role at a time and synchronize the
database role password with its SOPS value during a maintenance window.
The shared `backblaze/tobcloud-backup-env` credential affects both Bitcoin Pulse
and Nextcloud backups. Coordinate its rotation across both services, deploy the
new SOPS value once, and verify read, write, and retention access by running
both backup jobs during a maintenance window:
```console
sudo systemctl start restic-backups-bitcoin-pulse.service
sudo systemctl start restic-backups-nextcloud.service
sudo systemctl show \
restic-backups-bitcoin-pulse.service \
restic-backups-nextcloud.service \
--property=Id \
--property=Result \
--property=ExecMainStatus
sudo restic-bitcoin-pulse snapshots --latest 1
sudo restic-nextcloud snapshots --latest 1
```
Both units must report `Result=success` and `ExecMainStatus=0`, and both
repositories must contain a fresh snapshot.
A Restic repository password is independent of the Backblaze application key.
Retain each repository password in the approved password manager; losing one
makes that repository unreadable.
## First-response diagnostics
For an unhealthy deployment, gather state before restarting services:
```console
sudo systemctl status \
podman-bitcoin-pulse-postgres.service \
bitcoin-pulse-migrate.service \
bitcoin-pulse.service \
nginx.service \
--no-pager
sudo journalctl \
-u podman-bitcoin-pulse-postgres.service \
-u bitcoin-pulse-migrate.service \
-u bitcoin-pulse.service \
-u nginx.service \
-b \
-n 300 \
--no-pager
```
A migration failure intentionally prevents backend startup. Correct the
database readiness, migration, or credential error and then restart
`bitcoin-pulse.service`; its dependency starts the idempotent migration unit
again. Do not delete the PostgreSQL volume, bypass migrations, expose ports 3000
or 5432 publicly, or paste unredacted logs into an issue.

21
docs/bitcoin-pulse-postgresql.md
View file

@ -162,9 +162,9 @@ message, Migratus error, and systemd exit status.
## Deployment and verification ## Deployment and verification
A human must review and merge the change, provision the encrypted keys above, Use `bitcoin-pulse-operations.md` for the standard build, switch, and complete
and deploy the resulting NixOS generation manually. The agent does not access post-deployment procedure. The checks below cover the database-specific
or rebuild Tobserver. acceptance paths.
After deployment, verify the service and loopback binding: After deployment, verify the service and loopback binding:
@ -240,14 +240,15 @@ NixOS test host: both must make `bitcoin-pulse-migrate.service` fail, leave
`bitcoin-pulse.service` stopped, finish within the configured timeout, and emit `bitcoin-pulse.service` stopped, finish within the configured timeout, and emit
an actionable journal message. Never inject invalid credentials into production. an actionable journal message. Never inject invalid credentials into production.
## Rollback expectations ## Application compatibility constraints
A NixOS generation rollback changes the application package and service A full-host NixOS generation rollback is not part of the Bitcoin Pulse
configuration but does not reverse migrations in PostgreSQL. Do not use production procedure because it can affect unrelated Tobserver services and it
`bitcoin-pulse-migrate rollback` as an automatic deployment action. Every schema does not reverse PostgreSQL migrations. Do not use `bitcoin-pulse-migrate
change must remain compatible with the previous application generation. Use an rollback` as an automatic deployment action. Every schema change must remain
expand/migrate/contract sequence for breaking changes, and perform any reviewed compatible with the previously deployed application while that version remains
manual rollback only after assessing data loss and application compatibility. a possible recovery version. Use an expand/migrate/contract sequence for
breaking changes.
## Credential rotation for an existing database ## Credential rotation for an existing database