diff --git a/docs/bitcoin-pulse-ingress.md b/docs/bitcoin-pulse-ingress.md index cc438ef..84740aa 100644 --- a/docs/bitcoin-pulse-ingress.md +++ b/docs/bitcoin-pulse-ingress.md @@ -38,9 +38,9 @@ generation and verify locally on Tobserver: ```console systemctl is-active nginx.service bitcoin-pulse.service -nginx -t ss -ltn '( sport = :3000 )' -curl --fail --silent http://127.0.0.1:3000/health | jq -e '.status == "OK"' +curl --fail --show-error --silent http://127.0.0.1:3000/health \ + | jq -e '.status == "OK"' ``` The socket listing must show only `127.0.0.1:3000`, not `0.0.0.0:3000` or the @@ -49,11 +49,15 @@ endpoint, and each functional endpoint: ```console dig +short A bitcoin-pulse.tobiasostner.de -curl --fail --silent https://bitcoin-pulse.tobiasostner.de/health \ +curl --fail --show-error --silent \ + https://bitcoin-pulse.tobiasostner.de/health \ | jq -e '.status == "OK"' -curl --fail --silent https://bitcoin-pulse.tobiasostner.de/block-height | jq -curl --fail --silent https://bitcoin-pulse.tobiasostner.de/difficulty-adjustment | jq -curl --fail --silent https://bitcoin-pulse.tobiasostner.de/price | jq +curl --fail --show-error --silent \ + https://bitcoin-pulse.tobiasostner.de/block-height | jq -e . +curl --fail --show-error --silent \ + https://bitcoin-pulse.tobiasostner.de/difficulty-adjustment | jq -e . +curl --fail --show-error --silent \ + https://bitcoin-pulse.tobiasostner.de/price | jq -e . ``` The A lookup must return `162.55.103.218`. The HTTPS certificate must be valid diff --git a/docs/bitcoin-pulse-operations.md b/docs/bitcoin-pulse-operations.md new file mode 100644 index 0000000..eaad2b2 --- /dev/null +++ b/docs/bitcoin-pulse-operations.md @@ -0,0 +1,250 @@ +# Bitcoin Pulse production operations + +This runbook covers routine Bitcoin Pulse deployment and maintenance on +Tobserver. The declarative configuration lives in `modules/bitcoin-pulse.nix`; +PostgreSQL details are in `bitcoin-pulse-postgresql.md`, and public ingress +checks are in `bitcoin-pulse-ingress.md`. + +Run production commands from the Tobserver checkout unless noted otherwise. +Never place plaintext credentials in Git, the Nix store, command arguments, +shell history, logs, or issue comments. + +## Standard deployment + +Prepare and review every change on a workstation before touching the server: + +```console +nix flake check --no-build +nix build .#nixosConfigurations.tobserver.config.system.build.toplevel +``` + +The second command can run only on a compatible Linux builder. It is optional +on a workstation without one because the server build below performs the same +full configuration build before activation. + +Commit and push the reviewed change. On Tobserver, update the clean checkout and +build without changing the running system: + +```console +cd ~/tobserver +test -z "$(git status --porcelain)" || { + echo "Refusing to deploy from a dirty checkout" >&2 + exit 1 +} +git pull --ff-only +git rev-parse HEAD +nix flake check --no-build +sudo nixos-rebuild build --flake .#tobserver +``` + +The cleanliness check stops the procedure before pulling when the checkout has +local changes. Compare the printed commit with the approved deployment commit. +Review build failures before continuing. A successful `build` creates the +ignored `result` symlink but does not restart any service. + +Activate the already-reviewed configuration: + +```console +sudo nixos-rebuild switch --flake .#tobserver +``` + +Do not use a live full-host NixOS generation rollback as a Bitcoin Pulse test or +routine recovery action. Tobserver runs unrelated production services that a +full generation switch can also change. + +## Post-deployment verification + +Verify service state, migration completion, loopback-only listeners, and local +responses: + +```console +sudo systemctl is-active \ + podman-bitcoin-pulse-postgres.service \ + bitcoin-pulse.service \ + nginx.service + +sudo systemctl show bitcoin-pulse-migrate.service \ + --property=ActiveState \ + --property=Result \ + --property=ExecMainStatus + +sudo ss -ltnp | grep -E ':(3000|5432)\b' + +curl --fail --show-error --silent http://127.0.0.1:3000/health \ + | jq -e '.status == "OK"' +curl --fail --show-error --silent http://127.0.0.1:3000/block-height \ + | jq -e . +curl --fail --show-error --silent http://127.0.0.1:3000/difficulty-adjustment \ + | jq -e . +curl --fail --show-error --silent http://127.0.0.1:3000/price \ + | jq -e . +``` + +All three long-running services must be active. The migration unit is a +completed one-shot, so `ActiveState=inactive`, `Result=success`, and +`ExecMainStatus=0` are expected. PostgreSQL and the backend must listen only on +`127.0.0.1:5432` and `127.0.0.1:3000` respectively. + +From a machine outside Tobserver, run the public checks in +`bitcoin-pulse-ingress.md`. Inspect relevant logs without copying credentials: + +```console +sudo journalctl \ + -u podman-bitcoin-pulse-postgres.service \ + -u bitcoin-pulse-migrate.service \ + -u bitcoin-pulse.service \ + -u nginx.service \ + --since "-15 minutes" \ + --no-pager +``` + +## Updating the Bitcoin Pulse backend + +Update only the pinned backend input from a workstation: + +```console +cd ~/Code/Tobserver +nix flake update bitcoin-pulse +git diff -- flake.lock +nix flake check --no-build +``` + +Confirm that `flake.lock` points to the intended reviewed BitcoinPulseAPI +revision. Commit and push the lock-file update, follow the standard deployment, +and perform all post-deployment checks. The switch runs the packaged version's +pending migrations before starting that backend version. + +Every database migration must remain compatible with the previously deployed +application while it is still a possible recovery version. Do not run +`bitcoin-pulse-migrate rollback` automatically. + +## Backups + +The daily logical backup, retention policy, credentials, and initial +verification procedure are documented in `bitcoin-pulse-postgresql.md`. + +Before a database image change or other maintenance with data risk, create and +verify a fresh snapshot: + +```console +sudo systemctl start restic-backups-bitcoin-pulse.service +sudo systemctl show restic-backups-bitcoin-pulse.service \ + --property=Result \ + --property=ExecMainStatus +sudo restic-bitcoin-pulse snapshots --latest 1 +sudo restic-bitcoin-pulse ls latest | grep bitcoin-pulse.dump +sudo find /var/backup/bitcoin-pulse -maxdepth 1 -type f -ls +``` + +Expect `Result=success`, `ExecMainStatus=0`, a snapshot containing +`bitcoin-pulse.dump`, and no local staging file after completion. Alerting and +routine restore tests are separate backlog work. + +## PostgreSQL image updates within the current major version + +Automatic image updates are disabled. For a PostgreSQL 18 patch or minor image +update: + +1. Read the official PostgreSQL image release notes and confirm the data + directory remains compatible. +2. Create and verify a fresh Bitcoin Pulse backup. +3. Test the candidate image with the backend package and migrations outside + production where practical. +4. Verify that the candidate digest identifies the intended PostgreSQL 18 + Bookworm patch release, then replace the immutable digest in + `modules/bitcoin-pulse.nix`. +5. Update the adjacent exact-version and tested-date comment and the declared + version in `bitcoin-pulse-postgresql.md` in the same change. +6. Run `nixpkgs-fmt --check modules/bitcoin-pulse.nix` and + `nix flake check --no-build`. +7. Review, commit, push, and follow the standard deployment procedure. +8. Repeat all post-deployment and backup checks. + +Never enable Podman automatic updates or deploy a floating image tag. + +## PostgreSQL major upgrades + +A major PostgreSQL upgrade is a separate maintenance project, not a routine +image-digest change. A new major version must never start against the existing +`bitcoin-pulse-db` volume because PostgreSQL data directories are not +major-version compatible. + +Before approving a major upgrade, prepare a version-specific plan that: + +1. Defines a maintenance window and stops the Bitcoin Pulse backend from + accepting writes during the final export. +2. Creates and verifies a fresh logical backup with the old PostgreSQL version. +3. Provisions a new, separately named Podman volume for the new major version. +4. Initializes the administrator, migration, and application roles in the new + cluster without changing or deleting the old volume. +5. Restores the logical dump into the new cluster before normal backend startup. +6. Runs Migratus, starts the pinned backend, and repeats database, endpoint, + collector, ingress, and backup verification. +7. Keeps the old volume untouched until the new cluster has been explicitly + accepted and a new backup has completed. +8. Documents a version-specific recovery path before removing any old data. + +Do not improvise this procedure directly on production. The exact export, +restore, and compatibility steps depend on the source and target PostgreSQL +versions and require a separately reviewed implementation change. + +## Secret rotation + +PostgreSQL role rotation is documented in the credential-rotation section of +`bitcoin-pulse-postgresql.md`. Rotate one role at a time and synchronize the +database role password with its SOPS value during a maintenance window. + +The shared `backblaze/tobcloud-backup-env` credential affects both Bitcoin Pulse +and Nextcloud backups. Coordinate its rotation across both services, deploy the +new SOPS value once, and verify read, write, and retention access by running +both backup jobs during a maintenance window: + +```console +sudo systemctl start restic-backups-bitcoin-pulse.service +sudo systemctl start restic-backups-nextcloud.service + +sudo systemctl show \ + restic-backups-bitcoin-pulse.service \ + restic-backups-nextcloud.service \ + --property=Id \ + --property=Result \ + --property=ExecMainStatus + +sudo restic-bitcoin-pulse snapshots --latest 1 +sudo restic-nextcloud snapshots --latest 1 +``` + +Both units must report `Result=success` and `ExecMainStatus=0`, and both +repositories must contain a fresh snapshot. + +A Restic repository password is independent of the Backblaze application key. +Retain each repository password in the approved password manager; losing one +makes that repository unreadable. + +## First-response diagnostics + +For an unhealthy deployment, gather state before restarting services: + +```console +sudo systemctl status \ + podman-bitcoin-pulse-postgres.service \ + bitcoin-pulse-migrate.service \ + bitcoin-pulse.service \ + nginx.service \ + --no-pager + +sudo journalctl \ + -u podman-bitcoin-pulse-postgres.service \ + -u bitcoin-pulse-migrate.service \ + -u bitcoin-pulse.service \ + -u nginx.service \ + -b \ + -n 300 \ + --no-pager +``` + +A migration failure intentionally prevents backend startup. Correct the +database readiness, migration, or credential error and then restart +`bitcoin-pulse.service`; its dependency starts the idempotent migration unit +again. Do not delete the PostgreSQL volume, bypass migrations, expose ports 3000 +or 5432 publicly, or paste unredacted logs into an issue. diff --git a/docs/bitcoin-pulse-postgresql.md b/docs/bitcoin-pulse-postgresql.md index f51199e..3acfcc2 100644 --- a/docs/bitcoin-pulse-postgresql.md +++ b/docs/bitcoin-pulse-postgresql.md @@ -162,9 +162,9 @@ message, Migratus error, and systemd exit status. ## Deployment and verification -A human must review and merge the change, provision the encrypted keys above, -and deploy the resulting NixOS generation manually. The agent does not access -or rebuild Tobserver. +Use `bitcoin-pulse-operations.md` for the standard build, switch, and complete +post-deployment procedure. The checks below cover the database-specific +acceptance paths. After deployment, verify the service and loopback binding: @@ -240,14 +240,15 @@ NixOS test host: both must make `bitcoin-pulse-migrate.service` fail, leave `bitcoin-pulse.service` stopped, finish within the configured timeout, and emit an actionable journal message. Never inject invalid credentials into production. -## Rollback expectations +## Application compatibility constraints -A NixOS generation rollback changes the application package and service -configuration but does not reverse migrations in PostgreSQL. Do not use -`bitcoin-pulse-migrate rollback` as an automatic deployment action. Every schema -change must remain compatible with the previous application generation. Use an -expand/migrate/contract sequence for breaking changes, and perform any reviewed -manual rollback only after assessing data loss and application compatibility. +A full-host NixOS generation rollback is not part of the Bitcoin Pulse +production procedure because it can affect unrelated Tobserver services and it +does not reverse PostgreSQL migrations. Do not use `bitcoin-pulse-migrate +rollback` as an automatic deployment action. Every schema change must remain +compatible with the previously deployed application while that version remains +a possible recovery version. Use an expand/migrate/contract sequence for +breaking changes. ## Credential rotation for an existing database