oibot/Tobserver
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
Tobserver/docs/bitcoin-pulse-postgresql.md
Tobserver Agent 2015aac8e4 BIT-28 Gate backend startup on migrations
2026-07-28 11:35:54 +02:00

9 KiB
Raw Blame History

Bitcoin Pulse PostgreSQL operations

Declared production layout

modules/bitcoin-pulse.nix declares PostgreSQL 18 as the rootless bitcoin-pulse-postgres Podman container. The corresponding system service is podman-bitcoin-pulse-postgres.service.

  • The official PostgreSQL 18.4 Bookworm image is pinned by immutable digest.
  • Podman runs under the dedicated bitcoin-pulse-postgres system account.
  • The bitcoin-pulse-db named volume is mounted at /var/lib/postgresql, the persistent volume location required by PostgreSQL 18 images.
  • Host port 5432 is published only as 127.0.0.1:5432 and is not opened in the NixOS firewall.
  • Podman reports the service ready only after pg_isready succeeds.
  • PostgreSQL receives SIGINT for a clean fast shutdown, with 60 seconds for Podman and 90 seconds for the enclosing systemd service.
  • Automatic image updates are disabled. Update the digest only in a reviewed change after testing backups, startup, and application compatibility.

The named volume belongs to the rootless Podman account and survives container removal, service recreation, and NixOS generation changes. Backup and restore portability is handled separately by BIT-23; do not copy Podman's private storage directly.

Human-managed secrets

Before deploying this module, use sops from an authorized workstation to add three independently generated passwords to secrets/tobserver.yaml:

bitcoin-pulse:
  postgres:
    admin-password: <administrator password>
    migration-password: <migration role password>
    app-password: <application role password>

Commit only the resulting encrypted SOPS payload. Never place a password in a Nix expression, command argument, shell history, log, or pull-request text. The agent change intentionally declares only the secret paths; it does not modify or decrypt secrets/tobserver.yaml.

sops-nix materializes each credential as a restricted runtime file. Before container startup, the service stages private copies owned by PostgreSQL's uid inside the rootless user namespace. The copies are mounted read-only and are removed from the host runtime directory after PostgreSQL reports healthy and again when the unit stops. The application and migration SOPS files remain available only through the dedicated groups of their respective host services.

Empty-database initialization and privileges

On the first start of an empty bitcoin-pulse-db volume, the official image creates:

  • bitcoin_pulse_admin, the bootstrap PostgreSQL superuser;
  • the bitcoin_pulse database;
  • bitcoin_pulse_migration, a non-superuser login that owns the database;
  • bitcoin_pulse_app, a non-superuser login with CONNECT only initially.

The role bootstrap SQL comes from the pinned BitcoinPulseAPI flake input. Migratus' database-foundation migration then grants the application role schema usage and default DML privileges for application objects. Later migrations can further restrict individual tables; the application role cannot create schema objects, roles, or databases and must never run migrations.

Initialization scripts run only for an empty volume. Changing a SOPS value does not change a role password in an existing PostgreSQL cluster.

Migration-gated backend startup

bitcoin-pulse.service requires bitcoin-pulse-migrate.service, and the migration service requires the health-gated PostgreSQL container service. The resulting startup order is:

PostgreSQL healthy -> bounded readiness check -> Migratus -> backend

The migration service waits up to 60 seconds for pg_isready, then runs the same packaged backend version's bitcoin-pulse-migrate migrate command as the unprivileged bitcoin-pulse-migration host user. It receives only the migration role and its SOPS password-file path. The backend separately receives the restricted bitcoin_pulse_app role and application password-file path.

The migration unit is a one-shot service without RemainAfterExit, so every new start transaction that pulls it in runs Migratus again. Migratus records completed versions in schema_migrations; already-applied migrations therefore make repeated starts harmless. A readiness timeout, invalid credential, or migration error fails the one-shot unit. Because the backend is ordered after and requires that unit, the backend process is not started after such a failure. Inspect journalctl -u bitcoin-pulse-migrate.service for the bounded readiness message, Migratus error, and systemd exit status.

Deployment and verification

A human must review and merge the change, provision the encrypted keys above, and deploy the resulting NixOS generation manually. The agent does not access or rebuild Tobserver.

After deployment, verify the service and loopback binding:

systemctl status podman-bitcoin-pulse-postgres.service
ss -ltn '( sport = :5432 )'
journalctl -u podman-bitcoin-pulse-postgres.service --since today

ss must show only 127.0.0.1:5432. No credential should appear in the unit's command line or journal.

Connect with psql -h 127.0.0.1 -U <role> -d bitcoin_pulse; allow psql to prompt for the password. As the administrator, inspect role attributes with \du and verify:

SELECT has_database_privilege(
  'bitcoin_pulse_app', 'bitcoin_pulse', 'CONNECT');
SELECT has_database_privilege(
  'bitcoin_pulse_app', 'bitcoin_pulse', 'CREATE');
SELECT has_schema_privilege(
  'bitcoin_pulse_app', 'public', 'CREATE');

The expected results are true, false, and false. Verify the application can perform only the DML allowed by the migrated schema and that only the migration role can apply DDL.

To test recreation and persistence, create a temporary probe table as the administrator or migration role, restart the unit, confirm the row remains, and then drop the probe table:

CREATE TABLE bit29_recreation_probe (value integer PRIMARY KEY);
INSERT INTO bit29_recreation_probe VALUES (1);

systemctl restart podman-bitcoin-pulse-postgres.service

TABLE bit29_recreation_probe;
DROP TABLE bit29_recreation_probe;

A normal systemctl stop must complete within 90 seconds. Review the journal to confirm a clean fast PostgreSQL shutdown and no forced kill. Start the unit again and confirm it becomes healthy. Perform these production checks during a reviewed maintenance window.

After deploying the migration gate, verify the one-shot and backend:

systemctl is-active podman-bitcoin-pulse-postgres.service
systemctl is-active bitcoin-pulse.service
systemctl show bitcoin-pulse-migrate.service \
  --property=Result --property=ExecMainStatus
journalctl -u bitcoin-pulse-migrate.service -b --no-pager
ss -ltn '( sport = :3000 )'

A successful migration unit becomes inactive after it exits; Result=success and ExecMainStatus=0 are expected. The backend must be active and port 3000 must be bound only to 127.0.0.1. Restart bitcoin-pulse.service once and confirm the migration unit runs successfully again without reapplying completed migrations.

Successful and repeated paths may be verified during deployment. Exercise the database-unavailable and invalid-migration-credential paths only on a disposable NixOS test host: both must make bitcoin-pulse-migrate.service fail, leave bitcoin-pulse.service stopped, finish within the configured timeout, and emit an actionable journal message. Never inject invalid credentials into production.

Rollback expectations

A NixOS generation rollback changes the application package and service configuration but does not reverse migrations in PostgreSQL. Do not use bitcoin-pulse-migrate rollback as an automatic deployment action. Every schema change must remain compatible with the previous application generation. Use an expand/migrate/contract sequence for breaking changes, and perform any reviewed manual rollback only after assessing data loss and application compatibility.

Credential rotation for an existing database

Rotate one role at a time during a maintenance window. Keep all plaintext in an approved password manager and interactive prompts.

  1. Stop the backend and migration units that use the role being rotated. Leave PostgreSQL running.
  2. Connect over loopback as bitcoin_pulse_admin; enter the current admin password at the psql prompt.
  3. Run \password bitcoin_pulse_app, \password bitcoin_pulse_migration, or \password bitcoin_pulse_admin as appropriate. Enter the newly generated value only at the hidden prompts.
  4. Update the matching value in secrets/tobserver.yaml with sops, commit only the encrypted result, and deploy that reviewed NixOS generation.
  5. Restart the affected client unit and verify it connects. For administrator rotation, restart PostgreSQL once to verify the synchronized bootstrap file still permits normal startup of the existing cluster.
  6. Revoke the old value in the password manager and record the rotation date.

If client verification fails, keep the client stopped, reconnect as the administrator, and correct either the database role or encrypted runtime value. Do not reset or remove bitcoin-pulse-db: first-start initialization is not a credential-rotation mechanism.