BIT-29 Provision Bitcoin Pulse PostgreSQL (#5)

## Summary
- Provision a digest-pinned PostgreSQL 18 container through rootless Podman with health-gated systemd readiness, loopback-only publishing, graceful shutdown, and the persistent `bitcoin-pulse-db` volume.
- Declare separate SOPS credential paths for the administrator, migration, and application roles, stage them privately for first-start initialization, and reuse the role bootstrap SQL from the pinned BitcoinPulseAPI input.
- Document human-managed secret provisioning, least-privilege verification, container recreation, graceful shutdown, and coordinated credential rotation.

## Validation
- `tobserver-agent check` passed:
  - `nixpkgs-fmt --check`
  - `git diff --check`
  - `nix flake check --no-build`
  - Tobserver system derivation path evaluation
  - Gitleaks patch scan
- Evaluated the OCI container, systemd service, rootless user, and NixOS assertions; all assertions pass and the generated configuration contains the expected loopback port, health readiness, persistent volume, and shutdown settings.

## Risks and limitations
- Runtime startup, shutdown, persistence, and role behavior require Linux deployment verification and were not executed on macOS.
- The encrypted SOPS keys are intentionally absent from this agent-authored change; the service must not be deployed before a human provisions them.
- Static uid/gid 400 is reserved for the dedicated rootless container account and should be checked for conflicts during review.

## Manual follow-up
- Add `bitcoin-pulse/postgres/admin-password`, `bitcoin-pulse/postgres/migration-password`, and `bitcoin-pulse/postgres/app-password` to `secrets/tobserver.yaml` using `sops`; commit only encrypted values.
- Review and merge this pull request.
- Deploy manually on Tobserver after merge and encrypted secret provisioning.
- Run the startup, loopback binding, role privilege, restart/persistence, and graceful-shutdown checks in `docs/bitcoin-pulse-postgresql.md`.

Reviewed-on: https://codeberg.org/oibot/Tobserver/pulls/5

docs/bitcoin-pulse-postgresql.md

@@ -0,0 +1,149 @@
+# Bitcoin Pulse PostgreSQL operations
+
+## Declared production layout
+
+`modules/bitcoin-pulse.nix` declares PostgreSQL 18 as the rootless
+`bitcoin-pulse-postgres` Podman container. The corresponding system service is
+`podman-bitcoin-pulse-postgres.service`.
+
+- The official PostgreSQL 18.4 Bookworm image is pinned by immutable digest.
+- Podman runs under the dedicated `bitcoin-pulse-postgres` system account.
+- The `bitcoin-pulse-db` named volume is mounted at `/var/lib/postgresql`, the
+  persistent volume location required by PostgreSQL 18 images.
+- Host port 5432 is published only as `127.0.0.1:5432` and is not opened in the
+  NixOS firewall.
+- Podman reports the service ready only after `pg_isready` succeeds.
+- PostgreSQL receives `SIGINT` for smart shutdown, with 60 seconds for Podman
+  and 90 seconds for the enclosing systemd service.
+- Automatic image updates are disabled. Update the digest only in a reviewed
+  change after testing backups, startup, and application compatibility.
+
+The named volume belongs to the rootless Podman account and survives container
+removal, service recreation, and NixOS generation changes. Backup and restore
+portability is handled separately by BIT-23; do not copy Podman's private
+storage directly.
+
+## Human-managed secrets
+
+Before deploying this module, use `sops` from an authorized workstation to add
+three independently generated passwords to `secrets/tobserver.yaml`:
+
+```yaml
+bitcoin-pulse:
+  postgres:
+    admin-password: <administrator password>
+    migration-password: <migration role password>
+    app-password: <application role password>
+```
+
+Commit only the resulting encrypted SOPS payload. Never place a password in a
+Nix expression, command argument, shell history, log, or pull-request text. The
+agent change intentionally declares only the secret paths; it does not modify
+or decrypt `secrets/tobserver.yaml`.
+
+`sops-nix` materializes each credential as a restricted runtime file. Before
+container startup, the service stages private copies owned by PostgreSQL's uid
+inside the rootless user namespace. The copies are mounted read-only and are
+removed from the host runtime directory after PostgreSQL reports healthy and
+again when the unit stops. The application and migration SOPS files remain
+available only through their dedicated host groups for the later BIT-28 units.
+
+## Empty-database initialization and privileges
+
+On the first start of an empty `bitcoin-pulse-db` volume, the official image
+creates:
+
+- `bitcoin_pulse_admin`, the bootstrap PostgreSQL superuser;
+- the `bitcoin_pulse` database;
+- `bitcoin_pulse_migration`, a non-superuser login that owns the database;
+- `bitcoin_pulse_app`, a non-superuser login with `CONNECT` only initially.
+
+The role bootstrap SQL comes from the pinned BitcoinPulseAPI flake input.
+Migratus' database-foundation migration then grants the application role schema
+usage and default DML privileges for application objects. Later migrations can
+further restrict individual tables; the application role cannot create schema
+objects, roles, or databases and must never run migrations.
+
+Initialization scripts run only for an empty volume. Changing a SOPS value does
+not change a role password in an existing PostgreSQL cluster.
+
+## Deployment and verification
+
+A human must review and merge the change, provision the encrypted keys above,
+and deploy the resulting NixOS generation manually. The agent does not access
+or rebuild Tobserver.
+
+After deployment, verify the service and loopback binding:
+
+```console
+systemctl status podman-bitcoin-pulse-postgres.service
+ss -ltn '( sport = :5432 )'
+journalctl -u podman-bitcoin-pulse-postgres.service --since today
+```
+
+`ss` must show only `127.0.0.1:5432`. No credential should appear in the unit's
+command line or journal.
+
+Connect with `psql -h 127.0.0.1 -U <role> -d bitcoin_pulse`; allow `psql` to
+prompt for the password. As the administrator, inspect role attributes with
+`\du` and verify:
+
+```sql
+SELECT has_database_privilege(
+  'bitcoin_pulse_app', 'bitcoin_pulse', 'CONNECT');
+SELECT has_database_privilege(
+  'bitcoin_pulse_app', 'bitcoin_pulse', 'CREATE');
+SELECT has_schema_privilege(
+  'bitcoin_pulse_app', 'public', 'CREATE');
+```
+
+The expected results are `true`, `false`, and `false`. After BIT-28 runs the
+foundation migration, also verify the application can perform only the DML
+allowed by the migrated schema and that only the migration role can apply DDL.
+
+To test recreation and persistence, create a temporary probe table as the
+administrator or migration role, restart the unit, confirm the row remains, and
+then drop the probe table:
+
+```sql
+CREATE TABLE bit29_recreation_probe (value integer PRIMARY KEY);
+INSERT INTO bit29_recreation_probe VALUES (1);
+```
+
+```console
+systemctl restart podman-bitcoin-pulse-postgres.service
+```
+
+```sql
+TABLE bit29_recreation_probe;
+DROP TABLE bit29_recreation_probe;
+```
+
+A normal `systemctl stop` must complete within 90 seconds. Review the journal to
+confirm a smart PostgreSQL shutdown and no forced kill. Start the unit again and
+confirm it becomes healthy. Perform these production checks during a reviewed
+maintenance window.
+
+## Credential rotation for an existing database
+
+Rotate one role at a time during a maintenance window. Keep all plaintext in an
+approved password manager and interactive prompts.
+
+1. Stop the backend and migration units that use the role being rotated. Leave
+   PostgreSQL running.
+2. Connect over loopback as `bitcoin_pulse_admin`; enter the current admin
+   password at the `psql` prompt.
+3. Run `\password bitcoin_pulse_app`, `\password bitcoin_pulse_migration`, or
+   `\password bitcoin_pulse_admin` as appropriate. Enter the newly generated
+   value only at the hidden prompts.
+4. Update the matching value in `secrets/tobserver.yaml` with `sops`, commit only
+   the encrypted result, and deploy that reviewed NixOS generation.
+5. Restart the affected client unit and verify it connects. For administrator
+   rotation, restart PostgreSQL once to verify the synchronized bootstrap file
+   still permits normal startup of the existing cluster.
+6. Revoke the old value in the password manager and record the rotation date.
+
+If client verification fails, keep the client stopped, reconnect as the
+administrator, and correct either the database role or encrypted runtime value.
+Do not reset or remove `bitcoin-pulse-db`: first-start initialization is not a
+credential-rotation mechanism.

modules/bitcoin-pulse.nix

@@ -1,15 +1,165 @@
-{ bitcoin-pulse, ... }:
+{ bitcoin-pulse, config, lib, pkgs, ... }:
 
+let
+  containerName = "bitcoin-pulse-postgres";
+  serviceName = "podman-${containerName}";
+  runtimeDirectory = "/run/${containerName}";
+
+  secretFiles = {
+    admin = config.sops.secrets."bitcoin-pulse/postgres/admin-password".path;
+    app = config.sops.secrets."bitcoin-pulse/postgres/app-password".path;
+    migration = config.sops.secrets."bitcoin-pulse/postgres/migration-password".path;
+  };
+
+  prepareSecrets = pkgs.writeShellApplication {
+    name = "bitcoin-pulse-postgresql-prepare-secrets";
+    runtimeInputs = [
+      config.virtualisation.podman.package
+      pkgs.coreutils
+    ];
+    text = ''
+      copy_secret() {
+        source="$1"
+        destination="$2"
+
+        podman unshare rm -f "$destination"
+        # PostgreSQL runs as uid 999 in the container. Store each runtime copy
+        # with the corresponding rootless subordinate uid and a private mode.
+        podman unshare install -m 0400 -o 999 -g 999 "$source" "$destination"
+      }
+
+      copy_secret ${lib.escapeShellArg secretFiles.admin} \
+        ${lib.escapeShellArg "${runtimeDirectory}/admin-password"}
+      copy_secret ${lib.escapeShellArg secretFiles.app} \
+        ${lib.escapeShellArg "${runtimeDirectory}/app-password"}
+      copy_secret ${lib.escapeShellArg secretFiles.migration} \
+        ${lib.escapeShellArg "${runtimeDirectory}/migration-password"}
+    '';
+  };
+
+  cleanupSecrets = pkgs.writeShellApplication {
+    name = "bitcoin-pulse-postgresql-cleanup-secrets";
+    runtimeInputs = [
+      config.virtualisation.podman.package
+      pkgs.coreutils
+    ];
+    text = ''
+      podman unshare rm -f \
+        ${lib.escapeShellArg "${runtimeDirectory}/admin-password"} \
+        ${lib.escapeShellArg "${runtimeDirectory}/app-password"} \
+        ${lib.escapeShellArg "${runtimeDirectory}/migration-password"}
+    '';
+  };
+in
 {
   imports = [ bitcoin-pulse.nixosModules.default ];
 
   services.bitcoin-pulse = {
-    # BIT-28 and BIT-29 will enable the production instance after PostgreSQL,
+    # BIT-28 will enable the production instance after PostgreSQL readiness
-    # migrations, and credentials have been configured.
+    # and migrations have been integrated with these credentials.
     enable = false;
 
     bindAddress = "127.0.0.1";
     port = 3000;
     mempool.baseUrl = "http://127.0.0.1:8999";
   };
+
+  users.groups = {
+    bitcoin-pulse = { };
+    bitcoin-pulse-migration = { };
+    bitcoin-pulse-postgres.gid = 400;
+  };
+
+  users.users.bitcoin-pulse-postgres = {
+    description = "Rootless PostgreSQL container account for Bitcoin Pulse";
+    isSystemUser = true;
+    uid = 400;
+    group = "bitcoin-pulse-postgres";
+    extraGroups = [
+      "bitcoin-pulse"
+      "bitcoin-pulse-migration"
+    ];
+    home = "/var/lib/bitcoin-pulse-postgres";
+    homeMode = "0700";
+    createHome = true;
+    linger = true;
+    autoSubUidGidRange = true;
+  };
+
+  sops.secrets = {
+    "bitcoin-pulse/postgres/admin-password" = {
+      group = "bitcoin-pulse-postgres";
+      mode = "0440";
+    };
+
+    "bitcoin-pulse/postgres/app-password" = {
+      group = "bitcoin-pulse";
+      mode = "0440";
+    };
+
+    "bitcoin-pulse/postgres/migration-password" = {
+      group = "bitcoin-pulse-migration";
+      mode = "0440";
+    };
+  };
+
+  virtualisation.oci-containers = {
+    backend = "podman";
+
+    containers.${containerName} = {
+      # Official PostgreSQL 18.4 Bookworm multi-architecture image, pinned to
+      # the immutable digest tested by BitcoinPulseAPI on 2026-07-16.
+      image = "docker.io/library/postgres@sha256:1961f96e6029a02c3812d7cb329a3b03a3ac2bb067058dec17b0f5596aca9296";
+      pull = "missing";
+      autoStart = true;
+      autoRemoveOnStop = true;
+
+      podman = {
+        user = "bitcoin-pulse-postgres";
+        sdnotify = "healthy";
+      };
+
+      environment = {
+        POSTGRES_DB = "bitcoin_pulse";
+        POSTGRES_USER = "bitcoin_pulse_admin";
+        POSTGRES_PASSWORD_FILE = "/run/secrets/admin-password";
+        POSTGRES_HOST_AUTH_METHOD = "scram-sha-256";
+        POSTGRES_INITDB_ARGS = "--auth-host=scram-sha-256 --data-checksums";
+      };
+
+      ports = [ "127.0.0.1:5432:5432" ];
+
+      volumes = [
+        "bitcoin-pulse-db:/var/lib/postgresql"
+        "${runtimeDirectory}/admin-password:/run/secrets/admin-password:ro"
+        "${runtimeDirectory}/app-password:/run/secrets/app-password:ro"
+        "${runtimeDirectory}/migration-password:/run/secrets/migration-password:ro"
+        "${bitcoin-pulse}/resources/postgres/initdb/010-roles.sql:/docker-entrypoint-initdb.d/010-roles.sql:ro"
+      ];
+
+      extraOptions = [
+        "--health-cmd=pg_isready --host=127.0.0.1 --port=5432 --username=bitcoin_pulse_admin --dbname=bitcoin_pulse"
+        "--health-interval=10s"
+        "--health-timeout=5s"
+        "--health-retries=6"
+        "--health-start-period=60s"
+        "--stop-signal=SIGINT"
+        "--stop-timeout=60"
+        "--pids-limit=256"
+        "--security-opt=no-new-privileges"
+      ];
+    };
+  };
+
+  systemd.services.${serviceName}.serviceConfig = {
+    ExecStartPre = lib.mkAfter [ "${prepareSecrets}/bin/bitcoin-pulse-postgresql-prepare-secrets" ];
+    ExecStartPost = [ "${cleanupSecrets}/bin/bitcoin-pulse-postgresql-cleanup-secrets" ];
+    ExecStopPost = lib.mkAfter [ "${cleanupSecrets}/bin/bitcoin-pulse-postgresql-cleanup-secrets" ];
+    RuntimeDirectoryMode = "0700";
+    RestartSec = "5s";
+    TimeoutStopSec = lib.mkForce "90s";
+    UMask = "0077";
+  };
+
+  environment.systemPackages = [ pkgs.postgresql_18 ];
 }