Tobserver/docs/bitcoin-pulse-postgresql.md

# Bitcoin Pulse PostgreSQL operations

## Declared production layout

`modules/bitcoin-pulse.nix` declares PostgreSQL 18 as the rootless
`bitcoin-pulse-postgres` Podman container. The corresponding system service is
`podman-bitcoin-pulse-postgres.service`.

- The official PostgreSQL 18.4 Bookworm image is pinned by immutable digest.
- Podman runs under the dedicated `bitcoin-pulse-postgres` system account.
- The `bitcoin-pulse-db` named volume is mounted at `/var/lib/postgresql`, the
  persistent volume location required by PostgreSQL 18 images.
- Host port 5432 is published only as `127.0.0.1:5432` and is not opened in the
  NixOS firewall.
- Podman reports the service ready only after `pg_isready` succeeds.
- PostgreSQL receives `SIGINT` for smart shutdown, with 60 seconds for Podman
  and 90 seconds for the enclosing systemd service.
- Automatic image updates are disabled. Update the digest only in a reviewed
  change after testing backups, startup, and application compatibility.

The named volume belongs to the rootless Podman account and survives container
removal, service recreation, and NixOS generation changes. Backup and restore
portability is handled separately by BIT-23; do not copy Podman's private
storage directly.

## Human-managed secrets

Before deploying this module, use `sops` from an authorized workstation to add
three independently generated passwords to `secrets/tobserver.yaml`:

```yaml
bitcoin-pulse:
  postgres:
    admin-password: <administrator password>
    migration-password: <migration role password>
    app-password: <application role password>
```

Commit only the resulting encrypted SOPS payload. Never place a password in a
Nix expression, command argument, shell history, log, or pull-request text. The
agent change intentionally declares only the secret paths; it does not modify
or decrypt `secrets/tobserver.yaml`.

`sops-nix` materializes each credential as a restricted runtime file. Before
container startup, the service stages private copies owned by PostgreSQL's uid
inside the rootless user namespace. The copies are mounted read-only and are
removed from the host runtime directory after PostgreSQL reports healthy and
again when the unit stops. The application and migration SOPS files remain
available only through their dedicated host groups for the later BIT-28 units.

## Empty-database initialization and privileges

On the first start of an empty `bitcoin-pulse-db` volume, the official image
creates:

- `bitcoin_pulse_admin`, the bootstrap PostgreSQL superuser;
- the `bitcoin_pulse` database;
- `bitcoin_pulse_migration`, a non-superuser login that owns the database;
- `bitcoin_pulse_app`, a non-superuser login with `CONNECT` only initially.

The role bootstrap SQL comes from the pinned BitcoinPulseAPI flake input.
Migratus' database-foundation migration then grants the application role schema
usage and default DML privileges for application objects. Later migrations can
further restrict individual tables; the application role cannot create schema
objects, roles, or databases and must never run migrations.

Initialization scripts run only for an empty volume. Changing a SOPS value does
not change a role password in an existing PostgreSQL cluster.

## Deployment and verification

A human must review and merge the change, provision the encrypted keys above,
and deploy the resulting NixOS generation manually. The agent does not access
or rebuild Tobserver.

After deployment, verify the service and loopback binding:

```console
systemctl status podman-bitcoin-pulse-postgres.service
ss -ltn '( sport = :5432 )'
journalctl -u podman-bitcoin-pulse-postgres.service --since today
```

`ss` must show only `127.0.0.1:5432`. No credential should appear in the unit's
command line or journal.

Connect with `psql -h 127.0.0.1 -U <role> -d bitcoin_pulse`; allow `psql` to
prompt for the password. As the administrator, inspect role attributes with
`\du` and verify:

```sql
SELECT has_database_privilege(
  'bitcoin_pulse_app', 'bitcoin_pulse', 'CONNECT');
SELECT has_database_privilege(
  'bitcoin_pulse_app', 'bitcoin_pulse', 'CREATE');
SELECT has_schema_privilege(
  'bitcoin_pulse_app', 'public', 'CREATE');
```

The expected results are `true`, `false`, and `false`. After BIT-28 runs the
foundation migration, also verify the application can perform only the DML
allowed by the migrated schema and that only the migration role can apply DDL.

To test recreation and persistence, create a temporary probe table as the
administrator or migration role, restart the unit, confirm the row remains, and
then drop the probe table:

```sql
CREATE TABLE bit29_recreation_probe (value integer PRIMARY KEY);
INSERT INTO bit29_recreation_probe VALUES (1);
```

```console
systemctl restart podman-bitcoin-pulse-postgres.service
```

```sql
TABLE bit29_recreation_probe;
DROP TABLE bit29_recreation_probe;
```

A normal `systemctl stop` must complete within 90 seconds. Review the journal to
confirm a smart PostgreSQL shutdown and no forced kill. Start the unit again and
confirm it becomes healthy. Perform these production checks during a reviewed
maintenance window.

## Credential rotation for an existing database

Rotate one role at a time during a maintenance window. Keep all plaintext in an
approved password manager and interactive prompts.

1. Stop the backend and migration units that use the role being rotated. Leave
   PostgreSQL running.
2. Connect over loopback as `bitcoin_pulse_admin`; enter the current admin
   password at the `psql` prompt.
3. Run `\password bitcoin_pulse_app`, `\password bitcoin_pulse_migration`, or
   `\password bitcoin_pulse_admin` as appropriate. Enter the newly generated
   value only at the hidden prompts.
4. Update the matching value in `secrets/tobserver.yaml` with `sops`, commit only
   the encrypted result, and deploy that reviewed NixOS generation.
5. Restart the affected client unit and verify it connects. For administrator
   rotation, restart PostgreSQL once to verify the synchronized bootstrap file
   still permits normal startup of the existing cluster.
6. Revoke the old value in the password manager and record the rotation date.

If client verification fails, keep the client stopped, reconnect as the
administrator, and correct either the database role or encrypted runtime value.
Do not reset or remove `bitcoin-pulse-db`: first-start initialization is not a
credential-rotation mechanism.