# Bitcoin Pulse PostgreSQL operations

## Declared production layout

`modules/bitcoin-pulse.nix` declares PostgreSQL 18 as the rootless
`bitcoin-pulse-postgres` Podman container. The corresponding system service is
`podman-bitcoin-pulse-postgres.service`.

- The official PostgreSQL 18.4 Bookworm image is pinned by immutable digest.
- Podman runs under the dedicated `bitcoin-pulse-postgres` system account.
- The `bitcoin-pulse-db` named volume is mounted at `/var/lib/postgresql`, the
  persistent volume location required by PostgreSQL 18 images.
- Host port 5432 is published only as `127.0.0.1:5432` and is not opened in the
  NixOS firewall.
- Podman reports the service ready only after `pg_isready` succeeds.
- PostgreSQL receives `SIGINT` for a clean fast shutdown, with 60 seconds for
  Podman and 90 seconds for the enclosing systemd service.
- Automatic image updates are disabled. Update the digest only in a reviewed
  change after testing backups, startup, and application compatibility.

The named volume belongs to the rootless Podman account and survives container
removal, service recreation, and NixOS generation changes. Backup and restore
portability is handled separately by BIT-23; do not copy Podman's private
storage directly.

## Human-managed secrets

Before deploying this module, use `sops` from an authorized workstation to add
three independently generated passwords to `secrets/tobserver.yaml`:

```yaml
bitcoin-pulse:
  postgres:
    admin-password: <administrator password>
    migration-password: <migration role password>
    app-password: <application role password>
```

Commit only the resulting encrypted SOPS payload. Never place a password in a
Nix expression, command argument, shell history, log, or pull-request text. The
agent change intentionally declares only the secret paths; it does not modify
or decrypt `secrets/tobserver.yaml`.

`sops-nix` materializes each credential as a restricted runtime file. Before
container startup, the service stages private copies owned by PostgreSQL's uid
inside the rootless user namespace. The copies are mounted read-only and are
removed from the host runtime directory after PostgreSQL reports healthy and
again when the unit stops. The application and migration SOPS files remain
available only through the dedicated groups of their respective host services.

## Empty-database initialization and privileges

On the first start of an empty `bitcoin-pulse-db` volume, the official image
creates:

- `bitcoin_pulse_admin`, the bootstrap PostgreSQL superuser;
- the `bitcoin_pulse` database;
- `bitcoin_pulse_migration`, a non-superuser login that owns the database;
- `bitcoin_pulse_app`, a non-superuser login with `CONNECT` only initially.

The role bootstrap SQL comes from the pinned BitcoinPulseAPI flake input.
Migratus' database-foundation migration then grants the application role schema
usage and default DML privileges for application objects. Later migrations can
further restrict individual tables; the application role cannot create schema
objects, roles, or databases and must never run migrations.

Initialization scripts run only for an empty volume. Changing a SOPS value does
not change a role password in an existing PostgreSQL cluster.

## Migration-gated backend startup

`bitcoin-pulse.service` requires `bitcoin-pulse-migrate.service`, and the
migration service requires the health-gated PostgreSQL container service. The
resulting startup order is:

```text
PostgreSQL healthy -> bounded readiness check -> Migratus -> backend
```

The migration service waits up to 60 seconds for `pg_isready`, then runs the
same packaged backend version's `bitcoin-pulse-migrate migrate` command as the
unprivileged `bitcoin-pulse-migration` host user. It receives only the migration
role and its SOPS password-file path. The backend separately receives the
restricted `bitcoin_pulse_app` role and application password-file path.

The migration unit is a one-shot service without `RemainAfterExit`, so every
new start transaction that pulls it in runs Migratus again. Migratus records
completed versions in `schema_migrations`; already-applied migrations therefore
make repeated starts harmless. A readiness timeout, invalid credential, or
migration error fails the one-shot unit. Because the backend is ordered after
and requires that unit, the backend process is not started after such a failure.
Inspect `journalctl -u bitcoin-pulse-migrate.service` for the bounded readiness
message, Migratus error, and systemd exit status.

## Deployment and verification

A human must review and merge the change, provision the encrypted keys above,
and deploy the resulting NixOS generation manually. The agent does not access
or rebuild Tobserver.

After deployment, verify the service and loopback binding:

```console
systemctl status podman-bitcoin-pulse-postgres.service
ss -ltn '( sport = :5432 )'
journalctl -u podman-bitcoin-pulse-postgres.service --since today
```

`ss` must show only `127.0.0.1:5432`. No credential should appear in the unit's
command line or journal.

Connect with `psql -h 127.0.0.1 -U <role> -d bitcoin_pulse`; allow `psql` to
prompt for the password. As the administrator, inspect role attributes with
`\du` and verify:

```sql
SELECT has_database_privilege(
  'bitcoin_pulse_app', 'bitcoin_pulse', 'CONNECT');
SELECT has_database_privilege(
  'bitcoin_pulse_app', 'bitcoin_pulse', 'CREATE');
SELECT has_schema_privilege(
  'bitcoin_pulse_app', 'public', 'CREATE');
```

The expected results are `true`, `false`, and `false`. Verify the application
can perform only the DML allowed by the migrated schema and that only the
migration role can apply DDL.

To test recreation and persistence, create a temporary probe table as the
administrator or migration role, restart the unit, confirm the row remains, and
then drop the probe table:

```sql
CREATE TABLE bit29_recreation_probe (value integer PRIMARY KEY);
INSERT INTO bit29_recreation_probe VALUES (1);
```

```console
systemctl restart podman-bitcoin-pulse-postgres.service
```

```sql
TABLE bit29_recreation_probe;
DROP TABLE bit29_recreation_probe;
```

A normal `systemctl stop` must complete within 90 seconds. Review the journal to
confirm a clean fast PostgreSQL shutdown and no forced kill. Start the unit again
and confirm it becomes healthy. Perform these production checks during a
reviewed maintenance window.

After deploying the migration gate, verify the one-shot and backend:

```console
systemctl is-active podman-bitcoin-pulse-postgres.service
systemctl is-active bitcoin-pulse.service
systemctl show bitcoin-pulse-migrate.service \
  --property=Result --property=ExecMainStatus
journalctl -u bitcoin-pulse-migrate.service -b --no-pager
ss -ltn '( sport = :3000 )'
```

A successful migration unit becomes inactive after it exits; `Result=success`
and `ExecMainStatus=0` are expected. The backend must be active and port 3000
must be bound only to `127.0.0.1`. Restart `bitcoin-pulse.service` once and
confirm the migration unit runs successfully again without reapplying completed
migrations.

Successful and repeated paths may be verified during deployment. Exercise the
database-unavailable and invalid-migration-credential paths only on a disposable
NixOS test host: both must make `bitcoin-pulse-migrate.service` fail, leave
`bitcoin-pulse.service` stopped, finish within the configured timeout, and emit
an actionable journal message. Never inject invalid credentials into production.

## Rollback expectations

A NixOS generation rollback changes the application package and service
configuration but does not reverse migrations in PostgreSQL. Do not use
`bitcoin-pulse-migrate rollback` as an automatic deployment action. Every schema
change must remain compatible with the previous application generation. Use an
expand/migrate/contract sequence for breaking changes, and perform any reviewed
manual rollback only after assessing data loss and application compatibility.

## Credential rotation for an existing database

Rotate one role at a time during a maintenance window. Keep all plaintext in an
approved password manager and interactive prompts.

1. Stop the backend and migration units that use the role being rotated. Leave
   PostgreSQL running.
2. Connect over loopback as `bitcoin_pulse_admin`; enter the current admin
   password at the `psql` prompt.
3. Run `\password bitcoin_pulse_app`, `\password bitcoin_pulse_migration`, or
   `\password bitcoin_pulse_admin` as appropriate. Enter the newly generated
   value only at the hidden prompts.
4. Update the matching value in `secrets/tobserver.yaml` with `sops`, commit only
   the encrypted result, and deploy that reviewed NixOS generation.
5. Restart the affected client unit and verify it connects. For administrator
   rotation, restart PostgreSQL once to verify the synchronized bootstrap file
   still permits normal startup of the existing cluster.
6. Revoke the old value in the password manager and record the rotation date.

If client verification fails, keep the client stopped, reconnect as the
administrator, and correct either the database role or encrypted runtime value.
Do not reset or remove `bitcoin-pulse-db`: first-start initialization is not a
credential-rotation mechanism.