Compare commits
10 commits
801201565f
...
b49138d95a
| Author | SHA1 | Date | |
|---|---|---|---|
|
b49138d95a | ||
|
|
ff560ea84c | ||
|
|
1bdf25bf06 | ||
|
|
81fea16376 | ||
|
|
532a8ed613 | ||
|
|
5753bf9e33 | ||
|
|
f54963b249 | ||
|
|
133384660e | ||
|
|
2015aac8e4 | ||
|
|
0296280ac9 |
13 changed files with 1333 additions and 272 deletions
297
.plans/deploy-forgejo-with-backups.md
Normal file
297
.plans/deploy-forgejo-with-backups.md
Normal file
|
|
@ -0,0 +1,297 @@
|
|||
# Implementation Plan
|
||||
|
||||
> **Status:** Phase 1 is ready for implementation; Phase 2 is blocked on the post-trial **KEEP** decision.
|
||||
|
||||
## Outcome and boundaries
|
||||
|
||||
- **Problem and target:** First deploy a minimal, usable Forgejo instance at `https://git.tobiasostner.de` so it can be evaluated without committing to the backup and recovery design. Add that surrounding infrastructure only after an explicit decision to keep Forgejo.
|
||||
- **Phase 1 — trial deployment:** Native NixOS Forgejo LTS service; SQLite; `/var/lib/forgejo` state; loopback HTTP; nginx/ACME ingress; host OpenSSH on port 22; closed registration; first-admin bootstrap; and disposable HTTPS/SSH repository checks. Keep all declarative core deployment configuration—Forgejo service, database settings, state paths, nginx virtual host, and package exposure—together in `modules/forgejo.nix`.
|
||||
- **Phase 1 non-goals:** No new SOPS secret, B2/Restic repository, backup service or timer, staging/runtime backup directories, archive or manifest logic, configuration-revision plumbing, backup VM check, retention policy, production backup, backup/recovery documentation, or restore/protection claim.
|
||||
- **Trial safety boundary:** Data created during Phase 1 has no new backup protection. Use only disposable repositories and retain any authoritative copy elsewhere until Phase 2's first production snapshot has passed its content checks.
|
||||
- **Decision gate:** After the trial, **KEEP** preserves `/var/lib/forgejo` and authorizes Phase 2. **ABANDON** stops this plan, skips every Phase 2 task, and triggers a separate reviewed removal plan whose default is to preserve trial state. No trial duration is prescribed.
|
||||
- **Phase 2 — retained deployment:** Dedicated encrypted Restic password; daily complete-state backup to the existing Backblaze B2 bucket; 7 daily, 4 weekly, and 12 monthly snapshots; deterministic backup failure-path tests; deployment, backup, recovery, and rollback documentation; and inspection of the first production snapshot.
|
||||
- **Out of scope in both phases:** Migration, LFS, Actions/runners, package registry, SMTP, SSO, containers, PostgreSQL, HA, external object storage, additional firewall ports, backup alerting, SSH-service interruption during backup, and initial or recurring restore tests.
|
||||
- **Phase 2 accepted limitations:** sshd remains available during backup. A Git-over-SSH push beginning after the active-process check can overlap archive creation and make that snapshot inconsistent. Snapshot and archive inspection will not prove recovery because restore testing remains excluded.
|
||||
|
||||
## Key files, evidence, and decisions
|
||||
|
||||
| File or source | Why it matters | Decision or plan impact |
|
||||
|---|---|---|
|
||||
| `flake.nix:1-38`, `flake.lock` node `nixpkgs_2` | The host uses `nixos-25.11`; host modules are listed in one place; the evaluated `forgejo-lts` package is currently 11.0.15. | Import only the Forgejo service module in Phase 1. Add backup modules, the VM check, and `self` revision wiring in Phase 2. Re-evaluate the lock-specific package version before each activation. |
|
||||
| Pinned nixpkgs `nixos/modules/services/misc/forgejo.nix` | The native module supports SQLite, external OpenSSH, declarative settings, generated persistent secrets, and state rooted at `/var/lib/forgejo`. | Use the native pinned module rather than a container or new flake input. |
|
||||
| `configuration.nix:35-68` | Ports 22, 80, and 443 are already open; OpenSSH, nginx, and ACME are enabled. | Reuse those services and do not expose the Forgejo backend port. |
|
||||
| `modules/bitcoin-pulse.nix:101-106` | Bitcoin Pulse occupies `127.0.0.1:3000`. | Bind Forgejo only to `127.0.0.1:3001`. |
|
||||
| `configuration.nix:118-127`, `secrets/.sops.yaml` | SOPS is established and `backblaze/tobcloud-backup-env` is a shared root-only secret. | Do not touch secrets in Phase 1. In Phase 2, reuse the B2 credential and add only a dedicated root-only `restic/forgejo-password`. |
|
||||
| `modules/bitcoin-pulse.nix:115-174,340-388` | This is the strongest local Restic/private-staging/timer/retention pattern and an existing nginx proxy pattern. | Reuse only the nginx conventions in Phase 1. Add Forgejo-specific Restic orchestration in a separable backup module in Phase 2. |
|
||||
| `docs/bitcoin-pulse-operations.md` | Defines clean-checkout, build-before-switch, runtime verification, and multi-service rollback conventions. | Apply the same deployment discipline in both phases; avoid casual full-generation rollback. |
|
||||
| [Forgejo backup guidance](https://forgejo.org/docs/latest/admin/upgrade/#backup) | Complete state needs synchronized point-in-time capture; shutdown provides the consistency boundary used by the existing design. | Phase 2 stops Forgejo and requires no remaining Forgejo-owned process before archiving all state. |
|
||||
| [Forgejo reverse-proxy guidance](https://forgejo.org/docs/latest/admin/setup/reverse-proxy/) | Documents dedicated-origin, `ROOT_URL`, encoded-slash, proxy-header, and upload-size behavior. | Phase 1 uses a dedicated hostname, exact proxy behavior, and a 512 MiB body limit. |
|
||||
| Verified baseline on 2026-08-01 | `git.tobiasostner.de` resolves to `162.55.103.218` with no AAAA record, and `nix flake check --no-build` passes. | Historical DNS, Codeberg-fetch, and Hermes evaluation failures are not current blockers. Recheck DNS and perform complete builds before activation. |
|
||||
| User decisions captured by the existing plan | New personal instance; public HTTPS; closed registration; SQLite; host SSH; initial admin `oibot`; daily B2 backup with 7/4/12 retention; no restore tests, alerting, or SSH interruption. | Preserve these boundaries. Supply the admin email privately at deployment time; Phase 2 still requires an explicit KEEP decision. |
|
||||
|
||||
- **Open gate:** Phase 2 must not start until the user records an explicit **KEEP** decision after completing the Phase 1 trial. This does not block implementing or deploying Phase 1.
|
||||
- **Trial decision record:** PENDING
|
||||
|
||||
## Phase 1 — Deploy only the Forgejo trial
|
||||
|
||||
#### P1-T1 — Add the native Forgejo service and ingress
|
||||
|
||||
- **Change:**
|
||||
- Create `modules/forgejo.nix` and import it once from `flake.nix`. Keep the complete Phase 1 deployment in this one module: Forgejo service, SQLite/database settings, state and repository paths, package exposure, and nginx/ACME ingress. Do not distribute those settings across `configuration.nix` or additional Forgejo deployment modules.
|
||||
- Enable `services.forgejo` with `pkgs.forgejo-lts`, SQLite, the default `forgejo` account, `/var/lib/forgejo` state, and `/var/lib/forgejo/repositories` repository storage.
|
||||
- Bind HTTP only to `127.0.0.1:3001`; set `DOMAIN`, `ROOT_URL`, and `SSH_DOMAIN` to `git.tobiasostner.de`; set `SSH_PORT = 22`, `SSH_USER = "forgejo"`, and `START_SSH_SERVER = false`.
|
||||
- Enable secure cookies and disable registration. Keep LFS disabled and explicitly disable Actions, package registry, and mailer facilities.
|
||||
- Make the selected Forgejo package available in the system profile so administrative commands use the deployed version.
|
||||
- Add an nginx virtual host with ACME/forced TLS, `merge_slashes off`, `client_max_body_size 512M`, HTTP/1.1, and Forgejo's required Host, real-IP, forwarding, Connection, and Upgrade headers.
|
||||
- Do not add secrets, Restic options, backup paths, revision metadata, tests, timers, or firewall ports.
|
||||
- **Starts at:** `flake.nix:25-38`; new `modules/forgejo.nix`; `modules/bitcoin-pulse.nix:354-383`.
|
||||
- **Tests:** Protect the SQLite/state model, closed registration, excluded features, public and SSH URLs, loopback-only backend, proxy behavior, pinned package choice, and absence of a new firewall port.
|
||||
- **Verify:**
|
||||
- `nix eval --raw .#nixosConfigurations.tobserver.config.services.forgejo.package.version` prints `11.0.15` for the current lock.
|
||||
- `nix eval --json .#nixosConfigurations.tobserver.config.services.forgejo.settings | jq -e '.database.DB_TYPE == "sqlite3" and .server.HTTP_ADDR == "127.0.0.1" and .server.HTTP_PORT == 3001 and .server.DOMAIN == "git.tobiasostner.de" and .server.ROOT_URL == "https://git.tobiasostner.de/" and .server.SSH_DOMAIN == "git.tobiasostner.de" and .server.SSH_PORT == 22 and .server.SSH_USER == "forgejo" and .server.START_SSH_SERVER == false and .service.DISABLE_REGISTRATION == true and .session.COOKIE_SECURE == true and .actions.ENABLED == false and .packages.ENABLED == false and .mailer.ENABLED == false'` prints `true` and exits 0.
|
||||
- `nix eval --json .#nixosConfigurations.tobserver.config.networking.firewall.allowedTCPPorts | jq -e 'sort == [22,80,443,50001]'` prints `true` and exits 0, proving the complete existing port set is unchanged.
|
||||
- `nix eval --json .#nixosConfigurations.tobserver.config.services.restic.backups | jq -e 'has("forgejo") | not'` and `nix eval --json .#nixosConfigurations.tobserver.config.sops.secrets | jq -e 'has("restic/forgejo-password") | not'` both print `true` and exit 0.
|
||||
- **Risk/recovery:** Wrong URL, SSH, database, or proxy settings can break startup, redirects, authentication, or clone URLs. Keeping the complete core deployment in `modules/forgejo.nix` makes review and a narrow revert straightforward. A reviewed removal of its single import must remove only Forgejo's service and virtual host; never delete `/var/lib/forgejo` as an automatic rollback action.
|
||||
|
||||
#### P1-T2 — Write the trial deployment and administration runbook
|
||||
|
||||
- **Change:**
|
||||
- Create `docs/forgejo-operations.md` with Phase 1-only instructions for DNS confirmation, evaluation, clean-checkout build-before-switch, activation, service/listener/nginx/journal checks, and rollback boundaries for a multi-service host.
|
||||
- Document first-admin creation with the deployed `forgejo` binary, explicit `HOME`, `USER`, work/custom paths, `--config /var/lib/forgejo/custom/conf/app.ini`, `--username "oibot"`, a privately supplied runtime `admin_email` passed as `--email "$admin_email"`, `--admin`, and `--random-password`. Do not record the email in this plan.
|
||||
- Require immediate password-manager capture and prohibit operator-selected passwords in argv/history.
|
||||
- Document SSH-key registration, creation of a disposable smoke repository, an HTTPS clone/push, and an SSH clone/fetch.
|
||||
- State prominently that Phase 1 data is unprotected and should remain disposable.
|
||||
- Do not add backup, retention, manifest, recovery, or restore instructions yet.
|
||||
- **Starts at:** New `docs/forgejo-operations.md`; `docs/bitcoin-pulse-operations.md`.
|
||||
- **Depends on:** P1-T1.
|
||||
- **Tests:** Protect secret-safe/user-correct administration, exact deployed paths, both clone transports, multi-service rollback safety, and the no-protection warning.
|
||||
- **Verify:**
|
||||
- `git diff --check -- docs/forgejo-operations.md` exits 0.
|
||||
- Exact searches confirm the runbook contains `git.tobiasostner.de`, the `sudo -u forgejo env HOME=/var/lib/forgejo USER=forgejo FORGEJO_WORK_DIR=/var/lib/forgejo FORGEJO_CUSTOM=/var/lib/forgejo/custom forgejo --config /var/lib/forgejo/custom/conf/app.ini admin user create --username "oibot" --email "$admin_email" --admin --random-password` flow, HTTPS and SSH checks, and the words `unprotected` and `disposable`.
|
||||
- The Phase 1 document makes no claim that trial data is backed up or recoverable.
|
||||
- **Risk/recovery:** Preserve the generated password immediately; never place it in Git, Nix expressions, logs, or shell history.
|
||||
|
||||
#### P1-T3 — Build the trial without activation
|
||||
|
||||
- **Change:**
|
||||
- Reconfirm DNS and the lock-specific Forgejo version.
|
||||
- Track this plan with the Phase 1 implementation commit so the later decision record exists in every clean deployment checkout.
|
||||
- Format and validate only Phase 1 Nix files, evaluate the full flake, and build the complete host before switching.
|
||||
- On Tobserver, require a clean reviewed checkout and build the host configuration without activation.
|
||||
- Before the first switch, require `/var/lib/forgejo` to be absent or empty; stop for review if residual state exists because migration/reuse is out of scope.
|
||||
- **Starts at:** Completed P1-T1 and P1-T2 changes.
|
||||
- **Depends on:** P1-T1, P1-T2.
|
||||
- **Tests:** Protect DNS/ACME routing, new-empty-instance scope, formatting, full evaluation, and build-before-switch.
|
||||
- **Verify:**
|
||||
- `test "$(dig +short git.tobiasostner.de A)" = "162.55.103.218"` and `test -z "$(dig +short git.tobiasostner.de AAAA)"` exit 0.
|
||||
- `nixpkgs-fmt --check flake.nix modules/forgejo.nix`, `git diff --check`, and `nix flake check --no-build` exit 0.
|
||||
- On a compatible Linux builder, `nix build .#nixosConfigurations.tobserver.config.system.build.toplevel` exits 0.
|
||||
- `git ls-files --error-unmatch .plans/deploy-forgejo-with-backups.md` exits 0 after the reviewed Phase 1 commit, proving the plan is tracked rather than hidden by a cleanliness exception.
|
||||
- On Tobserver, `test -z "$(git status --porcelain)"`, `git ls-files --error-unmatch .plans/deploy-forgejo-with-backups.md`, `nix flake check --no-build`, `sudo test ! -e /var/lib/forgejo || ! sudo find /var/lib/forgejo -mindepth 1 -print -quit | grep -q .`, and `sudo nixos-rebuild build --flake .#tobserver` all exit 0 before any switch.
|
||||
- **Risk/recovery:** A full-build failure blocks activation but is not currently a known baseline failure. Do not bypass a failed build or overwrite unexpected Forgejo state.
|
||||
|
||||
#### P1-T4 — Activate and verify the public trial
|
||||
|
||||
- **Change:**
|
||||
- Switch to the reviewed generation, create the first admin with the documented `--admin --random-password` flow, and verify that Forgejo reports the account as an administrator.
|
||||
- Register an SSH key, create a disposable smoke repository, push a disposable commit, and exercise both HTTPS and host-SSH transports.
|
||||
- Verify service health, TLS ingress, closed registration, listener isolation, and rendered nginx behavior.
|
||||
- **Starts at:** `docs/forgejo-operations.md`; deployed Tobserver checkout.
|
||||
- **Depends on:** P1-T3.
|
||||
- **Tests:** Protect service health, TLS/redirect, proxy configuration, loopback-only backend, closed registration, first-admin login, host OpenSSH integration, and HTTPS/SSH repository use.
|
||||
- **Verify:**
|
||||
- `sudo systemctl is-active forgejo.service nginx.service` reports both `active`.
|
||||
- `sudo ss -H -ltn 'sport = :3001'` contains `127.0.0.1:3001` and no non-loopback listener; an external connection to port 3001 fails.
|
||||
- `sudo nginx -t` exits 0; `sudo nginx -T 2>&1` contains the `git.tobiasostner.de` server, `merge_slashes off`, `client_max_body_size 512M`, and required proxy headers.
|
||||
- `curl --fail --silent --show-error -H 'Host: git.tobiasostner.de' http://127.0.0.1:3001/api/v1/version | jq -e .` exits 0.
|
||||
- `curl --fail --silent --show-error --location --output /dev/null --write-out '%{http_code} %{url_effective}\n' http://git.tobiasostner.de/` reports `200` at `https://git.tobiasostner.de/`.
|
||||
- `sudo -u forgejo env HOME=/var/lib/forgejo USER=forgejo FORGEJO_WORK_DIR=/var/lib/forgejo FORGEJO_CUSTOM=/var/lib/forgejo/custom forgejo --config /var/lib/forgejo/custom/conf/app.ini admin user list --admin | grep -F -- "oibot"` exits 0, and the UI offers no registration flow.
|
||||
- From an external client, `! nc -z git.tobiasostner.de 3001` succeeds; the documented HTTPS clone/push and `forgejo@git.tobiasostner.de:<admin>/clone-smoke.git` SSH clone/fetch all exit 0 for the disposable repository.
|
||||
- **Risk/recovery:** If unhealthy, collect Forgejo/nginx journals before changing state. Deploy a reviewed Forgejo-only revert rather than casually switching the multi-service host to an older full generation. Preserve `/var/lib/forgejo`.
|
||||
|
||||
### Phase 1 acceptance
|
||||
|
||||
- Forgejo is publicly usable over HTTPS and host SSH, listens only on `127.0.0.1:3001`, permits no open registration, exposes no new firewall port, and supports disposable repository use through both transports.
|
||||
- No Forgejo backup secret, backup service/timer, staging path, manifest/revision plumbing, VM backup test, backup/recovery instructions, or restore claim has been added.
|
||||
- **The trial remains unprotected. Its data must remain disposable until Phase 2's first production snapshot passes.**
|
||||
|
||||
## Trial decision gate
|
||||
|
||||
#### G1 — Record KEEP or ABANDON
|
||||
|
||||
- **Change:**
|
||||
- Require the user to record one explicit outcome after Phase 1; do not infer it from elapsed time.
|
||||
- Record the decision durably in this plan by replacing `PENDING` above with exactly `KEEP | YYYY-MM-DD | approved by Tobias Ostner` or `ABANDON | YYYY-MM-DD | approved by Tobias Ostner`, then commit that decision before making any Phase 2 change.
|
||||
- **KEEP:** Preserve `/var/lib/forgejo` in place and authorize P2-T1 through P2-T6. Do not store the only authoritative copy of data in Forgejo until P2-T6 succeeds.
|
||||
- **ABANDON:** Stop this implementation plan and skip every Phase 2 task. Prepare a separate, narrowly reviewed removal plan that handles host-SSH write exclusion, service/ingress removal, state preservation, validation, and any later destruction decision; do not improvise removal from this plan.
|
||||
- **Starts at:** This plan's `Trial decision record`; deployed Phase 1 instance.
|
||||
- **Depends on:** P1-T4.
|
||||
- **Tests:** Protect explicit owner authorization and prevent Phase 2 from starting on an inferred or undocumented decision.
|
||||
- **Verify — KEEP:** From a clean checkout, run the following before P2-T1 begins; it requires exactly one KEEP record, validates its calendar date, rejects unresolved/ABANDON records, and proves the decision is tracked and committed:
|
||||
|
||||
```console
|
||||
set -euo pipefail
|
||||
plan=.plans/deploy-forgejo-with-backups.md
|
||||
decision_lines="$(rg -x -- '- \*\*Trial decision record:\*\* KEEP \| [0-9]{4}-[0-9]{2}-[0-9]{2} \| approved by Tobias Ostner' "$plan" || true)"
|
||||
test "$(printf '%s\n' "$decision_lines" | grep -c '^-')" -eq 1
|
||||
decision_date="$(printf '%s\n' "$decision_lines" | sed -E 's/^.*KEEP \| ([0-9-]+) \|.*$/\1/')"
|
||||
test "$(date -d "$decision_date" +%F)" = "$decision_date"
|
||||
! rg -F -- '- **Trial decision record:** PENDING' "$plan"
|
||||
! rg -F -- '- **Trial decision record:** ABANDON' "$plan"
|
||||
git ls-files --error-unmatch "$plan"
|
||||
git diff --quiet -- "$plan"
|
||||
git diff --cached --quiet -- "$plan"
|
||||
test -n "$(git log -1 --format=%H -- "$plan")"
|
||||
sudo test -d /var/lib/forgejo
|
||||
sudo systemctl is-active forgejo.service
|
||||
```
|
||||
|
||||
Expected: every command exits 0 and Forgejo remains active with its state in place.
|
||||
- **Verify — ABANDON:** The decision record contains the dated ABANDON form, no Phase 2 file has been implemented, and a separate removal plan is reviewed before any removal action.
|
||||
- **Risk/recovery:** A broad generation rollback or ad hoc state move can affect unrelated services or race host-SSH writes. Removal is deliberately deferred to a separate plan if ABANDON is chosen.
|
||||
|
||||
## Phase 2 — Add backups and surrounding infrastructure after KEEP
|
||||
|
||||
#### P2-T1 — Add separable backup configuration, revision wiring, and secret
|
||||
|
||||
- **Change:**
|
||||
- Create `modules/forgejo-backup.nix` and import it separately from `modules/forgejo.nix`, so backup infrastructure can be removed without removing Forgejo.
|
||||
- Pass `self` through `flake.nix` only in this phase and set `system.configurationRevision` from clean `self.rev`, allowing `dirtyRev` only for non-production evaluation. Production backup preparation must reject a missing, unknown, or dirty revision.
|
||||
- Declare `sops.secrets."restic/forgejo-password"` as `root:root` mode `0400`.
|
||||
- Add `restic.forgejo-password` to `secrets/tobserver.yaml` through SOPS using a high-entropy password retained in the approved password manager.
|
||||
- Reuse `backblaze/tobcloud-backup-env` only for B2 access; never duplicate its plaintext value.
|
||||
- **Starts at:** `flake.nix:16-38`; new `modules/forgejo-backup.nix`; `configuration.nix:118-127`; `secrets/tobserver.yaml`.
|
||||
- **Depends on:** G1 = KEEP.
|
||||
- **Tests:** Protect explicit KEEP authorization, encrypted Git content, authorized SOPS decryption, declarative permissions, clean revision identity, Restic-password independence, and backup-module separability.
|
||||
- **Verify:**
|
||||
- Re-run G1's exact decision-record command before making any Phase 2 change.
|
||||
- `sops --decrypt secrets/tobserver.yaml >/dev/null` exits 0 without printing plaintext.
|
||||
- The encrypted diff contains the new key only as an `ENC[...]` value.
|
||||
- `nix eval --json '.#nixosConfigurations.tobserver.config.sops.secrets."restic/forgejo-password"' | jq -e '.owner == "root" and .group == "root" and .mode == "0400"'` prints `true` and exits 0; runtime permissions are checked after activation in P2-T6.
|
||||
- **Risk/recovery:** Losing this password makes all Forgejo snapshots unreadable. It must never enter Nix strings, argv, history, logs, or plaintext Git content.
|
||||
|
||||
#### P2-T2 — Implement the complete-state archive and Restic job
|
||||
|
||||
- **Change:**
|
||||
- Configure `services.restic.backups.forgejo` with `initialize = true`, repository `b2:tobcloud-backup:forgejo`, the dedicated password, the shared B2 environment file, and only `/var/backup/forgejo/forgejo-state.tar` as its Restic path.
|
||||
- Create `/var/backup/forgejo` as `root:root` mode `0700`; keep Restic's generated runtime directory private; create independent tmpfiles-managed `/run/forgejo-backup` as `root:root` mode `0700`; use `UMask = "0077"` and a six-hour timeout.
|
||||
- Remove stale temporary/final archives before preparation and require `forgejo.service` active. Keep `du`/`df` capacity comparison advisory because Btrfs free-space accounting makes a fixed threshold unreliable.
|
||||
- Before claiming ownership, require `/run/forgejo-backup/forgejo-stopped` to be absent. If it already exists, run only the idempotent recovery path, clear it only after Forgejo is confirmed active, and fail this invocation without archiving; retain the marker and fail if recovery does not succeed.
|
||||
- Create `/run/forgejo-backup/forgejo-stopped` only after clean prechecks, stop Forgejo, and wait with a bounded timeout until no process owned by `forgejo` remains. If an existing Git-over-SSH process remains, fail before archiving and restart Forgejo through cleanup.
|
||||
- Do not stop or reconfigure sshd. Operate the job in an idle/no-push window and disclose the accepted race from a new SSH push starting after the process check.
|
||||
- While Forgejo is stopped, create root-level `forgejo-backup-manifest.txt` with exactly four ordered newline-delimited fields: `format_version=1`, `forgejo_version=<deployed version>`, `system_path=<readlink -f /run/current-system>`, and `configuration_revision=<deployed clean flake revision>`. Fail if any value is missing or dirty/unknown.
|
||||
- Archive all `/var/lib/forgejo` as `forgejo/` plus the manifest into a temporary tar while preserving numeric ownership, ACLs, and extended attributes; atomically rename it to the final staging path.
|
||||
- Restart Forgejo, require it active, and clear the marker before allowing Restic upload.
|
||||
- Preserve preparation's original status. Its EXIT trap must remove temporary artifacts, remove the final archive when preparation failed, and retry Forgejo restart only while this run owns the marker.
|
||||
- Make `backupCleanupCommand` idempotently retry a marked restart, clear the marker only after Forgejo is confirmed active, retain the marker after terminal restart failure, and unconditionally remove temporary/final staging after the Restic attempt. The next run removes persistent stale staging.
|
||||
- Do not configure `Requires=forgejo.service`; stopping a required unit would cancel the backup.
|
||||
- Schedule daily at 04:00 with `Persistent = true`, up to 30 minutes random delay, and prune `7/4/12`.
|
||||
- **Starts at:** New `modules/forgejo-backup.nix`; `modules/bitcoin-pulse.nix:115-174,340-350,386-388`.
|
||||
- **Depends on:** P2-T1.
|
||||
- **Tests:** Protect complete state/manifest scope, bounded active-process rejection, stale-artifact prevention, atomic publication, status-preserving restart/cleanup, no stale upload after failure, private staging, exact schedule/retention, and absence of `Requires=forgejo.service`.
|
||||
- **Verify:**
|
||||
- `nix eval --raw .#nixosConfigurations.tobserver.config.services.restic.backups.forgejo.repository` prints `b2:tobcloud-backup:forgejo`.
|
||||
- `nix eval --json .#nixosConfigurations.tobserver.config.services.restic.backups.forgejo.paths | jq -e '. == ["/var/backup/forgejo/forgejo-state.tar"]'` prints `true`.
|
||||
- `nix eval --json .#nixosConfigurations.tobserver.config.services.restic.backups.forgejo.pruneOpts | jq -e '. == ["--keep-daily 7","--keep-weekly 4","--keep-monthly 12"]'` prints `true`.
|
||||
- `nix eval --json .#nixosConfigurations.tobserver.config.systemd.services.restic-backups-forgejo.requires | jq -e 'index("forgejo.service") | not'` prints `true` and exits 0.
|
||||
- `nix eval --json .#nixosConfigurations.tobserver.config.systemd.services.restic-backups-forgejo.serviceConfig | jq -e '.RuntimeDirectoryMode == "0700" and .UMask == "0077" and .TimeoutStartSec == "6h"'` prints `true` and exits 0.
|
||||
- Evaluate `systemd.tmpfiles.rules` and require the exact `/run/forgejo-backup 0700 root root` rule; inspect the generated preparation/cleanup scripts and require the exact stop/check/archive/restart-before-upload and status-preserving cleanup order.
|
||||
- **Risk/recovery:** A newly racing SSH push can make a snapshot inconsistent; this is accepted. Any process, archive, restart, or upload failure must fail visibly without uploading stale content. Removing the separate backup module must leave Forgejo and `/var/lib/forgejo` intact.
|
||||
|
||||
#### P2-T3 — Add a disposable backup-mechanism VM check
|
||||
|
||||
- **Change:**
|
||||
- Factor orchestration generation into `modules/forgejo-backup-script.nix`. Permit build-time-only phase hooks for the VM check; production must instantiate it without hooks, production options, or runtime fault-injection variables.
|
||||
- Add `tests/forgejo-backup.nix` and expose `checks.x86_64-linux.forgejo-backup` with a local Restic repository.
|
||||
- Prove normal ordering and deterministically exercise pre-stop rejection, a lingering Forgejo-owned process, archive failure, one-shot restart failure recovered by cleanup, terminal restart failure with retained marker, Restic failure after restart, forced backup-unit termination, and stale staging on the next run.
|
||||
- Assert this expected outcome matrix; `empty` means no entry of any type remains in staging:
|
||||
|
||||
| Case | Backup unit | Forgejo | Marker | Staging | Snapshot | sshd |
|
||||
|---|---|---|---|---|---|---|
|
||||
| Normal | success | active | absent | empty | +1 | active |
|
||||
| Pre-stop rejection | failed | active | absent | empty | unchanged | active |
|
||||
| Lingering Forgejo process | failed | active after cleanup | absent | empty | unchanged | active |
|
||||
| Archive failure | failed | active after cleanup | absent | empty | unchanged | active |
|
||||
| One-shot restart failure | failed | active after cleanup retry | absent | empty | unchanged | active |
|
||||
| Terminal restart failure | failed | failed/inactive | retained | empty | unchanged | active |
|
||||
| Restic failure after restart | failed | active | absent | empty | unchanged | active |
|
||||
| Forced unit termination | failed | active after cleanup | absent | empty | unchanged | active |
|
||||
| Pre-existing retained marker | failed after recovery-only path | active after remediation | absent | empty | unchanged | active |
|
||||
| Next run with stale staging | success | active | absent | empty | +1 only | active |
|
||||
|
||||
- After the terminal restart case, exercise explicit operator remediation that starts Forgejo, confirms it active, and clears the retained marker before continuing.
|
||||
- Do not extract/restore a backup or claim protection from a newly racing SSH push.
|
||||
- **Starts at:** New `modules/forgejo-backup-script.nix`; new `tests/forgejo-backup.nix`; `flake.nix`; P2-T2 behavior.
|
||||
- **Depends on:** P2-T2.
|
||||
- **Tests:** Protect destructive failure paths and ordering that static evaluation cannot establish.
|
||||
- **Verify:** `nix build .#checks.x86_64-linux.forgejo-backup` exits 0 and the log shows every deterministic success/failure case completed; production evaluation contains no test hook.
|
||||
- **Risk/recovery:** Fault injection must remain build-time-only. A failing case blocks backup activation; never reproduce destructive test failures against trial data.
|
||||
|
||||
#### P2-T4 — Expand the runbook with backup and recovery operations
|
||||
|
||||
- **Change:**
|
||||
- Expand `docs/forgejo-operations.md` without weakening its Phase 1 deployment/admin instructions.
|
||||
- Add advisory capacity inspection, an idle/no-push backup window, manual backup invocation, snapshot freshness, service restart and timer checks, rendered-unit inspection, private streamed archive/manifest inspection, staging cleanup, and shared B2 credential rotation.
|
||||
- Update the canonical shared-credential section in `docs/bitcoin-pulse-operations.md` so rotation names and validates Bitcoin Pulse, Nextcloud, and Forgejo rather than only the existing two jobs.
|
||||
- Document recovery using the manifest's Forgejo version, system path, clean configuration revision, and matching committed flake/lock. Enter a maintenance boundary that denies only the `forgejo` SSH account without stopping sshd; stop Forgejo; wait for zero Forgejo-owned processes; restore privately; retain current state under a timestamped rollback path; extract with ownership/ACLs/xattrs; start and verify Forgejo; and restore Forgejo SSH access only after acceptance or rollback. Retain rollback data and the source snapshot until acceptance.
|
||||
- Require preservation or reproducible rebuilding of the matching generation before restoration.
|
||||
- State that content inspection is not a restore test, recoverability remains unproven, sshd stays active as a service, and the overlapping-push race remains accepted for backup creation only.
|
||||
- **Starts at:** `docs/forgejo-operations.md`; `docs/bitcoin-pulse-operations.md:191-222`; P2-T2 paths and units.
|
||||
- **Depends on:** P2-T2, P2-T3.
|
||||
- **Tests:** Protect exact paths/units, complete recovery scope, version-compatible startup, reversible recovery, credential-rotation scope, no-push instructions, and honest claims.
|
||||
- **Verify:**
|
||||
- `git diff --check -- docs/forgejo-operations.md docs/bitcoin-pulse-operations.md` exits 0.
|
||||
- Exact searches confirm the Forgejo runbook names `restic-backups-forgejo.service`, `forgejo-backup-manifest.txt`, the no-push window, snapshot freshness, manifest inspection, `Recovery`, `Rollback`, Forgejo-only SSH denial during recovery, and the unproven-recoverability warning.
|
||||
- Exact searches in `docs/bitcoin-pulse-operations.md` require all three unit names: `restic-backups-bitcoin-pulse.service`, `restic-backups-nextcloud.service`, and `restic-backups-forgejo.service`.
|
||||
- **Risk/recovery:** A newer Forgejo can migrate SQLite and make downgrade unsafe. Recovery must identify/rebuild the matching generation and preserve old/current state and the Restic snapshot until acceptance.
|
||||
|
||||
#### P2-T5 — Build Phase 2 without activation
|
||||
|
||||
- **Change:**
|
||||
- Format all Phase 2 Nix files, run the backup VM check, evaluate the full flake, and build the complete host.
|
||||
- On Tobserver, require a clean reviewed checkout, verify `system.configurationRevision` equals the deployment commit, and build without switching.
|
||||
- Confirm Forgejo remains active and `/var/lib/forgejo` remains intact; Phase 2 augments the trial instance rather than reinitializing it.
|
||||
- **Starts at:** Completed P2-T1 through P2-T4 changes.
|
||||
- **Depends on:** P2-T1, P2-T2, P2-T3, P2-T4.
|
||||
- **Tests:** Protect formatting, VM failure behavior, clean revision identity, full evaluation/build, and trial-state preservation.
|
||||
- **Verify:**
|
||||
- `nixpkgs-fmt --check flake.nix modules/forgejo.nix modules/forgejo-backup.nix modules/forgejo-backup-script.nix tests/forgejo-backup.nix`, `git diff --check`, `nix build .#checks.x86_64-linux.forgejo-backup`, and `nix flake check --no-build` all exit 0.
|
||||
- On a compatible Linux builder, `nix build .#nixosConfigurations.tobserver.config.system.build.toplevel` exits 0.
|
||||
- On Tobserver, re-run G1's exact KEEP-record command, then require `test -z "$(git status --porcelain)"`, `test "$(nix eval --raw .#nixosConfigurations.tobserver.config.system.configurationRevision)" = "$(git rev-parse HEAD)"`, `sudo systemctl is-active forgejo.service`, `sudo test -d /var/lib/forgejo`, and `sudo nixos-rebuild build --flake .#tobserver` to exit 0 without activation.
|
||||
- **Risk/recovery:** A failed check/build blocks backup activation but does not invalidate the running trial. Do not deploy from a dirty revision or bypass the VM check.
|
||||
|
||||
#### P2-T6 — Activate backups and inspect the first production snapshot
|
||||
|
||||
- **Change:**
|
||||
- Switch to the already-built Phase 2 generation.
|
||||
- Record Forgejo's activation timestamp and latest snapshot ID, manually run the backup, and prove Forgejo restarted before upload and a new snapshot was created.
|
||||
- Verify job result, timer, rendered unit, stored tar path, required tar entries and exact manifest, Forgejo activity, marker absence, and local staging cleanup.
|
||||
- Inspect redacted journals for orderly stop/process-check/archive/restart/upload behavior. Do not extract the archive or describe these checks as recovery validation.
|
||||
- **Starts at:** Deployed `restic-backups-forgejo.service`; expanded `docs/forgejo-operations.md`.
|
||||
- **Depends on:** P2-T5.
|
||||
- **Tests:** Protect fresh encrypted upload, complete content, restart-before-upload, cleanup, timer/retention activation, and honest reporting.
|
||||
- **Verify:**
|
||||
- Capture `before_service=$(systemctl show forgejo.service --property=ActiveEnterTimestampMonotonic --value)` and `before_snapshot=$(sudo restic-forgejo snapshots --json --latest 1 2>/dev/null | jq -r '.[0].id // empty')`; run `sudo systemctl start restic-backups-forgejo.service`.
|
||||
- `systemctl show restic-backups-forgejo.service --property=Result --property=ExecMainStatus` reports `Result=success` and `ExecMainStatus=0`; Forgejo is active; its activation timestamp increased; `/run/forgejo-backup/forgejo-stopped` is absent.
|
||||
- Capture `after_snapshot=$(sudo restic-forgejo snapshots --json --latest 1 | jq -er '.[0].id')`; require it to be non-empty and different from `$before_snapshot`.
|
||||
- `sudo restic-forgejo ls "$after_snapshot" | grep -Fx '/var/backup/forgejo/forgejo-state.tar'` succeeds.
|
||||
- Under `set -o pipefail`, privately stream the tar listing and require `forgejo/data/forgejo.db`, `forgejo/repositories/`, `forgejo/custom/conf/`, `forgejo/.ssh/`, and `forgejo-backup-manifest.txt`. Privately extract only the manifest and compare its exact four ordered lines, with no extras, to the deployed package version, `readlink -f /run/current-system`, and `nixos-version --configuration-revision`.
|
||||
- `sudo stat --format='%U %G %a' /run/secrets/restic/forgejo-password` prints `root root 400`.
|
||||
- `test -z "$(sudo find /var/backup/forgejo -mindepth 1 -print -quit)"` exits 0, proving no file, directory, or symlink remains. `systemctl is-enabled restic-backups-forgejo.timer` prints `enabled`; `systemctl show restic-backups-forgejo.timer --property=TimersCalendar --property=Persistent --property=RandomizedDelayUSec` reports daily 04:00, `Persistent=yes`, and 30 minutes randomized delay.
|
||||
- **Risk/recovery:** Upload/content inspection does not prove restoration or close the SSH race. On failure, first ensure Forgejo is active, inspect the marker and redacted journals, and revert only the separate backup module if necessary. The deployment remains unprotected until this task fully passes.
|
||||
|
||||
### Phase 2 acceptance
|
||||
|
||||
- The dedicated Restic password is encrypted in SOPS and deployed root-only; the shared B2 credential is reused without duplication.
|
||||
- The job stops Forgejo, rejects remaining Forgejo-owned processes, archives all `/var/lib/forgejo` plus the exact version/system/revision manifest, restarts Forgejo before upload, and cleans staging.
|
||||
- The daily 04:00 persistent timer, up to 30 minutes random delay, and 7/4/12 retention are active.
|
||||
- The build-only VM check passes all approved failure paths and cannot inject faults into production.
|
||||
- The runbook matches deployed paths and clearly states that restore testing was not performed.
|
||||
- The first production snapshot is fresh and its archive/manifest content has been inspected successfully.
|
||||
- Only after these checks pass is the retained deployment considered backup-protected. Recoverability remains unproven because restore tests are excluded.
|
||||
|
||||
## Final acceptance
|
||||
|
||||
- **Checks:** Phase 1 passes DNS revalidation, formatting, full evaluation/build, empty-state, service, TLS, registration, listener, admin, and disposable HTTPS/SSH repository checks without adding backup infrastructure. Exactly one dated trial decision is then recorded in this plan. KEEP authorizes Phase 2 and preserves state; ABANDON stops this plan and skips Phase 2 pending a separate removal plan. After KEEP, the VM check, full build, first production backup, snapshot freshness/content, manifest, timer, retention, restart, and cleanup checks all pass.
|
||||
- **End state:** Before the gate, a minimal but explicitly unprotected Forgejo trial is available. After KEEP and Phase 2, the same retained instance has encrypted daily off-server backups and documented recovery/rollback procedures. After ABANDON, no backup infrastructure is added and removal is handled separately.
|
||||
- **Deferrals or blockers:** Phase 1 has no known external blocker beyond its normal pre-activation checks. Phase 2 is intentionally blocked on the future KEEP decision. Restore testing and alerting remain deferred; sshd remains available during backup, leaving the accepted overlapping-push consistency race.
|
||||
|
|
@ -8,6 +8,8 @@
|
|||
enableSSHSupport = true;
|
||||
|
||||
};
|
||||
|
||||
mosh.enable = true;
|
||||
};
|
||||
|
||||
nix.settings.experimental-features = [ "nix-command" "flakes" ];
|
||||
|
|
@ -117,5 +119,11 @@
|
|||
defaultSopsFile = ./secrets/tobserver.yaml;
|
||||
defaultSopsFormat = "yaml";
|
||||
age.keyFile = "/var/lib/sops-nix/key.txt";
|
||||
|
||||
secrets."backblaze/tobcloud-backup-env" = {
|
||||
owner = "root";
|
||||
group = "root";
|
||||
mode = "0400";
|
||||
};
|
||||
};
|
||||
}
|
||||
|
|
|
|||
67
docs/bitcoin-pulse-ingress.md
Normal file
67
docs/bitcoin-pulse-ingress.md
Normal file
|
|
@ -0,0 +1,67 @@
|
|||
# Bitcoin Pulse public ingress
|
||||
|
||||
Bitcoin Pulse uses the dedicated API hostname
|
||||
`bitcoin-pulse.tobiasostner.de`. The native iOS client calls this hostname
|
||||
without a path prefix. A same-origin path proxy is unnecessary because there is
|
||||
no browser frontend, and no CORS headers are configured because native iOS
|
||||
requests are not subject to browser CORS enforcement.
|
||||
|
||||
## DNS and TLS
|
||||
|
||||
The `bitcoin-pulse` A record under `tobiasostner.de` points to Tobserver's public
|
||||
IPv4 address, `162.55.103.218`. There is no AAAA record until Tobserver has
|
||||
working public IPv6 connectivity.
|
||||
|
||||
The NixOS nginx virtual host obtains and renews its certificate through the
|
||||
existing ACME configuration and redirects plaintext HTTP to HTTPS. Only ports
|
||||
80 and 443 are public ingress for the API. The application listens on
|
||||
`127.0.0.1:3000`; port 3000 is not opened by the host firewall, so nginx is the
|
||||
only public path to the backend.
|
||||
|
||||
## Proxy policy
|
||||
|
||||
nginx preserves the original request path and forwards it to
|
||||
`http://127.0.0.1:3000`. It supplies `Host`, `X-Real-IP`, `X-Forwarded-For`, and
|
||||
`X-Forwarded-Proto`, and uses short local-upstream timeouts: two seconds to
|
||||
connect and ten seconds to read or send.
|
||||
|
||||
Requests are limited per source IP to an average of 300 requests per minute,
|
||||
with a burst of 50 requests, and rejected with HTTP 429 when the burst is
|
||||
exhausted. This allows normal polling and multiple devices behind a shared NAT
|
||||
while bounding abusive traffic. Request bodies are limited to 16 KiB; the
|
||||
current public API consists only of small GET requests.
|
||||
|
||||
## Deployment verification
|
||||
|
||||
After a human reviews and merges the change, deploy the resulting NixOS
|
||||
generation and verify locally on Tobserver:
|
||||
|
||||
```console
|
||||
systemctl is-active nginx.service bitcoin-pulse.service
|
||||
ss -ltn '( sport = :3000 )'
|
||||
curl --fail --show-error --silent http://127.0.0.1:3000/health \
|
||||
| jq -e '.status == "OK"'
|
||||
```
|
||||
|
||||
The socket listing must show only `127.0.0.1:3000`, not `0.0.0.0:3000` or the
|
||||
public address. From a machine outside Tobserver, verify DNS, HTTPS, the health
|
||||
endpoint, and each functional endpoint:
|
||||
|
||||
```console
|
||||
dig +short A bitcoin-pulse.tobiasostner.de
|
||||
curl --fail --show-error --silent \
|
||||
https://bitcoin-pulse.tobiasostner.de/health \
|
||||
| jq -e '.status == "OK"'
|
||||
curl --fail --show-error --silent \
|
||||
https://bitcoin-pulse.tobiasostner.de/block-height | jq -e .
|
||||
curl --fail --show-error --silent \
|
||||
https://bitcoin-pulse.tobiasostner.de/difficulty-adjustment | jq -e .
|
||||
curl --fail --show-error --silent \
|
||||
https://bitcoin-pulse.tobiasostner.de/price | jq -e .
|
||||
```
|
||||
|
||||
The A lookup must return `162.55.103.218`. The HTTPS certificate must be valid
|
||||
for `bitcoin-pulse.tobiasostner.de`; `/health` must return HTTP 200 and
|
||||
`{"status":"OK"}`. The functional endpoints must return HTTP 200 and JSON after
|
||||
the collectors have populated their data. A connection from outside Tobserver
|
||||
to TCP port 3000 must fail.
|
||||
250
docs/bitcoin-pulse-operations.md
Normal file
250
docs/bitcoin-pulse-operations.md
Normal file
|
|
@ -0,0 +1,250 @@
|
|||
# Bitcoin Pulse production operations
|
||||
|
||||
This runbook covers routine Bitcoin Pulse deployment and maintenance on
|
||||
Tobserver. The declarative configuration lives in `modules/bitcoin-pulse.nix`;
|
||||
PostgreSQL details are in `bitcoin-pulse-postgresql.md`, and public ingress
|
||||
checks are in `bitcoin-pulse-ingress.md`.
|
||||
|
||||
Run production commands from the Tobserver checkout unless noted otherwise.
|
||||
Never place plaintext credentials in Git, the Nix store, command arguments,
|
||||
shell history, logs, or issue comments.
|
||||
|
||||
## Standard deployment
|
||||
|
||||
Prepare and review every change on a workstation before touching the server:
|
||||
|
||||
```console
|
||||
nix flake check --no-build
|
||||
nix build .#nixosConfigurations.tobserver.config.system.build.toplevel
|
||||
```
|
||||
|
||||
The second command can run only on a compatible Linux builder. It is optional
|
||||
on a workstation without one because the server build below performs the same
|
||||
full configuration build before activation.
|
||||
|
||||
Commit and push the reviewed change. On Tobserver, update the clean checkout and
|
||||
build without changing the running system:
|
||||
|
||||
```console
|
||||
cd ~/tobserver
|
||||
test -z "$(git status --porcelain)" || {
|
||||
echo "Refusing to deploy from a dirty checkout" >&2
|
||||
exit 1
|
||||
}
|
||||
git pull --ff-only
|
||||
git rev-parse HEAD
|
||||
nix flake check --no-build
|
||||
sudo nixos-rebuild build --flake .#tobserver
|
||||
```
|
||||
|
||||
The cleanliness check stops the procedure before pulling when the checkout has
|
||||
local changes. Compare the printed commit with the approved deployment commit.
|
||||
Review build failures before continuing. A successful `build` creates the
|
||||
ignored `result` symlink but does not restart any service.
|
||||
|
||||
Activate the already-reviewed configuration:
|
||||
|
||||
```console
|
||||
sudo nixos-rebuild switch --flake .#tobserver
|
||||
```
|
||||
|
||||
Do not use a live full-host NixOS generation rollback as a Bitcoin Pulse test or
|
||||
routine recovery action. Tobserver runs unrelated production services that a
|
||||
full generation switch can also change.
|
||||
|
||||
## Post-deployment verification
|
||||
|
||||
Verify service state, migration completion, loopback-only listeners, and local
|
||||
responses:
|
||||
|
||||
```console
|
||||
sudo systemctl is-active \
|
||||
podman-bitcoin-pulse-postgres.service \
|
||||
bitcoin-pulse.service \
|
||||
nginx.service
|
||||
|
||||
sudo systemctl show bitcoin-pulse-migrate.service \
|
||||
--property=ActiveState \
|
||||
--property=Result \
|
||||
--property=ExecMainStatus
|
||||
|
||||
sudo ss -ltnp | grep -E ':(3000|5432)\b'
|
||||
|
||||
curl --fail --show-error --silent http://127.0.0.1:3000/health \
|
||||
| jq -e '.status == "OK"'
|
||||
curl --fail --show-error --silent http://127.0.0.1:3000/block-height \
|
||||
| jq -e .
|
||||
curl --fail --show-error --silent http://127.0.0.1:3000/difficulty-adjustment \
|
||||
| jq -e .
|
||||
curl --fail --show-error --silent http://127.0.0.1:3000/price \
|
||||
| jq -e .
|
||||
```
|
||||
|
||||
All three long-running services must be active. The migration unit is a
|
||||
completed one-shot, so `ActiveState=inactive`, `Result=success`, and
|
||||
`ExecMainStatus=0` are expected. PostgreSQL and the backend must listen only on
|
||||
`127.0.0.1:5432` and `127.0.0.1:3000` respectively.
|
||||
|
||||
From a machine outside Tobserver, run the public checks in
|
||||
`bitcoin-pulse-ingress.md`. Inspect relevant logs without copying credentials:
|
||||
|
||||
```console
|
||||
sudo journalctl \
|
||||
-u podman-bitcoin-pulse-postgres.service \
|
||||
-u bitcoin-pulse-migrate.service \
|
||||
-u bitcoin-pulse.service \
|
||||
-u nginx.service \
|
||||
--since "-15 minutes" \
|
||||
--no-pager
|
||||
```
|
||||
|
||||
## Updating the Bitcoin Pulse backend
|
||||
|
||||
Update only the pinned backend input from a workstation:
|
||||
|
||||
```console
|
||||
cd ~/Code/Tobserver
|
||||
nix flake update bitcoin-pulse
|
||||
git diff -- flake.lock
|
||||
nix flake check --no-build
|
||||
```
|
||||
|
||||
Confirm that `flake.lock` points to the intended reviewed BitcoinPulse
|
||||
revision. Commit and push the lock-file update, follow the standard deployment,
|
||||
and perform all post-deployment checks. The switch runs the packaged version's
|
||||
pending migrations before starting that backend version.
|
||||
|
||||
Every database migration must remain compatible with the previously deployed
|
||||
application while it is still a possible recovery version. Do not run
|
||||
`bitcoin-pulse-migrate rollback` automatically.
|
||||
|
||||
## Backups
|
||||
|
||||
The daily logical backup, retention policy, credentials, and initial
|
||||
verification procedure are documented in `bitcoin-pulse-postgresql.md`.
|
||||
|
||||
Before a database image change or other maintenance with data risk, create and
|
||||
verify a fresh snapshot:
|
||||
|
||||
```console
|
||||
sudo systemctl start restic-backups-bitcoin-pulse.service
|
||||
sudo systemctl show restic-backups-bitcoin-pulse.service \
|
||||
--property=Result \
|
||||
--property=ExecMainStatus
|
||||
sudo restic-bitcoin-pulse snapshots --latest 1
|
||||
sudo restic-bitcoin-pulse ls latest | grep bitcoin-pulse.dump
|
||||
sudo find /var/backup/bitcoin-pulse -maxdepth 1 -type f -ls
|
||||
```
|
||||
|
||||
Expect `Result=success`, `ExecMainStatus=0`, a snapshot containing
|
||||
`bitcoin-pulse.dump`, and no local staging file after completion. Alerting and
|
||||
routine restore tests are separate backlog work.
|
||||
|
||||
## PostgreSQL image updates within the current major version
|
||||
|
||||
Automatic image updates are disabled. For a PostgreSQL 18 patch or minor image
|
||||
update:
|
||||
|
||||
1. Read the official PostgreSQL image release notes and confirm the data
|
||||
directory remains compatible.
|
||||
2. Create and verify a fresh Bitcoin Pulse backup.
|
||||
3. Test the candidate image with the backend package and migrations outside
|
||||
production where practical.
|
||||
4. Verify that the candidate digest identifies the intended PostgreSQL 18
|
||||
Bookworm patch release, then replace the immutable digest in
|
||||
`modules/bitcoin-pulse.nix`.
|
||||
5. Update the adjacent exact-version and tested-date comment and the declared
|
||||
version in `bitcoin-pulse-postgresql.md` in the same change.
|
||||
6. Run `nixpkgs-fmt --check modules/bitcoin-pulse.nix` and
|
||||
`nix flake check --no-build`.
|
||||
7. Review, commit, push, and follow the standard deployment procedure.
|
||||
8. Repeat all post-deployment and backup checks.
|
||||
|
||||
Never enable Podman automatic updates or deploy a floating image tag.
|
||||
|
||||
## PostgreSQL major upgrades
|
||||
|
||||
A major PostgreSQL upgrade is a separate maintenance project, not a routine
|
||||
image-digest change. A new major version must never start against the existing
|
||||
`bitcoin-pulse-db` volume because PostgreSQL data directories are not
|
||||
major-version compatible.
|
||||
|
||||
Before approving a major upgrade, prepare a version-specific plan that:
|
||||
|
||||
1. Defines a maintenance window and stops the Bitcoin Pulse backend from
|
||||
accepting writes during the final export.
|
||||
2. Creates and verifies a fresh logical backup with the old PostgreSQL version.
|
||||
3. Provisions a new, separately named Podman volume for the new major version.
|
||||
4. Initializes the administrator, migration, and application roles in the new
|
||||
cluster without changing or deleting the old volume.
|
||||
5. Restores the logical dump into the new cluster before normal backend startup.
|
||||
6. Runs Migratus, starts the pinned backend, and repeats database, endpoint,
|
||||
collector, ingress, and backup verification.
|
||||
7. Keeps the old volume untouched until the new cluster has been explicitly
|
||||
accepted and a new backup has completed.
|
||||
8. Documents a version-specific recovery path before removing any old data.
|
||||
|
||||
Do not improvise this procedure directly on production. The exact export,
|
||||
restore, and compatibility steps depend on the source and target PostgreSQL
|
||||
versions and require a separately reviewed implementation change.
|
||||
|
||||
## Secret rotation
|
||||
|
||||
PostgreSQL role rotation is documented in the credential-rotation section of
|
||||
`bitcoin-pulse-postgresql.md`. Rotate one role at a time and synchronize the
|
||||
database role password with its SOPS value during a maintenance window.
|
||||
|
||||
The shared `backblaze/tobcloud-backup-env` credential affects both Bitcoin Pulse
|
||||
and Nextcloud backups. Coordinate its rotation across both services, deploy the
|
||||
new SOPS value once, and verify read, write, and retention access by running
|
||||
both backup jobs during a maintenance window:
|
||||
|
||||
```console
|
||||
sudo systemctl start restic-backups-bitcoin-pulse.service
|
||||
sudo systemctl start restic-backups-nextcloud.service
|
||||
|
||||
sudo systemctl show \
|
||||
restic-backups-bitcoin-pulse.service \
|
||||
restic-backups-nextcloud.service \
|
||||
--property=Id \
|
||||
--property=Result \
|
||||
--property=ExecMainStatus
|
||||
|
||||
sudo restic-bitcoin-pulse snapshots --latest 1
|
||||
sudo restic-nextcloud snapshots --latest 1
|
||||
```
|
||||
|
||||
Both units must report `Result=success` and `ExecMainStatus=0`, and both
|
||||
repositories must contain a fresh snapshot.
|
||||
|
||||
A Restic repository password is independent of the Backblaze application key.
|
||||
Retain each repository password in the approved password manager; losing one
|
||||
makes that repository unreadable.
|
||||
|
||||
## First-response diagnostics
|
||||
|
||||
For an unhealthy deployment, gather state before restarting services:
|
||||
|
||||
```console
|
||||
sudo systemctl status \
|
||||
podman-bitcoin-pulse-postgres.service \
|
||||
bitcoin-pulse-migrate.service \
|
||||
bitcoin-pulse.service \
|
||||
nginx.service \
|
||||
--no-pager
|
||||
|
||||
sudo journalctl \
|
||||
-u podman-bitcoin-pulse-postgres.service \
|
||||
-u bitcoin-pulse-migrate.service \
|
||||
-u bitcoin-pulse.service \
|
||||
-u nginx.service \
|
||||
-b \
|
||||
-n 300 \
|
||||
--no-pager
|
||||
```
|
||||
|
||||
A migration failure intentionally prevents backend startup. Correct the
|
||||
database readiness, migration, or credential error and then restart
|
||||
`bitcoin-pulse.service`; its dependency starts the idempotent migration unit
|
||||
again. Do not delete the PostgreSQL volume, bypass migrations, expose ports 3000
|
||||
or 5432 publicly, or paste unredacted logs into an issue.
|
||||
|
|
@ -13,15 +13,15 @@
|
|||
- Host port 5432 is published only as `127.0.0.1:5432` and is not opened in the
|
||||
NixOS firewall.
|
||||
- Podman reports the service ready only after `pg_isready` succeeds.
|
||||
- PostgreSQL receives `SIGINT` for smart shutdown, with 60 seconds for Podman
|
||||
and 90 seconds for the enclosing systemd service.
|
||||
- PostgreSQL receives `SIGINT` for a clean fast shutdown, with 60 seconds for
|
||||
Podman and 90 seconds for the enclosing systemd service.
|
||||
- Automatic image updates are disabled. Update the digest only in a reviewed
|
||||
change after testing backups, startup, and application compatibility.
|
||||
|
||||
The named volume belongs to the rootless Podman account and survives container
|
||||
removal, service recreation, and NixOS generation changes. Backup and restore
|
||||
portability is handled separately by BIT-23; do not copy Podman's private
|
||||
storage directly.
|
||||
removal, service recreation, and NixOS generation changes. Backup portability
|
||||
is provided by the logical Restic backup described below; do not copy Podman's
|
||||
private storage directly.
|
||||
|
||||
## Human-managed secrets
|
||||
|
||||
|
|
@ -41,12 +41,80 @@ Nix expression, command argument, shell history, log, or pull-request text. The
|
|||
agent change intentionally declares only the secret paths; it does not modify
|
||||
or decrypt `secrets/tobserver.yaml`.
|
||||
|
||||
`sops-nix` materializes each credential as a restricted runtime file. Before
|
||||
container startup, the service stages private copies owned by PostgreSQL's uid
|
||||
## Logical off-server backups
|
||||
|
||||
`modules/bitcoin-pulse.nix` declares the daily
|
||||
`restic-backups-bitcoin-pulse.service` and its systemd timer. The job:
|
||||
|
||||
1. Requires the PostgreSQL container service to be available.
|
||||
2. Creates a consistent custom-format `pg_dump` through the loopback database
|
||||
port using the administrator password as a systemd credential.
|
||||
3. Uploads the dump to the separate Restic repository
|
||||
`b2:tobcloud-backup:bitcoin-pulse`.
|
||||
4. Removes the local staging dump whether the upload succeeds or fails.
|
||||
5. Retains 7 daily, 4 weekly, and 12 monthly snapshots.
|
||||
|
||||
The timer starts the job daily between 03:00 and 03:30 Europe/Berlin time. It is
|
||||
persistent, so systemd starts a missed job after the next boot. A failed dump,
|
||||
upload, or prune operation makes the unit fail and leaves details in the system
|
||||
journal. A run that exceeds six hours is stopped and marked failed so it cannot
|
||||
block future timer runs indefinitely. Alert delivery and automated restore
|
||||
testing are intentionally outside this deployment scope.
|
||||
|
||||
### Backup credentials
|
||||
|
||||
The existing `tobcloud-backup` bucket and its Backblaze application key are
|
||||
shared with the Nextcloud backup. Before deploying, confirm that the key is not
|
||||
restricted only to the `nextcloud/` prefix and can also access
|
||||
`bitcoin-pulse/`. For the native B2 backend, it needs `listBuckets`,
|
||||
`listFiles`, `readFiles`, `writeFiles`, and `deleteFiles` because Restic must
|
||||
find the bucket and retention must remove expired repository files.
|
||||
|
||||
Generate a separate, strong Restic repository password and retain it in the
|
||||
approved password manager. Losing this password makes every backup in the
|
||||
repository unusable.
|
||||
|
||||
Use `sops secrets/tobserver.yaml` from an authorized workstation to add:
|
||||
|
||||
```yaml
|
||||
restic:
|
||||
bitcoin-pulse-password: <new Restic repository password>
|
||||
backblaze:
|
||||
tobcloud-backup-env: |
|
||||
B2_ACCOUNT_ID=<shared application key ID>
|
||||
B2_ACCOUNT_KEY=<shared application key>
|
||||
```
|
||||
|
||||
Rename the existing `backblaze/nextcloud-env` SOPS entry to
|
||||
`backblaze/tobcloud-backup-env`; do not keep a second plaintext copy of the same
|
||||
credential. Commit only the encrypted result. The NixOS configuration will not
|
||||
activate successfully until the renamed Backblaze key and the new Restic
|
||||
password exist.
|
||||
|
||||
### Initial backup verification
|
||||
|
||||
After adding the credentials and deploying the reviewed NixOS generation, run
|
||||
the first backup manually rather than waiting for the timer:
|
||||
|
||||
```console
|
||||
sudo systemctl start restic-backups-bitcoin-pulse.service
|
||||
sudo systemctl status restic-backups-bitcoin-pulse.service
|
||||
sudo journalctl -u restic-backups-bitcoin-pulse.service -b --no-pager
|
||||
sudo restic-bitcoin-pulse snapshots
|
||||
systemctl list-timers restic-backups-bitcoin-pulse.timer
|
||||
```
|
||||
|
||||
The service must exit successfully, `snapshots` must show the new off-server
|
||||
snapshot, and `/var/backup/bitcoin-pulse` must not retain a dump after the job.
|
||||
The journal must not contain any credential value.
|
||||
|
||||
For PostgreSQL, `sops-nix` materializes each credential as a restricted runtime
|
||||
file. Before container startup, the service stages private copies owned by
|
||||
PostgreSQL's uid
|
||||
inside the rootless user namespace. The copies are mounted read-only and are
|
||||
removed from the host runtime directory after PostgreSQL reports healthy and
|
||||
again when the unit stops. The application and migration SOPS files remain
|
||||
available only through their dedicated host groups for the later BIT-28 units.
|
||||
available only through the dedicated groups of their respective host services.
|
||||
|
||||
## Empty-database initialization and privileges
|
||||
|
||||
|
|
@ -58,7 +126,7 @@ creates:
|
|||
- `bitcoin_pulse_migration`, a non-superuser login that owns the database;
|
||||
- `bitcoin_pulse_app`, a non-superuser login with `CONNECT` only initially.
|
||||
|
||||
The role bootstrap SQL comes from the pinned BitcoinPulseAPI flake input.
|
||||
The role bootstrap SQL comes from the pinned BitcoinPulse flake input.
|
||||
Migratus' database-foundation migration then grants the application role schema
|
||||
usage and default DML privileges for application objects. Later migrations can
|
||||
further restrict individual tables; the application role cannot create schema
|
||||
|
|
@ -67,11 +135,36 @@ objects, roles, or databases and must never run migrations.
|
|||
Initialization scripts run only for an empty volume. Changing a SOPS value does
|
||||
not change a role password in an existing PostgreSQL cluster.
|
||||
|
||||
## Migration-gated backend startup
|
||||
|
||||
`bitcoin-pulse.service` requires `bitcoin-pulse-migrate.service`, and the
|
||||
migration service requires the health-gated PostgreSQL container service. The
|
||||
resulting startup order is:
|
||||
|
||||
```text
|
||||
PostgreSQL healthy -> bounded readiness check -> Migratus -> backend
|
||||
```
|
||||
|
||||
The migration service waits up to 60 seconds for `pg_isready`, then runs the
|
||||
same packaged backend version's `bitcoin-pulse-migrate migrate` command as the
|
||||
unprivileged `bitcoin-pulse-migration` host user. It receives only the migration
|
||||
role and its SOPS password-file path. The backend separately receives the
|
||||
restricted `bitcoin_pulse_app` role and application password-file path.
|
||||
|
||||
The migration unit is a one-shot service without `RemainAfterExit`, so every
|
||||
new start transaction that pulls it in runs Migratus again. Migratus records
|
||||
completed versions in `schema_migrations`; already-applied migrations therefore
|
||||
make repeated starts harmless. A readiness timeout, invalid credential, or
|
||||
migration error fails the one-shot unit. Because the backend is ordered after
|
||||
and requires that unit, the backend process is not started after such a failure.
|
||||
Inspect `journalctl -u bitcoin-pulse-migrate.service` for the bounded readiness
|
||||
message, Migratus error, and systemd exit status.
|
||||
|
||||
## Deployment and verification
|
||||
|
||||
A human must review and merge the change, provision the encrypted keys above,
|
||||
and deploy the resulting NixOS generation manually. The agent does not access
|
||||
or rebuild Tobserver.
|
||||
Use `bitcoin-pulse-operations.md` for the standard build, switch, and complete
|
||||
post-deployment procedure. The checks below cover the database-specific
|
||||
acceptance paths.
|
||||
|
||||
After deployment, verify the service and loopback binding:
|
||||
|
||||
|
|
@ -97,9 +190,9 @@ SELECT has_schema_privilege(
|
|||
'bitcoin_pulse_app', 'public', 'CREATE');
|
||||
```
|
||||
|
||||
The expected results are `true`, `false`, and `false`. After BIT-28 runs the
|
||||
foundation migration, also verify the application can perform only the DML
|
||||
allowed by the migrated schema and that only the migration role can apply DDL.
|
||||
The expected results are `true`, `false`, and `false`. Verify the application
|
||||
can perform only the DML allowed by the migrated schema and that only the
|
||||
migration role can apply DDL.
|
||||
|
||||
To test recreation and persistence, create a temporary probe table as the
|
||||
administrator or migration role, restart the unit, confirm the row remains, and
|
||||
|
|
@ -120,9 +213,42 @@ DROP TABLE bit29_recreation_probe;
|
|||
```
|
||||
|
||||
A normal `systemctl stop` must complete within 90 seconds. Review the journal to
|
||||
confirm a smart PostgreSQL shutdown and no forced kill. Start the unit again and
|
||||
confirm it becomes healthy. Perform these production checks during a reviewed
|
||||
maintenance window.
|
||||
confirm a clean fast PostgreSQL shutdown and no forced kill. Start the unit again
|
||||
and confirm it becomes healthy. Perform these production checks during a
|
||||
reviewed maintenance window.
|
||||
|
||||
After deploying the migration gate, verify the one-shot and backend:
|
||||
|
||||
```console
|
||||
systemctl is-active podman-bitcoin-pulse-postgres.service
|
||||
systemctl is-active bitcoin-pulse.service
|
||||
systemctl show bitcoin-pulse-migrate.service \
|
||||
--property=Result --property=ExecMainStatus
|
||||
journalctl -u bitcoin-pulse-migrate.service -b --no-pager
|
||||
ss -ltn '( sport = :3000 )'
|
||||
```
|
||||
|
||||
A successful migration unit becomes inactive after it exits; `Result=success`
|
||||
and `ExecMainStatus=0` are expected. The backend must be active and port 3000
|
||||
must be bound only to `127.0.0.1`. Restart `bitcoin-pulse.service` once and
|
||||
confirm the migration unit runs successfully again without reapplying completed
|
||||
migrations.
|
||||
|
||||
Successful and repeated paths may be verified during deployment. Exercise the
|
||||
database-unavailable and invalid-migration-credential paths only on a disposable
|
||||
NixOS test host: both must make `bitcoin-pulse-migrate.service` fail, leave
|
||||
`bitcoin-pulse.service` stopped, finish within the configured timeout, and emit
|
||||
an actionable journal message. Never inject invalid credentials into production.
|
||||
|
||||
## Application compatibility constraints
|
||||
|
||||
A full-host NixOS generation rollback is not part of the Bitcoin Pulse
|
||||
production procedure because it can affect unrelated Tobserver services and it
|
||||
does not reverse PostgreSQL migrations. Do not use `bitcoin-pulse-migrate
|
||||
rollback` as an automatic deployment action. Every schema change must remain
|
||||
compatible with the previously deployed application while that version remains
|
||||
a possible recovery version. Use an expand/migrate/contract sequence for
|
||||
breaking changes.
|
||||
|
||||
## Credential rotation for an existing database
|
||||
|
||||
|
|
|
|||
216
docs/forgejo-operations.md
Normal file
216
docs/forgejo-operations.md
Normal file
|
|
@ -0,0 +1,216 @@
|
|||
# Forgejo trial operations
|
||||
|
||||
This runbook covers the initial Forgejo trial on Tobserver. The complete
|
||||
service, SQLite, package, and nginx configuration lives in
|
||||
`modules/forgejo.nix`.
|
||||
|
||||
> **Warning:** This trial is unprotected. It has no Forgejo backup job, and its
|
||||
> data must be treated as disposable. Keep the authoritative copy of anything
|
||||
> important somewhere else.
|
||||
|
||||
Run deployment commands from the Tobserver checkout unless noted otherwise.
|
||||
Never place passwords or other credentials in Git, Nix expressions, command
|
||||
arguments, shell history, logs, or issue comments.
|
||||
|
||||
## Pre-deployment checks
|
||||
|
||||
Confirm that the dedicated hostname points only to Tobserver:
|
||||
|
||||
```console
|
||||
test "$(dig +short git.tobiasostner.de A)" = "162.55.103.218"
|
||||
test -z "$(dig +short git.tobiasostner.de AAAA)"
|
||||
```
|
||||
|
||||
On a workstation, evaluate the configuration and build it on a compatible Linux
|
||||
builder when one is available:
|
||||
|
||||
```console
|
||||
nix flake check --no-build
|
||||
nix build .#nixosConfigurations.tobserver.config.system.build.toplevel
|
||||
```
|
||||
|
||||
Commit and push the reviewed changes. On Tobserver, update a clean checkout and
|
||||
build without activating it:
|
||||
|
||||
```console
|
||||
cd ~/tobserver
|
||||
test -z "$(git status --porcelain)" || {
|
||||
echo "Refusing to deploy from a dirty checkout" >&2
|
||||
exit 1
|
||||
}
|
||||
git pull --ff-only
|
||||
git rev-parse HEAD
|
||||
nix flake check --no-build
|
||||
|
||||
if sudo test -e /var/lib/forgejo \
|
||||
&& sudo find /var/lib/forgejo -mindepth 1 -print -quit | grep -q .; then
|
||||
echo "Refusing to reuse non-empty Forgejo state" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
sudo nixos-rebuild build --flake .#tobserver
|
||||
```
|
||||
|
||||
Compare the printed commit with the reviewed deployment commit. Do not continue
|
||||
if evaluation, the full host build, or the empty-state check fails.
|
||||
|
||||
## Activation and service checks
|
||||
|
||||
Activate the configuration only after reviewing the successful build. Immediately
|
||||
before switching, recheck that the checkout is clean and still at the reviewed
|
||||
commit printed during the build step:
|
||||
|
||||
```console
|
||||
read -r -p 'Reviewed commit hash: ' reviewed_commit
|
||||
test -z "$(git status --porcelain)" || {
|
||||
echo "Refusing to switch from a dirty checkout" >&2
|
||||
exit 1
|
||||
}
|
||||
current_commit="$(git rev-parse HEAD)"
|
||||
printf 'Switching commit: %s\n' "$current_commit"
|
||||
test "$current_commit" = "$reviewed_commit" || {
|
||||
echo "Refusing to switch an unreviewed commit" >&2
|
||||
exit 1
|
||||
}
|
||||
sudo nixos-rebuild switch --flake .#tobserver
|
||||
unset current_commit reviewed_commit
|
||||
```
|
||||
|
||||
Verify the services, loopback-only backend, local API, rendered nginx
|
||||
configuration, and public HTTPS endpoint:
|
||||
|
||||
```console
|
||||
sudo systemctl is-active forgejo.service nginx.service
|
||||
sudo ss -H -ltn 'sport = :3001'
|
||||
|
||||
curl --fail --silent --show-error \
|
||||
-H 'Host: git.tobiasostner.de' \
|
||||
http://127.0.0.1:3001/api/v1/version \
|
||||
| jq -e .
|
||||
|
||||
sudo nginx -t
|
||||
(
|
||||
set -euo pipefail
|
||||
nginx_config="$(sudo nginx -T 2>&1)"
|
||||
for required_line in \
|
||||
'git.tobiasostner.de' \
|
||||
'merge_slashes off' \
|
||||
'client_max_body_size 512M' \
|
||||
'proxy_set_header Host $host' \
|
||||
'proxy_set_header X-Real-IP $remote_addr' \
|
||||
'proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for' \
|
||||
'proxy_set_header X-Forwarded-Proto $scheme' \
|
||||
'proxy_set_header Connection $http_connection' \
|
||||
'proxy_set_header Upgrade $http_upgrade'
|
||||
do
|
||||
printf '%s\n' "$nginx_config" | grep -F -- "$required_line" >/dev/null
|
||||
done
|
||||
)
|
||||
|
||||
curl --fail --silent --show-error --location \
|
||||
--output /dev/null \
|
||||
--write-out '%{http_code} %{url_effective}\n' \
|
||||
http://git.tobiasostner.de/
|
||||
```
|
||||
|
||||
The listener output must contain `127.0.0.1:3001` and no non-loopback address.
|
||||
The public request must report HTTP 200 at `https://git.tobiasostner.de/`.
|
||||
Port 3001 must not be reachable from an external machine:
|
||||
|
||||
```console
|
||||
! nc -z git.tobiasostner.de 3001
|
||||
```
|
||||
|
||||
If a check fails, inspect logs before restarting or changing state:
|
||||
|
||||
```console
|
||||
sudo journalctl \
|
||||
-u forgejo.service \
|
||||
-u nginx.service \
|
||||
--since '-15 minutes' \
|
||||
--no-pager
|
||||
```
|
||||
|
||||
## Create the first administrator
|
||||
|
||||
Run this once in an interactive shell on Tobserver. Enter the administrator
|
||||
email when prompted; it stays in the shell variable and is not recorded in this
|
||||
runbook or shell history.
|
||||
|
||||
```console
|
||||
read -r -p 'Admin email: ' admin_email
|
||||
sudo -u forgejo env \
|
||||
HOME=/var/lib/forgejo \
|
||||
USER=forgejo \
|
||||
FORGEJO_WORK_DIR=/var/lib/forgejo \
|
||||
FORGEJO_CUSTOM=/var/lib/forgejo/custom \
|
||||
forgejo --config /var/lib/forgejo/custom/conf/app.ini \
|
||||
admin user create \
|
||||
--username "oibot" \
|
||||
--email "$admin_email" \
|
||||
--admin \
|
||||
--random-password
|
||||
unset admin_email
|
||||
```
|
||||
|
||||
The command prints a generated password once. Copy it directly into the
|
||||
approved password manager immediately, then clear the terminal. Do not redirect,
|
||||
pipe, log, or save the output in a temporary file, and do not choose a password
|
||||
on the command line.
|
||||
|
||||
Confirm that the account is an administrator:
|
||||
|
||||
```console
|
||||
sudo -u forgejo env \
|
||||
HOME=/var/lib/forgejo \
|
||||
USER=forgejo \
|
||||
FORGEJO_WORK_DIR=/var/lib/forgejo \
|
||||
FORGEJO_CUSTOM=/var/lib/forgejo/custom \
|
||||
forgejo --config /var/lib/forgejo/custom/conf/app.ini \
|
||||
admin user list --admin \
|
||||
| grep -F -- 'oibot'
|
||||
```
|
||||
|
||||
Sign in at `https://git.tobiasostner.de/` and confirm that the interface does
|
||||
not offer public registration.
|
||||
|
||||
## Disposable HTTPS and SSH smoke test
|
||||
|
||||
In the Forgejo interface:
|
||||
|
||||
1. Add the external client's public SSH key to the `oibot` account.
|
||||
2. Create an empty repository named `clone-smoke`.
|
||||
|
||||
From an external client, exercise HTTPS clone and push with a disposable commit:
|
||||
|
||||
```console
|
||||
workdir="$(mktemp -d)"
|
||||
git clone https://git.tobiasostner.de/oibot/clone-smoke.git \
|
||||
"$workdir/clone-smoke-https"
|
||||
cd "$workdir/clone-smoke-https"
|
||||
printf '%s\n' 'Forgejo trial' > README.md
|
||||
git add README.md
|
||||
git commit -m 'Test Forgejo trial'
|
||||
git push origin HEAD:main
|
||||
```
|
||||
|
||||
Then exercise host OpenSSH clone and fetch:
|
||||
|
||||
```console
|
||||
git clone forgejo@git.tobiasostner.de:oibot/clone-smoke.git \
|
||||
"$workdir/clone-smoke-ssh"
|
||||
git -C "$workdir/clone-smoke-ssh" fetch origin
|
||||
rm -rf -- "$workdir"
|
||||
```
|
||||
|
||||
All clone, push, and fetch commands must exit successfully. The repository and
|
||||
commit are disposable trial data.
|
||||
|
||||
## Trial rollback boundary
|
||||
|
||||
Tobserver hosts unrelated services, so do not use a broad live NixOS generation
|
||||
rollback as a routine Forgejo test. If the service is unhealthy, collect its
|
||||
status and journals first. Prepare and review a narrow change that removes only
|
||||
the `modules/forgejo.nix` import and its Forgejo service and nginx virtual host.
|
||||
Preserve `/var/lib/forgejo`; never delete trial state as an automatic rollback
|
||||
action.
|
||||
172
flake.lock
generated
172
flake.lock
generated
|
|
@ -8,36 +8,17 @@
|
|||
]
|
||||
},
|
||||
"locked": {
|
||||
"lastModified": 1785063525,
|
||||
"narHash": "sha256-sQ2VQzADqvZMo1I4mmQ08e8UZCsD/OQv80naVZRRWr8=",
|
||||
"ref": "refs/heads/main",
|
||||
"rev": "74af564b41aa1d96d675b3861af00276ebad6ed9",
|
||||
"revCount": 25,
|
||||
"type": "git",
|
||||
"url": "https://codeberg.org/oibot/BitcoinPulseAPI.git"
|
||||
"lastModified": 1785422356,
|
||||
"narHash": "sha256-UYlO43kzV3C7M0Jw/Vu621gpP7TDGR9frtEy2eYMT50=",
|
||||
"owner": "oibot",
|
||||
"repo": "BitcoinPulse",
|
||||
"rev": "7150f9813b7cda3add1c539533e9a1020cb941f6",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
"type": "git",
|
||||
"url": "https://codeberg.org/oibot/BitcoinPulseAPI.git"
|
||||
}
|
||||
},
|
||||
"checker": {
|
||||
"inputs": {
|
||||
"clj-nix": "clj-nix_2",
|
||||
"nixpkgs": "nixpkgs"
|
||||
},
|
||||
"locked": {
|
||||
"lastModified": 1776095659,
|
||||
"narHash": "sha256-3xXKVCwzjNKjctilYZWXcejxeE+9FPVLZaVuciFQmAI=",
|
||||
"ref": "refs/heads/main",
|
||||
"rev": "57fe47b20e85ab3b8eba87a1a24d5ae46f1beaa7",
|
||||
"revCount": 32,
|
||||
"type": "git",
|
||||
"url": "https://codeberg.org/oibot/checker.git"
|
||||
},
|
||||
"original": {
|
||||
"type": "git",
|
||||
"url": "https://codeberg.org/oibot/checker.git"
|
||||
"owner": "oibot",
|
||||
"repo": "BitcoinPulse",
|
||||
"type": "github"
|
||||
}
|
||||
},
|
||||
"clj-nix": {
|
||||
|
|
@ -63,29 +44,6 @@
|
|||
"type": "github"
|
||||
}
|
||||
},
|
||||
"clj-nix_2": {
|
||||
"inputs": {
|
||||
"devshell": "devshell_2",
|
||||
"nix-fetcher-data": "nix-fetcher-data_2",
|
||||
"nixpkgs": [
|
||||
"checker",
|
||||
"nixpkgs"
|
||||
]
|
||||
},
|
||||
"locked": {
|
||||
"lastModified": 1773151887,
|
||||
"narHash": "sha256-YUiehwe2iTlwYSrnJ1pcw7KDNX+U42xlTx/2k/mo0P8=",
|
||||
"owner": "jlesquembre",
|
||||
"repo": "clj-nix",
|
||||
"rev": "27dac4466c9d3939f6a4925bc09e0cb1d8f32d9c",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
"owner": "jlesquembre",
|
||||
"repo": "clj-nix",
|
||||
"type": "github"
|
||||
}
|
||||
},
|
||||
"devshell": {
|
||||
"inputs": {
|
||||
"nixpkgs": [
|
||||
|
|
@ -108,28 +66,6 @@
|
|||
"type": "github"
|
||||
}
|
||||
},
|
||||
"devshell_2": {
|
||||
"inputs": {
|
||||
"nixpkgs": [
|
||||
"checker",
|
||||
"clj-nix",
|
||||
"nixpkgs"
|
||||
]
|
||||
},
|
||||
"locked": {
|
||||
"lastModified": 1768818222,
|
||||
"narHash": "sha256-460jc0+CZfyaO8+w8JNtlClB2n4ui1RbHfPTLkpwhU8=",
|
||||
"owner": "numtide",
|
||||
"repo": "devshell",
|
||||
"rev": "255a2b1725a20d060f566e4755dbf571bbbb5f76",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
"owner": "numtide",
|
||||
"repo": "devshell",
|
||||
"type": "github"
|
||||
}
|
||||
},
|
||||
"flake-parts": {
|
||||
"inputs": {
|
||||
"nixpkgs-lib": "nixpkgs-lib"
|
||||
|
|
@ -149,24 +85,6 @@
|
|||
}
|
||||
},
|
||||
"flake-parts_2": {
|
||||
"inputs": {
|
||||
"nixpkgs-lib": "nixpkgs-lib_2"
|
||||
},
|
||||
"locked": {
|
||||
"lastModified": 1719745305,
|
||||
"narHash": "sha256-xwgjVUpqSviudEkpQnioeez1Uo2wzrsMaJKJClh+Bls=",
|
||||
"owner": "hercules-ci",
|
||||
"repo": "flake-parts",
|
||||
"rev": "c3c5ecc05edc7dafba779c6c1a61cd08ac6583e9",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
"owner": "hercules-ci",
|
||||
"repo": "flake-parts",
|
||||
"type": "github"
|
||||
}
|
||||
},
|
||||
"flake-parts_3": {
|
||||
"inputs": {
|
||||
"nixpkgs-lib": [
|
||||
"hermes-agent",
|
||||
|
|
@ -189,19 +107,19 @@
|
|||
},
|
||||
"hermes-agent": {
|
||||
"inputs": {
|
||||
"flake-parts": "flake-parts_3",
|
||||
"nixpkgs": "nixpkgs_2",
|
||||
"flake-parts": "flake-parts_2",
|
||||
"nixpkgs": "nixpkgs",
|
||||
"npm-lockfile-fix": "npm-lockfile-fix",
|
||||
"pyproject-build-systems": "pyproject-build-systems",
|
||||
"pyproject-nix": "pyproject-nix",
|
||||
"uv2nix": "uv2nix"
|
||||
},
|
||||
"locked": {
|
||||
"lastModified": 1785135041,
|
||||
"narHash": "sha256-BesiGRVPWppwnJdkSYhfVy0Skq1vNwd3nfVmHJQZhHk=",
|
||||
"lastModified": 1785392178,
|
||||
"narHash": "sha256-kftnbY/YA69Haqe7IfJGFJB2hbE9ANljXUR7i1a+m4o=",
|
||||
"owner": "NousResearch",
|
||||
"repo": "hermes-agent",
|
||||
"rev": "825069004a09023452a50c649f79e9a39bef40b5",
|
||||
"rev": "8e1debd5ed6cb0fe5737c59682fb0036d79c29f5",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
|
|
@ -233,41 +151,18 @@
|
|||
"type": "github"
|
||||
}
|
||||
},
|
||||
"nix-fetcher-data_2": {
|
||||
"inputs": {
|
||||
"flake-parts": "flake-parts_2",
|
||||
"nixpkgs": [
|
||||
"checker",
|
||||
"clj-nix",
|
||||
"nixpkgs"
|
||||
]
|
||||
},
|
||||
"locked": {
|
||||
"lastModified": 1755022803,
|
||||
"narHash": "sha256-/QtBdVfZlrRJW5enUoWlBE2wrLXJBMJ45X0rZh0jiaU=",
|
||||
"owner": "jlesquembre",
|
||||
"repo": "nix-fetcher-data",
|
||||
"rev": "9da3926b1459d6ff15268072d1c51351b82811b9",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
"owner": "jlesquembre",
|
||||
"repo": "nix-fetcher-data",
|
||||
"type": "github"
|
||||
}
|
||||
},
|
||||
"nixpkgs": {
|
||||
"locked": {
|
||||
"lastModified": 1775811116,
|
||||
"narHash": "sha256-t+HZK42pB6N+i5RGbuy7Xluez/VvWbembBdvzsc23Ss=",
|
||||
"lastModified": 1775036866,
|
||||
"narHash": "sha256-ZojAnPuCdy657PbTq5V0Y+AHKhZAIwSIT2cb8UgAz/U=",
|
||||
"owner": "NixOS",
|
||||
"repo": "nixpkgs",
|
||||
"rev": "54170c54449ea4d6725efd30d719c5e505f1c10e",
|
||||
"rev": "6201e203d09599479a3b3450ed24fa81537ebc4e",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
"owner": "NixOS",
|
||||
"ref": "nixos-25.11",
|
||||
"ref": "nixos-unstable",
|
||||
"repo": "nixpkgs",
|
||||
"type": "github"
|
||||
}
|
||||
|
|
@ -284,35 +179,7 @@
|
|||
"url": "https://github.com/NixOS/nixpkgs/archive/eb9ceca17df2ea50a250b6b27f7bf6ab0186f198.tar.gz"
|
||||
}
|
||||
},
|
||||
"nixpkgs-lib_2": {
|
||||
"locked": {
|
||||
"lastModified": 1717284937,
|
||||
"narHash": "sha256-lIbdfCsf8LMFloheeE6N31+BMIeixqyQWbSr2vk79EQ=",
|
||||
"type": "tarball",
|
||||
"url": "https://github.com/NixOS/nixpkgs/archive/eb9ceca17df2ea50a250b6b27f7bf6ab0186f198.tar.gz"
|
||||
},
|
||||
"original": {
|
||||
"type": "tarball",
|
||||
"url": "https://github.com/NixOS/nixpkgs/archive/eb9ceca17df2ea50a250b6b27f7bf6ab0186f198.tar.gz"
|
||||
}
|
||||
},
|
||||
"nixpkgs_2": {
|
||||
"locked": {
|
||||
"lastModified": 1775036866,
|
||||
"narHash": "sha256-ZojAnPuCdy657PbTq5V0Y+AHKhZAIwSIT2cb8UgAz/U=",
|
||||
"owner": "NixOS",
|
||||
"repo": "nixpkgs",
|
||||
"rev": "6201e203d09599479a3b3450ed24fa81537ebc4e",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
"owner": "NixOS",
|
||||
"ref": "nixos-unstable",
|
||||
"repo": "nixpkgs",
|
||||
"type": "github"
|
||||
}
|
||||
},
|
||||
"nixpkgs_3": {
|
||||
"locked": {
|
||||
"lastModified": 1782847189,
|
||||
"narHash": "sha256-twXPFqFsrrY5r28Zh7Homgcp2gUMBgQ6WDS98Q/3xFI=",
|
||||
|
|
@ -402,9 +269,8 @@
|
|||
"root": {
|
||||
"inputs": {
|
||||
"bitcoin-pulse": "bitcoin-pulse",
|
||||
"checker": "checker",
|
||||
"hermes-agent": "hermes-agent",
|
||||
"nixpkgs": "nixpkgs_3",
|
||||
"nixpkgs": "nixpkgs_2",
|
||||
"sops-nix": "sops-nix"
|
||||
}
|
||||
},
|
||||
|
|
|
|||
|
|
@ -3,15 +3,14 @@
|
|||
nixpkgs.url = "github:NixOS/nixpkgs/nixos-25.11";
|
||||
sops-nix.url = "github:Mic92/sops-nix";
|
||||
sops-nix.inputs.nixpkgs.follows = "nixpkgs";
|
||||
checker.url = "git+https://codeberg.org/oibot/checker.git";
|
||||
hermes-agent.url = "github:NousResearch/hermes-agent";
|
||||
bitcoin-pulse = {
|
||||
url = "git+https://codeberg.org/oibot/BitcoinPulseAPI.git";
|
||||
url = "github:oibot/BitcoinPulse";
|
||||
inputs.nixpkgs.follows = "nixpkgs";
|
||||
};
|
||||
};
|
||||
|
||||
outputs = { self, nixpkgs, sops-nix, checker, hermes-agent, bitcoin-pulse, ... }:
|
||||
outputs = { self, nixpkgs, sops-nix, hermes-agent, bitcoin-pulse, ... }:
|
||||
let
|
||||
system = "x86_64-linux";
|
||||
in
|
||||
|
|
@ -20,20 +19,20 @@
|
|||
inherit system;
|
||||
|
||||
specialArgs = {
|
||||
inherit bitcoin-pulse checker;
|
||||
inherit bitcoin-pulse;
|
||||
};
|
||||
|
||||
modules = [
|
||||
sops-nix.nixosModules.sops
|
||||
./hardware-configuration.nix
|
||||
./configuration.nix
|
||||
./modules/forgejo.nix
|
||||
./modules/bitcoin-pulse.nix
|
||||
./modules/bitcoin.nix
|
||||
./modules/electrs.nix
|
||||
./modules/mempool.nix
|
||||
./modules/nextcloud.nix
|
||||
./modules/totap.nix
|
||||
./modules/checker.nix
|
||||
hermes-agent.nixosModules.default
|
||||
./modules/hermes-agent.nix
|
||||
./modules/lnd.nix
|
||||
|
|
|
|||
|
|
@ -1,14 +1,22 @@
|
|||
{ bitcoin-pulse, config, lib, pkgs, ... }:
|
||||
|
||||
let
|
||||
backupDirectory = "/var/backup/bitcoin-pulse";
|
||||
backupDump = "${backupDirectory}/bitcoin-pulse.dump";
|
||||
backupRepository = "b2:tobcloud-backup:bitcoin-pulse";
|
||||
backupServiceName = "restic-backups-bitcoin-pulse";
|
||||
containerName = "bitcoin-pulse-postgres";
|
||||
databaseUrl = "jdbc:postgresql://127.0.0.1:5432/bitcoin_pulse?tcpKeepAlive=true";
|
||||
migrationServiceName = "bitcoin-pulse-migrate";
|
||||
serviceName = "podman-${containerName}";
|
||||
runtimeDirectory = "/run/${containerName}";
|
||||
|
||||
secretFiles = {
|
||||
admin = config.sops.secrets."bitcoin-pulse/postgres/admin-password".path;
|
||||
app = config.sops.secrets."bitcoin-pulse/postgres/app-password".path;
|
||||
backblaze = config.sops.secrets."backblaze/tobcloud-backup-env".path;
|
||||
migration = config.sops.secrets."bitcoin-pulse/postgres/migration-password".path;
|
||||
restic = config.sops.secrets."restic/bitcoin-pulse-password".path;
|
||||
};
|
||||
|
||||
prepareSecrets = pkgs.writeShellApplication {
|
||||
|
|
@ -50,18 +58,119 @@ let
|
|||
${lib.escapeShellArg "${runtimeDirectory}/migration-password"}
|
||||
'';
|
||||
};
|
||||
|
||||
waitForPostgresql = pkgs.writeShellApplication {
|
||||
name = "bitcoin-pulse-wait-for-postgresql";
|
||||
runtimeInputs = [
|
||||
pkgs.coreutils
|
||||
pkgs.postgresql_18
|
||||
];
|
||||
text = ''
|
||||
deadline=$((SECONDS + 60))
|
||||
echo "Waiting up to 60 seconds for PostgreSQL on 127.0.0.1:5432"
|
||||
|
||||
while (( SECONDS < deadline )); do
|
||||
if pg_isready \
|
||||
--host=127.0.0.1 \
|
||||
--port=5432 \
|
||||
--username=bitcoin_pulse_migration \
|
||||
--dbname=bitcoin_pulse \
|
||||
--timeout=2 \
|
||||
--quiet; then
|
||||
echo "PostgreSQL is accepting connections"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
sleep 2
|
||||
done
|
||||
|
||||
echo "PostgreSQL did not become ready on 127.0.0.1:5432 within 60 seconds" >&2
|
||||
pg_isready \
|
||||
--host=127.0.0.1 \
|
||||
--port=5432 \
|
||||
--username=bitcoin_pulse_migration \
|
||||
--dbname=bitcoin_pulse \
|
||||
--timeout=2 || true
|
||||
exit 1
|
||||
'';
|
||||
};
|
||||
in
|
||||
{
|
||||
imports = [ bitcoin-pulse.nixosModules.default ];
|
||||
|
||||
services.bitcoin-pulse = {
|
||||
# BIT-28 will enable the production instance after PostgreSQL readiness
|
||||
# and migrations have been integrated with these credentials.
|
||||
enable = false;
|
||||
enable = true;
|
||||
|
||||
bindAddress = "127.0.0.1";
|
||||
port = 3000;
|
||||
mempool.baseUrl = "http://127.0.0.1:8999";
|
||||
|
||||
database = {
|
||||
url = databaseUrl;
|
||||
username = "bitcoin_pulse_app";
|
||||
passwordFile = secretFiles.app;
|
||||
};
|
||||
};
|
||||
|
||||
services.restic.backups.bitcoin-pulse = {
|
||||
initialize = true;
|
||||
repository = backupRepository;
|
||||
|
||||
passwordFile = secretFiles.restic;
|
||||
environmentFile = secretFiles.backblaze;
|
||||
|
||||
paths = [ backupDump ];
|
||||
|
||||
backupPrepareCommand = ''
|
||||
set -euo pipefail
|
||||
|
||||
install -d -m 0700 -o root -g root ${lib.escapeShellArg backupDirectory}
|
||||
|
||||
dump_path=${lib.escapeShellArg backupDump}
|
||||
temporary_dump="$dump_path.tmp"
|
||||
pgpass_file="$RUNTIME_DIRECTORY/postgresql.pgpass"
|
||||
|
||||
rm -f "$dump_path" "$temporary_dump" "$pgpass_file"
|
||||
|
||||
password="$(cat "$CREDENTIALS_DIRECTORY/postgres-password")"
|
||||
escaped_password="''${password//\\/\\\\}"
|
||||
escaped_password="''${escaped_password//:/\\:}"
|
||||
printf '127.0.0.1:5432:bitcoin_pulse:bitcoin_pulse_admin:%s\n' \
|
||||
"$escaped_password" > "$pgpass_file"
|
||||
chmod 0600 "$pgpass_file"
|
||||
|
||||
trap 'rm -f "$temporary_dump" "$pgpass_file"' EXIT
|
||||
|
||||
PGPASSFILE="$pgpass_file" ${pkgs.postgresql_18}/bin/pg_dump \
|
||||
--host=127.0.0.1 \
|
||||
--port=5432 \
|
||||
--username=bitcoin_pulse_admin \
|
||||
--dbname=bitcoin_pulse \
|
||||
--format=custom \
|
||||
--no-password \
|
||||
--file="$temporary_dump"
|
||||
|
||||
mv "$temporary_dump" "$dump_path"
|
||||
'';
|
||||
|
||||
backupCleanupCommand = ''
|
||||
rm -f \
|
||||
${lib.escapeShellArg backupDump} \
|
||||
${lib.escapeShellArg "${backupDump}.tmp"} \
|
||||
"$RUNTIME_DIRECTORY/postgresql.pgpass"
|
||||
'';
|
||||
|
||||
timerConfig = {
|
||||
OnCalendar = "*-*-* 03:00:00";
|
||||
Persistent = true;
|
||||
RandomizedDelaySec = "30m";
|
||||
};
|
||||
|
||||
pruneOpts = [
|
||||
"--keep-daily 7"
|
||||
"--keep-weekly 4"
|
||||
"--keep-monthly 12"
|
||||
];
|
||||
};
|
||||
|
||||
users.groups = {
|
||||
|
|
@ -70,20 +179,28 @@ in
|
|||
bitcoin-pulse-postgres.gid = 400;
|
||||
};
|
||||
|
||||
users.users.bitcoin-pulse-postgres = {
|
||||
description = "Rootless PostgreSQL container account for Bitcoin Pulse";
|
||||
isSystemUser = true;
|
||||
uid = 400;
|
||||
group = "bitcoin-pulse-postgres";
|
||||
extraGroups = [
|
||||
"bitcoin-pulse"
|
||||
"bitcoin-pulse-migration"
|
||||
];
|
||||
home = "/var/lib/bitcoin-pulse-postgres";
|
||||
homeMode = "0700";
|
||||
createHome = true;
|
||||
linger = true;
|
||||
autoSubUidGidRange = true;
|
||||
users.users = {
|
||||
bitcoin-pulse-migration = {
|
||||
description = "Bitcoin Pulse database migration account";
|
||||
isSystemUser = true;
|
||||
group = "bitcoin-pulse-migration";
|
||||
};
|
||||
|
||||
bitcoin-pulse-postgres = {
|
||||
description = "Rootless PostgreSQL container account for Bitcoin Pulse";
|
||||
isSystemUser = true;
|
||||
uid = 400;
|
||||
group = "bitcoin-pulse-postgres";
|
||||
extraGroups = [
|
||||
"bitcoin-pulse"
|
||||
"bitcoin-pulse-migration"
|
||||
];
|
||||
home = "/var/lib/bitcoin-pulse-postgres";
|
||||
homeMode = "0700";
|
||||
createHome = true;
|
||||
linger = true;
|
||||
autoSubUidGidRange = true;
|
||||
};
|
||||
};
|
||||
|
||||
sops.secrets = {
|
||||
|
|
@ -101,6 +218,12 @@ in
|
|||
group = "bitcoin-pulse-migration";
|
||||
mode = "0440";
|
||||
};
|
||||
|
||||
"restic/bitcoin-pulse-password" = {
|
||||
owner = "root";
|
||||
group = "root";
|
||||
mode = "0400";
|
||||
};
|
||||
};
|
||||
|
||||
virtualisation.oci-containers = {
|
||||
|
|
@ -108,7 +231,7 @@ in
|
|||
|
||||
containers.${containerName} = {
|
||||
# Official PostgreSQL 18.4 Bookworm multi-architecture image, pinned to
|
||||
# the immutable digest tested by BitcoinPulseAPI on 2026-07-16.
|
||||
# the immutable digest tested by BitcoinPulse on 2026-07-16.
|
||||
image = "docker.io/library/postgres@sha256:1961f96e6029a02c3812d7cb329a3b03a3ac2bb067058dec17b0f5596aca9296";
|
||||
pull = "missing";
|
||||
autoStart = true;
|
||||
|
|
@ -151,15 +274,118 @@ in
|
|||
};
|
||||
};
|
||||
|
||||
systemd.services.${serviceName}.serviceConfig = {
|
||||
ExecStartPre = lib.mkAfter [ "${prepareSecrets}/bin/bitcoin-pulse-postgresql-prepare-secrets" ];
|
||||
ExecStartPost = [ "${cleanupSecrets}/bin/bitcoin-pulse-postgresql-cleanup-secrets" ];
|
||||
ExecStopPost = lib.mkAfter [ "${cleanupSecrets}/bin/bitcoin-pulse-postgresql-cleanup-secrets" ];
|
||||
RuntimeDirectoryMode = "0700";
|
||||
RestartSec = "5s";
|
||||
TimeoutStopSec = lib.mkForce "90s";
|
||||
UMask = "0077";
|
||||
systemd.services = {
|
||||
${serviceName}.serviceConfig = {
|
||||
ExecStartPre = lib.mkAfter [ "${prepareSecrets}/bin/bitcoin-pulse-postgresql-prepare-secrets" ];
|
||||
ExecStartPost = [ "${cleanupSecrets}/bin/bitcoin-pulse-postgresql-cleanup-secrets" ];
|
||||
ExecStopPost = lib.mkAfter [ "${cleanupSecrets}/bin/bitcoin-pulse-postgresql-cleanup-secrets" ];
|
||||
RuntimeDirectoryMode = "0700";
|
||||
RestartSec = "5s";
|
||||
TimeoutStopSec = lib.mkForce "90s";
|
||||
UMask = "0077";
|
||||
};
|
||||
|
||||
${migrationServiceName} = {
|
||||
description = "Migrate the Bitcoin Pulse PostgreSQL database";
|
||||
requires = [ "${serviceName}.service" ];
|
||||
after = [ "${serviceName}.service" ];
|
||||
before = [ "bitcoin-pulse.service" ];
|
||||
|
||||
environment = {
|
||||
MIGRATUS_DATABASE_URL = databaseUrl;
|
||||
MIGRATUS_USERNAME = "bitcoin_pulse_migration";
|
||||
MIGRATUS_PASSWORD_FILE = secretFiles.migration;
|
||||
};
|
||||
|
||||
serviceConfig = {
|
||||
Type = "oneshot";
|
||||
ExecStartPre = [ "${waitForPostgresql}/bin/bitcoin-pulse-wait-for-postgresql" ];
|
||||
ExecStart = "${config.services.bitcoin-pulse.package}/bin/bitcoin-pulse-migrate migrate";
|
||||
User = "bitcoin-pulse-migration";
|
||||
Group = "bitcoin-pulse-migration";
|
||||
Restart = "no";
|
||||
TimeoutStartSec = "5min";
|
||||
|
||||
AmbientCapabilities = "";
|
||||
CapabilityBoundingSet = "";
|
||||
LockPersonality = true;
|
||||
NoNewPrivileges = true;
|
||||
PrivateDevices = true;
|
||||
PrivateTmp = true;
|
||||
ProcSubset = "pid";
|
||||
ProtectClock = true;
|
||||
ProtectControlGroups = true;
|
||||
ProtectHome = true;
|
||||
ProtectHostname = true;
|
||||
ProtectKernelLogs = true;
|
||||
ProtectKernelModules = true;
|
||||
ProtectKernelTunables = true;
|
||||
ProtectProc = "invisible";
|
||||
ProtectSystem = "strict";
|
||||
RemoveIPC = true;
|
||||
RestrictAddressFamilies = [ "AF_INET" "AF_INET6" "AF_UNIX" ];
|
||||
RestrictNamespaces = true;
|
||||
RestrictRealtime = true;
|
||||
RestrictSUIDSGID = true;
|
||||
SystemCallArchitectures = "native";
|
||||
UMask = "0077";
|
||||
};
|
||||
};
|
||||
|
||||
bitcoin-pulse = {
|
||||
requires = [ "${migrationServiceName}.service" ];
|
||||
after = [ "${migrationServiceName}.service" ];
|
||||
};
|
||||
|
||||
${backupServiceName} = {
|
||||
requires = [ "${serviceName}.service" ];
|
||||
after = [ "${serviceName}.service" ];
|
||||
|
||||
serviceConfig = {
|
||||
LoadCredential = [
|
||||
"postgres-password:${secretFiles.admin}"
|
||||
];
|
||||
TimeoutStartSec = "6h";
|
||||
UMask = "0077";
|
||||
};
|
||||
};
|
||||
};
|
||||
|
||||
services.nginx = {
|
||||
appendHttpConfig = ''
|
||||
limit_req_zone $binary_remote_addr zone=bitcoin_pulse_api:10m rate=300r/m;
|
||||
'';
|
||||
|
||||
virtualHosts."bitcoin-pulse.tobiasostner.de" = {
|
||||
enableACME = true;
|
||||
forceSSL = true;
|
||||
|
||||
locations."/" = {
|
||||
proxyPass = "http://127.0.0.1:3000";
|
||||
extraConfig = ''
|
||||
limit_req zone=bitcoin_pulse_api burst=50 nodelay;
|
||||
limit_req_status 429;
|
||||
client_max_body_size 16k;
|
||||
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_set_header Connection "";
|
||||
|
||||
proxy_connect_timeout 2s;
|
||||
proxy_read_timeout 10s;
|
||||
proxy_send_timeout 10s;
|
||||
proxy_next_upstream off;
|
||||
'';
|
||||
};
|
||||
};
|
||||
};
|
||||
|
||||
systemd.tmpfiles.rules = [
|
||||
"d ${backupDirectory} 0700 root root - -"
|
||||
];
|
||||
|
||||
environment.systemPackages = [ pkgs.postgresql_18 ];
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,53 +0,0 @@
|
|||
{ config, pkgs, checker, ... }:
|
||||
|
||||
let
|
||||
checkerPkg = checker.packages.${pkgs.stdenv.hostPlatform.system}.default;
|
||||
in
|
||||
{
|
||||
users.users.checker = {
|
||||
isSystemUser = true;
|
||||
group = "checker";
|
||||
home = "/var/lib/checker";
|
||||
createHome = true;
|
||||
};
|
||||
|
||||
users.groups.checker = {};
|
||||
|
||||
sops.secrets."checker/env" = {
|
||||
owner = "checker";
|
||||
group = "checker";
|
||||
mode = "0400";
|
||||
};
|
||||
|
||||
systemd.services.checker = {
|
||||
description = "Checker a personal assistant";
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
after = [ "network.target" ];
|
||||
|
||||
serviceConfig = {
|
||||
User = "checker";
|
||||
Group = "checker";
|
||||
WorkingDirectory = "/var/lib/checker";
|
||||
ExecStart = "${checkerPkg}/bin/checker";
|
||||
EnvironmentFile = config.sops.secrets."checker/env".path;
|
||||
|
||||
Restart = "always";
|
||||
RestartSec = 5;
|
||||
|
||||
NowNewPrivileges = true;
|
||||
PrivateTmp = true;
|
||||
ProtectSystem = "strict";
|
||||
ProtectHome = true;
|
||||
ReadWritePaths = [ "/var/lib/checker" ];
|
||||
};
|
||||
};
|
||||
|
||||
services.nginx.virtualHosts."checker.tobiasostner.de" = {
|
||||
enableACME = true;
|
||||
forceSSL = true;
|
||||
|
||||
locations."/" = {
|
||||
proxyPass = "http://127.0.0.1:3001";
|
||||
};
|
||||
};
|
||||
}
|
||||
66
modules/forgejo.nix
Normal file
66
modules/forgejo.nix
Normal file
|
|
@ -0,0 +1,66 @@
|
|||
{ pkgs, ... }:
|
||||
|
||||
{
|
||||
services.forgejo = {
|
||||
enable = true;
|
||||
package = pkgs.forgejo-lts;
|
||||
|
||||
stateDir = "/var/lib/forgejo";
|
||||
repositoryRoot = "/var/lib/forgejo/repositories";
|
||||
|
||||
database = {
|
||||
type = "sqlite3";
|
||||
path = "/var/lib/forgejo/data/forgejo.db";
|
||||
};
|
||||
|
||||
lfs.enable = false;
|
||||
|
||||
settings = {
|
||||
server = {
|
||||
PROTOCOL = "http";
|
||||
HTTP_ADDR = "127.0.0.1";
|
||||
HTTP_PORT = 3001;
|
||||
|
||||
DOMAIN = "git.tobiasostner.de";
|
||||
ROOT_URL = "https://git.tobiasostner.de/";
|
||||
|
||||
SSH_DOMAIN = "git.tobiasostner.de";
|
||||
SSH_PORT = 22;
|
||||
SSH_USER = "forgejo";
|
||||
START_SSH_SERVER = false;
|
||||
};
|
||||
|
||||
service.DISABLE_REGISTRATION = true;
|
||||
session.COOKIE_SECURE = true;
|
||||
|
||||
actions.ENABLED = false;
|
||||
packages.ENABLED = false;
|
||||
mailer.ENABLED = false;
|
||||
};
|
||||
};
|
||||
|
||||
services.nginx.virtualHosts."git.tobiasostner.de" = {
|
||||
enableACME = true;
|
||||
forceSSL = true;
|
||||
extraConfig = ''
|
||||
merge_slashes off;
|
||||
'';
|
||||
|
||||
locations."/" = {
|
||||
proxyPass = "http://127.0.0.1:3001";
|
||||
extraConfig = ''
|
||||
client_max_body_size 512M;
|
||||
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_set_header Connection $http_connection;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
'';
|
||||
};
|
||||
};
|
||||
|
||||
environment.systemPackages = [ pkgs.forgejo-lts ];
|
||||
}
|
||||
|
|
@ -22,12 +22,6 @@
|
|||
mode = "0400";
|
||||
};
|
||||
|
||||
sops.secrets."backblaze/nextcloud-env" = {
|
||||
owner = "root";
|
||||
group = "root";
|
||||
mode = "0400";
|
||||
};
|
||||
|
||||
services = {
|
||||
|
||||
nextcloud = {
|
||||
|
|
@ -41,7 +35,7 @@
|
|||
config = {
|
||||
dbtype = "mysql";
|
||||
adminuser = "tobi";
|
||||
adminpassFile = config.sops.secrets."nextcloud/admin-pass".path;
|
||||
adminpassFile = config.sops.secrets."nextcloud/admin-pass".path;
|
||||
};
|
||||
|
||||
autoUpdateApps.enable = true;
|
||||
|
|
@ -54,9 +48,9 @@
|
|||
maintenance_window_start = 2;
|
||||
|
||||
overwritewebroot = "";
|
||||
overwrite.cli.url = "https://cloud.tobiasostner.de";
|
||||
overwritehost = "cloud.tobiasostner.de";
|
||||
overwriteprotocol = "https";
|
||||
overwrite.cli.url = "https://cloud.tobiasostner.de";
|
||||
overwritehost = "cloud.tobiasostner.de";
|
||||
overwriteprotocol = "https";
|
||||
|
||||
## Email settings
|
||||
mail_smtpmode = "smtp";
|
||||
|
|
@ -99,7 +93,7 @@
|
|||
repository = "b2:tobcloud-backup:nextcloud";
|
||||
|
||||
passwordFile = config.sops.secrets."restic/nextcloud-password".path;
|
||||
environmentFile = config.sops.secrets."backblaze/nextcloud-env".path;
|
||||
environmentFile = config.sops.secrets."backblaze/tobcloud-backup-env".path;
|
||||
|
||||
paths = [
|
||||
"/nextcloud-data"
|
||||
|
|
@ -128,7 +122,7 @@
|
|||
enable = true;
|
||||
virtualHosts."cloud.tobiasostner.de" = {
|
||||
forceSSL = true;
|
||||
enableACME = true;
|
||||
enableACME = true;
|
||||
};
|
||||
};
|
||||
|
||||
|
|
|
|||
|
|
@ -5,10 +5,9 @@ nextcloud:
|
|||
admin-pass: ENC[AES256_GCM,data:3xufFHvcm5g5+jn+B4qxRljuWDqDXo1mLu4IlSuax7Cz+3ZoOuNXoCXoiaY=,iv:WaWeu/a45Vuz+HvO7m0dUlCGCexk4IX/ijURmb/7jPU=,tag:ytRjS89AmHiGOhKymViTrg==,type:str]
|
||||
restic:
|
||||
nextcloud-password: ENC[AES256_GCM,data:fbH++0KWr9V4QR90t4kmBdhwyF+dbtBdf9RFarv2E9MyrH7QMO4yxdppaeE=,iv:VnS2936xTO2sTl51lqXl8/3MBeGldARc4/ZFZYL/hG4=,tag:QGp6OFqEmt6iP11lG6GwmA==,type:str]
|
||||
bitcoin-pulse-password: ENC[AES256_GCM,data:OAm3stWlJIcOEa/Gp5zkNq+z/gejyw5okWR0UAEKZXa6UXT/XAXGIYcnzOtbmBZK0G3Lt9OpxOyEACjL8xp+Xw==,iv:JddNNeJN5aRzvizHl/iSi3vjqVHrzrax1IPE7Fmh8bU=,tag:ckYd4yqaYjHROB3zlubleQ==,type:str]
|
||||
backblaze:
|
||||
nextcloud-env: ENC[AES256_GCM,data:fZsW4+R0nDTqhx6pKQCOKULrheseS1j6JOBvHVCuOPaVqkkZv8/TsgtDJdWnKXTNS9/RBTqhLFPIaUSUrfH/bh6rIc4iDe0BehPzHCxh/0CBwirmu2iS,iv:6ggfA8R+zpj+kxcfaWlG4OH1Iteex6zGIin8+TUemGw=,tag:VQW3hbrbgNBQXzmeSrMaAw==,type:str]
|
||||
checker:
|
||||
env: ENC[AES256_GCM,data:TkeIRm8CXS9AiFpOrMD04dX5L/kL1XAYwOavDwtAuaCsg8bhAsmI6L9rdHsVuMnL46EaUwG4mAgn6EK6kfiSzkzgZ63MKrhcwhD3B4ybMMjj1opIWvifKBY2+wJ9bLI1ygZtQToOgti5oWLtcCt7zlDx8RqXKbAYNq9j0SBDhZBiZK/m+YJPPHclxr8cn9TlKfEhW78sZ2fyH8V65vT1Rz8gVDARdaz3G12CO5jLVY0q5ucqXrS8YRsuYczMSI3aUcBwRyf2UESsjDBXqfg7OKsgabGqvEwIK+rlbvo1ylIWFJZGC80FEcn/dAIsxC8ew8vhHwDch+dlbgrdyUxktb0qNBJU+7HdyP9QZ+mfU2Y/FIvRbicZXFIVdTnwls9tt2p+uxJQkLCp7OTyZgDHTECDQBRmg5hAdrvyyJUZAUvbcFVYkK984kSNyELbGsZBz+SGnxbQLuQDYi98T6HbIk448pDaWy43QGiEZERcFfLAVHnhK8wYsw6ta4BJSf95t+Ka9zMDhR/j0v3mkTrvXpprnTTp727jUY8AM+dUfavETfi9VtrD9P73UZWkL/Wy4PezBGGhm/8l6PU8C+6cfKfKswLGkKW8tDzxVKgvAHiWcK3adyW1/wLdTOHP5XMgDxWEGc3EZRJ4hrw1cQRrVqkxHI9q8a2pmQZxi7l1u61V/EXBKQSSHGDX8pg1HFwGSkj4a39llz5auFk3QsWIntDFYIYpQKDXns2pH+A+PbTJEv+UfMvDQnMKMG99njn2r8KKXf88VWs64tCq/uylaslx3f/1CnVwTlr3REI=,iv:5nbRJpjNReztdAaTlHHjh7MH0PAN9eSSEEhE1Z8ynA4=,tag:9gYKrH/0g0bca/Q67LchFA==,type:str]
|
||||
tobcloud-backup-env: ENC[AES256_GCM,data:Lo2Xhjz/Wc1wPJljp7/2Vp2pSNAqGW/oOKvxc7s7q4LwJoGbOv1B2B5YqeJAgLR2LzB7cMwxpyK02uoUf7muFofYJmUC0A1CWuWe0pAsSkez7E13aOGh,iv:LEfrirUCbq4BQP3u9keo3nb6uI+geaTjjhSfhCSnRM4=,tag:A2xd+I+Mam8eXdu1r3sRpA==,type:str]
|
||||
hermes-env: ENC[AES256_GCM,data:WcN3pP4BIrMoKfehixAHZFTI2rk0js2mwi0m4+NsfgqamDH8IqHpbMJGhIg4zUmJBUlyz+4ZhjtNmvlxtj/i7UxIkPNq4qg95eK59gEM3npW9euRoX+BziKvsM6fFMRbWt60GaKigiI05z83M9I9/A8qbZC/PEWPfj6a5wlqHSVSe2qJjg5tGpthVhApaJ9ZmuTzhyHjs9zRvlkC9SFp9WsdyWWxNfChIKj8MySeAGs1YlDAGFVzsjX3+JlPVlCwPw==,iv:euK36Gk50/NQfjkspCP7tv3/a/gknzrDM21GUqVhEBY=,tag:jPF67Qa6e1VrY7byXKQw0w==,type:str]
|
||||
hermes:
|
||||
auth.json: ENC[AES256_GCM,data:sIp98d9dYqSe5vjTIaGQTgN7zl2XGRtT8CLMmqpf79DKQ2KDbbSdYyzhbbajccwH+S4T7vFYqs41hoNdf5a5zsrNBq68nQYzmT2Ja+hUCvxN0KsIF8xk/yjC56btlHBBH7ApzsDq8IQg31j1+hwXZSmLRNB4KB9fC1gWTyPedJDfVpTSmv0y37m6MauqgkJDZc1JWcYnswqW7qSqcNAAm0CwBycwEOvcxKRF9Xqg+KP6fYc+2MUagbzqWuv7JO14ZaoSrub8ob0bCHg6z21fEVkYCD4EyqTUmyOQlbmyfA6h2kjamD81sFAC/87crQ22bnTy6pUydAs85X2Ea7GGx9lirnzLzneJ5LDcFuPzxi2Aryp1/BY4bOPa8cD2kWAOH/zmqicwmATtMJPn8XVyfDLrOG+0Z6ADCFrUaKKfqJ5gzNoeF7gy3L4qYvHnvHLbpnWTWQETmMP8Y3eHTWfzqaH3m+u3yuH32Uh4NqEL7WmxROPyQhxmOjxWm4zSdvby4qsGA0ILZ39zKtzo7XTMRvKZScPql2zCmubvrFmV1pofEbqP3BN5b4kIYVa1JabpR3/saxO3+IqyoD8HsRaCsmvCSu3i4sMmxkfqLCsa8RmtpMMRgNQCie9VSDXqciAyHg6zMW3c+tZPX20gIEdgsAgis5Y2Rdbxnooh5HiU1r05p0pci1+X9sTsWs+0tyjnFwV4bYyPq/DwoPBJmiw4IyjfSgZOaiCJddWg4QnO5hVz/eWI5/g53/mlgF7AN6DRuobiepShvyZuDy7m32uBuYLOjgtvIfi+seAp+kzaPgIctjxxwRdGRs24ajuAHW69OIbwyk/zdANrK0K/5NDPHQ9vw6xUed+TCZnBZvKmyCtSNxgsnEk2cjiMxENawXTNpWbdfXAelRJP8f+hBBDJOo7FFkVLWS2OzE6m3ZHMyJf+dnQoZXLynwLTGq5aZKeP2YOVtQtjUlRnsNjKjzw8A/gAiwbu6xtIAummC9OYC1L0aX3Kh/cSg3eZw8z8Aaw7jposWXDhJaOvYYWp6RzMOtwpKU3Om27xGrsTQNBfweyX8/gucOEo70zE+XQXhH0VXwbdrGZVF1qU1bd5bx9P/GaHidkESkQrpM28L3ahJSJSCfwV+9Qd7toCotn14QPs1g4Je0GF4dd+KhyU0EKdlGW7s4+ozFfJS8PfmPVQOuXi0ul0cPPuID8CtqFjtaanWSuGRL/16CRPK71cf4/kY+JRIw/hTuQW/9lYXWW93pq2v/WLxG5505KWu+Lrmdx8MpuLKfyXPP39GkvMT1jT/zWfh9ZsmLs0emIEEoK1IZfSnFsiqcxmg6zpUBffATyBDellCFYvnDFkfpXyLYWi5uhr4hgcn5QyrXk0M98mggdEFrJtzDdm68/cNvs0pta7A6WHV5OtkjneIzsj7p4EJcEkeST6iuv/43vscL6lGI/nbCby7U+yXZdngf1nGkkJ5tpkIS/dWdnA00HIVLk9c8iPhp1HneeUB9GIKgaHRglGu7UVocqiAeRnZQoKp9giRNcKT6yPbLqVAYqjH+00C34F3DGXLaISS+wxHDGqGfxCeAj8PCC54e9XuYEbkes3BJSDsCbfeP0MAzgZfyU5RZKPyykKzseR9/5We3MZfyoXjO2FzgpKPYfUq2gHclMYvwgN6inXZ9NtFi+NZCLxs14k4fneaUI6Wgysq8GFRQ3inLjE3gWCCtt1RbyCmfFZ/8KLo8O16j0rIHvz83LLtN+N2NjibHE2f8j2WkrLd79htPGu9i61ETD19hGhx63f0EtZFgE/KUPz6EGGOzEyzwpm4VXIBNt0S53FxToVl0dNF1f7DD/RekNhBN5jkrRnkGf5wpgIceGlr7pjQUSrhArWhxAPAlsFlqE5elPdon2KfJFltSboeIDFtt7z6YEkhXL6eW3C6UM/p5WLL3Fo3s7mtyxaAq7+Yq2XJnrAvUhRuTgSEX4SgXLpIj4/xUxk5iWZcxwvjC+O7GaQ2V2JgMibIg8vhcCD/InYB+iydMegRaSCUPcbBsRUsUlwhnLlEmP9sNkzjV3qMHoemBDcKOpOudbnOUOTd9W2SYBduXSXAohWTgyCmkPTd4txMJ/bA4aXddTxkzRhkXLW8Q1lve2yhn4gS1+1xtRUjk9PgpvVKNrNXiI/nOH3sJX76Dbcrkh9f06PkB8Kt04pFVACR2JVmubcE04HQpzBmMy369iqDYQk+t9kxaJ32kTiOApV8YdbnW5xIdA9nrhZccgzP6PjY5G2uljLVlJWOB5qhk5OHiXGLy6QemW4RmM0Pe4TuF4onK61FugUan7D7X7g79Slcg8VNgOie1aaMgMJwu3RxjMKBmhbY4u5y8hx9PNPs+b8gmgn3IHfqf1GmfzOAu6VtUUxyZa+bZ+d+7C7PeotHgKz2OL/MPOykxknNmTblqzuLiwtVg1UN1SKmwQNH1Bydxxc7qCb7jTH/NAHB9D9ABrZ5WFJqZd/Ls87XJWe6VlF/8tYTuBanXFrWxd8ogh4kDEhSr/FIB1Y8Ub6qHMQJoPttmmHSIZEDSZ4QzShDocMtAPBjkMOx/UB+qbzoJq9yihszJN13+fgcERngSTXTozX+Z9+rxOVPP0PPCfiylH2d0kUS8AUejxzpVzgkycZ8bW4mkEWf6WiF0YDtEfmQBwHoGs0H6ddtamHOXZDyoi4ddxoif5CXVVbU7xbqX0T+mP/DRtQULgnwR3I/QC3q5F7B8d/zvpxHf5Mk+WCID+9zIZrRFOk3XO63OnKbndXHdCCjAqIDTPpp9yb8PGZkuTKU+jC98pgKj4kA/+FWMdCdIAKoLfepXwicHN9SoXNMDo6XZimmEzZPavQAizQF02UXDQTnsPclo7ulLrnn+J81LABcjawgkS5CGE5xioz3Z9akFY6rQ+LDlDN/88zuBIunOwconcTTstM/hNe6tC+6eleTBc2frpbKh43cB25sK6qIxTYgEP9hXTpq5PWtTMp3Jox6pgAMIP1d/Rg3Pf7MzdlIEp1d/XKPawIuvLuy6sBxJYi3k/rpDbfDKydSiKQN8Jja6uZNNrzGV6KMd5q6aXDVOyEOVlahILIYVFW8a9YvYggzL5TaAg9bOpxflqcqGg/roSHvy/e6j8Hib+R6+LLZMtqNTtT0YaQBt78i0SUo7lIuzNpAUPpHBbPJleBIUs5x6b0GIvgU7bp56UA9ZunX4ySxAO8wwhhKSP4iNpjrHCdaoJk4pxl/4ook8u9ve6Pwt9LDJeet1GMn1nHAQRXtrhx2qFrn+yrT1R48QWXi4O/eWvAKYfFp9NvMLPFcdM3qmQ6LkwbqzcZZnmck0jPmhYqaPaRxWepkdQorBO5t32lNaguYc0X5PUtqfm6e1+u1LE438tIP/9xEpDjqnBDIYO1Yw86V/SZs5THKOx88t5Mx3u9oiFFYk9+ULxgwRbCbSekdnlAdzyjueujP2dh9iGGTTNcm6ZzeVqIJCErGky2+Ffp4nl+6fbHJ9u0zjKZNtCP/AHskS7PTnI4adXDoQcJUh4pdtvkz6DFQhk0z+Af2qLu2PtWR7aVwE221PSVSdITOoj2PS4lEMuN91gkUGPf0oMl46kF9fRLMyuWqlcqJaom+Un4MrHnETf0udDicSB5hXwAHhNHPxOf4VfUUkkg3psPOZA4Jz/MIPM+at5hNAvf4qvz0TT4wl5y7MtqdDNnr1Qyo9trx+OfwYUwNZpcfEDeNtrt67EYM9L5hYY7+AUA6RKG2SnBArH6UCUl4zkgYPxecQ3aInXyvJAMAK6bfgo6IZoLxz98skByESBOkFwamqriKARt8CTC+apf9E00KnVhIHBaQx1cq5ANXU6ItydyIsT28LIcG+Wlyu1B6xONwhZTRJE+kowdRs6Znl26GjAjqh/lNAcbvGRmpmec7UfEXJyGoDhtbUQ1/QZKcXpYLwr3ABoldNKHwO1hlePWdZQ/9/p4ATkcGO3JqHwfrj2V5bSCU/79z66TS8XtRVmkic81WUjGGXiU4Z3cY1V8OFpYovwQL/opdtVjahSSuMVBAl7JXuJcmB8cqlxPTuAXT26NfbqzvDlc7ehf9nUnZOZvK30/QEh72lpeFCgPVS1Rgvt/K4oIexGPBnBkU5Wqgw==,iv:2laQqhM+pxSbClSOJBX7P9K87Iaxb23dhDqNIgy6oMM=,tag:iO4IWEZ2PJliahaVZtELzA==,type:str]
|
||||
|
|
@ -39,7 +38,7 @@ sops:
|
|||
cVE5VG56cFAzT1dsRFN4cCtnVEJJdU0KSBWm4blGhnVlOJ3k7lh8aJVOEH8NfFOh
|
||||
EwU/4Up6zWYBoe2+GKiQf8imnVLSSxMR7dlNd06ttuFD4B5baqFRcA==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-07-27T08:06:30Z"
|
||||
mac: ENC[AES256_GCM,data:Ix0g+VpLtA6vtiunLOa209iHic/+XWJJyyeEcn5UiLNLjAgdlEN2etpIlE+FEdJ0Zq6DwgG909qgEyTJ8buCMtnJ0D0W80kjFYXYxJfVLnQ7EQJwyuc0X7kpl8fKtIp3Momy8+u9DoFxMNyDgeaC2fP1V9ypRHT8MN+woQgTWxc=,iv:sc0SYyJCHBswcHGOMJrVHKqG9RtkOchZ1F9B4jLdCbE=,tag:Q1pPREj0qQ7ibZ5XWXaHLA==,type:str]
|
||||
lastmodified: "2026-07-30T05:44:20Z"
|
||||
mac: ENC[AES256_GCM,data:zpu938KpSjicnCJicrxSwV4sbdxRW02HnD7m1GXVSRH+j2Kuv3eKfB7JF4lBx339mnnOIasBZbcIgv5k/F7wX8EXGc7008AmM7Q5LwbBxPQSbIf3amGePoxL/pNw2i+h2MhWCNRadjdd1TJpGPFrFfTqI3FO0MLOacxAdxwvMF0=,iv:l5PLV2fOvukJFwPz5XitnGPICsIYC5JB9MxL1Ak+2q8=,tag:7cAhK9+R1xNIyEH9/6pCdA==,type:str]
|
||||
unencrypted_suffix: _unencrypted
|
||||
version: 3.12.1
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue