From b49138d95a4ac28989cd17f59f346c8f2e494115 Mon Sep 17 00:00:00 2001 From: Tobias Ostner Date: Sat, 1 Aug 2026 13:56:44 +0700 Subject: [PATCH] Deploy Forgejo trial instance --- .plans/deploy-forgejo-with-backups.md | 297 ++++++++++++++++++++++++++ docs/forgejo-operations.md | 216 +++++++++++++++++++ flake.nix | 1 + modules/forgejo.nix | 66 ++++++ 4 files changed, 580 insertions(+) create mode 100644 .plans/deploy-forgejo-with-backups.md create mode 100644 docs/forgejo-operations.md create mode 100644 modules/forgejo.nix diff --git a/.plans/deploy-forgejo-with-backups.md b/.plans/deploy-forgejo-with-backups.md new file mode 100644 index 0000000..c7f5c49 --- /dev/null +++ b/.plans/deploy-forgejo-with-backups.md @@ -0,0 +1,297 @@ +# Implementation Plan + +> **Status:** Phase 1 is ready for implementation; Phase 2 is blocked on the post-trial **KEEP** decision. + +## Outcome and boundaries + +- **Problem and target:** First deploy a minimal, usable Forgejo instance at `https://git.tobiasostner.de` so it can be evaluated without committing to the backup and recovery design. Add that surrounding infrastructure only after an explicit decision to keep Forgejo. +- **Phase 1 — trial deployment:** Native NixOS Forgejo LTS service; SQLite; `/var/lib/forgejo` state; loopback HTTP; nginx/ACME ingress; host OpenSSH on port 22; closed registration; first-admin bootstrap; and disposable HTTPS/SSH repository checks. Keep all declarative core deployment configuration—Forgejo service, database settings, state paths, nginx virtual host, and package exposure—together in `modules/forgejo.nix`. +- **Phase 1 non-goals:** No new SOPS secret, B2/Restic repository, backup service or timer, staging/runtime backup directories, archive or manifest logic, configuration-revision plumbing, backup VM check, retention policy, production backup, backup/recovery documentation, or restore/protection claim. +- **Trial safety boundary:** Data created during Phase 1 has no new backup protection. Use only disposable repositories and retain any authoritative copy elsewhere until Phase 2's first production snapshot has passed its content checks. +- **Decision gate:** After the trial, **KEEP** preserves `/var/lib/forgejo` and authorizes Phase 2. **ABANDON** stops this plan, skips every Phase 2 task, and triggers a separate reviewed removal plan whose default is to preserve trial state. No trial duration is prescribed. +- **Phase 2 — retained deployment:** Dedicated encrypted Restic password; daily complete-state backup to the existing Backblaze B2 bucket; 7 daily, 4 weekly, and 12 monthly snapshots; deterministic backup failure-path tests; deployment, backup, recovery, and rollback documentation; and inspection of the first production snapshot. +- **Out of scope in both phases:** Migration, LFS, Actions/runners, package registry, SMTP, SSO, containers, PostgreSQL, HA, external object storage, additional firewall ports, backup alerting, SSH-service interruption during backup, and initial or recurring restore tests. +- **Phase 2 accepted limitations:** sshd remains available during backup. A Git-over-SSH push beginning after the active-process check can overlap archive creation and make that snapshot inconsistent. Snapshot and archive inspection will not prove recovery because restore testing remains excluded. + +## Key files, evidence, and decisions + +| File or source | Why it matters | Decision or plan impact | +|---|---|---| +| `flake.nix:1-38`, `flake.lock` node `nixpkgs_2` | The host uses `nixos-25.11`; host modules are listed in one place; the evaluated `forgejo-lts` package is currently 11.0.15. | Import only the Forgejo service module in Phase 1. Add backup modules, the VM check, and `self` revision wiring in Phase 2. Re-evaluate the lock-specific package version before each activation. | +| Pinned nixpkgs `nixos/modules/services/misc/forgejo.nix` | The native module supports SQLite, external OpenSSH, declarative settings, generated persistent secrets, and state rooted at `/var/lib/forgejo`. | Use the native pinned module rather than a container or new flake input. | +| `configuration.nix:35-68` | Ports 22, 80, and 443 are already open; OpenSSH, nginx, and ACME are enabled. | Reuse those services and do not expose the Forgejo backend port. | +| `modules/bitcoin-pulse.nix:101-106` | Bitcoin Pulse occupies `127.0.0.1:3000`. | Bind Forgejo only to `127.0.0.1:3001`. | +| `configuration.nix:118-127`, `secrets/.sops.yaml` | SOPS is established and `backblaze/tobcloud-backup-env` is a shared root-only secret. | Do not touch secrets in Phase 1. In Phase 2, reuse the B2 credential and add only a dedicated root-only `restic/forgejo-password`. | +| `modules/bitcoin-pulse.nix:115-174,340-388` | This is the strongest local Restic/private-staging/timer/retention pattern and an existing nginx proxy pattern. | Reuse only the nginx conventions in Phase 1. Add Forgejo-specific Restic orchestration in a separable backup module in Phase 2. | +| `docs/bitcoin-pulse-operations.md` | Defines clean-checkout, build-before-switch, runtime verification, and multi-service rollback conventions. | Apply the same deployment discipline in both phases; avoid casual full-generation rollback. | +| [Forgejo backup guidance](https://forgejo.org/docs/latest/admin/upgrade/#backup) | Complete state needs synchronized point-in-time capture; shutdown provides the consistency boundary used by the existing design. | Phase 2 stops Forgejo and requires no remaining Forgejo-owned process before archiving all state. | +| [Forgejo reverse-proxy guidance](https://forgejo.org/docs/latest/admin/setup/reverse-proxy/) | Documents dedicated-origin, `ROOT_URL`, encoded-slash, proxy-header, and upload-size behavior. | Phase 1 uses a dedicated hostname, exact proxy behavior, and a 512 MiB body limit. | +| Verified baseline on 2026-08-01 | `git.tobiasostner.de` resolves to `162.55.103.218` with no AAAA record, and `nix flake check --no-build` passes. | Historical DNS, Codeberg-fetch, and Hermes evaluation failures are not current blockers. Recheck DNS and perform complete builds before activation. | +| User decisions captured by the existing plan | New personal instance; public HTTPS; closed registration; SQLite; host SSH; initial admin `oibot`; daily B2 backup with 7/4/12 retention; no restore tests, alerting, or SSH interruption. | Preserve these boundaries. Supply the admin email privately at deployment time; Phase 2 still requires an explicit KEEP decision. | + +- **Open gate:** Phase 2 must not start until the user records an explicit **KEEP** decision after completing the Phase 1 trial. This does not block implementing or deploying Phase 1. +- **Trial decision record:** PENDING + +## Phase 1 — Deploy only the Forgejo trial + +#### P1-T1 — Add the native Forgejo service and ingress + +- **Change:** + - Create `modules/forgejo.nix` and import it once from `flake.nix`. Keep the complete Phase 1 deployment in this one module: Forgejo service, SQLite/database settings, state and repository paths, package exposure, and nginx/ACME ingress. Do not distribute those settings across `configuration.nix` or additional Forgejo deployment modules. + - Enable `services.forgejo` with `pkgs.forgejo-lts`, SQLite, the default `forgejo` account, `/var/lib/forgejo` state, and `/var/lib/forgejo/repositories` repository storage. + - Bind HTTP only to `127.0.0.1:3001`; set `DOMAIN`, `ROOT_URL`, and `SSH_DOMAIN` to `git.tobiasostner.de`; set `SSH_PORT = 22`, `SSH_USER = "forgejo"`, and `START_SSH_SERVER = false`. + - Enable secure cookies and disable registration. Keep LFS disabled and explicitly disable Actions, package registry, and mailer facilities. + - Make the selected Forgejo package available in the system profile so administrative commands use the deployed version. + - Add an nginx virtual host with ACME/forced TLS, `merge_slashes off`, `client_max_body_size 512M`, HTTP/1.1, and Forgejo's required Host, real-IP, forwarding, Connection, and Upgrade headers. + - Do not add secrets, Restic options, backup paths, revision metadata, tests, timers, or firewall ports. +- **Starts at:** `flake.nix:25-38`; new `modules/forgejo.nix`; `modules/bitcoin-pulse.nix:354-383`. +- **Tests:** Protect the SQLite/state model, closed registration, excluded features, public and SSH URLs, loopback-only backend, proxy behavior, pinned package choice, and absence of a new firewall port. +- **Verify:** + - `nix eval --raw .#nixosConfigurations.tobserver.config.services.forgejo.package.version` prints `11.0.15` for the current lock. + - `nix eval --json .#nixosConfigurations.tobserver.config.services.forgejo.settings | jq -e '.database.DB_TYPE == "sqlite3" and .server.HTTP_ADDR == "127.0.0.1" and .server.HTTP_PORT == 3001 and .server.DOMAIN == "git.tobiasostner.de" and .server.ROOT_URL == "https://git.tobiasostner.de/" and .server.SSH_DOMAIN == "git.tobiasostner.de" and .server.SSH_PORT == 22 and .server.SSH_USER == "forgejo" and .server.START_SSH_SERVER == false and .service.DISABLE_REGISTRATION == true and .session.COOKIE_SECURE == true and .actions.ENABLED == false and .packages.ENABLED == false and .mailer.ENABLED == false'` prints `true` and exits 0. + - `nix eval --json .#nixosConfigurations.tobserver.config.networking.firewall.allowedTCPPorts | jq -e 'sort == [22,80,443,50001]'` prints `true` and exits 0, proving the complete existing port set is unchanged. + - `nix eval --json .#nixosConfigurations.tobserver.config.services.restic.backups | jq -e 'has("forgejo") | not'` and `nix eval --json .#nixosConfigurations.tobserver.config.sops.secrets | jq -e 'has("restic/forgejo-password") | not'` both print `true` and exit 0. +- **Risk/recovery:** Wrong URL, SSH, database, or proxy settings can break startup, redirects, authentication, or clone URLs. Keeping the complete core deployment in `modules/forgejo.nix` makes review and a narrow revert straightforward. A reviewed removal of its single import must remove only Forgejo's service and virtual host; never delete `/var/lib/forgejo` as an automatic rollback action. + +#### P1-T2 — Write the trial deployment and administration runbook + +- **Change:** + - Create `docs/forgejo-operations.md` with Phase 1-only instructions for DNS confirmation, evaluation, clean-checkout build-before-switch, activation, service/listener/nginx/journal checks, and rollback boundaries for a multi-service host. + - Document first-admin creation with the deployed `forgejo` binary, explicit `HOME`, `USER`, work/custom paths, `--config /var/lib/forgejo/custom/conf/app.ini`, `--username "oibot"`, a privately supplied runtime `admin_email` passed as `--email "$admin_email"`, `--admin`, and `--random-password`. Do not record the email in this plan. + - Require immediate password-manager capture and prohibit operator-selected passwords in argv/history. + - Document SSH-key registration, creation of a disposable smoke repository, an HTTPS clone/push, and an SSH clone/fetch. + - State prominently that Phase 1 data is unprotected and should remain disposable. + - Do not add backup, retention, manifest, recovery, or restore instructions yet. +- **Starts at:** New `docs/forgejo-operations.md`; `docs/bitcoin-pulse-operations.md`. +- **Depends on:** P1-T1. +- **Tests:** Protect secret-safe/user-correct administration, exact deployed paths, both clone transports, multi-service rollback safety, and the no-protection warning. +- **Verify:** + - `git diff --check -- docs/forgejo-operations.md` exits 0. + - Exact searches confirm the runbook contains `git.tobiasostner.de`, the `sudo -u forgejo env HOME=/var/lib/forgejo USER=forgejo FORGEJO_WORK_DIR=/var/lib/forgejo FORGEJO_CUSTOM=/var/lib/forgejo/custom forgejo --config /var/lib/forgejo/custom/conf/app.ini admin user create --username "oibot" --email "$admin_email" --admin --random-password` flow, HTTPS and SSH checks, and the words `unprotected` and `disposable`. + - The Phase 1 document makes no claim that trial data is backed up or recoverable. +- **Risk/recovery:** Preserve the generated password immediately; never place it in Git, Nix expressions, logs, or shell history. + +#### P1-T3 — Build the trial without activation + +- **Change:** + - Reconfirm DNS and the lock-specific Forgejo version. + - Track this plan with the Phase 1 implementation commit so the later decision record exists in every clean deployment checkout. + - Format and validate only Phase 1 Nix files, evaluate the full flake, and build the complete host before switching. + - On Tobserver, require a clean reviewed checkout and build the host configuration without activation. + - Before the first switch, require `/var/lib/forgejo` to be absent or empty; stop for review if residual state exists because migration/reuse is out of scope. +- **Starts at:** Completed P1-T1 and P1-T2 changes. +- **Depends on:** P1-T1, P1-T2. +- **Tests:** Protect DNS/ACME routing, new-empty-instance scope, formatting, full evaluation, and build-before-switch. +- **Verify:** + - `test "$(dig +short git.tobiasostner.de A)" = "162.55.103.218"` and `test -z "$(dig +short git.tobiasostner.de AAAA)"` exit 0. + - `nixpkgs-fmt --check flake.nix modules/forgejo.nix`, `git diff --check`, and `nix flake check --no-build` exit 0. + - On a compatible Linux builder, `nix build .#nixosConfigurations.tobserver.config.system.build.toplevel` exits 0. + - `git ls-files --error-unmatch .plans/deploy-forgejo-with-backups.md` exits 0 after the reviewed Phase 1 commit, proving the plan is tracked rather than hidden by a cleanliness exception. + - On Tobserver, `test -z "$(git status --porcelain)"`, `git ls-files --error-unmatch .plans/deploy-forgejo-with-backups.md`, `nix flake check --no-build`, `sudo test ! -e /var/lib/forgejo || ! sudo find /var/lib/forgejo -mindepth 1 -print -quit | grep -q .`, and `sudo nixos-rebuild build --flake .#tobserver` all exit 0 before any switch. +- **Risk/recovery:** A full-build failure blocks activation but is not currently a known baseline failure. Do not bypass a failed build or overwrite unexpected Forgejo state. + +#### P1-T4 — Activate and verify the public trial + +- **Change:** + - Switch to the reviewed generation, create the first admin with the documented `--admin --random-password` flow, and verify that Forgejo reports the account as an administrator. + - Register an SSH key, create a disposable smoke repository, push a disposable commit, and exercise both HTTPS and host-SSH transports. + - Verify service health, TLS ingress, closed registration, listener isolation, and rendered nginx behavior. +- **Starts at:** `docs/forgejo-operations.md`; deployed Tobserver checkout. +- **Depends on:** P1-T3. +- **Tests:** Protect service health, TLS/redirect, proxy configuration, loopback-only backend, closed registration, first-admin login, host OpenSSH integration, and HTTPS/SSH repository use. +- **Verify:** + - `sudo systemctl is-active forgejo.service nginx.service` reports both `active`. + - `sudo ss -H -ltn 'sport = :3001'` contains `127.0.0.1:3001` and no non-loopback listener; an external connection to port 3001 fails. + - `sudo nginx -t` exits 0; `sudo nginx -T 2>&1` contains the `git.tobiasostner.de` server, `merge_slashes off`, `client_max_body_size 512M`, and required proxy headers. + - `curl --fail --silent --show-error -H 'Host: git.tobiasostner.de' http://127.0.0.1:3001/api/v1/version | jq -e .` exits 0. + - `curl --fail --silent --show-error --location --output /dev/null --write-out '%{http_code} %{url_effective}\n' http://git.tobiasostner.de/` reports `200` at `https://git.tobiasostner.de/`. + - `sudo -u forgejo env HOME=/var/lib/forgejo USER=forgejo FORGEJO_WORK_DIR=/var/lib/forgejo FORGEJO_CUSTOM=/var/lib/forgejo/custom forgejo --config /var/lib/forgejo/custom/conf/app.ini admin user list --admin | grep -F -- "oibot"` exits 0, and the UI offers no registration flow. + - From an external client, `! nc -z git.tobiasostner.de 3001` succeeds; the documented HTTPS clone/push and `forgejo@git.tobiasostner.de:/clone-smoke.git` SSH clone/fetch all exit 0 for the disposable repository. +- **Risk/recovery:** If unhealthy, collect Forgejo/nginx journals before changing state. Deploy a reviewed Forgejo-only revert rather than casually switching the multi-service host to an older full generation. Preserve `/var/lib/forgejo`. + +### Phase 1 acceptance + +- Forgejo is publicly usable over HTTPS and host SSH, listens only on `127.0.0.1:3001`, permits no open registration, exposes no new firewall port, and supports disposable repository use through both transports. +- No Forgejo backup secret, backup service/timer, staging path, manifest/revision plumbing, VM backup test, backup/recovery instructions, or restore claim has been added. +- **The trial remains unprotected. Its data must remain disposable until Phase 2's first production snapshot passes.** + +## Trial decision gate + +#### G1 — Record KEEP or ABANDON + +- **Change:** + - Require the user to record one explicit outcome after Phase 1; do not infer it from elapsed time. + - Record the decision durably in this plan by replacing `PENDING` above with exactly `KEEP | YYYY-MM-DD | approved by Tobias Ostner` or `ABANDON | YYYY-MM-DD | approved by Tobias Ostner`, then commit that decision before making any Phase 2 change. + - **KEEP:** Preserve `/var/lib/forgejo` in place and authorize P2-T1 through P2-T6. Do not store the only authoritative copy of data in Forgejo until P2-T6 succeeds. + - **ABANDON:** Stop this implementation plan and skip every Phase 2 task. Prepare a separate, narrowly reviewed removal plan that handles host-SSH write exclusion, service/ingress removal, state preservation, validation, and any later destruction decision; do not improvise removal from this plan. +- **Starts at:** This plan's `Trial decision record`; deployed Phase 1 instance. +- **Depends on:** P1-T4. +- **Tests:** Protect explicit owner authorization and prevent Phase 2 from starting on an inferred or undocumented decision. +- **Verify — KEEP:** From a clean checkout, run the following before P2-T1 begins; it requires exactly one KEEP record, validates its calendar date, rejects unresolved/ABANDON records, and proves the decision is tracked and committed: + + ```console + set -euo pipefail + plan=.plans/deploy-forgejo-with-backups.md + decision_lines="$(rg -x -- '- \*\*Trial decision record:\*\* KEEP \| [0-9]{4}-[0-9]{2}-[0-9]{2} \| approved by Tobias Ostner' "$plan" || true)" + test "$(printf '%s\n' "$decision_lines" | grep -c '^-')" -eq 1 + decision_date="$(printf '%s\n' "$decision_lines" | sed -E 's/^.*KEEP \| ([0-9-]+) \|.*$/\1/')" + test "$(date -d "$decision_date" +%F)" = "$decision_date" + ! rg -F -- '- **Trial decision record:** PENDING' "$plan" + ! rg -F -- '- **Trial decision record:** ABANDON' "$plan" + git ls-files --error-unmatch "$plan" + git diff --quiet -- "$plan" + git diff --cached --quiet -- "$plan" + test -n "$(git log -1 --format=%H -- "$plan")" + sudo test -d /var/lib/forgejo + sudo systemctl is-active forgejo.service + ``` + + Expected: every command exits 0 and Forgejo remains active with its state in place. +- **Verify — ABANDON:** The decision record contains the dated ABANDON form, no Phase 2 file has been implemented, and a separate removal plan is reviewed before any removal action. +- **Risk/recovery:** A broad generation rollback or ad hoc state move can affect unrelated services or race host-SSH writes. Removal is deliberately deferred to a separate plan if ABANDON is chosen. + +## Phase 2 — Add backups and surrounding infrastructure after KEEP + +#### P2-T1 — Add separable backup configuration, revision wiring, and secret + +- **Change:** + - Create `modules/forgejo-backup.nix` and import it separately from `modules/forgejo.nix`, so backup infrastructure can be removed without removing Forgejo. + - Pass `self` through `flake.nix` only in this phase and set `system.configurationRevision` from clean `self.rev`, allowing `dirtyRev` only for non-production evaluation. Production backup preparation must reject a missing, unknown, or dirty revision. + - Declare `sops.secrets."restic/forgejo-password"` as `root:root` mode `0400`. + - Add `restic.forgejo-password` to `secrets/tobserver.yaml` through SOPS using a high-entropy password retained in the approved password manager. + - Reuse `backblaze/tobcloud-backup-env` only for B2 access; never duplicate its plaintext value. +- **Starts at:** `flake.nix:16-38`; new `modules/forgejo-backup.nix`; `configuration.nix:118-127`; `secrets/tobserver.yaml`. +- **Depends on:** G1 = KEEP. +- **Tests:** Protect explicit KEEP authorization, encrypted Git content, authorized SOPS decryption, declarative permissions, clean revision identity, Restic-password independence, and backup-module separability. +- **Verify:** + - Re-run G1's exact decision-record command before making any Phase 2 change. + - `sops --decrypt secrets/tobserver.yaml >/dev/null` exits 0 without printing plaintext. + - The encrypted diff contains the new key only as an `ENC[...]` value. + - `nix eval --json '.#nixosConfigurations.tobserver.config.sops.secrets."restic/forgejo-password"' | jq -e '.owner == "root" and .group == "root" and .mode == "0400"'` prints `true` and exits 0; runtime permissions are checked after activation in P2-T6. +- **Risk/recovery:** Losing this password makes all Forgejo snapshots unreadable. It must never enter Nix strings, argv, history, logs, or plaintext Git content. + +#### P2-T2 — Implement the complete-state archive and Restic job + +- **Change:** + - Configure `services.restic.backups.forgejo` with `initialize = true`, repository `b2:tobcloud-backup:forgejo`, the dedicated password, the shared B2 environment file, and only `/var/backup/forgejo/forgejo-state.tar` as its Restic path. + - Create `/var/backup/forgejo` as `root:root` mode `0700`; keep Restic's generated runtime directory private; create independent tmpfiles-managed `/run/forgejo-backup` as `root:root` mode `0700`; use `UMask = "0077"` and a six-hour timeout. + - Remove stale temporary/final archives before preparation and require `forgejo.service` active. Keep `du`/`df` capacity comparison advisory because Btrfs free-space accounting makes a fixed threshold unreliable. + - Before claiming ownership, require `/run/forgejo-backup/forgejo-stopped` to be absent. If it already exists, run only the idempotent recovery path, clear it only after Forgejo is confirmed active, and fail this invocation without archiving; retain the marker and fail if recovery does not succeed. + - Create `/run/forgejo-backup/forgejo-stopped` only after clean prechecks, stop Forgejo, and wait with a bounded timeout until no process owned by `forgejo` remains. If an existing Git-over-SSH process remains, fail before archiving and restart Forgejo through cleanup. + - Do not stop or reconfigure sshd. Operate the job in an idle/no-push window and disclose the accepted race from a new SSH push starting after the process check. + - While Forgejo is stopped, create root-level `forgejo-backup-manifest.txt` with exactly four ordered newline-delimited fields: `format_version=1`, `forgejo_version=`, `system_path=`, and `configuration_revision=`. Fail if any value is missing or dirty/unknown. + - Archive all `/var/lib/forgejo` as `forgejo/` plus the manifest into a temporary tar while preserving numeric ownership, ACLs, and extended attributes; atomically rename it to the final staging path. + - Restart Forgejo, require it active, and clear the marker before allowing Restic upload. + - Preserve preparation's original status. Its EXIT trap must remove temporary artifacts, remove the final archive when preparation failed, and retry Forgejo restart only while this run owns the marker. + - Make `backupCleanupCommand` idempotently retry a marked restart, clear the marker only after Forgejo is confirmed active, retain the marker after terminal restart failure, and unconditionally remove temporary/final staging after the Restic attempt. The next run removes persistent stale staging. + - Do not configure `Requires=forgejo.service`; stopping a required unit would cancel the backup. + - Schedule daily at 04:00 with `Persistent = true`, up to 30 minutes random delay, and prune `7/4/12`. +- **Starts at:** New `modules/forgejo-backup.nix`; `modules/bitcoin-pulse.nix:115-174,340-350,386-388`. +- **Depends on:** P2-T1. +- **Tests:** Protect complete state/manifest scope, bounded active-process rejection, stale-artifact prevention, atomic publication, status-preserving restart/cleanup, no stale upload after failure, private staging, exact schedule/retention, and absence of `Requires=forgejo.service`. +- **Verify:** + - `nix eval --raw .#nixosConfigurations.tobserver.config.services.restic.backups.forgejo.repository` prints `b2:tobcloud-backup:forgejo`. + - `nix eval --json .#nixosConfigurations.tobserver.config.services.restic.backups.forgejo.paths | jq -e '. == ["/var/backup/forgejo/forgejo-state.tar"]'` prints `true`. + - `nix eval --json .#nixosConfigurations.tobserver.config.services.restic.backups.forgejo.pruneOpts | jq -e '. == ["--keep-daily 7","--keep-weekly 4","--keep-monthly 12"]'` prints `true`. + - `nix eval --json .#nixosConfigurations.tobserver.config.systemd.services.restic-backups-forgejo.requires | jq -e 'index("forgejo.service") | not'` prints `true` and exits 0. + - `nix eval --json .#nixosConfigurations.tobserver.config.systemd.services.restic-backups-forgejo.serviceConfig | jq -e '.RuntimeDirectoryMode == "0700" and .UMask == "0077" and .TimeoutStartSec == "6h"'` prints `true` and exits 0. + - Evaluate `systemd.tmpfiles.rules` and require the exact `/run/forgejo-backup 0700 root root` rule; inspect the generated preparation/cleanup scripts and require the exact stop/check/archive/restart-before-upload and status-preserving cleanup order. +- **Risk/recovery:** A newly racing SSH push can make a snapshot inconsistent; this is accepted. Any process, archive, restart, or upload failure must fail visibly without uploading stale content. Removing the separate backup module must leave Forgejo and `/var/lib/forgejo` intact. + +#### P2-T3 — Add a disposable backup-mechanism VM check + +- **Change:** + - Factor orchestration generation into `modules/forgejo-backup-script.nix`. Permit build-time-only phase hooks for the VM check; production must instantiate it without hooks, production options, or runtime fault-injection variables. + - Add `tests/forgejo-backup.nix` and expose `checks.x86_64-linux.forgejo-backup` with a local Restic repository. + - Prove normal ordering and deterministically exercise pre-stop rejection, a lingering Forgejo-owned process, archive failure, one-shot restart failure recovered by cleanup, terminal restart failure with retained marker, Restic failure after restart, forced backup-unit termination, and stale staging on the next run. + - Assert this expected outcome matrix; `empty` means no entry of any type remains in staging: + + | Case | Backup unit | Forgejo | Marker | Staging | Snapshot | sshd | + |---|---|---|---|---|---|---| + | Normal | success | active | absent | empty | +1 | active | + | Pre-stop rejection | failed | active | absent | empty | unchanged | active | + | Lingering Forgejo process | failed | active after cleanup | absent | empty | unchanged | active | + | Archive failure | failed | active after cleanup | absent | empty | unchanged | active | + | One-shot restart failure | failed | active after cleanup retry | absent | empty | unchanged | active | + | Terminal restart failure | failed | failed/inactive | retained | empty | unchanged | active | + | Restic failure after restart | failed | active | absent | empty | unchanged | active | + | Forced unit termination | failed | active after cleanup | absent | empty | unchanged | active | + | Pre-existing retained marker | failed after recovery-only path | active after remediation | absent | empty | unchanged | active | + | Next run with stale staging | success | active | absent | empty | +1 only | active | + + - After the terminal restart case, exercise explicit operator remediation that starts Forgejo, confirms it active, and clears the retained marker before continuing. + - Do not extract/restore a backup or claim protection from a newly racing SSH push. +- **Starts at:** New `modules/forgejo-backup-script.nix`; new `tests/forgejo-backup.nix`; `flake.nix`; P2-T2 behavior. +- **Depends on:** P2-T2. +- **Tests:** Protect destructive failure paths and ordering that static evaluation cannot establish. +- **Verify:** `nix build .#checks.x86_64-linux.forgejo-backup` exits 0 and the log shows every deterministic success/failure case completed; production evaluation contains no test hook. +- **Risk/recovery:** Fault injection must remain build-time-only. A failing case blocks backup activation; never reproduce destructive test failures against trial data. + +#### P2-T4 — Expand the runbook with backup and recovery operations + +- **Change:** + - Expand `docs/forgejo-operations.md` without weakening its Phase 1 deployment/admin instructions. + - Add advisory capacity inspection, an idle/no-push backup window, manual backup invocation, snapshot freshness, service restart and timer checks, rendered-unit inspection, private streamed archive/manifest inspection, staging cleanup, and shared B2 credential rotation. + - Update the canonical shared-credential section in `docs/bitcoin-pulse-operations.md` so rotation names and validates Bitcoin Pulse, Nextcloud, and Forgejo rather than only the existing two jobs. + - Document recovery using the manifest's Forgejo version, system path, clean configuration revision, and matching committed flake/lock. Enter a maintenance boundary that denies only the `forgejo` SSH account without stopping sshd; stop Forgejo; wait for zero Forgejo-owned processes; restore privately; retain current state under a timestamped rollback path; extract with ownership/ACLs/xattrs; start and verify Forgejo; and restore Forgejo SSH access only after acceptance or rollback. Retain rollback data and the source snapshot until acceptance. + - Require preservation or reproducible rebuilding of the matching generation before restoration. + - State that content inspection is not a restore test, recoverability remains unproven, sshd stays active as a service, and the overlapping-push race remains accepted for backup creation only. +- **Starts at:** `docs/forgejo-operations.md`; `docs/bitcoin-pulse-operations.md:191-222`; P2-T2 paths and units. +- **Depends on:** P2-T2, P2-T3. +- **Tests:** Protect exact paths/units, complete recovery scope, version-compatible startup, reversible recovery, credential-rotation scope, no-push instructions, and honest claims. +- **Verify:** + - `git diff --check -- docs/forgejo-operations.md docs/bitcoin-pulse-operations.md` exits 0. + - Exact searches confirm the Forgejo runbook names `restic-backups-forgejo.service`, `forgejo-backup-manifest.txt`, the no-push window, snapshot freshness, manifest inspection, `Recovery`, `Rollback`, Forgejo-only SSH denial during recovery, and the unproven-recoverability warning. + - Exact searches in `docs/bitcoin-pulse-operations.md` require all three unit names: `restic-backups-bitcoin-pulse.service`, `restic-backups-nextcloud.service`, and `restic-backups-forgejo.service`. +- **Risk/recovery:** A newer Forgejo can migrate SQLite and make downgrade unsafe. Recovery must identify/rebuild the matching generation and preserve old/current state and the Restic snapshot until acceptance. + +#### P2-T5 — Build Phase 2 without activation + +- **Change:** + - Format all Phase 2 Nix files, run the backup VM check, evaluate the full flake, and build the complete host. + - On Tobserver, require a clean reviewed checkout, verify `system.configurationRevision` equals the deployment commit, and build without switching. + - Confirm Forgejo remains active and `/var/lib/forgejo` remains intact; Phase 2 augments the trial instance rather than reinitializing it. +- **Starts at:** Completed P2-T1 through P2-T4 changes. +- **Depends on:** P2-T1, P2-T2, P2-T3, P2-T4. +- **Tests:** Protect formatting, VM failure behavior, clean revision identity, full evaluation/build, and trial-state preservation. +- **Verify:** + - `nixpkgs-fmt --check flake.nix modules/forgejo.nix modules/forgejo-backup.nix modules/forgejo-backup-script.nix tests/forgejo-backup.nix`, `git diff --check`, `nix build .#checks.x86_64-linux.forgejo-backup`, and `nix flake check --no-build` all exit 0. + - On a compatible Linux builder, `nix build .#nixosConfigurations.tobserver.config.system.build.toplevel` exits 0. + - On Tobserver, re-run G1's exact KEEP-record command, then require `test -z "$(git status --porcelain)"`, `test "$(nix eval --raw .#nixosConfigurations.tobserver.config.system.configurationRevision)" = "$(git rev-parse HEAD)"`, `sudo systemctl is-active forgejo.service`, `sudo test -d /var/lib/forgejo`, and `sudo nixos-rebuild build --flake .#tobserver` to exit 0 without activation. +- **Risk/recovery:** A failed check/build blocks backup activation but does not invalidate the running trial. Do not deploy from a dirty revision or bypass the VM check. + +#### P2-T6 — Activate backups and inspect the first production snapshot + +- **Change:** + - Switch to the already-built Phase 2 generation. + - Record Forgejo's activation timestamp and latest snapshot ID, manually run the backup, and prove Forgejo restarted before upload and a new snapshot was created. + - Verify job result, timer, rendered unit, stored tar path, required tar entries and exact manifest, Forgejo activity, marker absence, and local staging cleanup. + - Inspect redacted journals for orderly stop/process-check/archive/restart/upload behavior. Do not extract the archive or describe these checks as recovery validation. +- **Starts at:** Deployed `restic-backups-forgejo.service`; expanded `docs/forgejo-operations.md`. +- **Depends on:** P2-T5. +- **Tests:** Protect fresh encrypted upload, complete content, restart-before-upload, cleanup, timer/retention activation, and honest reporting. +- **Verify:** + - Capture `before_service=$(systemctl show forgejo.service --property=ActiveEnterTimestampMonotonic --value)` and `before_snapshot=$(sudo restic-forgejo snapshots --json --latest 1 2>/dev/null | jq -r '.[0].id // empty')`; run `sudo systemctl start restic-backups-forgejo.service`. + - `systemctl show restic-backups-forgejo.service --property=Result --property=ExecMainStatus` reports `Result=success` and `ExecMainStatus=0`; Forgejo is active; its activation timestamp increased; `/run/forgejo-backup/forgejo-stopped` is absent. + - Capture `after_snapshot=$(sudo restic-forgejo snapshots --json --latest 1 | jq -er '.[0].id')`; require it to be non-empty and different from `$before_snapshot`. + - `sudo restic-forgejo ls "$after_snapshot" | grep -Fx '/var/backup/forgejo/forgejo-state.tar'` succeeds. + - Under `set -o pipefail`, privately stream the tar listing and require `forgejo/data/forgejo.db`, `forgejo/repositories/`, `forgejo/custom/conf/`, `forgejo/.ssh/`, and `forgejo-backup-manifest.txt`. Privately extract only the manifest and compare its exact four ordered lines, with no extras, to the deployed package version, `readlink -f /run/current-system`, and `nixos-version --configuration-revision`. + - `sudo stat --format='%U %G %a' /run/secrets/restic/forgejo-password` prints `root root 400`. + - `test -z "$(sudo find /var/backup/forgejo -mindepth 1 -print -quit)"` exits 0, proving no file, directory, or symlink remains. `systemctl is-enabled restic-backups-forgejo.timer` prints `enabled`; `systemctl show restic-backups-forgejo.timer --property=TimersCalendar --property=Persistent --property=RandomizedDelayUSec` reports daily 04:00, `Persistent=yes`, and 30 minutes randomized delay. +- **Risk/recovery:** Upload/content inspection does not prove restoration or close the SSH race. On failure, first ensure Forgejo is active, inspect the marker and redacted journals, and revert only the separate backup module if necessary. The deployment remains unprotected until this task fully passes. + +### Phase 2 acceptance + +- The dedicated Restic password is encrypted in SOPS and deployed root-only; the shared B2 credential is reused without duplication. +- The job stops Forgejo, rejects remaining Forgejo-owned processes, archives all `/var/lib/forgejo` plus the exact version/system/revision manifest, restarts Forgejo before upload, and cleans staging. +- The daily 04:00 persistent timer, up to 30 minutes random delay, and 7/4/12 retention are active. +- The build-only VM check passes all approved failure paths and cannot inject faults into production. +- The runbook matches deployed paths and clearly states that restore testing was not performed. +- The first production snapshot is fresh and its archive/manifest content has been inspected successfully. +- Only after these checks pass is the retained deployment considered backup-protected. Recoverability remains unproven because restore tests are excluded. + +## Final acceptance + +- **Checks:** Phase 1 passes DNS revalidation, formatting, full evaluation/build, empty-state, service, TLS, registration, listener, admin, and disposable HTTPS/SSH repository checks without adding backup infrastructure. Exactly one dated trial decision is then recorded in this plan. KEEP authorizes Phase 2 and preserves state; ABANDON stops this plan and skips Phase 2 pending a separate removal plan. After KEEP, the VM check, full build, first production backup, snapshot freshness/content, manifest, timer, retention, restart, and cleanup checks all pass. +- **End state:** Before the gate, a minimal but explicitly unprotected Forgejo trial is available. After KEEP and Phase 2, the same retained instance has encrypted daily off-server backups and documented recovery/rollback procedures. After ABANDON, no backup infrastructure is added and removal is handled separately. +- **Deferrals or blockers:** Phase 1 has no known external blocker beyond its normal pre-activation checks. Phase 2 is intentionally blocked on the future KEEP decision. Restore testing and alerting remain deferred; sshd remains available during backup, leaving the accepted overlapping-push consistency race. diff --git a/docs/forgejo-operations.md b/docs/forgejo-operations.md new file mode 100644 index 0000000..9a6c954 --- /dev/null +++ b/docs/forgejo-operations.md @@ -0,0 +1,216 @@ +# Forgejo trial operations + +This runbook covers the initial Forgejo trial on Tobserver. The complete +service, SQLite, package, and nginx configuration lives in +`modules/forgejo.nix`. + +> **Warning:** This trial is unprotected. It has no Forgejo backup job, and its +> data must be treated as disposable. Keep the authoritative copy of anything +> important somewhere else. + +Run deployment commands from the Tobserver checkout unless noted otherwise. +Never place passwords or other credentials in Git, Nix expressions, command +arguments, shell history, logs, or issue comments. + +## Pre-deployment checks + +Confirm that the dedicated hostname points only to Tobserver: + +```console +test "$(dig +short git.tobiasostner.de A)" = "162.55.103.218" +test -z "$(dig +short git.tobiasostner.de AAAA)" +``` + +On a workstation, evaluate the configuration and build it on a compatible Linux +builder when one is available: + +```console +nix flake check --no-build +nix build .#nixosConfigurations.tobserver.config.system.build.toplevel +``` + +Commit and push the reviewed changes. On Tobserver, update a clean checkout and +build without activating it: + +```console +cd ~/tobserver +test -z "$(git status --porcelain)" || { + echo "Refusing to deploy from a dirty checkout" >&2 + exit 1 +} +git pull --ff-only +git rev-parse HEAD +nix flake check --no-build + +if sudo test -e /var/lib/forgejo \ + && sudo find /var/lib/forgejo -mindepth 1 -print -quit | grep -q .; then + echo "Refusing to reuse non-empty Forgejo state" >&2 + exit 1 +fi + +sudo nixos-rebuild build --flake .#tobserver +``` + +Compare the printed commit with the reviewed deployment commit. Do not continue +if evaluation, the full host build, or the empty-state check fails. + +## Activation and service checks + +Activate the configuration only after reviewing the successful build. Immediately +before switching, recheck that the checkout is clean and still at the reviewed +commit printed during the build step: + +```console +read -r -p 'Reviewed commit hash: ' reviewed_commit +test -z "$(git status --porcelain)" || { + echo "Refusing to switch from a dirty checkout" >&2 + exit 1 +} +current_commit="$(git rev-parse HEAD)" +printf 'Switching commit: %s\n' "$current_commit" +test "$current_commit" = "$reviewed_commit" || { + echo "Refusing to switch an unreviewed commit" >&2 + exit 1 +} +sudo nixos-rebuild switch --flake .#tobserver +unset current_commit reviewed_commit +``` + +Verify the services, loopback-only backend, local API, rendered nginx +configuration, and public HTTPS endpoint: + +```console +sudo systemctl is-active forgejo.service nginx.service +sudo ss -H -ltn 'sport = :3001' + +curl --fail --silent --show-error \ + -H 'Host: git.tobiasostner.de' \ + http://127.0.0.1:3001/api/v1/version \ + | jq -e . + +sudo nginx -t +( + set -euo pipefail + nginx_config="$(sudo nginx -T 2>&1)" + for required_line in \ + 'git.tobiasostner.de' \ + 'merge_slashes off' \ + 'client_max_body_size 512M' \ + 'proxy_set_header Host $host' \ + 'proxy_set_header X-Real-IP $remote_addr' \ + 'proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for' \ + 'proxy_set_header X-Forwarded-Proto $scheme' \ + 'proxy_set_header Connection $http_connection' \ + 'proxy_set_header Upgrade $http_upgrade' + do + printf '%s\n' "$nginx_config" | grep -F -- "$required_line" >/dev/null + done +) + +curl --fail --silent --show-error --location \ + --output /dev/null \ + --write-out '%{http_code} %{url_effective}\n' \ + http://git.tobiasostner.de/ +``` + +The listener output must contain `127.0.0.1:3001` and no non-loopback address. +The public request must report HTTP 200 at `https://git.tobiasostner.de/`. +Port 3001 must not be reachable from an external machine: + +```console +! nc -z git.tobiasostner.de 3001 +``` + +If a check fails, inspect logs before restarting or changing state: + +```console +sudo journalctl \ + -u forgejo.service \ + -u nginx.service \ + --since '-15 minutes' \ + --no-pager +``` + +## Create the first administrator + +Run this once in an interactive shell on Tobserver. Enter the administrator +email when prompted; it stays in the shell variable and is not recorded in this +runbook or shell history. + +```console +read -r -p 'Admin email: ' admin_email +sudo -u forgejo env \ + HOME=/var/lib/forgejo \ + USER=forgejo \ + FORGEJO_WORK_DIR=/var/lib/forgejo \ + FORGEJO_CUSTOM=/var/lib/forgejo/custom \ + forgejo --config /var/lib/forgejo/custom/conf/app.ini \ + admin user create \ + --username "oibot" \ + --email "$admin_email" \ + --admin \ + --random-password +unset admin_email +``` + +The command prints a generated password once. Copy it directly into the +approved password manager immediately, then clear the terminal. Do not redirect, +pipe, log, or save the output in a temporary file, and do not choose a password +on the command line. + +Confirm that the account is an administrator: + +```console +sudo -u forgejo env \ + HOME=/var/lib/forgejo \ + USER=forgejo \ + FORGEJO_WORK_DIR=/var/lib/forgejo \ + FORGEJO_CUSTOM=/var/lib/forgejo/custom \ + forgejo --config /var/lib/forgejo/custom/conf/app.ini \ + admin user list --admin \ + | grep -F -- 'oibot' +``` + +Sign in at `https://git.tobiasostner.de/` and confirm that the interface does +not offer public registration. + +## Disposable HTTPS and SSH smoke test + +In the Forgejo interface: + +1. Add the external client's public SSH key to the `oibot` account. +2. Create an empty repository named `clone-smoke`. + +From an external client, exercise HTTPS clone and push with a disposable commit: + +```console +workdir="$(mktemp -d)" +git clone https://git.tobiasostner.de/oibot/clone-smoke.git \ + "$workdir/clone-smoke-https" +cd "$workdir/clone-smoke-https" +printf '%s\n' 'Forgejo trial' > README.md +git add README.md +git commit -m 'Test Forgejo trial' +git push origin HEAD:main +``` + +Then exercise host OpenSSH clone and fetch: + +```console +git clone forgejo@git.tobiasostner.de:oibot/clone-smoke.git \ + "$workdir/clone-smoke-ssh" +git -C "$workdir/clone-smoke-ssh" fetch origin +rm -rf -- "$workdir" +``` + +All clone, push, and fetch commands must exit successfully. The repository and +commit are disposable trial data. + +## Trial rollback boundary + +Tobserver hosts unrelated services, so do not use a broad live NixOS generation +rollback as a routine Forgejo test. If the service is unhealthy, collect its +status and journals first. Prepare and review a narrow change that removes only +the `modules/forgejo.nix` import and its Forgejo service and nginx virtual host. +Preserve `/var/lib/forgejo`; never delete trial state as an automatic rollback +action. diff --git a/flake.nix b/flake.nix index df63841..6d08d6a 100644 --- a/flake.nix +++ b/flake.nix @@ -26,6 +26,7 @@ sops-nix.nixosModules.sops ./hardware-configuration.nix ./configuration.nix + ./modules/forgejo.nix ./modules/bitcoin-pulse.nix ./modules/bitcoin.nix ./modules/electrs.nix diff --git a/modules/forgejo.nix b/modules/forgejo.nix new file mode 100644 index 0000000..07a8eba --- /dev/null +++ b/modules/forgejo.nix @@ -0,0 +1,66 @@ +{ pkgs, ... }: + +{ + services.forgejo = { + enable = true; + package = pkgs.forgejo-lts; + + stateDir = "/var/lib/forgejo"; + repositoryRoot = "/var/lib/forgejo/repositories"; + + database = { + type = "sqlite3"; + path = "/var/lib/forgejo/data/forgejo.db"; + }; + + lfs.enable = false; + + settings = { + server = { + PROTOCOL = "http"; + HTTP_ADDR = "127.0.0.1"; + HTTP_PORT = 3001; + + DOMAIN = "git.tobiasostner.de"; + ROOT_URL = "https://git.tobiasostner.de/"; + + SSH_DOMAIN = "git.tobiasostner.de"; + SSH_PORT = 22; + SSH_USER = "forgejo"; + START_SSH_SERVER = false; + }; + + service.DISABLE_REGISTRATION = true; + session.COOKIE_SECURE = true; + + actions.ENABLED = false; + packages.ENABLED = false; + mailer.ENABLED = false; + }; + }; + + services.nginx.virtualHosts."git.tobiasostner.de" = { + enableACME = true; + forceSSL = true; + extraConfig = '' + merge_slashes off; + ''; + + locations."/" = { + proxyPass = "http://127.0.0.1:3001"; + extraConfig = '' + client_max_body_size 512M; + + proxy_http_version 1.1; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header Connection $http_connection; + proxy_set_header Upgrade $http_upgrade; + ''; + }; + }; + + environment.systemPackages = [ pkgs.forgejo-lts ]; +}