oibot/Tobserver
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
Tobserver/.plans/deploy-forgejo-with-backups.md
Tobias Ostner b49138d95a Deploy Forgejo trial instance
2026-08-01 13:56:44 +07:00

37 KiB
Raw Blame History

Implementation Plan

Status: Phase 1 is ready for implementation; Phase 2 is blocked on the post-trial KEEP decision.

Outcome and boundaries

  • Problem and target: First deploy a minimal, usable Forgejo instance at https://git.tobiasostner.de so it can be evaluated without committing to the backup and recovery design. Add that surrounding infrastructure only after an explicit decision to keep Forgejo.
  • Phase 1 — trial deployment: Native NixOS Forgejo LTS service; SQLite; /var/lib/forgejo state; loopback HTTP; nginx/ACME ingress; host OpenSSH on port 22; closed registration; first-admin bootstrap; and disposable HTTPS/SSH repository checks. Keep all declarative core deployment configuration—Forgejo service, database settings, state paths, nginx virtual host, and package exposure—together in modules/forgejo.nix.
  • Phase 1 non-goals: No new SOPS secret, B2/Restic repository, backup service or timer, staging/runtime backup directories, archive or manifest logic, configuration-revision plumbing, backup VM check, retention policy, production backup, backup/recovery documentation, or restore/protection claim.
  • Trial safety boundary: Data created during Phase 1 has no new backup protection. Use only disposable repositories and retain any authoritative copy elsewhere until Phase 2's first production snapshot has passed its content checks.
  • Decision gate: After the trial, KEEP preserves /var/lib/forgejo and authorizes Phase 2. ABANDON stops this plan, skips every Phase 2 task, and triggers a separate reviewed removal plan whose default is to preserve trial state. No trial duration is prescribed.
  • Phase 2 — retained deployment: Dedicated encrypted Restic password; daily complete-state backup to the existing Backblaze B2 bucket; 7 daily, 4 weekly, and 12 monthly snapshots; deterministic backup failure-path tests; deployment, backup, recovery, and rollback documentation; and inspection of the first production snapshot.
  • Out of scope in both phases: Migration, LFS, Actions/runners, package registry, SMTP, SSO, containers, PostgreSQL, HA, external object storage, additional firewall ports, backup alerting, SSH-service interruption during backup, and initial or recurring restore tests.
  • Phase 2 accepted limitations: sshd remains available during backup. A Git-over-SSH push beginning after the active-process check can overlap archive creation and make that snapshot inconsistent. Snapshot and archive inspection will not prove recovery because restore testing remains excluded.

Key files, evidence, and decisions

File or source Why it matters Decision or plan impact
flake.nix:1-38, flake.lock node nixpkgs_2 The host uses nixos-25.11; host modules are listed in one place; the evaluated forgejo-lts package is currently 11.0.15. Import only the Forgejo service module in Phase 1. Add backup modules, the VM check, and self revision wiring in Phase 2. Re-evaluate the lock-specific package version before each activation.
Pinned nixpkgs nixos/modules/services/misc/forgejo.nix The native module supports SQLite, external OpenSSH, declarative settings, generated persistent secrets, and state rooted at /var/lib/forgejo. Use the native pinned module rather than a container or new flake input.
configuration.nix:35-68 Ports 22, 80, and 443 are already open; OpenSSH, nginx, and ACME are enabled. Reuse those services and do not expose the Forgejo backend port.
modules/bitcoin-pulse.nix:101-106 Bitcoin Pulse occupies 127.0.0.1:3000. Bind Forgejo only to 127.0.0.1:3001.
configuration.nix:118-127, secrets/.sops.yaml SOPS is established and backblaze/tobcloud-backup-env is a shared root-only secret. Do not touch secrets in Phase 1. In Phase 2, reuse the B2 credential and add only a dedicated root-only restic/forgejo-password.
modules/bitcoin-pulse.nix:115-174,340-388 This is the strongest local Restic/private-staging/timer/retention pattern and an existing nginx proxy pattern. Reuse only the nginx conventions in Phase 1. Add Forgejo-specific Restic orchestration in a separable backup module in Phase 2.
docs/bitcoin-pulse-operations.md Defines clean-checkout, build-before-switch, runtime verification, and multi-service rollback conventions. Apply the same deployment discipline in both phases; avoid casual full-generation rollback.
Forgejo backup guidance Complete state needs synchronized point-in-time capture; shutdown provides the consistency boundary used by the existing design. Phase 2 stops Forgejo and requires no remaining Forgejo-owned process before archiving all state.
Forgejo reverse-proxy guidance Documents dedicated-origin, ROOT_URL, encoded-slash, proxy-header, and upload-size behavior. Phase 1 uses a dedicated hostname, exact proxy behavior, and a 512 MiB body limit.
Verified baseline on 2026-08-01 git.tobiasostner.de resolves to 162.55.103.218 with no AAAA record, and nix flake check --no-build passes. Historical DNS, Codeberg-fetch, and Hermes evaluation failures are not current blockers. Recheck DNS and perform complete builds before activation.
User decisions captured by the existing plan New personal instance; public HTTPS; closed registration; SQLite; host SSH; initial admin oibot; daily B2 backup with 7/4/12 retention; no restore tests, alerting, or SSH interruption. Preserve these boundaries. Supply the admin email privately at deployment time; Phase 2 still requires an explicit KEEP decision.
  • Open gate: Phase 2 must not start until the user records an explicit KEEP decision after completing the Phase 1 trial. This does not block implementing or deploying Phase 1.
  • Trial decision record: PENDING

Phase 1 — Deploy only the Forgejo trial

P1-T1 — Add the native Forgejo service and ingress

  • Change:
    • Create modules/forgejo.nix and import it once from flake.nix. Keep the complete Phase 1 deployment in this one module: Forgejo service, SQLite/database settings, state and repository paths, package exposure, and nginx/ACME ingress. Do not distribute those settings across configuration.nix or additional Forgejo deployment modules.
    • Enable services.forgejo with pkgs.forgejo-lts, SQLite, the default forgejo account, /var/lib/forgejo state, and /var/lib/forgejo/repositories repository storage.
    • Bind HTTP only to 127.0.0.1:3001; set DOMAIN, ROOT_URL, and SSH_DOMAIN to git.tobiasostner.de; set SSH_PORT = 22, SSH_USER = "forgejo", and START_SSH_SERVER = false.
    • Enable secure cookies and disable registration. Keep LFS disabled and explicitly disable Actions, package registry, and mailer facilities.
    • Make the selected Forgejo package available in the system profile so administrative commands use the deployed version.
    • Add an nginx virtual host with ACME/forced TLS, merge_slashes off, client_max_body_size 512M, HTTP/1.1, and Forgejo's required Host, real-IP, forwarding, Connection, and Upgrade headers.
    • Do not add secrets, Restic options, backup paths, revision metadata, tests, timers, or firewall ports.
  • Starts at: flake.nix:25-38; new modules/forgejo.nix; modules/bitcoin-pulse.nix:354-383.
  • Tests: Protect the SQLite/state model, closed registration, excluded features, public and SSH URLs, loopback-only backend, proxy behavior, pinned package choice, and absence of a new firewall port.
  • Verify:
    • nix eval --raw .#nixosConfigurations.tobserver.config.services.forgejo.package.version prints 11.0.15 for the current lock.
    • nix eval --json .#nixosConfigurations.tobserver.config.services.forgejo.settings | jq -e '.database.DB_TYPE == "sqlite3" and .server.HTTP_ADDR == "127.0.0.1" and .server.HTTP_PORT == 3001 and .server.DOMAIN == "git.tobiasostner.de" and .server.ROOT_URL == "https://git.tobiasostner.de/" and .server.SSH_DOMAIN == "git.tobiasostner.de" and .server.SSH_PORT == 22 and .server.SSH_USER == "forgejo" and .server.START_SSH_SERVER == false and .service.DISABLE_REGISTRATION == true and .session.COOKIE_SECURE == true and .actions.ENABLED == false and .packages.ENABLED == false and .mailer.ENABLED == false' prints true and exits 0.
    • nix eval --json .#nixosConfigurations.tobserver.config.networking.firewall.allowedTCPPorts | jq -e 'sort == [22,80,443,50001]' prints true and exits 0, proving the complete existing port set is unchanged.
    • nix eval --json .#nixosConfigurations.tobserver.config.services.restic.backups | jq -e 'has("forgejo") | not' and nix eval --json .#nixosConfigurations.tobserver.config.sops.secrets | jq -e 'has("restic/forgejo-password") | not' both print true and exit 0.
  • Risk/recovery: Wrong URL, SSH, database, or proxy settings can break startup, redirects, authentication, or clone URLs. Keeping the complete core deployment in modules/forgejo.nix makes review and a narrow revert straightforward. A reviewed removal of its single import must remove only Forgejo's service and virtual host; never delete /var/lib/forgejo as an automatic rollback action.

P1-T2 — Write the trial deployment and administration runbook

  • Change:
    • Create docs/forgejo-operations.md with Phase 1-only instructions for DNS confirmation, evaluation, clean-checkout build-before-switch, activation, service/listener/nginx/journal checks, and rollback boundaries for a multi-service host.
    • Document first-admin creation with the deployed forgejo binary, explicit HOME, USER, work/custom paths, --config /var/lib/forgejo/custom/conf/app.ini, --username "oibot", a privately supplied runtime admin_email passed as --email "$admin_email", --admin, and --random-password. Do not record the email in this plan.
    • Require immediate password-manager capture and prohibit operator-selected passwords in argv/history.
    • Document SSH-key registration, creation of a disposable smoke repository, an HTTPS clone/push, and an SSH clone/fetch.
    • State prominently that Phase 1 data is unprotected and should remain disposable.
    • Do not add backup, retention, manifest, recovery, or restore instructions yet.
  • Starts at: New docs/forgejo-operations.md; docs/bitcoin-pulse-operations.md.
  • Depends on: P1-T1.
  • Tests: Protect secret-safe/user-correct administration, exact deployed paths, both clone transports, multi-service rollback safety, and the no-protection warning.
  • Verify:
    • git diff --check -- docs/forgejo-operations.md exits 0.
    • Exact searches confirm the runbook contains git.tobiasostner.de, the sudo -u forgejo env HOME=/var/lib/forgejo USER=forgejo FORGEJO_WORK_DIR=/var/lib/forgejo FORGEJO_CUSTOM=/var/lib/forgejo/custom forgejo --config /var/lib/forgejo/custom/conf/app.ini admin user create --username "oibot" --email "$admin_email" --admin --random-password flow, HTTPS and SSH checks, and the words unprotected and disposable.
    • The Phase 1 document makes no claim that trial data is backed up or recoverable.
  • Risk/recovery: Preserve the generated password immediately; never place it in Git, Nix expressions, logs, or shell history.

P1-T3 — Build the trial without activation

  • Change:
    • Reconfirm DNS and the lock-specific Forgejo version.
    • Track this plan with the Phase 1 implementation commit so the later decision record exists in every clean deployment checkout.
    • Format and validate only Phase 1 Nix files, evaluate the full flake, and build the complete host before switching.
    • On Tobserver, require a clean reviewed checkout and build the host configuration without activation.
    • Before the first switch, require /var/lib/forgejo to be absent or empty; stop for review if residual state exists because migration/reuse is out of scope.
  • Starts at: Completed P1-T1 and P1-T2 changes.
  • Depends on: P1-T1, P1-T2.
  • Tests: Protect DNS/ACME routing, new-empty-instance scope, formatting, full evaluation, and build-before-switch.
  • Verify:
    • test "$(dig +short git.tobiasostner.de A)" = "162.55.103.218" and test -z "$(dig +short git.tobiasostner.de AAAA)" exit 0.
    • nixpkgs-fmt --check flake.nix modules/forgejo.nix, git diff --check, and nix flake check --no-build exit 0.
    • On a compatible Linux builder, nix build .#nixosConfigurations.tobserver.config.system.build.toplevel exits 0.
    • git ls-files --error-unmatch .plans/deploy-forgejo-with-backups.md exits 0 after the reviewed Phase 1 commit, proving the plan is tracked rather than hidden by a cleanliness exception.
    • On Tobserver, test -z "$(git status --porcelain)", git ls-files --error-unmatch .plans/deploy-forgejo-with-backups.md, nix flake check --no-build, sudo test ! -e /var/lib/forgejo || ! sudo find /var/lib/forgejo -mindepth 1 -print -quit | grep -q ., and sudo nixos-rebuild build --flake .#tobserver all exit 0 before any switch.
  • Risk/recovery: A full-build failure blocks activation but is not currently a known baseline failure. Do not bypass a failed build or overwrite unexpected Forgejo state.

P1-T4 — Activate and verify the public trial

  • Change:
    • Switch to the reviewed generation, create the first admin with the documented --admin --random-password flow, and verify that Forgejo reports the account as an administrator.
    • Register an SSH key, create a disposable smoke repository, push a disposable commit, and exercise both HTTPS and host-SSH transports.
    • Verify service health, TLS ingress, closed registration, listener isolation, and rendered nginx behavior.
  • Starts at: docs/forgejo-operations.md; deployed Tobserver checkout.
  • Depends on: P1-T3.
  • Tests: Protect service health, TLS/redirect, proxy configuration, loopback-only backend, closed registration, first-admin login, host OpenSSH integration, and HTTPS/SSH repository use.
  • Verify:
    • sudo systemctl is-active forgejo.service nginx.service reports both active.
    • sudo ss -H -ltn 'sport = :3001' contains 127.0.0.1:3001 and no non-loopback listener; an external connection to port 3001 fails.
    • sudo nginx -t exits 0; sudo nginx -T 2>&1 contains the git.tobiasostner.de server, merge_slashes off, client_max_body_size 512M, and required proxy headers.
    • curl --fail --silent --show-error -H 'Host: git.tobiasostner.de' http://127.0.0.1:3001/api/v1/version | jq -e . exits 0.
    • curl --fail --silent --show-error --location --output /dev/null --write-out '%{http_code} %{url_effective}\n' http://git.tobiasostner.de/ reports 200 at https://git.tobiasostner.de/.
    • sudo -u forgejo env HOME=/var/lib/forgejo USER=forgejo FORGEJO_WORK_DIR=/var/lib/forgejo FORGEJO_CUSTOM=/var/lib/forgejo/custom forgejo --config /var/lib/forgejo/custom/conf/app.ini admin user list --admin | grep -F -- "oibot" exits 0, and the UI offers no registration flow.
    • From an external client, ! nc -z git.tobiasostner.de 3001 succeeds; the documented HTTPS clone/push and forgejo@git.tobiasostner.de:<admin>/clone-smoke.git SSH clone/fetch all exit 0 for the disposable repository.
  • Risk/recovery: If unhealthy, collect Forgejo/nginx journals before changing state. Deploy a reviewed Forgejo-only revert rather than casually switching the multi-service host to an older full generation. Preserve /var/lib/forgejo.

Phase 1 acceptance

  • Forgejo is publicly usable over HTTPS and host SSH, listens only on 127.0.0.1:3001, permits no open registration, exposes no new firewall port, and supports disposable repository use through both transports.
  • No Forgejo backup secret, backup service/timer, staging path, manifest/revision plumbing, VM backup test, backup/recovery instructions, or restore claim has been added.
  • The trial remains unprotected. Its data must remain disposable until Phase 2's first production snapshot passes.

Trial decision gate

G1 — Record KEEP or ABANDON

  • Change:

    • Require the user to record one explicit outcome after Phase 1; do not infer it from elapsed time.
    • Record the decision durably in this plan by replacing PENDING above with exactly KEEP | YYYY-MM-DD | approved by Tobias Ostner or ABANDON | YYYY-MM-DD | approved by Tobias Ostner, then commit that decision before making any Phase 2 change.
    • KEEP: Preserve /var/lib/forgejo in place and authorize P2-T1 through P2-T6. Do not store the only authoritative copy of data in Forgejo until P2-T6 succeeds.
    • ABANDON: Stop this implementation plan and skip every Phase 2 task. Prepare a separate, narrowly reviewed removal plan that handles host-SSH write exclusion, service/ingress removal, state preservation, validation, and any later destruction decision; do not improvise removal from this plan.

  • Starts at: This plan's Trial decision record; deployed Phase 1 instance.

  • Depends on: P1-T4.

  • Tests: Protect explicit owner authorization and prevent Phase 2 from starting on an inferred or undocumented decision.

  • Verify — KEEP: From a clean checkout, run the following before P2-T1 begins; it requires exactly one KEEP record, validates its calendar date, rejects unresolved/ABANDON records, and proves the decision is tracked and committed:

    set -euo pipefail
plan=.plans/deploy-forgejo-with-backups.md
decision_lines="$(rg -x -- '- \*\*Trial decision record:\*\* KEEP \| [0-9]{4}-[0-9]{2}-[0-9]{2} \| approved by Tobias Ostner' "$plan" || true)"
test "$(printf '%s\n' "$decision_lines" | grep -c '^-')" -eq 1
decision_date="$(printf '%s\n' "$decision_lines" | sed -E 's/^.*KEEP \| ([0-9-]+) \|.*$/\1/')"
test "$(date -d "$decision_date" +%F)" = "$decision_date"
! rg -F -- '- **Trial decision record:** PENDING' "$plan"
! rg -F -- '- **Trial decision record:** ABANDON' "$plan"
git ls-files --error-unmatch "$plan"
git diff --quiet -- "$plan"
git diff --cached --quiet -- "$plan"
test -n "$(git log -1 --format=%H -- "$plan")"
sudo test -d /var/lib/forgejo
sudo systemctl is-active forgejo.service

    Expected: every command exits 0 and Forgejo remains active with its state in place.

  • Verify — ABANDON: The decision record contains the dated ABANDON form, no Phase 2 file has been implemented, and a separate removal plan is reviewed before any removal action.

  • Risk/recovery: A broad generation rollback or ad hoc state move can affect unrelated services or race host-SSH writes. Removal is deliberately deferred to a separate plan if ABANDON is chosen.

Phase 2 — Add backups and surrounding infrastructure after KEEP

P2-T1 — Add separable backup configuration, revision wiring, and secret

  • Change:
    • Create modules/forgejo-backup.nix and import it separately from modules/forgejo.nix, so backup infrastructure can be removed without removing Forgejo.
    • Pass self through flake.nix only in this phase and set system.configurationRevision from clean self.rev, allowing dirtyRev only for non-production evaluation. Production backup preparation must reject a missing, unknown, or dirty revision.
    • Declare sops.secrets."restic/forgejo-password" as root:root mode 0400.
    • Add restic.forgejo-password to secrets/tobserver.yaml through SOPS using a high-entropy password retained in the approved password manager.
    • Reuse backblaze/tobcloud-backup-env only for B2 access; never duplicate its plaintext value.
  • Starts at: flake.nix:16-38; new modules/forgejo-backup.nix; configuration.nix:118-127; secrets/tobserver.yaml.
  • Depends on: G1 = KEEP.
  • Tests: Protect explicit KEEP authorization, encrypted Git content, authorized SOPS decryption, declarative permissions, clean revision identity, Restic-password independence, and backup-module separability.
  • Verify:
    • Re-run G1's exact decision-record command before making any Phase 2 change.
    • sops --decrypt secrets/tobserver.yaml >/dev/null exits 0 without printing plaintext.
    • The encrypted diff contains the new key only as an ENC[...] value.
    • nix eval --json '.#nixosConfigurations.tobserver.config.sops.secrets."restic/forgejo-password"' | jq -e '.owner == "root" and .group == "root" and .mode == "0400"' prints true and exits 0; runtime permissions are checked after activation in P2-T6.
  • Risk/recovery: Losing this password makes all Forgejo snapshots unreadable. It must never enter Nix strings, argv, history, logs, or plaintext Git content.

P2-T2 — Implement the complete-state archive and Restic job

  • Change:
    • Configure services.restic.backups.forgejo with initialize = true, repository b2:tobcloud-backup:forgejo, the dedicated password, the shared B2 environment file, and only /var/backup/forgejo/forgejo-state.tar as its Restic path.
    • Create /var/backup/forgejo as root:root mode 0700; keep Restic's generated runtime directory private; create independent tmpfiles-managed /run/forgejo-backup as root:root mode 0700; use UMask = "0077" and a six-hour timeout.
    • Remove stale temporary/final archives before preparation and require forgejo.service active. Keep du/df capacity comparison advisory because Btrfs free-space accounting makes a fixed threshold unreliable.
    • Before claiming ownership, require /run/forgejo-backup/forgejo-stopped to be absent. If it already exists, run only the idempotent recovery path, clear it only after Forgejo is confirmed active, and fail this invocation without archiving; retain the marker and fail if recovery does not succeed.
    • Create /run/forgejo-backup/forgejo-stopped only after clean prechecks, stop Forgejo, and wait with a bounded timeout until no process owned by forgejo remains. If an existing Git-over-SSH process remains, fail before archiving and restart Forgejo through cleanup.
    • Do not stop or reconfigure sshd. Operate the job in an idle/no-push window and disclose the accepted race from a new SSH push starting after the process check.
    • While Forgejo is stopped, create root-level forgejo-backup-manifest.txt with exactly four ordered newline-delimited fields: format_version=1, forgejo_version=<deployed version>, system_path=<readlink -f /run/current-system>, and configuration_revision=<deployed clean flake revision>. Fail if any value is missing or dirty/unknown.
    • Archive all /var/lib/forgejo as forgejo/ plus the manifest into a temporary tar while preserving numeric ownership, ACLs, and extended attributes; atomically rename it to the final staging path.
    • Restart Forgejo, require it active, and clear the marker before allowing Restic upload.
    • Preserve preparation's original status. Its EXIT trap must remove temporary artifacts, remove the final archive when preparation failed, and retry Forgejo restart only while this run owns the marker.
    • Make backupCleanupCommand idempotently retry a marked restart, clear the marker only after Forgejo is confirmed active, retain the marker after terminal restart failure, and unconditionally remove temporary/final staging after the Restic attempt. The next run removes persistent stale staging.
    • Do not configure Requires=forgejo.service; stopping a required unit would cancel the backup.
    • Schedule daily at 04:00 with Persistent = true, up to 30 minutes random delay, and prune 7/4/12.
  • Starts at: New modules/forgejo-backup.nix; modules/bitcoin-pulse.nix:115-174,340-350,386-388.
  • Depends on: P2-T1.
  • Tests: Protect complete state/manifest scope, bounded active-process rejection, stale-artifact prevention, atomic publication, status-preserving restart/cleanup, no stale upload after failure, private staging, exact schedule/retention, and absence of Requires=forgejo.service.
  • Verify:
    • nix eval --raw .#nixosConfigurations.tobserver.config.services.restic.backups.forgejo.repository prints b2:tobcloud-backup:forgejo.
    • nix eval --json .#nixosConfigurations.tobserver.config.services.restic.backups.forgejo.paths | jq -e '. == ["/var/backup/forgejo/forgejo-state.tar"]' prints true.
    • nix eval --json .#nixosConfigurations.tobserver.config.services.restic.backups.forgejo.pruneOpts | jq -e '. == ["--keep-daily 7","--keep-weekly 4","--keep-monthly 12"]' prints true.
    • nix eval --json .#nixosConfigurations.tobserver.config.systemd.services.restic-backups-forgejo.requires | jq -e 'index("forgejo.service") | not' prints true and exits 0.
    • nix eval --json .#nixosConfigurations.tobserver.config.systemd.services.restic-backups-forgejo.serviceConfig | jq -e '.RuntimeDirectoryMode == "0700" and .UMask == "0077" and .TimeoutStartSec == "6h"' prints true and exits 0.
    • Evaluate systemd.tmpfiles.rules and require the exact /run/forgejo-backup 0700 root root rule; inspect the generated preparation/cleanup scripts and require the exact stop/check/archive/restart-before-upload and status-preserving cleanup order.
  • Risk/recovery: A newly racing SSH push can make a snapshot inconsistent; this is accepted. Any process, archive, restart, or upload failure must fail visibly without uploading stale content. Removing the separate backup module must leave Forgejo and /var/lib/forgejo intact.

P2-T3 — Add a disposable backup-mechanism VM check

  • Change:

    • Factor orchestration generation into modules/forgejo-backup-script.nix. Permit build-time-only phase hooks for the VM check; production must instantiate it without hooks, production options, or runtime fault-injection variables.

    • Add tests/forgejo-backup.nix and expose checks.x86_64-linux.forgejo-backup with a local Restic repository.

    • Prove normal ordering and deterministically exercise pre-stop rejection, a lingering Forgejo-owned process, archive failure, one-shot restart failure recovered by cleanup, terminal restart failure with retained marker, Restic failure after restart, forced backup-unit termination, and stale staging on the next run.

    • Assert this expected outcome matrix; empty means no entry of any type remains in staging:

      Case Backup unit Forgejo Marker Staging Snapshot sshd
      Normal success active absent empty +1 active
      Pre-stop rejection failed active absent empty unchanged active
      Lingering Forgejo process failed active after cleanup absent empty unchanged active
      Archive failure failed active after cleanup absent empty unchanged active
      One-shot restart failure failed active after cleanup retry absent empty unchanged active
      Terminal restart failure failed failed/inactive retained empty unchanged active
      Restic failure after restart failed active absent empty unchanged active
      Forced unit termination failed active after cleanup absent empty unchanged active
      Pre-existing retained marker failed after recovery-only path active after remediation absent empty unchanged active
      Next run with stale staging success active absent empty +1 only active

    • After the terminal restart case, exercise explicit operator remediation that starts Forgejo, confirms it active, and clears the retained marker before continuing.

    • Do not extract/restore a backup or claim protection from a newly racing SSH push.

  • Starts at: New modules/forgejo-backup-script.nix; new tests/forgejo-backup.nix; flake.nix; P2-T2 behavior.
  • Depends on: P2-T2.
  • Tests: Protect destructive failure paths and ordering that static evaluation cannot establish.
  • Verify: nix build .#checks.x86_64-linux.forgejo-backup exits 0 and the log shows every deterministic success/failure case completed; production evaluation contains no test hook.
  • Risk/recovery: Fault injection must remain build-time-only. A failing case blocks backup activation; never reproduce destructive test failures against trial data.

P2-T4 — Expand the runbook with backup and recovery operations

  • Change:
    • Expand docs/forgejo-operations.md without weakening its Phase 1 deployment/admin instructions.
    • Add advisory capacity inspection, an idle/no-push backup window, manual backup invocation, snapshot freshness, service restart and timer checks, rendered-unit inspection, private streamed archive/manifest inspection, staging cleanup, and shared B2 credential rotation.
    • Update the canonical shared-credential section in docs/bitcoin-pulse-operations.md so rotation names and validates Bitcoin Pulse, Nextcloud, and Forgejo rather than only the existing two jobs.
    • Document recovery using the manifest's Forgejo version, system path, clean configuration revision, and matching committed flake/lock. Enter a maintenance boundary that denies only the forgejo SSH account without stopping sshd; stop Forgejo; wait for zero Forgejo-owned processes; restore privately; retain current state under a timestamped rollback path; extract with ownership/ACLs/xattrs; start and verify Forgejo; and restore Forgejo SSH access only after acceptance or rollback. Retain rollback data and the source snapshot until acceptance.
    • Require preservation or reproducible rebuilding of the matching generation before restoration.
    • State that content inspection is not a restore test, recoverability remains unproven, sshd stays active as a service, and the overlapping-push race remains accepted for backup creation only.
  • Starts at: docs/forgejo-operations.md; docs/bitcoin-pulse-operations.md:191-222; P2-T2 paths and units.
  • Depends on: P2-T2, P2-T3.
  • Tests: Protect exact paths/units, complete recovery scope, version-compatible startup, reversible recovery, credential-rotation scope, no-push instructions, and honest claims.
  • Verify:
    • git diff --check -- docs/forgejo-operations.md docs/bitcoin-pulse-operations.md exits 0.
    • Exact searches confirm the Forgejo runbook names restic-backups-forgejo.service, forgejo-backup-manifest.txt, the no-push window, snapshot freshness, manifest inspection, Recovery, Rollback, Forgejo-only SSH denial during recovery, and the unproven-recoverability warning.
    • Exact searches in docs/bitcoin-pulse-operations.md require all three unit names: restic-backups-bitcoin-pulse.service, restic-backups-nextcloud.service, and restic-backups-forgejo.service.
  • Risk/recovery: A newer Forgejo can migrate SQLite and make downgrade unsafe. Recovery must identify/rebuild the matching generation and preserve old/current state and the Restic snapshot until acceptance.

P2-T5 — Build Phase 2 without activation

  • Change:
    • Format all Phase 2 Nix files, run the backup VM check, evaluate the full flake, and build the complete host.
    • On Tobserver, require a clean reviewed checkout, verify system.configurationRevision equals the deployment commit, and build without switching.
    • Confirm Forgejo remains active and /var/lib/forgejo remains intact; Phase 2 augments the trial instance rather than reinitializing it.
  • Starts at: Completed P2-T1 through P2-T4 changes.
  • Depends on: P2-T1, P2-T2, P2-T3, P2-T4.
  • Tests: Protect formatting, VM failure behavior, clean revision identity, full evaluation/build, and trial-state preservation.
  • Verify:
    • nixpkgs-fmt --check flake.nix modules/forgejo.nix modules/forgejo-backup.nix modules/forgejo-backup-script.nix tests/forgejo-backup.nix, git diff --check, nix build .#checks.x86_64-linux.forgejo-backup, and nix flake check --no-build all exit 0.
    • On a compatible Linux builder, nix build .#nixosConfigurations.tobserver.config.system.build.toplevel exits 0.
    • On Tobserver, re-run G1's exact KEEP-record command, then require test -z "$(git status --porcelain)", test "$(nix eval --raw .#nixosConfigurations.tobserver.config.system.configurationRevision)" = "$(git rev-parse HEAD)", sudo systemctl is-active forgejo.service, sudo test -d /var/lib/forgejo, and sudo nixos-rebuild build --flake .#tobserver to exit 0 without activation.
  • Risk/recovery: A failed check/build blocks backup activation but does not invalidate the running trial. Do not deploy from a dirty revision or bypass the VM check.

P2-T6 — Activate backups and inspect the first production snapshot

  • Change:
    • Switch to the already-built Phase 2 generation.
    • Record Forgejo's activation timestamp and latest snapshot ID, manually run the backup, and prove Forgejo restarted before upload and a new snapshot was created.
    • Verify job result, timer, rendered unit, stored tar path, required tar entries and exact manifest, Forgejo activity, marker absence, and local staging cleanup.
    • Inspect redacted journals for orderly stop/process-check/archive/restart/upload behavior. Do not extract the archive or describe these checks as recovery validation.
  • Starts at: Deployed restic-backups-forgejo.service; expanded docs/forgejo-operations.md.
  • Depends on: P2-T5.
  • Tests: Protect fresh encrypted upload, complete content, restart-before-upload, cleanup, timer/retention activation, and honest reporting.
  • Verify:
    • Capture before_service=$(systemctl show forgejo.service --property=ActiveEnterTimestampMonotonic --value) and before_snapshot=$(sudo restic-forgejo snapshots --json --latest 1 2>/dev/null | jq -r '.[0].id // empty'); run sudo systemctl start restic-backups-forgejo.service.
    • systemctl show restic-backups-forgejo.service --property=Result --property=ExecMainStatus reports Result=success and ExecMainStatus=0; Forgejo is active; its activation timestamp increased; /run/forgejo-backup/forgejo-stopped is absent.
    • Capture after_snapshot=$(sudo restic-forgejo snapshots --json --latest 1 | jq -er '.[0].id'); require it to be non-empty and different from $before_snapshot.
    • sudo restic-forgejo ls "$after_snapshot" | grep -Fx '/var/backup/forgejo/forgejo-state.tar' succeeds.
    • Under set -o pipefail, privately stream the tar listing and require forgejo/data/forgejo.db, forgejo/repositories/, forgejo/custom/conf/, forgejo/.ssh/, and forgejo-backup-manifest.txt. Privately extract only the manifest and compare its exact four ordered lines, with no extras, to the deployed package version, readlink -f /run/current-system, and nixos-version --configuration-revision.
    • sudo stat --format='%U %G %a' /run/secrets/restic/forgejo-password prints root root 400.
    • test -z "$(sudo find /var/backup/forgejo -mindepth 1 -print -quit)" exits 0, proving no file, directory, or symlink remains. systemctl is-enabled restic-backups-forgejo.timer prints enabled; systemctl show restic-backups-forgejo.timer --property=TimersCalendar --property=Persistent --property=RandomizedDelayUSec reports daily 04:00, Persistent=yes, and 30 minutes randomized delay.
  • Risk/recovery: Upload/content inspection does not prove restoration or close the SSH race. On failure, first ensure Forgejo is active, inspect the marker and redacted journals, and revert only the separate backup module if necessary. The deployment remains unprotected until this task fully passes.

Phase 2 acceptance

  • The dedicated Restic password is encrypted in SOPS and deployed root-only; the shared B2 credential is reused without duplication.
  • The job stops Forgejo, rejects remaining Forgejo-owned processes, archives all /var/lib/forgejo plus the exact version/system/revision manifest, restarts Forgejo before upload, and cleans staging.
  • The daily 04:00 persistent timer, up to 30 minutes random delay, and 7/4/12 retention are active.
  • The build-only VM check passes all approved failure paths and cannot inject faults into production.
  • The runbook matches deployed paths and clearly states that restore testing was not performed.
  • The first production snapshot is fresh and its archive/manifest content has been inspected successfully.
  • Only after these checks pass is the retained deployment considered backup-protected. Recoverability remains unproven because restore tests are excluded.

Final acceptance

  • Checks: Phase 1 passes DNS revalidation, formatting, full evaluation/build, empty-state, service, TLS, registration, listener, admin, and disposable HTTPS/SSH repository checks without adding backup infrastructure. Exactly one dated trial decision is then recorded in this plan. KEEP authorizes Phase 2 and preserves state; ABANDON stops this plan and skips Phase 2 pending a separate removal plan. After KEEP, the VM check, full build, first production backup, snapshot freshness/content, manifest, timer, retention, restart, and cleanup checks all pass.
  • End state: Before the gate, a minimal but explicitly unprotected Forgejo trial is available. After KEEP and Phase 2, the same retained instance has encrypted daily off-server backups and documented recovery/rollback procedures. After ABANDON, no backup infrastructure is added and removal is handled separately.
  • Deferrals or blockers: Phase 1 has no known external blocker beyond its normal pre-activation checks. Phase 2 is intentionally blocked on the future KEEP decision. Restore testing and alerting remain deferred; sshd remains available during backup, leaving the accepted overlapping-push consistency race.