Forgejo trial operations

This runbook covers the initial Forgejo trial on Tobserver. The complete service, SQLite, package, and nginx configuration lives in modules/forgejo.nix.

Warning: This trial is unprotected. It has no Forgejo backup job, and its data must be treated as disposable. Keep the authoritative copy of anything important somewhere else.

Run deployment commands from the Tobserver checkout unless noted otherwise. Never place passwords or other credentials in Git, Nix expressions, command arguments, shell history, logs, or issue comments.

Pre-deployment checks

Confirm that the dedicated hostname points only to Tobserver:

test "$(dig +short git.tobiasostner.de A)" = "162.55.103.218"
test -z "$(dig +short git.tobiasostner.de AAAA)"

On a workstation, evaluate the configuration and build it on a compatible Linux builder when one is available:

nix flake check --no-build
nix build .#nixosConfigurations.tobserver.config.system.build.toplevel

Commit and push the reviewed changes. On Tobserver, update a clean checkout and build without activating it:

cd ~/tobserver
test -z "$(git status --porcelain)" || {
  echo "Refusing to deploy from a dirty checkout" >&2
  exit 1
}
git pull --ff-only
git rev-parse HEAD
nix flake check --no-build

if sudo test -e /var/lib/forgejo \
  && sudo find /var/lib/forgejo -mindepth 1 -print -quit | grep -q .; then
  echo "Refusing to reuse non-empty Forgejo state" >&2
  exit 1
fi

sudo nixos-rebuild build --flake .#tobserver

Compare the printed commit with the reviewed deployment commit. Do not continue if evaluation, the full host build, or the empty-state check fails.

Activation and service checks

Activate the configuration only after reviewing the successful build. Immediately before switching, recheck that the checkout is clean and still at the reviewed commit printed during the build step:

read -r -p 'Reviewed commit hash: ' reviewed_commit
test -z "$(git status --porcelain)" || {
  echo "Refusing to switch from a dirty checkout" >&2
  exit 1
}
current_commit="$(git rev-parse HEAD)"
printf 'Switching commit: %s\n' "$current_commit"
test "$current_commit" = "$reviewed_commit" || {
  echo "Refusing to switch an unreviewed commit" >&2
  exit 1
}
sudo nixos-rebuild switch --flake .#tobserver
unset current_commit reviewed_commit

Verify the services, loopback-only backend, local API, rendered nginx configuration, and public HTTPS endpoint:

sudo systemctl is-active forgejo.service nginx.service
sudo ss -H -ltn 'sport = :3001'

curl --fail --silent --show-error \
  -H 'Host: git.tobiasostner.de' \
  http://127.0.0.1:3001/api/v1/version \
  | jq -e .

sudo nginx -t
(
  set -euo pipefail
  nginx_config="$(sudo nginx -T 2>&1)"
  for required_line in \
    'git.tobiasostner.de' \
    'merge_slashes off' \
    'client_max_body_size 512M' \
    'proxy_set_header Host $host' \
    'proxy_set_header X-Real-IP $remote_addr' \
    'proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for' \
    'proxy_set_header X-Forwarded-Proto $scheme' \
    'proxy_set_header Connection $http_connection' \
    'proxy_set_header Upgrade $http_upgrade'
  do
    printf '%s\n' "$nginx_config" | grep -F -- "$required_line" >/dev/null
  done
)

curl --fail --silent --show-error --location \
  --output /dev/null \
  --write-out '%{http_code} %{url_effective}\n' \
  http://git.tobiasostner.de/

The listener output must contain 127.0.0.1:3001 and no non-loopback address. The public request must report HTTP 200 at https://git.tobiasostner.de/. Port 3001 must not be reachable from an external machine:

! nc -z git.tobiasostner.de 3001

If a check fails, inspect logs before restarting or changing state:

sudo journalctl \
  -u forgejo.service \
  -u nginx.service \
  --since '-15 minutes' \
  --no-pager

Create the first administrator

Run this once in an interactive shell on Tobserver. Enter the administrator email when prompted; it stays in the shell variable and is not recorded in this runbook or shell history.

read -r -p 'Admin email: ' admin_email
sudo -u forgejo env \
  HOME=/var/lib/forgejo \
  USER=forgejo \
  FORGEJO_WORK_DIR=/var/lib/forgejo \
  FORGEJO_CUSTOM=/var/lib/forgejo/custom \
  forgejo --config /var/lib/forgejo/custom/conf/app.ini \
  admin user create \
  --username "oibot" \
  --email "$admin_email" \
  --admin \
  --random-password
unset admin_email

The command prints a generated password once. Copy it directly into the approved password manager immediately, then clear the terminal. Do not redirect, pipe, log, or save the output in a temporary file, and do not choose a password on the command line.

Confirm that the account is an administrator:

sudo -u forgejo env \
  HOME=/var/lib/forgejo \
  USER=forgejo \
  FORGEJO_WORK_DIR=/var/lib/forgejo \
  FORGEJO_CUSTOM=/var/lib/forgejo/custom \
  forgejo --config /var/lib/forgejo/custom/conf/app.ini \
  admin user list --admin \
  | grep -F -- 'oibot'

Disposable HTTPS and SSH smoke test

In the Forgejo interface:

Add the external client's public SSH key to the oibot account.
Create an empty repository named clone-smoke.

From an external client, exercise HTTPS clone and push with a disposable commit:

workdir="$(mktemp -d)"
git clone https://git.tobiasostner.de/oibot/clone-smoke.git \
  "$workdir/clone-smoke-https"
cd "$workdir/clone-smoke-https"
printf '%s\n' 'Forgejo trial' > README.md
git add README.md
git commit -m 'Test Forgejo trial'
git push origin HEAD:main

Then exercise host OpenSSH clone and fetch:

git clone forgejo@git.tobiasostner.de:oibot/clone-smoke.git \
  "$workdir/clone-smoke-ssh"
git -C "$workdir/clone-smoke-ssh" fetch origin
rm -rf -- "$workdir"

All clone, push, and fetch commands must exit successfully. The repository and commit are disposable trial data.

Trial rollback boundary

Tobserver hosts unrelated services, so do not use a broad live NixOS generation rollback as a routine Forgejo test. If the service is unhealthy, collect its status and journals first. Prepare and review a narrow change that removes only the modules/forgejo.nix import and its Forgejo service and nginx virtual host. Preserve /var/lib/forgejo; never delete trial state as an automatic rollback action.

6.4 KiB Raw Permalink Blame History