Tobserver/docs/forgejo-operations.md

# Forgejo trial operations

This runbook covers the initial Forgejo trial on Tobserver. The complete
service, SQLite, package, and nginx configuration lives in
`modules/forgejo.nix`.

> **Warning:** This trial is unprotected. It has no Forgejo backup job, and its
> data must be treated as disposable. Keep the authoritative copy of anything
> important somewhere else.

Run deployment commands from the Tobserver checkout unless noted otherwise.
Never place passwords or other credentials in Git, Nix expressions, command
arguments, shell history, logs, or issue comments.

## Pre-deployment checks

Confirm that the dedicated hostname points only to Tobserver:

```console
test "$(dig +short git.tobiasostner.de A)" = "162.55.103.218"
test -z "$(dig +short git.tobiasostner.de AAAA)"
```

On a workstation, evaluate the configuration and build it on a compatible Linux
builder when one is available:

```console
nix flake check --no-build
nix build .#nixosConfigurations.tobserver.config.system.build.toplevel
```

Commit and push the reviewed changes. On Tobserver, update a clean checkout and
build without activating it:

```console
cd ~/tobserver
test -z "$(git status --porcelain)" || {
  echo "Refusing to deploy from a dirty checkout" >&2
  exit 1
}
git pull --ff-only
git rev-parse HEAD
nix flake check --no-build

if sudo test -e /var/lib/forgejo \
  && sudo find /var/lib/forgejo -mindepth 1 -print -quit | grep -q .; then
  echo "Refusing to reuse non-empty Forgejo state" >&2
  exit 1
fi

sudo nixos-rebuild build --flake .#tobserver
```

Compare the printed commit with the reviewed deployment commit. Do not continue
if evaluation, the full host build, or the empty-state check fails.

## Activation and service checks

Activate the configuration only after reviewing the successful build. Immediately
before switching, recheck that the checkout is clean and still at the reviewed
commit printed during the build step:

```console
read -r -p 'Reviewed commit hash: ' reviewed_commit
test -z "$(git status --porcelain)" || {
  echo "Refusing to switch from a dirty checkout" >&2
  exit 1
}
current_commit="$(git rev-parse HEAD)"
printf 'Switching commit: %s\n' "$current_commit"
test "$current_commit" = "$reviewed_commit" || {
  echo "Refusing to switch an unreviewed commit" >&2
  exit 1
}
sudo nixos-rebuild switch --flake .#tobserver
unset current_commit reviewed_commit
```

Verify the services, loopback-only backend, local API, rendered nginx
configuration, and public HTTPS endpoint:

```console
sudo systemctl is-active forgejo.service nginx.service
sudo ss -H -ltn 'sport = :3001'

curl --fail --silent --show-error \
  -H 'Host: git.tobiasostner.de' \
  http://127.0.0.1:3001/api/v1/version \
  | jq -e .

sudo nginx -t
(
  set -euo pipefail
  nginx_config="$(sudo nginx -T 2>&1)"
  for required_line in \
    'git.tobiasostner.de' \
    'merge_slashes off' \
    'client_max_body_size 512M' \
    'proxy_set_header Host $host' \
    'proxy_set_header X-Real-IP $remote_addr' \
    'proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for' \
    'proxy_set_header X-Forwarded-Proto $scheme' \
    'proxy_set_header Connection $http_connection' \
    'proxy_set_header Upgrade $http_upgrade'
  do
    printf '%s\n' "$nginx_config" | grep -F -- "$required_line" >/dev/null
  done
)

curl --fail --silent --show-error --location \
  --output /dev/null \
  --write-out '%{http_code} %{url_effective}\n' \
  http://git.tobiasostner.de/
```

The listener output must contain `127.0.0.1:3001` and no non-loopback address.
The public request must report HTTP 200 at `https://git.tobiasostner.de/`.
Port 3001 must not be reachable from an external machine:

```console
! nc -z git.tobiasostner.de 3001
```

If a check fails, inspect logs before restarting or changing state:

```console
sudo journalctl \
  -u forgejo.service \
  -u nginx.service \
  --since '-15 minutes' \
  --no-pager
```

## Create the first administrator

Run this once in an interactive shell on Tobserver. Enter the administrator
email when prompted; it stays in the shell variable and is not recorded in this
runbook or shell history.

```console
read -r -p 'Admin email: ' admin_email
sudo -u forgejo env \
  HOME=/var/lib/forgejo \
  USER=forgejo \
  FORGEJO_WORK_DIR=/var/lib/forgejo \
  FORGEJO_CUSTOM=/var/lib/forgejo/custom \
  forgejo --config /var/lib/forgejo/custom/conf/app.ini \
  admin user create \
  --username "oibot" \
  --email "$admin_email" \
  --admin \
  --random-password
unset admin_email
```

The command prints a generated password once. Copy it directly into the
approved password manager immediately, then clear the terminal. Do not redirect,
pipe, log, or save the output in a temporary file, and do not choose a password
on the command line.

Confirm that the account is an administrator:

```console
sudo -u forgejo env \
  HOME=/var/lib/forgejo \
  USER=forgejo \
  FORGEJO_WORK_DIR=/var/lib/forgejo \
  FORGEJO_CUSTOM=/var/lib/forgejo/custom \
  forgejo --config /var/lib/forgejo/custom/conf/app.ini \
  admin user list --admin \
  | grep -F -- 'oibot'
```

Sign in at `https://git.tobiasostner.de/` and confirm that the interface does
not offer public registration.

## Disposable HTTPS and SSH smoke test

In the Forgejo interface:

1. Add the external client's public SSH key to the `oibot` account.
2. Create an empty repository named `clone-smoke`.

From an external client, exercise HTTPS clone and push with a disposable commit:

```console
workdir="$(mktemp -d)"
git clone https://git.tobiasostner.de/oibot/clone-smoke.git \
  "$workdir/clone-smoke-https"
cd "$workdir/clone-smoke-https"
printf '%s\n' 'Forgejo trial' > README.md
git add README.md
git commit -m 'Test Forgejo trial'
git push origin HEAD:main
```

Then exercise host OpenSSH clone and fetch:

```console
git clone forgejo@git.tobiasostner.de:oibot/clone-smoke.git \
  "$workdir/clone-smoke-ssh"
git -C "$workdir/clone-smoke-ssh" fetch origin
rm -rf -- "$workdir"
```

All clone, push, and fetch commands must exit successfully. The repository and
commit are disposable trial data.

## Trial rollback boundary

Tobserver hosts unrelated services, so do not use a broad live NixOS generation
rollback as a routine Forgejo test. If the service is unhealthy, collect its
status and journals first. Prepare and review a narrow change that removes only
the `modules/forgejo.nix` import and its Forgejo service and nginx virtual host.
Preserve `/var/lib/forgejo`; never delete trial state as an automatic rollback
action.