Job search

A Clojure application for collecting software-engineering job postings. The goal is to gather both European jobs and fully remote jobs, then filter and rank them later.

The CLI collects Himalayas and Arbeitnow postings into SQLite with normalization, deduplication, and collection-run tracking. Search and expiration handling are still to come.

Development

Enter the pinned Java 21 development environment with direnv:

direnv allow

Or use an explicit shell:

nix develop

The shell provides Clojure CLI, Babashka, clj-kondo, clojure-lsp, zprint, Git, Java, and the sqlite3 CLI. .envrc also loads an optional ignored .env file. No credentials are needed. .env.example documents database, source-configuration, and build overrides.

Run these commands from the project root:

bb format
bb lint
bb test
bb run -- help
bb db:pending
bb db:migrate
bb build

bb run help also works. The task forwards CLI exit codes. Help, an empty argument list, and successful commands return 0. Invalid arguments return 2; operational failures return 1. Errors and failed collection summaries go to stderr.

Tests use clojure.test and the Cognitect test runner. They do not contact job APIs or touch the configured development database. SQLite tests use temporary files and remove them after closing connections. They cover migrations, persistence, idempotency, run transitions, transactional rollback, runtime cleanup, search pagination, retries and limits, normalization, and source-payload preservation.

Runtime profiles and lifecycle

Select a profile before the command:

bb run -- --profile dev collect himalayas
bb run -- --profile prod collect himalayas

Omitting --profile selects dev, so existing commands still work. An invalid or missing profile value returns exit code 2. Production invocations should select prod explicitly. Migrate the selected database before collecting. Collection fetches the configured scope and then saves its unique postings.

job-search.runtime/read-config reads resources/config.edn with Aero. The system contains one SQLite connection, owned and closed by Integrant. Configuration supports #profile and #ig/ref.

Help and argument errors bypass configuration and system initialization. Database and collection commands initialize the system, execute the command, and halt the system before printing the result. Collection rejects pending migrations before starting a run or contacting the source; it does not migrate automatically. A failed startup halts components that already initialized. A failed command also triggers cleanup. Runtime exceptions return exit code 1.

Each component must release its own partially allocated resources if its initializer throws before returning them to Integrant. Halting follows Integrant's reverse dependency order. A halt failure is reported rather than treated as success; if execution already failed, the cleanup failure is attached to that exception as suppressed information. Component halt functions should release their resources reliably because Integrant stops traversal when one throws. Thread interruption during collection triggers best-effort run finalization. Abrupt process termination can leave a run marked running; no signal handler or automatic stale-run recovery is implemented.

Database

The default paths are data/jobs.db for dev and /var/lib/job-collector/jobs.db for prod. JOBS_DB_PATH overrides either profile. Parent directories are created when the connection opens, so production needs permission to write its configured directory.

bb db:pending
bb db:migrate
bb run -- --profile prod db migrate

Migrations do not run automatically at startup. db pending does not apply application migrations, but Migratus may create its metadata table and the database file. Repeating db migrate is safe. There is no destructive rollback CLI.

SQLite uses WAL, foreign keys, and a 5000 ms busy timeout. Override the timeout with JOBS_DB_BUSY_TIMEOUT_MS. There is one connection per system, without a connection pool; use it serially.

See database details for the schema, repository API, run tracking, and backup considerations.

REPL workflow

For Emacs/CIDER, .dir-locals.el selects Clojure CLI with the :dev alias. Alternatively, start clojure -M:dev. The user namespace provides:

(start)
integrant.repl.state/system
(stop)
(restart)

Run bb db:migrate before using the repository. The REPL uses the dev profile; start opens the connection without applying migrations. restart refreshes namespaces under src/ and dev/ through integrant.repl/reset, then resumes the system with fresh configuration. No system starts merely by loading the namespace.

job-search.core/run-cli returns an exit code without terminating the REPL. It owns a separate short-lived system and does not stop or reuse integrant.repl.state/system. Do not call -main from the REPL because it exits the JVM.

Collect Himalayas jobs

These commands use the development database and make live API requests:

bb db:migrate
bb run -- collect himalayas

The initial scope uses four broad role queries for candidates in Germany, including worldwide jobs: software engineer, software developer, DevOps engineer, and engineering manager. Requests are serial with a one-second default delay. The adapter completes every query before persistence begins. It deduplicates overlapping results and fails on inconsistent pagination or exceeded budgets. Non-empty short pages produce warnings and do not end pagination; all reported pages are still requested. The normalizer preserves raw payloads and full descriptions.

Collection records its scope and run status, saves postings, and prints counts and aggregated normalization warnings. Repeated collections update existing postings rather than duplicate them. A 24-hour guard skips repeat attempts in the same database without fetching or creating another run, including after a failed attempt. Use bb run -- collect himalayas --force only for an intentional rerun; it cannot bypass an existing running owner. A fetch failure saves no postings; a later normalization or database failure retains completed saves and marks the run failed where possible. Missing or expired postings are not closed in this increment.

JOBS_HIMALAYAS_CONFIG can point to a complete Aero configuration file copied from resources/sources/himalayas.edn. Otherwise the CLI uses the packaged defaults. Existing external configuration files keep their own query list and delay, so update them too if needed. Keep personal preferences out of these source filters.

The confirmed salary preference is at least €60,000 gross annually, using the advertised range's minimum; undisclosed salaries remain eligible. This is recorded for future local search/ranking and is not applied during collection.

See collection behavior and adapter configuration and REPL usage. The adapter itself remains independent of SQLite.

Collect Arbeitnow jobs

After migrating the selected database, run bb run -- collect arbeitnow (or use --profile prod). This is a separate bounded source run with its own 24-hour guard; --force bypasses only its cooldown, not a running owner. JOBS_ARBEITNOW_CONFIG can point to a complete configuration copied from resources/sources/arbeitnow.edn. The default walks up to 20 pages (5000 unique jobs, 30 attempts, five minutes of fetch time) at 1.5-second spacing. The API offers a general job-board feed, not an engineering-only scope: these bounds may fail rather than silently truncate the feed. Arbeitnow remote does not establish worldwide eligibility. Collection stores raw objects privately in the local database for re-normalization; vendor retention rights and coverage/terms are not verified. Review the source terms before using collected data outside personal local use. No relevance filters or missing-record closures are applied. See collection behavior.

Search stored jobs

After upgrading this checkout, apply the new FTS5 migration to your existing database once:

bb db:pending
bb db:migrate
bb run -- search "clojure"
bb run -- search "ios swift" --page 2
bb run -- show 123

The migration indexes existing postings locally; it does not collect jobs or change run counters. Replace 123 with a local ID printed by search. search returns 20 active rows per page, ordered by full-text relevance then local ID. It searches title, company, and plain-text description. Words are matched literally and combined with AND; punctuation separates words, so C++ currently searches C, not the programming language. Empty or punctuation-only queries are rejected. Pagination is capped at page 10000. There is no salary or location preference filter yet, and an active posting may have expired because expiration handling is deferred.

show prints the full plain-text description, original location labels, normalized location restrictions, salary units, dates, and source and application URLs. It does not render source HTML. Neither command contacts job APIs. Use the same profile and JOBS_DB_PATH used for collection.

Local uberjar

bb build uses tools.build to create:

target/job-search-<version>.jar

The version comes from VERSION, the current short Git revision, or dev when no revision is available. The jar includes its main class and dependencies:

java -jar target/job-search-dev.jar help

Use the actual filename printed by the build. A local build may download Maven and Git dependencies on its first run.

Nix package

Build and verify the package:

nix build
nix flake check
./result/bin/jobs help

Or run it directly:

nix run . -- help

The package uses the same build.clj as bb build. Its build and tests use content-addressed dependencies from deps-lock.json, without fetching dependencies during compilation. Nix also normalizes jar timestamps so repeated builds produce identical output. Use nix build --rebuild to check reproducibility. The installed jobs command includes a Nix-managed Java runtime and does not need Clojure CLI or a local Maven cache.

The flake supports Apple Silicon macOS, x86-64 Linux, and ARM64 Linux. nix flake check verifies the current platform. Other platforms require a corresponding builder. There is no NixOS module or timer yet.

After changing deps.edn, regenerate the dependency lock:

nix run .#deps-lock
nix build
nix flake check

flake.lock starts from BitcoinPulse's pinned Nix inputs. Update Nix inputs separately with nix flake update, then rebuild and review the lock changes.

If this directory is placed under Git, new files must be added to Git before Nix's Git-based flake source can see them. No commit is needed to build staged files.

Next steps

• Overall design
• First collector milestone

Next are local salary/location filtering and explicit expiration handling. Collection remains a configured, broad software-engineering scope, without personal preference filtering or a full-feed mirror.