Bitcoin Pulse backend

Clojure backend for Bitcoin Pulse. The JVM process runs directly on the host and uses PostgreSQL 18 in a rootless Podman container.

Backend modules

• collection/ acquires datasets and writes shared storage, without feature callbacks or HTTP dependencies. Its shared scheduler gives each collector a separate single-thread daemon executor. Spot prices, hourly market summaries, and network collectors run immediately and then with millisecond fixed delay; daily market history runs on startup and then at 00:20 UTC. Halt interrupts cancellation, shuts down the executor, and waits up to five seconds before warning.
• providers/ owns the existing domain protocols, live/fake adapters, and upstream HTTP options. data/ owns price observations, repository and deterministic fixture generation, plus network caches and shared snapshot schemas. Neutral config validates against those data schemas, never feature schemas.
• features/ owns Price, Price Detail, Block Height, Difficulty Adjustment, and Halving. Each exposes its handler and routes; Price and Halving own their response schemas and calculations. The simple network routes use shared data schemas directly. Block Height and Halving consume the same cache and freshness policy, without another collector or cache.
• Price's composition service coordinates the existing change/history services on each request, with no response cache. Collection does not know which feature consumes its data. Existing short chart history is not a Price Detail dataset.
• Root routes composes health and feature routes; http/contracts contains only shared health/error schemas. core and http remain runtime composition entry points. Operational db.seed uses core configuration but initializes only datasource/repository; it is deliberately outside data/. db.migrations remains the migration entry point.

The architecture tests check transitive project-local namespace dependencies, including paths through neutral helpers. A separate bounded fresh-JVM test runs all five collectors with real caches and deterministic doubles, verifies Integrant init/halt, and checks that no feature or HTTP composition namespaces load.

Development

Create an ignored local configuration and allow direnv to enter the Nix development shell:

cp .env.example .env
# Replace every placeholder password in .env.
direnv allow

.envrc loads .env into the local process environment. Without direnv, run nix develop and export the same variables before using the application or database tasks.

On macOS, initialize Podman's Linux VM once and start it when needed:

podman machine init # once only
podman machine start

Start PostgreSQL, apply pending migrations, seed a deterministic rolling price history, and run the application with continuously updating fake USD/EUR prices:

bb dev

This default workflow uses deterministic in-process price and network data. It does not contact CoinGecko or Mempool. To skip fake seeding and run against the real CoinGecko and Mempool APIs instead, use:

bb dev:live

Fake and live spot observations share the production-compatible coingecko source identity. New daily market-history fixtures instead use source fake, separate from live coingecko history. Use bb db:reset before live development when fake history must not mix with real history. PostgreSQL remains running when the application stops. On macOS, podman machine start is still a separate host-specific step.

Useful tasks:

bb db:logs       # follow PostgreSQL logs
bb db:pending    # show pending migrations
bb db:rollback   # roll back the latest migration when safe
bb db:seed       # seed an already migrated database with fake price history
bb db:down       # stop PostgreSQL; preserve the named volume
bb db:reset      # explicitly and destructively replace development data
bb test:unit     # container-free tests
bb test          # complete suite with an isolated temporary PostgreSQL

Application and Integrant/REPL restarts recreate HikariCP, not PostgreSQL. Development data survives db:down and db:up; only db:reset deletes its volume.

Nix package

The flake builds a reproducible uberjar from content-addressed dependencies and installs it with a Nix-managed headless Java 21 runtime. The package has two entry points:

Command	Purpose
bitcoin-pulse <dev|dev-live|prod>	Start the backend with an explicit Aero runtime profile. It reads the application environment documented below and in docs/database.md.
bitcoin-pulse-migrate [migrate|pending|rollback]	Run Migratus with MIGRATUS_DATABASE_URL, MIGRATUS_USERNAME, and MIGRATUS_PASSWORD or MIGRATUS_PASSWORD_FILE.

Build and check the package on the current system:

nix build
nix flake check
./result/bin/bitcoin-pulse prod
./result/bin/bitcoin-pulse-migrate pending

The production output is packages.x86_64-linux.default. Verifying it from a non-Linux workstation requires a configured Linux builder:

nix build .#packages.x86_64-linux.default
nix build .#checks.x86_64-linux.entry-points

The package version is the pinned source revision, or the dirty revision for a local checkout. Maven and Git dependencies are fetched as content-addressed Nix inputs from the committed deps-lock.json; the package build itself does not access the network and the installed application does not need Clojure CLI or a Maven cache.

Whenever deps.edn changes, regenerate and review the dependency lock with the flake-pinned clj-nix tool:

nix run .#deps-lock
nix flake check

Update the Nix inputs separately and rebuild before committing the resulting flake.lock:

nix flake update
nix flake check

A consuming Tobserver flake can pin this repository and select its package without keeping a mutable application checkout:

{
  inputs.bitcoin-pulse.url =
    "git+https://codeberg.org/oibot/BitcoinPulseAPI.git";

  # In the NixOS configuration:
  # inputs.bitcoin-pulse.packages.${pkgs.system}.default
}

Run nix flake update bitcoin-pulse in the Tobserver repository to review and pin a newer backend revision.

NixOS module

The flake exports the reusable module as both nixosModules.default and nixosModules.bitcoin-pulse. A consuming flake imports it and supplies only instance-specific configuration:

{
  imports = [ inputs.bitcoin-pulse.nixosModules.default ];

  services.bitcoin-pulse = {
    enable = true;
    bindAddress = "127.0.0.1";
    port = 3000;

    mempool.baseUrl = "http://127.0.0.1:8999";
    database = {
      url = "jdbc:postgresql://127.0.0.1:5432/bitcoin_pulse";
      username = "bitcoin_pulse_app";
      passwordFile = "/run/secrets/bitcoin-pulse/database-password";
    };
  };
}

The module defaults to loopback binding and the host-local Mempool backend. It also exposes the package, CoinGecko URL and optional API-key file, upstream and database timeouts, Hikari pool sizing, collector intervals, and price-history tolerance. Credential files must be readable by the bitcoin-pulse system user; the module passes only their paths to the application.

The generated service explicitly starts the packaged JVM with the prod profile under a dedicated unprivileged identity. It restarts on failure, sends SIGTERM to exercise the application shutdown hook, allows 30 seconds for shutdown, and applies systemd hardening without blocking PostgreSQL or upstream HTTP access. The Linux flake check checks.x86_64-linux.nixos-module tests startup, hardening, stop/restart behavior, and collector termination.

The reusable module does not create PostgreSQL, decrypt secrets, configure migrations or nginx, or enable the production Tobserver instance. Those host-specific concerns remain in the Tobserver deployment configuration and follow-up deployment issues.

Runtime profiles and network configuration

The backend requires exactly one positional runtime profile and fails clearly when it is missing or unknown:

Profile	Price history startup	Price provider	Bitcoin network data
dev	Seeded separately by bb dev	Deterministic fake provider	Fixed in-process values
dev-live	No automatic seed	Real CoinGecko	Real Mempool collectors
prod	No automatic seed	Real CoinGecko	Real Mempool collectors

The NixOS module always supplies prod; bb dev and bb dev:live supply their corresponding profiles.

The HTTP server listens on PORT (3000 by default) and binds to BIND_ADDRESS (0.0.0.0 by default). The all-interface default preserves convenient access from development containers, VMs, and other local devices. Production must set BIND_ADDRESS=127.0.0.1 so that only a same-host reverse proxy can reach the backend directly; the reusable NixOS module is responsible for applying that production setting.

MEMPOOL_BASE_URL is required in dev-live and prod, must not have a trailing slash, and has no application default. The fixed dev profile does not read Mempool URL, timeout, or collection-interval configuration. COINGECKO_BASE_URL optionally replaces the default https://api.coingecko.com/api/v3/simple/price endpoint, which is useful for isolated deployments and tests. Upstream requests have separate connection and overall request deadlines:

Provider	Connection timeout	Request timeout
Mempool	MEMPOOL_CONNECTION_TIMEOUT_MS (5000 ms)	MEMPOOL_REQUEST_TIMEOUT_MS (10000 ms)
CoinGecko	COINGECKO_CONNECTION_TIMEOUT_MS (5000 ms)	COINGECKO_REQUEST_TIMEOUT_MS (10000 ms)

A connection timeout bounds establishing the upstream connection. A request timeout bounds the complete HTTP request. Timeout failures are logged as failed collection attempts; they do not stop the application or collector. The collector tries again after its normal collection interval.

Block height collection

In dev, GET /block-height immediately returns a fixed height of 900000 with five deterministic recent blocks. Its fetched-at value records when the in-process fixture was initialized, and its source remains mempool to preserve the production response schema. This fixed snapshot never expires: the profile does not read BLOCK_HEIGHT_MAX_AGE_MS and creates no Mempool provider or block-height collector.

In dev-live and prod, the block height collector immediately requests GET /api/v1/blocks from the configured Mempool server. It projects the five newest entries to only hash, height, and Unix-second timestamp fields, derives the top-level height from the newest entry, and caches the complete snapshot in memory. It repeats every BLOCK_HEIGHT_COLLECTION_INTERVAL_MS (30 seconds by default). BLOCK_HEIGHT_MAX_AGE_MS controls how long a cached snapshot is publicly available. Its 120000 ms default allows four missed collections at the default interval. Configure the upstream with the required MEMPOOL_BASE_URL. Mempool requests use the shared Mempool connection and request timeouts documented above.

A successful snapshot requires five blocks with valid hexadecimal hashes, positive integer heights and timestamps, and consecutive newest-first heights. Only the five consumed entries are validated; older entries returned by Mempool are ignored. GET /block-height reads the in-memory cache without making a request-time upstream call. It returns 503 until the first successful fetch and whenever the stored snapshot is strictly older than the configured maximum age. A snapshot whose age is exactly the maximum remains available. Failed and malformed upstream responses are logged and leave the last coherent snapshot stored internally; once that snapshot becomes too old it is unavailable publicly but is not cleared. A later successful collection immediately replaces it and restores availability. The cache is intentionally cleared whenever the application restarts.

Halving countdown

GET /halving derives a countdown from one snapshot in the existing block-height cache. Its response contains exactly halving-height, remaining-blocks, progress-percent, estimated-at, and fetched-at. The next halving height is the first 210,000-block boundary strictly after the current height, remaining-blocks is that target minus the current height, and progress is ((210000 - remaining-blocks) / 210000) * 100. Progress is an unrounded finite number in the inclusive API range from 0 through 100. At a boundary it resets to zero; for example, height 840,000 targets 1,050,000 with 210,000 blocks remaining.

estimated-at adds 600 seconds per remaining block to the snapshot's unchanged fetched-at. This fixed-spacing projection does not use recent block cadence and is not a consensus-scheduled timestamp. The endpoint shares the block-height freshness policy and returns 503 with {"error":"Halving unavailable"} when no fresh snapshot is available. HTTP requests only read the cache and never make a request-time Mempool call.

Difficulty adjustment collection

In dev, GET /difficulty-adjustment immediately returns a fixture with a +2% adjustment estimate, 1000 blocks remaining, 50.4% progress, and a network hashrate of 897400000000000000000 hashes per second. The profile creates no Mempool provider or difficulty-adjustment collector.

In dev-live and prod, each difficulty adjustment collection requests GET /api/v1/difficulty-adjustment and GET /api/v1/mining/hashrate/3d from the configured Mempool server. The collector reads currentHashrate from the second response and caches it with the difficulty change, progress percentage, and remaining block count. Hashrate remains an unformatted numeric hashes-per-second value. Collection repeats every DIFFICULTY_ADJUSTMENT_COLLECTION_INTERVAL_MS (120 seconds by default) and uses the same Mempool URL and timeout settings as block height collection.

GET /difficulty-adjustment serves the combined snapshot from the live cache without making an upstream request. It returns 503 until the first successful collection. Both Mempool responses must succeed and validate before the collector replaces the cache. Failed and malformed upstream responses are logged, while the last successful combined snapshot remains available. The cache is intentionally cleared whenever the application restarts.

Market summary collection

A separate collector fetches Bitcoin USD/EUR market summaries on startup and hourly, with one CoinGecko Markets request per currency. Each validated summary contains current price, ATH price/date/change, and upstream 24-hour, 7-day, 30-day and 1-year percentages. Missing upstream period values remain null. The latest summary and acquisition status live in a per-currency/source in-memory cache. Restart clears the cache; failed refreshes preserve previous data internally without claiming success. There are no immediate retries.

Simple Price remains the source for /price, now collected every ten minutes. Hourly Markets prices do not write into spot history, and their upstream 1d change does not replace local change-24h. Combined with daily history, the defaults use approximately 6,014 requests per 31-day month for one continuously running instance, before startup and shared-key usage. The default historical-reference tolerance is now 15 minutes to support the slower spot cadence. Explicit overrides remain available.

MARKET_SUMMARY_COLLECTION_INTERVAL_MS defaults to 3600000; COINGECKO_MARKETS_URL overrides the full Markets endpoint without changing existing spot or daily URLs. The dev provider uses deterministic source-fake summaries without upstream access. NixOS exposes collectors.marketSummaryIntervalMs and coinGecko.marketsUrl. Price Detail reads these summaries independently and enforces its own availability policy. See docs/market-summary.md for exact mappings, cache reads, restart/failure semantics and quota assumptions.

Daily market history collection

A separate collector persists Bitcoin USD/EUR midnight prices and quote-currency 24-hour volumes. It fetches 365 days on first successful initialization, then normally three days for new data and overlapping corrections. It expands the window for retrievable missing days after downtime and retains older observations indefinitely. Bootstrap completion and failure/freshness state survive restart independently of whether history has gaps.

Collection runs on startup and at MARKET_HISTORY_COLLECTION_TIME_UTC, default 00:20 UTC. There are no automatic retries or late-publication polling. Failed attempts preserve valid data and wait for the next ordinary scheduled run or startup. Each currency gets at most one request per run. The dev profile uses deterministic daily fixtures; dev-live and prod use CoinGecko's market-chart endpoint with explicit daily granularity. COINGECKO_MARKET_CHART_URL overrides that complete URL without changing the existing spot-price URL setting.

Price Detail reads this dataset without changing /price. Optional intraday points are validated but not persisted; the detail chart currently contains only midnight observations. See docs/market-history.md for scheduling, coverage, repository metadata, failure behavior, retention, and quota limits. Apply the new migration before starting the updated backend; rolling it back destroys retained daily history.

Price Detail

GET /price/detail?currency=USD|EUR returns prepared daily prices/volumes, MA200, upstream ATH/period changes and nullable yearly performance. Its independent local refresh runs on startup and every 60 seconds, without upstream requests. HTTP only reads prepared state and checks acquisition status/freshness.

Price Detail returns 503 until both current-process startup acquisitions and preparation succeed, after known acquisition/preparation failures, and while newly accepted source versions await reconciliation. Summaries must be no older than 75 minutes by both fetch and observation time. Today's midnight becomes required at 00:30 UTC. These strict rules do not change /price or network-cache availability.

PRICE_DETAIL_REFRESH_INTERVAL_MS, PRICE_DETAIL_SUMMARY_MAX_AGE_SECONDS and PRICE_DETAIL_DAILY_DEADLINE_UTC default to 60000, 4500 and 00:30. NixOS exposes them under priceDetail. See docs/price-detail.md for the API, calculations, configuration and single-instance restart/failure limits.

Price collection

The generic collector runs immediately and repeats after PRICE_COLLECTION_INTERVAL_MS (ten minutes by default). In dev, it obtains Bitcoin's USD and EUR prices from a deterministic in-process fake provider and performs no CoinGecko HTTP request. The fake provider derives smooth, positive BigDecimal values from the absolute observation timestamp and currency using the same generator as bb db:seed, so the UI keeps updating against a coherent 48-hour fixture series.

bb db:seed writes 577 observations per currency: both endpoints of an inclusive 48-hour window at five-minute intervals, ending at the current five-minute UTC boundary. Re-running it is safe because identical timestamp/currency observations produce identical values and the repository preserves the existing observation identity.

In dev-live and prod, the collector requests both currencies in one real CoinGecko request. Set the optional COINGECKO_API_KEY to a CoinGecko Demo API key, or use COINGECKO_API_KEY_FILE to read it from a credential file. CoinGecko requests use the connection and request timeouts documented above. Failed requests and writes are logged without stopping later collection attempts.

Real observations use CoinGecko's shared source timestamp, which is stored separately from the application's shared fetch time. Fake and real observations use source coingecko to remain compatible with the Price feature service lookup. Repeated provider responses are idempotent, including when multiple application instances collect the same observations.

Clients must select a canonical uppercase quote currency with GET /price?currency=USD or GET /price?currency=EUR. A missing, lowercase, or unsupported currency returns HTTP 400. A supported currency without a latest persisted observation returns HTTP 503. Otherwise the response includes that currency's latest price and its independently calculated change-24h, which is null until sufficient history exists.

Add include-history=true to request approximately hourly points from the complete trailing 24-hour window; include-history=false or omission preserves the response without a history key. Insufficient coverage or a history-only failure returns an empty history while preserving the current price. The endpoint reads PostgreSQL directly and has no application-level server cache. The collector updates persisted data approximately every ten minutes by default, while clients may refresh chart history less frequently. The exact history selection, sampling, and failure policy is documented in docs/price-change.md.

See docs/database.md for datasource configuration, local PostgreSQL, migrations, and integration tests. The persisted-history selection and calculation policy is documented in docs/price-change.md. NixOS production deployment is tracked separately in BIT-10.