Repository navigation
Offline test framework - #344
Closed
nikbpetrov wants to merge 9 commits into
Closed
nikbpetrov wants to merge 9 commits into
nikbpetrov wants to merge 9 commits into
Conversation
nikbpetrov
force-pushed
the
feat/offline-test-framework
branch
from
October 25, 2026 17:28
a014fa3 to
845978e
Compare
Introduce the level-is-the-directory layout (`unit/`, `contract/`, `integration/`, `e2e/`, `live/`) and move the existing tests into the level they already test at. Moved test bodies are unchanged apart from the import path, with these exceptions: * the builders move out of `conftest.py` into `tests/factories.py` (type hints added), leaving `conftest.py` with the fixtures only; * the leaderboard, nightly and naive-forecaster tests import `leaderboard.main` / the jobs directly now that secrets, env and the LLM clients are read lazily, so `leaderboard/_leaderboard_import.py` and `leaderboard/conftest.py` go (`_module_stubs.py` stays: the serpapi job tests still import `func_serpapi_fetch` under stand-in `helpers.keys` / `helpers.slack` modules); * tests that pinned implementation details rather than behaviour are pruned: `test_constants.py` as a whole, and the `hasattr` / source-text assertions inside the moved `llm_forecaster`, `orchestration`, `cloud_run`, `leaderboard` and `metadata` tests; * two relocations inside the move: `test_runner.py`'s transcript tests get their own `unit/llm_forecaster/test_model_run_transcripts.py`, and `test_llm_identities.py`'s registry-wide release-date check joins `contract/test_shared_llm_model_runs.py` next to the tests `test_model_runs.py` already sent there; `test_types_and_schemas.py`'s `SourceType` assertion is tightened to the exact member set; * the LLM smoke-test CLI moves from `tests/llm_forecaster/smoke_test/` to `src/llm_forecaster/smoke_test.py`, since it is a tool rather than a test (that package is copied into the Cloud Run image, so this is the one production file the test commits touch). The sampling tests land in `unit/question_sampling/`, after the package they test, among them `test_question_sampling.py`, `test_retire_questions_from_sampling_pool.py` and `test_get_resolution_criteria.py` (`tests.conftest` builder import changed to `tests.factories`). The publish job's test joins the other job tests in `unit/orchestration/` (its repository-root path adjusted for the deeper folder). Not moved: `test_integration.py` stays at the root until its e2e successor lands; `test_legacy_llm_cleanup.py` and `test_utils_cleanup.py` stay at the root pending a decision to keep or prune them. `test_orchestration_boundary.py` is deleted here and returns re-homed as `contract/test_llm_forecaster_conventions.py` when that layer is added; likewise the deploy-staging and GitPython assertions pruned from the moved orchestration / question-set IO tests return as `contract/test_deploy_staging.py`. `test_real_question_set_forecast.py` moves to `unit/llm_forecaster/` unchanged and is re-homed as `e2e/test_llm_forecaster_pipeline.py` when that layer is added. `pyproject.toml` registers the `live` marker and deselects it by default; `.gitignore` admits the `tests/golden/*.csv` snapshots the e2e layer will commit (replacing a stray `--` line).
Framework code only. Its autouse guards apply to the whole suite from this commit on, so the moved tests now run offline by construction; nothing else consumes the harness yet. * `_harness/network.py`: a socket-level guard that turns any outbound connection into a test failure, and disables `curl_cffi` transfers (the client `yfinance` uses, which bypasses `socket`). `BlockedNetworkError` derives from `BaseException` so production's tolerant `except Exception` blocks (404-tolerant downloads, modified-time fallbacks, Slack) cannot swallow it and read a stale `/tmp` copy instead. `conftest.py` installs it in `pytest_configure`, before collection imports any job module, and keeps it on for the whole session; because Google's clients use native gRPC the socket guard cannot see, an autouse fixture also makes the `google.cloud.storage.Client`, `SecretManagerServiceClient` and `run_v2.JobsClient` constructors raise. A fake `keys.get_secret` keeps key access offline, and the metadata jobs' shared-registry key prefetch (`metadata_llm.configure_api_keys` / `validate_provider_keys`) is faked the same way. All of it is lifted for `@pytest.mark.live`. * `_harness/buckets.py`: `LocalBucket`, a temp-dir stand-in for the GCS buckets. The `local_bucket` fixture points `GCP_STORAGE_LOCAL_ROOT` (the offline opt-in of the pinned `fri-utils` build), `BUCKET_MOUNT_POINT` and the `*_BUCKET` variables at it, so the real storage code reads and writes there, and makes a tolerated 404 leave no stale `/tmp` file behind. Seed/read helpers cover question banks, fetch files (incl. Wikipedia's per-page files), resolution files, hash mappings, metadata and forecast sets; writes are atomic. * `_sources.py`: the registry partition the parametrized layers iterate over — `IMPLEMENTED` (real fetch + update, listed explicitly), `UPDATE_ONLY` (no longer fetched, still updated: manifold) and `RETIRED` (frozen dfq: INFER). Derived from the registry metadata, so importing it never calls a source. * `_golden.py`: `check_golden()`, a CSV snapshot comparator (canonicalise → round-trip → `assert_frame_equal`, sentinel strings such as `N/A` compared literally); a missing golden is written and the test fails, never silently passes; `UPDATE_GOLDEN=1` re-blesses. * `conftest.py` also gains `fresh_source` and the `offline_update_case` adapter, which patches each market source's own network seam (`_build_resolution_df`, `_get_market`) so `update()` runs offline; the sources whose `update()` reads everything out of the fetch data (acled, dbnomics, fred, polymarket, serpapi, wikipedia) run unpatched. `factories.py` gains the forecast-set, leaderboard-entry, serpapi and Wikipedia fetch-result builders and a `FakeJevClient` for the English-language classifier. `unit/llm_forecaster/test_real_question_set_forecast.py` reads the published question set over the network, which the guard now blocks, so it carries `@pytest.mark.live` from this commit until the e2e layer re-homes it.
System-wide guarantees, parametrized over the source registry or the job list so a new source or job fails until it is classified and conforms. * `test_offline_imports.py`: every job entry point and the lazy helpers import with no network, no Secret Manager client and zero `get_secret` calls; the heavy import chains are re-checked in a cold subprocess with bogus credentials. Each module is evicted and re-imported so its top-level body really re-executes under the guards. Also pins that keys, env and the storage root are read at call time, that the network guard itself blocks, and the smoke-test tooling's import layering. * `test_registry_coverage.py`: the implemented / update-only / retired partition covers the registry exactly once, and each bucket behaves like its label (implemented sources are scheduled for both fetch and update, retired and update-only sources refuse to fetch, retired `update()` is a no-op). * `test_update_conformance.py`: `update()` of every implemented or update-only source returns a `QuestionFrame` with exactly `QUESTION_FILE_COLUMNS` (serpapi's per-question `resolution_criteria` is the one sanctioned extra) plus `ResolutionFrame` resolution files (acled, whose questions resolve off its fetch file, must return none), run offline through the `offline_update_case` adapter. * `test_resolve_contract.py`: `resolve()` of every market source fails fast on a missing id and an unknown source; nullified rows are dropped (surviving ids and counts asserted, so an empty frame cannot pass); values stay in [0, 1]; no resolution date precedes the due date. * `test_deploy_staging.py` and `test_llm_forecaster_conventions.py`: the deploy copy lists and the GitPython-free requirements, and the LLM forecaster's import layering — the assertions pruned or deleted from the moved tests in the scaffold commit, re-homed at the level they belong to.
One real IO boundary per test: the production `driver()` or IO module runs against the `local_bucket` mount with its external call (`fetch()`, `update()`, the LLM, Slack) mocked at the source's own seam, proving the read → call → write wiring. * `test_source_drivers.py`: every implemented source's fetch and update drivers, parametrized (bank read, inputs handed to the source — bound through the real method signature, so the call shape is not pinned — outputs persisted); `polymarket.update()` runs unmocked as a real chain since it needs no network. Drivers that diverge from the uniform wiring have their own tests: serpapi's self-inspecting fetch and strict history load; acled's update (the raw events reach `update()` with `event_date` unparsed, questions and the hash mapping are persisted, never a resolution file; also run unmocked); wikipedia's one-file-per-page fetch with its failed-page alert and the update that reads the pages back as one frame each (also run unmocked); the update-only manifold driver; the retired INFER driver. * `test_resolve_driver.py`: `func_resolve` from raw forecast set to processed forecast set and the resolution set published to the public bucket. * `test_leaderboard_compile.py`: `download_and_compile` date, min-resolved and eligibility filters down to exact `question_pk` keys. * `test_metadata_drivers.py`: tag and validate drivers writing `question_metadata.jsonl`, including the `english` flag and the stateful behaviour (idempotent rerun, incremental tagging, stale-row prune, a failed language classification left blank and retried next run). * `test_orchestration_io.py`: `_source_io` and `_io` round-trips, including the empty-file edge cases and the files `read_forecast_file` skips (invalid JSON; a missing, non-string or date-less `question_set`; a filename date that differs from it), with the Slack alert recorded.
Three whole offline flows, each pinned two ways: semantic anchors (what is correct) and a golden
CSV (what is produced), so a wrong re-bless still fails.
* `test_resolution_pipeline.py`: bucket update → `explode_question_set` → `resolve_all` → impute →
leaderboard scoring, with and without a nullified market; the forecasts are question-dependent
so a wrong join or weighting shows in the scores; replaces the root `test_integration.py`,
deleted here.
* `test_question_set_pipeline.py`: the `func_question_set_create` `driver()` on seeded banks +
metadata → the published `<date>-{llm,human}.json`, deterministic through `env.RANDOM_SEED` and
golden-pinned; plus the composed flow seeded banks → tag `driver()` → validate `driver()` (LLM
and the English classifier mocked) → create `driver()`. The eligible pool equals the configured
totals, so every exclusion (invalid, resolved, not English, unclassified, `Other`, no freeze
value) is load-bearing. The driver reads only the source keys and sampling shares from
`question_curation`; intros and resolution criteria come from the source registry, and the
per-type totals are bound on the sampler call.
* `test_llm_forecaster_pipeline.py`: question set → `runner.run_model` with the LLM boundary
mocked → the written forecast set, which is the freeze-values variant only (the zero-shot
variant is forecast and shared, not written); re-homes `test_real_question_set_forecast.py`,
whose read of the published set becomes the `@live` companion.
Goldens live in `tests/golden/`; `UPDATE_GOLDEN=1` re-blesses after an intended change.
`@pytest.mark.live` tests that call a source's own request methods against the real API and validate the fields the parser consumes, so upstream drift surfaces as a `KeyError` in the real code path rather than in a hand-maintained schema. Covers polymarket, manifold (the update path: market detail + bets), yfinance and metaculus; the dataset sources have no live contract yet. Deselected by default (`addopts = -m 'not live'`); run with `make test ARGS="-m live"`. Keyed endpoints resolve their credential through `helpers.keys` and skip, not fail, when none is available.
…nd curation Pure in-memory tests of the domain logic, one concern per test, HTTP mocked ad hoc per source and the LLM mocked at `metadata_llm.get_metadata_model_response`. * sources: ACLED and Wikipedia `resolve()` orchestration against hand-computed oracles (the moved files grow from their main versions), the inclusive resolution-date boundary of every source, and the ACLED naive forecaster, whose 29-of-30-days sum is pinned as a strict xfail. * resolve: the strict "today is not yet resolvable" boundary of `explode_question_set`, and the cross-source invariants in `test_invariants.py`. * leaderboard: scoring, masks, intervals and p-values, artifact writers, and a real two-way-fixed-effects + bootstrap run on a seeded fixture whose output is pinned by the `leaderboard_2fe*.csv` goldens. * metadata: tag and validate response parsing (for validate: `ok`, `flag`, a missing `Classification:`, an ambiguous verdict and a model error). * question_sampling: the share-based allocation wrappers with unequal shares, every market and horizon bin edge, stratified and seeded sampling, against `question_sampling._allocation`, `_binning`, `_prepare` and `_sampling`; the binning and market samplers take the forecast due date as `forecast_datetime`, so the tests anchor on a fixed tz-aware date. Two stratified cases that `test_bin_quota_apportionment.py` already pins (per-bin counts follow the weights; a bin that runs out hands its slots to the others) are not repeated here.
`tests/README.md` is the full per-level, per-job coverage reference (what each layer guarantees, where a new test goes, what is still open).
The offline test suite needs every `utils.gcp.storage` function to serve a bucket from a local directory, with no network and no credentials. That is forecastingresearch/utils PR forecastingresearch#32 (`GCP_STORAGE_LOCAL_ROOT`: when set, `<root>/<bucket>` is the authoritative copy of the bucket, with GCS semantics, and a missing bucket directory raises instead of reaching GCS); until it merges, the runtime pin points at the PR's head on the `nikbpetrov/utils` fork. The build is upstream `8fc3087` (`main`'s pin) plus that one commit, whose only runtime change is `utils/gcp/storage.py`; nothing in deployment sets the variable, so the jobs behave as before. The line goes back to the upstream pin when forecastingresearch#32 merges. `test_runtime_requirements.py` matched the pin by the upstream owner's URL; it now matches any `github.com/<owner>/utils@` pin, since the contract it guards is that the shared utils pin lives only in the root `requirements.runtime.txt`.
nikbpetrov
force-pushed
the
feat/offline-test-framework
branch
from
October 30, 2026 08:01
845978e to
35df984
Compare
nikbpetrov
marked this pull request as ready for review
October 30, 2026 08:12
Member
|
merged in e8d3c94 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Not mergable until this utils PR merges!! Must edit latest commit here thereafter.
Commits split for reviewability. Squash before merging.
Start with
src/tests/README.mdto get a sense of what's going on. Unit tests are almost entirely unreviewed in their implementation. Must be checked as work is done for a given part. Ideally, start seperate issues when this merges.