Improve noxfile: install consistency, reproducibility, and CI time #900

Tracking issue for improvements to py/noxfile.py and the related sharding and dependency setup. Each item below was checked against main (c93a167) by running the sessions locally.

Items are grouped by PR. PRs 1–6 have been merged; optional PR 7 remains.

PR 1: Fix session install correctness — merged in #903

Make each session's environment consistent and fail loudly when it isn't.

• Sessions can end up with conflicting packages. Each session runs 3–4 separate install commands (-e ., the test group, the provider pin, the session's extra group), and none of them checks what the earlier ones installed. In test_pydantic_ai_logfire(latest), uv pip check reports a conflict: pydantic-ai 2.54 installs opentelemetry-instrumentation-httpx, which requires opentelemetry-instrumentation==0.65b0 and opentelemetry-semantic-conventions==0.65b0. The test-pydantic-ai-logfire lockfile install then downgrades both to 0.60b1. The tests still pass, but on a mismatched environment.
  • Fix: do one install per session where possible, and run uv pip check at the end of setup so these conflicts fail loudly. Fix the logfire session's conflict as part of this.
• test_cli replaces the copy of braintrust it's supposed to test. session.install(".[cli]") runs after _install_test_deps:
  • In source mode it swaps the editable install for a plain copy (direct_url.json no longer has "editable": true), so subprocesses import a stale build.
  • In --wheel mode it replaces the built wheel with a build from source, so wheel mode doesn't test the wheel for this session.
  • Fix: pass the extra through _install_test_deps (-e .[cli] or <wheel>[cli]).
• A missing matrix key silently installs nothing. _install_matrix_dep returns when the key isn't there. boto3/botocore, the three OpenTelemetry packages and autogen-agentchat/autogen-ext are kept in step through separate keys, so a typo or a version added to only one of them silently drops a package. It should call session.error, or the matrix could allow a list of requirements per version.
• Some installs aren't pinned at all: pyyaml (test_btx_openai), llama-index-llms-openai and llama-index-embeddings-openai (test_llamaindex), and the .[cli] extra (test_cli). Move these into dependency groups.

PR 2: Run pylint against the matrix latest pins — merged in #905

• pylint checks against much older provider SDKs than the tests use. The lint group gets its versions from uv.lock, not [tool.braintrust.matrix]:

  package	lint (uv.lock)	matrix latest
  openai	1.99.9 / 2.31.0 / 2.32.0	3.24.0
  anthropic	0.125.0	1.11.0
  openrouter	0.10.8	1.3.22
  huggingface-hub	1.33.0	2.1.1
  pydantic-ai	2.8.0	2.54.0
  litellm	1.88.6	1.104.0
  opentelemetry-*	1.38.0 / 1.39.1	1.45.0

  The same is true for openai-agents, livekit-agents, agentscope, autoevals and google-cloud-discoveryengine. The [tool.braintrust.vendor-packages] comment in pyproject.toml says pylint installs each package at its matrix latest pin, but the code doesn't.

  • Fix: install the matrix latest pins in the pylint session (or update the comment and bump the lint group). Expect some new pylint errors from the newer SDKs.

• The pylint session installs the .[all] extra unpinned. Move it into a dependency group (or install the matrix pins instead). Moved here from PR 1 so that only this PR changes the pylint session.

PR 3: Fix shard weights — merged in #904

Touches scripts/check-session-weights.py, scripts/session-weights.json and .github/workflows/update-session-weights.yaml; no noxfile changes.

• The shard weights in scripts/session-weights.json are out of date in both directions.
  • 8 entries are for sessions that no longer exist, e.g. test_anthropic(0.49.0), test_litellm(1.83.0), test_openai(1.71), test_openai_http2_streaming. check-session-weights.py --update merges new numbers in but never removes old keys.
  • 14 sessions have no weight. Several skip on the machine that measures weights (Windows, Python 3.10): test_ai_sdk, test_deepagents, test_harbor, test_pipecat, test_livekit_agents. The duration regex also only records sessions that succeed, so these always fall back to the 15s default.
  • Fix: drop keys that nox -l doesn't list, and measure on Linux with Python 3.12+ (or record per platform).

PR 4: Reproducible installs with exclude-newer — merged in #906

• Only the provider itself is pinned; everything it depends on is resolved fresh from PyPI on every run. _install_matrix_dep runs e.g. uv pip install anthropic==1.11.0 with nothing pinned underneath it. That's where test-mcp-v1, test-sqlalchemy-2-0, test-pydantic-ai-otel-events, the genai-prices pin (chore(pydantic_ai): pin compatible genai-prices #636) and chore: Fix CI due to transitive deps #625 ("Fix CI due to transitive deps") came from.
  • Fix: set exclude-newer (uv's option to ignore anything published after a date) and pass it to every session install. Have dependency-updates.yml move the date forward along with uv.lock, so upstream breakage shows up in the weekly PR instead of on random feature PRs.
  • Easier to land after PR 1, since uv pip check will show whether the date-pinned resolutions are consistent.

PR 5: noxfile cleanup and small speedups — merged in #901

No change to what runs or which versions are covered.

• Some sessions start pytest several times. test_openai and test_langchain each call _run_tests 3 times, so collection runs 3 times, and if the first run fails the later files never run. Let _run_tests take several paths and call pytest once.
• The test_core check that no provider SDK is installed starts ~30 Python subprocesses (~1.6s). One importlib.util.find_spec script is faster and stricter: today an import that fails for any other reason also passes.
• Small fixes:
  • The module docstring says --no-vcr; the real flag is --disable-vcr.
  • The "Version matrices — derived from dependency groups" comment is out of date (they come from [tool.braintrust.matrix]).
  • The ignore_path parameter of _run_tests is unused.
  • Wheel mode uses os.chdir(tmp) instead of session.chdir(tmp). It's harmless today because nox resets the working directory for each session, but it leaves the cwd pointing at a deleted directory.
  • test_harbor checks the Python version with Version(platform.python_version()); every other session uses sys.version_info.
  • test_temporal and test_pytest_plugin don't pass version= to _run_tests.

PR 6: Trim redundant test runs — merged in #902

This changes what CI covers, so it needs agreement on the trade-off.

• Three sessions re-run the whole core suite: test_autoevals ×2 and test_braintrust_core, about 155s together (~9% of the ~1800s total in session weights). Run only the tests that behave differently when those packages are installed, e.g. via a marker.
• Some tests run against every openai version when they probably don't need to. test_openai_http2_streaming and test_btx_openai each run on all 4 openai versions (8 sessions). Latest plus oldest is probably enough.

PR 7 (optional): Table-driven sessions

• About 45 session functions are nearly identical (install test deps → install matrix dep → run tests). A small table plus a helper would cut a few hundred lines. If Python/OS skips were written as data, nox-matrix.py could leave out sessions that will skip on a given job, instead of giving them shard slots. Easier after PRs 1 and 5, since it builds on the install and _run_tests changes.

Not scheduled

• (Smaller) With a warm cache, test_anthropic(latest) takes 16s: about 5s installing and 11s in pytest. Rebuilding the editable install costs ~1.9s in every one of the 98 sessions. PR 1's combined installs should cut some of this; revisit afterwards.
• pytest-vcr hasn't had a release since 2020; _test_cassette_usage.py already refers to pytest-recording. Switching touches every cassette-backed test, so it's out of scope for the noxfile work.
• The wheel includes 143 test_*.py files. This is a packaging decision, but wheel mode (_run_tests with --wheel) currently relies on it.

Which PRs can run in parallel

• Wave 1: PRs 1, 2, 3, 5, and 6 have merged. PR 6's coverage trade-off was approved.
• PR 4: merged in #906, after PRs 1 and 2.
• PR 7 goes last: it rewrites every session function, so it would conflict with any open PR that touches sessions, and it should build on the final install and _run_tests helpers. PRs 2 and 3 don't block it, since the pylint session wouldn't be part of the table.

Historical suggested order

PRs 1–6 have since merged; PR 7 remains optional.

1. PRs 1–3: each fixes a confirmed bug and is small and contained.
2. PR 4 and PR 6: these help most with flaky CI and CI time.
3. PR 5 any time, then PR 7 if we want it.

🤖 Generated with Claude Code