[CarlosMion]

Fixes Issue

PER-16361 (opal-server log volume). This ports #973 to release/0.9.x, the line the 0.9.x images are built from. Three changes.

Changes proposed

1. The per-data-update line is DEBUG

opal_server/data/data_update_publisher.py::DataUpdatePublisher.publish_data_updates:

• "[pid] Publishing data update to topics: …, reason: …, entries: …" is now DEBUG. Its message and kwargs are unchanged.
• Kept at WARNING: "[pid] No topics were provided for the entry with url: …" in the same function.

2. The gunicorn master configures logging before it logs anything

scripts/gunicorn_conf.py::when_ready now calls configure_logs() first, before it imports opal_server.scopes.task and runs preload_scopes(). preload_scopes' docstring now states that precondition.

On this line nothing configured logging in the master until now. Its output, including the SCOPES preload, went through loguru's default handler: DEBUG level, plaintext, diagnose=True and no credential scrubbing. After this change:

• The preload's per-scope DEBUG lines no longer appear at the default LOG_LEVEL=INFO. These include "Sync scope:", "Initializing git fetcher", "Repo found at", "Fetching remote", "Fetch completed" and the backoff-skip lines.
• Master lines use the configured format (JSON with LOG_SERIALIZE=true) and carry their real level.
• Master lines are now scrubbed by the configured patcher and stream. That includes "Could not clone repo at" (still ERROR, same text) and redis_utils' "Connecting to Redis: {url}" DEBUG line, which on this line passes the URL unredacted. At INFO that line is not emitted at all.
• The opal-client images use the same gunicorn_conf.py, so their master is configured too before when_ready returns at the ImportError.
• If you use your own GUNICORN_CONF and call preload_scopes() from it, call configure_logs() first.

Side effects:

• configure_logs() also takes over stdlib logging in the master (root InterceptHandler at level 0), so third-party records reach the configured sink at their own level, INFO included. Example: datadog.dogstatsd "Error submitting packet" WARNINGs, where no statsd agent is reachable. Workers already do this.
• With LOG_TO_FILE=true, the master opens its own file sink.
• A configure_logs() error now fails in the master, so gunicorn exits with code 1. Before, a worker failed to boot and gunicorn exited with code 3.

3. One OPAL INFO line per scopes policy push

Line	file::symbol	Level
"Triggering policy update for scope …"	opal_server/scopes/service.py::NewCommitsCallbacks.trigger_notification	INFO, kept: one per publish per pod leader
"scope '…': Found new commits …"	opal_server/scopes/service.py::NewCommitsCallbacks.on_update	INFO → DEBUG
"Publishing to topics: …"	opal_common/topics/publisher.py::ScopedServerSideTopicPublisher.publish	INFO → DEBUG
"Refresh scope: …"	opal_server/scopes/api.py::init_scope_router.refresh_scope	INFO → DEBUG
"Webhook listener triggered (N)"	opal_server/policy/watcher/task.py::BasePolicyWatcherTask._on_webhook	INFO → DEBUG
"Repo was fetched after refresh request, …"	opal_server/git_fetcher.py::GitPolicyFetcher._should_fetch	INFO → DEBUG

• Lines that keep their level:
  • "Webhook trigger task failed" (ERROR), opal_server/policy/watcher/task.py::BasePolicyWatcherTask._on_webhook
  • "repo path is not found" and "Got exception for repo" (ERROR), opal_server/scopes/service.py::NewCommitsCallbacks.on_update
  • "no tracked files were affected" (WARN), opal_server/policy/watcher/callbacks.py::create_policy_update
  • "Could not clone repo at" (ERROR), opal_server/git_fetcher.py::GitPolicyFetcher._clone
  • "Target branch was not found" and "Hinted commit hash was not found" (INFO), opal_server/git_fetcher.py::GitPolicyFetcher._should_fetch
• Other INFO lines that stay by design:
  • per pass: "Periodic sync" and "OPAL Scopes: syncing N scopes"
  • a scope's first sync: the clone lines and "Publishing policy update, directories"
  • a hinted refresh: "Hinted commit hash was not found", which normally fires because the hinted commit isn't in the clone yet
  • fetch anomalies, and recovery or deletion notices
  • third-party lines: the pubsub broadcaster and uvicorn's access log
• Data updates: the scoped publisher also carries POST /scopes/{id}/data/update, so together with change 1, data updates have no OPAL INFO line at the default level. Use OPAL_LOG_LEVEL=DEBUG to see them.
• Non-scopes mode: "Webhook listener triggered (N)" sits in a shared base class, so non-scopes mode loses it too. That mode still logs its own INFO per check from GitPolicySource.
• The N count: N, the length of the webhook task list, is now visible only at DEBUG.

Docs

The two tutorials quoted the data-update line as default-level output. They now show the remaining lines and mention OPAL_LOG_LEVEL=DEBUG. The compose example's comments drop the line. The healthcheck sample also drops two "calling subscription callbacks" lines, which fastapi_websocket_pubsub logs at DEBUG.

Also

• scopes/task.py::_periodic_polling drops an unused except … as e and a placeholder-less f-string (flake8 F841/F541). The message is unchanged.
• The preload comment cites scripts/gunicorn_conf.py::when_ready.

Not addressed (flagged)

• "Skipping sync for {url}: a previous git operation is still running after its timeout." (git_fetcher.py::GitPolicyFetcher.fetch_and_notify_on_changes) causes most per-restart WARNING lines on a large fleet, and the workers log it too. Its text is unchanged here.
• scopes/task.py::trigger's WARN uses %s, so the data is never interpolated.
• refresh_scope's ERROR says "Unauthorized to delete scope".

Differences from #973

• scopes/task.py only gains the docstring precondition. This line has no fix: redact auth values in error logs, callback reports and server responses #968, so there was no configure_logs() call in preload_scopes to remove.
• server_log_redaction_test.py doesn't exist on this line. The preload_*_test.py files never stubbed configure_logs.
• test_data_update_publisher.py gains _RecordingPublisher and the level test. Master's pydantic-v2 serialization test (fix(deps): support Starlette 1.x to address CVE-2026-48710 (#956) #959) is not ported.
• These test commits are also in fix(opal-server): cut per-update and policy-push INFO lines, configure master logging first #973 or are being added there: the thread filter in the master-logging test, the credential-masking pins (DEBUG entries URLs, credentialed REDIS_URL) and the stdlib-routing pin. Their test files are identical on both branches.

Tests

• opal_common/tests/log_capture.py: records at DEBUG and through a real INFO sink.
• opal_server/tests/gunicorn_master_logging_test.py:
  • runs the real configure_logs() against a fresh loguru state whose only sink stands in for the default handler;
  • asserts nothing reaches that stand-in, with SCOPES on (INFO and DEBUG) and off, at import time, and in the client path;
  • asserts credentials are masked: git remote, plus a credentialed REDIS_URL logged as a kwarg the way RedisDB does;
  • asserts stdlib logging is routed into the configured sink;
  • asserts preload_scopes doesn't reconfigure logging;
  • each check has a control arm; most of them skip configure_logs.
• opal_server/tests/scopes_push_log_levels_test.py:
  • exactly one OPAL INFO line per push, via on_update and end to end via ScopesService.sync_scope against a local clone;
  • the levels of the WARN/ERROR siblings, refresh_scope, _should_fetch's three lines and "Could not clone repo at".
• Level tests in test_data_update_publisher.py (which also checks redaction of the DEBUG entries and the WARN URL), webhook_task_cleanup_test.py and opal_common/tests/scoped_publisher_log_level_test.py.
• Verified on Python 3.9.6 with pydantic 1.10.26:
  • pytest packages/opal-server packages/opal-common: 568 passed;
  • 100% diff coverage;
  • every demotion and kept sibling is mutation-checked: reverting any of them fails a test.

Check List (Check all the applicable boxes)

• I sign off on contributing this submission to open-source
• My code follows the code style of this project.
• My change requires changes to the documentation.
• I have updated the documentation accordingly.
• All new and existing tests passed.
• This PR does not contain plagiarized content.
• The title of my pull request is a short description of the requested changes.

Note to reviewers

opal_client/tests/data_updater_test.py::test_data_updater_with_report_callback fails intermittently on macOS. It fails the same way on an untouched release/0.9.x, and this PR doesn't touch the client.

🤖 Generated with Claude Code

[linear-code]

PER-16361

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.