Celery Diagnostics Observer

The data-collection component for the Celery Diagnostics service.

Celery Diagnostics is an independent product and is not affiliated with, sponsored by, or endorsed by the Celery project or its maintainers.

The Observer runs as a separate process beside your Celery deployment. It reads Celery events and bounded operational signals, sanitizes them locally, and sends the resulting evidence to the Celery Diagnostics backend. The backend stores and reconstructs that evidence, runs diagnostic rules, and presents the results in the web dashboard.

Celery workers + Redis broker -> Observer -> Celery Diagnostics backend -> dashboard

The Observer collects evidence; the backend turns that evidence into task timelines and diagnoses. No diagnostics code runs in customer web or worker processes. An optional scheduler adapter runs inside Celery Beat only when periodic-fire diagnosis is required.

Before You Start

You need:

• a Celery Diagnostics account and project;
• the project's ingest key (CD_PROJECT_KEY), created in the dashboard;
• Python 3.11 or newer;
• network access from the Observer process to the Celery broker and the Celery Diagnostics ingest endpoint;
• Celery task events enabled for full lifecycle evidence.

Redis is the currently supported broker for queue-depth sampling and bounded message-presence checks.

Capabilities

• A CLI package that provides celery-diagnostics.
• The primary customer integration path for Celery Diagnostics.
• A separate observer process for Redis broker queue depth, Celery task and worker events, Celery control inspect snapshots, Observer health, transport retry, and an optional sanitized local spool.
• An executor for bounded, read-only diagnostic checks requested by Celery Diagnostics. Checks cover task presence, reservation, scheduling, worker capacity and presence, queue consumers, status-only JSON result records in Redis, and task-specific presence in configured Redis queues.
• An optional project-aware mode when run with -A myproject.celery:app; this loads the Celery app inside the observer process to explain routing, visibility timeout, task_track_started, and beat schedule coverage.
• An optional Celery Beat scheduler adapter that reports bounded schedule inventory, due decisions, and publish failures without task payloads.

Boundaries

• It does not instrument producer, web, or worker processes.
• It does not require changes to task definitions or a custom Celery Task base.
• It never transmits or persists task args, kwargs, result values, raw tracebacks, frame locals, or task message bodies.
• Redis queue depth alone cannot establish worker topology or prove that a queue has no consumers.

Install

python -m pip install --upgrade celery-diagnostics

If the shell cannot find celery-diagnostics, activate the virtual environment where it was installed or use:

python -m celery_diagnostics_observer --help

Quick Start

1. Create a project key

Sign in at app.celerydiagnostics.com, create or select a project, then open Integration and create a project key. The raw key starts with cf_ and is shown only once, so copy it directly to your deployment's secret manager.

CD_PROJECT_KEY is not a Celery setting or a key generated by this package. It is a project-scoped ingest credential issued by the Celery Diagnostics backend. The backend uses it to authenticate the Observer and associate incoming evidence with the correct project. Revoking or rotating the key in the dashboard does not change your Celery configuration.

The key is intentionally environment-only. The CLI has no --project-key option, so the key cannot be exposed as a process argument. Provide the real value through your deployment's secret manager instead of typing it into a shell command. The Observer sends it only as an HTTP Authorization: Bearer credential; it is never added to event bodies or the local spool.

2. Enable Celery events

Add these settings to the configuration used by your Celery app, then restart the workers. For a standard celeryconfig.py or another config object that uses Celery's lowercase setting names:

worker_send_task_events = True
task_send_sent_event = True
task_track_started = True

If you configure the app directly in Python:

app.conf.update(
    worker_send_task_events=True,
    task_send_sent_event=True,
    task_track_started=True,
)

For Django projects that load settings with app.config_from_object("django.conf:settings", namespace="CELERY"), add these to settings.py:

CELERY_WORKER_SEND_TASK_EVENTS = True
CELERY_TASK_SEND_SENT_EVENT = True
CELERY_TASK_TRACK_STARTED = True

Without task events, Redis queue sampling can still provide limited backlog evidence, but the backend cannot reconstruct complete task lifecycles.

3. Start the Observer

The values below are placeholders. Supply the real project key through your secret manager.

CD_PROJECT_KEY=cf_xxx \
CELERY_BROKER_URL=redis://YOUR_REDIS_HOST:6379/0 \
celery-diagnostics observe \
  --queues default,emails \
  --ingest-url https://ingest.celerydiagnostics.com

Use a broker address reachable from the Observer process. For example, redis://redis:6379/0 works only when redis resolves in the Observer's container or network; a process running on the host will commonly use redis://127.0.0.1:6379/0 with a published Redis port.

Keep the Observer running, trigger one known task, and confirm in the dashboard that the Observer, worker, and task event have been seen. Run celery-diagnostics doctor if the expected evidence does not appear.

Commands

observe

Run the long-lived observer process:

CD_PROJECT_KEY=cf_xxx \
CD_INGEST_URL=https://ingest.celerydiagnostics.com \
CELERY_BROKER_URL=redis://localhost:6379/0 \
celery-diagnostics observe --queues default

Project-aware mode loads your Celery app in the Observer process only:

CD_PROJECT_KEY=cf_xxx \
CD_INGEST_URL=https://ingest.celerydiagnostics.com \
celery-diagnostics observe \
  --mode project-aware \
  -A myproject.celery:app

observe prints a startup coverage summary to stderr before the runtime loops start. To inspect representative sanitized events locally without sending anything to the backend, use:

celery-diagnostics observe \
  --broker memory:// \
  --queues default \
  --dry-run \
  --print-sanitized-events

Dry-run output is machine-readable JSON Lines and never includes the project key. --dry-run does not require CD_PROJECT_KEY.

Read-only diagnostic checks are enabled by default. Disable them explicitly when the Observer must remain passive by adding --no-active-probes to the observe command.

For Redis message-presence checks, the Observer scans only configured queues, uses the default Kombu priority layout, and applies a bounded scan limit. It parses message envelopes locally only to match the protocol-v2 task ID; message bodies are immediately discarded and are never transmitted or persisted. A missing task is reported only when all scanned lists were stable and decodable; partial or malformed observations remain inconclusive.

Result-backend status checks are advertised only for a Redis backend with the JSON result serializer. A server-side Redis script returns the status field alone, so the Observer does not fetch the task's result value. Other result backends and serializers remain unsupported rather than silently loading a private result record through AsyncResult.state.

Periodic schedules (optional Celery Beat adapter)

The standalone Observer can follow a periodic task after publication, but it cannot know that Celery Beat should have fired an entry and did not. Run Beat with the package's scheduler wrapper when that distinction matters:

CD_PROJECT_KEY=cf_xxx \
CD_INGEST_URL=https://ingest.celerydiagnostics.com \
celery -A myproject.celery:app beat \
  --scheduler celery_diagnostics_observer.beat:ObserverPersistentScheduler

This remains Celery's PersistentScheduler; the wrapper adds sanitized schedule snapshots and evidence for due and failed publish attempts. It does not collect task args, kwargs, result values, broker URLs, or credentials. Schedules above the snapshot cap are reported as an incomplete inventory, so the backend will not infer that omitted entries were deleted.

Without the adapter, ordinary task diagnosis continues to work and the Periodic schedules page explicitly reports that Beat evidence is unavailable.

doctor

Explain what Celery Diagnostics can and cannot currently know:

CELERY_BROKER_URL=redis://localhost:6379/0 \
celery-diagnostics doctor

For project-aware coverage:

celery-diagnostics doctor --mode project-aware -A myproject.celery:app

doctor does not require CD_PROJECT_KEY. It reports telemetry coverage, safe claims, blocked claims, and next steps. It is a diagnostic coverage explanation, not a fake health check.

status

Show lightweight local integration status:

celery-diagnostics status

Use doctor when you need detailed telemetry coverage.

replay-events

Sanitize and time-shift a retained Celery JSONL event capture for controlled diagnostic testing:

CD_PROJECT_KEY=cf_xxx \
celery-diagnostics replay-events \
  --input capture.jsonl \
  --output sanitized-replay.jsonl \
  --cutoff 1710000000

The command applies the same privacy filter used by the live Observer. Use it only with event captures you are authorized to process. The output file is owner-readable only (0600). --cutoff is the latest source-event Unix timestamp to include; --anchor can map that cutoff to a specific ISO-8601 time and otherwise defaults to now.

Configuration

Variable	Purpose
CD_PROJECT_KEY	Project-scoped ingest credential. Required by live observe, replay-events, and resolve; not required by doctor, status, or dry-run.
CELERY_BROKER_URL	Celery broker URL. Redis is the current observer target.
CD_QUEUES	Comma-separated queue names to sample when --queues is not provided.
CD_INGEST_URL	Celery Diagnostics ingest base URL. Defaults to loopback development at http://127.0.0.1:8000; hosted use should set https://ingest.celerydiagnostics.com.
CD_TELEMETRY_POLICY	Identity visibility: readable or local-only. Defaults to readable.
CD_IDENTITY_KEY	Customer-managed identity key. Required only for local-only; never sent to the backend.
CD_OBSERVER_MODE	standalone or project-aware. Defaults to standalone.
CELERY_APP	Celery app import path used by project-aware mode, equivalent to -A.
CD_OBSERVER_ID	Observer instance identifier. Set a stable value in production; otherwise one is generated at startup.
CD_SAMPLE_INTERVAL	Redis queue sample interval in seconds.
CD_INSPECT_INTERVAL	Celery control inspect interval in seconds.
CD_BATCH_SIZE	HTTP transport batch size.
CD_FLUSH_INTERVAL	HTTP transport flush interval in seconds.
CD_SPOOL_PATH	Optional sanitized JSONL local spool path.
CD_LOG_LEVEL	Python logging level.
CD_ACTIVE_PROBES	Enable bounded read-only diagnostic checks. Defaults to 1.
CD_BROKER_MESSAGE_SCAN_LIMIT	Maximum Redis messages inspected by one task-presence check. Defaults to 10000, bounded to 100000.
CD_BEAT_OBSERVER_ID	Optional stable source label for the Beat adapter. Defaults to beat@<hostname>.
CD_BEAT_SNAPSHOT_INTERVAL	Beat schedule snapshot interval in seconds. Defaults to 30, bounded to 5..3600.
CD_BEAT_SPOOL_PATH	Optional sanitized JSONL spool used only by the Beat adapter.

Most Observer settings have equivalent CLI options. Secrets such as CD_PROJECT_KEY and CD_IDENTITY_KEY remain environment-only. For example:

CD_PROJECT_KEY=cf_xxx \
CD_INGEST_URL=https://ingest.celerydiagnostics.com \
celery-diagnostics observe \
  --broker redis://localhost:6379/0 \
  --queues default \
  --policy readable

Privacy Defaults

The Observer sanitizes telemetry before it leaves the customer environment.

readable sends operational task, queue, routing, and worker identifiers. local-only sends stable HMAC references plus an authenticated encrypted identity capsule. The customer-managed identity key never leaves the Observer, and both modes collect the same lifecycle evidence. The identity key must be at least 16 characters, remain stable across the project's Observer instances, and be stored in the customer's secret manager.

To identify a local-only run, execute this inside the customer environment:

CD_PROJECT_KEY=cf_xxx \
CD_IDENTITY_KEY='customer-managed-secret' \
CD_INGEST_URL=https://ingest.celerydiagnostics.com \
celery-diagnostics resolve R-XXXXXXXXXXXX

Default behavior:

• no task args;
• no task kwargs;
• no task results;
• no task payload body;
• no raw tracebacks;
• no frame locals;
• project keys appear only in the Authorization header and are never written to event bodies or the local spool;
• broker and ingest URL credentials are redacted in CLI reports;
• non-loopback ingest endpoints must use HTTPS;
• local spool and replay files are written for the owner only (0600).

Redis queue sampling uses safe queue depth checks. Redis-only evidence can show queue pressure and backlog symptoms, but it cannot prove that no worker is consuming a queue.

Project-aware mode uses an allowlist of app configuration facts. It does not dump arbitrary Celery config, task payloads, broker credentials, or exception messages from failed app imports.

The external Observer cannot directly witness an exception that prevents a producer from reaching the broker. In that case it reports the boundary of its evidence instead of inferring a publish failure from absence alone. Progress events are consumed when they already exist in the Celery event stream; this package does not add task-side progress instrumentation.

The optional Beat adapter observes only the Beat-to-broker publication boundary. It does not broaden access to task payloads or application data.

Production Checklist

• Store CD_PROJECT_KEY and, when used, CD_IDENTITY_KEY in a secret manager.
• Set a stable CD_OBSERVER_ID for each Observer instance.
• Configure CD_SPOOL_PATH on durable storage writable only by the Observer service account.
• Keep the Observer running under systemd, Docker Compose, Kubernetes, or an equivalent process supervisor.
• Allow outbound HTTPS to the ingest endpoint and only the broker/control access required by the enabled checks.
• Run celery-diagnostics doctor after changes to Celery routing, workers, queues, the broker, or the result backend.

Troubleshooting

If the dashboard receives no evidence, check that:

• the project key is active and belongs to the selected dashboard project;
• CD_INGEST_URL points to the Celery Diagnostics ingest endpoint;
• the Observer can resolve and reach both the broker and the ingest endpoint;
• the broker URL is reachable from the Observer's network namespace;
• workers were restarted after Celery task events were enabled;
• --queues or CD_QUEUES includes the queues you expect to sample.

Start with celery-diagnostics doctor. It reports available evidence sources, diagnostic limits, and configuration steps without requiring a project key or contacting the backend.

Local Development

Install editable dependencies:

python -m pip install --upgrade -e ".[dev]"

Run the test and static checks:

python -m pytest tests -q
python -m ruff check celery_diagnostics_observer tests
python -m py_compile celery_diagnostics_observer/*.py

Build the source and wheel distributions:

python -m build

Repository Layout

celery_diagnostics_observer/
  active_probes.py       # runs bounded read-only checks requested by the backend
  app_context.py         # reads allowlisted facts from a project Celery app
  app_loader.py          # loads the standalone or project-aware Celery app
  beat.py                # optional privacy-safe Celery Beat scheduler adapter
  capabilities.py        # reports which read-only checks are currently available
  cli.py                 # implements the celery-diagnostics commands
  config.py              # loads and validates environment and CLI configuration
  coverage.py            # determines what the Observer can reliably diagnose
  event_receiver.py      # receives Celery task and worker events
  health.py              # reports Observer health and available capabilities
  identity.py            # protects operational identities in local-only mode
  inspect_sampler.py     # samples worker state through Celery control inspect
  policy.py              # defines readable and local-only telemetry policies
  redis_message_probe.py # safely checks whether a task is present in Redis queues
  redis_result_probe.py  # reads task status without fetching Redis result values
  redis_sampler.py       # samples Redis queue depth
  replay.py              # sanitizes and time-shifts retained Celery event captures
  sanitizer.py           # removes private fields and normalizes outgoing evidence
  spool.py               # stores sanitized evidence locally during delivery outages
  transport.py           # batches, retries, and sends evidence to the backend
  version.py             # package version
tests/
  test_*.py              # Observer test suite

Release Notes

The package name is celery-diagnostics; its code imports as celery_diagnostics_observer.

Releases are built in GitHub Actions and published to PyPI with Trusted Publishing. No long-lived PyPI API token is stored in GitHub.

Before creating a GitHub release, update pyproject.toml, version.py, and CHANGELOG.md to the same version, then run:

python -m pytest tests -q
python -m ruff check celery_diagnostics_observer tests
python -m py_compile celery_diagnostics_observer/*.py
python -m build
python -m twine check dist/*

Publish by creating a GitHub release tagged vX.Y.Z. The protected pypi environment must approve the publish job. The release workflow verifies that the tag, package metadata, and version.py contain the same version before it publishes with PyPI Trusted Publishing.

See the changelog for version notes.

Security and License

Report vulnerabilities through GitHub private vulnerability reporting, not a public issue. Do not include production credentials, task payloads, or customer data in a report. See the security policy for details.

This package is source-available under the Celery Diagnostics Observer Proprietary License. It may be used as an unmodified integration with the Celery Diagnostics service, subject to the Celery Diagnostics Terms of Service.