[BUG] Docker Compose reports Data Prepper ready while OpenSearch sinks are still uninitialized

[kangkyunghyun]

What is the bug?

Docker Compose can report Data Prepper as ready even when its OpenSearch sinks have not initialized and its OTLP source is not accepting connections.

The Data Prepper service currently has no Docker Compose health check, and the official opensearchproject/data-prepper:2.16.0 image does not provide a built-in health check.

When an OpenSearch sink cannot initialize, Data Prepper keeps its process and Core API running while retrying the sink initialization. As a result:

• docker compose up --wait exits successfully.
• The container remains in the running state with no health status.
• http://localhost:4900/list returns the configured pipeline.
• The OTLP source port 21890 continues to refuse connections.
• Telemetry cannot be ingested.

This is separate from #407 and #408. PR #408 removes the snapshot-image-specific failure trigger, but the Docker Compose readiness gap is still reproducible with the official stable image when a sink fails for another reason, such as invalid credentials or an unreachable OpenSearch endpoint.

How can one reproduce the bug?

The issue can be reproduced with the official Data Prepper image and a healthy OpenSearch instance by deliberately configuring an unreachable sink endpoint.

From a clean Data Prepper container:

docker compose up -d opensearch

docker compose rm -sf data-prepper

DATA_PREPPER_DOCKER_REPO=opensearchproject \
DATA_PREPPER_IMAGE=data-prepper \
DATA_PREPPER_VERSION=2.16.0 \
OPENSEARCH_PORT=1 \
docker compose up -d --wait --wait-timeout 30 data-prepper

echo $?

Docker Compose exits successfully even though the sink cannot initialize.

Check the container state:

docker inspect \
  --format '{{.State.Status}} {{if .State.Health}}{{.State.Health.Status}}{{else}}no-healthcheck{{end}}' \
  data-prepper

Observed result:

running no-healthcheck

The Core API still lists the pipeline:

docker exec data-prepper \
  curl -s http://localhost:4900/list

However, the actual OTLP source is unavailable:

docker exec data-prepper \
  timeout 3 /bin/bash -c 'exec 3<>/dev/tcp/127.0.0.1/21890'

The connection is refused while the logs repeatedly show the OpenSearch sink initialization retrying.

I also reproduced the same behavior with intentionally invalid OpenSearch credentials.

What is the expected behavior?

docker compose up --wait should not succeed until Data Prepper is able to accept telemetry on its configured OTLP source port.

When a required sink cannot initialize, the Data Prepper container should remain unhealthy so that Compose and dependent tooling do not treat the stack as ready.

What is your host/environment?

• Host OS: macOS 26.5.1, arm64
• Docker CLI: 29.4.0
• Docker Compose: v5.1.2
• OpenSearch image: opensearchproject/opensearch:3.8.0
• Data Prepper image: opensearchproject/data-prepper:2.16.0
• Data Prepper platform: linux/amd64
• Default memory limits:
  • OpenSearch: 2 GiB
  • Data Prepper: 1 GiB

Are there any screenshots?

N/A. The relevant state, log output, and connection checks are included above.

Do you have any additional context?

The following validation matrix was tested in isolated Docker Compose projects connected to a healthy OpenSearch instance:

Scenario	docker compose up --wait	Container health	OTLP port 21890
Valid sink, no health check	Exit 0	No health status	Opens after initialization
Invalid sink credentials, no health check	Exit 0	No health status	Remains closed
Valid sink with TCP health check	Exit 0	Healthy	Open
Invalid sink with TCP health check	Exit 1	Unhealthy	Closed
Two sinks, one failing, with TCP health check	Exit 1	Unhealthy	Closed

The management /list API is not sufficient as a readiness check because it remains available and lists the pipeline while sink initialization is still failing.

The Helm chart already uses the OTLP source port as its Data Prepper readiness signal:

readinessProbe:
  tcpSocket:
    port: 21890

A matching Docker Compose health check would align both deployment methods. The following check was validated against the normal, complete-failure, and partial-sink-failure cases:

healthcheck:
  test:
    - CMD-SHELL
    - timeout 3 /bin/bash -c 'exec 3<>/dev/tcp/127.0.0.1/21890'
  start_period: 10s
  interval: 5s
  timeout: 4s
  retries: 12

This health check is intended to verify initial startup readiness. Runtime sink failures after a successful startup should continue to be monitored through exporter, queue, and pipeline failure metrics.