[BUG] Data Prepper sinks repeatedly fail with "Request execution cancelled" on initial Docker Compose startup

[kangkyunghyun]

What is the bug?

On a fresh local deployment using the repository's default Docker Compose configuration, OpenSearch becomes healthy and the Data Prepper container reports Up, but some Data Prepper OpenSearch sinks repeatedly fail during initialization with:

Failed to initialize OpenSearch sink with a retryable exception.
java.lang.RuntimeException: Request execution cancelled
Caused by: java.util.concurrent.CancellationException: Request execution cancelled

In my reproduction:

• traces-raw-pipeline successfully initialized.
• otel-logs-pipeline and service-map-pipeline repeatedly failed.
• The failed sinks retried approximately every 200 ms.
• Waiting did not make the affected sinks recover.
• Restarting only Data Prepper, without changing any configuration, caused all three sinks to initialize successfully.

This leaves the stack partially ready even though docker compose ps shows Data Prepper as running.

How can one reproduce the bug?

Start from a fresh OpenSearch volume.

Warning: docker compose down -v removes local test data created by this Compose project.

docker compose down -v --remove-orphans

docker compose up -d opensearch
docker compose ps opensearch

docker compose up -d data-prepper
docker compose ps opensearch data-prepper

Inspect the Data Prepper logs:

docker compose logs --no-log-prefix data-prepper |
  grep -E 'Initializing OpenSearch sink|Initialized OpenSearch sink|Failed to initialize OpenSearch sink|Request execution cancelled'

The affected pipelines repeatedly produce output similar to:

[service-map-pipeline-sink-worker-...] INFO  OpenSearchSink - Initializing OpenSearch sink
[otel-logs-pipeline-sink-worker-...] INFO  OpenSearchSink - Initializing OpenSearch sink

[service-map-pipeline-sink-worker-...] WARN  OpenSearchSink - Failed to initialize OpenSearch sink with a retryable exception.
java.lang.RuntimeException: Request execution cancelled
    at org.opensearch.client.RestClient.extractAndWrapCause(...)
    at org.opensearch.client.RestClient.performRequest(...)
    at org.opensearch.client.opensearch.cluster.OpenSearchClusterClient.getSettings(...)
    at org.opensearch.dataprepper.plugins.sink.opensearch.index.AbstractIndexManager.checkISMEnabled(...)
    at org.opensearch.dataprepper.plugins.sink.opensearch.index.AbstractIndexManager.checkAndCreateIndexTemplate(...)
    at org.opensearch.dataprepper.plugins.sink.opensearch.index.AbstractIndexManager.setupIndex(...)
Caused by: java.util.concurrent.CancellationException: Request execution cancelled

The Data Prepper API still lists all configured pipelines:

docker compose exec data-prepper \
  curl -s http://localhost:4900/list

Example response:

{
  "pipelines": [
    {"name": "otlp-pipeline"},
    {"name": "otel-traces-pipeline"},
    {"name": "service-map-pipeline"},
    {"name": "traces-raw-pipeline"},
    {"name": "otel-logs-pipeline"},
    {"name": "service-metrics-cortex-pipeline"}
  ]
}

Restart only Data Prepper:

docker compose restart data-prepper

After the restart, without any configuration changes, all OpenSearch sinks initialize and the OTLP source starts successfully.

What is the expected behavior?

Once OpenSearch reports healthy and Data Prepper starts:

• All configured OpenSearch sinks should initialize successfully.
• The stack should not require a manual Data Prepper restart.
• The service status should distinguish a running container from a fully initialized Data Prepper instance.
• Telemetry should not be accepted before the required sinks are ready.

What is your host/environment?

• Host OS: macOS 26.5.1, arm64
• Docker CLI: 29.4.0
• Docker Compose: v5.1.2
• Repository commit: 6e5b56e
• OpenSearch image: opensearchproject/opensearch:3.8.0
• Data Prepper image: sgguruda62324/opensearch-data-prepper:2.16.0-SNAPSHOT-rc1
• Data Prepper platform: linux/amd64
• Default memory limits:
  • OpenSearch: 2 GiB
  • Data Prepper: 1 GiB

Are there any screenshots?

N/A. The relevant log output and reproduction commands are included above.

Do you have any additional context?

During the first startup, all three OpenSearch sinks appeared to initialize concurrently. Two HTTP 409 responses involving an already-existing .opendistro-ism-config index were also observed during that initialization period.

This may indicate a race during concurrent ISM or index-template initialization, but the root cause has not yet been confirmed. Other possibilities include the Data Prepper snapshot image itself or compatibility with OpenSearch 3.8.0.

The following behavior has been confirmed:

1. The issue is reproducible with a fresh OpenSearch volume.
2. OpenSearch is healthy when it occurs.
3. Data Prepper remains Up while some sinks continuously fail.
4. Restarting only Data Prepper resolves the problem without configuration or resource changes.

Data Prepper currently has no Docker Compose health check that verifies whether all sinks have initialized. As a result, dependent services or users may treat the stack as ready while log ingestion and service-map generation are unavailable.

[kangkyunghyun]

Investigation update:

I ran an A/B test using a fresh OpenSearch volume.

• The currently configured snapshot image reproduced the repeated Request execution cancelled sink initialization failure.
• The official opensearchproject/data-prepper:2.16.0 image completed three consecutive clean startups without the failure.
• Log, trace, and service-map ingestion also succeeded with the official image.

This isolates the failure to the previously configured snapshot image, although the internal root cause within that image has not been identified.

PR #408 replaces the snapshot default with the official stable image.

[kangkyunghyun]

Follow-up: I opened #409 to track the separate Docker Compose readiness gap identified during this investigation.

#407 covers the repeated Request execution cancelled sink initialization failure associated with the previously configured snapshot image, which #408 addresses by switching to the official stable image.

#409 covers the remaining readiness issue: Docker Compose can report Data Prepper as ready while its sinks are still uninitialized and the OTLP source port is closed. This was reproduced independently with the official stable image using deliberately invalid sink configurations.