tracking: Test demo upgrades on nightly versions for 26.3.0

@NickLarsenNZ

This is testing that upgrades from the stable release to the nightly release of the operators
and products do not negatively impact the products.

Note

Record the results (issues, anomalies, or success) during the process in a comment on this issue.
Eg:

:green_circle: **airflow-scheduled-job**

The CRD had been updated and I needed to change the following in the manifest:
...

The following emojis can be used to indicate the status:

hourglass: In progress / waiting
red_circle: In progress with errors
orange_circle: Completed with minor issues or anomalies
green_circle: Completed without issues or errors

List of demos

Replace the items in the lists below with the applicable Pull Requests (if any). Also put your
GitHub handle next to the demo to indicate a particular demo was picked up for testing.

List of stacks

Similar to the demos above we need to test all stacks that don't have a demo.
Replace the items in the lists below with the applicable Pull Requests (if any). Also put your
GitHub handle next to the stack to indicate a particular stack was picked up for testing.

Tip

Some of the stacks have a tutorial to follow.

~~airflow-trino-dbt~~ (this was merged into the airflow demo and was only removed from the stacks recently: it has also been removed from the template: @adwk67 )
~~monitoring~~ The only thing that get's upgraded is Prometheus, which is not a supported tool, hence not super important to test. tracking: Test demos on nightly versions for 26.3.0 #372 makes sure that a fresh install works
~~observability~~ @NickLarsenNZ Will do it ias part of tracking: Test demos on nightly versions for 26.3.0 #372
openldap (comes up)
tutorial-openldap @Techassi

Stable to Nightly Upgrade Testing Instructions

These instructions are for deploying and completing the stable demo, and then
upgrading operators, CRDs, and products to the nightly versions well as upgrading
the operators and CRDS.

Tip

Be sure to select the stable docs version on https://docs.stackable.tech/home/stable/demos/.

# For demo testing, , install demo (stable operators) for the stable release (25.11).
stackablectl demo install <DEMO_NAME>

# For stack testing, install stack via
stackablectl stack install <STACK_NAME>

# --- IMPORTANT ---
# Run through the (still) nightly demo/stack instructions (refer to the list above).

# Install nightly version of operators
stackablectl release upgrade dev

# Bump every stacklet to the version that used by the nightly demo.
# Typically, this is the latest supported version.
kubectl patch hbaseclusters/hbase --type='json' -p='[{"op": "replace", "path": "/spec/image/productVersion", "value":"x.x.x"}]' # changed

# Check that the operator logs look good (no unexpected errors etc.)

# Run through the (still) nightly demo/stack instructions again.

@siegfriedweber

🟠 logging

🟢 Stable from scratch
- PR to adjust image: docs(demos/logging): Adjust OpenSearch dashboard image #379
🟠 Upgrade to nightly
- Initial upgrade worked
- Bumping OpenSearch 3.4.0 results in a weird mount error (@siegfriedweber wrote a guide on migration steps needed: doc: Add SDP upgrade instructions opensearch-operator#124)

🟢 airflow-scheduled-job

🟢 Stable from scratch
- Docs PR: docs(demo/airflow-scheduled-job): Update instructions #378
🟢 Upgrade to nightly
- When airflow is upgraded, you need to replace image refs (and rollout STSs) for additional containers to:
```
image: oci.stackable.tech/sdp/airflow:3.1.6-stackable0.0.0-dev
```
- Also need to edit the ConfigMap to update Spark version used by the DAG.

product upgrade steps

# Airflow 3.0.6 -> 3.1.6
kubectl -n airflow-demo patch airflowclusters/airflow --type='json' -p='[{"op": "replace", "path": "/spec/image/productVersion", "value":"3.1.6"}]'
# Kafka 4.1.0 -> 4.1.1
kubectl -n airflow-demo patch kafkaclusters/kafka --type='json' -p='[{"op": "replace", "path": "/spec/image/productVersion", "value":"4.1.1"}]'
# Opa 1.8.0 -> 1.12.3
kubectl -n airflow-demo patch opaclusters/opa-airflow --type='json' -p='[{"op": "replace", "path": "/spec/image/productVersion", "value":"1.12.3"}]'
# Spark 4.0.1 -> 4.1.1
# Need to edit the ConfigMap

🟢 signal-processing

🟢 Stable from scratch
🟢 Upgrade to nightly

🟠 end-to-end-security

🟢 Stable from scratch
- The stacklet output needs updating (I'll wait for nightly from scratch testing to get the most up2date output)
- Docs PR: docs(demos/end-to-end-security): Add adjustments for 26.3.0 #384.

🟠 Upgrade to nightly

2026-03-12T14:08:36.137Z    ERROR    opa-access-control-http-1    io.trino.plugin.opa.OpaHttpClient    OPA response for URI "http://opa-server.default.svc.cluster.local:8081/v1/data/trino/batchColumnMasks" with status code = 200 could not be deserialized
java.lang.IllegalArgumentException: Unable to create class io.trino.plugin.opa.schema.OpaBatchColumnMaskQueryResult from JSON response

This issue is fixed by manually applying: chore: Update Trino Rego rules and disable column masking otherwise #361. A release note is available here: Batch OPA column masks calls trino-operator#814 (comment)

🟢 spark-k8s-anomaly-detection-taxi-data

🟢 Stable from scratch

trino-coordinator-default-0 and trino-worker-default-0:

ERROR    main    io.trino.server.Server    Environment variable is not set: CATALOG_PREDICTION_S3_AWS_SECRET_KEY

Fixed with #382 and #383

🟢 Upgrade to nightly

upgrades

# Trino 477 -> 479
kubectl patch trinoclusters/trino --type='json' -p='[{"op": "replace", "path": "/spec/image/productVersion", "value":"479"}]'
# Opa 1.8.0 -> 1.12.3
kubectl patch opaclusters/opa --type='json' -p='[{"op": "replace", "path": "/spec/image/productVersion", "value":"1.12.3"}]'
# Superset 4.1.4 -> 6.0.0
kubectl patch supersetclusters/superset --type='json' -p='[{"op": "replace", "path": "/spec/image/productVersion", "value":"6.0.0"}]'
# Spark 4.0.1 has to stay
# kubectl patch sparkapplications/spark-ad --type='json' -p='[{"op": "replace", "path": "/spec/sparkImage/productVersion", "value":"4.1.1"}]'
# Hive: Keep on 4.0.0
# https://github.com/stackabletech/demos/blob/1f8b9349d6bbdc4cb0c94f8fa8d208c4c2c7c1f2/stacks/spark-trino-superset-s3/hive-metastore.yaml#L8-L10

kubectl port-forward spark-ad-driver 4040
- Error from server (NotFound): pods "spark-ad-driver" not found
Superset: Error: Failed to deserialize OPA policy response -> Need to set spec.clusterConfig.authorization.opa.enableColumnMasking: false on the TrinoCluster (Batch OPA column masks calls trino-operator#814).

⏳ trino-iceberg

🟢 Stable from scratch
🟢 Upgrade to nightly => Had to disable enableColumnMasking according to chore: Update Trino Rego rules and disable column masking otherwise #361

🟢 jupyterhub-pyspark-hdfs-anomaly-detection-taxi-data

🟢 Stable from scratch
🟢 Upgrade to nightly
- hdfs stays on 3.4.2
- zookeeper stays on 3.9.4
- jupyterlab references spark-connect but is still on the same version in dev although the spark-connect manifest has bumped to 4.1.1. Updated both manually, we are using a custom image. Notebook also has a reference to 4.0.1 that was bumped manually. Will open a PR to update these in dev.
fix(jupyterhub-pyspark) : Bump spark-connect-client to 4.1.1 #389

🟢 nifi-kafka-druid-water-level-data

🟢 Stable from scratch
🟢 Upgrade to nightly

🟢 hbase-hdfs-load-cycling-data

🟢 Stable from scratch
🟢 Upgrade to nightly

🔴 data-lakehouse-iceberg-trino-spark

🟢 Stable from scratch
🔴 Upgrade to nightly: Found a bug in kafka-op, as only upgrading the SPD version already broke Kafka. Reported in https://stackable-workspace.slack.com/archives/C0A777V696H/p1773654397526609. We will likely accept this and document it as known issue

Demo and stack testing completed

Navigation Menu

Repository navigation

tracking: Test demo upgrades on nightly versions for 26.3.0 #371

List of demos

List of stacks

Stable to Nightly Upgrade Testing Instructions

🟢 Stable from scratch

🟢 Upgrade to nightly

5 remaining items

🟢 jupyterhub-pyspark-hdfs-anomaly-detection-taxi-data

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

tracking: Test demo upgrades on nightly versions for 26.3.0 #371

Description

List of demos

List of stacks

Stable to Nightly Upgrade Testing Instructions

Activity

Techassi commented on Mar 12, 2026

NickLarsenNZ commented on Mar 12, 2026

Techassi commented on Mar 12, 2026

Techassi commented on Mar 12, 2026

NickLarsenNZ commented on Mar 12, 2026

🟢 Stable from scratch

🟢 Upgrade to nightly

sbernauer commented on Mar 12, 2026

5 remaining items

adwk67 commented on Mar 13, 2026

🟢 jupyterhub-pyspark-hdfs-anomaly-detection-taxi-data

NickLarsenNZ commented on Mar 13, 2026

NickLarsenNZ commented on Mar 13, 2026

sbernauer commented on Mar 16, 2026

sbernauer commented on Mar 16, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions