question: how to make trino independent of opa restarts #817

Activity

1. dervoeti commented on Mar 17, 2026

   

   dervoeti

   on Mar 17, 2026

   Member

   More actions

   Hi @maxgruber19,

   thanks for reporting this! You're right, this is a real problem and we've been looking into it.

   OPA is deployed as a DaemonSet because policy decisions need to be fast. Products like Trino make multiple serial OPA queries per request, so having an OPA agent on every node (with internalTrafficPolicy: Local) avoids cross-node network hops. The current downside is exactly what you're seeing: when the local OPA pod restarts, there's no fallback.

   We're working on a fix that will tell Kubernetes to start the new OPA pod and wait for it to become ready before terminating the old one. This eliminates the availability gap during rolling updates. We will ship that in a future SDP release, and even further in the future we can leverage a new Kubernetes feature which tells Kubernetes to prefer the local pod, but fall back to pods on other nodes if that pod is unavailable. That would be the proper fix, but we need to rely on Kubernetes 1.35 to be the minimum required version in SDP.

   What you can do very soon:
   Our release of SDP 26.3.0 is just around the corner, sadly it won't yet include the fix, but I can already tell you that we added the concept of objectOverrides in that release, which will allow you to do the following in your OpaCluster spec:

   spec:
     objectOverrides:
       - apiVersion: apps/v1
         kind: DaemonSet
         metadata:
           name: <your-opa-name>-server-<rolegroup>
           namespace: <namespace>
         spec:
           updateStrategy:
             type: RollingUpdate
             rollingUpdate:
               maxSurge: 1
               maxUnavailable: 0

   Replace <your-opa-name>, <rolegroup> and <namespace> with the actual values from your DaemonSet(s) (check with kubectl get daemonsets -A).

   That does exactly what the fix we're implementing for a future release will do (start the new pod before the old one terminates) and should fix your problem. As said, you'll have to wait a bit for the official SDP 26.3 release and then upgrade to SDP 26.3 to make use of objectOverrides, then you should be able to solve the problem that way.

   👍React with 👍1Reacted by Max Gruber
2. maxgruber19 commented on Mar 17, 2026

   

   maxgruber19

   on Mar 17, 2026

   Author

   More actions

   Thank you for that good and quick reply as usual. We're looking forward to that feature. I'll let the team know
3. Maleware commented on Mar 17, 2026

   

   Maleware

   on Mar 17, 2026

   Member

   More actions

   Hey, I'm late to the party my apologies.

   If upgrading to 26.3 is no option for you or the solution given won't fit there is another path.

   You could define and deploy a svc similar to:

   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: opa-cluster
     namespace: <namespace>  
   spec:
     internalTrafficPolicy: Cluster
     ports:
     - name: http
       port: 8081
       protocol: TCP
       targetPort: 8081
     selector:
       app.kubernetes.io/component: server
       app.kubernetes.io/instance: opa
       app.kubernetes.io/name: opa
     type: ClusterIP
   

   which noteworthy sets internalTrafficPolicy: Cluster and defines the product this svc adheres to.

   Along this you'd need to deploy a configMap like:

   ---
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: opa-cluster
     namespace: <trino-namespace>
   data:
     OPA: http://opa-cluster/.<namespace>.svc.cluster.local:8081/
   

   and point trino to it with

   authorization:
         opa:
           configMapName: opa-cluster
   

   This will make the opa-cluster svc a LoadBalancer across all pods of a specific DaemonSet. Since load will be distributed, restarts of single nodes won't degrade the whole TrinoCluster.

   👍React with 👍1Reacted by Max Gruber
4. maxgruber19 commented on Mar 17, 2026

   

   maxgruber19

   on Mar 17, 2026

   Author

   More actions

   @Maleware thats a very good point, we'll check on that as well.

   Thank you as well! I think waiting for the new release is fine for us, it's not that critical right now
5. stackable-bot

   added

   customer-request

   on Mar 18, 2026
6. sbernauer commented on Jul 15, 2026

   

   sbernauer

   on Jul 15, 2026

   Member

   More actions

   Given we implemented #819 this should work out of the box in 26.7, can we close this issue? :)

   👍React with 👍1Reacted by Max Gruber
7. maxgruber19 commented on Jul 16, 2026

   

   maxgruber19

   on Jul 16, 2026

   Author

   More actions

   Looks good, thank you for implementing that :-)

   👍React with 👍1Reacted by Sebastian Bernauer
8. maxgruber19

   closed this as completedon Jul 16, 2026