Skip to content

fix(ci): unblock disconnected OCP nightlies - #5709

Merged
openshift-merge-bot[bot] merged 16 commits into
redhat-developer:mainfrom
zdrapela:fix/ci-disconnected-nightlies
Oct 27, 2026
Merged

openshift-merge-bot[bot] merged 16 commits into
redhat-developer:mainfrom
zdrapela:fix/ci-disconnected-nightlies

Conversation

@zdrapela

@zdrapela zdrapela commented Oct 23, 2026 •

Copy link
Copy Markdown
Member

Summary

The disconnected OCP Helm and Operator nightlies (main and release-2.1) have not passed for over five weeks. Four independent gaps kept them red:

 .ci/pipelines/
 ├── lib/common.sh                       # get_release_stream moved from helm.sh
 ├── lib/disconnected/namespace.sh      # + allow_mirror_registry_egress
 ├── lib/disconnected/plugins.sh        # homepage-only -> minimal NFS plugin set
 ├── resources/disconnected/
 │   ├── dynamic-plugins-homepage.yaml  # -> dynamic-plugins-smoke.yaml (5 plugins)
 │   └── app-config-rhdh-disconnected-smoke.yaml  # + NFS app.extensions, signInPage
 ├── value_files/values_disconnected-smoke.yaml   # same plugin set + NFS config
 └── jobs/
     ├── ocp-disconnected-helm.sh       # apply egress rule; chart-name auth secret
     └── ocp-disconnected-operator.sh   # apply egress rule; CI operator catalog
  1. Backend egress to the mirror registry. The chart and the Operator default-deny backend egress except DNS, HTTPS, Redis and PostgreSQL. The CI mirror (bastion) serves TLS on port 5000, so the install-dynamic-plugins skopeo pull of the plugin catalog timed out. The CI-only allow-all rule added in fix(e2e): harden NFS settings pin and Docs coverage #5503 is applied by apply_yaml_files only, which the disconnected handlers skip. The new disconnected::allow_mirror_registry_egress applies a targeted rule for the mirror's port (Helm + Operator pod selectors) and is skipped when the mirror is served on 443 (LOCAL_DISCONNECTED).

  2. Operator job: mirror the RHDH CI catalog, not the published one. Since rhdh-operator#3822 (chore(deps): update tombi-toml/setup-tombi digest to f7cb38e #3821 on release-2.1) a filtered oc-mirror run rejects the published catalog because its default fast channel is filtered out. That catalog also has no 2.x bundles (max 1.10.6), so even an unfiltered run installed the stale 1.10 operator. CI runs now use quay.io/rhdh/iib:<stream>-v<ocp>-x86_64, where <stream> is the same common::get_release_stream value passed to --filter-versions (2.2 on main, 2.1 on release-2.1), so the catalog and the fast-X.Y channel it is filtered to cannot disagree and a stream without a published catalog fails at the pre-flight skopeo inspect. CONTAINERS_REGISTRIES_CONF redirects its registry.redhat.io/rhdh relatedImages to quay.io/rhdh for the prepare-restricted-environment.sh call only. CI runs mirror the newest CI bundle of the branch's stream: the hub image comes from the bundle (RELATED_IMAGE_backstage), so pinned TAG_NAME, IMAGE_* and CHART_VERSION settings do not apply to them. A GA verification run (registry.redhat.io with a released x.y.z chart) keeps the published Red Hat catalog, with --filter-versions '*': filtering it to fast-X.Y drops the catalog's default fast channel, which oc-mirror rejects.

  3. Smoke setup for the 2.x New Frontend System hub. The homepage-only plugin set predates the NFS migration. On 2.x the sign-in page (app-auth), sidebar (app-defaults), SCM integrations (app-integrations) and guest auth provider are dynamic plugins, so guest sign-in could not render. Both install methods now mirror that minimal digest-pinned set, plus the NFS app.extensions config and an explicit signInPage: guest.

  4. Helm registry auth secret name. The chart mounts <fullname>-dynamic-plugins-registry-auth with optional: true, where fullname is <release>-<nameOverride>. The job created <release>-dynamic-plugins-registry-auth, so install-dynamic-plugins had no auth.json and the mirror pull failed with "authentication required". The secret now uses the chart's name and the values file pins nameOverride: developer-hub.

Second round: what the first re-run exposed

The fixes above got both jobs past mirroring, plugin install and the operator catalog. Two more problems surfaced, both in pre-existing helpers, plus one upstream bug:

  1. Node drains during the deploy (Helm). disconnected::wait_machineconfigpool_updated used oc wait --for=condition=Updated=True, which returns while the MCO has not rendered the IDMS/ITMS config yet. The workers were drained a few seconds later, evicting the hub and PostgreSQL pods mid-startup. It now snapshots each pool's rendered config before the change (retrying failed reads) and waits for the MCO to pick it up before waiting for convergence.

  2. The Helm recovery restart started two backends. ensure_helm_hub_after_postgres ran oc rollout restart, keeping the old pod serving while a second backend booted against a freshly initialised database. They raced on the knex migration lock and the losing backend stayed up with readiness 503 forever. The bounce is now scale-to-zero then scale-to-one, the wait gives a pod that is still extracting plugins time to start its backend first, and the bounce refuses to scale up while an old pod remains.

  3. Operator: OLM v1 installer RBAC (upstream). OLM installs the bundle with rhdh-operator-installer, which cannot grant RBAC it does not hold. The installer ClusterRole rendered by prepare-restricted-environment.sh lags behind the bundle (pods, apiservers/ingresses, devhubplugincatalogs), so the operator ClusterRole creation is refused and the operator runs with no permissions. Tracked in RHDHBUGS-4056, fixed in fix: grant the bundle RBAC to the OLM v1 installer ServiceAccount rhdh-operator#3923 (backport chore(e2e): topology flaky test fix #3927). feat(e2e): add orchestrator RBAC tests and deployment fixes #3923 is merged: the job downloads prepare-restricted-environment.sh from rhdh-operator main at runtime, so this branch installs with the fixed installer role (the temporary verification workaround has been removed). Our wait now requires ClusterExtension Installed=True as well as the CRD, so a half-applied release fails fast with the real error instead of a misleading "deployment not created". deploy_rhdh_operator also returns as soon as rendering or applying the CR fails.

Scope

The disconnected jobs exist only on main and release-2.1, and both now target 2.x hubs: the smoke plugin set is the minimal New Frontend System set and the operator stream follows the branch. Verification of 1.x releases is not covered by these jobs.

Which issue(s) does this PR fix

Jira: https://redhat.atlassian.net/browse/RHDHBUGS-4056 (installer RBAC, fixed in redhat-developer/rhdh-operator#3923)

Failing job: https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/periodic-ci-redhat-developer-rhdh-main-e2e-ocp-disconnected-helm-nightly/2112001170120642560

Evidence

  • Before: install-dynamic-plugins times out on dial tcp 10.0.x.x:5000; the Operator job fails earlier on the default channel fast was filtered out.
  • After: live OCP 4.21 cluster, 2.x hub image (quay.io/rhdh/rhdh-hub-rhel10:next), chart 2.1-169-CI with the new values:
    • all five plugins install from OCI,
    • / renders "Select a sign-in method" with the Guest Enter button,
    • after guest sign-in the sidebar renders and the homepage shows Welcome back!.

Testing

  • shellcheck --severity=warning over all tracked .sh files: clean.
  • prettier --check over every changed .sh/.yaml: clean.
  • helm template with the rendered values: chart app-config deep-merges to the expected app.extensions + auth.providers.guest + signInPage.
  • Plugin list/package export exercised against the real :next and :2.1 catalogs (5 digest-pinned refs each).
  • Egress NetworkPolicy passes oc apply --dry-run=server.
  • registry.redhat.io/rhdh and registry.stage.redhat.io/rhdh relatedImage digests resolve through the registries.conf redirect (and fail without it).
  • Catalog check: quay.io/rhdh/iib:2.2-v4.21-x86_64 has defaultChannel: fast-2.2 and channel fast-2.2 (same digest as next-v4.21); 2.1-v4.21 mirrors that for fast-2.1.
  • MachineConfigPool wait, hub bounce and smoke plugin render helpers exercised with a stubbed oc (snapshot retry, failed reads ignored, render detected, nothing-to-roll, degraded pool, scale 0/1, no restart when Available, no scale-up while a pod remains, empty package: rejected).
  • deploy_rhdh_operator returns immediately on a failed oc apply instead of polling for five minutes.

What the checks would miss: an oc-mirror m2m regression specific to the bastion mirror. /test e2e-ocp-disconnected-{helm,operator}-nightly covers it.

Merge Danger

Door: two-way (revert restores the previous jobs).

Blast radius: CI-only, disconnected nightlies.

How to revert

git revert <commit>. The helpers are used only by the disconnected jobs.

PR acceptance criteria

  • GitHub Actions are completed and successful
  • Unit Tests are updated and passing (no unit tests cover .ci/pipelines)
  • E2E Tests are updated and passing (/test e2e-ocp-disconnected-helm-nightly, /test e2e-ocp-disconnected-operator-nightly)
  • Documentation is updated if necessary (no docs describe the disconnected jobs)
  • Add a screenshot if the change is UX/UI related

@openshift-ci
openshift-ci Bot requested review from josephca and teknaS47 October 23, 2026 14:10
@zdrapela

Copy link
Copy Markdown
Member Author

/test e2e-ocp-disconnected-helm-nightly
/test e2e-ocp-disconnected-operator-nightly

@rhdh-qodo-merge

Copy link
Copy Markdown

PR Summary by Qodo

Unblock disconnected OCP Helm and Operator nightlies

🐞 Bug fix ⚙️ Configuration changes 🕐 40+ Minutes


AI Description

• Allow disconnected backends to reach the CI mirror registry for plugin installation.
• Mirror the matching CI Operator catalog instead of the outdated published catalog for CI runs.
• Install and configure the minimal 2.x plugins needed for guest sign-in and homepage smoke checks.
Diagram

graph TD
  H["Helm nightly"] --> P["Plugin catalog"] --> M[("Mirror registry")] --> N["Backend egress"] --> B["Hub deployment"] --> S["Guest smoke"]
  O["Operator nightly"] --> C["CI catalog"] --> M
  O --> P
  H --> N
  O --> N
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Mirror the full plugin catalog
  • ➕ Would exercise more of the disconnected plugin inventory.
  • ➖ Unpublished tag references can break startup, and mirroring is substantially larger than the smoke check requires.
2. Reuse the broad CI egress policy
  • ➕ Would avoid generating a dedicated mirror-port policy.
  • ➖ The disconnected handlers do not apply that policy, and unrestricted egress weakens the test's network isolation.

Recommendation: Keep the targeted mirror-port policy and digest-pinned smoke set. They address the failing paths without broadening egress or making the nightlies depend on the full, potentially fragile plugin catalog.

Files changed (11) +292 / -71

Bug fix (5) +223 / -57
ocp-disconnected-helm.shEnable mirror egress and require the expanded smoke plugin set +7/-3

Enable mirror egress and require the expanded smoke plugin set

• Applies mirror-registry backend egress policy after namespace setup. Requires all five mirrored plugin package references before rendering Helm smoke values.

.ci/pipelines/jobs/ocp-disconnected-helm.sh

ocp-disconnected-operator.shUse the CI Operator catalog and prepare disconnected smoke resources +57/-8

Use the CI Operator catalog and prepare disconnected smoke resources

• Selects the stream- and OCP-matched CI catalog for non-published runs while retaining the published catalog for GA/RC verification. Scopes related-image registry redirects to Operator preparation, then applies mirror egress and creates the expanded smoke plugin ConfigMap.

.ci/pipelines/jobs/ocp-disconnected-operator.sh

plugins.shMirror all smoke plugins in local disconnected runs +5/-7

Mirror all smoke plugins in local disconnected runs

• Switches the local mirroring override from a homepage-only list to the shared digest-pinned smoke list. Retains its registry retry and catalog-copy behavior.

.ci/pipelines/lib/disconnected/local/plugins.sh

namespace.shAdd mirror-port backend egress NetworkPolicies +69/-0

Add mirror-port backend egress NetworkPolicies

• Generates and applies separate Helm and Operator pod policies for a mirror registry on a non-443 port. Skips the additional rule for mirrors served on port 443 and saves the manifest as a CI artifact.

.ci/pipelines/lib/disconnected/namespace.sh

plugins.shResolve and mirror five digest-pinned smoke plugins +85/-39

Resolve and mirror five digest-pinned smoke plugins

• Filters the plugin catalog to the minimal 2.x smoke set and exports each package reference for template rendering. Uses that set in standard mirroring and creates the Operator smoke plugin ConfigMap.

.ci/pipelines/lib/disconnected/plugins.sh

Refactor (1) +5 / -2
disconnected.shExport the new disconnected smoke and egress helpers +5/-2

Export the new disconnected smoke and egress helpers

• Replaces homepage-only helper exports with smoke-plugin list and package helpers. Exports the mirror-registry egress helper for both disconnected jobs.

.ci/pipelines/lib/disconnected.sh

Documentation (2) +5 / -3
local.shDescribe the local smoke-plugin mirroring path +1/-1

Describe the local smoke-plugin mirroring path

• Updates the local disconnected module description to reflect the expanded plugin set.

.ci/pipelines/lib/disconnected/local.sh

rhdh-start-disconnected-smoke.yamlClarify CI and released hub image sources +4/-2

Clarify CI and released hub image sources

• Documents how the CI and published Operator catalogs supply different hub image references and how mirrored pulls are resolved.

.ci/pipelines/resources/rhdh-operator/rhdh-start-disconnected-smoke.yaml

Other (3) +59 / -9
app-config-rhdh-disconnected-smoke.yamlConfigure Operator guest sign-in and NFS homepage +13/-0

Configure Operator guest sign-in and NFS homepage

• Explicitly selects guest sign-in and configures homepage extensions. Disables the competing community home page.

.ci/pipelines/resources/disconnected/app-config-rhdh-disconnected-smoke.yaml

dynamic-plugins-smoke.yamlDefine the Operator's minimal 2.x smoke plugin set +18/-0

Define the Operator's minimal 2.x smoke plugin set

• Adds a digest-pinned plugin template for app-auth, app-defaults, app-integrations, guest auth, and homepage. Keeps catalog defaults excluded to avoid unavailable plugin tags.

.ci/pipelines/resources/disconnected/dynamic-plugins-smoke.yaml

values_disconnected-smoke.yamlConfigure Helm guest sign-in and NFS smoke plugins +28/-9

Configure Helm guest sign-in and NFS smoke plugins

• Adds the four plugins needed alongside homepage for 2.x smoke checks. Selects guest sign-in and configures homepage extensions while continuing to exclude catalog plugin defaults.

.ci/pipelines/value_files/values_disconnected-smoke.yaml

@codecov

codecov Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 69.00%. Comparing base (36f066e) to head (0cc0355).
⚠️ Report is 5 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5709      +/-   ##
==========================================
- Coverage   75.70%   69.00%   -6.70%     
==========================================
  Files          65       58       -7     
  Lines         708      555     -153     
  Branches      100       83      -17     
==========================================
- Hits          536      383     -153     
  Misses        171      171              
  Partials        1        1              
Flag Coverage Δ
rhdh 69.00% <ø> (-6.70%) ⬇️
Components Coverage Δ
Backend plugins ∅ <ø> (∅)
Backend app 78.94% <ø> (ø)
Frontend app 61.03% <ø> (ø)

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 36f066e...0cc0355. Read the comment docs.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@rhdh-qodo-merge

rhdh-qodo-merge Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (1)



View medium (1)
🟠 **Medium**
1. Older release smoke jobs stop at mirroring 🐞 Bug
Description
DISCONNECTED_SMOKE_PLUGIN_NAMES unconditionally requires the five-plugin set described here as
necessary for the 2.x frontend, and set_smoke_plugin_packages_from_list aborts if any package is
absent. Disconnected jobs can still run against older release catalogs, including the documented 1.9
and 1.9.4 verification inputs, so those catalogs must satisfy the new 2.x package requirement before
either deployment can start.
Code

.ci/pipelines/lib/disconnected/plugins.sh[R12-15]

+readonly -a DISCONNECTED_SMOKE_PLUGIN_NAMES=(
+  "red-hat-developer-hub-backstage-plugin-app-auth"
+  "red-hat-developer-hub-backstage-plugin-app-defaults"
+  "red-hat-developer-hub-backstage-plugin-app-integrations"
Evidence
The new list is identified as a 2.x frontend requirement, every name is mandatory during export, and
both deployment modes call that export before rendering. The CI trigger documents older release
catalog inputs, while release branches also supply their version directly.

.ci/pipelines/lib/disconnected/plugins.sh[8-18]
.ci/pipelines/lib/disconnected/plugins.sh[203-240]
.ci/pipelines/trigger-nightly-job.sh[101-111]
.ci/pipelines/env_variables.sh[25-34]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The disconnected smoke plugin list requires 2.x frontend packages even when the job targets an older release catalog.
## Fix Focus Areas
- .ci/pipelines/lib/disconnected/plugins.sh[8-18]
- .ci/pipelines/lib/disconnected/plugins.sh[203-240]
- .ci/pipelines/value_files/values_disconnected-smoke.yaml[16-59]
- .ci/pipelines/resources/disconnected/dynamic-plugins-smoke.yaml[7-18]
## Recommended Fix
Select the smoke plugin set and corresponding deployment configuration according to the target catalog or hub version. Keep the older homepage configuration for releases that do not publish the five-package set.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗



Resolved findings
1. A slow pod deletion restores the backend race ✓ Resolved
Description
disconnected::ensure_helm_hub_after_postgres scales the Deployment back to one replica even when
its wait for the old pod to disappear times out. If that pod is still running after 180 seconds, the
replacement can start alongside it and recreate the concurrent database-initialization condition
this recovery path is meant to avoid.
Code

.ci/pipelines/lib/disconnected/helm.sh[R94-97]

+  if ! oc wait --for=delete pod -l "${hub_selector}" -n "${namespace}" --timeout=180s; then
+    log::warn "Old hub pods still present after 180s — scaling back up anyway"
+  fi
+  if ! oc scale "deployment/${hub_deploy}" --replicas=1 -n "${namespace}"; then
Evidence
The function scales down, warns rather than stopping when pod deletion times out, and then
immediately scales up. The disconnected Helm job invokes this function as its post-deployment
recovery gate.

.ci/pipelines/lib/disconnected/helm.sh[85-102]
.ci/pipelines/jobs/ocp-disconnected-helm.sh[238-242]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The recovery scales a new backend up even when the old backend pod has not disappeared.
## Fix Focus Areas
- .ci/pipelines/lib/disconnected/helm.sh[88-99]
## Recommended Fix
Do not initiate scale-up while the old backend remains. Treat deletion timeout as a recovery failure or continue waiting with a bounded, explicit failure path.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


2. GA operator verification on main fails at mirroring ✓ Resolved
Description
filter_versions now always comes from common::get_release_stream, which reads package.json
(2.2) on main, so a pinned GA chart or image no longer affects it, while non-CI runs still mirror
the published redhat-operator-index. A documented GA verification run on main (`--image-registry
registry.redhat.io --tag 1.9.4`) therefore asks oc-mirror for 2.2 bundles from a catalog whose
newest bundle is 1.10.6, so the operator job either fails or installs nothing; before this change
the filter came from CHART_VERSION (1.9).
Code

.ci/pipelines/jobs/ocp-disconnected-operator.sh[R62-64]

+  local filter_versions
+  filter_versions=$(common::get_release_stream) || return 1
+  log::info "filter-versions=${filter_versions} (from RELEASE_BRANCH_NAME=${RELEASE_BRANCH_NAME})"
Evidence
On main, env_variables.sh sets RELEASE_VERSION=next, and package.json has version 2.2.0, so
get_release_stream returns 2.2. trigger-nightly-job.sh documents GA verification on the main
nightly job with registry.redhat.io and tag 1.9.4. That makes use_ci_catalog false, so the
published index is used with filter 2.2. The PR description itself says the published catalog's
highest bundle is 1.10.6.

.ci/pipelines/trigger-nightly-job.sh[28-32]
.ci/pipelines/env_variables.sh[30-34]
package.json[3-3]
.ci/pipelines/jobs/ocp-disconnected-operator.sh[57-114]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
When the published Red Hat operator catalog is used (GA/RC verification, `use_ci_catalog=false`), `filter_versions` still comes from the checkout branch stream (2.2 on main). The published catalog has no 2.2 bundles, so GA verification of a pinned older release breaks.

## Fix Focus Areas
- .ci/pipelines/jobs/ocp-disconnected-operator.sh[57-114]

## Recommended Fix
Work out `use_ci_catalog` first. When it is false and CHART_VERSION (or TAG_NAME) matches `^([0-9]+\.[0-9]+)`, set `filter_versions` to that major.minor, as the code did before. Use `common::get_release_stream` only for CI-catalog runs. Log which source was used.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


3. Deployments can start before nodes finish rolling ✓ Resolved
Description
disconnected::wait_mcp_updated treats any rendered-config difference as the change it is waiting
for, then returns as soon as the currently rendered configs converge. When the Helm job applies IDMS
and ITMS separately, the first change can finish rolling before the second has been rendered,
allowing workload deployment to begin ahead of that later node rollout.
Code

.ci/pipelines/lib/disconnected/mirror.sh[R310-312]

+      if [[ "$(disconnected::snapshot_mcp_rendered_configs)" != "${before_snapshot}" ]]; then
+        rendered=1
+        break
Evidence
The Helm job applies IDMS and, when present, ITMS as separate operations but invokes the wait only
once. The new wait accepts the first snapshot difference and subsequently checks only whether the
pools have converged to their current spec, without confirming that both applied changes were
incorporated.

.ci/pipelines/jobs/ocp-disconnected-helm.sh[127-144]
.ci/pipelines/lib/disconnected/mirror.sh[305-332]
.ci/pipelines/lib/disconnected/mirror.sh[270-285]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The MCP wait can finish after the first of two separately applied mirror configurations rolls out, before the second configuration is rendered.
## Fix Focus Areas
- .ci/pipelines/lib/disconnected/mirror.sh[305-332]
- .ci/pipelines/jobs/ocp-disconnected-helm.sh[127-144]
## Recommended Fix
Track each applied change through rendering and convergence, or otherwise establish that the final desired mirror configuration has been observed before declaring the rollout complete.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


4. Failed pool reads can bypass the node wait ✓ Resolved
Description
disconnected::snapshot_mcp_rendered_configs suppresses a failed oc get and returns an empty
successful result. If the pre-change read fails, the wait skips render detection; if a read during
detection fails, its empty result looks like a changed configuration, so either path can accept an
already-converged state before the requested change rolls out.
Code

.ci/pipelines/lib/disconnected/mirror.sh[R262-265]

+disconnected::snapshot_mcp_rendered_configs() {
+  oc get machineconfigpool \
+    -o jsonpath='{range .items[*]}{.metadata.name}={.spec.configuration.name}{"\n"}{end}' \
+    2> /dev/null || true
Evidence
The snapshot helper forces success on oc get failure. Both jobs capture its output without
validation, while the wait skips its render phase for an empty initial snapshot and compares later
empty results against the nonempty baseline as if they were a new render.

.ci/pipelines/lib/disconnected/mirror.sh[259-266]
.ci/pipelines/lib/disconnected/mirror.sh[300-315]
.ci/pipelines/jobs/ocp-disconnected-helm.sh[124-144]
.ci/pipelines/jobs/ocp-disconnected-operator.sh[145-165]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
An API read failure becomes an empty MCP snapshot, which can skip or falsely satisfy render detection.
## Fix Focus Areas
- .ci/pipelines/lib/disconnected/mirror.sh[262-265]
- .ci/pipelines/lib/disconnected/mirror.sh[301-315]
- .ci/pipelines/jobs/ocp-disconnected-helm.sh[124-125]
- .ci/pipelines/jobs/ocp-disconnected-operator.sh[145-165]
## Recommended Fix
Return an error for failed or empty MCP snapshots, check the initial snapshot in both jobs, and retry unsuccessful reads during render detection without interpreting them as configuration changes.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


5. RC operator runs select the wrong stream ✓ Resolved
Description
The new non-Red-Hat-registry branch builds the operator catalog tag from RELEASE_VERSION rather
than the pinned image or chart version. For the documented RC invocation on main, the registry
remains quay.io and RELEASE_VERSION is next, while the image and derived operator-bundle
filter are 1.9, so preparation is given a next catalog for a 1.9 verification run.
Code

.ci/pipelines/jobs/ocp-disconnected-operator.sh[R99-100]

+    index_image="quay.io/rhdh/iib:${RELEASE_VERSION}-v${ocp_version}-x86_64"
+    skopeo inspect --raw "docker://${index_image}" > /dev/null || {
Evidence
The documented RC trigger leaves the image registry at its Quay default while pinning a 1.9 image.
On main, RELEASE_VERSION is next; the job derives a 1.9 filter from the pinned chart but now
selects an IIB tag using next.

.ci/pipelines/trigger-nightly-job.sh[101-105]
.ci/pipelines/env_variables.sh[25-34]
.ci/pipelines/env_variables.sh[59-66]
.ci/pipelines/lib/helm.sh[146-161]
.ci/pipelines/jobs/ocp-disconnected-operator.sh[58-76]
.ci/pipelines/jobs/ocp-disconnected-operator.sh[93-100]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Pinned RC runs from main choose an operator catalog from the checkout branch instead of the release being verified.
## Fix Focus Areas
- .ci/pipelines/jobs/ocp-disconnected-operator.sh[58-76]
- .ci/pipelines/jobs/ocp-disconnected-operator.sh[91-103]
## Recommended Fix
Derive or explicitly accept the operator catalog stream from the pinned verification target. Ensure it agrees with the version passed to the operator-bundle filter, while retaining `next` for unpinned main nightlies.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


6. Backend pods gain unrestricted port egress ✗ Dismissed
Description
allow_mirror_registry_egress limits its new policies by port but supplies no destination, so the
selected backend pods can connect to any address on the mirror’s port. When the CI mirror uses port
5000, this widens the disconnected namespace’s default-deny boundary beyond traffic to the bastion.
Code

.ci/pipelines/lib/disconnected/namespace.sh[R159-162]

+  egress:
+    - ports:
+        - port: ${registry_port}
+          protocol: TCP
Evidence
The function parses the mirror host, but both emitted NetworkPolicy egress rules contain only
ports and no to clause. The existing CI policy documents that these backends otherwise have
restricted egress.

.ci/pipelines/lib/disconnected/namespace.sh[130-142]
.ci/pipelines/lib/disconnected/namespace.sh[150-178]
.ci/pipelines/resources/network_policy/netpol-ci-allow-backend-egress.yaml[1-5]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The new backend egress policies allow connections to any destination on the mirror port, rather than only to the mirror.
## Fix Focus Areas
- .ci/pipelines/lib/disconnected/namespace.sh[130-181]
## Recommended Fix
Add destination restrictions for both generated policies, using the mirror’s reachable IP addresses or suitable pod/namespace selectors. Preserve the port restriction and account for every address the bastion can resolve to.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


7. Exported smoke helpers lose the plugin list in child shells ✓ Resolved
Description
DISCONNECTED_SMOKE_PLUGIN_NAMES replaces the exported scalar DISCONNECTED_HOMEPAGE_PLUGIN_NAME
with a readonly bash array, and bash cannot export arrays. Yet write_smoke_plugin_list and
set_smoke_plugin_packages_from_list are still exported with export -f so they can run in child
shells. If one of them is ever called through timeout bash -c (the reason the export block
exists), the array is empty there. write_digest_plugin_list then gets an empty filter and keeps
every digest-pinned catalog plugin, and the package loop exits early. It logs 'Exported 0' without
setting any *_PLUGIN_PACKAGE variable.
Code

.ci/pipelines/lib/disconnected/plugins.sh[R12-18]

+readonly -a DISCONNECTED_SMOKE_PLUGIN_NAMES=(
+  "red-hat-developer-hub-backstage-plugin-app-auth"
+  "red-hat-developer-hub-backstage-plugin-app-defaults"
+  "red-hat-developer-hub-backstage-plugin-app-integrations"
+  "red-hat-developer-hub-backstage-plugin-homepage"
+  "backstage-plugin-auth-backend-module-guest-provider"
+)
Evidence
The export block says functions are exported for subshell usage (e.g. timeout bash -c), and the
old code exported DISCONNECTED_HOMEPAGE_PLUGIN_NAME for exactly that reason. The new array is
readonly and never exported, and bash cannot export arrays anyway. The functions that read it are
exported in disconnected.sh lines 69-71. No current caller runs them in a child shell, so this is a
latent hazard rather than a live failure.

.ci/pipelines/lib/disconnected.sh[46-72]
.ci/pipelines/lib/disconnected/plugins.sh[199-241]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`DISCONNECTED_SMOKE_PLUGIN_NAMES` is a bash array, and bash cannot export arrays. The exported functions `disconnected::write_smoke_plugin_list` and `disconnected::set_smoke_plugin_packages_from_list` would see an empty list in a child bash. They would then mirror the full catalog and export no package variables, without reporting an error.

## Fix Focus Areas
- .ci/pipelines/lib/disconnected/plugins.sh[12-18]
- .ci/pipelines/lib/disconnected/plugins.sh[203-228]

## Recommended Fix
Also export a space-separated scalar, for example `export DISCONNECTED_SMOKE_PLUGIN_NAMES_STR="${DISCONNECTED_SMOKE_PLUGIN_NAMES[*]}"`, and have the functions read from it. Alternatively, make `set_smoke_plugin_packages_from_list` fail when the list is empty or when `exported` is 0.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

@zdrapela

Copy link
Copy Markdown
Member Author

/test e2e-ocp-disconnected-helm-nightly
/test e2e-ocp-disconnected-operator-nightly

@github-actions

Copy link
Copy Markdown
Contributor

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

@zdrapela

Copy link
Copy Markdown
Member Author

/test e2e-ocp-disconnected-helm-nightly
/test e2e-ocp-disconnected-operator-nightly

@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

@github-actions

Copy link
Copy Markdown
Contributor

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

@zdrapela

Copy link
Copy Markdown
Member Author

/test e2e-ocp-disconnected-helm-nightly
/test e2e-ocp-disconnected-operator-nightly

@zdrapela
zdrapela force-pushed the fix/ci-disconnected-nightlies branch from 3a2bd03 to 18f4dc8 Compare October 26, 2026 07:57
@zdrapela

Copy link
Copy Markdown
Member Author

/test e2e-ocp-disconnected-helm-nightly
/test e2e-ocp-disconnected-operator-nightly

@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

@zdrapela
zdrapela force-pushed the fix/ci-disconnected-nightlies branch from 6164795 to edd3dba Compare October 26, 2026 13:54
@github-actions

Copy link
Copy Markdown
Contributor

The container image build workflow finished with status: cancelled.

The Helm and Operator disconnected nightlies have not passed for over five
weeks. Three independent gaps kept them red:

- Backend egress to the mirror registry. The chart and the Operator both
  default-deny backend egress except DNS, HTTPS, Redis, and PostgreSQL, while
  the CI mirror serves plain HTTP on port 5000. install-dynamic-plugins timed
  out pulling the catalog index from the bastion.
- Operator catalog. A filtered oc-mirror run against the published Red Hat
  catalog rejects it since rhdh-operator#3822 removed the default `fast`
  channel, and that catalog carries no 2.x bundles anyway. CI runs now mirror
  the RHDH CI catalog (quay.io/rhdh/iib:<stream>-v<ocp>-x86_64); its
  registry.redhat.io/rhdh relatedImages are redirected to quay.io/rhdh with
  CONTAINERS_REGISTRIES_CONF for the prepare-restricted-environment.sh call.
- Smoke plugin set. On the 2.x New Frontend System hub the sign-in page,
  sidebar, SCM integrations, and guest auth provider are dynamic plugins, so
  the homepage-only set could not render a login. Mirror the minimal digest
  pinned set and configure the NFS extensions and signInPage in the app
  configs.

The 2.x hub image and the new plugin set were verified on a live OCP cluster:
guest sign-in, sidebar, and the "Welcome back!" homepage heading all render.

Assisted-by: OpenCode
- Pick the operator catalog from the chart build type instead of the image
  registry alone: a CI chart (x.y-N-CI) mirrors the RHDH CI catalog, a
  released chart (x.y.z) keeps the published catalog. A pinned RC/GA
  verification on main (TAG_NAME and CHART_VERSION 1.9.4) was handed a `next`
  catalog for a 1.9 channel filter.
- Export the smoke plugin list as a space-separated scalar. bash cannot export
  arrays, so the `export -f` helpers saw an empty list in a child shell and
  would have mirrored every digest-pinned plugin. Fail when no package
  variable was exported.
- Document why the mirror egress rule is scoped by port and not by
  destination: the bastion address is assigned per run and reachable by DNS
  name only, so an ipBlock would break whenever the bastion is replaced.

Assisted-by: OpenCode
The chart mounts <fullname>-dynamic-plugins-registry-auth, and fullname is
<release>-<nameOverride> with nameOverride pinned to developer-hub. The job
created rhdh-dynamic-plugins-registry-auth instead, so the optional volume was
never mounted, install-dynamic-plugins ran without auth.json, and skopeo failed
the mirror pull with "authentication required". Pin nameOverride in the values
and create the secret under the chart's name.

Verified on a live OCP 4.21 cluster (LOCAL_DISCONNECTED): the init container
now pulls the catalog index and all five smoke plugins through the mirror.

Assisted-by: OpenCode
Helm: after the IDMS/ITMS apply, `oc wait machineconfigpool
--for=condition=Updated=True` returned while the MCO had not rendered the new
config yet. The nodes were drained a few seconds later, while the hub and
PostgreSQL pods were starting, so they were evicted mid-startup. The hub
recovery path then ran `oc rollout restart`, starting a second backend next to
the first against a freshly initialised database; the two raced on the knex
migration lock and the losing backend stayed up with readiness 503 forever.

- snapshot each pool's rendered config before the IDMS/ITMS and
  additionalTrustedCA changes, and wait for the MCO to render before waiting
  for convergence (poll Updated + updatedMachineCount; warn and continue when
  nothing is rolling)
- start the hub restart grace period only once the backend container has
  started, and replace the rollout restart with scale to 0, wait for the pods
  to be gone, scale to 1, so two backends never overlap

Operator: OLM v1 installs the bundle with the rhdh-operator-installer
ServiceAccount, which cannot grant RBAC it does not hold itself. The installer
ClusterRole rendered by prepare-restricted-environment.sh lags behind
config/rbac/role.yaml (pods, apiservers/ingresses, devhubplugincatalogs), so
the operator ClusterRole creation is refused and the operator ends up running
with no permissions. The half-applied release still creates the CRD, so the
CRD-only wait passed and the job failed later with a misleading message.

- require ClusterExtension Installed=True as well as the CRD, log the install
  condition at info level, and dump OLM v1 status when the Backstage CR
  produces no deployment
- TEMPORARY: grant the installer the operator's own ClusterRole rules from the
  same release branch before prepare runs, so the first install attempt
  succeeds. Remove once redhat-developer/rhdh-operator ships the fix.

Verified the RBAC workaround on a live OCP cluster: the same ClusterRole
creation is refused before the grant and succeeds after it.

Assisted-by: OpenCode
Point the temporary OLM v1 installer workaround at
redhat-developer/rhdh-operator#3923 (backport redhat-developer#3924) and RHDHBUGS-4056, so
the patch is easy to find and remove once the fix is released.

Assisted-by: OpenCode
redhat-developer/rhdh-operator#3923 is merged, so the
prepare-restricted-environment.sh fetched from rhdh-operator main renders an
installer ClusterRole that covers the bundle permissions (pods,
apiservers/ingresses, devhubplugincatalogs). The disconnected operator job
downloads that script at runtime, so the workaround is no longer needed on
main.

release-2.1 keeps the workaround until redhat-developer#3924 merges, because that job fetches
the script from rhdh-operator release-2.1.

Assisted-by: OpenCode
The operator job took the bundle filter from CHART_VERSION while the catalog tag
came from RELEASE_VERSION, so a pinned run on main (for example an RC tag) asked
oc-mirror for bundles the `next` catalog does not carry.

Both now come from the branch: RELEASE_VERSION for the catalog tag (next on
main, 2.1 on release-2.1) and common::get_release_stream for the filter, which
needs a major.minor and resolves main to the package.json stream (2.2).

Move that helper out of lib/helm.sh while at it: the operator and upgrade jobs
use it too, so it lives in lib/common.sh as common::get_release_stream.

Assisted-by: OpenCode
Drop the comments that only describe the PR history or restate the code
(merged rhdh-operator PRs, the NFS plugin rationale repeated in three files,
snapshot double-comments, a duplicated secret-mount paragraph). Keep the notes
that explain non-obvious behaviour: the guest sign-in page, nameOverride
coupling, CI vs released catalogs, the registries.conf redirect and the
additionalTrustedCA node roll.

Also log the configured MachineConfigPool timeout instead of a hardcoded 20m,
and correct the mirror egress rationale: MIRROR_REGISTRY_URL is the bastion's
public EC2 DNS name, which resolves differently from inside the cluster, so
there is no stable ipBlock to use.

Assisted-by: OpenCode
@zdrapela
zdrapela force-pushed the fix/ci-disconnected-nightlies branch from edd3dba to cdae8e9 Compare October 26, 2026 14:00
@github-actions

Copy link
Copy Markdown
Contributor

The container image build workflow finished with status: cancelled.

@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

@zdrapela

Copy link
Copy Markdown
Member Author

/agentic_review

Comment thread .ci/pipelines/lib/disconnected/helm.sh Resolved
The operator job picked the catalog tag from RELEASE_VERSION and the
--filter-versions value from common::get_release_stream. On main those are
"next" and the package.json stream (2.2), which agree only while rhdh and
rhdh-operator bump together; when one side lags, prepare keeps a fast-X.Y
channel the catalog does not carry, or oc-mirror rejects the catalog because
its default channel is filtered out.

Use filter_versions for the tag as well. Both catalog tags resolve to the same
digest today (iib:next-v4.21 == iib:2.2-v4.21), so nightlies are unchanged,
and a stream without a published catalog now fails at the skopeo check with a
readable error.

Assisted-by: OpenCode
- list jq in common.sh dependencies: the moved get_release_stream uses it
- rename mcp_all_converged to the underscore prefix the other private
  helpers use
- drop the unreachable exported -eq 0 branch in the smoke plugin export
- document the hub grace period as it behaves: it covers the backend start
  wait, then the Available wait

Assisted-by: OpenCode
@zdrapela

Copy link
Copy Markdown
Member Author

/test e2e-ocp-disconnected-helm-nightly
/test e2e-ocp-disconnected-operator-nightly

@github-actions

Copy link
Copy Markdown
Contributor

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

@zdrapela

Copy link
Copy Markdown
Member Author

/config

@rhdh-qodo-merge

Copy link
Copy Markdown
🛠️ Wiki configuration file settings:

🛠️ Local configuration file settings:
 

🛠️ Global configuration file settings:

🛠️ PR-Agent final configurations:
==================== CONFIG ====================
config.fixer_agent_provenance = {'enable': 'platform', 'enable_auto_fix': 'platform', 'enable_fixer_agent': 'platform'}  
config.portal_base_url = 'https://app.qodo.ai'  
config.is_new_pr = False  
config.is_auto_command = False  
config.organization_id = 'redhat.com'  
config.qodo_llm_gateway_metadata_tags = True  
config.fallback_models = ['openai/anthropic/claude-sonnet-5-5_thinking', 'openai/multi-provider/gpt-5.6-terra_thinking', 'openai/xai/grok-4.6_thinking']  
config.pr_compliance = {'ENABLE_RULES_PLATFORM': True}  
config.model_medium = 'openai/multi-provider/gpt-5.6-terra_thinking'  
config.model_turbo = 'openai/multi-provider/gpt-5.6-luna_thinking'  
config.fetch_github_apps_from_platform = True  
config.ignore_pr_authors = 'REDACTED'  
config.model_reasoning = 'openai/vertex_ai/gemini-4-argon'  
config.invite_only_mode = True  
config.model = 'openai/multi-provider/gpt-6-sol'  
config.use_async_clone = True  
config.treat_llm_quota_error_as_healthy = False  
config.git_provider = 'github'  
config.publish_output = True  
config.publish_output_no_suggestions = True  
config.publish_output_progress = True  
config.enable_v1_deprecation_banner = True  
config.enable_legacy_command_qodo_banner = True  
config.sunset_bypass_commands = []  
config.ignore_commands = []  
config.verbosity_level = 0  
config.log_level = 'INFO'  
config.publish_logs = False  
config.debug_mode = False  
config.use_wiki_settings_file = True  
config.use_repo_settings_file = True  
config.use_global_settings_file = True  
config.use_global_wiki_settings_file = False  
config.global_settings_file_overrides_platform = False  
config.gate_pr_supplied_ambient_models = True  
config.pr_supplied_ambient_model_allowlist = []  
config.disable_auto_feedback = False  
config.ai_timeout = 300  
config.obfuscation_max_input_chars = 200000  
config.response_language = 'en-US'  
config.clone_repo_instead_of_fetch = True  
config.always_clone = False  
config.add_repo_metadata = True  
config.add_repo_metadata_resolve_references = False  
config.add_repo_metadata_max_resolved_lines = 1500  
config.clone_repo_time_limit = 600  
config.clone_transport_retry = False  
config.publish_inline_comments_fallback_batch_size = 5  
config.publish_inline_comments_fallback_sleep_time = 2  
config.max_model_tokens = 32000  
config.custom_model_max_tokens = -1  
config.patch_extension_skip_types = ['.md', '.txt']  
config.extra_allowed_extensions = []  
config.allow_dynamic_context = True  
config.allow_forward_dynamic_context = True  
config.max_extra_lines_before_dynamic_context = 12  
config.patch_extra_lines_before = 5  
config.patch_extra_lines_after = 1  
config.ai_handler = 'litellm'  
config.qodo_llm_gateway_user_header = ''  
config.qodo_llm_gateway_user_header_field = 'sender'  
config.cli_mode = False  
config.disable_platform_features = False  
config.trial_git_org_max_invokes_per_month = 30  
config.trial_ratio_close_to_limit = 0.8  
config.quota_just_exceeded_message = '### Qodo reviews are paused for this user.\n\nTroubleshooting steps vary by plan [Learn more →](https://docs.qodo.ai/subscription-plans#what-you%E2%80%99ll-see-when-reviews-are-paused)\n\n\n**On a Teams plan?**\nReviews resume once this user has a paid seat *and* their Git account is linked in Qodo.\n[Link Git account →](https://docs.qodo.ai/subscription-plans#linking-a-git-account)\n\n**Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?**\nThese require an Enterprise plan - Contact us\n[Contact us →](https://docs.qodo.ai/qodo-support#support)\n'  
config.billing_trial_expiring_message = '**ⓘ Your Qodo trial ends soon.** Ask your workspace admin to set up billing to keep reviews running after the trial. [Manage billing](https://app.qodo.ai/account/billing/manage-subscription?traffic_source=pr_comment)'  
config.billing_payment_failure_message = "**ⓘ A recent Qodo payment didn't go through.** Ask your workspace admin to update payment details to keep reviews running. [Manage billing](https://app.qodo.ai/account/billing/manage-subscription?traffic_source=pr_comment)"  
config.trial_expiring_notice_enabled = True  
config.payment_failure_notice_enabled = True  
config.enable_per_org_invite_only = False  
config.enable_request_access_msg_on_new_pr = False  
config.check_also_invites_field = False  
config.allowed_users = []  
config.disable_checkboxes = False  
config.output_relevant_configurations = False  
config.large_patch_policy = 'clip'  
config.temperature = 0.2  
config.allow_dynamic_context_ab_testing = False  
config.choose_dynamic_context_ab_testing_ratio = 0.5  
config.ignore_pr_title = ['^\\[Auto\\]', '^Auto']  
config.ignore_pr_target_branches = []  
config.ignore_pr_source_branches = []  
config.ignore_pr_labels = []  
config.ignore_ticket_labels = []  
config.allow_only_specific_folders = []  
config.ignore_repositories = []  
config.rules_platform_excluded_repositories = []  
config.ignore_bot_pr = True  
config.ignore_language_framework = []  
config.enable_ai_metadata = True  
config.enable_description_image_reachability_probe = True  
config.read_file_max_image_bytes = 5242880  
config.max_tickets = 10  
config.max_tickets_chars = 8000  
config.extract_tickets_from_pr_title = False  
config.max_tickets_from_title = 1  
config.title_extraction_skip_patterns = ['^revert\\b', '^merge\\b', '^release\\b', '^backport\\b', '^bump\\b']  
config.prevent_any_approval = False  
config.prevent_any_block = False  
config.enable_comment_approval = False  
config.enable_auto_approval = False  
config.auto_approve_for_low_review_effort = -1  
config.auto_approve_for_no_suggestions = False  
config.ensure_ticket_compliance = False  
config.new_diff_format = True  
config.new_diff_format_add_external_references = True  
config.tasks_queue_ttl_from_dequeue_in_seconds = 1200  
config.pr_commands = ['/agentic_describe', '/agentic_review']  
config.push_commands = ['/agentic_review']  
config.handle_pr_actions = ['opened', 'reopened', 'ready_for_review']  
config.handle_push_trigger = False  
config.feedback_on_draft_pr = False  
config.enable_custom_labels = False  
config.custom_labels_discovery_prefixes = ['pr_agent:', 'qodo:']  

==================== PR_REVIEWER ====================
pr_reviewer.require_score_review = False  
pr_reviewer.require_tests_review = True  
pr_reviewer.require_estimate_effort_to_review = True  
pr_reviewer.require_can_be_split_review = False  
pr_reviewer.require_security_review = True  
pr_reviewer.require_todo_scan = False  
pr_reviewer.require_ticket_analysis_review = True  
pr_reviewer.require_ticket_labels = False  
pr_reviewer.require_no_ticket_labels = False  
pr_reviewer.check_pr_additional_content = False  
pr_reviewer.persistent_comment = True  
pr_reviewer.extra_instructions = ''  
pr_reviewer.final_update_message = True  
pr_reviewer.enable_review_labels_security = True  
pr_reviewer.enable_review_labels_effort = True  
pr_reviewer.enable_help_text = False  

==================== PR_COMPLIANCE ====================
pr_compliance.enabled = True  
pr_compliance.enable_rules_platform = True  
pr_compliance.rule_providers = []  
pr_compliance.rule_providers_max_rules = 0  
pr_compliance.filter_rules_by_history_band = False  
pr_compliance.assign_skill_rules_to_skill_findings = True  
pr_compliance.enable_security_section = True  
pr_compliance.enable_ticket_section = True  
pr_compliance.enable_codebase_duplication_section = True  
pr_compliance.enable_custom_compliance_section = True  
pr_compliance.require_ticket_analysis_review = True  
pr_compliance.allow_repo_pr_compliance = True  
pr_compliance.enable_global_pr_compliance = True  
pr_compliance.local_wiki_compliance_str = ''  
pr_compliance.global_wiki_pr_compliance = ''  
pr_compliance.local_repo_compliance_str = ''  
pr_compliance.global_repo_pr_compliance_str = ''  
pr_compliance.global_compliance_str = ''  
pr_compliance.enable_generic_custom_compliance_checklist = True  
pr_compliance.persist_generic_custom_compliance_checklist = False  
pr_compliance.display_no_compliance_only = False  
pr_compliance.enable_security_compliance = True  
pr_compliance.enable_update_pr_compliance_checkbox = True  
pr_compliance.enable_todo_scan = False  
pr_compliance.enable_ticket_labels = False  
pr_compliance.enable_no_ticket_labels = False  
pr_compliance.check_pr_additional_content = False  
pr_compliance.enable_compliance_labels_security = True  
pr_compliance.enable_user_defined_compliance_labels = True  
pr_compliance.enable_estimate_effort_to_review = True  
pr_compliance.max_rag_components_to_analyze = 5  
pr_compliance.min_component_size = 5  
pr_compliance.persistent_comment = True  
pr_compliance.enable_help_text = False  
pr_compliance.extra_instructions = ''  

==================== PR_DESCRIPTION ====================
pr_description.publish_labels = False  
pr_description.add_original_user_description = True  
pr_description.generate_ai_title = False  
pr_description.extra_instructions = ''  
pr_description.enable_pr_type = True  
pr_description.final_update_message = True  
pr_description.enable_help_text = False  
pr_description.enable_help_comment = False  
pr_description.bring_latest_tag = False  
pr_description.enable_pr_diagram = True  
pr_description.pr_diagram_direction = 'LR'  
pr_description.publish_description_as_comment = False  
pr_description.publish_description_as_comment_persistent = True  
pr_description.enable_semantic_files_types = True  
pr_description.collapsible_file_list = 'adaptive'  
pr_description.collapsible_file_list_threshold = 8  
pr_description.inline_file_summary = False  
pr_description.use_description_markers = False  
pr_description.include_generated_by_header = True  
pr_description.enable_large_pr_handling = True  
pr_description.max_ai_calls = 4  
pr_description.auto_create_ticket = False  

==================== PR_AGENTIC_DESCRIPTION ====================
pr_agentic_description.enable_file_changes_section = True  
pr_agentic_description.file_listing_style = 'cards'  
pr_agentic_description.publish_as_comment = True  

==================== PR_CODE_SUGGESTIONS ====================
pr_code_suggestions.suggestions_depth = 'regular'  
pr_code_suggestions.commitable_code_suggestions = False  
pr_code_suggestions.decouple_hunks = False  
pr_code_suggestions.dual_publishing_score_threshold = -1  
pr_code_suggestions.focus_only_on_problems = True  
pr_code_suggestions.allow_thumbs_up_down = False  
pr_code_suggestions.enable_suggestion_type_reuse = False  
pr_code_suggestions.enable_more_suggestions_checkbox = True  
pr_code_suggestions.high_level_suggestions_enabled = True  
pr_code_suggestions.extra_instructions = ''  
pr_code_suggestions.enable_help_text = False  
pr_code_suggestions.show_extra_context = False  
pr_code_suggestions.persistent_comment = True  
pr_code_suggestions.max_history_len = 5  
pr_code_suggestions.apply_suggestions_checkbox = True  
pr_code_suggestions.enable_chat_in_code_suggestions = True  
pr_code_suggestions.apply_limit_scope = True  
pr_code_suggestions.suggestions_score_threshold = 0  
pr_code_suggestions.new_score_mechanism = True  
pr_code_suggestions.new_score_mechanism_th_high = 9  
pr_code_suggestions.new_score_mechanism_th_medium = 7  
pr_code_suggestions.discard_unappliable_suggestions = False  
pr_code_suggestions.num_code_suggestions_per_chunk = 3  
pr_code_suggestions.num_best_practice_suggestions = 2  
pr_code_suggestions.max_number_of_calls = 3  
pr_code_suggestions.demand_code_suggestions_self_review = False  
pr_code_suggestions.code_suggestions_self_review_text = '**Author self-review**: I have reviewed the PR code suggestions, and addressed the relevant ones.'  
pr_code_suggestions.approve_pr_on_self_review = False  
pr_code_suggestions.fold_suggestions_on_self_review = True  
pr_code_suggestions.publish_post_process_suggestion_impact = True  
pr_code_suggestions.wiki_page_accepted_suggestions = True  
pr_code_suggestions.simplify_response = True  

==================== PR_CUSTOM_PROMPT ====================
pr_custom_prompt.prompt = 'The code suggestions should focus only on the following:\n- ...\n- ...\n...\n'  
pr_custom_prompt.suggestions_score_threshold = 0  
pr_custom_prompt.num_code_suggestions_per_chunk = 4  
pr_custom_prompt.self_reflect_on_custom_suggestions = True  
pr_custom_prompt.enable_help_text = False  

==================== PR_ADD_DOCS ====================
pr_add_docs.extra_instructions = ''  
pr_add_docs.docs_style = 'Sphinx'  
pr_add_docs.file = ''  
pr_add_docs.class_name = ''  

==================== PR_UPDATE_CHANGELOG ====================
pr_update_changelog.push_changelog_changes = False  
pr_update_changelog.extra_instructions = ''  
pr_update_changelog.add_pr_link = True  
pr_update_changelog.skip_ci_on_push = True  

==================== PR_ANALYZE ====================
pr_analyze.enable_help_text = False  

==================== PR_TEST ====================
pr_test.enable = True  
pr_test.extra_instructions = ''  
pr_test.testing_framework = ''  
pr_test.num_tests = 3  
pr_test.avoid_mocks = True  
pr_test.file = ''  
pr_test.class_name = ''  
pr_test.enable_help_text = False  

==================== PR_IMPROVE_COMPONENT ====================
pr_improve_component.num_code_suggestions = 4  
pr_improve_component.extra_instructions = ''  
pr_improve_component.file = ''  
pr_improve_component.class_name = ''  

==================== REVIEW_AGENT ====================
review_agent.enabled = True  
review_agent.subagent_llm_model = 'openai/multi-provider/gpt-5.6-luna_thinking'  
review_agent.llm_model = 'openai/multi-provider/gpt-6-sol_thinking'  
review_agent.smart_router_scope_provenance = False  
review_agent.ensemble_models = ['openai/multi-provider/gpt-6-sol_thinking', 'openai/anthropic/claude-opus-5-5_thinking']  
review_agent.enable_database_persistence = True  
review_agent.publish_output = True  
review_agent.enable_standard_mode = True  
review_agent.enable_skip_mode = True  
review_agent.enable_lite_mode = True  
review_agent.enable_extended_mode = True  
review_agent.override_effort = ''  
review_agent.publish_blast_radius_labels = False  
review_agent.push_trigger_dedup_enabled = True  
review_agent.push_trigger_dedup_ttl_seconds = 60  
review_agent.mr_trigger_dedup_ttl_seconds = 300  
review_agent.enable_issues_agent = True  
review_agent.enable_ticket_context = True  
review_agent.max_ticket_fetch_chars = 200000  
review_agent.max_ticket_tokens = 'REDACTED'  
review_agent.max_requirements_ticket_tokens = 'REDACTED'  
review_agent.max_spec_tokens = 'REDACTED'  
review_agent.context_summary_llm_model = 'turbo'  
review_agent.context_summary_timeout_seconds = 90  
review_agent.enable_compliance_agent = True  
review_agent.enable_spec_agent = True  
review_agent.enable_ui_agent = True  
review_agent.enable_figma_findings_revalidation = True  
review_agent.enable_rules_findings_revalidation = True  
review_agent.enable_persona_agent = False  
review_agent.persona_identifier = ''  
review_agent.persona_auto_select = True  
review_agent.persona_max_count = 2  
review_agent.persona_portal_base_url = ''  
review_agent.enable_finding_dismissal_link = True  
review_agent.enable_deduplication = True  
review_agent.enable_conversion_agent = True  
review_agent.apply_conversion = True  
review_agent.is_advanced_configuration = False  
review_agent.enable_precision_agent = False  
review_agent.enable_unobserved_citation_filter = False  
review_agent.citation_text_timeout_seconds = 60  
review_agent.enable_off_change_findings_filter = True  
review_agent.enable_cross_repo_agent = True  
review_agent.security_scan_timeout_seconds = 180  
review_agent.security_scan_max_concurrency = 2  
review_agent.enable_smart_router = True  
review_agent.a2a_pre_pr_review_enabled = True  
review_agent.qodo_review_skill_carry_window_days = 7  
review_agent.prepr_max_detailed_binding_checks = 100  
review_agent.enable_past_bugs_collector = False  
review_agent.enable_past_issues_context = True  
review_agent.enable_skills_agent = True  
review_agent.enable_skill_steering = True  
review_agent.enable_cross_repo_skill_steering = True  
review_agent.enable_web_search_tool = False  
review_agent.enable_review_md = False  
review_agent.serve_pr_repo_via_gitway = False  
review_agent.serve_pr_repo_via_gitway_ratio = 1.0  
review_agent.pr_repo_gitway_read_timeout_seconds = 30  
review_agent.pr_repo_gitway_read_retry_budget_seconds = 60  
review_agent.pr_repo_gitway_midrun_clone_timeout_seconds = 120  
review_agent.enable_sandbox_usage = False  
review_agent.sandbox_provider = 'langsmith'  
review_agent.sandbox_configs_repo_name = 'qodo-sandbox-configs'  
review_agent.sandbox_config_source = 'git_repo'  
review_agent.enable_sandbox_agent_tools = False  
review_agent.persistent_comment = True  
review_agent.persistent_comment_notification = False  
review_agent.add_history_audit = True  
review_agent.publish_in_progress_comment = False  
review_agent.attribution_runs_retention_days = 30  
review_agent.review_components_stats_retention_days = 90  
review_agent.enable_incremental_review = True  
review_agent.enable_fix_loop_optimization = True  
review_agent.fix_loop_min_rounds = 2  
review_agent.fix_loop_recency_window_minutes = 5760  
review_agent.fix_loop_attribution_wait_seconds = 15  
review_agent.fix_loop_intent_llm_model = 'turbo'  
review_agent.fix_loop_intent_timeout_seconds = 20  
review_agent.fix_loop_intent_fallback_models = []  
review_agent.fix_loop_intent_max_author_tokens = 'REDACTED'  
review_agent.fix_loop_drop_below_rank = 3  
review_agent.fix_loop_narrow_agents = True  
review_agent.enable_previous_findings = True  
review_agent.skip_review_on_empty_incremental_diff = True  
review_agent.enable_a2a_auto_fix = False  
review_agent.enable_review_on_fix_pr = False  
review_agent.rules_enabled = True  
review_agent.requirements_gap_enabled = True  
review_agent.llm_call_timeout = 180  
review_agent.sub_agent_timeout_seconds = 2400  
review_agent.llm_call_overall_timeout = 300  
review_agent.finalization_llm_call_timeout_seconds = 240  
review_agent.weak_models_harness_enabled = False  
review_agent.weak_models = ['laguna']  
review_agent.weak_models_llm_call_overall_timeout = 120  
review_agent.weak_models_vllm_thinking_switch = True  
review_agent.weak_models_max_llm_calls = 30  
review_agent.smart_router_llm_model = 'turbo'  
review_agent.smart_router_max_llm_calls = 4  
review_agent.smart_router_disabled_on_open_pr = False  
review_agent.smart_router_extra_instructions = ''  
review_agent.router_extended_floor_hunks = 18  
review_agent.router_extended_floor_lines = 200  
review_agent.lite_mode_llm_model = 'turbo'  
review_agent.compliance_ensemble_models = []  
review_agent.compliance_llm_model = 'turbo'  
review_agent.enable_explorer_subagent = True  
review_agent.enable_downstream_impact_subagent = False  
review_agent.downstream_impact_llm_model = 'turbo'  
review_agent.downstream_impact_timeout_seconds = 150  
review_agent.enable_split_review = False  
review_agent.split_review_min_changed_lines = 2500  
review_agent.split_review_min_changed_files = 15  
review_agent.split_review_llm_model = 'medium'  
review_agent.split_review_max_slices = 8  
review_agent.split_review_timeout_seconds = 900  
review_agent.spec_llm_model = ''  
review_agent.persona_llm_model = ''  
review_agent.persona_selector_llm_model = ''  
review_agent.conversion_llm_model = 'turbo'  
review_agent.conversion_batching_mode = 'batch'  
review_agent.conversion_batch_size = 10  
review_agent.conversion_exploration_rate = 0.1  
review_agent.cross_repo_llm_model = ''  
review_agent.precision_llm_model = 'turbo'  
review_agent.security_llm_model = ''  
review_agent.memory_llm_model = ''  
review_agent.precision_max_llm_calls = 10  
review_agent.precision_max_concurrency = 5  
review_agent.langsmith_project_name = 'review-agent'  
review_agent.max_tokens_for_file = 'REDACTED'  
review_agent.single_unified_diff_tokens_limit = 'REDACTED'  
review_agent.offload_test_files = True  
review_agent.offload_docs_files = True  
review_agent.max_llm_calls = 100  
review_agent.max_llm_calls_limit = 100  
review_agent.warn_when_remaining = 3  
review_agent.enable_agent_time_limit = True  
review_agent.max_agent_time = 900  
review_agent.time_warn_when_remaining_seconds = 120  
review_agent.compliance_batch_size = 35  
review_agent.compliance_batch_concurrency = 5  
review_agent.past_bugs_max_results = 10  
review_agent.past_bugs_llm_model = 'turbo'  
review_agent.skills_agent_dirs = ['.qodo/skills', '.claude/skills', '.cursor/skills', '.agents/skills', 'skills']  
review_agent.skill_discovery_timeout_seconds = 30  
review_agent.pass_previous_findings_to_agents = True  
review_agent.prepr_pass_previous_findings_to_agents = False  
review_agent.deduplication_llm_max_tokens = 'REDACTED'  
review_agent.publishing_action_level_rank_threshold = 0  
review_agent.comments_location_policy = 'both'  
review_agent.publish_inline_acknowledgment = False  
review_agent.enable_commitable_suggestions = False  
review_agent.commitable_suggestions_max_lines = 5  
review_agent.commitable_suggestions_max_concurrency = 5  
review_agent.prefer_single_line_comments = False  
review_agent.issues_user_guidelines = ''  
review_agent.compliance_user_guidelines = ''  
review_agent.enable_ticket_spec_hop = False  
review_agent.additional_repos = []  
review_agent.demand_self_review = False  
review_agent.self_review_text = '**Author self-review**: I have reviewed the code review findings, and addressed the relevant ones.'  
review_agent.approve_pr_on_self_review = False  

==================== REVIEW_AGENT_UX ====================
review_agent_ux.finding_overflow_count = 3  
review_agent_ux.resolved_overflow_count = 0  
review_agent_ux.expand_description = False  
review_agent_ux.expand_code = False  
review_agent_ux.expand_evidence = False  
review_agent_ux.expand_recommendation = False  
review_agent_ux.expand_prompt = False  
review_agent_ux.render_description = True  
review_agent_ux.render_code = True  
review_agent_ux.render_recommendation = True  
review_agent_ux.render_evidence = True  
review_agent_ux.show_prompt = True  
review_agent_ux.show_context_used = False  
review_agent_ux.sub_categories_display = 'verbose'  
review_agent_ux.use_images_and_animations = False  
review_agent_ux.resolved_badge_display = 'both'  
review_agent_ux.resolved_findings_placement = 'resolved_section'  
review_agent_ux.show_resolved_findings = True  
review_agent_ux.display_mode = 'default'  
review_agent_ux.findings_group_by = 'action_level'  
review_agent_ux.findings_label_coloring = 'simplified'  
review_agent_ux.severity_terminology = 'high_medium_low'  
review_agent_ux.findings_group_icon = False  
review_agent_ux.findings_show_empty_groups = False  
review_agent_ux.findings_overflow_include_group_label = True  
review_agent_ux.finding_labels = ['type']  
review_agent_ux.findings_show_counters = True  
review_agent_ux.findings_show_zero_counters = False  
review_agent_ux.findings_count_dimension = 'type'  
review_agent_ux.findings_order_by = 'relevance'  
review_agent_ux.findings_new_first = True  
review_agent_ux.show_comment_footer = False  
review_agent_ux.trial_banner_style = 'pre'  
review_agent_ux.trial_banner_position = 'footer'  
review_agent_ux.show_daily_tips = False  
review_agent_ux.show_dividers = True  
review_agent_ux.link_findings_to_inline_comments = True  
review_agent_ux.show_subsection_names = False  
review_agent_ux.show_persistent_review_section = False  
review_agent_ux.findings_overflow = {'bugs': 3, 'ux_issue': 3, 'ui_issues': 3, 'cross_repo': 3, 'informational': 0, 'team_insights': 3, 'skill_insights': 3, 'action_required': -1, 'rule_violations': 3, 'security_issues': 3, 'requirement_gaps': 3, 'remediation_recommended': 0}  

==================== PR_HELP ====================
pr_help.force_local_db = False  
pr_help.num_retrieved_snippets = 5  

==================== PR_NEW_ISSUE ====================
pr_new_issue.label_to_prompt_part = {'general': 'general question', 'feature': 'feature request (may already be addressed in the documentation)', 'bug': 'possible bug report (may be a by design behavior)'}  
pr_new_issue.supported_repos = ['qodo-ai/pr-agent']  

==================== PR_HELP_DOCS ====================
pr_help_docs.repo_url = ''  
pr_help_docs.repo_default_branch = 'main'  
pr_help_docs.docs_path = 'docs'  
pr_help_docs.exclude_root_readme = False  
pr_help_docs.supported_doc_exts = ['.md', '.mdx', '.rst']  
pr_help_docs.enable_help_text = False  

==================== PR_SIMILAR_ISSUE ====================
pr_similar_issue.skip_comments = False  
pr_similar_issue.force_update_dataset = False  
pr_similar_issue.max_issues_to_scan = 500  
pr_similar_issue.vectordb = 'pinecone'  

==================== PR_FIND_SIMILAR_COMPONENT ====================
pr_find_similar_component.class_name = ''  
pr_find_similar_component.file = ''  
pr_find_similar_component.search_from_org = False  
pr_find_similar_component.allow_fallback_less_words = True  
pr_find_similar_component.number_of_keywords = 5  
pr_find_similar_component.number_of_results = 5  

==================== BEST_PRACTICES ====================
best_practices.auto_best_practices_str = ''  
best_practices.wiki_best_practices_str = ''  
best_practices.global_wiki_best_practices = ''  
best_practices.local_repo_best_practices_str = ''  
best_practices.global_repo_best_practices_str = ''  
best_practices.global_best_practices_str = ''  
best_practices.organization_name = ''  
best_practices.max_lines_allowed = 2000  
best_practices.enable_global_best_practices = True  
best_practices.allow_repo_best_practices = True  
best_practices.enabled = True  

==================== AUTO_BEST_PRACTICES ====================
auto_best_practices.enable_auto_best_practices = True  
auto_best_practices.utilize_auto_best_practices = True  
auto_best_practices.extra_instructions = ''  
auto_best_practices.min_suggestions_to_auto_best_practices = 10  
auto_best_practices.number_of_days_to_update = 30  
auto_best_practices.max_patterns = 5  
auto_best_practices.minimal_date_to_update = '2025-01-26'  

==================== JIRA ====================
jira.jira_client_id = 'REDACTED'  
jira.jira_app_secret = 'REDACTED'  

==================== LINEAR ====================
linear.linear_client_id = 'REDACTED'  

==================== PR_TO_TICKET ====================
pr_to_ticket.default_base_url = ''  
pr_to_ticket.default_project_key = 'REDACTED'  
pr_to_ticket.fallback_to_git_provider_issues = True  
pr_to_ticket.direct_update_compliance = False  

==================== PR_LIFECYCLE_EVENTS ====================
pr_lifecycle_events.project_id = 'qodo-saas-shared-services'  
pr_lifecycle_events.transport = 'pubsub'  
pr_lifecycle_events.enabled = True  
pr_lifecycle_events.topic = 'events-v2'  
pr_lifecycle_events.publish_timeout_seconds = 5.0  

==================== github_app ====================
github_app.ignore_bot_pr = False  
github_app.bot_user = 'github-actions[bot]'  
github_app.override_deployment_type = True  
github_app.handle_pr_actions = ['opened', 'reopened', 'ready_for_review']  
github_app.pr_commands = ['/agentic_describe', '/agentic_review']  
github_app.feedback_on_draft_pr = False  
github_app.handle_push_trigger = False  
github_app.push_commands = ['/agentic_review']  
github_app.attribute_closed_pr_to_opener = False  

The operator job fell back to the published Red Hat catalog whenever the image
registry was not registry.redhat.io or the chart was not a CI tag. That path
cannot work: prepare-restricted-environment.sh writes only fast-X.Y channels
into the ImageSetConfiguration, so oc-mirror rejects the published catalog
because its default fast channel is filtered out, and the catalog carries no
2.x bundles at all.

Every disconnected run is branch-aligned, and the hub image comes from the
bundle's RELATED_IMAGE_backstage, so TAG_NAME, IMAGE_* and CHART_VERSION never
reached the deployment. Use the CI catalog
(quay.io/rhdh/iib:<stream>-v<ocp>-x86_64) and its registries.conf redirect
unconditionally, and log that only the newest bundle of the branch's stream is
tested.

Assisted-by: OpenCode
Putting deploy_rhdh_operator on the left of || turns errexit off for its whole
body, so a failed envsubst, yq or oc apply fell through to the 5-minute
deployment poll and then reported a missing deployment, which pointed at OLM.

Return from deploy_rhdh_operator as soon as rendering or applying the CR fails,
and use a neutral message in the operator job (the OLM v1 dump stays as
diagnostics).

The Helm hub bounce scaled the Deployment back up even when the old pod was
still present after 180s, recreating the two-backend race it is meant to
prevent. Treat that timeout as a recovery failure.

Assisted-by: OpenCode
disconnected::snapshot_mcp_rendered_configs hid oc failures by returning an
empty success: a failed pre-change read skipped render detection, and a failed
read during detection looked like a changed config. Retry the read three times
and fail on an error or an empty answer, skip failed reads in the render loop,
and require the pre-change snapshot. Both jobs now stop when it cannot be taken.

Rename the MCP identifiers to machineconfigpool, since MCP reads as Model
Context Protocol in this repository.

Document why the IDMS and ITMS applies share one wait: a pool's spec moves to
the newest rendered config, so a config rendered during the first rollout is
still awaited.

Assisted-by: OpenCode
The smoke set was spelled out in six places: the array, the case in
_smoke_plugin_package_var, two require_vars lists and two templates. A template
entry the array did not know about rendered an empty package: and only failed at
pod runtime.

Render both templates through one helper that maps the array to the package
variables, requires them, and rejects a package: left empty by envsubst.

Assisted-by: OpenCode
- the CI mirror serves TLS on port 5000, not plain HTTP: the openshift/release
  step curls https://<mirror>/v2/ with the vault CA
- nameOverride pins the chart fullname for the registry-auth secret; the egress
  NetworkPolicy selects on pod labels
- the smoke app config is a subset of dynamic-plugins-config.yaml, not a mirror
- the Backstage CR installs the minimal NFS plugin set, not only the homepage

Assisted-by: OpenCode
@zdrapela

Copy link
Copy Markdown
Member Author

/test e2e-ocp-disconnected-helm-nightly
/test e2e-ocp-disconnected-operator-nightly

A GA verification run (--image-registry registry.redhat.io with a released x.y.z
chart) should pair with the published Red Hat catalog, and the existing image and
chart overrides already select it. Restore that branch.

It cannot keep the version filter: the published catalog declares defaultChannel
fast, and oc-mirror rejects a channels list that leaves the default channel out,
which is why GA runs failed at mirroring before. Pass --filter-versions '*' for
the published catalog (prepare then mirrors every rhdh channel, still scoped to
the rhdh package) and keep the branch-stream filter for the CI catalog, so the
nightlies are unchanged.

Assisted-by: OpenCode
@zdrapela

Copy link
Copy Markdown
Member Author

/test e2e-ocp-disconnected-helm-nightly
/test e2e-ocp-disconnected-operator-nightly

@sonarqubecloud

Copy link
Copy Markdown

@github-actions

Copy link
Copy Markdown
Contributor

The container image build workflow finished with status: cancelled.

@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants