Feature branch sync - staging to main for Omnia 2.2.0.0 changes - #5004
Merged
Conversation
Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com>
Signed-off-by: mithileshreddy04 <mithilesh.reddy@dell.com>
------------- increased reboot timeout for slurm login_compiler_* in rollback restoring slurm conf fro rollback Signed-off-by: Jagadeesh N V <jagadeesh_n_v@dell.com>
Signed-off-by: Jagadeesh N V <39791839+jagadeeshnv@users.noreply.github.com>
Signed-off-by: Jagadeesh N V <39791839+jagadeeshnv@users.noreply.github.com>
…ubnet (OMN01D-2534) In multi-subnet deployments, service K8s control plane nodes may reside in an additional_subnet (e.g. 10.40.2.0/24) rather than the primary admin subnet (e.g. 10.40.1.0/24). The VIP for K8s HA must be in the same subnet as the control plane nodes, not the OIM admin NIC subnet. The fix: 1. In validate_service_k8s_cluster_ha(), extract control plane node IPs from PXE mapping (FUNCTIONAL_GROUP_NAME starts with service_kube_control_plane) and determine their subnet by checking the primary admin subnet and additional_subnets. 2. Pass the control plane subnet (kcp_subnet_ip, kcp_subnet_bits) to validate_vip_address(). 3. In validate_vip_address(), validate the VIP against the control plane subnet if provided, otherwise fall back to the primary admin subnet for backward compatibility. Fixes: OMN01D-2534 Signed-off-by: Sujit Jadhav <sujit.jadhav@dell.com>
Rollback: Fixed reading of build_stream_terminal for slurm
* localrepo checkmarx fixes Signed-off-by: pullan1 <sudha.pullalaravu@dell.com> * Checkmarx fixes Signed-off-by: pullan1 <sudha.pullalaravu@dell.com> * vast metrics relabeling fix Signed-off-by: pullan1 <sudha.pullalaravu@dell.com> * ansible 2.20 fixes Signed-off-by: pullan1 <sudha.pullalaravu@dell.com> --------- Signed-off-by: pullan1 <sudha.pullalaravu@dell.com>
Signed-off-by: Katakam-Rakesh <katakam.rakesh@dell.com>
Signed-off-by: Katakam-Rakesh <katakam.rakesh@dell.com>
Feature branch sync - pub/q2 upgrade to staging
Signed-off-by: Katakam-Rakesh <katakam.rakesh@dell.com>
Add a wait for kube controller pod to be created and then check for pod running
feat: Add custom callback plugin to suppress duplicate error output in ansible-core 2.20
…3_secret_key Signed-off-by: venu <236371043+Venu-p1@users.noreply.github.com>
Fix/cleanup image
Signed-off-by: Jagadeesh N V <jagadeesh_n_v@dell.com>
…led is true Two issues prevent nid hostname resolution on slurm and login nodes: 1. OIM firewall blocks port 53 (DNS) for external access CoreDNS on the OIM binds to admin_nic_ip:53, but firewalld only opens ports for DHCP/TFTP/HTTP/etc. Nodes querying 10.x.x.x:53 get their packets dropped. From the OIM itself, DNS works because podman interfaces are in the trusted zone (local traffic bypasses the firewall). Fix: Open port 53/tcp and 53/udp in the OIM firewall when dns_enabled is true. 2. NetworkManager overwrites /etc/resolv.conf after cloud-init set-ssh.sh runs nmcli con add/up which triggers NetworkManager to overwrite /etc/resolv.conf with DHCP-provided DNS servers, removing the CoreDNS nameserver entry. Fix: After set-ssh.sh completes, restore /etc/resolv.conf and lock it with chattr +i. Matches existing K8s template protection. Files changed: - prepare_oim/.../openchami/tasks/configs/firewall.yml (port 53) - ci-group-slurm_control_node_x86_64.yaml.j2 - ci-group-slurm_node_x86_64.yaml.j2 - ci-group-slurm_node_aarch64.yaml.j2 - ci-group-login_node_x86_64.yaml.j2 - ci-group-login_node_aarch64.yaml.j2 - ci-group-login_compiler_node_x86_64.yaml.j2 - ci-group-login_compiler_node_aarch64.yaml.j2 Only active when dns_enabled is true (no impact on non-DNS deployments). Signed-off-by: Sujit Jadhav <sujit.jadhav@dell.com>
Signed-off-by: sakshi-singla-1735 <sakshi.s@dell.com>
fix(provision): fix DNS resolution on slurm/login nodes when dns_enabled is true
…plate (OMN01D-2533) (#4729) The cloud-init template has two YAML literal block scalar levels: 1. Outer content: | (base indent 6sp) - strips 6 spaces 2. Inner runcmd - | (base indent 4sp after outer) - strips 4 spaces Total: 10 spaces stripped from template lines. Previous heredoc fix used 12sp indent with spaces embedded in the delimiter string (' PYEOF'). After YAML stripping, the terminator line became ' PYEOF' (2sp) but the shell expected ' PYEOF' (12sp literal) — heredoc never terminated. Fix: Place Python code and PYEOF terminator at 10sp in the template. After both YAML levels strip their indentation, these lines land at column 0 in the shell script. The simple delimiter 'PYEOF' matches the column-0 terminator exactly. Python receives column-0 code with correct relative indentation for with/if/else blocks. All lines >= 10sp > 6sp, so the outer YAML content: | block stays intact (lines at < 6sp would prematurely terminate it). Signed-off-by: Sujit Jadhav <sujit.jadhav@dell.com>
…ction (OMN01D-2532) (#4724) In multi-subnet deployments, service K8s control plane nodes may reside in an additional_subnet (e.g. 10.40.2.0/24) while the OIM admin NIC is in the primary subnet (e.g. 10.40.1.0/24). Calico's IP_AUTODETECTION_METHOD was hardcoded to admin_nic_cidr (the OIM subnet), causing Calico to fail IP auto-detection on nodes in different subnets with: 'Unable to auto-detect an IPv4 address using interface cidr [10.40.1.0/24]: no valid IPv4 addresses found' The fix: 1. In create_k8s_config_nfs.yml, read the PXE mapping to find the first service_kube_control_plane node's ADMIN_IP and determine which subnet (primary or additional) it belongs to. Set calico_cidr to that subnet's CIDR. 2. Update the cloud-init template to use calico_cidr instead of admin_nic_cidr for Calico's IP_AUTODETECTION_METHOD. The upgrade path is intentionally left unchanged (uses admin_nic_cidr) since multi-subnet is a fresh deployment feature and changing the upgrade flow could impact existing deployments. Fixes: OMN01D-2532 Signed-off-by: Sujit Jadhav <sujit.jadhav@dell.com>
fix(validation): validate HA VIP against service_kube_control_plane subnet
Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com>
Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com>
defct fix for input valdition and pxe mapping check
Push software_config..json from artifacts during deploy
…se) (#4964) The openchami-0.1.7 RPM was previously built from release/ which ships an incomplete/newer architecture (tokensmith-based) that conflicts with the container images actually deployed (opaal, bss, hydra, cloud-init). Rebuild from openchami/release/ which has: - Complete bootstrap_openchami.sh with all required podman secrets (bss_postgres_password, hydra_postgres_password, hydra_system_secret, hydra_dsn, and full postgres_multiple_databases) - Correct openchami.env with BSS, OPAAL, and cloud-init env vars - openchami-certificate-update with opaal.container FQDN update - openchami_profile.sh with hydra-based gen_access_token (not tokensmith) - Correct openchami.target without tokensmith/boot-service/metadata-service - Only the containers matching the deployed images (opaal, bss, hydra, etc.) - cloud-init container rebuilt with go-viper/mapstructure/v2 v2.5.0 (fixes CVE-2025-11065 sensitive info leak) Signed-off-by: Sujit Jadhav <sujit.jadhav@dell.com>
…p template (#4968) Since coresmd v0.6.3 is the only version shipped, the version check and the v0.4.x positional argument fallback in coredhcp.yaml.j2 are dead code. Remove them to simplify the template. Both single-subnet and multi-subnet deployments continue to work through the same key=value config format - the additional_subnets for-loops simply produce zero iterations for single-subnet setups. Signed-off-by: Sujit Jadhav <sujit.jadhav@dell.com>
* mysql upgrade and rollback fixes0 Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> * mysql upgrade and rollback changes for version update. Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> * Fix ansible-lint configuration Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> * lint fixes Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> * rollback backup handling fixes Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> * ansible-lint fixes Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> * Delete temp_upgrade_mysql Signed-off-by: Kratika Patidar <Kratika.Patidar@dell.com> * Update restore_mysql_data.yml Signed-off-by: Kratika Patidar <Kratika.Patidar@dell.com> --------- Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> Signed-off-by: Kratika Patidar <Kratika.Patidar@dell.com>
csi driver powerscale version is updated Signed-off-by: Nagachandan P <Nagachandan.p@dell.com>
Signed-off-by: Nagachandan P <Nagachandan.p@dell.com>
Signed-off-by: Nagachandan P <Nagachandan.p@dell.com>
csi driver powerscale version in examples
…m_config (#4977) The subnet membership check for aarch64_inventory_host_ip only validated against the primary admin subnet from network_spec.yml, ignoring additional_subnets. In multi-subnet environments, this incorrectly rejected valid IPs on additional subnets (e.g., 192.168.3.48 on 192.168.3.0/24 was rejected because it wasn't in the primary 192.168.0.0/24 subnet). Removed the subnet check entirely. IPv4 format validation and SSH reachability checks are retained, which provide sufficient validation. Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com>
Signed-off-by: Abhishek S A <abhishek.sa3@dell.com>
Feature branch sync - pub/omnia_2.2.0.0_fix to staging
Signed-off-by: balajikumaran.cs <balajikumaran.cs@dell.com>
chore(aarch64): update image-builder container version to 1.2
The upgrade role's vars/main.yml had outdated image tags that were out of sync with prepare_oim/roles/deploy_containers/openchami/vars/main.yml, causing "manifest unknown" errors during the 2.1→2.2 upgrade when podman tried to pull non-existent image versions. Updated tags to match the prepare_oim source of truth: - opaal: v0.3.12 → v0.3.14 - smd: v2.19.3 → v2.20.6 - bss: v1.32.2 → v1.32.4 - cloud-init: v1.3.0 → v1.4.9 - coresmd: v0.4.3 → v0.6.3 - postgres: 11.5-alpine → 17-alpine Co-Authored-By: Sujit Jadhav <sujit.jadhav@dell.com> Signed-off-by: Super User <root@demo.hpc.cluster>
Fix stale OpenCHAMI container image tags in upgrade vars
Fix OpenCHAMI upgrade failures for vulnerability container updates
…g kubernetes_addr env var (#4992) Problem: After upgrading kube-vip from v0.8.9 to v1.2.2, the API VIP fails to failover to another control-plane node when the VIP leader node reboots. All nodes go NotReady and kube-vip logs show "Error retrieving lease lock: dial tcp <VIP>:6443: connect: connection refused". Root Cause: In kube-vip v0.8.9, when cp_enable=true and admin.conf exists, the code overrode the API server URL to kubernetes:6443. Combined with hostAliases mapping kubernetes -> 127.0.0.1, kube-vip always connected to the local API server for leader election. In kube-vip v1.2.2, a new code path was added for explicit kubeconfig files (K8sConfigFile). Since the default value is /etc/kubernetes/admin.conf and the file exists, this new path matches before the old cp_enable logic runs. kube-vip reads admin.conf as-is (which points to the VIP), and the kubernetes:6443 override never executes. This creates a circular dependency: kube-vip needs to connect to the API server to acquire the leader lease, but the API server is only reachable via the VIP, which is only assigned by the kube-vip leader. Fix: Add the kubernetes_addr environment variable to the kube-vip static pod manifest, set to the node's own IP with port (<node_ip>:6443). This is a built-in kube-vip feature that overrides the API server address used by the Kubernetes client, regardless of what's in the kubeconfig file. Each control-plane node's kube-vip talks to its own local API server for leader election, breaking the circular dependency. Changes: - ci-group-service_kube_control_plane_first_x86_64.yaml.j2: Add kubernetes_addr env var - ci-group-service_kube_control_plane_x86_64.yaml.j2: Add kubernetes_addr env var - step_kube_vip_upgrade.yml: Add kubernetes_addr env var, remove admin.conf workaround Signed-off-by: Katakam-Rakesh <katakam.rakesh@dell.com>
…lience fixes (#4995) Tested: upgrade and rollback executed successfully with OpenCHAMI behaviour verified end-to-end. 1. PostgreSQL major version downgrade (PG 17 -> PG 11): Omnia 2.2 uses PostgreSQL 17; Omnia 2.1 uses PostgreSQL 11. After upgrading (PG 11->17 via PR #4990), rolling back to v2.1 fails because PG 11 cannot read PG 17 data files: FATAL: database files are incompatible with server Fix: Mirror the upgrade PG detection logic in reverse in start_postgres_only.yml - detect PG_VERSION in the data volume, clear it if mismatched, let PG 11 init fresh, then restore from pg_dump backup. 2. Upgrade OIM completion gate fix: The "Mark OIM upgrade as completed" task was guarded by `when: openchami_deployed`, causing OIM to stay as 'in-progress' when containers were not running. Downstream stages (build_image) then fail, leaving the system in an unrecoverable state. Fix: Remove the guard so OIM is always marked completed when the play reaches finalization. Re-read manifest before writing to capture intermediate state changes. 3. Rollback upgrade lock handling: Ansible playbooks are synchronous - if the user runs rollback, the upgrade has already exited. The stale upgrade lock file blocked rollback unnecessarily. Fix: Clear the stale lock with a warning instead of aborting. 4. Cloud-init readiness check compatibility: The v2.1 cloud-init-server does not expose the version endpoint that `ochami cloud-init service status` checks (returns 404). Fix: Use podman container-level check instead of the ochami CLI status endpoint for readiness verification. 5. Increased wait_time from 30s to 60s for both upgrade and rollback to allow OpenCHAMI services more time to stabilize. Files changed: rollback/roles/rollback_openchami/tasks/main.yml rollback/roles/rollback_openchami/tasks/start_postgres_only.yml rollback/roles/rollback_openchami/tasks/reload_cloud_init_data.yml rollback/roles/rollback_openchami/vars/main.yml rollback/rollback.yml upgrade/playbooks/upgrade_oim.yml upgrade/roles/upgrade_openchami/vars/main.yml Signed-off-by: Sujit Jadhav <sujit.jadhav@dell.com>
* Upgrade GitLab Runner to v19.2.0 to resolve CVEs - Update gitlab_runner_image: v18.8.0 -> v19.2.0 - Update gitlab_runner_helper_image_version: v18.8.0 -> v19.2.0 Confirmed fixes: - CVE-2026-34986 (HIGH) in golang.org/x/net - CVE-2026-33186 (CRITICAL) in google.golang.org/grpc Additional Go dependency CVEs expected to be resolved by the v19.2.0 base image and module updates. Signed-off-by: Rajeshkumar S <rajeshkumar.s2@dell.com> * Revert "Upgrade GitLab Runner to v19.2.0 to resolve CVEs" This reverts commit efaba87. * Remove stale/old runner images during upgrade/fresh installation Signed-off-by: Rajeshkumar S <rajeshkumar.s2@dell.com> * Fix ansible lint issues Signed-off-by: Rajeshkumar S <rajeshkumar.s2@dell.com> --------- Signed-off-by: Rajeshkumar S <rajeshkumar.s2@dell.com>
…aults to true (#4996) * fix(upgrade): lock file deletion requires force flag, dns_enabled defaults to true - upgrade.yml: upgrade lock removal now requires force_upgrade=true - rollback.yml: stale upgrade lock and rollback lock removal require force_rollback=true - provision_config.yml: dns_enabled default changed from false to true - pxe_mapping_file.csv: hostnames updated to nid001-nid012 - provision_config.j2: dns_enabled hardcoded to false during upgrade with clarifying comment - transform_provision_config.yml: added comment explaining dns_enabled is not carried forward * Update pxe_mapping_file.csv Signed-off-by: Abhishek S A <abhishek.sa3@dell.com> --------- Signed-off-by: Abhishek S A <abhishek.sa3@dell.com>
…driver (#4997) * initialize variable Signed-off-by: priti-parate <140157516+priti-parate@users.noreply.github.com> * Update create_telemetry_report.yml Signed-off-by: priti-parate <140157516+priti-parate@users.noreply.github.com> * Update create_telemetry_report.yml Signed-off-by: priti-parate <140157516+priti-parate@users.noreply.github.com> * revert vars and add defaults Signed-off-by: priti-parate <140157516+priti-parate@users.noreply.github.com> * add defaults Signed-off-by: priti-parate <140157516+priti-parate@users.noreply.github.com> * Update csi_driver_powerscale.json REmoving CSM encryption from CSI powerscale driver * csm encryption packaged removed Signed-off-by: priti_parate <priti_parate@dell.com> --------- Signed-off-by: priti-parate <140157516+priti-parate@users.noreply.github.com> Signed-off-by: priti_parate <priti_parate@dell.com>
… openchami stabilization wait (#4999)
images priority of setting in arm nodes fixed Signed-off-by: Nagachandan P <Nagachandan.p@dell.com>
…k exits (#5003) * fix(prepare_oim): ensure OpenCHAMI services are stable before exit After prepare_oim.yml completes, OpenCHAMI containers can still be restarting because omnia_service.yml triggers a cascade: 1. network-online.target is unconditionally restarted, which can cause dependent OpenCHAMI services to stop and restart via systemd 2. daemon_reload + start omnia.target re-evaluates all unit files 3. The playbook exits immediately with no stabilization check Fix both the trigger and the gap: - Change network services from state: restarted to state: started (no need to restart already-running network targets — avoids cascading restarts to OpenCHAMI services that depend on them) - Add verify_final_readiness.yml as a readiness gate before the completion message: wait for stabilization, verify openchami.target is active with retries, and fail with diagnostics if services remain unstable. Signed-off-by: Sujit Jadhav <sujit.jadhav@dell.com> * fix(lint): add noqa for systemctl command-instead-of-module The ansible-lint rule command-instead-of-module flags systemctl usage via ansible.builtin.command. Add noqa suppression comments matching the existing pattern in verify_ochami.yml — the systemd module cannot be used here because we need the exit code from is-active without changing service state. Signed-off-by: Sujit Jadhav <sujit.jadhav@dell.com> * fix(prepare_oim): revert stabilization wait from 60s to 30s The stabilization wait was bumped from 30s to 60s in PR #4999 as a workaround for containers restarting after playbook completion. PR #5003 fixed the root cause (preventing unnecessary network service restarts), so the longer wait is no longer needed. Signed-off-by: Sujit Jadhav <sujit.jadhav@dell.com> --------- Signed-off-by: Sujit Jadhav <sujit.jadhav@dell.com>
abhishek-sa1
marked this pull request as ready for review
August 17, 2026 10:19
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| # 2. - followed by multi-part digits (e.g. -2-16-0, -1.1.2) | ||
| # 3. - followed by single digit at the end (e.g. -3$) | ||
| # Preserves trailing non-version suffixes (e.g. -amd64, -chart) | ||
| version_regex = r'[-_](?:v\d+(?:[-.]\d+)*|\d+(?:[-.]\d+)+|\d+$)(?=[-_]|$)' |
|
|
||
| # pylint: disable=invalid-name | ||
| # PowerScale telemetry validation messages | ||
| POWERSCALE_VICTORIA_REQUIRED_MSG = ( |
| "'victoria_logs' must be included in collection_targets " | ||
| "(e.g., 'victoria_metrics,victoria_logs')." | ||
| ) | ||
| POWERSCALE_CSI_DRIVER_MISSING_MSG = ( |
sujit-jadhav
pushed a commit
that referenced
this pull request
Aug 20, 2026
Feature branch sync - staging to main for Omnia 2.2.0.0 changes
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
19 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Feature branch sync - staging to main for Omnia 2.2.0.0 changes