From 3fdd2b757bd6cf07162898cc6680b522b348caba Mon Sep 17 00:00:00 2001 From: Maksym Kucherov Date: Mon, 7 Sep 2026 17:26:54 +0200 Subject: [PATCH 01/17] CI: Move `run-performance-tests.ps1` (already coupled) from `common-ci/rust` to here. --- .github/workflows/nightly-performance.yml | 8 +- AGENTS.md | 6 ++ ci/missing-ci-scripts.md | 91 +++++++++++++++++++++++ ci/run-performance-tests.ps1 | 80 ++++++++++++++++++++ 4 files changed, 182 insertions(+), 3 deletions(-) create mode 100644 ci/missing-ci-scripts.md create mode 100644 ci/run-performance-tests.ps1 diff --git a/.github/workflows/nightly-performance.yml b/.github/workflows/nightly-performance.yml index 860f090..21d6fdf 100644 --- a/.github/workflows/nightly-performance.yml +++ b/.github/workflows/nightly-performance.yml @@ -41,8 +41,10 @@ jobs: path: rust submodules: recursive - # 2. The shared common-ci scripts (the Rust performance adapter and the - # comparison step). + # 2. common-ci, checked out for the shared comparison step + # (steps/compare-performance.ps1). The Rust performance adapter itself + # now lives in this repo at ci/run-performance-tests.ps1 (see + # ci/missing-ci-scripts.md); only the comparison step is shared. - name: Checkout common-ci uses: actions/checkout@v4 with: @@ -112,7 +114,7 @@ jobs: # rust/test-results/performance-summary in the common-ci schema. - name: Run performance and write results shell: pwsh - run: ./common-ci/rust/run-performance-tests.ps1 -RepoName rust + run: ./rust/ci/run-performance-tests.ps1 -RepoName rust - name: Upload Performance Results Artifact uses: actions/upload-artifact@v4 diff --git a/AGENTS.md b/AGENTS.md index 76c3af2..e0473a2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -111,6 +111,12 @@ manifest's `include`, so `cargo publish` packages it. builds and tests `examples/` against the just-released crates.io packages. - `utm-link-lint.yml` — see UTM conventions below. Runs on every PR. - `mustache-drift.yml` — guards the generated JavaScript resource template. +- `nightly-performance.yml` — nightly (and on-demand) on-premise throughput + run that feeds the documentation performance graphs. Its Rust adapter lives + in this repo at `ci/run-performance-tests.ps1`; only the shared comparison + step (`compare-performance.ps1`) comes from common-ci. See + [ci/missing-ci-scripts.md](ci/missing-ci-scripts.md) for how this relates to + the org-standard reusable-workflow contract and the planned direction. ## Conventions and gotchas diff --git a/ci/missing-ci-scripts.md b/ci/missing-ci-scripts.md new file mode 100644 index 0000000..cbfb656 --- /dev/null +++ b/ci/missing-ci-scripts.md @@ -0,0 +1,91 @@ +# Missing CI scripts and the road to the shared reusable workflows + +This note records why the Rust repository's CI diverges from the rest of the +51Degrees organisation today, what the organisation's standard contract looks +like, and the incremental direction for closing the gap. It exists so the next +person (or agent) does not have to re-derive the layout from scratch, and so the +docs that must be kept current (see `AGENTS.md`) point at a single source of +truth. + +## The organisation standard (how the other repos work) + +Consuming repositories keep their GitHub Actions YAML deliberately thin. They do +not `run:` common-ci PowerShell directly; instead they `uses:` common-ci's +*reusable workflows*. For example, `device-detection-cxx/.github/workflows`: + +``` +nightly-pipeline.yml + └─ uses: 51Degrees/common-ci/.github/workflows/nightly-pull-requests.yml@main + └─ uses: nightly-pull-request.yml (per PR: Configure → BuildAndTest + │ → ComparePerformance → Complete) + └─ runs: nightly-pull-request.build-and-test.ps1 +``` + +The orchestrator `nightly-pull-request.build-and-test.ps1` then calls back into +**repo-local** hook scripts by a fixed naming convention: + +``` +.//ci/fetch-assets.ps1 +.//ci/setup-environment.ps1 +.//ci/build-project.ps1 +.//ci/run-unit-tests.ps1 +.//ci/run-integration-tests.ps1 +.//ci/run-performance-tests.ps1 # only when Options.RunPerformance +``` + +plus a `ci/options.json` describing the build matrix. In short: common-ci owns +the *orchestration*; each repo owns a set of `ci/.ps1` *customization +entry points*. The only common-ci script a repo's YAML runs directly is the +generic linter `scripts/utm-lint.ps1`. + +## Where the Rust repo stands today + +The Rust repo has **not** joined the shared orchestration. It carries its own +self-contained `.github/workflows/nightly-performance.yml`, which the workflow +header itself notes is "kept self-contained here (rather than calling the shared +reusable workflow) until the repository joins the shared nightly orchestration." + +Historically its performance adapter also lived in the wrong repository: +`common-ci/rust/run-performance-tests.ps1`. That was an anomaly — common-ci +exists to hold code shared *across* repositories, and the Rust repo is the only +Rust consumer, so there is nothing to share. Every other language keeps its +`run-performance-tests.ps1` under its own `/ci/`. + +## What this change does (Phase 1) + +- Adds the Rust performance adapter to this repo at + `ci/run-performance-tests.ps1`, matching the org's `/ci/.ps1` + convention. This is the customization entry point that further repo-specific + work will build on. +- Repoints `nightly-performance.yml` to run `./rust/ci/run-performance-tests.ps1` + (the workflow checks this repo out into a `rust/` subdirectory) instead of + `./common-ci/rust/run-performance-tests.ps1`. +- Keeps the `Checkout common-ci` step, because the comparison step + `steps/compare-performance.ps1` is genuinely shared and still comes from + common-ci. + +The behaviour of the nightly run is unchanged; only the adapter's home moves. + +The copy in `common-ci/rust/run-performance-tests.ps1` is intentionally **left +in place** for now and will be removed in a separate, manually raised common-ci +PR. Until then the two copies are identical; this repo's copy is the one CI +uses. + +## The remaining gap (Phase 2, deferred) + +Fully aligning with the organisation standard would mean: + +- Providing the complete `ci/` verb set (`build-project.ps1`, + `run-unit-tests.ps1`, `run-integration-tests.ps1`, `setup-environment.ps1`, + `fetch-assets.ps1`, `run-performance-tests.ps1`) and a `ci/options.json` + build matrix. +- Replacing the bespoke `nightly-performance.yml` with a thin + `nightly-pipeline.yml` that `uses:` the common reusable workflows, letting + common-ci drive build, test, performance and publish uniformly with the other + languages. + +This is a much larger change that alters how the Rust repo is built and tested +in CI across the whole organisation, and it carries open-ended maintenance until +the shared workflows fully accommodate a Cargo-workspace consumer. It is +recorded here as a direction, not scheduled work. Weigh that cost explicitly +before starting it. diff --git a/ci/run-performance-tests.ps1 b/ci/run-performance-tests.ps1 new file mode 100644 index 0000000..781f70e --- /dev/null +++ b/ci/run-performance-tests.ps1 @@ -0,0 +1,80 @@ +param( + # The directory the rust workspace is checked out to. CI checks the repo out + # into a subdirectory named after the repository, matching the other + # languages; a local run can pass "." for the current directory. + [string]$RepoName = ".", + # Where the results files are written, relative to the repo directory. This is + # the path the nightly workflow uploads and the compare-performance step reads. + [string]$OutputDir = "test-results/performance-summary" +) +$ErrorActionPreference = "Stop" +$PSNativeCommandUseErrorActionPreference = $true + +# Runs the 51Degrees Rust on-premise performance examples in release and writes +# their throughput figures into results_.json files, in the same +# `{ HigherIsBetter = @{ metric = value } }` shape the shared +# steps/compare-performance.ps1 consumes. The Rust on-premise engines call the +# device-detection-cxx and ip-intelligence-cxx libraries through FFI, so the +# figures track the native C/C++ performance. + +Push-Location $RepoName +try { + $summaryDir = Join-Path (Get-Location) $OutputDir + New-Item -ItemType Directory -Force -Path $summaryDir | Out-Null + + # The example crates live in their own workspace under examples/ and depend + # on the published crates from crates.io by default. The nightly must + # benchmark this checkout's engine code, not the released packages, so the + # cargo commands run from examples/ with `--config source.toml`, the patch + # file that points every fiftyone-* dependency at its local path. + $examplesDir = Join-Path (Get-Location) "examples" + + # Run one performance example and parse the throughput it prints. The + # examples take the highest throughput figure they report (the multi-threaded + # pass), which is the headline number for the product. + function Get-Throughput { + param( + [Parameter(Mandatory)][string]$Package, + [Parameter(Mandatory)][string]$Bin, + [Parameter(Mandatory)][string]$Pattern + ) + Write-Host "Running performance example '$Bin'..." + Push-Location $examplesDir + try { + $output = cargo run --release --config source.toml -p $Package --bin $Bin 2>&1 | Out-String + } finally { + Pop-Location + } + Write-Host $output + $found = [regex]::Matches($output, $Pattern) + if ($found.Count -eq 0) { + Write-Error "Could not parse a throughput figure from '$Bin' output" + } + # Take the last match so the multi-threaded figure wins where an example + # prints both a single- and a multi-threaded result. + return [double]$found[$found.Count - 1].Groups[1].Value + } + + # Device Detection on-premise (Hash): detections per second. + $ddDetectionsPerSecond = Get-Throughput ` + -Package "device-detection-examples" ` + -Bin "dd-onprem-performance" ` + -Pattern "Detections per second\s*:\s*(\d+)" + @{ HigherIsBetter = @{ DetectionsPerSecond = $ddDetectionsPerSecond } } | + ConvertTo-Json -Depth 5 | + Out-File (Join-Path $summaryDir "results_DeviceDetection-OnPremise.json") -Encoding utf8 + + # IP Intelligence on-premise (IP graph): lookups per second. + $ipiLookupsPerSecond = Get-Throughput ` + -Package "ip-intelligence-examples" ` + -Bin "ipi-onprem-performance" ` + -Pattern "Throughput:\s*(\d+)\s*lookups/sec" + @{ HigherIsBetter = @{ LookupsPerSecond = $ipiLookupsPerSecond } } | + ConvertTo-Json -Depth 5 | + Out-File (Join-Path $summaryDir "results_IpIntelligence-OnPremise.json") -Encoding utf8 + + Write-Host "Wrote performance results to '$summaryDir':" + Get-ChildItem $summaryDir -Filter "results_*.json" | ForEach-Object { Write-Host " - $($_.Name)" } +} finally { + Pop-Location +} From 141cdfe3d03159dd667d730abc8a2c964ac35561 Mon Sep 17 00:00:00 2001 From: Maksym Kucherov Date: Thu, 10 Sep 2026 09:13:58 +0200 Subject: [PATCH 02/17] CI: Download Asn file from Azure. --- ci/run-performance-tests.ps1 | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/ci/run-performance-tests.ps1 b/ci/run-performance-tests.ps1 index 781f70e..4a8bcac 100644 --- a/ci/run-performance-tests.ps1 +++ b/ci/run-performance-tests.ps1 @@ -10,6 +10,23 @@ param( $ErrorActionPreference = "Stop" $PSNativeCommandUseErrorActionPreference = $true +Push-Location "ip-intelligence-cxx/ip-intelligence-data" +try { + Write-Host "Entering $PWD" + # Remove old Asn file (if exists) + $AsnFilePath = "51Degrees-IPIV4AsnIpiV41.ipi" + if (Test-Path -Type Leaf -Path $AsnFilePath) { + Remove-Item -Path $AsnFilePath + Write-Host "Deleted $AsnFilePath" + } + + Write-Host "Loading free IPI data files..." + & ./get-lite-file-from-azure.ps1 +} finally { + Write-Host "Leaving $PWD" + Pop-Location +} + # Runs the 51Degrees Rust on-premise performance examples in release and writes # their throughput figures into results_.json files, in the same # `{ HigherIsBetter = @{ metric = value } }` shape the shared From 308b0546cfbafdde6b019f4d707ee599fb61c843 Mon Sep 17 00:00:00 2001 From: Maksym Kucherov Date: Thu, 10 Sep 2026 09:29:57 +0200 Subject: [PATCH 03/17] CI: Add `dryrun` to `nightly-performance.yml`. --- .github/workflows/nightly-performance.yml | 25 ++++++++++++++++++++--- 1 file changed, 22 insertions(+), 3 deletions(-) diff --git a/.github/workflows/nightly-performance.yml b/.github/workflows/nightly-performance.yml index 21d6fdf..657b36c 100644 --- a/.github/workflows/nightly-performance.yml +++ b/.github/workflows/nightly-performance.yml @@ -18,6 +18,16 @@ on: # 02:00 UTC daily. - cron: '0 2 * * *' workflow_dispatch: + inputs: + dryrun: + description: >- + Render the performance graphs but do not push them anywhere outside + this CI job (no commit to the gh-images / perf-images branch). + Scheduled runs always publish; this toggle only affects manual + dispatch. + required: false + type: boolean + default: false jobs: performance: @@ -127,15 +137,23 @@ jobs: # `main`, commit them to the `gh-images` branch (other branches use a # `perf-images/` prefix). The documentation benchmarks page # embeds those images by raw URL, so without `-Publish` the Rust graphs - # never appear. Best-effort: on a repository without run history the - # comparison has too few points and simply records the current figures, - # so a failure here must not fail the job. + # never appear. `-DryRun` still renders and commits the graphs locally + # (so the full render path is exercised) but suppresses the final + # `git push`, so a manually dispatched dry run leaves nothing outside + # this CI job. The `dryrun` input is empty on scheduled runs, which + # parse to `false`, so the nightly cron always publishes. Best-effort: + # on a repository without run history the comparison has too few points + # and simply records the current figures, so a failure here must not + # fail the job. - name: Compare performance and render graphs continue-on-error: true shell: pwsh env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} run: | + # The dryrun input is only set on manual dispatch; default it to + # false so scheduled runs publish normally. + $DryRun = [bool]::Parse( "${{ inputs.dryrun || 'false' }}" ) # Committing the rendered graphs onto the gh-images branch needs a git # identity, which a bare runner checkout does not configure; use the # github-actions bot identity so the commit succeeds. @@ -154,6 +172,7 @@ jobs: -OrgName '51Degrees' ` -AllOptions $options ` -Branch '${{ github.ref_name }}' ` + -DryRun $DryRun ` -Publish - name: Escape Branch Name From 6aacc82aaf9e18fadfa9700152c0366ff4e1f6dc Mon Sep 17 00:00:00 2001 From: Maksym Kucherov Date: Thu, 10 Sep 2026 10:03:19 +0200 Subject: [PATCH 04/17] CI: Add `-ErrorAction Stop` to `compare-performance.ps1` invocation. --- .github/workflows/nightly-performance.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/nightly-performance.yml b/.github/workflows/nightly-performance.yml index 657b36c..fcf4d67 100644 --- a/.github/workflows/nightly-performance.yml +++ b/.github/workflows/nightly-performance.yml @@ -172,6 +172,7 @@ jobs: -OrgName '51Degrees' ` -AllOptions $options ` -Branch '${{ github.ref_name }}' ` + -ErrorAction Stop ` -DryRun $DryRun ` -Publish From ffc004107a28055cd0b15a7a6431a6065a79d7ed Mon Sep 17 00:00:00 2001 From: Maksym Kucherov Date: Thu, 10 Sep 2026 10:56:51 +0200 Subject: [PATCH 05/17] CI: Set `PSNativeCommandUseErrorActionPreference` before `compare-performance.ps1`. --- .github/workflows/nightly-performance.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/nightly-performance.yml b/.github/workflows/nightly-performance.yml index fcf4d67..f482d27 100644 --- a/.github/workflows/nightly-performance.yml +++ b/.github/workflows/nightly-performance.yml @@ -167,6 +167,7 @@ jobs: @{ Name = "DeviceDetection-OnPremise"; RunPerformance = $true }, @{ Name = "IpIntelligence-OnPremise"; RunPerformance = $true } ) + $PSNativeCommandUseErrorActionPreference = $true ./common-ci/steps/compare-performance.ps1 ` -RepoName 'rust' ` -OrgName '51Degrees' ` From ec892e74f5f777d44b07f8b35511db6f8e309b94 Mon Sep 17 00:00:00 2001 From: Maksym Kucherov Date: Thu, 10 Sep 2026 10:58:01 +0200 Subject: [PATCH 06/17] CI: Delete `dryrun` description. --- .github/workflows/nightly-performance.yml | 6 ------ 1 file changed, 6 deletions(-) diff --git a/.github/workflows/nightly-performance.yml b/.github/workflows/nightly-performance.yml index f482d27..c4e10e6 100644 --- a/.github/workflows/nightly-performance.yml +++ b/.github/workflows/nightly-performance.yml @@ -20,12 +20,6 @@ on: workflow_dispatch: inputs: dryrun: - description: >- - Render the performance graphs but do not push them anywhere outside - this CI job (no commit to the gh-images / perf-images branch). - Scheduled runs always publish; this toggle only affects manual - dispatch. - required: false type: boolean default: false From 387e446b17819e77fa8184f01062eb77ef8ee733 Mon Sep 17 00:00:00 2001 From: Maksym Kucherov Date: Thu, 10 Sep 2026 11:59:18 +0200 Subject: [PATCH 07/17] CI: Add `common-ci-ref` to `nightly-performance.yml` --- .github/workflows/nightly-performance.yml | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/.github/workflows/nightly-performance.yml b/.github/workflows/nightly-performance.yml index c4e10e6..4142f7e 100644 --- a/.github/workflows/nightly-performance.yml +++ b/.github/workflows/nightly-performance.yml @@ -22,6 +22,18 @@ on: dryrun: type: boolean default: false + common-ci-ref: + type: string + description: "common-ci branch/tag/sha to use (default: main)" + default: '' + dd-cxx-ref: + type: string + description: "device-detection-cxx branch/tag/sha to use (default: main)" + default: '' + ipi-cxx-ref: + type: string + description: "ip-intelligence-cxx branch/tag/sha to use (default: main)" + default: '' jobs: performance: @@ -54,6 +66,7 @@ jobs: with: repository: 51Degrees/common-ci path: common-ci + ref: ${{ inputs.common-ci-ref }} # 3. The C/C++ source repositories the -sys crates compile, plus the data # submodules the performance examples read. They are checked out as @@ -68,6 +81,7 @@ jobs: with: repository: 51Degrees/device-detection-cxx path: device-detection-cxx + ref: ${{ inputs.dd-cxx-ref }} submodules: recursive - name: Checkout ip-intelligence-cxx @@ -75,6 +89,7 @@ jobs: with: repository: 51Degrees/ip-intelligence-cxx path: ip-intelligence-cxx + ref: ${{ inputs.ipi-cxx-ref }} submodules: recursive # 3a. The data files (.hash, .ipi) and the User-Agent / evidence records are From d06848ec438d6cc2ac80d60890529e4f7c3f7add Mon Sep 17 00:00:00 2001 From: Maksym Kucherov Date: Thu, 10 Sep 2026 12:03:23 +0200 Subject: [PATCH 08/17] CI: Do not hide failure of `compare-performance.ps1` --- .github/workflows/nightly-performance.yml | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/.github/workflows/nightly-performance.yml b/.github/workflows/nightly-performance.yml index 4142f7e..b071a69 100644 --- a/.github/workflows/nightly-performance.yml +++ b/.github/workflows/nightly-performance.yml @@ -150,12 +150,16 @@ jobs: # (so the full render path is exercised) but suppresses the final # `git push`, so a manually dispatched dry run leaves nothing outside # this CI job. The `dryrun` input is empty on scheduled runs, which - # parse to `false`, so the nightly cron always publishes. Best-effort: - # on a repository without run history the comparison has too few points - # and simply records the current figures, so a failure here must not - # fail the job. + # parse to `false`, so the nightly cron always publishes. + # + # This step is allowed to fail the job. The shared + # compare-performance.ps1 distinguishes its outcomes by exit code: + # the benign "not enough history yet" case exits 0, a genuine + # performance regression exits 1, and an infrastructure failure (for + # example an inability to switch to the images branch) exits 2. None of + # those should be silently swallowed, so `continue-on-error` is false. - name: Compare performance and render graphs - continue-on-error: true + continue-on-error: false shell: pwsh env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} From e76633b734b9dd7819fbd4cc02f5fcf9268657df Mon Sep 17 00:00:00 2001 From: Maksym Kucherov Date: Thu, 10 Sep 2026 12:30:03 +0200 Subject: [PATCH 09/17] CI: Reset `Cargo.lock` before running `compare-performance.ps1` --- .github/workflows/nightly-performance.yml | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/.github/workflows/nightly-performance.yml b/.github/workflows/nightly-performance.yml index b071a69..ee235a0 100644 --- a/.github/workflows/nightly-performance.yml +++ b/.github/workflows/nightly-performance.yml @@ -181,6 +181,15 @@ jobs: @{ Name = "IpIntelligence-OnPremise"; RunPerformance = $true } ) $PSNativeCommandUseErrorActionPreference = $true + # The performance run resolves examples/Cargo.lock against source.toml + # (the local-path patch), which rewrites this tracked lockfile. The + # shared compare step publishes graphs by switching to an orphan images + # branch, and git aborts that switch while a tracked file has + # uncommitted changes. Restore the committed lockfile so the tree is + # clean before the switch. Only this one file is touched by the run; + # git's "changes would be overwritten" error enumerates every colliding + # tracked file, and it lists only examples/Cargo.lock. + git -C rust checkout -- examples/Cargo.lock ./common-ci/steps/compare-performance.ps1 ` -RepoName 'rust' ` -OrgName '51Degrees' ` From 8fcdeb98f80c9c8a6185728c20c05a1550be5b37 Mon Sep 17 00:00:00 2001 From: Maksym Kucherov Date: Thu, 10 Sep 2026 12:31:20 +0200 Subject: [PATCH 10/17] Revert "CI: Set `PSNativeCommandUseErrorActionPreference` before `compare-performance.ps1`." This reverts commit ffc004107a28055cd0b15a7a6431a6065a79d7ed. # Conflicts: # .github/workflows/nightly-performance.yml --- .github/workflows/nightly-performance.yml | 1 - 1 file changed, 1 deletion(-) diff --git a/.github/workflows/nightly-performance.yml b/.github/workflows/nightly-performance.yml index ee235a0..539aec5 100644 --- a/.github/workflows/nightly-performance.yml +++ b/.github/workflows/nightly-performance.yml @@ -180,7 +180,6 @@ jobs: @{ Name = "DeviceDetection-OnPremise"; RunPerformance = $true }, @{ Name = "IpIntelligence-OnPremise"; RunPerformance = $true } ) - $PSNativeCommandUseErrorActionPreference = $true # The performance run resolves examples/Cargo.lock against source.toml # (the local-path patch), which rewrites this tracked lockfile. The # shared compare step publishes graphs by switching to an orphan images From 8c8df10c1f240b5110d6087ae6ba8aee77a8201d Mon Sep 17 00:00:00 2001 From: Maksym Kucherov Date: Wed, 16 Sep 2026 12:02:48 +0200 Subject: [PATCH 11/17] FIX: Upload performance history even when the compare step fails The compare step is allowed to fail the job, but Escape Branch Name and Upload Graphed Performance Results had no if: always(), so a failing compare skipped the history upload. That artifact is the only source of future baselines, so the run's figure never entered history and every later nightly compared against the same stale baseline. Run both steps unconditionally. --- .github/workflows/nightly-performance.yml | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/.github/workflows/nightly-performance.yml b/.github/workflows/nightly-performance.yml index 1a353f1..8069b7b 100644 --- a/.github/workflows/nightly-performance.yml +++ b/.github/workflows/nightly-performance.yml @@ -201,12 +201,17 @@ jobs: -DryRun $DryRun ` -Publish + # Runs even when the compare step failed. That upload is the only source + # of future baselines, so skipping it on a failing compare would drop the + # run's own figure out of history and wedge the trend (see below). - name: Escape Branch Name id: escape_branch + if: always() shell: pwsh run: '"name=" + ($env:GITHUB_REF_NAME -replace ''[":<>|*?/\\\r\n]'', ''-'') | Out-File $env:GITHUB_OUTPUT -Append' - name: Upload Graphed Performance Results + if: always() uses: actions/upload-artifact@v4 with: name: publish_performance_results@${{ steps.escape_branch.outputs.name }} From e8790df5b0d60bee980463b1fd1c08e9da35396e Mon Sep 17 00:00:00 2001 From: Maksym Kucherov Date: Wed, 16 Sep 2026 12:03:52 +0200 Subject: [PATCH 12/17] FIX: Resolve the ASN download against the repo directory The Push-Location into ip-intelligence-cxx/ip-intelligence-data resolved against the caller's current directory, not RepoName. The documented local invocation (-RepoName '.' from the repo root) threw at Push-Location before any benchmark ran. Resolve the sibling data directory against RepoName so both the CI and local invocations find it. --- ci/run-performance-tests.ps1 | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/ci/run-performance-tests.ps1 b/ci/run-performance-tests.ps1 index 4a8bcac..a24e5d7 100644 --- a/ci/run-performance-tests.ps1 +++ b/ci/run-performance-tests.ps1 @@ -10,7 +10,13 @@ param( $ErrorActionPreference = "Stop" $PSNativeCommandUseErrorActionPreference = $true -Push-Location "ip-intelligence-cxx/ip-intelligence-data" +# The ip-intelligence-cxx checkout is a sibling of the repo directory (CI +# checks this repo out into $RepoName next to it; a local run passes "." from +# the repo root, whose parent is the workspace holding the sibling checkout). +# Resolve the ASN data directory against $RepoName rather than the caller's +# current directory, so both invocations find it. +$AsnDataDir = Join-Path $RepoName "../ip-intelligence-cxx/ip-intelligence-data" +Push-Location $AsnDataDir try { Write-Host "Entering $PWD" # Remove old Asn file (if exists) From 2830bc175531ff4d2b71a3fb33c820b87c0a0c94 Mon Sep 17 00:00:00 2001 From: Maksym Kucherov Date: Wed, 16 Sep 2026 12:04:06 +0200 Subject: [PATCH 13/17] FIX: Force the ASN refresh instead of re-extracting a stale archive The pre-delete could not force a refresh: get-lite-file-from-azure.ps1 gates its download on the .gz archive, not the .ipi, and was called without -Force, so on a persisted workspace a stale archive was silently re-extracted. Pass -Force so the archive is re-downloaded, and -Asn so it fetches the ASN file the performance example reads. --- ci/run-performance-tests.ps1 | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/ci/run-performance-tests.ps1 b/ci/run-performance-tests.ps1 index a24e5d7..990185a 100644 --- a/ci/run-performance-tests.ps1 +++ b/ci/run-performance-tests.ps1 @@ -27,7 +27,13 @@ try { } Write-Host "Loading free IPI data files..." - & ./get-lite-file-from-azure.ps1 + # -Force re-downloads the .gz archive rather than re-extracting whatever is + # already on disk: the Azure script gates its download on the .gz, not the + # .ipi, so on a persisted workspace a stale archive would otherwise be + # silently re-extracted and the delete above would refresh nothing. -Asn + # fetches the ASN file the performance example reads + # (51DEGREES_IPI_PATH). + & ./get-lite-file-from-azure.ps1 -Force -Asn } finally { Write-Host "Leaving $PWD" Pop-Location From dd2667e025aa0bebc188b857b5bd9a6abdcde377 Mon Sep 17 00:00:00 2001 From: Maksym Kucherov Date: Wed, 16 Sep 2026 12:04:30 +0200 Subject: [PATCH 14/17] DOC: Correct the compare-step exit-code comment The comment claimed compare-performance.ps1 exits 2 on infrastructure failure, but the shared script only ever exits 0 or 1. What matters here is only that a non-zero exit fails the job, so describe it as 0 on success and non-zero otherwise rather than citing a code that does not exist. --- .github/workflows/nightly-performance.yml | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/.github/workflows/nightly-performance.yml b/.github/workflows/nightly-performance.yml index 8069b7b..713281f 100644 --- a/.github/workflows/nightly-performance.yml +++ b/.github/workflows/nightly-performance.yml @@ -156,11 +156,11 @@ jobs: # parse to `false`, so the nightly cron always publishes. # # This step is allowed to fail the job. The shared - # compare-performance.ps1 distinguishes its outcomes by exit code: - # the benign "not enough history yet" case exits 0, a genuine - # performance regression exits 1, and an infrastructure failure (for - # example an inability to switch to the images branch) exits 2. None of - # those should be silently swallowed, so `continue-on-error` is false. + # compare-performance.ps1 exits 0 when it succeeds (including the benign + # "not enough history yet" case) and non-zero otherwise, whether that is + # a genuine performance regression or an infrastructure failure such as + # an inability to switch to the images branch. Neither should be + # silently swallowed, so `continue-on-error` is false. - name: Compare performance and render graphs continue-on-error: false shell: pwsh From 5808a3fc9b1ded084e7ce9d842efa722aa840547 Mon Sep 17 00:00:00 2001 From: Maksym Kucherov Date: Wed, 16 Sep 2026 12:04:44 +0200 Subject: [PATCH 15/17] REORG: Drop the no-op -ErrorAction Stop on the compare call compare-performance.ps1 sets $ErrorActionPreference = "Stop" itself, so passing -ErrorAction Stop to it has no effect. It was a leftover from the reverted $PSNativeCommandUseErrorActionPreference attempt; remove it. --- .github/workflows/nightly-performance.yml | 1 - 1 file changed, 1 deletion(-) diff --git a/.github/workflows/nightly-performance.yml b/.github/workflows/nightly-performance.yml index 713281f..95fcf1d 100644 --- a/.github/workflows/nightly-performance.yml +++ b/.github/workflows/nightly-performance.yml @@ -197,7 +197,6 @@ jobs: -OrgName '51Degrees' ` -AllOptions $options ` -Branch '${{ github.ref_name }}' ` - -ErrorAction Stop ` -DryRun $DryRun ` -Publish From c3e741b1102f7840641c3edd9cb0c4fd9bdcc0c8 Mon Sep 17 00:00:00 2001 From: Maksym Kucherov Date: Wed, 16 Sep 2026 12:05:02 +0200 Subject: [PATCH 16/17] FIX: Keep a dry run out of the real performance history dryrun only suppressed git push; the workflow still uploaded publish_performance_results@, so a dry run on main injected its figure into the real performance history, contradicting the step comment. Guard the history upload so it is skipped on a dry run. --- .github/workflows/nightly-performance.yml | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/.github/workflows/nightly-performance.yml b/.github/workflows/nightly-performance.yml index 95fcf1d..14548e1 100644 --- a/.github/workflows/nightly-performance.yml +++ b/.github/workflows/nightly-performance.yml @@ -209,8 +209,11 @@ jobs: shell: pwsh run: '"name=" + ($env:GITHUB_REF_NAME -replace ''[":<>|*?/\\\r\n]'', ''-'') | Out-File $env:GITHUB_OUTPUT -Append' + # Runs even when the compare step failed (see above), but not on a dry + # run: this artifact is the performance history, and a dry run must leave + # nothing outside its own CI job, so it must not inject a figure here. - name: Upload Graphed Performance Results - if: always() + if: always() && github.event.inputs.dryrun != 'true' uses: actions/upload-artifact@v4 with: name: publish_performance_results@${{ steps.escape_branch.outputs.name }} From 5dab92ffa75adb11520859050502473dd8f11b6c Mon Sep 17 00:00:00 2001 From: Maksym Kucherov Date: Wed, 16 Sep 2026 12:05:19 +0200 Subject: [PATCH 17/17] DOC: Correct the nightly-unchanged claims in missing-ci-scripts.md The note claimed the nightly run behaviour is unchanged and the two run-performance-tests.ps1 copies are identical. This PR falsifies both: the repo copy adds the ASN fetch, the dryrun input, the *-ref inputs and lets the compare step fail the job. Record that the copies have diverged and this repo owns the source of truth. --- ci/missing-ci-scripts.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/ci/missing-ci-scripts.md b/ci/missing-ci-scripts.md index cbfb656..e750d22 100644 --- a/ci/missing-ci-scripts.md +++ b/ci/missing-ci-scripts.md @@ -64,12 +64,18 @@ Rust consumer, so there is nothing to share. Every other language keeps its `steps/compare-performance.ps1` is genuinely shared and still comes from common-ci. -The behaviour of the nightly run is unchanged; only the adapter's home moves. +Moving the adapter's home is the structural part of the change. Alongside it, +this repo's copy has since diverged from the common-ci original with +repository-specific fixes (the ASN data fetch, the `dryrun` input, the +`common-ci-ref`/`dd-cxx-ref`/`ipi-cxx-ref` inputs, and letting the compare step +fail the job), so the behaviour of the nightly run is not identical to the old +`common-ci/rust` path. The copy in `common-ci/rust/run-performance-tests.ps1` is intentionally **left in place** for now and will be removed in a separate, manually raised common-ci -PR. Until then the two copies are identical; this repo's copy is the one CI -uses. +PR. This repo's copy is the one CI uses, and it is now the source of truth: the +two copies are no longer identical, so the common-ci copy should not be edited +in place — it is only awaiting deletion. ## The remaining gap (Phase 2, deferred)