From a230bbcf56870d56436944ac2804ae2bdb1997c6 Mon Sep 17 00:00:00 2001 From: Jens Geyer Date: Sun, 20 Sep 2026 12:42:47 +0200 Subject: [PATCH] THRIFT-6314: Publish the Windows Thrift compiler through WinGet Client: build Windows has a package manager, and the Thrift compiler is not in it. winget install Apache.Thrift Apache.Thrift is free: the whole winget-pkgs manifest tree was swept and holds no Thrift manifest. Apache is the publisher folder the ASF already uses there, for Directory Studio, Groovy, NetBeans and OpenOffice. Note the two decoys in that folder: ApacheFriends is XAMPP and ApacheLounge is a third party httpd build, and neither is the ASF. The manifest points at archive.apache.org rather than downloads.apache.org. The latter only carries the current release, so a manifest naming it stops resolving at the next release, and takes every older version in winget-pkgs with it. The archive is permanent. That makes this a post release step: it cannot run until the release has been promoted and the archive has picked it up, the same wait the Docker Official Image update has. The workflow runs when the release is published and can be run again from the Actions tab once the archive has the file. ReleaseManagement.md says so. The checksum is computed from the file downloaded from the very URL that goes into the manifest, so a manifest cannot claim a checksum the published file does not have. The manifest records when the release came out, taken from the release event, or given on the Actions tab when the workflow is run again later. Defaulting to today would be wrong on exactly the runs the documentation asks for: the re-run, days after the release, once the archive has the file. The manifest declares Microsoft.VCRedist.2015+.x64 as a dependency. The compiler links the shared runtime and needs the Visual C++ redistributable, which the ASF cannot ship; WinGet installs it first, so a "winget install Apache.Thrift" does not end in a compiler that will not start. The rendered manifests are validated against the WinGet JSON schemas, fetched for the schema version each manifest names. Without that a mistake is only found after the pull request against winget-pkgs has been opened. test-winget-manifests.ps1 covers the renderer separately: that values land where they belong, that no placeholder survives into a file that would be submitted verbatim, that the output is UTF-8 with no BOM and LF endings, and that a malformed version, checksum or date is refused. Submission uses wingetcreate, which forks winget-pkgs itself, so nothing has to be set up there. It needs a classic personal access token with the public_repo scope - fine grained tokens are not supported - read from the environment rather than passed on the command line, which is what Microsoft's own release workflows do so that it cannot reach a log. Without the token the manifests are still rendered, validated and uploaded, and the run says in its summary how to submit them by hand. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/winget.yml | 246 ++++++++++++++++++ build/windows/README.md | 39 +++ build/windows/build-winget-manifests.ps1 | 148 +++++++++++ .../winget/Apache.Thrift.installer.yaml.in | 46 ++++ .../winget/Apache.Thrift.locale.en-US.yaml.in | 59 +++++ build/windows/winget/Apache.Thrift.yaml.in | 24 ++ .../windows/winget/test-winget-manifests.ps1 | 182 +++++++++++++ build/windows/winget/validate_manifests.py | 138 ++++++++++ doc/ReleaseManagement.md | 27 ++ 9 files changed, 909 insertions(+) create mode 100644 .github/workflows/winget.yml create mode 100644 build/windows/build-winget-manifests.ps1 create mode 100644 build/windows/winget/Apache.Thrift.installer.yaml.in create mode 100644 build/windows/winget/Apache.Thrift.locale.en-US.yaml.in create mode 100644 build/windows/winget/Apache.Thrift.yaml.in create mode 100644 build/windows/winget/test-winget-manifests.ps1 create mode 100644 build/windows/winget/validate_manifests.py diff --git a/.github/workflows/winget.yml b/.github/workflows/winget.yml new file mode 100644 index 00000000000..ba3dc7daebb --- /dev/null +++ b/.github/workflows/winget.yml @@ -0,0 +1,246 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +# Submits the Windows Thrift compiler to the Windows Package Manager, so that +# it can be installed with "winget install Apache.Thrift". +# +# The manifest points at archive.apache.org, which keeps every release, rather +# than downloads.apache.org, which only carries the current one. So this cannot +# run until the release has been promoted and the archive has picked it up. If +# the release run is too early, run it again from the Actions tab once the +# archive has the file. + +name: "WinGet" + +on: + release: + types: [published] + pull_request: + branches: + - master + paths: + - ".github/workflows/winget.yml" + - "build/windows/winget/**" + - "build/windows/build-winget-manifests.ps1" + - "doc/ReleaseManagement.md" + workflow_dispatch: + inputs: + version: + description: "Released version to submit, for example 0.26.0. Taken from CMakeLists.txt when empty." + required: false + type: string + release_date: + description: "Release date as yyyy-MM-dd. Today when empty, which is only right on the day of the release." + required: false + type: string + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +permissions: + contents: read + +jobs: + manifests: + name: Render manifests + runs-on: ubuntu-latest + timeout-minutes: 20 + permissions: + contents: read + outputs: + version: ${{ steps.version.outputs.version }} + release_date: ${{ steps.version.outputs.release_date }} + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + + - name: Test the manifest renderer + shell: pwsh + run: ./build/windows/winget/test-winget-manifests.ps1 + + - name: Determine the version and release date + id: version + shell: pwsh + env: + RELEASE_TAG: ${{ github.event.release.tag_name }} + RELEASE_PUBLISHED_AT: ${{ github.event.release.published_at }} + INPUT_VERSION: ${{ inputs.version }} + INPUT_RELEASE_DATE: ${{ inputs.release_date }} + run: | + $version = '' + if ($env:INPUT_VERSION) { + $version = $env:INPUT_VERSION + } + elseif ($env:RELEASE_TAG) { + $version = $env:RELEASE_TAG -replace '^v', '' + } + else { + foreach ($line in Get-Content CMakeLists.txt) { + if ($line -match '^\s*set\(thrift_VERSION\s+"(\d+\.\d+\.\d+)"\)') { + $version = $Matches[1] + break + } + } + } + if ($version -notmatch '^\d+\.\d+\.\d+$') { + Write-Error "Could not determine a major.minor.patch version (got '$version')." + exit 1 + } + + # The manifest records when the release came out. Defaulting to + # today would be wrong on exactly the runs the documentation asks + # for: the re-run, days later, once the archive has the file. + $releaseDate = '' + if ($env:INPUT_RELEASE_DATE) { + $releaseDate = $env:INPUT_RELEASE_DATE + } + elseif ($env:RELEASE_PUBLISHED_AT) { + $releaseDate = ([datetime]$env:RELEASE_PUBLISHED_AT).ToUniversalTime().ToString('yyyy-MM-dd') + } + else { + $releaseDate = (Get-Date).ToUniversalTime().ToString('yyyy-MM-dd') + Write-Host "::notice::No release date given; using today, $releaseDate." + } + if ($releaseDate -notmatch '^\d{4}-\d{2}-\d{2}$') { + Write-Error "Release date '$releaseDate' is not yyyy-MM-dd." + exit 1 + } + + Write-Host "Version: $version" + Write-Host "Release date: $releaseDate" + "version=$version" | Out-File -FilePath $env:GITHUB_OUTPUT -Append + "release_date=$releaseDate" | Out-File -FilePath $env:GITHUB_OUTPUT -Append + + - name: Render the manifests + shell: pwsh + env: + VERSION: ${{ steps.version.outputs.version }} + RELEASE_DATE: ${{ steps.version.outputs.release_date }} + IS_RELEASE: ${{ github.event_name == 'release' }} + run: | + if ($env:IS_RELEASE -eq 'true') { + # No checksum given, so the renderer downloads the published + # installer and hashes that. The manifest then cannot claim a + # checksum the published file does not have. + ./build/windows/build-winget-manifests.ps1 -Version $env:VERSION ` + -ReleaseDate $env:RELEASE_DATE -OutputDir winget-manifests + } + else { + # Nothing is published for a pull request, so the manifests are + # rendered with an obviously fake checksum. They are still checked + # against the WinGet schemas below; they are just not submittable, + # and the job does not upload them. + Write-Host '::notice::Rendering with a placeholder checksum; this run cannot submit.' + ./build/windows/build-winget-manifests.ps1 -Version $env:VERSION ` + -ReleaseDate $env:RELEASE_DATE -Sha256 ('DEADBEEF' * 8) -OutputDir winget-manifests + } + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.12" + + - name: Validate the manifests + env: + VERSION: ${{ steps.version.outputs.version }} + run: | + python -m pip install --quiet pyyaml jsonschema + python build/windows/winget/validate_manifests.py \ + "winget-manifests/manifests/a/Apache/Thrift/${VERSION}" + + - name: Show the manifests + env: + VERSION: ${{ steps.version.outputs.version }} + run: | + for manifest in "winget-manifests/manifests/a/Apache/Thrift/${VERSION}"/*.yaml; do + { + echo "### $(basename "$manifest")" + echo '```yaml' + # The default shell is "bash -e" with no pipefail, but a grep + # that matches nothing still returns 1 and would end the step. + grep -v '^#' "$manifest" | grep -v '^$' || true + echo '```' + } >> "$GITHUB_STEP_SUMMARY" + done + + - name: Archive the manifests + if: ${{ github.event_name != 'pull_request' }} + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + with: + name: winget-manifests + if-no-files-found: error + path: winget-manifests/ + retention-days: 14 + + submit: + name: Submit to winget-pkgs + needs: manifests + if: ${{ github.event_name == 'release' && !github.event.release.prerelease }} + # wingetcreate only runs on Windows. + runs-on: windows-2025 + timeout-minutes: 20 + environment: release + permissions: + contents: read + steps: + - name: Download the manifests + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: winget-manifests + path: winget-manifests + + - name: Submit + shell: pwsh + env: + # Read from the environment rather than passed as --token, which + # Microsoft warn can end up in a log. wingetcreate forks + # microsoft/winget-pkgs itself, so nothing is set up here. + WINGET_CREATE_GITHUB_TOKEN: ${{ secrets.WINGET_TOKEN }} + VERSION: ${{ needs.manifests.outputs.version }} + run: | + $manifestDir = "winget-manifests/manifests/a/Apache/Thrift/$env:VERSION" + + if (-not $env:WINGET_CREATE_GITHUB_TOKEN) { + # Not a failure: the manifests are ready and someone can submit + # them by hand. Say so loudly enough to be noticed. + Write-Host '::warning::WINGET_TOKEN is not set, so nothing was submitted to winget-pkgs.' + @( + '### WinGet submission skipped', + '', + 'The `WINGET_TOKEN` secret is not set in the `release` environment, so the', + 'manifests were rendered but not submitted. They are attached to this run as the', + '`winget-manifests` artifact. To submit them by hand, from a Windows machine:', + '', + '```powershell', + 'curl.exe -JLO https://aka.ms/wingetcreate/latest', + ".\wingetcreate.exe submit --prtitle ""Apache.Thrift version $env:VERSION"" ", + '```', + '', + 'The token has to be a *classic* personal access token with the `public_repo`', + 'scope. Fine grained tokens are not supported by wingetcreate.' + ) | Out-File -FilePath $env:GITHUB_STEP_SUMMARY -Append + exit 0 + } + + curl.exe -JLO https://aka.ms/wingetcreate/latest + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + + ./wingetcreate.exe submit --prtitle "Apache.Thrift version $env:VERSION" $manifestDir + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } diff --git a/build/windows/README.md b/build/windows/README.md index f88b8d82f23..f3d4c00233d 100644 --- a/build/windows/README.md +++ b/build/windows/README.md @@ -189,3 +189,42 @@ Given `-Package`, it also looks inside a packed `.nupkg` - including that it carries no executable, since the compiler is downloaded at install time. Runs anywhere PowerShell does; only the `choco pack` step needs Windows. + +## `build-winget-manifests.ps1` and `winget/` + +Renders the three manifest files a submission to +[`microsoft/winget-pkgs`](https://github.com/microsoft/winget-pkgs) needs, from +the templates in `winget/`, so that the compiler can be installed with +`winget install Apache.Thrift`. + +```bash +$ pwsh build/windows/build-winget-manifests.ps1 -Version 0.26.0 +``` + +The manifest points at `archive.apache.org`, which keeps every release. +`downloads.apache.org` only carries the current one, so a manifest naming it +would stop working at the next release and take every older version in +winget-pkgs with it. + +Unless `-Sha256` is given, the installer is downloaded from the very URL that +goes into the manifest and hashed, so the manifest cannot claim a checksum the +published file does not have. That also means this cannot be run before the +release has reached the archive. + +### `winget/test-winget-manifests.ps1` + +Tests the renderer: that the values land where they belong, that no placeholder +survives into a file that would be submitted verbatim, that the output is UTF-8 +without a BOM and with LF endings, and that a malformed version, checksum or +date is refused. + +### `winget/validate_manifests.py` + +Validates rendered manifests against the WinGet JSON schemas, fetched from +Microsoft for the schema version each manifest names. A manifest that does not +match is otherwise rejected only after the pull request has been opened. + +```bash +$ python3 -m pip install pyyaml jsonschema +$ python3 build/windows/winget/validate_manifests.py winget-manifests/manifests/a/Apache/Thrift/0.26.0 +``` diff --git a/build/windows/build-winget-manifests.ps1 b/build/windows/build-winget-manifests.ps1 new file mode 100644 index 00000000000..063aa8660cb --- /dev/null +++ b/build/windows/build-winget-manifests.ps1 @@ -0,0 +1,148 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +<# +.SYNOPSIS + Renders the WinGet manifests for a released Windows Thrift compiler. + +.DESCRIPTION + Fills in the templates in build/windows/winget and writes the three files + a submission to microsoft/winget-pkgs needs. + + Unless a checksum is given, the installer is downloaded from the very URL + that goes into the manifest and hashed, so that the manifest cannot claim a + checksum the published file does not have. That also means this cannot run + before the release has reached the archive. + +.PARAMETER Version + The released version, for example 0.26.0. + +.PARAMETER InstallerUrl + Where the installer is published. Defaults to the Apache archive, which + keeps every release; downloads.apache.org only carries the current one, so + a manifest naming it stops working at the next release. + +.PARAMETER Sha256 + The installer's SHA-256. Computed from the downloaded file when omitted. + +.PARAMETER ReleaseDate + The release date, yyyy-MM-dd. Today when omitted. + +.PARAMETER OutputDir + Where to write the manifests. They land in a + manifests/a/Apache/Thrift/ subtree, which is the layout + winget-pkgs uses and wingetcreate expects. + +.EXAMPLE + pwsh build/windows/build-winget-manifests.ps1 -Version 0.26.0 +#> + +[CmdletBinding()] +param( + [Parameter(Mandatory = $true)] + [string] $Version, + [string] $InstallerUrl = '', + [string] $Sha256 = '', + [string] $ReleaseDate = '', + [string] $OutputDir = 'winget-manifests' +) + +$ErrorActionPreference = 'Stop' + +$PackageIdentifier = 'Apache.Thrift' + +if ($Version -notmatch '^\d+\.\d+\.\d+$') { + throw "Version '$Version' is not major.minor.patch." +} +if (-not $InstallerUrl) { + $InstallerUrl = "https://archive.apache.org/dist/thrift/$Version/thrift-$Version-setup.exe" +} +if (-not $ReleaseDate) { + $ReleaseDate = (Get-Date).ToString('yyyy-MM-dd') +} +if ($ReleaseDate -notmatch '^\d{4}-\d{2}-\d{2}$') { + throw "ReleaseDate '$ReleaseDate' is not yyyy-MM-dd." +} + +if ($Sha256) { + if ($Sha256 -notmatch '^[0-9a-fA-F]{64}$') { + throw "Sha256 '$Sha256' is not 64 hexadecimal digits." + } + $Sha256 = $Sha256.ToUpperInvariant() +} +else { + Write-Host "Downloading $InstallerUrl" + $download = Join-Path ([System.IO.Path]::GetTempPath()) ([System.Guid]::NewGuid().ToString('N') + '.exe') + try { + try { + Invoke-WebRequest -Uri $InstallerUrl -OutFile $download -MaximumRedirection 5 + } + catch { + throw ("Could not download $InstallerUrl : " + $_.Exception.Message + "`n" + + 'If the release was just promoted, the Apache archive may not have picked it up yet. Wait and try again.') + } + $Sha256 = (Get-FileHash -Algorithm SHA256 -LiteralPath $download).Hash.ToUpperInvariant() + Write-Host " sha256: $Sha256" + } + finally { + Remove-Item -LiteralPath $download -Force -ErrorAction SilentlyContinue + } +} + +$templateDir = Join-Path $PSScriptRoot 'winget' +$targetDir = Join-Path $OutputDir "manifests/a/Apache/Thrift/$Version" +New-Item -Path $targetDir -ItemType Directory -Force | Out-Null + +$substitutions = @{ + '__VERSION__' = $Version + '__INSTALLER_URL__' = $InstallerUrl + '__INSTALLER_SHA256__' = $Sha256 + '__RELEASE_DATE__' = $ReleaseDate +} + +$written = [System.Collections.Generic.List[string]]::new() +foreach ($suffix in @('', '.installer', '.locale.en-US')) { + $name = "$PackageIdentifier$suffix.yaml" + $template = Join-Path $templateDir "$name.in" + if (-not (Test-Path -LiteralPath $template)) { + throw "Template not found: $template" + } + + $content = Get-Content -LiteralPath $template -Raw + foreach ($key in $substitutions.Keys) { + $content = $content.Replace($key, $substitutions[$key]) + } + + # A placeholder that survived would be submitted verbatim, so stop here + # rather than open a pull request with "__VERSION__" in it. + $leftovers = [regex]::Matches($content, '__[A-Z0-9_]+__') + if ($leftovers.Count -gt 0) { + throw ("$name still holds placeholders after substitution: " + + (($leftovers | ForEach-Object { $_.Value } | Sort-Object -Unique) -join ', ')) + } + + $target = Join-Path $targetDir $name + # winget-pkgs manifests are UTF-8; no BOM, and no CRLF from PowerShell. + [System.IO.File]::WriteAllText($target, ($content -replace "`r`n", "`n"), + [System.Text.UTF8Encoding]::new($false)) + $written.Add($target) + Write-Host "Wrote $target" +} + +Write-Output $targetDir diff --git a/build/windows/winget/Apache.Thrift.installer.yaml.in b/build/windows/winget/Apache.Thrift.installer.yaml.in new file mode 100644 index 00000000000..c9dfe1e38e8 --- /dev/null +++ b/build/windows/winget/Apache.Thrift.installer.yaml.in @@ -0,0 +1,46 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# yaml-language-server: $schema=https://aka.ms/winget-manifest.installer.1.12.0.schema.json + +PackageIdentifier: Apache.Thrift +PackageVersion: __VERSION__ +MinimumOSVersion: 10.0.0.0 +InstallerType: inno +Scope: user +InstallModes: + - interactive + - silent + - silentWithProgress +UpgradeBehavior: install +# Inno Setup registers its uninstaller under "_is1". The AppId is fixed +# in build/windows/installer/thrift.iss and must never change. +ProductCode: '{064E0C5D-29EE-47DD-BB5E-172FE25FE936}_is1' +ReleaseDate: __RELEASE_DATE__ +# The compiler links the shared runtime, so it needs the Visual C++ +# redistributable, which the ASF cannot ship. WinGet installs this first. +Dependencies: + PackageDependencies: + - PackageIdentifier: Microsoft.VCRedist.2015+.x64 +Installers: + - Architecture: x64 + InstallerUrl: __INSTALLER_URL__ + # Quoted: a checksum that happened to be all digits would otherwise be + # read as a YAML number and rejected by the schema as not a string. + InstallerSha256: '__INSTALLER_SHA256__' +ManifestType: installer +ManifestVersion: 1.12.0 diff --git a/build/windows/winget/Apache.Thrift.locale.en-US.yaml.in b/build/windows/winget/Apache.Thrift.locale.en-US.yaml.in new file mode 100644 index 00000000000..5e73e5c864d --- /dev/null +++ b/build/windows/winget/Apache.Thrift.locale.en-US.yaml.in @@ -0,0 +1,59 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# yaml-language-server: $schema=https://aka.ms/winget-manifest.defaultLocale.1.12.0.schema.json + +PackageIdentifier: Apache.Thrift +PackageVersion: __VERSION__ +PackageLocale: en-US +Publisher: Apache Software Foundation +PublisherUrl: https://www.apache.org/ +PublisherSupportUrl: https://thrift.apache.org/docs/ +PrivacyUrl: https://privacy.apache.org/policies/privacy-policy-public.html +Author: Apache Thrift Developers +PackageName: Apache Thrift Compiler +PackageUrl: https://thrift.apache.org/ +License: Apache-2.0 +LicenseUrl: https://www.apache.org/licenses/LICENSE-2.0 +Copyright: Copyright (c) The Apache Software Foundation +CopyrightUrl: https://www.apache.org/licenses/LICENSE-2.0 +ShortDescription: The Apache Thrift IDL compiler +Description: |- + Apache Thrift is a software framework for scalable cross-language services + development. It combines a software stack with a code generation engine to + build services that work efficiently between a large number of languages. + + This package contains the Thrift compiler, which turns a .thrift interface + definition into client and server code. The runtime libraries are distributed + separately, through each language's own package manager. +Moniker: thrift +Tags: + - apache + - codegen + - compiler + - idl + - rpc + - serialization + - thrift +ReleaseNotesUrl: https://github.com/apache/thrift/blob/master/CHANGES.md +Documentations: + - DocumentLabel: Documentation + DocumentUrl: https://thrift.apache.org/docs/ + - DocumentLabel: Tutorial + DocumentUrl: https://thrift.apache.org/tutorial/ +ManifestType: defaultLocale +ManifestVersion: 1.12.0 diff --git a/build/windows/winget/Apache.Thrift.yaml.in b/build/windows/winget/Apache.Thrift.yaml.in new file mode 100644 index 00000000000..0e2f84e2584 --- /dev/null +++ b/build/windows/winget/Apache.Thrift.yaml.in @@ -0,0 +1,24 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# yaml-language-server: $schema=https://aka.ms/winget-manifest.version.1.12.0.schema.json + +PackageIdentifier: Apache.Thrift +PackageVersion: __VERSION__ +DefaultLocale: en-US +ManifestType: version +ManifestVersion: 1.12.0 diff --git a/build/windows/winget/test-winget-manifests.ps1 b/build/windows/winget/test-winget-manifests.ps1 new file mode 100644 index 00000000000..7ffe347e327 --- /dev/null +++ b/build/windows/winget/test-winget-manifests.ps1 @@ -0,0 +1,182 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +<# +.SYNOPSIS + Tests build/windows/build-winget-manifests.ps1. + +.DESCRIPTION + Checks what the renderer puts in the manifests and what it refuses to do. + Whether the result matches the WinGet schemas is a separate question, + answered by validate_manifests.py. + + Runs anywhere PowerShell does and needs no network: the checksum is passed + in rather than computed from a download. + +.EXAMPLE + pwsh build/windows/winget/test-winget-manifests.ps1 +#> + +[CmdletBinding()] +param() + +$ErrorActionPreference = 'Stop' + +$renderer = Join-Path $PSScriptRoot '..\build-winget-manifests.ps1' +if (-not (Test-Path -LiteralPath $renderer)) { + throw "build-winget-manifests.ps1 not found next to $PSCommandPath" +} + +$workDir = Join-Path ([System.IO.Path]::GetTempPath()) ("thrift-winget-tests-" + [System.Guid]::NewGuid().ToString('N')) + +$failures = [System.Collections.Generic.List[string]]::new() +$passed = 0 + +function Assert-True { + param([string] $Name, [bool] $Condition, [string] $Detail = '') + + if ($Condition) { + Write-Host " PASS $Name" + $script:passed++ + } + else { + Write-Host " FAIL $Name$(if ($Detail) { " - $Detail" })" + $script:failures.Add($Name) + } +} + +function Assert-Throws { + param([string] $Name, [scriptblock] $Action, [string] $Matching = '') + + try { + & $Action | Out-Null + Assert-True $Name $false 'it was accepted' + } + catch { + if ($Matching -and ($_.Exception.Message -notmatch $Matching)) { + Assert-True $Name $false "wrong message: $($_.Exception.Message)" + } + else { + Assert-True $Name $true + } + } +} + +$version = '1.2.3' +$sha = 'B' * 64 +$url = 'https://archive.apache.org/dist/thrift/1.2.3/thrift-1.2.3-setup.exe' +$date = '2026-09-20' + +Write-Host 'build-winget-manifests.ps1' + +try { + $outputDir = Join-Path $workDir 'out' + $targetDir = & $renderer -Version $version -Sha256 $sha -ReleaseDate $date -OutputDir $outputDir | + Select-Object -Last 1 + + # wingetcreate and winget-pkgs both expect this layout. + $expectedDir = Join-Path $outputDir "manifests/a/Apache/Thrift/$version" + Assert-True 'the manifests land in manifests/a/Apache/Thrift/' ` + ((Resolve-Path -LiteralPath $targetDir).Path -eq (Resolve-Path -LiteralPath $expectedDir).Path) ` + "got $targetDir" + + $names = @('Apache.Thrift.yaml', 'Apache.Thrift.installer.yaml', 'Apache.Thrift.locale.en-US.yaml') + foreach ($name in $names) { + Assert-True "$name is written" (Test-Path -LiteralPath (Join-Path $targetDir $name)) + } + + $installer = Get-Content -LiteralPath (Join-Path $targetDir 'Apache.Thrift.installer.yaml') -Raw + $locale = Get-Content -LiteralPath (Join-Path $targetDir 'Apache.Thrift.locale.en-US.yaml') -Raw + $versionManifest = Get-Content -LiteralPath (Join-Path $targetDir 'Apache.Thrift.yaml') -Raw + + # A placeholder left in place would be submitted verbatim. + foreach ($name in $names) { + $content = Get-Content -LiteralPath (Join-Path $targetDir $name) -Raw + Assert-True "$name holds no placeholders" ($content -notmatch '__[A-Z0-9_]+__') + } + + Assert-True 'the version reaches every manifest' ` + (($installer -match "PackageVersion: $([regex]::Escape($version))") -and + ($locale -match "PackageVersion: $([regex]::Escape($version))") -and + ($versionManifest -match "PackageVersion: $([regex]::Escape($version))")) + + Assert-True 'the checksum reaches the installer manifest' ` + ($installer -match "InstallerSha256: '$sha'") + Assert-True 'the release date reaches the installer manifest' ` + ($installer -match "ReleaseDate: $([regex]::Escape($date))") + + # The compiler needs the Visual C++ redistributable; without this a WinGet + # install would succeed and then not run. + Assert-True 'the redistributable is declared as a dependency' ` + ($installer -match 'PackageIdentifier:\s*Microsoft\.VCRedist\.2015\+\.x64') $installer + + # downloads.apache.org only carries the current release, so a manifest + # naming it stops working at the next one. + Assert-True 'the installer URL defaults to the Apache archive' ` + ($installer -match "InstallerUrl: $([regex]::Escape($url))") $installer + Assert-True 'the installer URL is not downloads.apache.org' ` + ($installer -notmatch 'downloads\.apache\.org') + + # A checksum of nothing but digits is still a string. Unquoted, YAML reads + # it as a number and the WinGet schema rejects it. + $numeric = '0' * 64 + $numericDir = & $renderer -Version $version -Sha256 $numeric -ReleaseDate $date ` + -OutputDir (Join-Path $workDir 'numeric') | Select-Object -Last 1 + $numericInstaller = Get-Content -LiteralPath (Join-Path $numericDir 'Apache.Thrift.installer.yaml') -Raw + Assert-True 'an all digit checksum stays quoted' ` + ($numericInstaller -match "InstallerSha256: '$numeric'") $numericInstaller + + $custom = & $renderer -Version $version -Sha256 $sha -ReleaseDate $date ` + -InstallerUrl 'https://example.invalid/thrift.exe' ` + -OutputDir (Join-Path $workDir 'custom') | Select-Object -Last 1 + $customInstaller = Get-Content -LiteralPath (Join-Path $custom 'Apache.Thrift.installer.yaml') -Raw + Assert-True 'a given installer URL is used' ` + ($customInstaller -match 'InstallerUrl: https://example\.invalid/thrift\.exe') + + # winget-pkgs manifests are UTF-8 without a BOM, and PowerShell is happy to + # write both a BOM and CRLF if left to itself. + $bytes = [System.IO.File]::ReadAllBytes((Join-Path $targetDir 'Apache.Thrift.yaml')) + Assert-True 'the manifests have no byte order mark' ` + (-not (($bytes.Length -ge 3) -and ($bytes[0] -eq 0xEF) -and ($bytes[1] -eq 0xBB) -and ($bytes[2] -eq 0xBF))) + Assert-True 'the manifests use LF line endings' ($versionManifest -notmatch "`r") + + # ---- what it refuses --------------------------------------------------- + + Assert-Throws 'a version that is not major.minor.patch is refused' ` + { & $renderer -Version '1.2' -Sha256 $sha -OutputDir (Join-Path $workDir 'bad1') } 'major.minor.patch' + + Assert-Throws 'a checksum that is not 64 hex digits is refused' ` + { & $renderer -Version $version -Sha256 'nope' -OutputDir (Join-Path $workDir 'bad2') } 'hexadecimal' + + Assert-Throws 'a release date that is not yyyy-MM-dd is refused' ` + { & $renderer -Version $version -Sha256 $sha -ReleaseDate '20.09.2026' -OutputDir (Join-Path $workDir 'bad3') } 'yyyy-MM-dd' +} +finally { + Remove-Item -LiteralPath $workDir -Recurse -Force -ErrorAction SilentlyContinue +} + +Write-Host '' +if ($failures.Count -gt 0) { + Write-Host "$($failures.Count) of $($failures.Count + $passed) checks failed:" + foreach ($failure in $failures) { Write-Host " $failure" } + exit 1 +} + +Write-Host "$passed checks passed." +exit 0 diff --git a/build/windows/winget/validate_manifests.py b/build/windows/winget/validate_manifests.py new file mode 100644 index 00000000000..fb081b223a4 --- /dev/null +++ b/build/windows/winget/validate_manifests.py @@ -0,0 +1,138 @@ +#!/usr/bin/env python3 +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +"""Validate WinGet manifests against the published JSON schemas. + +A manifest that does not match its schema is rejected by microsoft/winget-pkgs +after the pull request has been opened, which is an awkward place to find out. +This checks the rendered manifests before they are submitted. + +Each manifest names its own type and schema version, and the matching schema is +fetched from Microsoft, so this follows the manifests rather than pinning a +schema version here. + +Usage: + python3 validate_manifests.py + +Requires PyYAML and jsonschema. +""" + +import argparse +import datetime +import json +import pathlib +import sys +import urllib.request + +import yaml +from jsonschema import Draft7Validator + +SCHEMA_URL = "https://aka.ms/winget-manifest.{type}.{version}.schema.json" + +# Every manifest a multi file submission needs, and nothing else: a directory +# missing one of them is not submittable. +REQUIRED_TYPES = {"version", "installer", "defaultLocale"} + + +def load_schema(manifest_type, manifest_version, cache): + """Fetch and cache the schema for one manifest type and version.""" + key = (manifest_type, manifest_version) + if key not in cache: + url = SCHEMA_URL.format(type=manifest_type, version=manifest_version) + with urllib.request.urlopen(url, timeout=60) as response: + cache[key] = json.loads(response.read().decode("utf-8")) + return cache[key] + + +def jsonify(value): + """Turn YAML scalars into what JSON Schema expects. + + YAML resolves an unquoted 2026-09-20 to a date, and the schema wants the + string it was written as. + """ + if isinstance(value, dict): + return {k: jsonify(v) for k, v in value.items()} + if isinstance(value, list): + return [jsonify(v) for v in value] + if isinstance(value, (datetime.datetime, datetime.date)): + return value.isoformat() + return value + + +def validate(path, cache): + """Validate one manifest. Returns a list of human readable problems.""" + document = jsonify(yaml.safe_load(path.read_text(encoding="utf-8"))) + if not isinstance(document, dict): + return [f"{path.name}: not a YAML mapping"] + + manifest_type = document.get("ManifestType") + manifest_version = document.get("ManifestVersion") + if not manifest_type or not manifest_version: + return [f"{path.name}: ManifestType or ManifestVersion is missing"] + + schema = load_schema(manifest_type, manifest_version, cache) + problems = [] + for error in sorted(Draft7Validator(schema).iter_errors(document), key=str): + location = "/".join(str(part) for part in error.absolute_path) or "(document)" + problems.append(f"{path.name}: {location}: {error.message}") + return problems + + +def main(argv=None): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("directory", help="directory holding the rendered manifests") + args = parser.parse_args(argv) + + directory = pathlib.Path(args.directory) + manifests = sorted(directory.glob("*.yaml")) + if not manifests: + print(f"error: no .yaml manifests in {directory}", file=sys.stderr) + return 1 + + cache = {} + problems = [] + seen_types = set() + + for manifest in manifests: + found = validate(manifest, cache) + if found: + problems.extend(found) + else: + print(f"ok {manifest.name}") + document = yaml.safe_load(manifest.read_text(encoding="utf-8")) + if isinstance(document, dict): + seen_types.add(document.get("ManifestType")) + + missing = REQUIRED_TYPES - seen_types + if missing: + problems.append("missing manifest types: " + ", ".join(sorted(missing))) + + if problems: + print(f"\n{len(problems)} problem(s):", file=sys.stderr) + for problem in problems: + print(f" {problem}", file=sys.stderr) + return 1 + + print(f"\n{len(manifests)} manifests validated.") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/doc/ReleaseManagement.md b/doc/ReleaseManagement.md index 0e12b9a4dfc..a269a0b806d 100644 --- a/doc/ReleaseManagement.md +++ b/doc/ReleaseManagement.md @@ -508,6 +508,33 @@ refused, the free ids `apache-thrift` and `thrift-compiler` are the fallback; changing the id means editing `build/windows/chocolatey/thrift.nuspec.in` and the package name the build script expects. +##### WinGet + +The [`WinGet`](../.github/workflows/winget.yml) workflow submits the installer to +the [Windows Package Manager](https://github.com/microsoft/winget-pkgs) as +`Apache.Thrift`, so that it can be installed with `winget install Apache.Thrift`. + +It runs when the release is published, but it can only succeed once the release +has been promoted to `dist.apache.org` **and** `archive.apache.org` has picked it +up - the manifest points at the archive, because `downloads.apache.org` only +carries the current release and a manifest naming it would stop working at the +next one. This is the same wait the Docker Official Image update has. + +So expect to run it again, from the Actions tab, a while after the release: + +1. Actions → `WinGet` → *Run workflow*, entering the released version. +1. The workflow renders the manifests, downloads the published installer to + compute its checksum, validates the manifests against the WinGet schemas, and + opens a pull request against `microsoft/winget-pkgs`. +1. Watch that pull request. Submissions are reviewed, and validation failures + are reported there. + +For the submission step the `release` environment needs a secret named +`WINGET_TOKEN`: a **classic** GitHub personal access token with the `public_repo` +scope. Fine-grained tokens are not supported by `wingetcreate`. Without it the +workflow still renders and validates the manifests and leaves them as an +artifact, and says in its summary how to submit them by hand. + #### Third Party Package Managers See https://thrift.apache.org/lib/ for the current status of each external package manager's distribution. The information below is from the 0.12.0 release: