Skip to content

IoTEdgeDev Python 311-314 upgrade - #639

Open
DrewH-ms wants to merge 15 commits into
mainfrom
drew-python-311-314-upgrade
Open

DrewH-ms wants to merge 15 commits into
mainfrom
drew-python-311-314-upgrade

Conversation

@DrewH-ms

@DrewH-ms DrewH-ms commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator

Upgrades python_requires from >=3.6, <3.10 to >=3.11, <3.15 and moves CI to a 3.11–3.14 matrix. No product code changesgit diff main..HEAD -- iotedgedev/ is empty.

  • setup.py — new Python range and classifiers; dependency floors raised. Removes the upper caps that made the tree unresolvable on modern Python (azure-cli-core < 2.35.0, requests <= 2.25.1, more-itertools < 8.1.0, bcrypt <= 3.1.7). Swaps pypiwin32pywin32 and bumpversionbump2version (the old release dies on 3.12 via ConfigParser.readfp()).
  • vsts_ci/ — matrix is now 3.11–3.14 on windows-2022, ubuntu-22.04, ubuntu-24.04. Tests authenticate to the test registry with a short-lived, registry-scoped AAD token instead of admin credentials, which stay disabled.
  • azure-pipelines.yml — release build targets 3.11.
  • Docs / devcontainerCONTRIBUTING.md, setup docs and .devcontainer updated to the supported range.

Version bumped to 3.3.9 (patch — no product behavior change). 3.3.8 was set on main but never published to PyPI, where the latest release remains 3.3.7.

Release Docker image

docker/tool/linux no longer built at all on main — the NodeSource node_14.x signing key was retired and the Maven 3.5.4 URL now 404s — and its Python 3.9 could not have installed the new wheel. Re-based on Ubuntu 24.04, whose default python3 is 3.12:

  • Node 14 → 20, .NET SDK 2.1 → 8.0, Maven 3.5.4 → 3.8.7 (apt, no tarball), Java 8 retained.
  • docker-cedocker-ce-cli + docker-compose-plugin; the container talks to the host daemon over the mounted socket, so the daemon package was dead weight.
  • iotedgedev installs into a virtualenv at /opt/iotedgedev. Ubuntu 24.04 is PEP 668 managed, so the previous sudo pip3 install of the wheel would fail outright.

The step that was supposed to catch this ran docker run against an image with no CMD: the container started, exited 0, and passed unconditionally. It now asserts the interpreter is in range, prints every tool version, and scaffolds C#, Node.js, Python and Java modules. docs/test-coverage.md was also years out of date and now matches the real matrix.

Verification

  • CI build 8099: 12/12 jobs green, 119 passed, 1 skipped per job.
  • Release build 8102: builds and installs the wheel end to end under the new Requires-Python.
  • Locally on Python 3.14: pip install -e . succeeds and pytest -m unit gives 73 passed.
  • Docker image: toolchain layers build clean and report python 3.12.3, dotnet 8.0.131, node 20.20.2, mvn 3.8.7, java 1.8.0_502. The wheel installs into the venv; against the system interpreter the same install fails with externally-managed-environment, which is the behaviour the venv exists to avoid. The npm and PyPI layers can only be exercised on a build agent — corporate egress blocks both from a local container.

@DrewH-ms

Copy link
Copy Markdown
Collaborator Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 2 pipeline(s).

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The release Docker path must be updated from Python 3.9, and the linked test-coverage documentation needs correction.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Upgrades IoTEdgeDev for Python 3.11–3.14, refreshes dependencies and tooling, and prepares release 3.3.9.

Changes:

  • Updates Python and dependency requirements.
  • Expands CI and tox coverage across Python 3.11–3.14.
  • Modernizes ACR authentication and release tooling.
  • Updates documentation, containers, and version metadata.
File summaries
File Summary Review finding
vsts_ci/win32/continuous-build-win32.yml Windows CI and ACR authentication
vsts_ci/linux/continuous-build-linux.yml Linux CI and ACR authentication
vsts_ci/.vsts-ci.yml Expanded CI matrix
tox.ini Updated test environments
tests/__init__.py Version metadata
setup.py Python range, classifiers, dependencies, and version Critical (3 votes): Release Docker tooling still uses Python 3.9, which will reject this wheel’s Requires-Python metadata.
setup.cfg Version configuration
requirements.txt Runtime dependency updates
requirements_dev.txt Development dependency updates
iotedgedev/__init__.py Package version bump
docs/environment-setup/manual-dev-machine-setup.md Supported Python documentation Nit (1 vote): Linked test-coverage documentation still lists only older Python versions.
CONTRIBUTING.md Updated development instructions
CHANGELOG.md Release notes for 3.3.9
azure-pipelines.yml Release Python version
.devcontainer/Dockerfile Development container Python version
Review details

Suppressed comments (1)

docs/environment-setup/manual-dev-machine-setup.md:7

  • The linked docs/test-coverage.md still lists only Python 2.7.13, 3.6.5, and 3.7.0 as tested. This new setup instruction therefore contradicts its own test-coverage link and does not document the newly supported 3.11–3.14 range; update that page in the same change.
1. Install **Python 3.11-3.14** and **pip**
  • Files reviewed: 14/15 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread setup.py
@DrewH-ms

Copy link
Copy Markdown
Collaborator Author

/azp run

@DrewH-ms
DrewH-ms requested a lite review from Copilot September 11, 2026 18:15
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 2 pipeline(s).

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A critical Docker socket-access issue, a moderate Windows Python 3.9 incompatibility, and a nit in the legacy Compose setup remain unresolved.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (2)

docs/environment-setup/manual-dev-machine-setup.md:7

  • This update advertises Python 3.11–3.14 but leaves the Linux setup instructions' pip install -U docker-compose step unchanged. Legacy docker-compose 1.x requires docker<6, which conflicts with this PR's new docker>=7.1.0 requirement; following the documented steps leaves either the tool or Compose with an unsatisfied Docker SDK dependency. Replace the legacy package step with Docker Compose v2/plugin instructions or remove it.
1. Install **Python 3.11-3.14** and **pip**

setup.py:51

  • This package now excludes Python 3.9, but the supported Windows Docker build still passes PYTHON3=3.9.12 from build-docker.sh into docker/tool/windows/Dockerfile.base and installs this wheel with that interpreter. Any Windows (or default both-platform) Docker build will therefore fail at the wheel install with Requires-Python; update that image/build argument to Python 3.11+ or remove the Windows build path.
    python_requires='>=3.11, <3.15',
  • Files reviewed: 17/18 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment thread docker/tool/linux/Dockerfile.base
@DrewH-ms
DrewH-ms requested a lite review from Copilot September 11, 2026 23:00
@DrewH-ms

Copy link
Copy Markdown
Collaborator Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 2 pipeline(s).

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Address the unresolved critical token exposure, stale Python version source, and outdated virtual-environment example.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (2)

docker/tool/build-docker.sh:30

  • The active Windows build now passes Python 3.11.9, but docker/tool/deps.txt still declares PYTHON3_VERSION=3.9.12. Because this script still contains the TODO to read its version values from that file, the repository now has contradictory Python inputs for the Windows image; update that manifest or remove the unused version source before merging.
PYTHON3="3.11.9"

docs/environment-setup/manual-dev-machine-setup.md:7

  • The Python 3.11–3.14 claim is not carried through the virtual-environment documentation: docs/environment-setup/python-virtual-environment-setup.md:15 still tells contributors to create /usr/bin/python2.7 py27. Following the setup docs can therefore produce an interpreter rejected by the new python_requires; update that example in this change.
1. Install **Python 3.11-3.14** and **pip**
  • Files reviewed: 19/20 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment thread vsts_ci/linux/continuous-build-linux.yml Outdated
@DrewH-ms

Copy link
Copy Markdown
Collaborator Author

/azp run

@DrewH-ms
DrewH-ms requested a lite review from Copilot September 11, 2026 23:30
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 2 pipeline(s).

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The Windows Docker path has an incompatible Compose v1 dependency, and the manual setup documentation needs an explicit supported Python version.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (1)

docs/environment-setup/manual-dev-machine-setup.md:9

  • This Linux command installs only the distribution's default python3/pip; on Ubuntu 22.04 that is Python 3.10, outside the newly declared >=3.11,<3.15 range. Following these instructions can therefore make the subsequent pip install -U iotedgedev fail with Requires-Python; document an explicit supported interpreter and matching pip instead.
    - Linux: `sudo apt install python3-pip`
  • Files reviewed: 21/22 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment thread setup.py

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Address the critical Docker privilege issue and both moderate CI/runtime issues before approval.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (2)

docker/tool/linux/entrypoint.sh:21

  • printf '%q' can emit Bash-only $'...' quoting for arguments containing newlines or control characters, while sg re-parses this string with the account shell created by useradd (normally /bin/sh). As a result, commands such as docker run ... bash -c '<multiline script>' fail whenever the socket group switch is needed; preserve the original argv without serializing it through a shell, or invoke a compatible shell.
        exec sg "$group" -c "$(printf '%q ' "$@")"

vsts_ci/linux/continuous-build-linux.yml:69

  • This removes the explicit AZURE_CONFIG_DIR forwarding that the previous command supplied. Since tox is now launched through sudo, it can use root's Azure CLI cache instead of the authenticated cache created by AzureCLI@2, causing the login-mode IoT Hub tests to fail; preserve the variable explicitly when invoking tox.
            sudo -E `which tox` -e "$(TOXENV)"
  • Files reviewed: 22/23 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment thread docker/tool/linux/Dockerfile.base
@DrewH-ms

Copy link
Copy Markdown
Collaborator Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 2 pipeline(s).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants