Skip to content

release: 26.10.0a4 - #16130

Merged
fregataa merged 2 commits into
mainfrom
release/26.10.0a4
Oct 28, 2026
Merged

fregataa merged 2 commits into
mainfrom
release/26.10.0a4

Conversation

@HyeockJinKim

@HyeockJinKim HyeockJinKim commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

Release 26.10.0a4

  • Base branch: main
  • Tested commit: 60f7346f0d2fe9155f9263edfcca76308762db4a
  • WebUI bundle: unchanged (26.10.0-alpha.0)

The release commit only changes VERSION, the changelog, generated docs/configs and the bundled bssh binaries; the code is identical to the tested commit.

Live Test

Local stack (existing install; manager migrations upgraded to 8bb02f0415eb, supergraph regenerated). Services: mgr, agent, storage, web, proxy-coordinator, proxy-worker running.

# Item Result Evidence
0 Profiles ✅ PASS local-admin and local-user on http://127.0.0.1:8090, logged in
1 Service health ✅ PASS mgr 10/10, agent and storage all checks passed
2 DB schema ✅ PASS manager current = head 8bb02f0415eb, appproxy current = head
3 Superadmin ✅ PASS admin domain search succeeded
4 Basic read ✅ PASS domain get default
5 Agent ✅ PASS 1 agent, ALIVE, schedulable
6 Images ✅ PASS 155 images
7 Session ✅ PASS local-user, stable/python:3.13-ubuntu24.04-arm64 → RUNNING → TERMINATED
7-1 Deployment ✅ PASS local-user, create with 0 replicas → my deployment search → delete (STOPPED)
8 VFolder ✅ PASS local-user, create in project → project-search → delete → purge (delete-complete)
9 Regular user ✅ PASS my session search
10 Permission block ✅ PASS local-user admin domain search → 403
11 Error logs ✅ PASS Only OTLP export failures (optional otel-collector not running)

Overall: PASS

Changelog

Breaking Changes

  • Remove the endpoints field from the ServiceCatalog GraphQL node and the v2 ServiceCatalogNode, and drop the endpoints relationship from ServiceCatalogRow (#14881)
  • Remove the etcd-backed /manager/announcement API, its SDK methods and admin announcement CLI commands, and the client-side announcement_handler option, as WebUI now keeps the announcement in the domain app config (#15918)
  • Answer GET /v2/resource-allocation/keypair/my for the caller's default keypair instead of the keypair that signed the request (#16110)

Features

  • Publish the lablup/backend.ai-* service images to Docker Hub with a new multi-arch build workflow authenticating via OIDC (no stored registry credentials) (#13700)
  • Add a DOCKER install mode to the installer that deploys the published lablup/backend.ai-* service images with docker compose, including full bootstrap of the database schema, fixtures, etcd configuration, and kernel images (#13703)
  • Build the published service images from a shared backend.ai-base image installing every backend.ai package once (defined in docker-bake.hcl), and start them via the backend.ai <alias> start-server CLI entry points (#13709)
  • Make the DOCKER install mode emit a conventional compose deployment: per-file /etc/backend.ai config mounts with image default commands, host networking only for the manager and agent, bridge networking with published ports for the other services, the halfstack merged into the single generated compose file (one file, one project), daemon-visible state under fixed system paths (/var/lib/backend.ai, /tmp/backend.ai, /vfroot/local) so no running service mounts the install directory, and the storage-proxy running as the installing user (#13712)
  • Add the adminSessionLifecycleHandlers GraphQL query listing each session lifecycle handler's name, a valid key for a resource group's default session handler_options.by_handler, and its description (#15408)
  • Decide which values the manager injects into a session's environment from the system_session_environ rows at the public scope and the session's domain and project (#15606)
  • Add the system_session_environ table deciding which BACKENDAI_* environment variables sessions receive per public, domain and project scope (#15617)
  • Add the GraphQL schema for storage volumes: get, admin search, update, delete, restore and purge, with the storage volume service attachments (#15702)
  • Change the placement direction and enforcement of a session group through REST v2 and the GraphQL updateSessionGroup mutation (#15747)
  • Permanently remove a session group through REST v2 and the GraphQL purgeSessionGroup mutation, keeping its member sessions (#15751)
  • Add the AppProxy coordinator /v2/ingress API that pushes ingresses to workers with the api delivery mode (#15833)
  • Allow running the agent socket-relay container as a non-root user with no-new-privileges, all capabilities dropped, a read-only root filesystem, and pids, memory, and CPU limits through [agent.socket-relay]; each is off by default (#15849)
  • Add the volume status check registry and the probe check runner that runs each check on its own interval, and hold them in the storage proxy and agent. (#15863)
  • Report when a TUS upload session expires through the Upload-Expires header and advertise the TUS Expiration extension from the storage proxy (#15889)
  • Add max_header_field_size to the App Proxy worker and coordinator configs so operators can raise the request header size limit (#15910)
  • Add the AppProxy coordinator client that pushes ingresses to workers with the api delivery mode (#15915)
  • Add the kernel_groups, kernel_group_resource_slots and kernel_group_environ tables and a nullable kernels.kernel_group_id foreign key, so a session's kernel groups and their execution spec get their own rows. (#15919)
  • Add a service operation that makes one of a user's dotfiles the default of its path, as groundwork for the user dotfile v2 API (#15936)
  • Seed the system_session_environ bindings on a fresh install through mgr seed apply; public, domain and project members can read them and domain and project admins can manage them (#15945)
  • Add GET /v2/session-lifecycle-handlers, the SDK V2SessionLifecycleHandlerClient.admin_list(), and ./bai admin session-lifecycle-handler list to list the session lifecycle handlers (#15947)
  • Identify storage volumes by a system-unique name in the storage proxy volume pool and volume status check, and enforce unique storage volume names (#15967)
  • Record when each bootstrap script was created, and build bootstrap script data through its searchable fields (#15987)
  • Add compare_and_replace_prefixes to AbstractKVStore, which replaces etcd subtrees in one transaction only while every guard key holds its given value (#15988)
  • Add the v2 REST, GraphQL, SDK and CLI API that returns how many regular and private sessions the current user is running (#15993)
  • Store the kernel groups of a session with their resource requests and environ when it is enqueued (#15996)
  • Create a session group in a project through the v2 API, owned by the caller (#15997)
  • Keep every update of a bootstrap script as a numbered revision with its creator, and soft-delete bootstrap scripts (#16006)
  • Add the vfolder auto-mount entity that records which vfolders are mounted into every session of a user or project scope (#16009)
  • Show StructuredLogger fields after the message in console logs (#16014)
  • Add a superadmin search of the system session environ rows across every scope, with filtering, ordering and pagination (#16017)
  • Add v2 GraphQL and REST APIs to create, update, delete and search the vfolder auto-mounts of a project (#16020)
  • Resend Hive gateway requests that could not connect to a manager once to another manager, and end subscriptions with an error when their manager connection drops (#16031)
  • Make the webserver resend a request that failed before connecting to another healthy manager or Hive gateway endpoint, and exclude an endpoint that refuses connections right away (#16033)
  • Add the [proxy_coordinator.content_encoding] rejected_encodings App Proxy coordinator option, which makes workers and Traefik answer 415 to requests whose Content-Encoding contains a listed coding (for example ["br"]) instead of forwarding them to session apps. (#16041)
  • Let a domain's or a project's admins change the key of a system session environ row, soft-delete it and restore it (#16055)
  • Add the v2 REST, GraphQL, SDK and CLI API that returns the current user's resource usage summed across all of their keypairs, and deprecate the keypair-only query (#16056)
  • Add an id primary key to kernel_group_environ, keeping (kernel_group_id, key) unique, so environ rows can be paged by cursor. (#16058)
  • Replace the enabled flag of system_session_environ with a soft delete; disabled bindings become soft-deleted on upgrade, and a deleted binding no longer hides the same key at a broader scope (#16060)
  • Add a list of the value sources a system session environ row can carry, open to every logged-in user (#16067)
  • Add [agent.socket-relay.health-check], enabled by default, which gives the agent socket-relay container a healthcheck that reports it unhealthy when its agent socket file is missing, with configurable interval, timeout, and retries (#16075)
  • Make the App Proxy Traefik Go plugin answer 415 to requests whose Content-Encoding contains a coding listed in its contentencoding.rejectedencodings config instead of forwarding them to session apps. (#16078)
  • Add a user scope to the vfolder auto-mount v2 API, so a user can auto-mount a vfolder they own into every session they create (#16102)
  • Expose replicaCount on the deployment replica state, and make the desiredReplicas filter and order fall back to it until autoscaling sets a desired count (#16107)
  • Add mountPermission to the GraphQL VFolder node, reporting the level the current user mounts each folder at. (#16117)
  • Let a domain's or a project's admins create a system session environ row at their scope and a superadmin at the public scope, and purge a row for good (#16119)
  • Refuse adding a bootstrap script revision to a soft-deleted script, and let an ownerless field create spec declare preconditions (#16123)

Improvements

  • Resolve a kernel group's execution spec and vfolder mounts on the group itself before it expands into kernels, so a stored group and its kernels no longer disagree. (#15857)
  • Return the created dotfile from dotfile creation and let a dotfile search cover several users, as groundwork for the user dotfile v2 API (#15925)
  • Add a search for the SSH keypairs a user owns, scoped to that user and filtered through searchable-field conditions (#15964)
  • Retire the pickle-based kernel registry from the agent; the docker agent recovers running kernels from the per-kernel scratch records alone (#16007)
  • Build the Traefik router, service and middleware configs of an AppProxy ingress through TraefikRoutingConfigFactory (#16018)
  • Start the AppProxy coordinator server through CoordinatorDependencyComposer, which now also composes service discovery and the worker client pool (#16045)
  • Add the exception and the operation name to the retryable call failure log of the resilience retry policy (#16093)
  • Read project, domain and resource group usage and user concurrency through the resource allocation ops, taking known slot types from the database (#16104)
  • Delete the user, keypair row and storage host permission code nothing uses (#16120)

Fixes

  • Fix the dev helper script silently leaving an orphaned agent on restart when agent.pid is missing, which made the newly started agent die on a port conflict (#14958)
  • Insert every row of a fixture dataset with all of its own attributes, so rows that carry extra columns are no longer silently written with only the first row's columns (#15087)
  • Register CLI command groups before the subcommands that extend them, so a subcommand shipped in a different distribution than its group no longer crashes the CLI. (#15304)
  • Apply kernel-creation-concurrency per agent instead of per request, so concurrent sessions no longer multiply the number of containers an agent creates at once. (#15307)
  • Treat an unrecognized kernel lifecycle event reason as unknown instead of losing the kernel's terminated event. (#15308)
  • Bound every wait on a kernel runner's reply, so a runner slow to start no longer uses up the kernel creation retries in one attempt and the keepalive ping keeps running. (#15309)
  • Clean up a kernel's scratch directory and network even when its container is already gone. (#15311)
  • Fix alembic upgrade to 26.9 failing on the session group owner FK when a routed deployment has neither its owner nor its creator left (#15386)
  • Fix mgr ops list --concern dotfile rejecting the area the dotfile operations are wired under (#15899)
  • Fix the kernel entrypoint on Alpine images without the shadow package failing to add the work user to the shadow and grpread groups (#15903)
  • Fix Alpine-based images failing to start with UnsupportedBaseDistroError by matching the musllinux_* kernel runner volume of krunner-alpine 5.x and installing krunner-alpine with the agent (#15904)
  • Fix every mgr image command failing with a KernelRow mapper error (#15938)
  • Fix mgr image commands failing with GlobalEntityNotLoaded when writing images of a global container registry (#15942)
  • Fix a registry rescan failing a whole project when a customized image's owner label names a user that does not exist in this installation (#15958)
  • Fix the password-change requirement staying set after a user changes their own password through updateUserV2 (#15979)
  • Fix sessions stuck in CREATING when several kernels are created concurrently on one agent, by saving only the started kernel's recovery data before reporting it started (#16025)
  • Refuse restoring a user that was not deleted instead of activating it. (#16028)
  • Refuse password login of users whose status is purging or purge-error, as API request authentication already does (#16029)
  • Fix etcd reads failing for as long as one etcd member stops responding by dropping that member through HTTP/2 keep-alive and resending timed-out reads (#16032)
  • Fix vfolder REST and GraphQL responses failing when a vfolder is in the creating status by adding creating to the response status enums (#16088)
  • Show the unregistered slot type name and how to register it when an agent heartbeat reports a slot type missing from the resource slot type catalog (#16094)

Documentation Updates

  • Document the published lablup/backend.ai-* Docker images, their tagging scheme, a reference compose setup, and the required container privileges in docker/README.md (#13702)
  • Add BEP-1079, the VXLAN overlay cluster network: the manager allocates each multi-node session a subnet and a VNI, and each agent builds its node's half of the overlay through a runtime-neutral seam, with an optional privileged helper so the agent itself needs no CAP_NET_ADMIN. (#14906)
  • Require nullable return types on every GQL resolver field, including computed fields on types. (#16013)
  • Extend the NEXT_RELEASE_VERSION rule beyond GQL so new config fields and DTO fields also record their version through it (#16039)
  • Update the installation docs and install-dev.sh to match the current code and installation process, including keeping the agent port range out of the ephemeral port range (#16044)
  • Document that adapter scenarios leave out the logic every entity shares, such as filter AND / OR / NOT combinations, pagination and the default order direction. (#16047)

External Dependency Updates

  • Bump certifi, pip, uv and pex to drop trust in the compromised emSign root certificate, and bump aiohttp and OpenTelemetry for security fixes (#16021)
  • Bump CPython to 3.13.18 and setuptools to 84.2.1 (#16048)

Miscellaneous

  • Build and push the service Docker images automatically on every release tag as part of the CI release chain (#13701)
  • Build and push the seven service images in a single docker buildx bake run wiring the shared base image in as a bake target, with a drift check between docker-bake.hcl and the list-dockerfiles.sh publishing registry (#13710)
  • Give the cuda.unified resource slot type fixture row a uuid so the component tests can seed the slot types again (#15921)
  • Merge the two alembic heads left by two revisions written against the same parent, so alembic upgrade head resolves again. (#15995)
  • Fix the App Proxy worker decompressing request bodies it forwards, which truncated gzip/deflate bodies and rejected br bodies with 400. (#16037)
  • Bump the python-build-standalone release used for scie builds to 20261026, which ships CPython 3.13.18 (#16071)
  • Fix StructuredLogger lines inside deployment actions failing the request by normalizing str subclasses such as EntityType before the log relay packs them (#16097)

Test Updates

  • Expect the span's own trace flags in the storage proxy client traceparent test, which the OpenTelemetry bump broke by also setting the random trace ID flag (#16131)

🤖 Generated with Claude Code


📚 Documentation preview 📚: https://sorna--16130.org.readthedocs.build/en/16130/


📚 Documentation preview 📚: https://sorna-ko--16130.org.readthedocs.build/ko/16130/

@HyeockJinKim
HyeockJinKim requested a review from a team as a code owner October 28, 2026 09:01
Copilot AI balanced review requested due to automatic review settings October 28, 2026 09:01
@github-actions github-actions Bot added size:L 100~500 LoC area:docs Documentations labels Oct 28, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

jopemachine
jopemachine previously approved these changes Oct 28, 2026
@HyeockJinKim
HyeockJinKim enabled auto-merge (squash) October 28, 2026 09:07
seedspirit
seedspirit previously approved these changes Oct 28, 2026
@fregataa
fregataa disabled auto-merge October 28, 2026 09:44
@fregataa
fregataa merged commit dd18908 into main Oct 28, 2026
56 of 58 checks passed
@fregataa
fregataa deleted the release/26.10.0a4 branch October 28, 2026 09:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:docs Documentations size:L 100~500 LoC

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants