Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
__pycache__/
build/
/build/
dist/
*.egg-info/
.pytest_cache/
Expand Down
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,13 @@

### Added

- The build API client: `ukc.builds` creates, lists, cancels and deletes
builds on the remote build service, follows their progress with `logs()`
(reconnecting and skipping what it has seen) and `plain_logs()`, and takes
its sources from `git_source`, `url_source`, `kraftfile_from` and
`image_output`. The service's refusals carry their code on
`err.errors[0].code`. The plumbing is `ukc.api.build`, generated from
`build.yaml`; `build_url` (or `UKC_BUILD_SERVER`) names the service.
- The sandbox plugin client: `sandbox()` on an instance handle runs commands,
follows their output, feeds standard input and moves files, and `plugin(name)`
on one addresses any other plugin. The plumbing is `ukc.api.plugins.sandbox`.
Expand Down
23 changes: 19 additions & 4 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,15 @@ CHANNEL ?= prod-staging
SPEC_BASE ?= https://raw.githubusercontent.com/unikraft-cloud/openapi/refs/heads/$(CHANNEL)
PLATFORM_SPEC ?= $(SPEC_BASE)/platform.json
CONTROLPLANE_SPEC ?= $(SPEC_BASE)/controlplane.json
BUILD_SPEC ?= $(SPEC_BASE)/build.yaml

# Both specs also live in the proto repository, which is where the changes land
# first. Generate against a local checkout of it with:
# The platform and control-plane specs also live in the proto repository, which
# is where the changes land first, and the build spec in the build service's.
# Generate against a local checkout with:
# make generate \
# PLATFORM_SPEC=../proto/gen/openapi/platform/openapi.yaml \
# CONTROLPLANE_SPEC=../proto/gen/openapi/controlplane/openapi.yaml
# CONTROLPLANE_SPEC=../proto/gen/openapi/controlplane/openapi.yaml \
# BUILD_SPEC=../blacksmith/openapi.yaml

# The openapi-gen code generator, pinned so a regeneration is reproducible.
# Raise OPENAPI_GEN_VERSION to take a newer one.
Expand All @@ -29,7 +32,7 @@ all: generate test
# Each per-spec target clears its generated modules first, so a tag the spec
# drops leaves no stale client behind. The hand-written `__init__.py` files stay.
.PHONY: generate
generate: generate-platform generate-controlplane ## Regenerate the platform and control-plane plumbing from the OpenAPI specs.
generate: generate-platform generate-controlplane generate-build ## Regenerate the platform, control-plane and build plumbing from the OpenAPI specs.
$(MAKE) fmt

# Namespaced schema names (`Instances.Instance`) are not Python identifiers, so
Expand All @@ -52,6 +55,18 @@ generate-controlplane: ## Regenerate the control-plane API plumbing.
-t $(TEMPLATES) \
--namespace-flatten strip

# The progress stream of a build is newline-delimited JSON or plain text,
# which the templates cannot return, so its operation is left to the
# hand-written client.
.PHONY: generate-build
generate-build: ## Regenerate the build API plumbing.
rm -f $(OUTPUT)/build/*_gen.py
$(OPENAPI_GEN) \
-i $(BUILD_SPEC) \
-o $(OUTPUT)/build \
-t $(TEMPLATES) \
-v skip=GetBuildLogs

# A plugin's client is generated by plugin-sdk from the plugin's own
# specification, and installed as a package of its own.

Expand Down
46 changes: 40 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@ ukc = UnikraftCloud(
| `metros` | The metros operations cover by default: `"all"`, one metro, or a list. Creating a resource needs exactly one, so name a metro somewhere when you create. |
| `base_url` | Explicit platform API base URL. It settles where requests go, so it overrides `metro` and `UKC_METRO` alike, and pins the client to that one endpoint. |
| `control_plane_url` | Override the control-plane API base URL. |
| `build_url` | Override the build API base URL, an `http(s)://` URL. Falls back to `UKC_BUILD_SERVER`. |
| `headers` | Extra headers sent with every request. |
| `user_agent` | Override the default User-Agent. |
| `http` | An `httpx.AsyncClient` to send through. Supplying one makes its lifetime yours. |
Expand Down Expand Up @@ -229,7 +230,8 @@ except UnikraftCloudError as err:
`RateLimitError` (429) and `ServerError` (5xx) are raised for the statuses they name, and
all subclass `UnikraftCloudError`. The API reports some failures inside an otherwise-200
envelope, per item; those carry the API's own code on `err.errors[n].code` and are raised
with the status that says the same thing.
with the status that says the same thing. The build service refuses with a word instead,
such as `not_found` or `too_many_builds`, on the same field.

Two failures are caught before anything is sent, and are not `UnikraftCloudError`: a
property the resource does not have is a `TypeError`, and a request missing a field the
Expand Down Expand Up @@ -286,7 +288,8 @@ except MetroFanoutError as err:
## Resources

`instances`, `templates`, `images`, `volumes`, `services`, `certificates` and `users`
hang off any scope — `ukc`, `ukc.metro("fra")` or `ukc.metros([...])`.
hang off any scope — `ukc`, `ukc.metro("fra")` or `ukc.metros([...])`. `builds` hangs
off `ukc` alone: the build service is global.

Creating one takes the properties the API describes as keyword arguments, and a property
it does not have is a `TypeError` rather than a field the server quietly ignores.
Expand Down Expand Up @@ -362,6 +365,33 @@ job = await fra.templates.get(name="worker-v3").clone(
)
```

### Builds

A build runs on the remote build service, in a builder of your own account, from
sources the builder fetches itself: a git repository or a tarball, never a directory on
this machine. A create takes the properties the API describes; `kraftfile_from` reads
the Kraftfile out of a source, and `image_output` pushes the image with your token.
`logs()` follows the progress as buildkit events, every event once, asking again and
skipping what it has seen when the stream is cut, and ends with the event that carries
the `result` or the `error`: a failed build is that last event, not an exception.
`plain_logs()` is the text `docker buildx build --progress=plain` prints, asked for
once. The service is named by `build_url` or `UKC_BUILD_SERVER`, an `http(s)://` URL.

```python
from unikraft_cloud import git_source, image_output, kraftfile_from

src = git_source("https://github.com/acme/app.git", "main", subdir="app")
build = await ukc.builds.create(
kraftfile=kraftfile_from(src),
contexts={"context": src},
output=[image_output("index.unikraft.io/acme/app:1.2.3")],
)
async for event in build.logs():
if event.error:
raise SystemExit(f"build failed: {event.error.message}")
build = await build.refresh()
```

## The sandbox plugin

A plugin is a helper loaded into an instance beside its workload, and reached through the
Expand Down Expand Up @@ -469,12 +499,16 @@ async with PlatformApi(config) as api:
- [`examples/sandbox.py`](examples/sandbox.py) — run commands and move files through the sandbox plugin
- [`examples/templates.py`](examples/templates.py) — look an image up, prepare a template, clone it
- [`examples/plumbing.py`](examples/plumbing.py) — the raw API on its own
- [`examples/detached.py`](examples/detached.py) — build a git repository on the build service and push the image
- [`examples/parallel.py`](examples/parallel.py) — several builds at once, followed side by side

## Development

The `api/platform` and `api/controlplane` packages are generated from the OpenAPI
specification by `openapi-gen`, pinned in the Makefile, using the templates in
[`templates/`](templates). The sandbox plugin's plumbing
The `api/platform`, `api/controlplane` and `api/build` packages are generated from
the OpenAPI specifications by `openapi-gen`, pinned in the Makefile, using the
templates in [`templates/`](templates). The progress stream of a build is the one
operation the templates skip (`-v skip=...`): it is newline-delimited JSON or plain
text, which `ukc.builds` reads by hand. The sandbox plugin's plumbing
is generated elsewhere, by
[plugin-sdk](https://github.com/unikraft-cloud/plugin-sdk) from the plugin's own
specification, and installed as the `unikraft-cloud-plugin-sandbox-api` package.
Expand All @@ -490,7 +524,7 @@ and a `Request`-named schema that a response also carries fails to parse when th
leaves a required field out.

```sh
make generate # regenerate the platform and control-plane plumbing from the specs
make generate # regenerate the platform, control-plane and build plumbing from the specs
make lint # ruff check + format --check
make typecheck # mypy
make test # pytest
Expand Down
97 changes: 97 additions & 0 deletions examples/detached.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
# SPDX-License-Identifier: BSD-3-Clause
# Copyright (c) 2026, Unikraft GmbH. All rights reserved.
#
# A detached build on the remote build service: the source, the Kraftfile and
# the output all go in the request, and the build runs entirely server side.
# Every input but the token and the image to push has a default, which builds
# the nginx example of github.com/unikraft-cloud/examples:
# UKC_TOKEN=... uv run examples/detached.py index.unikraft.io/<user>/nginx:latest
#
# Progress goes to stderr and the pushed ref@digest to stdout, so that
# IMAGE=$(uv run examples/detached.py ...) captures the image.

from __future__ import annotations

import argparse
import asyncio
import contextlib
import sys

from unikraft_cloud import (
Build,
UnikraftCloud,
UnikraftCloudError,
git_source,
image_output,
kraftfile_from,
)


async def main(args: argparse.Namespace) -> None:
async with UnikraftCloud(build_url=args.server) as ukc: # token from UKC_TOKEN
src = git_source(args.git, args.ref or None, subdir=args.subdir or None)
build = await ukc.builds.create(
kraftfile=kraftfile_from(src),
contexts={"context": src},
output=[image_output(args.image)],
)
print(f"build {build.id}", file=sys.stderr)

try:
build = await follow(build)
except asyncio.CancelledError:
# Interrupted: the build would go on without us, so cancel it.
await build.cancel()
raise

if build.result is None:
# Failed or cancelled: the error is the build's, not the API's.
print(f"build {build.id} {build.status}: {build.error}", file=sys.stderr)
raise SystemExit(1)
for img in build.result.images or []:
print(f"{img.ref}@{img.digest}")


async def follow(build: Build) -> Build:
"""Print the build's progress as the service renders it, until it has ended."""
while True:
async with contextlib.aclosing(build.plain_logs()) as lines:
async for text in lines:
sys.stderr.write(text)
sys.stderr.flush()
build = await build.refresh()
# The text ends when the build does, or when the service's side of the
# stream broke off first; then it says so, and asking again replays it.
if build.status not in ("waiting", "running"):
return build


if __name__ == "__main__":
parser = argparse.ArgumentParser(
description="Build a git repository on the remote build service and push it to <image>."
)
parser.add_argument("image", help="the image reference to push, including its tag")
parser.add_argument("--server", default=None, help="build service URL, or $UKC_BUILD_SERVER")
parser.add_argument(
"--git",
default="https://github.com/unikraft-cloud/examples.git",
help="git repository to build",
)
parser.add_argument(
"--ref",
default="main",
help="branch, tag or commit to build, or empty for the default branch",
)
parser.add_argument(
"--subdir",
default="nginx",
help="directory of the repository that holds the Kraftfile and is the build context, "
"or empty for the root",
)
try:
asyncio.run(main(parser.parse_args()))
except KeyboardInterrupt:
sys.exit(130)
except UnikraftCloudError as err:
print(f"API error ({err.status or '?'}): {err}", file=sys.stderr)
raise SystemExit(1) from err
113 changes: 113 additions & 0 deletions examples/parallel.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
# SPDX-License-Identifier: BSD-3-Clause
# Copyright (c) 2026, Unikraft GmbH. All rights reserved.
#
# Several builds on the remote build service at once, followed side by side.
# Each argument after the image namespace is a directory of
# github.com/unikraft-cloud/examples, pushed to <namespace>/<directory>:latest:
# UKC_TOKEN=... uv run examples/parallel.py index.unikraft.io/<user> nginx redis
#
# Progress goes to stderr with each line prefixed by its directory, and the
# pushed ref@digest of every build to stdout.

from __future__ import annotations

import argparse
import asyncio
import contextlib
import sys

from unikraft_cloud import (
UnikraftCloud,
UnikraftCloudError,
git_source,
image_output,
kraftfile_from,
)

# The ANSI colours of the prefixes, in docker compose's order.
COLOURS = ["36", "33", "32", "35", "34", "36;1", "33;1", "32;1", "35;1", "34;1"]


async def main(args: argparse.Namespace) -> None:
dirs = args.dirs or ["mariadb", "neo4j", "visual-studio-code-server"]
colour = sys.stderr.isatty()
width = max(len(d) for d in dirs)
async with UnikraftCloud(build_url=args.server) as ukc: # token from UKC_TOKEN
builds = []
for index, subdir in enumerate(dirs):
prefix = f"{subdir:<{width}} | "
if colour:
prefix = f"\x1b[{COLOURS[index % len(COLOURS)]}m{prefix}\x1b[0m"
builds.append(run(ukc, args, subdir, prefix))
# Every build runs to its end; the failures are reported together.
failures = [
f"{subdir}: {outcome}"
for subdir, outcome in zip(
dirs, await asyncio.gather(*builds, return_exceptions=True), strict=True
)
if outcome is not None
]
if failures:
print("parallel: " + "; ".join(failures), file=sys.stderr)
raise SystemExit(1)


async def run(ukc: UnikraftCloud, args: argparse.Namespace, subdir: str, prefix: str) -> str | None:
"""Run one build to its end, printing its progress behind `prefix`; None once it pushed."""
src = git_source(args.git, args.ref or None, subdir=subdir)
build = await ukc.builds.create(
kraftfile=kraftfile_from(src),
contexts={"context": src},
output=[image_output(f"{args.namespace}/{subdir}:latest")],
)
print(f"{prefix}build {build.id}", file=sys.stderr)

try:
while True:
async with contextlib.aclosing(build.plain_logs()) as lines:
tail = ""
async for text in lines:
*whole, tail = (tail + text).split("\n")
for line in whole:
print(f"{prefix}{line}", file=sys.stderr)
if tail:
print(f"{prefix}{tail}", file=sys.stderr)
build = await build.refresh()
if build.status not in ("waiting", "running"):
break
except asyncio.CancelledError:
await build.cancel()
raise

if build.result is None:
return f"build {build.id} {build.status}: {build.error}"
for img in build.result.images or []:
print(f"{img.ref}@{img.digest}")
return None


if __name__ == "__main__":
parser = argparse.ArgumentParser(
description="Build directories of a git repository at once on the remote build service "
"and push each to <namespace>/<directory>:latest."
)
parser.add_argument("namespace", help="where to push, e.g. index.unikraft.io/<user>")
parser.add_argument("dirs", nargs="*", help="directories of the repository to build")
parser.add_argument("--server", default=None, help="build service URL, or $UKC_BUILD_SERVER")
parser.add_argument(
"--git",
default="https://github.com/unikraft-cloud/examples.git",
help="git repository to build",
)
parser.add_argument(
"--ref",
default="main",
help="branch, tag or commit to build, or empty for the default branch",
)
try:
asyncio.run(main(parser.parse_args()))
except KeyboardInterrupt:
sys.exit(130)
except UnikraftCloudError as err:
print(f"API error ({err.status or '?'}): {err}", file=sys.stderr)
raise SystemExit(1) from err
Loading
Loading