Skip to content

stack-files.md

Latest commit

 

History

History
388 lines (317 loc) · 19.1 KB

stack-files.md

File metadata and controls

388 lines (317 loc) · 19.1 KB
Raw

stack.yml

The stack.yml file defines the structure and configuration of a deployment stack. It specifies the containers, pods, and any pre/post-start commands required for the deployment.

How these files relate to each other — which one is authoritative for what, and how stack validate checks that they agree — is described in stack-integrity.md.

Example

# The name of the stack.
name: gitea
# A brief description of the stack (optional).
description: "Gitea SCM and Actions"
# A list of containers to be used in the stack.
containers:
    # The name of the container in the form `<organization>/<name>`.  The pod's composefile.yml will need to use the
    # same name for the image with the tag `stack`.  For example: `image: bozemanpass/act-runner:stack`
  - name: bozemanpass/act-runner
    # An (optional) reference to the container's repository.  The format is: [hostname/]organization/repo[@tag_or_branch]
    # The hostname is optional.  When omitted, github.com is assumed.  The tag is also optional.  If omitted,
    # the main repo branch is used.  If `ref` is omitted entirely, the current repo is assumed.
    ref: bozemanpass/gitea-containers
    # The relative path in the repo to the container build info.  This directory must contain one (or more) of:
    #   - container.yml descriptor file (more info below)
    #   - build.sh build script
    #        The result of execution should be a local image tagged `<name>:stack`.  The exact tag is available
    #        in the script build environment under ${STACK_DEFAULT_CONTAINER_IMAGE_TAG}.
    #   - Dockerfile
    #        The container will be built using the Dockerfile in this directory similar to:
    #             docker build -t ${STACK_DEFAULT_CONTAINER_IMAGE_TAG} .
    # This directory is also what gets built, unless `content-root` says otherwise.
    path: ./act-runner
  - name: bozemanpass/gitea
    ref: bozemanpass/gitea-containers
    path: ./gitea
    # An (optional) container wrapper scheme (see docs/wrappers.md).  When specified, the container image is built by
    # wrapping the contents of the repo at `ref` using the named wrapper, e.g. `static-content` serves the repo
    # contents with nginx, `nextjs` builds and serves a Next.js webapp, and `node-service` builds and runs a
    # long-running node service.  The repo does not need to contain any container build files.  Available
    # wrappers can be listed with `stack webapp wrappers`.
  - name: bozemanpass/my-static-site
    ref: myorg/my-static-site
    wrapper: static-content
    # An (optional) reference to the wrapper's repository, same format as `ref`.  When omitted,
    # the wrapper is resolved from already-fetched repos (or the default wrapper repos).
    wrapper-ref: bozemanpass/stack-wrapper-static-content@main
    # An (optional) subdirectory of the source repo holding what is actually built: the content
    # wrapped by `wrapper`, or the build context of the default build.  Not wrapper-specific.
    # When omitted it defaults to `path`.  See "path vs content-root" below.
    content-root: site
# Pods are groups of containers that are deployed together.  Each pod corresponds to one composefile.yml.
pods:
    # The name of the pod.
  - name: gitea
    # The relative path in this repo to the directory containing the pod composefile.yml and other files.
    path: ./gitea
    # An (optional) command to run just _before_ the pod starts. The command is executed on the host, and the location
    # is relative to the `path` specified above.  The deployment directory will be set in the environment under
    # ${STACK_DEPLOYMENT_DIR}, allowing a script to execute commands _inside_ the service containers with: 
    #     stack manage --dir ${STACK_DEPLOYMENT_DIR} exec <service> <command>
    pre_start_command: "run-this-first.sh"
    # Similar to pre_start_command, but executed _after_ the pod starts.
    post_start_command: "initialize-gitea.sh"
  - name: act-runner
    path: ./act-runner
    pre_start_command: "pre_start.sh"
    post_start_command: "post_start.sh"
# (Optional) environment variables that are secret: their values are injected at deploy
# time -- generated, or pulled from a reference recorded in the spec -- and never appear
# in the stack files or the deployment artifacts.  See secrets.md.
secrets:
  # Delivered to every container of the deployment; generated by default.
  GITEA_ADMIN_PASSWORD:
  # `external: true` means a generated value would be useless (its counterpart lives
  # outside the deployment), so a reference must be supplied at init time.
  SMTP_PASSWORD:
    external: true

path vs content-root

A container entry answers two questions that are easy to confuse:

  • Where is the recipe? — path, the directory holding the container build info (container.yml, build.sh, or a Dockerfile). A pod's path means the same kind of thing: where its composefile lives.
  • What gets built? — the source repo, narrowed by content-root to a subdirectory of it. This is the docker build context: the content a wrapper wraps, or what the default build builds.
path content-root
Answers where the recipe lives what gets built
Relative to the repo at the entry's ref the source repo (example 4 -- not always the same repo)
Names container.yml / build.sh / Dockerfile the docker build context
When omitted the repo root path, except in example 4
In the build ${STACK_BUILD_DIR} ${STACK_CONTENT_ROOT_DIR}

They coincide for a repository that carries a Dockerfile beside the code it builds, which is why path alone was enough for a long time. They come apart whenever the recipe lives somewhere other than the source. The four shapes:

1. Recipe and source together

The common case: build info sits in the same directory as what it builds.

gitea-containers/           <- ref
└── act-runner/             <- path, and the build context
    ├── build.sh
    └── Dockerfile

containers:
  - name: bozemanpass/act-runner
    ref: bozemanpass/gitea-containers
    path: ./act-runner

content-root is not needed: it defaults to path, so ./act-runner is what gets built — the build context for the Dockerfile, and ${STACK_CONTENT_ROOT_DIR} for the build.sh.

2. Wrapper, whole repo is the content

The repo contains no build files at all — the recipe comes from the wrapper repo (see wrappers.md), so there is nothing for path to point at.

my-static-site/             <- ref, and the build context
├── index.html
└── css/

containers:
  - name: bozemanpass/my-static-site
    ref: myorg/my-static-site
    wrapper: static-content

3. Wrapper, content in a subdirectory

The same, but the repo holds more than the site: the content is one directory within it.

my-static-site/             <- ref
├── README.md
├── stack-files/            <- the stack.yml itself lives here
└── site/                   <- content-root, and the build context
    ├── index.html
    └── pages/

containers:
  - name: bozemanpass/my-static-site
    ref: myorg/my-static-site
    wrapper: static-content
    content-root: site

Only ./site is sent to the container build, so README.md and stack-files/ are not baked into the image.

4. Recipe in one repo, source in another

A container.yml whose ref names a different repo builds source the entry's own repo doesn't contain. Here the two fields refer to two different repositories:

gitea-containers/           <- the entry's ref
└── act-runner/             <- path (recipe only)
    ├── container.yml       <- ref: gitea.com/gitea/act_runner
    └── build.sh

act_runner/                 <- the source repo, named by container.yml
└── src/                    <- content-root, and the build context

# stack.yml
containers:
  - name: bozemanpass/act-runner
    ref: bozemanpass/gitea-containers
    path: ./act-runner
# gitea-containers/act-runner/container.yml
container:
  name: bozemanpass/act-runner
  ref: gitea.com/gitea/act_runner
  content-root: src

This is the one case where content-root does not default to path: ./act-runner describes the layout of gitea-containers and says nothing about the layout of act_runner, so leaving content-root out builds the source repo's root. Set it explicitly to build from a subdirectory of the source.

Note: content-root may be given in either file. In stack.yml it applies to the entry; in container.yml it travels with the repo that declares it, and takes precedence.

In the build environment

A build.sh is free to build whatever it likes, so content-root does not constrain it — it is handed over instead, along with the other locations resolved for the container:

Variable Directory
STACK_BUILD_DIR the recipe: where build.sh was found (the default build's context)
STACK_CONTENT_ROOT_DIR the content root: the source repo narrowed by content-root
STACK_REPO_SOURCE_DIR the source repo, unnarrowed
STACK_REPO_CONTAINER_DIR the repo holding the container build info
STACK_REPO_STACK_DIR the repo holding the stack.yml

In example 1 STACK_BUILD_DIR and STACK_CONTENT_ROOT_DIR are the same directory. In example 4 every one of them is different.

container.yml

The container.yml file defines the build configuration for individual containers within a stack. It specifies the container's name, the repository reference, and the build script or command to be used for building the container image.

The build script path is relative to the container.yml file, not the target repo. In practice, this allows for build scripts to be located in a separate repo from the container's source code, which is very useful for building customized container images from repositories that are not under your control.

The following example from the bozemanpass/gitea-containers repo builds the bozemanpass/act-runner container directly from the gitea.com/gitea/act_runner repo, for example.

Example

container:
  # The name of the container in the form `<organization>/<name>`.  The pod's composefile.yml will need to use the
  # same name for the image with the tag `stack`.  For example: `image: bozemanpass/act-runner:stack`
  name: bozemanpass/act-runner
  # An optional reference to the container's repository.  The format is: [hostname/]organization/repo[@tag_or_branch]
  # The hostname is optional.  When omitted, github.com is assumed.  The tag is also optional.  If omitted,
  # the main repo branch is used.  If `ref` is omitted entirely, the current repo is assumed.
  ref: gitea.com/gitea/act_runner
  # Optional path to the container build script or command.  This path is relative to the `container.yml` file.
  # If no build script is provided, the default build command will be used.
  build: ./build.sh
  # An optional subdirectory of the repo at `ref` to build, rather than the whole repo.
  content-root: src

Example: a self-describing wrapped repo

A repository can also declare that it is built by wrapping, rather than being named as a wrapper in every stack that uses it (see wrappers.md). A container.yml at the root of a static site whose pages live in ./site:

container:
  name: bozemanpass/my-static-site
  wrapper: static-content
  # Optional: pin the wrapper repo, same format as `ref`.
  wrapper-ref: bozemanpass/stack-wrapper-static-content@main
  # Serve ./site, not the whole repo.
  content-root: site

A stack then needs only the name and the repo:

containers:
  - name: bozemanpass/my-static-site
    ref: myorg/my-static-site

Image identity and lock files

A container image is identified by the commit hash of its recipe repo — the repository hosting the container's build declaration: the repo carrying its container.yml when there is one, otherwise the repo carrying the stack.yml that declares it. When the recipe and the source it builds are the same repository (the common, colocated case) that is simply the source repo's hash. When they are different repositories, lock files committed in the recipe repo pin the other build inputs — the source ("payload") repo and the wrapper — so that a recipe commit fully determines image content. An image built from unpinned or deviating inputs, or from a recipe checkout with uncommitted changes (including a not-yet-committed lock file), gets a synthetic stackdev- tag instead, which is never published or matched remotely: committing the lock file is what stabilizes the image identity.

Lock files exist to make builds reproducible. To move a pin to a newer version, delete the lock file (or the relevant entry) and rebuild to regenerate it.

Building from a local checkout

When a stack is loaded from a filesystem path (rather than located by name under the repo base directory), containers whose source is the stack's own repository build directly from that checkout — the repo is not cloned again, and the image identity and the built content always come from the same tree. This is what a CI job wants: the pipeline's checkout is the build source, with no second clone (which for a private repository would need its own credentials). It also means local modifications in a developer checkout are what gets built — with tracked changes producing a stackdev- tagged image, since the checkout no longer matches any commit. Repositories referenced by ref that are not the stack's own are always cloned beneath the repo base directory, at their pinned versions.

container.lock

The container.lock file lives beside a container.yml whose ref names a different source repository, and pins that source: hash records the source repo commit (and wrapper records the wrapper repo, when one is used). If not already present, the file is automatically generated when the container is built, and when the source repository is cloned by stack the pinned commit is checked out. Commit it to make the build repeatable.

Note: Even when container.lock is present, any local code changes will be included when building the container, since the hash is used only when the repository is cloned or pulled.

stack.lock

The stack.lock file is generated next to stack.yml when a container declared there is built from inputs the stack's repo does not itself contain: a source repository other than the stack's own, and/or a wrapper (see wrappers.md). It has a containers section pinning each such container's source repo commit, and a wrappers section pinning each wrapper repo commit (which also identifies the exact prebuilt base image to pull). Commit it to make the stack's builds repeatable and its image tags stable.

Note: stack.lock supersedes the earlier wrapper.lock; an existing wrapper.lock is still read (as the wrappers section) when no stack.lock is present, and can be deleted once a stack.lock has been generated.

The stack.lock also carries an images section pinning each external image the stack's pod files name (anything not tagged :stack, e.g. postgres:14) to its manifest digest. stack prepare records the pins and stack deploy applies them to the deployment's copy of the pod files; an image annotated # @stack unpinned on its image: line is left floating. See stack-integrity.md.

composefile.yml

The composefile.yml file defines the structure and configuration of a pod. It specifies the containers, volumes, environment variables, and any other settings required for the pod's deployment.

It is compatible in syntax with docker-compose.yml.

Example

services:
  runner:
    image: bozemanpass/act-runner:stack
    restart: always
    privileged: true
    environment:
      - CONFIG_FILE=/config/act-runner-config.yml
      - GITEA_INSTANCE_URL=http://gitea:3000
    volumes:
      - act-runner-data:/data
      - act-runner-config:/config:ro
    ports:
      - 8088

volumes:
  act-runner-data:
  act-runner-config:

Environment Variable Precedence

A service can pick up the same variable from more than one place. Later sources in this list win:

  1. the deployment's config.env (values supplied at deployment time)
  2. each file listed in the service's env_file:, in the order listed
  3. the service's inline environment: block

This is the order Docker Compose uses, and it is applied identically for Kubernetes deployments, so a pod file hands its containers the same values whichever target it is deployed to. A pod file that wants a deployment-time value to reach the container can forward it explicitly, e.g. environment: {SOME_VAR: "${SOME_VAR}"}. In the sequence form, a bare - SOME_VAR does the same thing: it names a variable to take from the sources listed above it, and passes nothing at all when none of them set it.

The consequence worth knowing is that an inline default beats a value the deployer supplied with --config: the composefile wins, and the deployment is wrong rather than failed. So stack deploy reports it, naming the service and the key and both values, whenever a key in the deployment's config.env is shadowed by an inline literal that differs from it. A forwarding entry (above) is the fix and is not reported, nor is an inline literal that agrees with the config value, nor a sequence entry with no value of its own (- SOME_VAR).

Service Hostnames

Every service in a deployment is reachable from every other service by its service name as it appears in the composefile.yml, regardless of which pod it belongs to. In the example above, the runner service reaches the gitea service (located in another pod in the same deployment) at the hostname gitea.

This works the same way whether the deployment target is Docker Compose or Kubernetes: on Kubernetes each deployment gets its own namespace, so the unqualified service name resolves within it.

Declaring ports: is not what makes a service addressable. A service with no ports: at all -- a database whose port nothing outside the deployment should reach, say -- still answers to its own name on both targets; on Kubernetes it gets a headless Service (no cluster IP, no ports) whose name resolves to the pod. So there is no need to publish a port merely to obtain a hostname, which on the Compose target would also publish it on the host under --map-ports-to-host.