-
Notifications
You must be signed in to change notification settings - Fork 12
docs(huntsman): Add a Kubernetes deployment guide for Spider's Helm chart. #447
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -1,5 +1,228 @@ | ||||||
| # Kubernetes deployment | ||||||
|
|
||||||
| :::{warning} | ||||||
| 🚧 This section is still under construction. | ||||||
| This guide describes how to deploy a Spider cluster on Kubernetes using the [Spider Helm | ||||||
| chart][helm-chart]. | ||||||
|
|
||||||
| --- | ||||||
|
|
||||||
| ## Requirements | ||||||
|
|
||||||
| * [`kubectl`][kubectl] >= 1.30 | ||||||
| * [Helm] >= 4.0 | ||||||
| * A Kubernetes cluster (see [Setting up a cluster](#setting-up-a-cluster) below) | ||||||
|
|
||||||
| --- | ||||||
|
|
||||||
| ## Setting up a cluster | ||||||
|
|
||||||
| If you already have a cluster, skip to [Installing the chart](#installing-the-chart). Otherwise, | ||||||
| [`kind`][kind] (Kubernetes in Docker) runs a cluster inside Docker containers, making it ideal for | ||||||
| local testing and development. | ||||||
|
|
||||||
| `kind` requires: | ||||||
|
|
||||||
| * [Docker], which requires: | ||||||
| * `containerd.io` >= 1.7.18 | ||||||
| * `docker-ce` >= 27.0.3 | ||||||
| * `docker-ce-cli` >= 27.0.3 | ||||||
| * [`kind`][kind] >= 0.23 | ||||||
|
|
||||||
| Create a `kind` cluster: | ||||||
|
|
||||||
| ```shell | ||||||
| kind create cluster --name spider | ||||||
| ``` | ||||||
|
|
||||||
| --- | ||||||
|
|
||||||
| ## Installing the chart | ||||||
|
|
||||||
| ### Adding the Helm repository | ||||||
|
|
||||||
| The chart is published to a Helm repository hosted on the `gh-pages` branch of Spider's GitHub | ||||||
| repository: | ||||||
|
|
||||||
| ```shell | ||||||
| helm repo add spider https://github.com/y-scope/spider/raw/gh-pages | ||||||
| helm repo update spider | ||||||
| ``` | ||||||
|
|
||||||
| ### Basic installation | ||||||
|
|
||||||
| To install the chart with its default values: | ||||||
|
|
||||||
| ```shell | ||||||
| helm install spider spider/spider | ||||||
| ``` | ||||||
|
|
||||||
| ### Installation with custom values | ||||||
|
|
||||||
| For highly customized deployments, you can override the default values by creating a values file. | ||||||
| The chart's defaults live in [`values.yaml`][helm-values], and Helm deep-merges your file into | ||||||
| them, so you only need to list the keys you want to change: | ||||||
|
|
||||||
| ```{code-block} yaml | ||||||
| :caption: spider-values.yaml | ||||||
|
|
||||||
| # Use custom image tags. | ||||||
| image: | ||||||
| scheduler: | ||||||
| tag: "latest" | ||||||
| storage: | ||||||
| tag: "latest" | ||||||
| worker: | ||||||
| tag: "latest" | ||||||
|
|
||||||
| spiderConfig: | ||||||
| # Tune performance and resource usage. | ||||||
| scheduler: | ||||||
| runtime: | ||||||
| scheduler: | ||||||
| config: | ||||||
| active_job_queue_capacity: 64 | ||||||
| dispatch_queue_capacity: 64 | ||||||
|
|
||||||
| # Adjust worker horizontal scaling. | ||||||
| worker: | ||||||
| replicas: 8 | ||||||
| ``` | ||||||
|
|
||||||
| Install the chart with the custom values file: | ||||||
|
|
||||||
| ```shell | ||||||
| helm install spider spider/spider -f spider-values.yaml | ||||||
| ``` | ||||||
|
|
||||||
| :::{note} | ||||||
| The example above shows only a subset of the available settings. For the complete set of settings | ||||||
| that tune each component's runtime behavior, see the [Configuration][configuration] guides. For more | ||||||
| advanced deployment settings, see [Advanced deployment settings](#advanced-deployment-settings) | ||||||
| below. | ||||||
| ::: | ||||||
|
|
||||||
| --- | ||||||
|
|
||||||
| ## Verifying the deployment | ||||||
|
|
||||||
| After installing the Helm chart, you can verify that all components are running correctly as | ||||||
| follows. | ||||||
|
|
||||||
| ### Check pod status | ||||||
|
|
||||||
| Wait for all pods to be ready: | ||||||
|
|
||||||
| ```shell | ||||||
| # Watch pod status | ||||||
| kubectl get pods -w | ||||||
|
|
||||||
| # Wait for all pods to be ready | ||||||
| kubectl wait pods --all --for=condition=Ready --timeout=300s | ||||||
|
Comment on lines
+114
to
+119
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
# Identify the release-specific labels before replacing --all.
rg -n -C 4 'app\.kubernetes\.io/instance|selectorLabels|componentFullname' \
tools/deployment/spider-helmRepository: y-scope/spider Length of output: 24124 🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '%s\n' '--- deployment guide ---'
sed -n '90,135p' docs/huntsman/src/user-docs/guides-deployment/kubernetes.md
printf '%s\n' '--- release references in the guide ---'
rg -n -C 3 'helm (install|upgrade)|kubectl (get|wait)|spider|namespace|label-selector|selector' \
docs/huntsman/src/user-docs/guides-deployment/kubernetes.md
printf '%s\n' '--- chart release selectors and names ---'
sed -n '1,90p' tools/deployment/spider-helm/templates/_helpers.tpl
rg -n -C 3 'name:|selector:|matchLabels:|app\.kubernetes\.io/(name|instance|component)' \
tools/deployment/spider-helm/templatesRepository: y-scope/spider Length of output: 36549 🌐 Web query:
💡 Result: The kubectl get and kubectl wait commands serve distinct purposes in Kubernetes operations, typically used in sequence to monitor and then verify the state of resources [1][2]. kubectl get pods --watch The --watch (or -w) flag allows you to monitor changes to resources in real-time [3][1]. When used with kubectl get pods, it lists existing pods and then streams subsequent updates (such as additions, modifications, or deletions) to the console [3][4]. This is an interactive way to observe the lifecycle of pods as they transition through various states [1]. kubectl wait --for=condition=Ready [label selector] The kubectl wait command is used to block execution until a specific condition is met on one or more resources [2][5]. It is commonly used in automation scripts to ensure resources reach a desired state before proceeding [2]. When combined with a label selector, it monitors all pods matching that selector. For example: kubectl wait --for=condition=Ready -l app=myapp Key details regarding kubectl wait: - Condition Syntax: The --for flag supports several types of conditions, including status conditions (e.g., condition=Ready), lifecycle states (e.g., create, delete), and custom JSONPath expressions [2][5]. - Default Behavior: The default value for a condition is true [2]. You can specify explicit values using an equals sign, such as --for=condition=Ready=false [2][5]. - Timeout: You can control how long the command waits using the --timeout flag (e.g., --timeout=30s). If the condition is not met within the timeout period, the command exits with an error [2][5]. - Multiple Conditions: You can specify multiple --for flags to wait for multiple conditions to be met in a sequential, AND'ed order [2][5]. While kubectl get --watch provides a human-readable stream of events, kubectl wait is designed for programmatic synchronization, making it the standard choice for CI/CD pipelines and shell scripts requiring state verification [1][2]. Citations:
🏁 Script executed: #!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
import re
guide = Path("docs/huntsman/src/user-docs/guides-deployment/kubernetes.md").read_text()
section = guide[guide.index("### Check pod status"):guide.index(":::{note}", guide.index("### Check pod status"))]
commands = re.findall(r"^kubectl .+$", section, re.MULTILINE)
helpers = Path("tools/deployment/spider-helm/templates/_helpers.tpl").read_text()
selector = re.search(
r'define "spider\.selectorLabels".*?\n(.*?)\n\{\{- end \}\}',
helpers,
re.DOTALL,
).group(1)
print("commands:", commands)
print("watch precedes wait:", commands.index("kubectl get pods -w") < commands.index(
"kubectl wait pods --all --for=condition=Ready --timeout=300s"
))
print("watch is blocking:", "-w" in commands[0] or "--watch" in commands[0])
print("release selector uses .Release.Name:", ".Release.Name" in selector)
print("documented release name:", re.findall(r"helm install ([^ ]+)", guide))
PYRepository: y-scope/spider Length of output: 388 Separate the watcher from the readiness check.
🤖 Prompt for AI Agents |
||||||
| ``` | ||||||
|
|
||||||
| The output should show that all pods are in the `Running` state: | ||||||
|
|
||||||
| ```text | ||||||
| NAME READY STATUS RESTARTS AGE | ||||||
| spider-database-0 1/1 Running 0 2m | ||||||
| spider-scheduler-... 1/1 Running 2 2m | ||||||
| spider-storage-... 1/1 Running 2 2m | ||||||
| spider-worker-... 1/1 Running 0 2m | ||||||
| ``` | ||||||
|
|
||||||
| :::{note} | ||||||
| Spider's services fail fast when their dependencies are unreachable, so the storage and scheduler | ||||||
| pods may restart a few times while the database is initializing. A small number of restarts during | ||||||
| startup is expected. | ||||||
|
Comment on lines
+122
to
+135
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -eu
file="docs/huntsman/src/user-docs/guides-deployment/kubernetes.md"
printf '%s\n' '--- target lines ---'
sed -n '105,150p' "$file"
printf '%s\n' '--- relevant restart examples and Kubernetes commands ---'
rg -n -C 2 'RESTARTS|kubectl get pods|Running|restart|helm|database|scheduler|storage' "$file"
printf '%s\n' '--- repository references to the documented pod output ---'
rg -n -C 2 'spider-scheduler|spider-storage|spider-worker|RESTARTS' . --glob '!node_modules' --glob '!dist' --glob '!build'Repository: y-scope/spider Length of output: 19041 🏁 Script executed: #!/bin/bash
set -eu
python3 - <<'PY'
from pathlib import Path
import re
path = Path("docs/huntsman/src/user-docs/guides-deployment/kubernetes.md")
text = path.read_text()
table = re.search(
r"```text\n(?P<header>NAME\s+READY\s+STATUS\s+RESTARTS\s+AGE\n)"
r"(?P<rows>(?:.+\n)+?)```",
text,
)
assert table, "Pod-status example not found"
rows = [line for line in table.group("rows").splitlines() if line.strip()]
restart_values = {}
for row in rows:
fields = row.split()
assert len(fields) == 5, row
restart_values[fields[0]] = fields[3]
note = re.search(
r"Spider's services fail fast.*?may restart a few times.*?startup is expected\.",
text,
re.S,
)
assert note, "Restart-timing note not found"
print("documented restart values:", restart_values)
print("timing-dependent restart note: present")
print("hard-coded restart values:", sorted(set(restart_values.values())))
PYRepository: y-scope/spider Length of output: 365 Use variable restart counts in the sample output. The 🧰 Tools🪛 LanguageTool[style] ~134-~134: Specify a number, remove phrase, use “a few”, or use “some” (SMALL_NUMBER_OF) 🤖 Prompt for AI Agents |
||||||
| ::: | ||||||
|
|
||||||
| --- | ||||||
|
|
||||||
| ## Advanced deployment settings | ||||||
|
|
||||||
| ### Scaling the workers | ||||||
|
|
||||||
| `spiderConfig.worker.replicas` (default: `4`): Sets the number of worker pods. Increase it to | ||||||
| raise the number of tasks the cluster can execute concurrently. | ||||||
|
|
||||||
| ### Making task packages available to the workers | ||||||
|
|
||||||
| The default worker image ships with the execution manager and task executor, but no pre-installed | ||||||
| TDL packages. You can supply your TDL packages in one of two ways: | ||||||
|
|
||||||
| #### Option 1: Mount a volume | ||||||
|
|
||||||
| Mount a volume containing your built packages. Use `extra_volumes` to specify the package source | ||||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We should mention that mounted libraries should be compatible with the worker image's os and arch. |
||||||
| and `extra_volume_mounts` to define where the worker reads them. | ||||||
|
|
||||||
| :::{note} | ||||||
| The `mountPath` must match `spiderConfig.execution_manager.task_executor.package_dir` in your | ||||||
| values file (or its default: `/opt/spider/packages`). | ||||||
| ::: | ||||||
|
|
||||||
| ```{code-block} yaml | ||||||
| :caption: spider-values.yaml | ||||||
|
|
||||||
| spiderConfig: | ||||||
| worker: | ||||||
| extra_volumes: | ||||||
| - name: "task-packages" | ||||||
| hostPath: | ||||||
| path: "/path/to/your/packages" | ||||||
| type: "Directory" | ||||||
| extra_volume_mounts: | ||||||
| - name: "task-packages" | ||||||
| mountPath: "/opt/spider/packages" # Default package_dir. | ||||||
| readOnly: true | ||||||
|
Comment on lines
+154
to
+175
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -e
printf '%s\n' '--- candidate files ---'
git ls-files | rg '(^|/)(kubernetes\.md|kind|values|.*chart.*|.*deployment.*)$' | head -200
printf '%s\n' '--- relevant references ---'
rg -n -C 4 'kind|extraMounts|hostPath|extra_volumes|extra_volume_mounts|package_dir|task-packages' docs/huntsman/src charts . 2>/dev/null | head -500Repository: y-scope/spider Length of output: 43909 🏁 Script executed: #!/bin/bash
set -e
printf '%s\n' '--- kubernetes guide setup and package sections ---'
sed -n '1,50p;135,185p' docs/huntsman/src/user-docs/guides-deployment/kubernetes.md
printf '%s\n' '--- worker deployment template ---'
sed -n '1,75p' tools/deployment/spider-helm/templates/worker-deployment.yaml
printf '%s\n' '--- chart defaults and config paths ---'
sed -n '35,60p;84,100p' tools/deployment/spider-helm/values.yaml
printf '%s\n' '--- static configuration verifier ---'
python3 - <<'PY'
from pathlib import Path
guide = Path("docs/huntsman/src/user-docs/guides-deployment/kubernetes.md").read_text()
template = Path("tools/deployment/spider-helm/templates/worker-deployment.yaml").read_text()
values = Path("tools/deployment/spider-helm/values.yaml").read_text()
checks = {
"default_kind_create_command": "kind create cluster --name spider" in guide,
"kind_extra_mounts_documented": "extraMounts" in guide,
"host_path_package_example": 'path: "/path/to/your/packages"' in guide,
"directory_host_path": 'type: "Directory"' in guide,
"worker_extra_volumes_rendered": ".Values.spiderConfig.worker.extra_volumes" in template,
"worker_extra_mounts_rendered": ".Values.spiderConfig.worker.extra_volume_mounts" in template,
"default_package_dir": 'package_dir: "/opt/spider/packages"' in values,
}
for key, value in checks.items():
print(f"{key}={value}")
PYRepository: y-scope/spider Length of output: 5989 🌐 Web query:
💡 Result: To mount a host directory into a kind (Kubernetes in Docker) node, you use the extraMounts configuration in your kind cluster configuration file [1][2]. Configuration You define the extraMounts under the nodes section of your kind configuration YAML [1]. This maps a path on your host machine to a path inside the kind node container [1][2]. Example configuration: kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 nodes: - role: control-plane extraMounts: - hostPath: /path/on/your/host containerPath: /path/in/node readOnly: false # Optional: default is false Once the cluster is created with this configuration, the directory from your host will be available at the specified containerPath inside the kind node [2][3]. Using in Kubernetes After the host directory is mounted into the kind node, you can use it in your Kubernetes manifests by creating a PersistentVolume (PV) with a hostPath volume type, pointing to the containerPath you specified in the kind configuration [2][3]. Example PV manifest: apiVersion: v1 kind: PersistentVolume metadata: name: example-pv spec: capacity: storage: 1Gi accessModes: - ReadWriteOnce hostPath: path: /path/in/node # This must match the containerPath from your kind config Important Considerations - Docker Resource Limits: If you are using Docker Desktop (e.g., on macOS or Windows), ensure the directory you are mounting is included in the File Sharing settings of your Docker environment; otherwise, the directory may appear empty inside the container [4]. - Propagation: The extraMounts configuration supports an optional propagation field (None, HostToContainer, or Bidirectional), though it is generally not needed for basic mounting [1]. - Node-Specific: Since kind nodes are separate Docker containers, you must define the extraMounts for each specific node (control-plane or worker) where you want the host path to be accessible [3][5]. Citations:
Make the When users create the cluster with 🤖 Prompt for AI Agents |
||||||
| ``` | ||||||
|
|
||||||
| #### Option 2: Build a custom worker image | ||||||
|
|
||||||
| For TDL packages requiring complex dependencies or runtimes, build a custom container image using | ||||||
| `ghcr.io/y-scope/spider/worker` as the base: | ||||||
|
|
||||||
| 1. Create a `Dockerfile` starting `FROM ghcr.io/y-scope/spider/worker` that installs your TDL | ||||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
| package and required dependencies. | ||||||
|
|
||||||
| 2. Override `image.worker.repository` and `image.worker.tag` in your Helm values file to point to | ||||||
| your container image. | ||||||
|
Comment on lines
+180
to
+187
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -e
printf '%s\n' '--- target documentation ---'
sed -n '150,205p' docs/huntsman/src/user-docs/guides-deployment/kubernetes.md
printf '%s\n' '--- image references and chart defaults ---'
rg -n --glob '!node_modules' 'ghcr\.io/y-scope/spider/worker|image:|repository:|tag:' . | head -200Repository: y-scope/spider Length of output: 5664 🏁 Script executed: #!/bin/bash
set -e
printf '%s\n' '--- documented Helm image settings ---'
sed -n '30,85p' docs/huntsman/src/user-docs/guides-deployment/kubernetes.md
printf '%s\n' '--- chart worker image defaults and rendering ---'
sed -n '1,30p' tools/deployment/spider-helm/values.yaml
sed -n '65,95p' tools/deployment/spider-helm/templates/_helpers.tpl
printf '%s\n' '--- all worker image documentation and configuration references ---'
rg -n -C 3 'worker.*(repository|tag)|image\.worker|worker:' docs/huntsman/src/user-docs/guides-deployment/kubernetes.md tools/deployment/spider-helm
printf '%s\n' '--- deterministic reference check ---'
python3 - <<'PY'
from pathlib import Path
import re
doc = Path("docs/huntsman/src/user-docs/guides-deployment/kubernetes.md").read_text()
values = Path("tools/deployment/spider-helm/values.yaml").read_text()
dockerfile = re.search(r'FROM\s+(\S+)', doc[doc.index("#### Option 2"):])
worker = re.search(r'worker:\s*\n\s+repository:\s*"([^"]+)"\s*\n\s+tag:\s*"([^"]+)"', values)
print("documented FROM:", dockerfile.group(1) if dockerfile else None)
print("chart worker default:", f"{worker.group(1)}:{worker.group(2)}" if worker else None)
print("FROM has explicit tag:", bool(dockerfile and ":" in dockerfile.group(1).rsplit("/", 1)[-1]))
PYRepository: y-scope/spider Length of output: 8168 🏁 Script executed: #!/bin/bash
set -e
python3 - <<'PY'
from pathlib import Path
import re
doc = Path("docs/huntsman/src/user-docs/guides-deployment/kubernetes.md").read_text()
values = Path("tools/deployment/spider-helm/values.yaml").read_text()
section = doc[doc.index("#### Option 2"):doc.index("### Passing environment variables")]
from_refs = re.findall(r'\bFROM\s+([^\s`]+)', section)
worker_block = re.search(
r'(?ms)^ worker:\n(?:(?: .*)\n)*? repository:\s*"([^"]+)"\n tag:\s*"([^"]+)"',
values,
)
print("documented FROM references:", from_refs)
print("chart worker repository:", worker_block.group(1) if worker_block else None)
print("chart worker tag:", worker_block.group(2) if worker_block else None)
for ref in from_refs:
image = ref.rstrip("`")
last = image.rsplit("/", 1)[-1]
print(f"{image}: implicit latest =", ":" not in last)
PYRepository: y-scope/spider Length of output: 144 Pin the custom worker image base tag. The Helm chart defaults to 🤖 Prompt for AI Agents |
||||||
|
|
||||||
| ### Passing environment variables to tasks | ||||||
|
|
||||||
| Once a TDL package is available, you may pass environment variables through Spider's Helm chart | ||||||
| values file so tasks can consume them at runtime. Forwarding variables requires configuring two | ||||||
| settings: | ||||||
|
|
||||||
| * `spiderConfig.worker.extra_envs`: Adds environment variables to the execution manager container. | ||||||
|
|
||||||
| * `spiderConfig.execution_manager.task_executor.inherited_env`: Lists the variables to forward from | ||||||
| the execution manager to the task executors. | ||||||
|
|
||||||
| :::{important} | ||||||
| A variable must be listed in both fields to be accessible by a running task. | ||||||
| ::: | ||||||
|
Comment on lines
+197
to
+202
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Right now all environment variables received by executor manager are passed to task executor. |
||||||
|
|
||||||
| #### Example configuration | ||||||
|
|
||||||
| To pass the `AWS_REGION` environment variable to your tasks: | ||||||
|
|
||||||
| ```{code-block} yaml | ||||||
| :caption: spider-values.yaml | ||||||
|
|
||||||
| spiderConfig: | ||||||
| execution_manager: | ||||||
| task_executor: | ||||||
| inherited_env: ["AWS_REGION"] | ||||||
|
|
||||||
| worker: | ||||||
| extra_envs: | ||||||
| - name: "AWS_REGION" | ||||||
| value: "us-east-2" | ||||||
| ``` | ||||||
|
|
||||||
| [configuration]: ../guides-configuration/index.md | ||||||
| [Docker]: https://docs.docker.com/engine/install/ | ||||||
| [Helm]: https://helm.sh/ | ||||||
| [helm-chart]: https://github.com/y-scope/spider/tree/main/tools/deployment/spider-helm | ||||||
| [helm-values]: https://github.com/y-scope/spider/blob/main/tools/deployment/spider-helm/values.yaml | ||||||
| [kind]: https://kind.sigs.k8s.io/ | ||||||
| [kubectl]: https://kubernetes.io/docs/tasks/tools/ | ||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What do you mean by cluster? Is it a
kindcluster, or anyk8scluster?