FFNaaS for qwen3-30b

.agents/skills/deploy-afd-k8s/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: deploy-afd-k8s
-description: Use when the user asks to deploy, serve, or stand up an AFD GPU recipe on Kubernetes/OpenShift from a recipe .sh script - creating the model PVC, recipe ConfigMap, serve pod(s), and Service(s), and waiting until the endpoint answers. Placement across pods is never asked for or assumed -- it is read directly off the recipe's own top-of-file comment block (however that block describes it, strict or prose): a recipe whose comment describes a per-pod placement deploys exactly that many pods with exactly that per-pod setup; a recipe without one deploys as a single pod. Do not use for local (non-k8s) serving, NPU recipes, prefill-decode disaggregation recipes, or E2E correctness testing (see run-e2e).
+description: Use when the user asks to deploy, serve, or stand up an AFD GPU recipe on Kubernetes/OpenShift from a recipe .sh script - creating the model PVC, recipe ConfigMap, serve pod(s), and Service(s), and waiting until the endpoint answers. Placement across pods is never asked for or assumed -- it is the recipe's LAYOUT (the caller's setting, else the script's default): one pod per LAYOUT entry, a one-entry LAYOUT being a single pod. Do not use for local (non-k8s) serving, NPU recipes, prefill-decode disaggregation recipes, or E2E correctness testing (see run-e2e).
 ---
 
 # Deploy an AFD GPU recipe on Kubernetes
@@ -15,36 +15,38 @@ On success this skill guarantees, in the target namespace:
 
 | | |
 |---|---|
-| Endpoint | `http://vllm-service:${CLIENT_PORT}` (OpenAI-compatible, in-namespace; `CLIENT_PORT` defaults to `18305`) |
+| Endpoint | `http://vllm-service:${CLIENT_PORT}` (OpenAI-compatible, in-namespace; `CLIENT_PORT` is the launcher's Attention port, `18000`) |
 | Serving | the model named by `MODEL_ID` |
-| Pod(s) / Service | `vllm-pod`, or one pod per pod the recipe's header comment describes (named `vllm-pod-<pod>`); Service `vllm-service`, labels `app=afd-recipe,role=serve` |
+| Pod(s) / Service | `vllm-pod`, or one pod per `LAYOUT` entry (named `vllm-pod-afd-<k>`); Service `vllm-service`, labels `app=afd-recipe,role=serve` |
 | Model cache | `PVC_NAME`, mounted at `/models`, `HF_HOME=/models/.hf_home` |
-| Left running | yes, deliberately -- so weights stay warm for follow-up runs |
+| Left running | yes, deliberately -- so weights stay warm for follow-up runs; step 5 asks the caller when to delete the pods |
 
-Whenever a recipe's header spans more than one pod, two more internal-only
+Whenever a recipe's `LAYOUT` spans more than one pod, two more internal-only
 Services may also exist (`vllm-ffn-p2p-service` always; `vllm-attn-dp-service`
 and/or `vllm-ffn-dp-service` only if that role is itself split across more
 than one pod) -- callers never talk to these directly. See
-[services.md](references/services.md).
+[services.md](resources/services.md).
 
 ## Scope
 
-- Recipes under `recipe/gpu/P2pNcclAFDConnector/**` (GPU only, not NPU),
+- Recipes under `recipe/gpu/P2pNcclAFDConnector/**` or
+  `recipe/gpu/DeepEPM2NAFDConnector/**` (GPU only, not NPU; the M2N image
+  must contain afdEP-comm and DeepGEMM),
   **`prefill_decode_colocation` only** (attention + FFN workers, no prefill
   split).
   `prefill_decode_disaggregation` recipes are **out of scope**: they need a
   NIXL-enabled image plus a proxy, and resolve a `SCRIPT_DIR`-relative path
   that this skill's flat script mount (step 4b) breaks. Ask the user for a
   different workflow for disaggregation.
-- Recipes are hand-written shell scripts, not JSON configs -- there is no
-  generator step and no manifest to read.
-- A recipe is **placement-driven** (more than one pod) exactly when its
-  top-of-file comment block describes a per-pod placement -- read that
-  block, don't infer shape from flags. Today's recipes spell it as one
-  `# For pod N: POD=...` line per pod, but the comment isn't guaranteed to
-  use that exact grammar going forward. See
-  [resolve-recipe.md](references/resolve-recipe.md).
-- Not for `tools/benchmarks/decode_bench_server.sh` (local, non-k8s).
+- Recipes are short shell scripts on the shared launcher
+  `recipe/gpu/common/afd_colocation.sh` (see `recipe/gpu/common/README.md`);
+  the recipe ConfigMap ships both files (step 4b).
+- Placement is `LAYOUT` (one pod per `<a>A<f>F` entry) and run-specific
+  settings are env (`ATTN_TP`, `FFN_TP`, `VLLM_ARGS`, `DECODE_BENCH`, ...);
+  see [resolve-recipe.md](resources/resolve-recipe.md). This skill only
+  serves; load for the decode-only benchmark
+  (`tools/benchmarks/request_generator.sh`) is run separately against the
+  endpoint.
 
 ## Requirements
 
@@ -61,9 +63,9 @@ than one pod) -- callers never talk to these directly. See
   docker push "$IMAGE"
   ```
 
-  The image supplies only the `afd-plugin` install. The recipe script is
-  mounted fresh from local disk on every run (step 4b), so editing or adding
-  a recipe never requires a rebuild.
+  The image supplies only the `afd-plugin` install. The recipe script and
+  the shared launcher are mounted fresh from local disk on every run (step
+  4b), so editing or adding a recipe never requires a rebuild.
 - An `hf-token-secret` Secret with a `token` key, needed when `MODEL_ID` is
   a HF Hub repo id (not needed when weights are already staged on the PVC):
   ```bash
@@ -75,13 +77,13 @@ than one pod) -- callers never talk to these directly. See
 
 Load these in order; each covers one phase and nothing else. Services must
 exist **before** any Pod is deployed -- see
-[services.md](references/services.md) for why:
+[services.md](resources/services.md) for why:
 
-| Reference | Covers |
+| Resource | Covers |
 |---|---|
-| [resolve-recipe.md](references/resolve-recipe.md) | Step 1: read the recipe's serve-block fields and model id; read its placement (or lack of one) from its header comment -- never ask. |
-| [deploy.md](references/deploy.md) | Steps 2-4f: GPU_COUNT, confirm, PVC, ConfigMap, fsGroup, Services (step 4d -- which of the four, see [services.md](references/services.md)), pod(s) via `templates/pod.yaml`, wait for Running and for each pod's own readiness line. |
-| [report-and-teardown.md](references/report-and-teardown.md) | Step 5: report the endpoint/model/nodes. Step 6: teardown (pods/Services only -- never the PVC unless asked). |
+| [resolve-recipe.md](resources/resolve-recipe.md) | Step 1 (the only step that parses the recipe): model id, the `LAYOUT` plan (never ask), `GPU_COUNT` per pod, and the launcher's ports. |
+| [deploy.md](resources/deploy.md) | Steps 2-4f: take step 1's `GPU_COUNT`, confirm, PVC, ConfigMap, fsGroup, Services (step 4d -- which of the four, see [services.md](resources/services.md)), pod(s) via `templates/pod.yaml`, wait for Running and then for each pod's `Ready` condition. |
+| [report-and-teardown.md](resources/report-and-teardown.md) | Step 5: report the endpoint/model/nodes and ask when to delete the running pods. Step 6: teardown when the caller says so (pods/Services only -- never the PVC unless asked). |
 
 All k8s manifests live in `templates/*.yaml`, applied via `envsubst | kubectl
 apply -f -`; no manifest is inlined in these docs.

.agents/skills/deploy-afd-k8s/references/deploy.md → .agents/skills/deploy-afd-k8s/resources/deploy.md

@@ -1,31 +1,15 @@
-# 2-4. Compute GPU_COUNT, confirm, and deploy
+# 2-4. Confirm and deploy
 
 ## 2. GPU_COUNT
 
-Plain single-pod recipe (no plan from [resolve-recipe.md](resolve-recipe.md)):
-union of every literal `CUDA_VISIBLE_DEVICES=` in the script.
+Use the `GPU_COUNT` recorded by [resolve-recipe.md](resolve-recipe.md) --
+`a + f` per `LAYOUT` entry. Don't re-read the script here; all recipe
+parsing lives in step 1:
 
 ```bash
-GPU_COUNT="$(grep -oE 'CUDA_VISIBLE_DEVICES=[0-9,]+' "$RECIPE_SCRIPT_PATH" \
-  | cut -d= -f2 | tr ',' '\n' | sort -u | wc -l)"
-```
-
-Per plan entry: count the devices in the literal `CUDA_VISIBLE_DEVICES=` inside
-the recipe's `if` block that tests this pod's role (`ATTENTION` or `FFN` --
-whichever prefix `POD` starts with). Every pod sharing a role shares that
-literal, so this is a plain text extraction, not a per-pod computation:
-
-```bash
-role_block() {  # $1=script $2=ATTENTION|FFN
-  awk -v role="$2" '
-    /if.*\$POD/ && $0 ~ role { inblock=1; next }
-    inblock && /^fi/ { inblock=0 }
-    inblock { print }
-  ' "$1"
-}
-devices="$(role_block "$RECIPE_SCRIPT_PATH" "$ROLE" \
-  | grep -oE 'CUDA_VISIBLE_DEVICES=[0-9,]+' | head -1 | cut -d= -f2)"
-GPU_COUNT=$(( $(grep -o ',' <<<"$devices" | wc -l) + 1 ))
+LAYOUT=${LAYOUT:-$(grep -oE 'LAYOUT:-[0-9AF,]+' "$RECIPE_SCRIPT_PATH" | cut -d- -f2)}
+IFS=, read -ra ENTRIES <<<"$LAYOUT"
+for e in "${ENTRIES[@]}"; do f=${e#*A}; echo "$e -> $(( ${e%%A*} + ${f%F} )) GPUs"; done
 ```
 
 ## 3. Confirm before touching the cluster
@@ -41,7 +25,7 @@ existing Pod(s)** with the same name(s) about to be deployed.
 AFD_PLUGIN_IMAGE=<image>
 MODEL_ID=<model-id>
 PVC_NAME=<pvc-name>
-CLIENT_PORT=<n>   # from the script's --port; default 18305
+CLIENT_PORT=<n>   # the launcher's ATTN_PORT (18000), see resolve-recipe.md
 RECIPE_SCRIPT_PATH=<local-recipe-script-path>
 ```
 
@@ -94,11 +78,14 @@ rather than re-downloading.
 ## 4b. Recipe ConfigMap
 
 Lets the pod run any local recipe -- edited, uncommitted, or new -- without
-rebuilding the image. Run from wherever `RECIPE_SCRIPT_PATH` resolves:
+rebuilding the image. The recipe sources the shared launcher, so ship both;
+`templates/pod.yaml` points the recipe at `/recipe/afd_colocation.sh`
+(`AFD_RECIPE_LIB`). Run from the repository root:
 
 ```bash
 kubectl create configmap afd-recipe-script \
   --from-file=recipe.sh="${RECIPE_SCRIPT_PATH}" \
+  --from-file=afd_colocation.sh=recipe/gpu/common/afd_colocation.sh \
   --dry-run=client -o yaml | kubectl apply -f -
 ```
 
@@ -130,17 +117,19 @@ recipe) against the one shared `templates/pod.yaml`:
 ```bash
 deploy_pod() {
   local pod_name="$1" gpu="$2" attn_role="$3" ffn_role="$4" extra_env_pairs="$5"
-  # extra_env_pairs: space-separated VAR=value tokens straight off this
-  # plan entry's header line (including POD=... itself), empty for a plain
-  # recipe. Build the spliced env-entries block for templates/pod.yaml.
-  # Leading (not trailing) newlines keep the `# ${TEMPLATE_EXTRA_ENV}`
-  # placeholder line a harmless comment when this is empty, and valid
-  # sibling list items when it isn't.
-  local extra_env=""
-  for pair in $extra_env_pairs; do
+  # extra_env_pairs: one VAR=value per line -- this plan entry's overrides
+  # (including POD=... itself) plus the caller's settings -- empty for a plain
+  # recipe. Values may hold spaces and JSON (VLLM_ARGS, DECODE_BENCH); they
+  # become single-quoted YAML strings. Leading (not trailing) newlines keep
+  # the `# ${TEMPLATE_EXTRA_ENV}` placeholder line a harmless comment when
+  # this is empty, and valid sibling list items when it isn't.
+  local extra_env="" pair val q="'"
+  while IFS= read -r pair; do
+    [ -n "$pair" ] || continue
+    val=${pair#*=}
     extra_env="$extra_env
-        - {name: ${pair%%=*}, value: \"${pair#*=}\"}"
-  done
+        - {name: ${pair%%=*}, value: '${val//$q/$q$q}'}"
+  done <<<"$extra_env_pairs"
 
   kubectl delete pod "$pod_name" --ignore-not-found
 
@@ -153,27 +142,51 @@ deploy_pod() {
 }
 ```
 
-Plain recipe, one pod, both roles head, no overrides:
+One-entry `LAYOUT` (e.g. the default `2A2F`), one pod, both roles head,
+with or without the caller's settings:
 
 ```bash
 deploy_pod vllm-pod "$GPU_COUNT" head head ""
+deploy_pod vllm-pod 4 head head $'VLLM_ARGS=--max-num-seqs 32'
 ```
 
-Placement-driven recipe, one call per entry parsed in
-[resolve-recipe.md](resolve-recipe.md), e.g. the qwen3.5 3-pod plan:
+Several entries, one call per entry parsed in
+[resolve-recipe.md](resolve-recipe.md), e.g. `LAYOUT=2A0F,0A2F` (2 GPUs per
+pod) with the decode-only benchmark, `LAYOUT=4A1F,4A1F` (5 GPUs per pod),
+or the Qwen3.5 recipe's default `2A0F,2A0F,0A2F`:
 
 ```bash
-deploy_pod vllm-pod-attention-0 2 head none "POD=ATTENTION_0"
-deploy_pod vllm-pod-attention-1 2 worker none "POD=ATTENTION_1 ATTENTION_HEADLESS=1 ATTENTION_DP_START_RANK=2"
-deploy_pod vllm-pod-ffn-0 2 none head "POD=FFN_0"
+SETTINGS=$'LAYOUT=2A0F,0A2F\nDECODE_BENCH={}\nAFD_REPO=/opt/afd-plugin'
+deploy_pod vllm-pod-afd-0 2 head none $'POD=AFD_0\n'"$SETTINGS"
+deploy_pod vllm-pod-afd-1 2 none head $'POD=AFD_1\n'"$SETTINGS"
+
+deploy_pod vllm-pod-afd-0 5 head head $'POD=AFD_0\nLAYOUT=4A1F,4A1F'
+deploy_pod vllm-pod-afd-1 5 worker worker $'POD=AFD_1\nLAYOUT=4A1F,4A1F'
+
+deploy_pod vllm-pod-afd-0 2 head none 'POD=AFD_0'
+deploy_pod vllm-pod-afd-1 2 worker none 'POD=AFD_1'
+deploy_pod vllm-pod-afd-2 2 none head 'POD=AFD_2'
 ```
 
-## 4f. Wait for Running, then for each pod's own readiness line
+## 4f. Wait for Running, then for each pod's Ready condition
 
 Bounded 15-minute wait per pod for `phase=Running` (unschedulable pods never
-reach `Failed`, so an unbounded loop would spin forever), then poll logs for
-`=== pod <name> READY ===` (budget 60 minutes -- model download and graph
-capture dominate):
+reach `Failed`, so an unbounded wait would spin forever), then a 60-minute
+wait per pod for its `Ready` condition (model download and graph capture
+dominate).
+
+`Ready` is the single readiness signal. The pod's `readinessProbe` only
+turns it on after the in-pod startup script has seen every log marker, got
+`/health` from the attention head, and printed `=== pod <name> READY ===`;
+the same condition is what puts the attention head behind `vllm-service`.
+If the in-pod script gives up, it prints `ERROR:` with log tails and exits,
+the pod goes `Failed`, and the loop below stops at once.
+
+Timeouts are nested on purpose: the in-pod startup deadline
+(`STARTUP_TIMEOUT_S` in `templates/pod.yaml`, 55 minutes, one budget shared
+by every log and `/health`) is shorter than the 60-minute wait here, so a
+slow pod always fails with its own `ERROR:` reason instead of an outer
+timeout. Change the two together.
 
 ```bash
 for pod_name in <every deployed pod_name>; do
@@ -184,18 +197,22 @@ done
 for pod_name in <every deployed pod_name>; do
   deadline=$(( $(date +%s) + 3600 ))
   while true; do
-    logs="$(kubectl logs "pod/$pod_name" 2>/dev/null || true)"
-    echo "$logs" | grep -q "pod ${pod_name} READY" && break
-    echo "$logs" | grep -q "ERROR:" && { kubectl logs "$pod_name" --tail=200; exit 1; }
-    [ "$(date +%s)" -ge "$deadline" ] && { kubectl logs "$pod_name" --tail=200; exit 1; }
+    read -r phase ready < <(kubectl get pod "$pod_name" \
+      -o jsonpath='{.status.phase} {.status.conditions[?(@.type=="Ready")].status}' || true)
+    [ "$ready" = "True" ] && break
+    if [ "$phase" = "Failed" ] || [ "$(date +%s)" -ge "$deadline" ]; then
+      echo "pod $pod_name not Ready (phase=${phase:-unknown})"
+      kubectl logs "$pod_name" --tail=200
+      exit 1
+    fi
     sleep 10
   done
 done
 ```
 
-Only report the endpoint ready once every pod in the plan has printed its
-own `READY` line -- one pod becoming ready doesn't imply the others did.
+Only report the endpoint ready once every pod in the plan is `Ready` -- one
+pod becoming ready doesn't imply the others did.
 
-Startup markers are keyed on the fixed log names every colocation recipe
-uses today -- see `templates/pod.yaml`'s `ready_marker()` for the exact
-lines. A recipe using different log names needs that template updated.
+Startup markers are keyed on the launcher's fixed log names (`attn.log`,
+`ffn.log`) -- see `templates/pod.yaml`'s `ready_marker()` for the exact
+lines.

.agents/skills/deploy-afd-k8s/references/report-and-teardown.md → .agents/skills/deploy-afd-k8s/resources/report-and-teardown.md

@@ -12,6 +12,13 @@ kubectl get pods -l app=afd-recipe -o wide
 Every node matters if a follow-up pod needs to attach the same
 `ReadWriteOnce` PVC -- it can only attach from that node.
 
+**Ask when to delete the running pods.** End the report by asking the
+caller -- the user, or the task that invoked this skill (e.g.
+`run-vllm-bench-k8s`) -- when the pods should be deleted: right after its
+follow-up work, at a later point it names, or never (keep them warm). The
+caller owns that decision and triggers step 6 when the time comes; this
+skill never tears down on its own.
+
 # 6. Teardown
 
 Left running by design, so weights stay warm -- report what's up and how to