Skip to content

refactor: unify images into calicovpp/vpp and flatten repo layout#994

Open
aritrbas wants to merge 1 commit into
masterfrom
abasu-unify-image
Open

refactor: unify images into calicovpp/vpp and flatten repo layout#994
aritrbas wants to merge 1 commit into
masterfrom
abasu-unify-image

Conversation

@aritrbas

@aritrbas aritrbas commented Apr 30, 2026

Copy link
Copy Markdown
Collaborator

Summary:

This PR does two related things on the path to CalicoVPP release/v3.33.0:

  1. Unifies dataplane images and flattens the repo layout - consolidates calicovpp/agent, calicovpp/vpp, calicovpp/multinet-monitor and calicovpp/init-eks into a single calicovpp/vpp image and moves libraries under pkg/ with entrypoints under cmd/.
  2. Introduces a kustomize-based VPP kube-test integration - CalicoVPP now owns the VPP kube-test manifest and exposes a small Makefile API so VPP kube-test (and other consumers) no longer need to know CalicoVPP internal layout details (image names, in-repo VPP build paths, etc.).

Runtime behavior is unchanged - the agent and VPP still run as separate containers; the agent container reuses calicovpp/vpp with an explicit command: ["/bin/calico-vpp-agent"] override.

Changes:

  • Unified image: calicovpp/vpp (contains all dataplane binaries)
  • Single env var: CALICO_VPP_IMAGE (replaces CALICO_AGENT_IMAGE, CALICO_VPP_IMAGE, MULTINET_MONITOR_IMAGE)
  • calicovpp/vclsidecar and calicovpp/ctl remain separate images
  • Unified Dockerfiles (images/ubuntu/Dockerfile, Dockerfile.debug) ship all binaries: calico-vpp-agent, felix-api-proxy, gobgp, multinet-monitor, vpp-manager and init-eks scripts under images/ubuntu/bin/
  • Top-level Makefile delegates image build through pkg/vpp-manager
  • pkg/vpp-manager/Makefile builds all Go binaries into the unified image
  • Per-component image/push targets removed from agent and multinet-monitor Makefiles
  • Manifests updated to calicovpp/vpp with explicit command overrides for agent and multinet-monitor containers
  • Path references in test/ and docs/ updated for pkg/ move
  • Image contexts moved to top-level images/
  • Helper scripts moved to top-level scripts/
  • VPP clone/build directory moved to top-level vpp_build/

Directory Restructure:

├── cmd
│ ├── api-proxy
│ ├── calico-vpp-agent
│ ├── calicovppctl
│ ├── multinet-monitor
│ ├── replay-trace
│ └── vpp-manager
├── common.mk
├── docs
├── images
├── Makefile
├── pkg
│ ├── calico-vpp-agent
│ ├── config
│ ├── multinet-monitor
│ ├── vpplink
│ └── vpp-manager
├── scripts
├── test
├── vpp_build
└── yaml

VPP kube-test integration:

CalicoVPP now owns the kube-test manifest as kustomize sources under yaml/components/kube-test/ and yaml/overlays/kube-test-{kind,baremetal}/. Layout-specific values (unified image name, agent command override, VPP build path vpp_build/, registry/pull-policy) are baked into the component patches.

Only consumer-owned runtime placeholders survive in the emitted manifest:

Placeholder Owner / typical source
${CALICOVPP_VERSION} image tag (kt-master, v3.32.0, …)
${HOME} host path for calicovpp checkout bind mount
${ADDITIONAL_VPP_CONFIG} per-test extra VPP startup CLI
${CALICOVPP_ENABLE_MEMIF} per-test memif feature toggle
${CALICO_NETWORK_CONFIG} per-test calico-network override
${CALICOVPP_INTERFACE} baremetal-only: uplink NIC name

kustomize layout:

  • yaml/components/kube-test/ — shared kube-test component (daemonset patch, configmap defaults, tigera-operator snippet)
  • yaml/components/kube-test/kind/kind adjustments (eth0, af_packet, no hugepages)
  • yaml/components/kube-test/baremetal/baremetal adjustments (DPDK uplink, hugepages, larger CPU/buffer pool, docker.io registry)
  • yaml/overlays/kube-test-{kind,baremetal}/ — wires the base daemonset and the kube-test components.
  • yaml/generated/calico-vpp-kubetest.yaml — checked-in kind flavor output; baremetal is built on demand
  • yaml/Makefile — regenerates the kind kubetest yaml and sanity-checks the baremetal overlay

The tigera-operator Installation/APIServer CRs are not kustomize resources (they embed ${CALICO_NETWORK_CONFIG}, which is invalid YAML until envsubst runs). They are appended verbatim by make kube-test-template from yaml/components/kube-test/tigera-installation.snippet.yaml.

Makefile API (consumed by VPP kube-test)

Target Purpose
make vpp [VPP_DIR=…] [BASE=…] Build VPP; delegates to pkg/vpp-manager
make kube-test-template FLAVOR={kind,baremetal} Emit fully-resolved kube-test manifest
make kube-test-push-images CALICOVPP_VERSION=… Pull/tag/push images to local kind registry

KUBE_TEST_IMAGES defaults to calicovpp/vpp on this branch (unified layout). Older split-image release branches backport this with default value calicovpp/vpp calicovpp/agent calicovpp/multinet-monitor.

Benefits:

  • Single image tag to manage across all dataplane components.
  • Reduced bandwidth and storage requirements.
  • No risk of image version drift between components.
  • Simplified CI/CD pipeline.
  • Easier version management and releases.
  • Clear separation between CalicoVPP (layout owner) and VPP kube-test (consumer of generic Makefile).
  • kind and baremetal kube-test flavors supported from the same kustomize base.

VPP kube-test PR:

calicovpp-deployments PR:

@aritrbas aritrbas self-assigned this Apr 30, 2026
@aritrbas aritrbas force-pushed the abasu-unify-image branch from 274424b to f6fd812 Compare April 30, 2026 21:48
@aritrbas aritrbas changed the title refactor: unify images into calicovpp/vpp + restructure code into pkg/ refactor: unify images into calicovpp/vpp and flatten repo layout Apr 30, 2026
@aritrbas aritrbas force-pushed the abasu-unify-image branch 3 times, most recently from 8360e7e to b6dee79 Compare May 5, 2026 19:59
@aritrbas aritrbas force-pushed the abasu-unify-image branch 2 times, most recently from f1d81e2 to e50ee99 Compare May 13, 2026 19:10

@sknat sknat left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks neat ! Thanks a bunch for putting this together

@aritrbas aritrbas force-pushed the abasu-unify-image branch 2 times, most recently from cbb3065 to b1189d7 Compare May 22, 2026 08:18

@sknat sknat left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@aritrbas aritrbas force-pushed the abasu-unify-image branch from 2001663 to 072611e Compare June 18, 2026 00:04
- consolidate calicovpp/agent, calicovpp/vpp, calicovpp/multinet-monitor
  and calicovpp/init-eks into a single unified calicovpp/vpp image.
- restructure source directories under pkg/ for cleaner organization.
- move executable entrypoints to top-level cmd/ to allow for building
  all golang agents from the top-level Makefile.
- runtime behavior is unchanged - two separate daemons (agent and vpp)
  run in separate containers from the same image using command overrides.

Changes:
- unified image: calicovpp/vpp (contains all binaries)
- single env var: CALICO_VPP_IMAGE (replaces CALICO_AGENT_IMAGE,
  CALICO_VPP_IMAGE, MULTINET_MONITOR_IMAGE)
- unified dockerfiles: Dockerfile and Dockerfile.debug now contain all
  binaries - calico-vpp-agent, felix-api-proxy, gobgp, multinet-monitor,
  vpp-manager and init-eks scripts in images/ubuntu/bin/
- top-level Makefile delegates image build through vpp-manager
- vpp-manager Makefile builds all Go binaries into unified image
- image/push targets removed from agent and multinet-monitor Makefile
- manifests updated to calicovpp/vpp with explicit command overrides for
  agent and multinet-monitor containers
- path references in test/ and docs/ updated for pkg/ move
- image contexts moved to top-level images/
- helper scripts moved to top-level scripts/
- VPP clone/build directory relocated to top-level vpp_build

Changes for VPP kube-test integration:

CalicoVPP now owns the kube-test manifest as a set of kustomize sources
so VPP kube-test (and any other consumer) does not need to know which
image to use, where the VPP build lives in this repo or any other
CalicoVPP-internal layout detail.

- yaml/components/kube-test/ - kustomize coimponent shared by the kind
  and baremetal flavors. Defines the kube-test daemonset patch, the
  kube-test configmap defaults and the tigera-operator snippet.
- yaml/components/kube-test/kind/ - KinD-flavor adjustments
  (uplink eth0 via af_packet, no hugepages).
- yaml/components/kube-test/baremetal/ - baremetal-flavor adjustments
  (DPDK uplink driven by ${CALICOVPP_INTERFACE}, hugepages, larger
  CPU/buffer allocation, docker.io registry, IfNotPresent pull policy).
- yaml/overlays/kube-test-{kind,baremetal}/ - overlays wiring the
  base daemonset and the kube-test components together.
- yaml/generated/calico-vpp-kubetest.yaml - regenerated from the
  kind overlay; consumers see only the runtime placeholders
  ${CALICOVPP_VERSION}, ${HOME}, ${ADDITIONAL_VPP_CONFIG},
  ${CALICOVPP_ENABLE_MEMIF}, ${CALICO_NETWORK_CONFIG} and
  ${CALICOVPP_INTERFACE} (baremetal only).
- yaml/Makefile - regenerates the checked-in kubetest yaml for kind
  and sanity-checks the baremetal overlay.

Top-level Makefile targets (the public API consumed by VPP kube-test):
- make vpp - delegates to pkg/vpp-manager with VPP_DIR/BASE pass-through;
             consumers no longer need to know where VPP build sources live.
- make kube-test-template FLAVOR={kind,baremetal}
           - runs `kubectl kustomize` on the requested overlay and appends
             the tigera Installation/APIServer snippet (the snippet contains
             the ${CALICO_NETWORK_CONFIG} placeholder, so it is not a
             kustomize resource and is concatenated verbatim post-build).
- make kube-test-push-images
           - pulls/tags/pushes each image in $KUBE_TEST_IMAGES (default:
             calicovpp/vpp) into the local kind registry. Older split-image
             release branches backport this target with ${KUBE_TEST_IMAGES}
             as calicovpp/vpp calicovpp/agent calicovpp/multinet-monitor.

Images that remain separate: calicovpp/vclsidecar, calicovpp/ctl

Directory Restructure:
├── cmd
│   ├── api-proxy
│   ├── calico-vpp-agent
│   ├── calicovppctl
│   ├── multinet-monitor
│   ├── replay-trace
│   └── vpp-manager
├── common.mk
├── docs
├── images
├── Makefile
├── pkg
│   ├── calico-vpp-agent
│   ├── config
│   ├── multinet-monitor
│   ├── vpplink
│   └── vpp-manager
├── scripts
├── test
├── vpp_build
└── yaml

Benefits:
- Single image tag to manage across all dataplane components
- Reduced bandwidth and storage requirements
- No risk of image version drift between components
- Simplified CI/CD pipeline
- Easier version management and releases

Signed-off-by: Aritra Basu <aritrbas+gh@cisco.com>
@aritrbas aritrbas force-pushed the abasu-unify-image branch from 072611e to 4b2333d Compare June 18, 2026 00:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants