feat(service-graph): agent assembles + uploads the east-west graph#415
Merged
Conversation
Process-global, bounded, idle-TTL'd accumulator (mirrors flow_signals): records observed src->dst:port edges (flow_count) and per-IP passive fingerprints (server JA4TS, client JA4T, client software) on the capture path; drains the slice for the ServiceGraphWorker to resolve + upload. Identity is resolved at drain, never on the hot path.
…'s slice Drains the observed buffer each tick, resolves IPs to workloads via the identity MMDB (host/<ip> fallback on-prem), and PUTs two contributions to service-graph-api: a declared slice (classification from identity + declared edges folded from the policy-edges artifact) and an observed slice (this agent's edges + passive fingerprints). Pure, tested projection; on-prem degrades gracefully (no artifacts -> observed-only, host refs). Links hippocampus with the graph feature (synapse-core Cargo.toml).
- Record observed east-west edges (both endpoints in HOME_NET) + passive fingerprints (client JA4T->src, server JA4TS->dst) at BOTH capture sites (kernel_pump XDP + AF_PACKET fallback), beside flow_signals::record. - platform.service_graph upload config (enabled/url/interval_secs). - Register the ServiceGraphWorker when configured (off in PCAP replay).
…ation) 0.0.1 lacked the graph feature + classification fields this worker needs. CI is green once hippocampus 0.0.2 publishes (gen0sec/hippocampus#8); dev builds via the local [patch] (uncommitted).
The agent owns only the observed layer and has no Kubernetes identity (it must run on-prem), so the service-graph worker now drains its observed edges + passive fingerprints and uploads a single `observed` slice keyed by raw IP. service-graph-api resolves those IPs to workload refs server-side against the operator's declared IP index (host/<ip> when unresolved), so the agent no longer folds identity/policy-edges itself. Drops the declared upload, identity-MMDB resolution and policy-edges fold from the worker; ServiceGraphConfig loses policy_edges_path.
The SYN-ACK (ja4ts) collector emits ServerToClient events with the server (this host) as src, so the previous src=client / dst=server mapping recorded the server fingerprint against the client and gated it on dst_home — leaving ja4ts empty in practice while client ja4t worked. Normalize each observation to client->server by PacketDirection: ja4t -> client, ja4ts -> server, edge client->server:server_port. Draw edges only from real flows (synthetic fingerprint carriers have src_port 0). Applied at both the XDP (kernel_pump) and AF_PACKET (lib.rs) capture sites.
On overlay clusters the egress SYN-ACK map is empty — the server's SYN-ACK is handled by the CNI datapath and never traverses the physical-NIC TC egress hook, so iter_egress() yields nothing while iter_egress_syn() (host-originated SYNs) works. The same outbound connection's SYN-ACK does arrive at ingress, so look the server's JA4TS up in the ingress SYN-ACK map (keyed by the connection's dst ip/port) and attach it to the egress-SYN event. That event is client->server, so the service graph attributes the fingerprint to the server node.
The TC-based SynAckCollector sees no SYN-ACKs on overlay/Cilium clusters
(the CNI's TC datapath owns the reply path), so server JA4TS was always
empty there. XDP runs at the driver, before the CNI's TC programs, so
inbound SYN-ACKs are visible.
Add an `xdp_synack_capture` program (separate __noinline, observe-only, no
blocking) that extracts the SYN-ACK fingerprint and stores it in new
server-keyed simple maps (tcp_synack_simple{,_v6}). Implement the
previously-stubbed `lookup_synack_fingerprint`/`_force` to read those maps,
which also revives the proxy's upstream-JA4TS lookup. Wire the service
graph's server-fingerprint recording to fall back to this lookup, and
attribute it to the server node by flow direction.
The per-event JA4TS lookup only fires when a flow event coincides with a captured SYN-ACK, so it misses periodic short-lived handshakes (the SYN-ACK lands in the map a beat after the egress-SYN event is processed). Add `drain_synack` (batch-iterate the server-keyed synack maps) and call it on a throttled cadence (every 30s) from the kernel pump, recording JA4TS for every internal server the node has handshaked with since startup — independent of flow-event timing.
On an overlay (Cilium VXLAN) cluster the physical NIC only sees encapsulated node-level traffic, so the observed graph is node-to-node. Add `capture.cni_interface` — a passive AF_PACKET SYN-only tap on the CNI tunnel netdev (e.g. cilium_vxlan), where decapsulated inner pod packets are visible. It never attaches XDP/TC (can't clobber the CNI datapath) and runs alongside the primary XDP capture; copies feed the same enrich pipeline so pod-to-pod edges land in the service graph and resolve to pod workloads via the operator's declared pod IPs. SYN-only keeps volume low; internal pod IPs are HOME_NET so they're never banned. Also fix the service-graph worker agent_id default to fall back to HOSTNAME (not a shared "synapse" literal) so each agent owns a distinct OBSERVED_EDGE slice instead of resync-clobbering the others.
Adds the production path for east-west pod-to-pod visibility on overlay (Cilium VXLAN) clusters: a tc/sched_cls program (tcx_overlay.bpf.c) attached via tcx ingress on the CNI tunnel netdev (e.g. cilium_vxlan) at BPF_TCX_FIRST — ahead of the CNI's own tcx program, so it sees the decapsulated inner pod packets first. Observer-only (always TC_ACT_UNSPEC), so it coexists with Cilium's tcx programs and never touches the datapath. It emits SYN/SYN-ACK connection-setup packets (raw TCP options included) over a ringbuf; the userspace consumer records pod-to-pod edges + JA4T/JA4TS into the service-graph buffer, resolved to pod workloads server-side via the operator's declared pod IPs (pod CIDR is HOME_NET, so never banned). `capture.cni_mode` selects the backend: `tcx` (default, this) or `af_packet` (the portable passive tap). Only active when `capture.cni_interface` is set.
Wires hippocampus's scan matcher into the agent so graph nodes get a service/product label, not just fingerprints. The tcx overlay program now also captures each flow's FIRST payload packet (the banner) — deduped via an LRU map, up to 256 bytes via bpf_skb_load_bytes — alongside the SYN/SYN-ACK connection events, over the same ringbuf (kind discriminator). The userspace consumer runs ProbeDb::identify(NULL/GetRequest/GenericLines, banner); a match labels the sender (the server) with service + "product version" via record_node_service. synapse-core links hippocampus with the scan feature (pcre2, vendored); the DB (nmap-service-probes, NPSL/GPL) is loaded from /var/lib/synapse at startup and is external data, never bundled in the crate.
…tes ladder (verifier)
…hadows /var/lib/synapse)
…er), not client requests
A workload can listen on several ports (postgres:5432 + a metrics http port). Key detected services by server port instead of collapsing to one, so they no longer overwrite each other. The buffer keeps a port->service map; drain emits the full per-port list plus the lowest-port primary for the node label. The passive matcher and active prober pass the server port through; the observed payload carries a services[] array per node.
The tcx tap recorded a flow only at SYN/SYN-ACK, so long-lived connections whose handshake we never saw (DB streaming replication, pooled channels) were invisible as edges. Add a kind=2 'established flow' event: any non-SYN packet emits the 5-tuple, deduped per-flow to once per 60s window via a timestamp LRU map (seen_flow). Userspace records the canonical client->server edge (lower port = server), no fingerprint. Proven live: database/core-1/2 -> core-0 streaming replication and the kafka inter-broker mesh now surface.
The overlay tap (cilium_vxlan) only sees east-west; pod egress to the internet (e.g. download-api -> S3) goes out the node's primary NIC, masqueraded, so it was invisible. Add a second tcx EGRESS program on the primary NIC at BPF_TCX_FIRST (before Cilium's BPF masquerade, so the pod IP is intact), filtered to pod-CIDR sources. Same event stream; the edge gate is relaxed to require only the client to be a pod, so external destinations are recorded. Config: capture.cni_egress_interface (name | 'auto' = default-route dev). External peers resolve to host/<ip> server-side and external/<reverse-dns> in the exporter (teal nodes). Best-effort: egress attach failure doesn't disable the east-west tap.
copy_raw_tcp_options{,_short} copied TCP options with a break-terminated,
variable-index byte loop. On stricter (newer-kernel) verifiers the index
becomes a runtime value and the verifier over-approximates the write to the
buffer end, rejecting the program load ("R3 min value is outside of the
allowed memory range" / "invalid write to stack") — which prevents the
XDP program from loading at all.
Replace the loop with a constant-size copy ladder: pick the largest fixed
block that fits within the packet bounds and __builtin_memcpy it, using a
constant length in each 'src + N <= data_end' check so the verifier can
prove the access. opt_total is always a multiple of 4, so out_len stays
exact. Mirrors the constant-size copy idiom already used for banner capture.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Moves east-west service-graph building into the Synapse agent (so it works on-prem, not just k8s). The agent observes flows + passive fingerprints natively, folds in the operator's k8s context (identity classification + declared edges), and uploads its slice to service-graph-api. It also fingerprints the services running behind each node and labels the graph with them.
Changes
Graph assembly + upload
security/service_graph.rs) — process-global, bounded, idle-TTL'd accumulator of observedsrc→dst:portedges + per-IP passive fingerprints (server JA4TS, client JA4T, service/product). Mirrorsflow_signals.ServiceGraphWorker(worker/service_graph.rs) — each tick: drain → resolve IPs via the identity MMDB (host/<ip>fallback on-prem) → build adeclaredslice (classification + policy-edges folded) + anobservedslice (edges + fingerprints) → two PUTs. Pure, tested projection.record_edge+ fingerprint recording at both sites (kernel_pumpXDP + AF_PACKET fallback), gated east-west (both endpoints in HOME_NET).platform.service_graph{enabled,url,interval_secs}; worker registered when configured (off in PCAP replay).IdentityInfodecodesinternet_exposed/control_plane(the operator now emits them) — in hippocampus Fixing dev container #8.Pod-to-pod visibility on overlay CNIs
firewall/bpf/tcx_overlay.bpf.c,firewall/overlay_capture.rs) — XDP is single-owner and ingress-only, so on overlay CNIs (e.g. Ciliumcilium_vxlan) it can't coexist or see pod-to-pod traffic. This adds atcx-attached observer (SEC("tcx/ingress"), multi-program-safe,BPF_F_BEFORE) on the CNI interface that emits connection events (SYN/SYN-ACK + TCP options for JA4TS) and the first payload per flow. Configcni_interface/cni_mode: tcx.Passive + active service detection
security/passive_scan.rs) — matches each server's captured banner (first payload on a flow) against Nmap'snmap-service-probesDB via the hippocampusscanmatcher, and recordsservice/producton the node. Server-gated: only banners from a SYN-ACK sender are matched, so client requests aren't mislabelled.scan-active) — long-lived servers never re-handshake, so passive capture can't see them. Candidate server endpoints (from observed traffic) are periodically connected to and the service is elicited + identified — covering postgres/redis/http/etc. that are already established.nmap-service-probesDB ships at/opt/synapse(the/var/lib/synapsemmdb volume would otherwise shadow it).Verification
synapse-core+synapse-appcompile; worker + buffer unit tests pass; fmt + clippy clean.bpf_skb_load_bytesladder (variable length is rejected by the verifier).Depends on
graphfeature,IdentityInfoclassification, and thescan/scan-activefeatures (thenmap-service-probesmatcher + active prober) must be published.