[scotwells]

Requested by Scot Schuchert-Wells

Fixes datum-cloud/compute#630. Related: datum-cloud/enhancements#921.

Problem

A newly started instance cannot resolve names for its first few seconds. The node's service route controller installs a new consumer's route only when it next reconciles that consumer's ServiceRoutePolicy. Every VPCAttachment, VPC, ServiceEndpoint and producer report event in the cell requeued every policy. The controller has one worker and each reconcile reads every attachment. With a few dozen policies, the queue never drained and each policy was rebuilt only every few seconds. Measured in a test cell, the queue held about 55 policies at all times, with a median wait of about 5 s and a p99 of about 10 s. A new consumer waited one full pass before its route existed, and lookups to the service address timed out until then.

Change

New mappers in internal/controller/serviceroute_requeue.go:

• A VPCAttachment event, old and new object, requeues only the policies that select the attachment as a consumer (selector and consumerVPCRef) or whose endpoint selects it as a producer (selector or attachmentRef). A policy the attachment left is requeued too, so it withdraws its route.
• A ServiceEndpoint or ServiceEndpointProducerReport event requeues only the policies routing to that endpoint.
• A VPC event requeues only the policies whose consumerVPCRef names it.
• A policy whose selector does not parse, or whose endpoint is not in the cache, is still requeued for every attachment event.
• Policy, BGPRouter and BGPVRFInstance events still requeue every policy, because conflicts and SIDs are decided across all policies.
• A conflict between policies can start or end through an event that requeues only one of them. The reconciler now tracks the conflict set and requeues every policy that enters or leaves it.

The router architecture doc describes the mapping.

Testing

• New unit tests cover:
  • consumer, producer-by-selector and producer-by-ref matching
  • consumerVPCRef filtering
  • old and new labels on update
  • the fallback for policies the mapper cannot judge
  • endpoint, report and VPC mapping
  • delete events
  • requeueing policies that enter or leave a conflict
• go build ./..., go vet, golangci-lint (0 issues) and go test -race pass in a Linux build container for ./internal/controller/, ./cmd/galactic-router/, ./pkg/serviceplan/ and the instrumentation tests.
• Not yet run in a lab. After rollout, check:
  • the serviceroutepolicy workqueue depth and wait drop to near zero
  • a fresh instance's first lookup to the service address succeeds