Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 60 additions & 11 deletions docs/extension.md
Original file line number Diff line number Diff line change
Expand Up @@ -211,26 +211,75 @@ standalone Linux engine no address is both relay-reachable and off the LAN by de
resolves there to the bridge gateway, which a loopback-only listener cannot accept, while the wildcard exposes
the port on every host interface. (Docker Desktop has no such dilemma — its proxy reaches the host's loopback.)

`get-relay-info` resolves this: the provider asks, and Compose answers with one JSON line listing the networks
the relay would join — the dependents' networks, as selected for the relay deployment — each with the address a
locally-run endpoint should bind so the relay can reach it:
`get-relay-info` resolves this: sending it is itself the provider's declaration that it binds locally, and
Compose reacts by creating a **dedicated relay-link network** — an `internal:true` bridge, scoped to this one
provider service, joined by nothing but the relay container — then answers with one JSON line naming it
alongside the address a locally-run endpoint should bind so the relay can reach it:

```json
{ "type": "get-relay-info" }
```

```json
{"networks":[{"name":"myproject_default","gateway":"172.18.0.1"}]}
{"networks":[{"name":"myproject_database_relay","gateway":"172.20.0.1"}]}
```

Compose owns the platform knowledge behind that address: on a standalone engine it is the network's gateway —
an address the provider's host owns on that network's bridge, reachable from the relay (and from local
containers) but not from the LAN; under Docker Desktop it is `127.0.0.1` — the network lives inside the VM,
and the host's own loopback is, factually, where a host process is reached through the Desktop proxy. The
provider simply binds the announced gateway and publishes the endpoint exactly as bound: a routable address
passes to the relay untouched, a loopback one is announced as `localhost` (translated to
A provider backing a remote resource (an Amazon RDS instance, say) has no local endpoint to bind and simply
never sends `get-relay-info`: no relay-link network is created for it, and its `publish-endpoint` reports the
remote resource's own address unchanged.

```mermaid
sequenceDiagram
participant Compose
participant Provider
participant net as relay-link network<br/>(internal, per-service)
participant relay as relay container

rect rgb(235, 245, 255)
note over Compose,net: Provider running its service locally
Compose->>Provider: compose up --project-name=xx "database"
Provider->>Compose: json { "type": "get-relay-info" }
Compose->>net: create (or reuse) the dedicated<br/>relay-link network for "database"
Compose--)Provider: json {"networks":[{"name":"myproject_database_relay",<br/>"gateway":"172.20.0.1"}]}
Provider->>Provider: bind local endpoint to 172.20.0.1
Provider--)Compose: json { "type": "publish-endpoint",<br/>"message": "80=172.20.0.1:49152" }
Compose->>relay: deploy, join dependents' networks<br/>AND the relay-link network
end
```

```mermaid
sequenceDiagram
participant Compose
participant Provider
participant resource as remote resource<br/>(e.g. Amazon RDS)
participant relay as relay container

rect rgb(255, 245, 235)
note over Compose,resource: Provider backing a remote resource
Compose->>Provider: compose up --project-name=xx "database"
Provider->>resource: provision
note over Provider: no local endpoint to bind:<br/>get-relay-info is never sent
Provider--)Compose: json { "type": "publish-endpoint",<br/>"message": "80=resource.example.com:5432" }
Compose->>relay: deploy, join dependents' networks only<br/>(no relay-link network created)
end
```

Compose owns the platform knowledge behind the announced address: on a standalone engine it is the relay-link
network's IPv4 gateway — an address the provider's host owns on that dedicated bridge, reachable from the relay
(same-bridge local delivery) but joined by nothing else, so never reachable from the LAN or from a project's own
service networks; under Docker Desktop it is `127.0.0.1` — the network lives inside the VM, and the host's own
loopback is, factually, where a host process is reached through the Desktop proxy, so no dedicated network is
created there. The provider simply binds the announced gateway and publishes the endpoint exactly as bound: a
routable address passes to the relay untouched, a loopback one is announced as `localhost` (translated to
`host.docker.internal`). The `gateway` field may be absent when it cannot be resolved (exotic network drivers,
IPv6-only IPAM): fall back to a bind of your choice. Best-effort by design.
IPv6-only IPAM): fall back to a bind of your choice. Best-effort by design. The relay-link network is removed
along with the relay container when the service stops publishing endpoints, and by `down` like any other
project resource.

The `name` field is an opaque identifier, not a promise that a Docker network by that name exists: under Docker
Desktop it is the literal string `"desktop"`, which no `docker network inspect` or `NetworkConnect` call will
ever resolve. A provider must use it only for logging, never as an engine-level network reference — `gateway` is
the only field it needs to bind and publish correctly.

## Down lifecycle

Expand Down
6 changes: 6 additions & 0 deletions pkg/api/labels.go
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,12 @@ const (
// whether an existing relay can be kept on the next up. Commands that
// act on a service's process (exec, ...) refuse relay containers.
RelayLabel = "com.docker.compose.relay"
// RelayNetworkLabel marks the dedicated bridge network created for one
// provider-managed service's relay link — the sole channel between the
// relay container and the provider's own runtime (see get-relay-info).
// Never a project's user-declared network, and never joined by any
// dependent or sibling container: only the relay connects to it.
RelayNetworkLabel = "com.docker.compose.relay-network"
// SlugLabel stores unique slug used for one-off container identity
SlugLabel = "com.docker.compose.slug"
// ImageDigestLabel stores digest of the container image used to run service
Expand Down
6 changes: 6 additions & 0 deletions pkg/compose/compose.go
Original file line number Diff line number Diff line change
Expand Up @@ -543,6 +543,12 @@ func (s *composeService) actualNetworks(ctx context.Context, projectName string)

actual := types.Networks{}
for _, net := range networks.Items {
if _, ok := net.Labels[api.RelayNetworkLabel]; ok {
// a provider service's dedicated relay link, never a project's
// own declared network: it carries no NetworkLabel key at all,
// which would otherwise fold it into a bogus Networks[""] entry
continue
}
actual[net.Labels[api.NetworkLabel]] = types.NetworkConfig{
Name: net.Name,
Driver: net.Driver,
Expand Down
1 change: 1 addition & 0 deletions pkg/compose/down.go
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,7 @@ func (s *composeService) down(ctx context.Context, projectName string, options a
}

ops := s.ensureNetworksDown(ctx, project, limiter)
ops = append(ops, s.ensureRelayLinkNetworksDown(ctx, project)...)
Comment thread
ndeloof marked this conversation as resolved.

if options.Images != "" {
ops = append(ops, s.ensureImagesDown(ctx, project, options, limiter)...)
Expand Down
54 changes: 54 additions & 0 deletions pkg/compose/down_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,11 @@ func TestDown(t *testing.T) {
api.EXPECT().NetworkRemove(gomock.Any(), "abc123", gomock.Any()).Return(client.NetworkRemoveResult{}, nil)
api.EXPECT().NetworkRemove(gomock.Any(), "def456", gomock.Any()).Return(client.NetworkRemoveResult{}, nil)

// no relay-link network for this project (no provider service involved)
api.EXPECT().NetworkList(gomock.Any(), client.NetworkListOptions{
Filters: projectFilter(strings.ToLower(testProject)).Add("label", compose.RelayNetworkLabel),
}).Return(client.NetworkListResult{}, nil)

api.EXPECT().ContainerList(gomock.Any(), hookFilterListOpt()).Return(client.ContainerListResult{}, nil)

err = tested.Down(t.Context(), strings.ToLower(testProject), compose.DownOptions{})
Expand All @@ -117,6 +122,11 @@ func TestDown_ConcurrencyIsBoundedAcrossServices(t *testing.T) {
apiClient.EXPECT().ContainerList(gomock.Any(), gomock.Any()).
Return(client.ContainerListResult{}, nil) // removePreStartHookContainers lookup

// no relay-link network for this project (no provider service involved)
apiClient.EXPECT().NetworkList(gomock.Any(), client.NetworkListOptions{
Filters: projectFilter("prj").Add("label", compose.RelayNetworkLabel),
}).Return(client.NetworkListResult{}, nil)

apiClient.EXPECT().ContainerStop(gomock.Any(), gomock.Any(), gomock.Any()).
Return(client.ContainerStopResult{}, nil).
Times(numServices)
Expand Down Expand Up @@ -166,6 +176,11 @@ func TestDown_ImagePruningSharesConcurrencyBudgetAcrossOps(t *testing.T) {
{ID: "sha256:dangling2"},
}}, nil)

// no relay-link network for this project (no provider service involved)
apiClient.EXPECT().NetworkList(gomock.Any(), client.NetworkListOptions{
Filters: projectFilter("prj").Add("label", compose.RelayNetworkLabel),
}).Return(client.NetworkListResult{}, nil)

tracker := &peakConcurrencyTracker{}
apiClient.EXPECT().ImageRemove(gomock.Any(), gomock.Any(), gomock.Any()).
DoAndReturn(func(context.Context, string, client.ImageRemoveOptions) (client.ImageRemoveResult, error) {
Expand Down Expand Up @@ -238,6 +253,11 @@ func TestDown_NetworkAndImageRemovalShareConcurrencyBudget(t *testing.T) {
apiClient.EXPECT().NetworkRemove(gomock.Any(), "net1", gomock.Any()).
Return(client.NetworkRemoveResult{}, nil)

// no relay-link network for this project (no provider service involved)
apiClient.EXPECT().NetworkList(gomock.Any(), client.NetworkListOptions{
Filters: projectFilter("prj").Add("label", compose.RelayNetworkLabel),
}).Return(client.NetworkListResult{}, nil)

err := svc.down(t.Context(), "prj", compose.DownOptions{Project: project, Images: "local", RemoveOrphans: true})
assert.NilError(t, err)
assert.Assert(t, tracker.Peak() <= 2, "network- and image-removal ops must share the same concurrency budget, got peak %d", tracker.Peak())
Expand Down Expand Up @@ -286,6 +306,11 @@ func TestDownWithGivenServices(t *testing.T) {
api.EXPECT().NetworkInspect(gomock.Any(), "abc123", gomock.Any()).Return(client.NetworkInspectResult{Network: network.Inspect{Network: network.Network{ID: "abc123"}}}, nil)
api.EXPECT().NetworkRemove(gomock.Any(), "abc123", gomock.Any()).Return(client.NetworkRemoveResult{}, nil)

// no relay-link network for this project (no provider service involved)
api.EXPECT().NetworkList(gomock.Any(), client.NetworkListOptions{
Filters: projectFilter(strings.ToLower(testProject)).Add("label", compose.RelayNetworkLabel),
}).Return(client.NetworkListResult{}, nil)

api.EXPECT().ContainerList(gomock.Any(), hookFilterListOpt("service1")).Return(client.ContainerListResult{}, nil)

err = tested.Down(t.Context(), strings.ToLower(testProject), compose.DownOptions{
Expand Down Expand Up @@ -389,6 +414,11 @@ func TestDownRemoveOrphans(t *testing.T) {
}, nil)
api.EXPECT().NetworkRemove(gomock.Any(), "abc123", gomock.Any()).Return(client.NetworkRemoveResult{}, nil)

// no relay-link network for this project (no provider service involved)
api.EXPECT().NetworkList(gomock.Any(), client.NetworkListOptions{
Filters: projectFilter(strings.ToLower(testProject)).Add("label", compose.RelayNetworkLabel),
}).Return(client.NetworkListResult{}, nil)

api.EXPECT().ContainerList(gomock.Any(), hookFilterListOpt()).Return(client.ContainerListResult{}, nil)

err = tested.Down(t.Context(), strings.ToLower(testProject), compose.DownOptions{RemoveOrphans: true})
Expand Down Expand Up @@ -425,6 +455,11 @@ func TestDownRemoveVolumes(t *testing.T) {

api.EXPECT().VolumeRemove(gomock.Any(), "myProject_volume", client.VolumeRemoveOptions{Force: true}).Return(client.VolumeRemoveResult{}, nil)

// no relay-link network for this project (no provider service involved)
api.EXPECT().NetworkList(gomock.Any(), client.NetworkListOptions{
Filters: projectFilter(strings.ToLower(testProject)).Add("label", compose.RelayNetworkLabel),
}).Return(client.NetworkListResult{}, nil)

api.EXPECT().ContainerList(gomock.Any(), hookFilterListOpt()).Return(client.ContainerListResult{}, nil)

err = tested.Down(t.Context(), strings.ToLower(testProject), compose.DownOptions{Volumes: true})
Expand Down Expand Up @@ -505,6 +540,12 @@ func TestDownRemoveImages(t *testing.T) {
Return(client.ImageInspectResult{InspectResponse: image.InspectResponse{RepoTags: []string{"registry.example.com/remote-image-tagged:v1.0"}}}, nil).
AnyTimes()

// no relay-link network for this project (no provider service involved);
// down() runs twice in this test (--rmi=local then --rmi=all)
api.EXPECT().NetworkList(gomock.Any(), client.NetworkListOptions{
Filters: projectFilter(strings.ToLower(testProject)).Add("label", compose.RelayNetworkLabel),
}).Return(client.NetworkListResult{}, nil).AnyTimes()

localImagesToBeRemoved := []string{
"testproject-local-anonymous:latest",
"local-named-image:latest",
Expand Down Expand Up @@ -580,6 +621,11 @@ func TestDownRemoveImages_NoLabel(t *testing.T) {

api.EXPECT().ImageRemove(gomock.Any(), "testproject-service1:latest", client.ImageRemoveOptions{}).Return(client.ImageRemoveResult{}, nil)

// no relay-link network for this project (no provider service involved)
api.EXPECT().NetworkList(gomock.Any(), client.NetworkListOptions{
Filters: projectFilter(strings.ToLower(testProject)).Add("label", compose.RelayNetworkLabel),
}).Return(client.NetworkListResult{}, nil)

api.EXPECT().ContainerList(gomock.Any(), hookFilterListOpt()).Return(client.ContainerListResult{}, nil)

err = tested.Down(t.Context(), strings.ToLower(testProject), compose.DownOptions{Images: "local"})
Expand Down Expand Up @@ -1200,6 +1246,10 @@ func TestDownRemovesRetainedPreStartHookContainers(t *testing.T) {
api.EXPECT().NetworkList(gomock.Any(), client.NetworkListOptions{
Filters: projectFilter(strings.ToLower(testProject)),
}).Return(client.NetworkListResult{}, nil)
// no relay-link network for this project (no provider service involved)
api.EXPECT().NetworkList(gomock.Any(), client.NetworkListOptions{
Filters: projectFilter(strings.ToLower(testProject)).Add("label", compose.RelayNetworkLabel),
}).Return(client.NetworkListResult{}, nil)

// Hook container scan finds one retained pre_start container.
hookCtr := container.Summary{
Expand Down Expand Up @@ -1242,6 +1292,10 @@ func TestDownHookContainerRemovalFailureIsNonFatal(t *testing.T) {
api.EXPECT().NetworkList(gomock.Any(), client.NetworkListOptions{
Filters: projectFilter(strings.ToLower(testProject)),
}).Return(client.NetworkListResult{}, nil)
// no relay-link network for this project (no provider service involved)
api.EXPECT().NetworkList(gomock.Any(), client.NetworkListOptions{
Filters: projectFilter(strings.ToLower(testProject)).Add("label", compose.RelayNetworkLabel),
}).Return(client.NetworkListResult{}, nil)

// Hook scan finds one container.
hookCtr := container.Summary{
Expand Down
Loading
Loading