Skip to content

fix(media-server): enable shutdown draining in Compose - #3001

Open
07rjain wants to merge 9 commits into
CapSoftware:mainfrom
07rjain:split/media-drain-capacity
Open

07rjain wants to merge 9 commits into
CapSoftware:mainfrom
07rjain:split/media-drain-capacity

Conversation

@07rjain

@07rjain 07rjain commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Split out of #2580. Compose deployments now enable the media server's existing shutdown drain and allow enough time for cancellation webhooks and cleanup.

All three Compose variants set MEDIA_SERVER_SHUTDOWN_GRACE_MS=25000 and stop_grace_period: 60s. They forward both conversion tuning variables from .env; the self-hosting guide explains how to apply overrides for standard Compose, the template and Coolify.

Direct conversions retain main's default of one. Operators can explicitly raise MEDIA_SERVER_MAX_CONCURRENT_DIRECT_CONVERSIONS after measuring their deployment. Existing CPU/memory admission checks remain in effect. The audio-level integration regression occupies every configured direct slot.

Validation on 92835b607:

  • 87 tests passed: 16 audio-level integration, 37 video route and 34 lifecycle/drain tests.
  • Six separate processes verified unset, zero, negative, invalid and explicit conversion limits, slot admission, pressure rejection and idempotent release.
  • Scoped Biome, MDX compilation, diff whitespace checks and media-server bundling passed.
  • Full Docker builds remain for CI.

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; all previous findings are addressed.

Summary

Enables the media server’s existing shutdown drain in Compose deployments.

  • Compose stops let the media server drain before Docker ends it.
  • Compose operators can change how many video jobs run at once.

Reviews (4) · Last reviewed commit: "test(media-server): remove stale CPU def..." · Reviewed by Greptile

… the shutdown drain in compose

Direct /video/convert work defaulted to one conversion at a time on any machine. The default is now half the usable CPUs (container limit aware), between 1 and 4, and MEDIA_SERVER_MAX_CONCURRENT_DIRECT_CONVERSIONS still overrides it. The compose files set MEDIA_SERVER_SHUTDOWN_GRACE_MS to 25s under a 30s stop_grace_period so main's drain lets active work finish on docker compose stop.
@vercel

vercel Bot commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

@07rjain is attempting to deploy a commit to the Cap Software Inc Team on Vercel.

A member of the Team first needs to authorize it.

const MAX_DEFAULT_DIRECT_VIDEO_PROCESSES = 4;

export function getDefaultMaxConcurrentDirectVideoProcesses(
cpuCapacity: number = getContainerCpuLimit() || os.availableParallelism(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 CPU limits can overcount

If a container has a finite CPU quota larger than its assigned CPU set, getContainerCpuLimit() || os.availableParallelism() ignores the smaller available count. An eight-CPU quota with a two-CPU set selects four conversions instead of one, making conversions compete for the same CPUs. Use the smaller positive quota and os.availableParallelism() value before halving it, and test that combination.

Prompt To Fix With AI
This is a comment left during a code review.
Path: apps/media-server/src/lib/video-capacity.ts
Line: 16

Comment:
**CPU limits can overcount**

If a container has a finite CPU quota larger than its assigned CPU set, `getContainerCpuLimit() || os.availableParallelism()` ignores the smaller available count. An eight-CPU quota with a two-CPU set selects four conversions instead of one, making conversions compete for the same CPUs. Use the smaller positive quota and `os.availableParallelism()` value before halving it, and test that combination.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Comment thread docker-compose.yml Outdated
Comment on lines +52 to +55
stop_grace_period: 30s
environment:
# Finish active work for up to 25s on stop (below stop_grace_period).
MEDIA_SERVER_SHUTDOWN_GRACE_MS: "25000"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Docker cuts cleanup short

If a job outlasts the 25-second drain and its webhook is slow or unavailable, only five seconds remain before Docker kills the container. Cleanup starts after the drain, and abortAllJobs() can spend 16.5 seconds sending cancellation webhooks before the server reaches media and temporary-file cleanup. Docker can cut off both notifications and cleanup.

Increase stop_grace_period to cover both stages, or bound cleanup within the remaining time. Apply the same fix to docker-compose.template.yml and docker-compose.coolify.yml.

Prompt To Fix With AI
This is a comment left during a code review.
Path: docker-compose.yml
Line: 52-55

Comment:
**Docker cuts cleanup short**

If a job outlasts the 25-second drain and its webhook is slow or unavailable, only five seconds remain before Docker kills the container. Cleanup starts after the drain, and `abortAllJobs()` can spend 16.5 seconds sending cancellation webhooks before the server reaches media and temporary-file cleanup. Docker can cut off both notifications and cleanup.

Increase `stop_grace_period` to cover both stages, or bound cleanup within the remaining time. Apply the same fix to `docker-compose.template.yml` and `docker-compose.coolify.yml`.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Comment thread apps/web/content/docs/self-hosting.mdx Outdated
Comment on lines +111 to +116
The media server converts uploads in parallel based on the CPUs it can use: half the available CPUs, at least 1 and at most 4. It still turns work away when CPU or memory pressure is high. Override this with:

```bash
MEDIA_SERVER_MAX_CONCURRENT_DIRECT_CONVERSIONS=2 # direct /video/convert requests
MEDIA_SERVER_MAX_CONCURRENT_VIDEO_PROCESSES=4 # all video jobs
```

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Compose ignores these tuning settings

This page tells Compose users to put custom settings in .env, but none of the three Compose services forwards either tuning variable or loads that file with env_file. Adding the shown values to .env therefore leaves conversion limits unchanged. Show how to add them to the media-server service's environment mapping and recreate the container.

Prompt To Fix With AI
This is a comment left during a code review.
Path: apps/web/content/docs/self-hosting.mdx
Line: 111-116

Comment:
**Compose ignores these tuning settings**

This page tells Compose users to put custom settings in `.env`, but none of the three Compose services forwards either tuning variable or loads that file with `env_file`. Adding the shown values to `.env` therefore leaves conversion limits unchanged. Show how to add them to the media-server service's `environment` mapping and recreate the container.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

07rjain and others added 6 commits October 23, 2026 19:04
…cel webhooks and cleanup

A 25s drain plus webhooks for jobs aborted at shutdown (an in-flight attempt up to 5s, then 3 x 5s with 1.5s backoff) needs 46.5s before cleanup, more than the 30s stop_grace_period, so Docker could SIGKILL mid-cleanup. Raise stop_grace_period to 60s in all three compose files and document the budget.
@07rjain

07rjain commented Oct 24, 2026

Copy link
Copy Markdown
Contributor Author

@greptileai please re-review the latest changes.

@richiemcilroy

Copy link
Copy Markdown
Member

Thanks! The compose drain part looks good. Raising the default direct conversions from 1 to half the CPUs (up to 4) needs a measurement first, it's 4x on our boxes. Can you drop the default change or back it with numbers?

@07rjain 07rjain changed the title perf(media-server): size direct conversions from available CPUs and enable the shutdown drain in compose fix(media-server): enable shutdown draining in Compose Nov 5, 2026
@07rjain

07rjain commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

Note

🤖 gpt-6.1-sol responding on behalf of 07rjain

The direct conversion default is back to one in 92835b607583816ecbedc7f19804dde3a6939fcd, addressing the request to drop the unmeasured default increase. The explicit concurrency override and CPU/memory admission checks remain. The guide now recommends measuring deployment capacity before raising the limit.

The Compose drain, 60-second stop window and forwarded tuning variables remain. All 87 scoped tests, six process-level admission/configuration checks, scoped Biome, MDX compilation and bundling passed. Both architecture builds and self-hosting CI have passed on this head.

@greptileai please re-review this latest commit; the displayed 5/5 belongs to the previous head.

test("leaves an occupied conversion slot available to its current owner", async () => {
const slot = tryAcquireDirectVideoProcessSlot(() => true);
expect(slot).not.toBeNull();
// The default slot count follows the CPUs, so occupy every slot.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Comment describes the wrong default

This comment says the default slot count follows the CPUs, but DEFAULT_MAX_CONCURRENT_DIRECT_VIDEO_PROCESSES is now fixed at one. That gives future test maintainers the wrong explanation. The repository guide requires comments to capture needed, non-obvious context rather than describe the change. Remove this stale comment before merging to satisfy that requirement.

Suggested change
// The default slot count follows the CPUs, so occupy every slot.

Context Used: AGENTS.md (source)

Prompt To Fix With AI
This is a comment left during a code review.
Path: apps/media-server/src/__tests__/lib/audio-levels.integration.test.ts
Line: 242

Comment:
**Comment describes the wrong default**

This comment says the default slot count follows the CPUs, but `DEFAULT_MAX_CONCURRENT_DIRECT_VIDEO_PROCESSES` is now fixed at one. That gives future test maintainers the wrong explanation. The repository guide requires comments to capture needed, non-obvious context rather than describe the change. Remove this stale comment before merging to satisfy that requirement.

```suggestion

```

**Context Used:** AGENTS.md ([source](https://github.com/capsoftware/cap/blob/main/AGENTS.md))

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

@07rjain

07rjain commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

Note

🤖 gpt-6.1-sol responding on behalf of 07rjain

Removed the stale CPU-derived default comment in 64fe1dae4ab500af97dd796027bbdf0f19d658ad. The default remains one, and the regression still occupies every configured slot.

All 16 audio-level integration tests and the scoped Biome check passed. @greptileai please re-review this head.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants