Resourse check on event rebased - #137
Merged
Merged
Conversation
perk11
force-pushed
the
resourse-check-on-event-rebased
branch
3 times, most recently
from
July 28, 2026 01:50
3d63372 to
599981d
Compare
There was a problem hiding this comment.
Pull request overview
This PR updates large-model-proxy’s resource monitoring and management/status reporting to better represent asynchronous service lifecycle states (waiting/starting/running) and to make resource checks more event-driven, with corresponding UI and test updates.
Changes:
- Revamps
/statusoutput to exposestatus,waiting_connections,proxied_connections, and richer per-resource metrics (free,in_use,reserved_by_starting_services,total). - Reworks CheckCommand-based resource monitoring to support immediate rechecks via events/unpause signals and broadcasts.
- Updates management UI and expands/refactors tests to assert the new state machine and timing behavior more reliably (polling helpers instead of fixed sleeps in several places).
Reviewed changes
Copilot reviewed 9 out of 10 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| util_test.go | Adds new test helpers for resource/service state assertions and polling. |
| README.md | Documents CheckCommand behavior and new status/connection fields. |
| monitor_resources.go | Adds event-driven resource check triggering and change broadcasting helpers. |
| monitor_resources_test.go | Updates resource-check tests to use new status/resource fields and polling. |
| management-ui/index.html | Updates UI to display new status/connection/resource fields and new sort option. |
| management_api.go | Changes /status schema and populates new connection/resource fields. |
| management_api_test.go | Updates tests to use the new /status response types and helpers. |
| main.go | Implements waiting/proxied connection tracking, event-based resource rechecks, and new service state handling. |
| main_test.go | Adds regression tests for shutdown deadlocks, waiting-connection leaks, and resource-check edge cases. |
| config.go | Formatting-only adjustments for ResourceAvailable fields. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+68
to
+70
| } else { | ||
| free = resourceManager.resourcesAvailable[resourceName] | ||
| } |
| log.Printf("[%s] Failed to find a service to stop; will check every 100ms.", requestingService) | ||
| } else if iteration%logOutputIterationFrequency == 0 { | ||
| log.Printf("[%s] Failed to find a service to stop; continuing to check every 100ms.", requestingService) | ||
| resourceChangeServiceChannel := make(chan bool) |
|
|
||
| resourceChangeByResourceMutex *sync.Mutex //Also covers checkCommandFirstChangeByResourceChans | ||
| checkCommandFirstChangeByResourceChans map[string]map[string]chan struct{} | ||
| resourceChangeByResourceChans map[string]map[string]chan bool //true if amount is already updated, false if CheckCommands needs to run |
Comment on lines
+100
to
+106
| func (rm ResourceManager) broadcastResourceChanges(resources iter.Seq[string], recheckNeeded bool) { | ||
| resourceManager.resourceChangeByResourceMutex.Lock() | ||
| for resource := range resources { | ||
| rm.broadcastResourceChangeWhenResourceChangeByResourceMutexIsLocked(resource, recheckNeeded) | ||
| } | ||
| resourceManager.resourceChangeByResourceMutex.Unlock() | ||
| } |
Comment on lines
202
to
209
| let sortOrder = localStorage.getItem("sortOrder"); | ||
| if (sortOrder === null) { | ||
| sortOrder = "last_used"; | ||
| } else if (sortOrder === "proxied_connections") { | ||
| // proxied_connections is no longer a valid sort option but may exist in localStorage; this migration code can be removed in the future. | ||
| localStorage.setItem("sortOrder", "connections"); | ||
| sortOrder = "connections"; | ||
| } |
perk11
force-pushed
the
resourse-check-on-event-rebased
branch
from
July 28, 2026 02:44
599981d to
3981c4a
Compare
Closed
perk11
force-pushed
the
resourse-check-on-event-rebased
branch
10 times, most recently
from
August 1, 2026 02:06
9d2f678 to
a721bd5
Compare
A service is no longer started or kept waiting for a client that has already disconnected, and clients waiting on resources are unblocked the moment those resources free up rather than on the next poll. What changes in practice - A client that disconnects during startup — queued behind another start, waiting on a healthcheck, or blocked on resources — is dropped right away. - Contended resources (e.g. free VRAM reported by a check command) are monitored reactively: waiters wake the instant a service releases resources, and the check command only runs while something is actually waiting for it. - /status describes each service more precisely: stopped / waiting_for_resources / starting / running, with separate waiting vs. proxied connection counts, and resources broken down into Total / Reserved / In-use / Free. Implementation notes - Cancellation: manageMutex is now a channelMutex (a buffered-channel lock supporting select-based cancellation), and the client-disconnect signal threads through startService -> reserveResources / performHealthCheck / connectToService. - Monitoring: reserveResources no longer sleeps in a poll loop; service stops, the last connection closing, and monitor-measured changes broadcast on per-resource channels. The per-resource monitor runs its CheckCommand only while a waiter is registered and re-runs it on demand when a cached value may be stale; slow check commands run without holding the global serviceMutex. - Concurrency: runningServices is map[string]*RunningService; monitorProcess signals exitWaitGroup before acquiring serviceMutex (avoids a shutdown deadlock); idleTimer access and the post-cleanup nil check are serialized under serviceMutex; resourcesAvailable gains its own mutex. The process-exit test hook is gated behind the "testhooks" build tag.
… dashboard The dashboard now reports each service's lifecycle state and a per-resource breakdown of reserved/in-use/free capacity, making it easy to see at a glance what is running, what is still starting or starved for resources, and how much headroom is left. Dashboard - Service rows are colored by status (stopped / waiting_for_resources / starting / running), with waiting vs. proxied connections in separate columns — making it obvious when requests are queued behind a startup instead of silently counted as "active". - The resource section reports Reserved-by-starting / In-use / Free / Total instead of a single used/available pair; Total shows "N/A" for CheckCommand-backed resources whose capacity is only known dynamically. README: document the status, waiting_connections, and proxied_connections service fields and the resource fields (total, reserved_by_starting_services, in_use, free, usage_by_service), including the Total / "N/A" semantics.
perk11
force-pushed
the
resourse-check-on-event-rebased
branch
from
August 1, 2026 02:22
a721bd5 to
35e65bb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.