Skip to content

Feature/local execution backend - #14

Merged
toladnertum merged 6 commits into
mainfrom
feature/local-execution-backend
Sep 2, 2026
Merged

Feature/local execution backend#14
toladnertum merged 6 commits into
mainfrom
feature/local-execution-backend

Conversation

@pelin17

@pelin17 pelin17 commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds a new local execution backend for rapid local development. When developing or debugging tools, re-installing all packages from scratch on every submission is slow. The local backend skips provisioning and installation entirely, running benchmarks directly on the backend machine against a pre-installed local tool directory.

Changes

  • comp_eval_platform/compute/local.py — new LocalBackend class
  • comp_eval_platform/compute/base.py — registered LocalBackend in the backend registry
  • comp_eval_platform/compute/shell.pynode_exec and _ping route to local subprocess when IP is 127.0.0.1
  • comp_eval_platform/core/management/commands/init_settings.pylocal added as a valid execution_backend value
  • comp_eval_platform/settings.py — added LOCAL_TOOL_DIR and LOCAL_BENCHMARK_DIR config knobs

Behavior

Step local_docker / aws local
Assign Worker Docker/EC2 provision (~1-2 min) Instant (0s)
Install Tool git clone + install_tool.sh Skipped (0s)
Run Benchmark Via SSH to worker node Direct subprocess on backend

Configuration

EXECUTION_BACKEND=local
LOCAL_TOOL_DIR=/path/to/tool        # directory with install_tool.sh, run_instance.sh etc.
LOCAL_BENCHMARK_DIR=/path/to/benchmarks_repo

Proof of Local Backend Execution

As shown below, the Install Toolkit steps are skipped (0s) when using the new local backend.

ARCH tool submission VNN tool submission

Copilot AI lite review requested due to automatic review settings August 27, 2026 13:35

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@toladnertum
toladnertum merged commit 4dfde6b into main Sep 2, 2026
2 checks passed
@toladnertum

Copy link
Copy Markdown
Contributor

Looks very good, thanks for the detailed description!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants