Repository navigation
task-based execution: add option to increase max_tries #2227
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
22 commits
Select commit
Hold shift + click to select a range
7002d5c
add first thoughts for taskdb update
atravitz 18c8811
add minimal working version
atravitz e59c107
pull in test fixtures from exorcist
atravitz bf3f8ca
remove duplicate
atravitz 7d63a64
add test task status db
atravitz eb2bacc
add test for current (incomplete) behavior
atravitz 63f32e7
cover typical use cases
atravitz efbd10a
make sure only the updated columns datetimes change
atravitz 52617cb
remove todo
atravitz d41476b
Update src/openfe/orchestration/exorcist_utils.py
atravitz 79786f6
add additional filter suggestion from ialibay
atravitz e54c0c1
add input validation for max_tries
atravitz 85a333f
expose API and add to reference docs
atravitz e722589
positive int type hint
atravitz 22014e1
add CLI update-task-db
atravitz a77a775
add docs
atravitz 247fcd7
make --max-tries option required (for now)
atravitz 71e06da
fix docstring
atravitz 14fd11b
Apply batched suggestions from code review
atravitz aec1b41
fix example
atravitz 9620cbb
update type checking
atravitz 11b77b4
Merge branch 'main' into feat/add_max_tries_update
IAlibay File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -21,6 +21,7 @@ Execution Commands | |
|
|
||
| run_task | ||
| status | ||
| update_task_db | ||
| quickrun | ||
|
|
||
| Results Gathering Commands | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| .. _cli_update_task_db: | ||
|
|
||
| ``openfe update-task-db`` | ||
| ========================= | ||
|
|
||
| .. click:: openfecli.commands.update_task_db:update_task_db | ||
| :prog: openfe update-task-db |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| **Added:** | ||
|
|
||
| * Added ``openfe.update_max_tries()`` for updating a ``TaskStatusDB`` (`PR #2227 <https://github.com/OpenFreeEnergy/openfe/pull/2227/>`_). | ||
|
|
||
| **Changed:** | ||
|
|
||
| * <news item> | ||
|
|
||
| **Deprecated:** | ||
|
|
||
| * <news item> | ||
|
|
||
| **Removed:** | ||
|
|
||
| * <news item> | ||
|
|
||
| **Fixed:** | ||
|
|
||
| * <news item> | ||
|
|
||
| **Security:** | ||
|
|
||
| * <news item> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -4,5 +4,6 @@ | |
| get_dependency_df, | ||
| get_task_df, | ||
| setup_task_campaign, | ||
| update_max_tries, | ||
| ) | ||
| from .worker import Worker | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,106 @@ | ||
| # This code is part of OpenFE and is licensed under the MIT license. | ||
| # For details, see https://github.com/OpenFreeEnergy/openfe | ||
|
|
||
| from pathlib import Path | ||
|
|
||
| import click | ||
|
|
||
| from openfecli import OFECommandPlugin | ||
| from openfecli.utils import write | ||
|
|
||
|
|
||
| def update_task_db_main( | ||
| task_db_path: Path, | ||
| max_tries: int, | ||
| ): | ||
| """ | ||
| Parameters | ||
| ---------- | ||
| task_db_path : pathlib.Path | ||
| Path to a task.db | ||
|
|
||
| max_tries: int | ||
| Value to update "max_tries" column to, for applicable rows | ||
|
|
||
|
|
||
| Example | ||
| ------- | ||
| >>> openfe status --task-db task.db | ||
|
|
||
| ┌─────────────────────────────┬──────────────────┬─────────────────────┬───────┬───────────┐ | ||
| │ task_id │ status │ last_modified │ tries │ max_tries │ | ||
| ├─────────────────────────────┼──────────────────┼─────────────────────┼───────┼───────────┤ | ||
| │ SetupUnit-d568ebe569b445c7… │ COMPLETED │ 2026-08-14 11:19:16 │ 1 │ 3 │ | ||
| │ SetupUnit-17dc05e0d79747e7… │ COMPLETED │ 2026-08-14 11:19:21 │ 1 │ 3 │ | ||
| │ SetupUnit-78321eda905c4c2c… │ COMPLETED │ 2026-08-14 11:19:22 │ 1 │ 3 │ | ||
| │ SetupUnit-0238f65b55044b1e… │ COMPLETED │ 2026-08-14 11:19:22 │ 1 │ 3 │ | ||
| │ MultiStateSimulationUnit-2… │ COMPLETED │ 2026-08-14 11:25:10 │ 1 │ 3 │ | ||
| │ MultiStateSimulationUnit-1… │ COMPLETED │ 2026-08-14 11:25:34 │ 1 │ 3 │ | ||
| │ MultiStateSimulationUnit-f… │ COMPLETED │ 2026-08-14 11:26:22 │ 1 │ 3 │ | ||
| │ MultiStateSimulationUnit-c… │ TOO_MANY_RETRIES │ 2026-08-14 11:26:24 │ 3 │ 3 │ | ||
| │ MultiStateAnalysisUnit-a72… │ COMPLETED │ 2026-08-14 11:26:51 │ 1 │ 3 │ | ||
| │ MultiStateAnalysisUnit-e44… │ COMPLETED │ 2026-08-14 11:26:26 │ 1 │ 3 │ | ||
| │ MultiStateAnalysisUnit-7e9… │ COMPLETED │ 2026-08-14 11:29:07 │ 1 │ 3 │ | ||
| │ MultiStateAnalysisUnit-72c… │ BLOCKED │ NaT │ 0 │ 3 │ | ||
| └─────────────────────────────┴──────────────────┴─────────────────────┴───────┴───────────┘ | ||
|
|
||
| >>> openfe update-task-db task.db --max-tries=6 | ||
| ┌─────────────────────────────┬──────────────────┬─────────────────────┬───────┬───────────┐ | ||
| │ task_id │ status │ last_modified │ tries │ max_tries │ | ||
| ├─────────────────────────────┼──────────────────┼─────────────────────┼───────┼───────────┤ | ||
| │ SetupUnit-d568ebe569b445c7… │ COMPLETED │ 2026-08-14 11:19:16 │ 1 │ 3 │ | ||
| │ SetupUnit-17dc05e0d79747e7… │ COMPLETED │ 2026-08-14 11:19:21 │ 1 │ 3 │ | ||
| │ SetupUnit-78321eda905c4c2c… │ COMPLETED │ 2026-08-14 11:19:22 │ 1 │ 3 │ | ||
| │ SetupUnit-0238f65b55044b1e… │ COMPLETED │ 2026-08-14 11:19:22 │ 1 │ 3 │ | ||
| │ MultiStateSimulationUnit-2… │ COMPLETED │ 2026-08-14 11:25:10 │ 1 │ 3 │ | ||
| │ MultiStateSimulationUnit-1… │ COMPLETED │ 2026-08-14 11:25:34 │ 1 │ 3 │ | ||
| │ MultiStateSimulationUnit-f… │ COMPLETED │ 2026-08-14 11:26:22 │ 1 │ 3 │ | ||
| │ MultiStateSimulationUnit-c… │ AVAILABLE │ 2026-08-14 11:27:24 │ 3 │ 6 │ | ||
| │ MultiStateAnalysisUnit-a72… │ COMPLETED │ 2026-08-14 11:26:51 │ 1 │ 3 │ | ||
| │ MultiStateAnalysisUnit-e44… │ COMPLETED │ 2026-08-14 11:26:26 │ 1 │ 3 │ | ||
| │ MultiStateAnalysisUnit-7e9… │ COMPLETED │ 2026-08-14 11:29:07 │ 1 │ 3 │ | ||
| │ MultiStateAnalysisUnit-72c… │ BLOCKED │ NaT │ 0 │ 6 │ | ||
| └─────────────────────────────┴──────────────────┴─────────────────────┴───────┴───────────┘ | ||
|
|
||
| """ | ||
|
|
||
| from exorcist import TaskStatusDB | ||
|
|
||
| from openfe.orchestration.exorcist_utils import update_max_tries | ||
|
|
||
| task_db = TaskStatusDB.from_filename(task_db_path) | ||
| update_max_tries(task_db, max_tries) | ||
|
|
||
|
|
||
| @click.command("update-task-db", short_help="Update the 'max_tries' column.") | ||
| @click.option( | ||
| "--task-db", | ||
| type=click.Path( | ||
| exists=True, | ||
| readable=True, | ||
| file_okay=True, | ||
| dir_okay=False, | ||
| path_type=Path, | ||
| ), | ||
| required=True, | ||
| help="Path to a TaskDB file.", | ||
| ) | ||
| @click.option( | ||
| "--max-tries", | ||
| type=click.IntRange(min=1), | ||
| required=True, | ||
| help="The positive integer value to assign as 'max_tries' for the updated columns.", | ||
| ) | ||
| # NOTE: this is named intentionally broad so that we can add "update_task_type" in a future version | ||
| def update_task_db(task_db: Path, max_tries: int): | ||
| """ | ||
| Update the 'max_tries' column for applicable tasks. COMPLETED tasks and tasks with tries > the input 'max-tries' will not be updated. | ||
|
|
||
| """ | ||
|
|
||
| # TODO: add loading bar | ||
| write("Loading task db ...") | ||
| update_task_db_main(task_db_path=task_db, max_tries=max_tries) | ||
|
|
||
|
|
||
| PLUGIN = OFECommandPlugin(command=update_task_db, section="Execution", requires_ofe=(1, 13)) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| import exorcist | ||
| import networkx as nx | ||
| import pytest | ||
| import sqlalchemy as sqla | ||
| from click.testing import CliRunner | ||
|
|
||
| from openfecli.commands.update_task_db import update_task_db | ||
|
|
||
| from ..utils import assert_click_success | ||
|
|
||
|
|
||
| @pytest.fixture | ||
| def simple_task_graph(): | ||
| task_graph = nx.DiGraph() | ||
| node_ids = ["HybridTopologySetupUnit-123", "HybridTopologySetupUnit-456"] | ||
|
|
||
| for id in node_ids: | ||
| task_graph.add_node(id) | ||
|
|
||
| return task_graph, node_ids | ||
|
|
||
|
|
||
| def test_update_task_db(simple_task_graph): | ||
| runner = CliRunner() | ||
| with runner.isolated_filesystem(): | ||
| task_graph, _ = simple_task_graph | ||
| db_path = "test.db" | ||
| db = exorcist.TaskStatusDB.from_filename(db_path) | ||
| db.add_task_network(task_graph, max_tries=2) | ||
| input_max_tries = 7 | ||
| result = runner.invoke( | ||
| update_task_db, ["--task-db", db_path, "--max-tries", str(input_max_tries)] | ||
| ) | ||
| assert_click_success(result) | ||
|
|
||
| # just make sure the table is actually modified. all edge cases etc. are handled in the Python API tests | ||
| with db.engine.connect() as conn: | ||
| new_max_tries = set(conn.execute(sqla.select(db.tasks_table.c.max_tries))) | ||
|
|
||
| assert new_max_tries == {(input_max_tries,)} |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.