Hwp supervisor: actions do not abort action in prgress by default, additional spin check by ykyohei · Pull Request #1110 · simonsobs/socs

ykyohei · 2026-09-30T16:05:59Z

Description

Implemeted two safeties.
Modified request_new_actions not to abort action in progress by default. Only abort_action will abort it.
Added additional spin check for disable_driver_board and power_cycle_gripper.

Motivation and Context

This resolves #1105 and #1106

How Has This Been Tested?

Tested on satp3 on 2026/9/29

disable_driver_board fails when hwp is spinning and idle, due to spin check.

2026-09-29T23:08:03+0000 pid_to_freq:9 Status is now "done".
2026-09-29T23:08:49+0000 start called for disable_driver_board
2026-09-29T23:08:49+0000 disable_driver_board:10 Status is now "starting".
2026-09-29T23:08:49+0000 disable_driver_board:10 Status is now "running".
2026-09-29T23:08:49+0000 Setting state: ControlState.DisableDriverBoard(driver_power_agent_type='synaccess', outlets=[4, 5])
2026-09-29T23:08:49+0000 Error updating state:
Traceback (most recent call last):
  File "/so/home/ykyohei/repos/socs/socs/agents/hwp_supervisor/agent.py", line 1470, in update
    with ensure_spin_stop(hwp_state, self.log):
  File "/so/shared-software/miniconda3/envs/so-base-py310/lib/python3.10/contextlib.py", line 281, in helper
    return _GeneratorContextManager(func, args, kwds)
  File "/so/shared-software/miniconda3/envs/so-base-py310/lib/python3.10/contextlib.py", line 103, in __init__
    self.gen = func(*args, **kwds)
  File "/so/home/ykyohei/repos/socs/socs/agents/hwp_supervisor/agent.py", line 1054, in ensure_spin_stop
    raise RuntimeError("HWP is spinning. Rotation safery check is failed.")
RuntimeError: HWP is spinning. Rotation safery check is failed.

2026-09-29T23:08:49+0000 Setting state: ControlState.Error(traceback='Traceback (most recent call last):\n  File "/so/home/ykyohei/repos/socs/socs/agents/hwp_supervisor/agent.py", line 1470, in update\n    with ensure_spin_stop(hwp_state, self.log):\n  File "/so/shared-software/miniconda3/envs/so-base-py310/lib/python3.10/contextlib.py", line 281, in helper\n    return _GeneratorContextManager(func, args, kwds)\n  File "/so/shared-software/miniconda3/envs/so-base-py310/lib/python3.10/contextlib.py", line 103, in __init__\n    self.gen = func(*args, **kwds)\n  File "/so/home/ykyohei/repos/socs/socs/agents/hwp_supervisor/agent.py", line 1054, in ensure_spin_stop\n    raise RuntimeError("HWP is spinning. Rotation safery check is failed.")\nRuntimeError: HWP is spinning. Rotation safery check is failed.\n', start_time=1790723329.6353574)
2026-09-29T23:08:50+0000 disable_driver_board:10 Completed with state: ControlState.Error(traceback='Traceback (most recent call last):\n  File "/so/home/ykyohei/repos/socs/socs/agents/hwp_supervisor/agent.py", line 1470, in update\n    with ensure_spin_stop(hwp_state, self.log):\n  File "/so/shared-software/miniconda3/envs/so-base-py310/lib/python3.10/contextlib.py", line 281, in helper\n    return _GeneratorContextManager(func, args, kwds)\n  File "/so/shared-software/miniconda3/envs/so-base-py310/lib/python3.10/contextlib.py", line 103, in __init__\n    self.gen = func(*args, **kwds)\n  File "/so/home/ykyohei/repos/socs/socs/agents/hwp_supervisor/agent.py", line 1054, in ensure_spin_stop\n    raise RuntimeError("HWP is spinning. Rotation safery check is failed.")\nRuntimeError: HWP is spinning. Rotation safery check is failed.\n', start_time=1790723329.6353574)
2026-09-29T23:08:50+0000 disable_driver_board:10 Status is now "done".

new action (disable_driver_board) is rejected when hwp is spinning up.

2026-09-29T23:05:34+0000 pid_to_freq:5 Status is now "starting".
2026-09-29T23:05:34+0000 pid_to_freq:5 Status is now "running".
2026-09-29T23:05:34+0000 Setting state: ControlState.PIDToFreq(target_freq=1.0, direction='0', freq_tol=0.05, freq_tol_duration=10.0)
2026-09-29T23:05:35+0000 Starting op: name=set_direction, kwargs={'direction': '0'}
2026-09-29T23:05:36+0000 Completed op: name=set_direction, success=True, kwargs={'direction': '0'}
2026-09-29T23:05:36+0000 Starting op: name=declare_freq, kwargs={'freq': 1.0}
2026-09-29T23:05:36+0000 Completed op: name=declare_freq, success=True, kwargs={'freq': 1.0}
2026-09-29T23:05:36+0000 Starting op: name=use_ext, kwargs={}
2026-09-29T23:05:39+0000 Completed op: name=use_ext, success=True, kwargs={}
2026-09-29T23:05:39+0000 Starting op: name=set_on, kwargs={}
2026-09-29T23:05:43+0000 Completed op: name=set_on, success=True, kwargs={}
2026-09-29T23:05:43+0000 Starting op: name=tune_freq, kwargs={}
2026-09-29T23:05:48+0000 start called for pid_to_freq
2026-09-29T23:05:49+0000 Completed op: name=tune_freq, success=True, kwargs={}
2026-09-29T23:05:49+0000 Starting op: name=send_command, kwargs={'command': 'off'}
2026-09-29T23:05:50+0000 Completed op: name=send_command, success=True, kwargs={'command': 'off'}
2026-09-29T23:05:50+0000 Setting state: ControlState.CheckInitialRotation(target_freq=1.0, freq_tol=0.05, freq_tol_duration=10.0, direction='0', check_wait_time=15.0, start_time=1790723150.3290014)
2026-09-29T23:05:58+0000 start called for disable_driver_board
2026-09-29T23:05:58+0000 disable_driver_board:6 Status is now "starting".
2026-09-29T23:05:58+0000 disable_driver_board:6 Status is now "running".
2026-09-29T23:05:58+0000 disable_driver_board:6 DisableDriverBoard is rejected as CheckInitialRotation is in progress.
2026-09-29T23:05:58+0000 disable_driver_board:6 Status is now "done".
2026-09-29T23:06:05+0000 Starting op: name=send_command, kwargs={'command': 'on_2'}
2026-09-29T23:06:06+0000 Completed op: name=send_command, success=True, kwargs={'command': 'on_2'}
2026-09-29T23:06:09+0000 Starting op: name=send_command, kwargs={'command': 'off'}
2026-09-29T23:06:10+0000 Completed op: name=send_command, success=True, kwargs={'command': 'off'}

new action (disable_driver_board) is rejected when hwp is spinning down.

2026-09-29T23:09:48+0000 start called for brake
2026-09-29T23:09:48+0000 brake:11 Status is now "starting".
2026-09-29T23:09:48+0000 brake:11 Status is now "running".
2026-09-29T23:09:48+0000 Setting state: ControlState.Brake(freq_tol=0.05, freq_tol_duration=10.0, brake_voltage=10.0, max_brake_duration=None)
2026-09-29T23:09:48+0000 Starting op: name=send_command, kwargs={'command': 'off'}
2026-09-29T23:09:49+0000 Completed op: name=send_command, success=True, kwargs={'command': 'off'}
2026-09-29T23:09:49+0000 Starting op: name=get_state, kwargs={}
2026-09-29T23:09:51+0000 Completed op: name=get_state, success=True, kwargs={}
2026-09-29T23:09:51+0000 pid state: {'healthy': True, 'current_freq': 0.994, 'target_freq': 1.0, 'direction': 0}
2026-09-29T23:09:51+0000 Starting op: name=set_direction, kwargs={'direction': '1'}
2026-09-29T23:09:53+0000 start called for disable_driver_board
2026-09-29T23:09:53+0000 disable_driver_board:12 Status is now "starting".
2026-09-29T23:09:53+0000 disable_driver_board:12 Status is now "running".
2026-09-29T23:09:53+0000 Completed op: name=set_direction, success=True, kwargs={'direction': '1'}
2026-09-29T23:09:53+0000 Starting op: name=tune_stop, kwargs={}
2026-09-29T23:09:59+0000 Completed op: name=tune_stop, success=True, kwargs={}
2026-09-29T23:09:59+0000 Starting op: name=ign_ext, kwargs={}
2026-09-29T23:10:01+0000 Completed op: name=ign_ext, success=True, kwargs={}
2026-09-29T23:10:01+0000 Starting op: name=set_v, kwargs={'volt': 10.0}
2026-09-29T23:10:04+0000 Completed op: name=set_v, success=True, kwargs={'volt': 10.0}
2026-09-29T23:10:04+0000 Starting op: name=set_on, kwargs={}
2026-09-29T23:10:08+0000 Completed op: name=set_on, success=True, kwargs={}
2026-09-29T23:10:18+0000 Setting state: ControlState.WaitForBrake(freq_tol=0.05, freq_tol_duration=10.0, min_freq=0.5, max_duration=None, start_time=1790723418.4707317)
2026-09-29T23:10:18+0000 disable_driver_board:12 DisableDriverBoard is rejected as WaitForBrake is in progress.
2026-09-29T23:10:18+0000 disable_driver_board:12 Status is now "done".
2026-09-29T23:10:19+0000 Starting op: name=get_state, kwargs={}
2026-09-29T23:10:21+0000 Completed op: name=get_state, success=True, kwargs={}
2026-09-29T23:10:21+0000 pid state: {'healthy': True, 'current_freq': 0.883, 'target_freq': 0.0, 'direction': 1}
2026-09-29T23:10:26+0000 Starting op: name=get_state, kwargs={}
2026-09-29T23:10:27+0000 Completed op: name=get_state, success=True, kwargs={}
2026-09-29T23:10:27+0000 pid state: {'healthy': True, 'current_freq': 0.838, 'target_freq': 0.0, 'direction': 1}
2026-09-29T23:10:28+0000 Starting op: name=get_state, kwargs={}
2026-09-29T23:10:30+0000 Completed op: name=get_state, success=True, kwargs={}
2026-09-29T23:10:30+0000 pid state: {'healthy': True, 'current_freq': 0.815, 'target_freq': 0.0, 'direction': 1}
2026-09-29T23:10:35+0000 Starting op: name=get_state, kwargs={}
2026-09-29T23:10:37+0000 Completed op: name=get_state, success=True, kwargs={}

Checklist:

My code follows the code style of this project.
I have updated related documentation or an update to the documentation is not required.

AI Usage Disclosure

Not used

…r_spin_check

mhasself

Thanks -- couple of notes.

mhasself · 2026-10-01T10:10:46Z

                    self.run_and_validate(clients.driver_iboot.set_outlet, kwargs=kw)
-                for outlet in state.outlets:
-                    set_outlet_state(outlet, False)
+                with ensure_spin_stop(hwp_state, self.log):


I think this doesn't need to be a context manager -- I think this is the equivalent of just running an undecorated ensure_spin_stop(...).

A context manager would normally be used to express the following:

allocate / obtain / lock some resource

do some stuff that needs that resource

free / release / unlock that resource

But here you're just using it to raise an error if some condition is not met prior to "do some stuff" -- there's no cleanup step.

mhasself · 2026-10-01T10:16:54Z

+        if action is None:
+            requested = state.__class__.__name__
+            cur = self.control_state_machine.action
+            cur_state_type = cur.cur_state_info.state_type
+            return False, f"{requested} is rejected as {cur_state_type} is in progress."


These lines are repeated a bunch of times ... can you deduplicate it? E.g.

if action is None: return False, self._rejected_action_message(state)

There's actually a race condition here, though (state could go idle right before you generate the error message). It would be better to create this message inside the request_new_action function, where lock is held, and return message from there.

action, err_msg = self.control_state_machine.request_new_action(state) if action is None: return False, err_msg

ykyohei · 2026-10-01T14:48:56Z

Thanks for the suggestions @mhasself
I think I will find another time to test this, just to make sure.

ykyohei added 4 commits September 29, 2026 14:53

Add spin check

6695f46

Do not abort action in progess

d4a0d61

Merge remote-tracking branch 'origin/hwp_no_abort' into hwp_superviso…

1bcc812

…r_spin_check

fix typo

9d34123

ykyohei added enhancement New feature or request. agent: hwp supervisor labels Sep 30, 2026

ykyohei requested review from BrianJKoopman and bbixler500 September 30, 2026 16:06

mhasself reviewed Oct 1, 2026

View reviewed changes

ykyohei added 2 commits October 1, 2026 10:33

address comments

11aa2f0

fix

0d7073c

Navigation Menu

Hwp supervisor: actions do not abort action in prgress by default, additional spin check - #1110

Hwp supervisor: actions do not abort action in prgress by default, additional spin check#1110
ykyohei wants to merge 6 commits into
mainfrom
hwp_supervisor_spin_check

ykyohei commented Sep 30, 2026 •

edited

Loading

Uh oh!

mhasself left a comment

Uh oh!

mhasself Oct 1, 2026

Uh oh!

mhasself Oct 1, 2026

Uh oh!

mhasself Oct 1, 2026

Uh oh!

ykyohei commented Oct 1, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

ykyohei commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Motivation and Context

How Has This Been Tested?

Checklist:

AI Usage Disclosure

Uh oh!

mhasself left a comment

Choose a reason for hiding this comment

Uh oh!

mhasself Oct 1, 2026

Choose a reason for hiding this comment

Uh oh!

mhasself Oct 1, 2026

Choose a reason for hiding this comment

Uh oh!

mhasself Oct 1, 2026

Choose a reason for hiding this comment

Uh oh!

ykyohei commented Oct 1, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ykyohei commented Sep 30, 2026 •

edited

Loading