[Draft] Feature/add exposer - #65
Open
sbgaia wants to merge 95 commits into
Open
Conversation
Replace Poetry with uv as the package manager and dependency resolver: - pyproject.toml uses PEP 621 + PEP 735 dependency-groups - hatchling build backend - tox.ini uses tox-uv plugin (uv-venv-lock-runner) - CI workflow uses astral-sh/setup-uv - uv.lock replaces poetry.lock; requirements*.txt removed No library code changes; connector deps remain core (extras move in a follow-up phase).
- Move asyncua, aiomqtt, msgpack to [project.optional-dependencies] under extras: opcua, mqtt, all. - Introduce a connector plugin registry in machine_data_model/nodes/connectors/registry.py with a shared build_kwargs helper in _yaml_helpers.py. - Each connector subpackage (opcua, mqtt) self-registers at import: - As an available plugin when its optional deps are installed. - As an unavailable connector (with install hint) otherwise. - Move connector-specific YAML constructors/representers out of data_model_builder.py and data_model_dumper.py into per-protocol _yaml.py modules. The builder/dumper now consume the registry and no longer reference connector classes by name. - Adding a new connector requires only a new subpackage + an extra. - YAML files referencing unavailable connectors raise ImportError with the install hint instead of an opaque PyYAML error. - Split tests/builder/test_data_model_(builder|dumper).py into core (always runs) and connector-using (skipped in no-extras job). - New CI job test-no-extras verifies installation and the non-connector test suite without optional deps. - Update the existing CI test job to use --all-extras for Phase 2's new extras.
…er representer registration
- Add [tool.pyrefly] configuration in pyproject.toml with the strict preset and mypy-equivalent overrides; unmapped mypy flags listed as documented gaps. - Swap mypy for pyrefly in the dev dependency group. - tox.ini's type env now calls `pyrefly check ...`. - Remove mypy.ini. - Triage of new pyrefly findings: real bugs fixed inline, false positives silenced with `# pyrefly: ignore[...]` (logged in the audit handoff for follow-up).
Removes apply_cstyle.sh, check_typing.sh, gen_requirements.sh, and run_tox.sh — all hardwired to poetry/mypy and stale after the uv/pyrefly migration. radon.sh is updated to use 'uv run'. README sections that still referenced Poetry/mypy/run_tox.sh now point at uv and pyrefly.
…lpers The builder defined a verbatim copy of build_kwargs that was already implemented in machine_data_model.nodes.connectors._yaml_helpers (extracted during the Phase 2 connector refactor). Import the shared helper and drop the duplicate.
Both helpers are used by mqtt_connector.py despite the underscore prefix that marked them module-private. Drop the underscore so the cross-module usage is no longer flagged as private-access.
Replaces 89 'missing-override-decorator' suppress comments with typing_extensions.@OverRide decorators. Where the suppress was combined with 'bad-override-param-name', the decorator is added and the param suppress is repositioned inline on the def line so it stays adjacent. One remaining suppress on an attribute assignment in ObjectVariableNode is kept (decorators do not apply to attribute syntax).
The previous pattern logged 'msg' then 'exp' on separate error lines, losing the traceback. Switch to logger.exception so the full traceback is attached to the log record. Affects all eight Exception handlers in mqtt_connector.py (connect, disconnect, context-close, listener-cancel, listener-failure, publish, deserialize, subscription-callback).
ProtocolMng previously declared '_message_builder: MessageBuilder' as a mutable attribute and FrostProtocolMng narrowed it to FrostMessageBuilder on assignment — a Liskov violation suppressed with 'bad-override-mutable-attribute'. Parametrising ProtocolMng with a TypeVar bound to MessageBuilder makes the builder type part of the generic contract, so FrostProtocolMng(ProtocolMng[FrostMessageBuilder]) keeps the narrowed type soundly. The runtime isinstance() assertion is no longer needed.
CallMethodNode.execute (CC=16) and WaitConditionNode.execute (CC=17) were the two highest-complexity methods in the library. Extract focused helpers so each path is testable on its own: - CallMethodNode now defers to _resume_pending_composite, _register_pending_composite, _resolve_call_args, and _trace_step. execute drops to CC=8. - WaitConditionNode now defers to _evaluate_condition (operator dispatch via dict instead of an if/elif ladder), _get_or_create_subscription, _begin_wait, and _end_wait. execute drops to CC=5. No behaviour change; the 633 non-broker tests stay green.
The debug message interpolated 'path' twice, producing identical strings. Use resource.path for the local handle and the resolved remote path for the OPC UA target.
The exact == pin was inherited from the original Poetry config with no documented reason. Align with the rest of the dependency set by using a compatible-release range so patch updates flow through without re-releasing machine-data-model.
Adds the missing module-level docstrings flagged by the audit (data_model, mqtt_connector, mqtt_payload_codec, mqtt_remote_resource_spec, opcua_remote_resource_spec, remote_resource, frost_message_builder, message_builder). __main__.py now uses argparse, accepts an optional YAML path, and emits a helpful error when the path is missing instead of raising FileNotFoundError.
The class was being pulled in transitively via opcua_connector.py which only re-exports it. Import directly from the opcua_remote_resource_spec module so future cleanup of the re-export doesn't break this test.
If anyone narrows the no-extras CI --ignore path, the mqtt tests will now skip gracefully instead of failing with ImportError. Matches the existing pattern in test_data_model_builder_connectors.py.
Replaces three pyrefly suppressions with explicit narrowing: - bad-return on remote_node: wrap with _require_asyncua_node so the returned value carries the asserted asyncua.Node type. - bad-argument-type on get_node(node_id/parent_node_id): asyncua accepts these strings at runtime; cast(Any, ...) communicates that intent without silencing all type checks on the call site.
The two-line commented-out block referenced an unfinished template-variable resolution feature with no ticket. Deleting per the audit's 'just delete' option; reimplementation is out of scope for the post-migration cleanup.
ruff's commented-out-code detector flags every '# pyrefly: ignore[...]' comment as commented-out Python. Until ruff supports an external-pragma allow-list, the rule is more noise than signal in this codebase. The behaviour module has the most hits (17 of 18) but every ERA001 finding on the library tree is currently a false positive.
Library asserts are stripped under python -O, so the runtime guards they were providing silently disappeared. Convert 59 asserts across behavior/, builder/, data_model.py, nodes/, and protocols/frost_v1/ to explicit TypeError / RuntimeError raises so the invariants hold in optimised runs.
events.py shrinks from 1430 to 659 lines by leaning on the dataclass machinery: - TraceEvent becomes a kw_only=True dataclass with a default_factory for timestamp_ns and required event_type. - Each subclass declares only its own positional fields and overrides event_type with a kw_only default. The hand-written __init__ is gone. - _get_details() lives once on the base class and reflects via dataclass.fields(); per-subclass implementations are removed. - Convenience tracing functions now pass source/data_model_id as kwargs to the generated init.
Drops MqttConnector._async_connect complexity from B(6) to A(3) by pulling client construction and topic re-subscription into dedicated helpers. Combined with the earlier DI-2 logging cleanup, the module is no longer the lowest-MI outlier.
The no-extras CI job previously listed three --ignore paths; every new connector test would require updating that list. Replace with a pytest 'connector' marker auto-applied by tests/conftest.py based on file location, and switch CI to 'pytest -m "not connector"'.
Phase 2 introduced a self-registering connector plugin registry. The README pointed at no documentation and the only inline description lived in registry.py. Add docs/contributing/new-connector.md walking through subpackage layout, registration, optional extras, tests, and verification, and cross-link it from the README's contributing section.
docker was needed only by the MQTT integration tests but was being installed for every uv sync of the dev group (and every default tox env). Move it to a new 'integration' dependency group; tox envs that run the connector tests opt in explicitly. CI continues to install everything via 'uv sync --all-extras --all-groups'.
…bles dev's PR #64 added unsubscribe_from_node_changes as an abstract method on AbstractConnector. The exposer and mqtt test doubles predate it, so after rebasing onto dev they no longer satisfy the interface: _FakeConnector could not be instantiated and NullRemoteConnector's override lacked the @OverRide decorator required by pyrefly strict. Implement the method on both doubles.
Upgrade locked dependencies within existing pyproject constraints (aiohttp 3.13.5->3.14.1, msgpack 1.1->1.2, cryptography 48->49, pyrefly 1.0->1.1, tox/coverage/sphinx tooling, etc.). asyncua is deliberately held at 1.1.6: 1.1.7+ sends a ServerUri in CreateSessionRequest that the OPC-UA test server rejects with BadServerUriInvalid, failing the opcua integration suite. Needs investigation before un-pinning.
There was a problem hiding this comment.
Pull request overview
This draft PR modernizes the project’s packaging/tooling around uv + tox-uv + pyrefly, introduces an HTTP/WebSocket “exposer” subsystem (with coalesced backpressure semantics), and refactors connector handling into a plugin-style registry so optional connector extras can be cleanly absent in a “no-extras” CI job.
Changes:
- Add
ExposerManager,HttpExposer,WebSocketExposer, plus a sync→asyncNodeChangeCoalescer, with new tests and benchmarks. - Introduce a connector plugin registry and move connector YAML (constructor/representer) wiring behind discovery, improving optional-extra ergonomics and test collection.
- Migrate dev workflows from Poetry/mypy/requirements exports to
uv+tox-uv+pyrefly, updating CI and scripts accordingly.
Reviewed changes
Copilot reviewed 97 out of 103 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tox.ini | Switch tox envs to tox-uv runner and uv-managed dependency groups/extras. |
| tests/test_data_model.py | Tighten typing in subscription test helper state. |
| tests/protocols/frost_v1/test_frost_message_builder.py | Add Any-aware typing to args/kwargs fixtures. |
| tests/nodes/test_variable_node.py | Add typed lists and a new regression test for subscription dispatch snapshot semantics. |
| tests/nodes/measurement_unit/test_measure_builder.py | Replace type: ignore usage with explicit cast() to satisfy stricter type checking. |
| tests/nodes/connectors/test_registry.py | New tests for connector plugin registry behavior and discovery idempotence. |
| tests/nodes/connectors/test_abstract_async_connector.py | Add @override and casts to satisfy strict typing. |
| tests/nodes/connectors/opcua/conftest.py | Make container port extraction type-safe via cast(). |
| tests/nodes/connectors/opcua/init.py | Add OPC UA test helper utilities and reduce shared import-time dependency footprint. |
| tests/nodes/connectors/mqtt/test_mqtt_remote_resource_spec.py | Skip cleanly when aiomqtt missing; adjust import ordering. |
| tests/nodes/connectors/mqtt/test_mqtt_integration.py | Skip cleanly when aiomqtt/docker missing; reorganize imports; payload return simplification. |
| tests/nodes/connectors/mqtt/test_mqtt_data_model.py | Skip cleanly when aiomqtt missing; add @override and casts. |
| tests/nodes/connectors/mqtt/test_mqtt_connector.py | Skip cleanly when optional deps missing; reorganize imports for skip-at-top pattern. |
| tests/nodes/connectors/mqtt/conftest.py | Improve broker readiness wait with exponential backoff and better timeout diagnostics. |
| tests/nodes/connectors/mqtt/init.py | (Empty) Maintains package structure for pytest. |
| tests/nodes/connectors/init.py | Remove heavy imports so tests/nodes/connectors stays importable without extras. |
| tests/exposers/test_manager_lifecycle.py | New lifecycle tests for ExposerManager start/stop, threading, timeouts, executor shutdown semantics. |
| tests/exposers/test_coalescer.py | New unit tests for coalescer coalescing, pump dispatch, error isolation, post-close behavior. |
| tests/exposers/test_backpressure.py | New async test validating last-value-wins backpressure behavior under load. |
| tests/exposers/test_abstract_exposer.py | New tests for AbstractExposer ABC behavior. |
| tests/exposers/conftest.py | Skip exposer suite cleanly when aiohttp is not installed. |
| tests/exposers/init.py | (Empty) Maintains package structure for pytest. |
| tests/conftest.py | Centralize marker auto-application and optional-dep directory ignore-at-collection; register aiohttp pytest plugin when present. |
| tests/builder/test_data_model_dumper.py | Move connector-dependent dumper tests out of the dependency-free suite. |
| tests/builder/test_data_model_dumper_connectors.py | New connector-only dumper tests gated by importorskip. |
| tests/builder/test_data_model_builder.py | Remove connector build tests from dependency-free builder suite. |
| tests/builder/test_data_model_builder_connectors.py | New connector-only builder tests gated by importorskip. |
| tests/benchmarks/test_runner_smoke.py | New subprocess smoke tests covering benchmark runner CLI and JSON output paths. |
| tests/benchmarks/test_harness.py | New unit tests for benchmark harness percentile, aggregation, and scenario driving logic. |
| tests/benchmarks/test_baseline.py | New unit tests for baseline save/load and regression comparison logic. |
| tests/benchmarks/conftest.py | Skip benchmark-framework tests when aiohttp is missing. |
| tests/benchmarks/init.py | (Empty) Maintains package structure for pytest. |
| tests/behavior/test_bidirectional_linking.py | Remove a stale/unreachable type-ignore from an assertion. |
| setup.cfg | Removed legacy isort/flake8 config (now consolidated elsewhere). |
| scripts/run_tox.sh | Removed Poetry-based tox wrapper script. |
| scripts/radon.sh | Fix output dir name and switch to uv run radon. |
| scripts/gen_requirements.sh | Removed Poetry export script (requirements files removed). |
| scripts/check_typing.sh | Removed Poetry+mypy typing script (migrated to pyrefly/tox). |
| scripts/apply_cstyle.sh | Removed Poetry+ruff formatting wrapper script. |
| requirements.txt | Removed exported lock-style requirements file (uv-managed now). |
| requirements-dev.txt | Removed exported dev requirements file (uv-managed now). |
| README.md | Update contributor/dev instructions from Poetry to uv/tox-uv/pyrefly; link new connector guide. |
| pyproject.toml | Migrate to PEP 621 + Hatch build, define optional extras + dependency groups, configure Ruff + Pyrefly + pytest markers. |
| mypy.ini | Removed mypy config (replaced by pyrefly config in pyproject). |
| machine_data_model/tracing/tracing_core.py | Refactor TraceEvent defaults + details extraction; simplify boolean return. |
| machine_data_model/protocols/protocol_mng.py | Make ProtocolMng generic over concrete MessageBuilder type for narrower typing. |
| machine_data_model/protocols/message_builder.py | Add module docstring clarifying role of message builders. |
| machine_data_model/protocols/frost_v1/frost_message_builder.py | Replace assert validation with runtime exceptions for protocol version validation. |
| machine_data_model/nodes/subscription/variable_subscription.py | Add @override annotations on dunder methods. |
| machine_data_model/nodes/method_node.py | Replace asserts with typed exceptions; tighten callback typing; add @override; adjust return dict building logic. |
| machine_data_model/nodes/measurement_unit/measure_builder.py | Replace asserts/type-ignores with runtime checks and casts for strict typing. |
| machine_data_model/nodes/folder_node.py | Replace asserts with TypeError; add @override dunder annotations. |
| machine_data_model/nodes/data_model_node.py | Replace asserts with explicit exceptions; tighten register_children type validation. |
| machine_data_model/nodes/connectors/remote_resource.py | Add docstring and @override __repr__. |
| machine_data_model/nodes/connectors/registry.py | New connector plugin registry and discovery mechanism. |
| machine_data_model/nodes/connectors/opcua/opcua_remote_resource_spec.py | Add docstring and @override dunder annotations. |
| machine_data_model/nodes/connectors/opcua/opcua_connector.py | Typing cleanups (cast, @override), better debug logging, stricter path/type handling. |
| machine_data_model/nodes/connectors/opcua/_yaml.py | New OPC UA YAML constructors/representers using shared kwargs validation. |
| machine_data_model/nodes/connectors/opcua/init.py | Self-register OPC UA connector plugin or mark unavailable when deps missing. |
| machine_data_model/nodes/connectors/mqtt/mqtt_remote_resource_spec.py | Add docstring; promote topic helpers to public functions; add overrides. |
| machine_data_model/nodes/connectors/mqtt/mqtt_payload_codec.py | Add module docstring for codec responsibilities. |
| machine_data_model/nodes/connectors/mqtt/mqtt_connector.py | Add reconnect-with-backoff loop; improve exception logging; typing fixes; use public topic helpers. |
| machine_data_model/nodes/connectors/mqtt/_yaml.py | New MQTT YAML constructors/representers using shared kwargs validation. |
| machine_data_model/nodes/connectors/mqtt/init.py | Self-register MQTT connector plugin or mark unavailable when deps missing. |
| machine_data_model/nodes/connectors/_yaml_helpers.py | New helper to merge defaults with YAML and reject unexpected keys. |
| machine_data_model/exposers/websocket_exposer.py | New WebSocket exposer implementing subscriptions and coalesced broadcast with per-client timeouts. |
| machine_data_model/exposers/http_exposer.py | New HTTP exposer for node read/write and method invocation via aiohttp routes. |
| machine_data_model/exposers/exposer_manager.py | New manager owning aiohttp app, background loop thread, executor, and coalescer pump lifecycle. |
| machine_data_model/exposers/abstract_exposer.py | New ABC contract for exposers. |
| machine_data_model/exposers/_coalescer.py | New sync→async coalescing bridge with last-value-wins semantics and a drain pump. |
| machine_data_model/exposers/init.py | Public exposer package exports and security warning documentation. |
| machine_data_model/data_model.py | Add module docstring; type tweak for finalizer; replace assert with exception; add @override dunders. |
| machine_data_model/builder/data_model_dumper.py | Replace hard-coded connector representers with plugin-registry discovery and representer registration. |
| machine_data_model/behavior/remote_execution_node.py | Add @override annotations on various methods/dunders. |
| machine_data_model/behavior/control_flow_node.py | Add @override for __eq__; formatting/typing cleanup. |
| machine_data_model/main.py | Improve CLI ergonomics via argparse; return exit code; clearer usage/docs. |
| docs/contributing/new-connector.md | New contributor guide documenting connector registry pattern, extras, and test layout. |
| benchmarks/README.md | New documentation for running benchmark harness and baseline workflows. |
| benchmarks/bench_ws.py | New WS fanout + e2e latency benchmark scenarios. |
| benchmarks/bench_http.py | New HTTP read/write/method benchmark scenarios with concurrency sweeps. |
| benchmarks/bench_coalescer.py | New coalescer microbench scenarios (notify + drain). |
| benchmarks/baseline.json | Add checked-in baseline numbers for regression comparisons. |
| benchmarks/_harness.py | New harness primitives (samples, aggregation, percentile, run loop). |
| benchmarks/_baseline.py | New baseline load/save/compare utilities and regression reporting. |
| benchmarks/init.py | Package init for benchmark harness. |
| .github/workflows/ci.yml | Switch CI to uv; add a “no-extras” job verifying importability + non-connector test run. |
Comment on lines
+450
to
+454
| if isinstance(ret, Mapping): | ||
| raise RuntimeError("Return value cannot be a mapping.") | ||
| f" Received {ret} of type {type(ret)}." | ||
| ret = ret if isinstance(ret, list | tuple) else (ret,) | ||
| if not isinstance(ret, list) and not isinstance(ret, tuple): | ||
| ret = (ret,) |
Comment on lines
+82
to
+84
| tcp = connector.trust_store_certificates_paths | ||
| if tcp: | ||
| connector_dict["trusted_certificates_path"] = tcp |
| from machine_data_model.nodes.connectors.opcua.opcua_remote_resource_spec import ( # noqa: E501 | ||
| OpcuaRemoteResourceSpec, | ||
| ) | ||
| except ImportError: |
| from machine_data_model.nodes.connectors.mqtt.mqtt_remote_resource_spec import ( # noqa: E501 | ||
| MqttRemoteResourceSpec, | ||
| ) | ||
| except ImportError: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.