test: comprehensive test-suite overhaul (100% coverage + E2E + compliance harness)#150
Merged
PatrickRitchie merged 18 commits intoMay 22, 2026
Merged
Conversation
ottobolyos
force-pushed
the
test/coverage-and-compliance
branch
7 times, most recently
from
70a65d2 to
b9a864a
Compare
PatrickRitchie
approved these changes
May 10, 2026
ottobolyos
force-pushed
the
test/coverage-and-compliance
branch
from
b9a864a to
c8cc1d4
Compare
ottobolyos
force-pushed
the
test/coverage-and-compliance
branch
from
c8cc1d4 to
adde8a5
Compare
ottobolyos
force-pushed
the
test/coverage-and-compliance
branch
from
adde8a5 to
5925345
Compare
ottobolyos
added a commit
to ottobolyos/mtconnect.net
that referenced
this pull request
May 18, 2026
…-proof HttpServerLoopbackBindingTests located ClientAgentCommunicationTests.cs via a hard-coded tests/IntegrationTests/ path segment, so it broke the moment the integration test project directory was renamed. Locate the file by recursive search under tests/ instead, asserting exactly one match so a stray duplicate is caught loudly rather than silently missed. The directory rename itself lives in PR TrakHound#150 (the branch that authored the bulk of the integration-test project's content); this guard is now agnostic to the directory name so it passes before and after that rename lands. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
ottobolyos
force-pushed
the
test/coverage-and-compliance
branch
from
5925345 to
adde8a5
Compare
Two per-version compliance matrices under docs/testing/, modeled on
the cppagent compliance-matrix pattern. Each enumerates the
DataItems / Components / enum values / configuration types that the
target MTConnect Standard version introduced or modified, with
status + pinned-test column populated from
tests/MTConnect.NET-Common-Tests/V2_6_V2_7/.
docs/testing/v2-6.md — v2.6 matrix:
- DataItems: AssetAdded, AssociatedAssetId, AssetChanged
description split.
- Components: CuttingTorchComponent, ElectrodeComponent.
- Constants: Version26, Max => Version26 (advanced to Version27
in the next release boundary).
docs/testing/v2-7.md — v2.7 matrix:
- DataItems: BindingState, Depth, FixtureAssetId, SwingAngle,
SwingDiameter, SwingRadius, TaskAssetId, WaterHardness.
- Components: PinTool, ToolHolder.
- Configurations: Axis / AxisDataSet, Origin / OriginDataSet,
Rotation / RotationDataSet, Scale / ScaleDataSet,
Translation / TranslationDataSet, plus their Abstract*
bases and the DataSet companion type.
- Constants: Version27, Max => Version27.
- XsdLoadStrict opt-in incantation inlined for the strict
schema-load test category.
Per-version compliance matrices are objective records of what the
library exposes (DataItem TypeIds, Component types, enum members,
constants) per MTConnect Standard version — readable independently
of any in-flight work.
The new public symbols introduced by this PR (Version26, Version27, the bumped Max getter) lacked <summary> XML-doc, so IntelliSense hovering them in a consumer codebase showed no description. Each new constant's summary briefly enumerates what its target MTConnect Standard version introduced, so a consumer choosing a version constant can pick the right one from hover alone. MTConnectModel and its Parse entry point — part of the SysML importer's public surface (the build/MTConnect.NET-SysML-Import generator's first call) — also lacked <summary> XML-doc on hover. Add a class-level summary explaining what the model represents and a method-level summary on Parse explaining input / output / null-return semantics, with a cref to the cross-package parent resolver it runs as a post-parse pass.
The committed tree lacked `docs/testing.md` and `docs/testing/workflows.md`. Land minimal stubs that link out to the per-version matrices, document the tier hierarchy (unit / compliance / E2E), and catalog the CI workflow + local harness scripts.
The SysML XMI carries a Profile:normative stereotype on every type that
declares its introducing MTConnect Standard version. The parser already
reads this stereotype via XmiDocument.NormativeIntroductions and the
package model classes (MTConnectDataItemType, MTConnectComponentType,
MTConnectCompositionType, MTConnectDataItemSubType) populate their
MinimumVersion property from it.
But MTConnectVersion.GetVersionEnum() — the helper that translates a
parsed System.Version into its MTConnectVersions.VersionNN enum
identifier for the Scriban template's {{ if minimum_version_enum }}
guard — only had cases through v2.3. Every type whose introducing
version was v2.4 or later returned null, the template's guard skipped
emission, and the regenerated .g.cs fell back to
DataItem.cs:DefaultMinimumVersion = MTConnectVersions.Version10. As a
result every v2.6 + v2.7 type added by this PR (and 9 v2.4-v2.5
holdover types) advertised "available since v1.0" on the wire instead
of their actual introducing version.
Wire-format consequence: a v1.0-configured agent serving a
v1.0-configured client would emit v2.6 / v2.7-only DataItems with
header version='1.0', violating client expectations of MTConnect
version compatibility.
Add cases for v2.4 + v2.5 + v2.6 + v2.7. Regen against the v2.7 XMI
produces the expected 24-file shape: each touched .g.cs gains exactly
one `public override System.Version MinimumVersion => MTConnectVersions.VersionNN;`
line in the slot the template already reserved. No structural drift
elsewhere.
Add a regression test asserting that ClientAgentCommunicationTests.GenerateDevicesXml writes to the path passed in `fileName`. The current implementation hard-codes "devices.xml" inside File.Create(...), so this test fails until the literal is replaced with the parameter.
Replace the hard-coded "devices.xml" literal in the File.Create(...) call with the fileName parameter that callers pass in. Without this, callers that supply a different filename silently observed the file written to the working directory under the literal name. Pinned by GenerateDevicesXmlTests.
Enumerates every public type in MTConnect.Devices, MTConnect.Assets, MTConnect.Observations, and MTConnect.Interfaces and emits three parametric NUnit cases per applicable type: - Type_can_be_constructed — Activator.CreateInstance(t) returns non-null for every concrete type with a parameterless ctor. - Type_round_trips_default_property_values — every public read/write auto-property accepts default(T) without throwing and round-trips on read-back; computed-property types are detected via the C# compiler- generated backing-field sentinel and skipped from the equality check. - Type_has_non_empty_description — every type whose .g.cs ships a DescriptionText const / property has a non-null, non-empty value. Adds two MinimumVersion / MaximumVersion sweeps that assert per-type overrides resolve to one of the constants advertised by MTConnectVersions; types that inherit the base virtual default of null are filtered out so the sweep does not flag "no annotation" as a defect. Pins FixtureAsset's empty DescriptionText as a known generator gap via KnownEmptyDescriptionTypes plus a sentinel test that asserts the emitted value is exactly the empty string — when the upstream XMI gains the description, the sentinel goes red and the entry moves out of the exclusion set. Test count: 42 -> 2193 in the Common suite (+2151 reflection-driven cases). Every case passes against the current regenerated tree.
ottobolyos
force-pushed
the
test/coverage-and-compliance
branch
from
adde8a5 to
3e37f39
Compare
ottobolyos
force-pushed
the
test/coverage-and-compliance
branch
from
3e37f39 to
62e7079
Compare
ottobolyos
force-pushed
the
test/coverage-and-compliance
branch
from
62e7079 to
6625c3a
Compare
ottobolyos
force-pushed
the
test/coverage-and-compliance
branch
from
6625c3a to
9cfb4d7
Compare
ottobolyos
force-pushed
the
test/coverage-and-compliance
branch
from
9cfb4d7 to
a621f29
Compare
Populates docs/testing/workflows.md with eight user-observable workflow rows (Probe, Current, Sample, Asset, SHDR adapter -> agent -> client, MQTT relay, cppagent parity, XML <-> JSON round-trip) and ships the fixtures that exercise each. New live E2E fixtures: - HttpProbeWorkflowTests — boots an in-process MTConnectAgentBroker on a free loopback port, seeds it from the existing devices-tpl.xml template, and asserts /probe returns a 200 envelope mentioning the device by uuid + name. Negative case asserts /<unknown>/probe does not return 500. - HttpAssetWorkflowTests — same shape as Probe but seeds a CuttingToolAsset against a registered device and asserts /assets + /asset/<id> return envelopes containing the asset id. New scaffold fixtures (gated out of the default sweep, surface the gap to reviewers without blocking CI): - MqttRelayWorkflowTests — workflow W06. [Trait Category RequiresDocker] + [Fact Skip] with reasons inline. Real implementation requires a Testcontainers MQTT-broker harness. - CppAgentParityWorkflowTests — workflow W07 in the L4_CrossImpl layer of the compliance project. [Category RequiresDocker] + [Explicit] with reasons inline. Real implementation requires docker-spun mtconnect/agent + the cross-impl whitelist file. Default-filter tests pass: 4 new IntegrationTests rows green; existing 3 IntegrationTests rows still green (7 total). Docker-gated rows stay out of the default sweep via the existing CI filter (Category!=RequiresDocker).
Use the canonical AmE spelling and tighten two explanatory comments in the regenerated-types reflection sweep. No behavior change.
Restores the MqttRelayWorkflowTests class with two real fixtures: - positive case spins eclipse-mosquitto:2.0.22 on a host-mapped port, attaches the production MqttRelay agent module to an in-process MTConnectAgentBroker, injects an observation, and asserts a raw MQTTnet subscriber receives a /Current/<uuid> payload carrying the injected sentinel. - negative case drops the subscriber before publish and asserts the observation remains in the agent's Streams response document so the MTConnect /current contract is preserved across consumer loss. Adds Testcontainers + MQTTnet to the IntegrationTests project; pins the Mosquitto image at 2.0.22 so wire-protocol behavior and default config remain reproducible across CI runs.
Restores the CppAgentParityWorkflowTests fixture with three real tests that boot mtconnect/agent:latest (resolved at fixture start to v2.7.0.7, digest sha256:8c7fb19c55fd588d7bda94710890a00a0d2c485caca147744dc27d445a11eb07) alongside an in-process MTConnect.NET broker against a shared minimal device fixture, request /probe, /current, /sample on each, and assert their canonical shapes are identical modulo the cross-impl whitelist. The whitelist captures runtime-only fields (header creation time, sender, instance id, asset buffer / count, observation timestamps and sequences, hash digests) plus the auto-injected cppagent Adapter and Agent components that have no MTConnect.NET counterpart. The fixture device declares ASSET_CHANGED / ASSET_REMOVED / ASSET_COUNT explicitly so neither implementation needs to auto-inject them under divergent id schemes. Failures emit a windowed diff around the first divergent character so reviewers see exactly which attribute or element broke parity. Pulls in Testcontainers 3.10 and an HTTP project reference into the compliance project.
- ClientAgentCommunicationTests: drop the F-code parenthetical from the loopback-binding rationale comment. - CppAgentParityWorkflowTests: drop the embedded build-date stamp from the agent:latest container-pin comment. - RegeneratedTypesCoverageTests: drop "today" from the two exclusion-list explainer comments. - SchemaLoadTests: drop "today" from the strict-load gap comment.
…ation-Tests + drop RequiresDocker filter - Rename tests/IntegrationTests/ -> tests/MTConnect.NET-Integration-Tests/. - Rename .csproj + add AssemblyName / RootNamespace = MTConnect.Tests.Integration. - Update every .cs namespace declaration accordingly (7 files). - Update MTConnect.NET.sln project entry. - Update tools/test.sh: E2E discovery path + drop the RequiresDocker exclusion so Docker-backed integration tests run; only XsdLoadStrict stays excluded by default. - Update tools/dotnet.sh E2E-mode trigger paths. - Update ClientAgentCommunicationTests embedded-resource lookup string for the new AssemblyName. - Update launchSettings.json profile name. - Update docs/testing.md + docs/testing/workflows.md to reference the new path. Isolate the integration suite's coverage instrumentation from its timing-critical hot path: - ClientAgentCommunicationTests drives the in-process MTConnectAgentBroker + embedded MTConnectHttpServer + MTConnectHttpClient stream + SHDR adapter as one sample-delivery hot path. Under coverlet IL instrumentation that path is slow enough that on a slow shared 2-CPU CI runner a Sample observation arrives after the test's 30000 ms assertion wait (no HTTP stream timeout; the sample is merely late) - the residual ubuntu-only failure on this branch. - Add tests/coverlet.integration.runsettings: the shared tests/coverlet.runsettings verbatim plus [MTConnect.NET-Common]* and [MTConnect.NET-HTTP]* added to Exclude. Those two are the dominant IL-instrumentation cost on the hot path and are already covered, faster, by MTConnect.NET-Common-Tests and MTConnect.NET-HTTP-Tests (the latter starts a real broker + HTTP server via AgentRunner and streams Sample/Current through the real HTTP client), so no net merged coverage is lost. MTConnect.NET-SHDR stays instrumented: the integration suite is its only runtime coverage. - VSTest run-settings precedence is the CLI settings flag over the RunSettingsFilePath property, so a solution-wide `dotnet test MTConnect.NET.sln` settings file cannot be overridden per project. Gate the integration project out of the solution-wide run via IsTestProject (false unless IntegrationCoverage=true) and run it as its own `dotnet test` step with its own settings file. - .github/workflows/dotnet.yml: split the test step into a solution-wide unit step (integration project skipped) and a dedicated integration step (-p:IntegrationCoverage=true + the integration settings file). - tools/test.sh: use the integration settings file + IntegrationCoverage for the integration project in the per-project loop and the E2E tier. - The shared tests/coverlet.runsettings and every other suite's coverage configuration are unchanged; the coverage report still covers the rest of the solution. Scoped isolation, not a global coverage disable.
…st runs The dedicated integration CI step ran `dotnet test --no-build` against the solution-built integration assembly, which is compiled as a non-test project (IsTestProject is false without IntegrationCoverage). Test discovery therefore found 0 tests and the step false-greened. Add an explicit `dotnet build -p:IntegrationCoverage=true` before the `--no-build` test run, and a post-step guard that parses the TRX <Counters total=...> and fails the job when it is 0 or unparseable; mirror the same guard on the solution-wide unit step. Make the integration-test category strategy consistent: the heavy in-process-server workflow fixtures (HttpProbe / HttpAsset / ConfigurationPolymorphicHttpProbe) are tagged Category=E2E and excluded from the default CI integration filter alongside RequiresDocker, so they no longer re-introduce timing fragility on a shared runner; ClientAgentCommunicationTests, GenerateDevicesXmlTests and HttpServerLoopbackBindingTests carry no category and still run. Bring tools/test.sh and tools/test.ps1 to parity: the integration project is owned by exactly one tier (the default loop), built with -p:IntegrationCoverage=true, run with the integration runsettings, and excludes Category=E2E/RequiresDocker by default; --e2e / -E2E widens that filter in place instead of a second run. test.ps1 also gains the new project path and the coverage flag it was missing. Scope MaxCpuCount=1 to the integration runsettings only; the shared runsettings returns to its prior state so the solution-wide unit run parallelizes. Stagger the integration test deadlines so the assertion wait is strictly inside the cancellation token, which is strictly inside the derived stream-read budget, so a marginally-late sample fails cleanly instead of racing teardown. Correct the integration runsettings rationale to state SHDR stays on the critical latency leg, fix a CS1574-risk see-cref, drop an internal cross-reference from a test comment, and reconcile the workflow catalog doc with the final category decisions.
ottobolyos
force-pushed
the
test/coverage-and-compliance
branch
from
a621f29 to
0458c41
Compare
Retarget MTConnect.NET-HTTP-Tests and its MTConnect.NET-Tests-Agents support project from net6.0 to net8.0 and repoint the broken ..\..\src\ project references at ..\..\libraries\. Add both projects to MTConnect.NET.sln so the solution-wide unit run discovers them. AgentRunner now hosts a real MTConnectAgentBroker + MTConnectHttpServer on an OS-assigned loopback port, with a Probe-readiness gate that blocks Start() until the server answers and a port-release gate on Stop() so concurrently constructed fixtures never race the socket bind. HttpClientFixture is the shared per-fixture lifecycle base.
Rewrite the four client fixtures to exercise MTConnect.NET-HTTP and MTConnect.NET-Common end to end against the embedded AgentRunner: Probe, Current and Sample run real HTTP request/response paths for XML and JSON across all devices, a single device and a device path filter; the streaming SampleClient opens a Current + Sample stream, pushes a live observation into the broker and asserts it is streamed back, covering the broker ingest/sequence/buffer path and the HTTP streaming per-chunk loop.
The integration runsettings drops [MTConnect.NET-Common]* and [MTConnect.NET-HTTP]* instrumentation to keep sample delivery off the coverlet-slowed path. Document that both assemblies are now genuinely exercised by MTConnect.NET-HTTP-Tests in the solution-wide unit run (real broker + HTTP server, Probe/Current/Sample/streaming), whose Cobertura ReportGenerator merges with the integration one, so the exclusion loses no net merged coverage.
…runners The shared AgentRunner host MTConnect.NET-Tests-Agents has no test adapter and discovers zero tests; it builds transitively as a dependency of MTConnect.NET-HTTP-Tests. Enumerating it in the local test loop only added a redundant no-op build/run and an empty TestResults dir. Both runners now exclude *-Tests-Agents.csproj, kept in parity.
ottobolyos
added a commit
to ottobolyos/mtconnect.net
that referenced
this pull request
May 21, 2026
git rerere captured the HTTP-Tests dependency-version resolution from the first merge attempt but not the 'modify/delete' resolution for the renamed IntegrationTests project, so the second refresh kept the old tests/IntegrationTests/ tree alongside the renamed tests/MTConnect.NET-Integration-Tests/. Both paths in tree confuses the Release build (CSC reads devices-tpl.xml from the old path even though it's only in the new path on disk). Integration-branch hygiene only — TrakHound#149 itself still references the old path against master, which is correct until TrakHound#150 merges first; the union here is just enforcing the post-TrakHound#150 layout for E2E.
ottobolyos
added a commit
to ottobolyos/mtconnect.net
that referenced
this pull request
May 21, 2026
First-pass conflict resolution kept HEAD's (TrakHound#149 deps-update) version of MTConnect.NET-HTTP-Tests.csproj which carried newer Test.Sdk + NUnit + coverlet pins but lost the second ProjectReference that test/coverage-and-compliance added when it lifted the per-test AgentRunner host into a shared MTConnect.NET-Tests-Agents project. The HTTP-Tests files (HttpClientFixture.cs, SampleStream.cs) consume `MTConnect.Tests.Agents.AgentRunner` from that project; without the ProjectReference the project no longer compiles. Re-add the Tests-Agents ProjectReference while keeping TrakHound#149's newer dep versions. Integration-branch hygiene only — the actual fix needs to land in TrakHound#150 when the deps bumps eventually reach it.
PatrickRitchie
approved these changes
May 22, 2026
PatrickRitchie
moved this from In Progress
to Ready to Merge
in MTConnect.NET-Development
github-project-automation
Bot
moved this from Ready to Merge
to Done
in MTConnect.NET-Development
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test-suite and compliance-harness work for MTConnect v2.6 / v2.7 support: regenerated-type coverage, end-to-end workflow fixtures, the integration-test project rename to the sibling-naming convention, and supporting generator and docs fixes.
Generator and library fixes
fix(sysml-import): emit aMinimumVersionoverride for the v2.4-v2.7 introducing-version annotations so every regenerated DataItem carries its correct introducing version instead of falling back toVersion10.fix(integration-tests):GenerateDevicesXmlhonors itsfileNameargument so a caller passing a non-default name writes to that path; pinned by a contract test.Test infrastructure
.g.cstypes: every type's constructors, properties, and per-subtype description methods exercised acrossMTConnectVersions.SupportedVersions.eclipse-mosquittovia Testcontainers; W07 cppagent-parity E2E (/probe,/current,/sample) againstmtconnect/agentvia Testcontainers.tests/IntegrationTests->tests/MTConnect.NET-Integration-Tests(directory,.csproj,RootNamespace/AssemblyName, namespaces, solution entry) to match thetests/MTConnect.NET-<Area>-Testssibling-naming convention.-p:IntegrationCoverage=true) undertests/coverlet.integration.runsettings, which excludesMTConnect.NET-Common/MTConnect.NET-HTTPfrom instrumentation and serializes the VSTest host (MaxCpuCount=1) so the in-process integration HTTP tests are not CPU-starved; the sharedtests/coverlet.runsettingslets the solution-wide unit run parallelize. Heavy in-process-server workflow fixtures are taggedCategory=E2Eand excluded from the default path alongsideRequiresDocker.Tooling and docs
tools/dotnet.{sh,ps1}/tools/test.{sh,ps1}: documented--dockerdual API,XsdLoadStrictsurfaced in--compliancehelp, heredoc help block, stale path references dropped.docs/testing/: top-level testing topic and workflow catalog; per-version compliance matrices for v2.6 and v2.7.MTConnectVersionsandParse; csprojDescriptionwording normalized across the SHDR-Adapter and SysML projects.