Skip to content

Fix Test Runner count to include specification items only #74

Description

@redcatbear

Problem

The IntelliJ Test Runner view currently mixes two different concepts:

  • Specification items are the logical trace results users expect to count.
  • Trace links are useful detail nodes beneath those results.

This produces an inflated total and, for clean traces with linked items, a green progress bar that can remain below 100%.

Current tree and counting

A source file is a suite. A specification item becomes expandable when it has link children, and each displayed trace link is currently treated as an additional test.

doc/requirements.md                         suite — not counted
└─ User can log in                          specification item — counted
   └─ ⊙← Login test                         trace link — counted

src/LoginTest.java                          suite — not counted
└─ Login test                               specification item — counted
   └─ ⊙→ User can log in                    trace link — counted

For this one requirement-to-test relationship, the current total is 4:

2 specification items
+ 2 displayed link children (the same relationship is visible from both ends)
= 4 reported tests

The current implementation calculates each item as 1 + links.size(), then reports that value to the IntelliJ result form.

There is an additional lifecycle mismatch:

linked specification item  → suite-started / suite-finished
trace link                 → test-started / test-finished

The IntelliJ runner does not increment completed-test progress for onSuiteFinished(...). Therefore a clean, expandable specification-item node raises the expected count but may not raise the completed count, leaving the green bar below 100%.

Desired tree and counting

Keep the useful visual hierarchy exactly as it is, but count only specification items.

doc/requirements.md                         suite — not counted
└─ User can log in                          specification item — counted
   └─ ⊙← Login test                         trace-link detail — not counted

src/LoginTest.java                          suite — not counted
└─ Login test                               specification item — counted
   └─ ⊙→ User can log in                    trace-link detail — not counted

Expected total for the example: 2 tests, and a clean trace completes at 100%.

Proposed approach

Use the IntelliJ SM runner's custom-progress support for the logical result count:

  1. Set custom total to the number of visible specification items.
  2. For each specification item, report custom started, failed (when the item itself or one of its visible links is defective), and finished.
  3. Retain the existing normal SM events for file suites, expandable item nodes, and link child nodes, so the tree, navigation, details, and failure presentation remain unchanged.

While a custom-progress category is active, IntelliJ ignores normal test-counter events. This separates item-only progress/counting from rendering the link detail nodes.

Acceptance criteria

  • The tree still shows source-file suites, specification-item entries, and trace-link child entries as today.
  • The Test Runner total equals the number of visible specification-item entries only.
  • Trace-link children do not increase the displayed total.
  • A clean trace with linked items shows 100% completion.
  • An item with a defective trace link contributes one failed logical result (the owning specification item).
  • Existing navigation and details for suites, items, and links remain available.

Investigation evidence

  • OftTraceTestTree.OftTraceItemNode#testCount() currently returns 1 + links.size().
  • OftTraceTestRunnerOutputPresenter#showTrace(...) sends that count via onTestsCountInSuite(...).
  • Linked item nodes use suite lifecycle events; link nodes use test lifecycle events.
  • IntelliJ's SMTestRunnerResultsForm#onSuiteFinished(...) does not update the finished-test counter; onTestFinished(...) does.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions