Skip to content

Test App listener's POST failure shifts the log of the test, so robotstatuschecker fails it #5360

Open

Description

@aaltat

Problem
A slow Test App makes an acceptance test fail even though its keywords passed.

atest/library/test_app_listener.py POSTs every Robot Framework event to the Test App with timeout=2. When the POST fails, it logs the error with logger.debug from start_test, so the message becomes the first item in the test's body, ahead of its first keyword. A test whose [Documentation] checks LOG 1.1:... then points at that message, and robotstatuschecker fails it.

Seen in
#5358, job "Test wheels on windows-latest for BrowserBatteries tests" (run 39054424650, job 117461288070): Robot reported 941 tests, 940 passed, 1 skipped, 0 failed, then DONE rc=(1). The status checker failed Test.01 Browser Management.Load Report.New Page Reports What The Stalled Page Was Still Loading with Locator '1' matches message and it cannot have child '1'. Body item 1 was:

DEBUG  Listener: Failed to POST event: start_test, id=s1-s2-s16-t2, name=New Page Reports What The Stalled Page Was Still Loading, status=None, server=localhost:52519 - ReadTimeout(... read timeout=2)

Any test with a LOG 1... check can fail this way. The cause is not in tools/ci_failures/known_causes.json yet.

To look at

  • How the listener should report a failed POST without adding a message to the test body (for example not through robot.api.logger from start_test), within the decisions in atest/CONTEXT.md and ADRs 0001 to 0003.
  • Whether a 2 s timeout is right for the Windows runners.
  • After the fix is merged on main, record the known cause in tools/ci_failures/known_causes.json.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions