[Improvement] Support rerun/retry history by linking separate original + rerun uploads

[timdegroot1996]

Problem

With --rerunfailed, a suite produces an original output.xml and a separate rerun output.xml. If only the merged result is uploaded, the dashboard sees one final status per test and loses which tests failed on the first attempt. Consistent first-attempt failures (that pass on retry) signal a real problem, not just flakiness - that signal is valuable and currently invisible.

Why not parse merged output

rebot --merge does not store a structured attempt history. The merged output.xml keeps only the final test body plus a message annotation (Old status: FAIL -> New status: PASS). That yields a single first-status flag at best, requires users to actually run the merge step, and gives no per-attempt timings or error detail. So this issue takes the linked-uploads approach instead.

Proposed direction

Upload the original run and each rerun as separate runs, and link them so the dashboard can group and diff attempts.

• Linking mechanism - a CLI/API field on upload, e.g. --rerun-of <run_start|alias> (or a tag convention like rerun:<parent-id>). The rerun upload references its parent so they form an attempt chain.
• Storage - add parent_run (or rerun_group) to the run/test data so linked attempts are queryable.
• Display - in the statistics / test graph, group linked attempts under one logical run; mark tests that failed on attempt 1 but passed later (flaky), and tests that failed every attempt (hard fail). Optional toggle "Show first-attempt failures".

Benefits over merge-parsing

• No dependency on users running rebot --merge.
• Full per-attempt data (status, timing, message) for every attempt, not just a flag.
• Natural fit with existing run-based model.

Open questions

• Link via explicit CLI/API field, or a tag convention?
• Reference parent by run_start, alias, or a user-supplied group id?
• How to render attempt chains in the existing graphs without cluttering non-rerun runs?

[HuntTheSun]

Hi Tim!

That is a great feature, me and my colleague were just talking about this yesterday :)

Link via explicit CLI/API field, or a tag convention?

I have no real horse in the race, but to me it would be nicer to have a dedicated field instead of a tag convention,
it would save the tag splitting and processing in order to check if it is related to sth.
It would also not hinder the implementation of #290

Reference parent by run_start, alias, or a user-supplied group id?

Either seems fine IMO, I would prefer alias, but thats just for my specific use case.
What do you mean with group-id? Would the user need to supply a group ID for the first run also in order to associate them or can this be added post-hoc (in case someone needs to unexpectedly rerun something but didn't pass a group-id in the first run)

How to render attempt chains in the existing graphs without cluttering non-rerun runs?

Good question :)

[timdegroot1996]

@HuntTheSun as you probably noticed I let some AI generate that feature idea based on requests from 2 separate requests of "some implementation" of rerun handling.

I think I will go with the second suggestion it did just like you say as well regarding the "linking". I think --rerun-of is kind of the direction I would like to go. I don't like the name yet, but I'll play around with the implementation for it.

It would also not hinder the implementation of #290

That's good to hear! I also think it doesn't overlap but if it does let me know so I can fix potential issues...

Either seems fine IMO, I would prefer alias, but thats just for my specific use case.
What do you mean with group-id? Would the user need to supply a group ID for the first run also in order to associate them or can this be added post-hoc (in case someone needs to unexpectedly rerun something but didn't pass a group-id in the first run)

AI slop regarding the "group-id" that could maybe be useful if you have more than 1 rerun but I think I'll start out with just going for a 1 rerun implementation. I think linking based on "run_start" makes the most sense. So adding a DB field "first_run_start"... which is the value of the first run. Which can then be used in the graphs.

How to render attempt chains in the existing graphs without cluttering non-rerun runs?

My initial idea is to add a toggle to the Test Statistics graph like Show first run results or something. Toggling this will show the initial status which can help you find the "flaky" tests after reruns, over multiple runs if you have added all reruns. It's gonna be a bit tricky implementing this but I'll see how it works out when I'm building it.

Thanks for the feedback regarding this! I'll keep you updated if I have progress 😄

[HuntTheSun]

AI slop regarding the "group-id" that could maybe be useful if you have more than 1 rerun but I think I'll start out with just going for a 1 rerun implementation. I think linking based on "run_start" makes the most sense. So adding a DB field "first_run_start"... which is the value of the first run. Which can then be used in the graphs

I though it over and you may have meant this already with your suggestion, maybe some grouping would be good, doing it like this:

all reruns share a column "original run" with the run_start of the original run, chronological hierarchy is calculated by the run_start of the reruns.
This way you dont build a "single-linked list" if you have reruns of reruns of reruns, python just need to check if the parent run passed via --rerun-of $run_start has sth set in the column original_run and if yes, apply the same value to "original_run" for the new run.
It may require some extra js processing to order the reruns chronologically since you have no direct "this is my next parent" data,
but saves having to traverse the rerun-series "pointer-jumping" style.
Idk what results in better performance across more use cases, just wanted to mention this.

[timdegroot1996]

rerun_dashboard.html

@HuntTheSun a status update: implemented in #332, but with a different approach than the issue proposed.

Why not linked uploads. The "why not parse merged output" section of the issue turned out to be wrong: rebot --merge keeps the full attempt history — every merge prepends the previous status and message to the test message, so a 3-attempt chain survives intact (only per-attempt timing is lost). Parsing that is a single processor change and one column, with no new CLI/API/listener surface, no --rerun-of, and a run stays one complete run. Linking separate uploads would have meant a rerun output showing up as a "5 of 100 tests" run in every graph and on the overview until every consumer learned to collapse it. So: run robot --rerunfailed + rebot --merge as the RF docs describe and upload only the merged output.xml.

What you get:

• tests.attempts column (JSON, first attempt → last) — also fixes the merge HTML that used to land in the test message / messages graph.
• Test section filter bar → Reruns select: Mark Reruns (default, blue border on re-executed tests, tooltip lists every attempt with its message), Final Result (as before), First Attempt (status of the first try — your "run 1 status").
• Applies to Test Statistics (timeline + scatter), Most Flaky / Recent Most Flaky (a recovery on rerun counts as a flip, so a test that only passes on retry every night finally shows up there), Most Failed / Recent Most Failed and Messages timelines.
• On the overview page you can see what was rerun, with extra information on the relevant run cards (Rerun: 49 (fixed 22)).
• Compare page has its own Reruns select; stat widgets Re-executed Tests / Recovered On Rerun / Failed All Attempts; attempts column on the Tables page.
• Docs: docs/reruns.md.

I attached an example output so feel free to have a look and let me know what you think!