Prevent exponential compile time for reconverging workflows

[1fanwang]

Why are the changes needed?

Reconverging acyclic workflows can take exponentially longer to compile because cycle detection revisits checked descendants.

Before, each reconvergence multiplied the remaining traversal. After, each node is expanded once.

What changes were proposed in this pull request?

The traversal checks the active stack first, then skips completed nodes. Back edges still report cycles. A regression test counts expansions.

How was this patch tested?

The regression fails against the exact parent and passes at this PR's head.

$ git worktree add /tmp/flyte-pre-fix d8b6a3b49ec3ecdb7c2f3d6b9c2fa8da5635b850
$ git show 689b446:flytepropeller/pkg/compiler/utils_test.go > /tmp/flyte-pre-fix/flytepropeller/pkg/compiler/utils_test.go
$ (cd /tmp/flyte-pre-fix/flytepropeller && go test ./pkg/compiler -run '^TestDetectCycle/Reconverging$' -count=1)
$ git worktree remove /tmp/flyte-pre-fix
$ cd flytepropeller
$ go test ./pkg/compiler -run '^TestDetectCycle/Reconverging$' -count=1
$ go test ./pkg/compiler -count=1
$ go test ./pkg/compiler -race -coverprofile=/tmp/flyte-compiler-race-cover.out -covermode=atomic -count=1

Raw logs

Parent implementation with the regression test applied:

--- FAIL: TestDetectCycle (0.00s)
    --- FAIL: TestDetectCycle/Reconverging (0.00s)
        expected: map[string]int{"leaf":1, "left":1, "right":1, "root":1, "shared-a":1, "shared-b":1}
        actual  : map[string]int{"leaf":4, "left":1, "right":1, "root":1, "shared-a":2, "shared-b":2}
FAIL
FAIL	github.com/flyteorg/flyte/flytepropeller/pkg/compiler	0.882s
FAIL

This patch:

ok  	github.com/flyteorg/flyte/flytepropeller/pkg/compiler	1.175s
ok  	github.com/flyteorg/flyte/flytepropeller/pkg/compiler	1.068s
ok  	github.com/flyteorg/flyte/flytepropeller/pkg/compiler	4.919s	coverage: 82.0% of statements

At 69e00fb, the race run reports 82.0% compiler package coverage and 100% coverage for utils.go. codecov/project and codecov/patch both pass.

The unrelated sandbox job failed during go mod download from proxy.golang.org (stream ID 433; INTERNAL_ERROR; received from peer).

Labels

• fixed

Setup process

N/A.

Screenshots

N/A.

Check all the applicable boxes

• Documentation is not applicable because no public API or configuration changes.
• All new and existing compiler tests passed.
• All commits are signed-off.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 57.28%. Comparing base (d8b6a3b) to head (69e00fb).

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #7950   +/-   ##
=======================================
  Coverage   57.28%   57.28%           
=======================================
  Files         931      931           
  Lines       58311    58313    +2     
=======================================
+ Hits        33403    33405    +2     
- Misses      21849    21851    +2     
+ Partials     3059     3057    -2     

Flag	Coverage Δ
unittests-datacatalog	53.51% <ø> (ø)
unittests-flyteadmin	53.23% <ø> (ø)
unittests-flytecopilot	48.05% <ø> (ø)
unittests-flytectl	64.11% <ø> (-0.05%)	⬇️
unittests-flyteidl	76.63% <ø> (ø)
unittests-flyteplugins	60.45% <ø> (ø)
unittests-flytepropeller	53.84% <100.00%> (+0.03%)	⬆️
unittests-flytestdlib	64.39% <ø> (-0.03%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[ttitsworth-lila]

I literally just ran into the issue, thank you for fixing it! Can't wait for the next v1 release :)