Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
943 changes: 943 additions & 0 deletions docs/redesign/PROPOSAL.md

Large diffs are not rendered by default.

81 changes: 81 additions & 0 deletions docs/redesign/REVIEWS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
# Review record

[PROPOSAL.md](PROPOSAL.md) went through two review rounds. Full
reports: [reviews/](reviews/).

## Reviewers

| Report | Brief | Verdict |
|---|---|---|
| [01-adversarial-r1.md](reviews/01-adversarial-r1.md) | Break rev 1 | Not approvable |
| [02-consumer-fit-r1.md](reviews/02-consumer-fit-r1.md) | Official issues + consumers | Placement yes, contracts no |
| [03-go-api-r1.md](reviews/03-go-api-r1.md) | Signatures, inference, names | Changes required |
| [04-adversarial-r2.md](reviews/04-adversarial-r2.md) | Break rev 2 | Scheduler and shutdown still open |
| [05-consumer-fit-r2.md](reviews/05-consumer-fit-r2.md) | Re-check issues | Conditional pass; lifecycle not closed |

## Revision 2: findings that changed the design

| Finding | Source | Disposition |
|---|---|---|
| Formation unspecified once workers exist | adversarial 1, consumer #73 | 5.2 state machine: inbound ≤ n, released ≤ W, handlers ≤ W, no third buffer |
| Same Policy, two grouping laws | adversarial 2 | 5.3: Batcher greedy drain vs Loader light-load groups of one |
| `Do`/`Add` before `Run` deadlocks; 6.5 never starts Loaders | adversarial 3 | `Add` before `Run` does not wait for `Run`; `Do` before `Run` is `ErrNotRunning`; 6.5 starts `go loader.Run(procCtx)` |
| Shared Loader ctx vs run ctx | adversarial 4 | `Loader.Run` ctx is process-scoped and must outlive every `Do` |
| Three stop rules; errgroup sample self-aborts | adversarial 5 | 5.6 table; cancel wins over Close; two-context sample does not use `errgroup.WithContext` |
| Re-entry into the same worker | adversarial 6 | `ErrReentry` on Batcher, Loader, and Runner |
| Shutdown budget leaks waiters | adversarial 7, consumer #71/#95 | Budget default 0; on expiry Loader settles waiters; `Wait` joins leftover handlers |
| Call is not a state machine | adversarial 8 | `queued \| running \| settled`; no reuse; `Fail(nil)` = `ErrUnanswered` |
| Flow dispatch / `If` / `Result` unimplementable | adversarial 9 | Dispatch predicate written; `Condition *bool`; `GraphError` defined; ready nodes wait for a slot |
| Shared mutable envelope (`#97`/`#98`) | consumer 1–2, API 1 | Immutable `In` + owned `any` results + `View` + explicit join node |
| Unmeasured persist linger in the example | consumer 2 | Persist `MaxItems: 1` |
| `Run` panics if called twice | adversarial 10, consumer 7, API 5 | `ErrUsed` |
| Silent `MinItems` rewrite; `SetPolicy` hole | adversarial 11, consumer 6 | Zero means default 1; negatives rejected; Loader `SetPolicy` uses `NewLoader` rules |
| Default 30s handler timeout | all three | Default 0 |
| `flow` imports `batch` via shared types | adversarial 13, API 6 | Duplicated `Event` / `ErrClosed` / `Stats` |
| Close-drain / “started” undefined | adversarial 14 | Started = handler invoked; Close keeps forming until empty |
| `batch.Group` vs `errgroup` | API 3 | Renamed `Info` / `InfoFromContext`; dropped `Attempt`/`Partial` |
| Helpers disagree with Graph | API 4 | Deleted `Sequence`/`Parallel`; `Map` → `ForEach`; ForEach does not cancel the caller's ctx |
| errgroup panic attribution | API 9 | Batcher does not recover; flow recovers to `*PanicError` |
| `#100` linger × node bound | consumer 9 | Stated as required: blocked `Do` occupies a node slot |
| File-level over-claim | consumer 10 | 8.1 is path-level only |
| Structured errors incomplete | API 2 | `Error`/`Unwrap` on every struct; `StatusUnknown` is zero |
| `flow.Run` as a method | API 7 | Package function; open question closed |
| Observer re-entry / Flush blocks | adversarial 18, API 6 | Observer on `Run` goroutine; must not call back |

## Kept (reviewers agreed)

Drain ≠ abort. No implicit key coalescing. Missing answer ≠ zero `Out`.
Caller `Do` cancel ≠ group cancel. `flow` does not own Loaders. Finite
DAG, no Promise/Watermark/retry. Policy clocks from the first item.
Max < min is rejected. Do not put this library on capture / Helius /
normalizer maps / recorder sequencing. No shims. Go 1.25.

## Revision 3: findings that changed the design

| Finding | Source | Disposition |
|---|---|---|
| Two schedulers (cut only when worker free vs cut≠dispatch) | adversarial r2 1 | 5.3 restated on the 5.2 machine: cut is independent of a free worker |
| `Add` abort returns nil then drops | adversarial r2 2 | Blocked `Add` sees abort as `Run`'s `ctx.Err()`; `nil` means accepted |
| `Do` during drain refills inbound | adversarial r2 3 | `Do` after `Close` is always `ErrClosed` |
| Budgeted `flow.Run` vs immutable `Result` | adversarial r2 4 | Expiry fills `Result` with Canceled; late returns ignored |
| Budget clock unspecified | adversarial r2 5 | Starts on abort, or when Close has nothing left to dispatch |
| `SetPolicy` cuts from the wrong goroutine | adversarial r2 6 | Stores + wakes `Run`; only `Run` cuts |
| 6.5 `ErrNotRunning` race | adversarial r2 7 | `Started()`; example waits before `Do` |
| `Do` full-queue unspecified | adversarial r2 8 | Same wait rules as `Add` |
| Negative durations silently mean “none” | adversarial r2 9, consumer r2 5 | `< 0` is `ErrInvalidPolicy`; `== 0` means none / default |
| Failure + cancel, two `err` values | adversarial r2 10 | Failed wins; `err` is `*GraphError` |
| `Wait` only after budgeted return | adversarial r2 11, consumer r2 1 | `Wait` always joins `Run` + handlers |
| 6.5 Close ∪ cancel | consumer r2 1 | Defers: Close+Wait first, `stop` last; abort recipe is cancel first |
| Budget expiry does not settle `Do` | consumer r2 2, adversarial r2 Adv7 | Expiry settles every still-unsettled `Do` with `*ShutdownError` |
| Default infinite budget vs `#95` | consumer r2 3 | Mechanism required; default 0 is a house-rule exception; redeploy sets it |
| `#98` Close does not cancel | consumer r2 4 | `Runner.Close` stops admission, then grace, then cancels node children |
| `#71` finite handler timeout | consumer r2 6 | Option exists; default 0 documented as the exception |
| `View` heap sharing | consumer r2 7 | Written: same `any`, mutating slices/maps/pointers is a user bug |
| Goroutine per waiting node | consumer r2 8 | One dispatcher per run; ready list, not a goroutine per node |
| `GraphError.Unwraps` | adversarial r2 12 | Removed; `Error` uses `errors.Join` |

## Still open after revision 3

Honesty tag number. Optional later `WithCoalesce`. `View.Get` type
assertions vs generated joins. Whether a production default shutdown
budget should be non-zero (kept 0 so a deadline never invents failure).
27 changes: 27 additions & 0 deletions docs/redesign/reviews/01-adversarial-r1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Adversarial review of revision 1

Independent review. The full report is preserved here. Verdict: not
approvable as written for `batch` or `flow`. Ranked blockers that
revision 2 must close:

1. Formation unspecified once workers exist (`MaxItems`, `WithQueue`,
greedy drain, and “one Do per worker” cannot all be true).
2. Same `Policy`, two grouping laws (Batcher greedy vs Loader light load).
3. `Do`/`Add` before `Run` deadlocks; the 6.5 example never starts
`Loader.Run`.
4. Shared Loader `Run` ctx vs per-run `Do`/`flow.Run` ctx.
5. Three stop rules, no winner; the two-context sample self-aborts.
6. Re-entry into the same worker (`Do` from a LoadHandler, `Add` from a
Handler, `flow.Run` from a node).
7. Shutdown budget leaks waiters by spec (`Do` not settled on expiry).
8. “Exactly one terminal outcome” is not a Call state machine.
9. Flow dispatch, `If`, `Result`, and bounds cannot be implemented from
the text.
10. `Run` panics if called twice (v0.5 already had `ErrBatchUsed`).
11. Silent `MinItems` rewrite; `SetPolicy` punches a hole in `NewLoader`.
12. Default 30s handler timeout invents persist failure.
13. `flow` depends on nothing in `batch` is already false (`Event`,
`ErrClosed`).
14. Close-drain grouping and “started” undefined.

Rules the review said to keep are listed in [REVIEWS.md](../REVIEWS.md).
26 changes: 26 additions & 0 deletions docs/redesign/reviews/02-consumer-fit-r1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Consumer-fit review of revision 1

Independent review against official issues `#71`, `#73`, `#95`, `#97`–`#100`
and ShitQuant house rules. Verdict: placement yes, contracts no.

Must-fix for revision 2:

1. The recommended flow model is the shared mutable analysis object
`#97` forbade. `#98` wants owned results, immutable views, an
explicit join.
2. The Decode/Metadata/Prices/Persist example writes `*Work` concurrently
and puts an unmeasured linger on persist.
3. Batcher abort can hang forever (no budget / no `Wait`).
4. Loader shutdown does not settle waiters on budget expiry (`#71`).
5. Formation and safety are still one knob; formed-not-started groups
are unbounded (`#73`).
6. Silent `MinItems` clamp contradicts §1 and §4.
7. `Run` panics if called twice (`#95`).
8. Flow `Close` does not implement `#98` shutdown.
9. `#100` linger × executing-node bound is unspecified.
10. Section 8.1 over-claims file-level operational detail the environment
could not verify.

What to keep: product cut B; path-level Yes/No for ShitQuant; OnyxCore
and shitlock boundaries; no implicit key coalescing; cancel ≠ drain;
missing result ≠ zero `Out`.
20 changes: 20 additions & 0 deletions docs/redesign/reviews/03-go-api-r1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Go API review of revision 1

Independent review. Signatures were compiled as stubs. Verdict: changes
to insist on before implementation.

1. Envelope-by-value (`flow.New[Work]()`) compiles and loses writes.
Consumer-fit then forbade the pointer-envelope workaround. Revision 2
uses immutable `In` plus owned `any` results instead.
2. Structured errors lack `Error()` / `Unwrap()`; zero `Status` is
success; `*GraphError` is undefined.
3. `batch.Group` collides with `errgroup.Group`. Rename to `Info`.
Drop `Attempt` / `Partial`.
4. `Sequence` / `Parallel` / `Map` disagree with Graph on cancel.
Delete the first two; rename `Map` to `ForEach`.
5. Options contradict “never panic / never silent rewrite”.
6. Duplicate `Event` / `ErrClosed` across packages; plan it.
7. `flow.Run` must stay a package function (no generic methods on 1.25).
8. Default handler timeout 0 (30s invents failure and breaks synctest).
9. Do not attribute conc’s panic-recover to errgroup (reverted).
10. `Loader` godoc must lead with “does not coalesce by key”.
9 changes: 9 additions & 0 deletions docs/redesign/reviews/04-adversarial-r2.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Adversarial review of revision 2

Verdict: neither package approvable as written. Most r1 text holes
were closed. Remaining blockers: two schedulers in one spec; `Add`
abort can return nil then drop; `Do` during drain can refill inbound;
budgeted `flow.Run` cannot fill an immutable `Result`; budget clock
unspecified; `Wait` defined only after budgeted return.

Revision 3 is the response.
10 changes: 10 additions & 0 deletions docs/redesign/reviews/05-consumer-fit-r2.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# Consumer-fit review of revision 2

Verdict: conditional pass. Placement and the `#98` graph shape are
right. Lifecycle is not closed: 6.5 `Close ∪ cancel`, budget expiry
does not settle waiters, default infinite wait, `#98` close-then-cancel
inverted, `MinItems <= 0` still reads as a clamp.

6.5 graph: legal. 6.5 snippet: not legal.

Revision 3 is the response.
Loading