ci(pyamber): measure branch coverage - #8040
Open
aglinxinyuan wants to merge 1 commit into
Open
Conversation
Contributor
Automated Reviewer SuggestionsBased on the
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8040 +/- ##
============================================
- Coverage 93.36% 93.24% -0.12%
- Complexity 4687 4690 +3
============================================
Files 1179 1179
Lines 47713 47713
Branches 5314 5761 +447
============================================
- Hits 44547 44491 -56
Misses 1715 1715
- Partials 1451 1507 +56
☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🟢 | bs=10 sw=10 sl=64 | 391 | 0.239 | 25,320/31,975/31,975 us | 🟢 -9.4% / 🔴 +131.0% |
| 🔴 | bs=100 sw=10 sl=64 | 812 | 0.495 | 123,639/142,312/142,312 us | 🔴 +7.7% / 🔴 +51.9% |
| ⚪ | bs=1000 sw=10 sl=64 | 921 | 0.562 | 1,086,572/1,167,912/1,167,912 us | ⚪ within ±5% / 🔴 +29.4% |
Baseline details
Latest main 97e3585 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 391 tuples/sec | 411 tuples/sec | 894.86 tuples/sec | -4.9% | -56.3% |
| bs=10 sw=10 sl=64 | MB/s | 0.239 MB/s | 0.251 MB/s | 0.546 MB/s | -4.8% | -56.2% |
| bs=10 sw=10 sl=64 | p50 | 25,320 us | 25,476 us | 11,139 us | -0.6% | +127.3% |
| bs=10 sw=10 sl=64 | p95 | 31,975 us | 35,302 us | 13,843 us | -9.4% | +131.0% |
| bs=10 sw=10 sl=64 | p99 | 31,975 us | 35,302 us | 16,839 us | -9.4% | +89.9% |
| bs=100 sw=10 sl=64 | throughput | 812 tuples/sec | 836 tuples/sec | 1,166 tuples/sec | -2.9% | -30.3% |
| bs=100 sw=10 sl=64 | MB/s | 0.495 MB/s | 0.51 MB/s | 0.711 MB/s | -2.9% | -30.4% |
| bs=100 sw=10 sl=64 | p50 | 123,639 us | 114,779 us | 87,509 us | +7.7% | +41.3% |
| bs=100 sw=10 sl=64 | p95 | 142,312 us | 143,338 us | 93,675 us | -0.7% | +51.9% |
| bs=100 sw=10 sl=64 | p99 | 142,312 us | 143,338 us | 102,153 us | -0.7% | +39.3% |
| bs=1000 sw=10 sl=64 | throughput | 921 tuples/sec | 933 tuples/sec | 1,198 tuples/sec | -1.3% | -23.1% |
| bs=1000 sw=10 sl=64 | MB/s | 0.562 MB/s | 0.569 MB/s | 0.731 MB/s | -1.2% | -23.1% |
| bs=1000 sw=10 sl=64 | p50 | 1,086,572 us | 1,069,900 us | 859,766 us | +1.6% | +26.4% |
| bs=1000 sw=10 sl=64 | p95 | 1,167,912 us | 1,123,190 us | 902,897 us | +4.0% | +29.4% |
| bs=1000 sw=10 sl=64 | p99 | 1,167,912 us | 1,123,190 us | 937,957 us | +4.0% | +24.5% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,511.65,200,128000,391,0.239,25320.46,31975.42,31975.42
1,100,10,64,20,2463.79,2000,1280000,812,0.495,123639.10,142312.30,142312.30
2,1000,10,64,20,21718.10,20000,12800000,921,0.562,1086572.45,1167911.83,1167911.83
This was referenced Aug 27, 2026
renovate-bot
pushed a commit
to renovate-bot/apache-_-texera
that referenced
this pull request
Aug 28, 2026
### What changes were proposed in this PR? `test_run_python_worker.py` goes from 27 tests to 42, covering the worker entry point's configuration-failure paths. | Metric | Before | After | |---|---|---| | Codecov fully-covered lines **today** | 32/39 | **39/39** | | Codecov fully-covered lines **after apache#8040** | 31/39 | **39/39** | | Branch arms | 9/12 | **12/12** | **`texera_run_python_worker.py` reaches 100% on both lines and branches.** +7 fully-covered lines today, +8 and +3 arms after apache#8040 — line 155 is a partial that the current branch-less report already counts covered, because it executes at import. The uncovered set was the argument-parsing failures (26–28), the loguru level configuration (35), and the missing-dependency and R_HOME paths (48, 51, 155–156). These are what run when the worker starts with a malformed or incomplete configuration, so a regression there surfaces as a worker that fails to start with an unhelpful message. One correction to the assessment worth recording: **line 143, the R_HOME guard, was already at 2/2** — it was not part of the gap. ### Verification Two reviewers returned nine findings and reported **thirteen surviving mutants** against the first draft. All thirteen were independently reproduced on a pristine tree before being fixed, and all thirteen now die. **39 mutations in the final table: 38 killed, 1 equivalent, 0 real survivors.** The equivalent one is `frozenset` to `set` for `EXPECTED_CONFIG_KEYS` — reported rather than omitted. The repair also confirmed something worth stating about the measurement itself: `pytest`'s `importlib` import mode re-executes this module per test, and there is **exactly one** `<class filename="texera_run_python_worker.py">` entry in every report, on both sides and in the full-suite run. So the fresh execs attribute to the same source path rather than splitting the coverage entry — checked rather than assumed. Measured from `amber/` with an identical single-spec test-path scope on both sides, both with and without `--cov-branch` (the no-branch figure measured directly, not derived), per-line `hits` and `condition-coverage` read out of `coverage.xml` rather than from the summary percentage. Full pyamber suite: `5 failed, 1174 passed, 7 errors` against main's `1159 passed`. Failure sets sorted and diffed — **identical by identity**, 12 entries. `ruff check` and `ruff format --check` pass on CI's exact scope. No production file is touched; `git diff -- 'amber/src/main/*'` is empty. ### Any related issues, documentation, discussions? Closes apache#8046 ### How was this PR tested? ``` cd amber && python -m pytest -m "not integration" -q src/test/python/test_run_python_worker.py ``` ``` 42 passed ``` ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 5)
renovate-bot
pushed a commit
to renovate-bot/apache-_-texera
that referenced
this pull request
Aug 28, 2026
…he#8048) ### What changes were proposed in this PR? A new `TestIcebergDocumentWithMockCatalog` class in `test_iceberg_document.py`, taking the file from 9 collected items to 22. | Metric | Before | After | |---|---|---| | Codecov fully-covered lines **today** | 118/124 | **124/124** | | Codecov fully-covered lines **after apache#8040** | 115/124 | **124/124** | | Branch arms | 21/26 | **26/26** | **`iceberg_document.py` reaches 100% on both lines and branches.** +6 fully-covered lines today, +9 after apache#8040 — lines 90, 108 and 189 are `50% (1/2)` partials, invisible to the current branch-less report. The nine lines were exactly {79, 80, 81, 84, 90, 108, 109, 189, 190}: `get_uri`, `get_count`, `clear`, and the `get_after` skip guard. ### The measurement problem, and how it was handled The existing `TestIcebergDocument` needs a live postgres Iceberg catalog. CI supplies one; on this machine the `texera` role cannot read `iceberg_tables`, so **all 9 of those tests fail locally** and a raw local coverage run under-reports the file badly — enough to suggest ~50 winnable lines instead of 9. A throwaway sqlite-backed `SqlCatalog` proxy was used to reproduce the same code paths, applied identically to both sides and then deleted. The honest gap is 9. **The new tests need none of that.** They patch `IcebergCatalogInstance.get_instance` rather than mutating it, so they touch no process-global state and need no postgres, filesystem or network. They run in 1.25s. ### Two hazards found in the existing fixture - **`StorageConfig.initialize` is a process-wide one-shot** that raises on a second call, and `test_iceberg_document.py:45` calls it **unguarded at module level**. That works only because this module happens to be the first module-level initializer imported — two sibling modules guard with `if not StorageConfig._initialized`. A new test module initializing at module level and sorting before it would break collection for the whole suite. Appending a class to the existing file avoids this; a separate file would have been a trap. - **`IcebergCatalogInstance._catalogs` is a class-level dict** and `replace_instance` mutates it for the rest of the session, poisoning every later suite in the shared process. The new tests sidestep it by patching `get_instance`. ### Honest discounts - **`get_uri` and `get_count` have zero callers anywhere in `src/main/python`**, and `clear` has none either — the Scala side has its own implementation. These are real statements with real logic, not `@abstractmethod` stubs, but the value is interface-contract conformance rather than live behaviour. - **Lines 189–190 need a negative `from_index`.** `num_of_skipped_records` is provably 0 when the guard first runs, so the only route is `get_after(-1)`. It is drivable through the public API, but no production caller produces a negative offset — a defensive-guard test. Counted strictly, the bundle is 7 lines and 4 arms. - **One mutant survives this class**: `sum(f.file.record_count ...)` to `sum(1 ...)` at line 110, killed only by the pre-existing postgres-gated `test_get_counts`. Line 110 is already covered in CI so it is not a coverage gap, but the kill is not this bundle's and one was not manufactured. ### Verification Two reviewers returned seven findings; all repaired. **The repair pass added zero coverage lines and zero arms** — the builder's bundle already reached 124/124. Its entire value was mutation strength: 14 mutants that passed the delivered spec now fail. Stated plainly rather than sold as more coverage. Full pyamber suite: `5 failed, 1172 passed, 7 errors` against main's `1159 passed`. Failure sets sorted and diffed — **identical by identity**, 12 entries. `ruff check` and `ruff format --check` pass on CI's scope. The throwaway proxy is deleted and `git status --porcelain` is clean. No production file is touched; `git diff -- 'amber/src/main/*'` is empty. ### Any related issues, documentation, discussions? Closes apache#8044 ### How was this PR tested? ``` cd amber && python -m pytest -m "not integration" -q src/test/python/core/storage/iceberg/test_iceberg_document.py -k MockCatalog ``` ``` 13 passed, 1 warning in 1.25s ``` The 9 pre-existing `TestIcebergDocument` tests still require the postgres catalog that CI provides. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 5)
renovate-bot
pushed a commit
to renovate-bot/apache-_-texera
that referenced
this pull request
Aug 28, 2026
…che#8047) ### What changes were proposed in this PR? `test_operator.py` goes from 32 tests to 44, covering `SourceOperator.on_finish` and `BatchOperator`'s output-conversion arms. | Metric | Before | After | |---|---|---| | Codecov fully-covered lines **today** | 173/183 | **178/183** | | Codecov fully-covered lines **after apache#8040** | 171/183 | **178/183** | | Branch arms | 26/30 | **30/30** | **+5 fully-covered lines today, +7 after apache#8040, and the file reaches 100% branch coverage.** Lines 233 and 234 are line-hit with one arm untaken, so today's branch-less CI report already scores them covered; closing their second arm is mutation-proven but invisible until apache#8040 lands. **This is a small PR and I would rather say so than dress it up.** What makes it worth having is that it closes the file permanently: the five remaining gap lines (153, 177, 259, 293, 515) are `@abstractmethod` bodies, and a grep of `src/main/python` for `super().process_tuple|produce|process_batch|process_table|condition` finds **zero** production call sites. 178/183 is the ceiling, so nobody needs to re-derive this target. **Scope narrowed during assessment, and the original plan is worth recording.** This began as a three-file bundle also covering the four concrete partitioners and `proxy_server.py`. Both were dropped on measurement: the partitioners' base class is bare-`pass` stubs, and `proxy_server.py` is already at 90/95 with a 268-line spec, leaving 2 honest lines. Only `operator.py` survived. ### What matters about the two covered paths `SourceOperator.on_finish` is the uncovered production path of **every** Python source UDF. `BatchOperator._process_batch`'s `None` and non-DataFrame arms decide what a batch operator emits when its handler returns nothing, or returns something that is not a DataFrame. ### Verification Measured from `amber/` with the identical full-suite scope on both sides, `--cov-branch` passed explicitly (this branch lacks apache#8040's config), per-line `hits` and `condition-coverage` read out of `coverage.xml` by script rather than from the summary percentage. The before-state used main's spec extracted via `git show HEAD:` into a scratch dir and restored from that snapshot — never `git restore`. Two reviewers returned nine findings against the first draft; all repaired. **Three mutants survive and are reported rather than dropped**, all three genuinely equivalent: `!= 0` to `> 0` on a list length (non-negative by construction), and deleting `@overrides.final` from either method — `overrides.final` is `typing.final`, which enforces nothing at runtime. The last two prompted a docstring correction, since the original text claimed finality as the reason the method was worth testing. Full pyamber suite: `5 failed, 1171 passed, 7 errors` against main's `5 failed, 1159 passed, 7 errors`. The `FAILED`/`ERROR` node-id sets were sorted and diffed — **identical by identity**, 12 entries. `ruff check` and `ruff format --check` pass on CI's exact scope. `git diff --numstat` is `226 0` — zero deletions. No production file is touched; `git diff -- 'amber/src/main/*'` is empty. ### Any related issues, documentation, discussions? Closes apache#8043 ### How was this PR tested? ``` cd amber && python -m pytest -m "not integration" -q src/test/python/core/models/test_operator.py ``` ``` 44 passed, 1 warning ``` ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 5)
renovate-bot
pushed a commit
to renovate-bot/apache-_-texera
that referenced
this pull request
Aug 28, 2026
… paths (apache#8042) ### What changes were proposed in this PR? `test_main_loop.py` goes from 34 tests to 44, covering the console-message RPC path, the end-channel completion rules, the per-element exception backstop, and the ECM/loop-counter forwarding paths. | Metric | Before | After | |---|---|---| | Codecov fully-covered lines **today** | 270/285 | **284/285** | | Codecov fully-covered lines **after apache#8040** | 259/285 | **280/285** | | Branch arms newly taken | — | **+11** | **+14 fully-covered lines move on Codecov today; the full +21 only shows up once apache#8040 lands.** Lines 176, 397, 658, 661, 706, 714 and 837 are line-hit with one arm untaken, so today's branch-less report already scores them covered — closing their second arm is real and mutation-proven but invisible until branch coverage is on. I would rather state both figures than quote the larger one and have it not materialise. The `[tool.coverage.run] branch = true` setting is deliberately **not** added here; it belongs to apache#8040 and adding it would conflict. Newly fully covered: 176, 177, 397, 400, 658, 661, 662, 706, 714, 715, 718, 719, 725, 813, 814, 817, 837, 838, 839, 849, 850. Nothing lost. Line 795 remains uncovered and four lines (349, 650, 740, 794) remain branch-partial. ### What the reviewers found Two adversarial reviewers reported eight surviving mutants against the first draft. Every one was reproduced before being fixed — seven gave a clean `43 passed` on the delivered spec and one produced no summary at all. **Eight of the claimed lines were covered vacuously.** The ones worth naming: - **The console-message test blocked on `output_queue.get()` instead of failing** — a hang, not an assertion failure, which is the worst way for a test to be wrong. - **The two-input-port end-channel test could not distinguish the port lookup from a constant**, so the rule it exists to pin was unconstrained. - **The no-output-ports test could not see an `EndChannel` broadcast at all.** - **`assert switched == [True]` claimed a statement *order* it cannot observe.** - A `emitted_counter == 0` assertion was a degenerate literal, and a `reset_calls == []` assertion was call-graph-guaranteed to hold. All eight now die. **33 mutations applied one at a time, 32 killed, 1 judged equivalent** (a flush/pause order exchange, with the reasoning recorded). Four of the 32 kills are honest about their conditions: the line-706, line-661, line-837 negations and the 393–394 order swap die only when the run is scoped to a single test, because they make **pre-existing** thread-driven tests block, and pytest-timeout's Windows `thread` method then kills the session before the killing test runs. They are stated as scoped kills rather than credited as clean whole-file kills. One further mutant was discarded rather than reported as surviving: negating line 661 makes the pre-existing `test_main_loop_thread_can_process_messages` block forever, yielding no per-test signal. The same statement is covered by a `return`→`pass` mutant that dies exclusively against a new test, so nothing is left unproven. ### A defect found and deliberately not pinned `_process_data_element`'s per-element backstop (813–814) is `except Exception as err: logger.exception(err)` — it logs and continues **without calling `context.report_exception`**. A runtime failure on one element therefore never reaches the coordinator, and the workflow can report SUCCESS on a short result. The new test asserts only that iteration continues and that nothing propagates. It does **not** assert that nothing is reported, because that half is arguably a silent-wrong-results bug and pinning it would cement it. The test carries a comment saying so; anyone strengthening it should keep that boundary. ### Verification Measured with the full CI-shaped suite from `amber/`, `--cov-branch` passed explicitly on both sides (this branch is on `main` and so lacks apache#8040's config), `LOGURU_LEVEL=WARNING`, per-line `hits` and `condition-coverage` read out of `coverage.xml` by script rather than from the summary percentage. The before-state was obtained by copying the original spec back from a scratch snapshot — never via `git restore` — with the production tree verified pristine throughout. **No regression.** Full pyamber suite: `5 failed, 1169 passed, 7 errors` against a baseline of `5 failed, 1159 passed, 7 errors`. The `FAILED`/`ERROR` node-id sets were extracted from both runs, sorted and diffed: **identical by identity**, 12 entries (7 `test_iceberg_document` errors, 2 iceberg failures, `test_tuple::test_hash`, 2 `test_expression_evaluator` repr-formatting failures). +10 passing. The scoped spec passes `44 passed` on five consecutive runs. `ruff check` and `ruff format --check` pass on CI's exact scope (`src/main/python src/test/python`). `git diff --numstat` shows `666 0` — 666 added, **zero deleted**. **A flake seen once and reported rather than buried:** one no-branch after-run came back 282/285 with lines 324–325 at zero hits; three consecutive re-runs all gave 284/285 with both covered. Those two lines are in `_check_and_process_control`'s while body, reached from a daemon thread, so a scheduling-dependent miss is plausible. The published figure is the reproducible one. ### On overlap with open PRs This file is not untouched work. **apache#7624** (open) and **apache#7388** (draft) both modify `main_loop.py` *and* `test_main_loop.py`. I checked both diffs: none of the 21 lines here falls inside either PR's edited hunks, and the new tests are appended at the end of `TestMainLoop`, away from the test-file hunks. The collision risk is a routine rebase rather than a semantic conflict — but it is not zero, and the earlier claim that this target was free was wrong. ### Deliberately not included Line 795 and the four remaining partials (349, 650, 740, 794) are left. Reaching 813/814 already required monkeypatching a `MainLoop` private — established practice in this file but white-box, and flagged as the one such test in the bundle. No production file is touched; `git diff -- 'amber/src/main/*'` is empty, checked after every mutation revert and as the final action. ### Any related issues, documentation, discussions? Closes apache#8041 ### How was this PR tested? ``` cd amber && python -m pytest -m "not integration" -q src/test/python/core/runnables/test_main_loop.py ``` ``` 44 passed, 1 warning ``` ``` cd amber && ruff check src/main/python src/test/python && ruff format --check src/main/python src/test/python ``` `amber/junit.xml`, `amber/coverage.xml` and `amber/.coverage` are regenerated by every run and are not committed. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 5) --------- Signed-off-by: Xinyuan Lin <xinyual3@uci.edu> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this PR?
Enables branch coverage for pyamber. Two files, no production or test code:
amber/pyproject.toml— adds[tool.coverage.run] branch = true..github/workflows/build.yml— a comment on the pytest step recording that branch coverage is on and where it is configured. Therun:command itself is unchanged.Why the config rather than
--cov-branch--cov-branchon the CI command would work, but it would put CI and a developer's laptop on different metrics: a localpytest --covwould keep reporting statement-only coverage, so a half-takenifwould read as covered locally and partial on Codecov. Setting it inpyproject.tomlgives both the same metric from one switch.The comment in
build.ymlexists so the pipeline is not silent about the change, and it says explicitly not to add the flag back — two switches for one boolean invites someone to remove one and believe it is off.What this surfaces
Measured on
97e3585a6a, withcodecov.yml's ignore rules applied (proto/**,test_*.py):62 partial lines and 80 untaken branch arms become visible. No line's actual execution changed — the same 1159 tests run and the same statements execute. This only stops the report from rounding a half-taken branch up to "covered".
The reported number will go down, and that is the point
pyamber's flag figure drops 97.27% → 95.89%, about 1.4 points. That is expected: partial lines stop counting as hits.
codecov.ymlsetscoverage.status.projecttotarget: auto, threshold: 1%, so it is worth being precise about whether this trips the check. It does not:flag_managementincodecov.ymlsets onlycarryforward— no per-flag statuses are configured, so pyamber's own 1.4-point drop is not itself gated.patchstatus is unaffected: this PR changes no coverable lines.If a per-flag pyamber status is ever added, it should be added after this lands so its baseline is the branch-aware figure.
Verification
Run locally on Python 3.12.10 with coverage 7.13.5 and pytest 7.4.0 — the same coverage version CI resolves.
--cov-branchflag, andcoverage.xmlcame back withbranch-rate="0.9105"and 447 lines carryingcondition-coveragedata. Before the change the same command givesbranch-rate="0"and zero such lines. coverage.py findsamber/pyproject.tomlbecause CI doescd amberfirst, which is also how the repo documents running pytest.main) are identical in identity, not merely in count: the twoFAILED/ERRORsets diff clean at 12 entries.build.ymlstill parses as YAML (9 jobs) and the pyamber step'srun:command is byte-identical tomain.amber/pyproject.tomlstill parses viatomllib, with[tool.ruff]and[tool.pytest.ini_options]intact.ruff check src/main/python src/test/python && ruff format --check src/main/python src/test/python→All checks passed!/211 files already formatted.One note in case someone reproduces this: running
ruff format --check .fromamber/reports two files needing reformatting, both.pyfiles living undersrc/main/scala/.../aiassistant/. They are pre-existing (identical output with this change reverted) and outside CI's ruff scope, so they are unrelated to this PR — mentioned only so the broader command's output is not mistaken for a regression.Any related issues, documentation, discussions?
Closes #8039
How was this PR tested?
CI's own lint scope:
Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5)