[aayushbaluni]

Which issue does this PR close?

Closes #8376.

Rationale for this change

FlatLister::next and ServicerFlatLister::next take the pending directory
out of next_dir before listing it, and two error paths returned without
putting it back:

if let Some(de) = self.next_dir.take() {
    let mut l = match self.srv.list(&self.ctx, de.path(), OpList::new()) {
        ...
        Err(e) => return Err(e),          // `de` dropped here
    };
    let first = loop {
        match l.next().await {
            ...
            Err(e) => return Err(e),      // and here
        }
    };
    if let Some(v) = first {
        self.active_lister.push((Some(de.clone()), l));

At both points the entry has not reached active_lister yet, so returning
drops it and nothing remembers that the directory still needs listing.

Without a retry this is harmless — the error reaches the caller. Under
RetryLayer it becomes silent data loss. RetryWrapper::next
(core/layers/retry/src/lib.rs) hands the same lister to backon, which
calls next() on it again:

|mut p: P| async move { let res = p.next().await; (p, res) }
    .retry(self.builder).when(|e| e.is_temporary()).context(inner)

The second call finds next_dir empty and no new lister on the stack, so it
resumes from the parent, moves on to the next sibling, and eventually returns
Ok. The temporary error never surfaces and the failed directory's entries
are simply absent. A recursive list over a tree where one child directory
hiccups once comes back short and successful, which is the worst shape for
a caller: there is nothing to retry on and nothing to log.

That matches every row of the table in #8376, including why the non-recursive
case is unaffected (it does not go through the flat lister) and why /a/ is
requested exactly once.

What changes are included in this PR?

Restore next_dir before returning the error, so a retry re-lists that
directory and a caller that gives up propagates the error instead of
under-reporting. Neither point has yielded anything from the child lister
yet, so listing it again from the start cannot duplicate entries.

Both listers have the same defect and both are fixed:

• ServicerFlatLister in core/core/src/layers/simulate.rs is the reachable
  one — SimulateLayer builds it for recursive listing — and is what bug: recursive listing silently skips a directory after a temporary error with RetryLayer #8376
  exercises through WebDAV. It gets the regression tests.
• FlatLister in core/core/src/raw/oio/list/flat_list.rs has no in-repo
  caller, but it is re-exported as raw::oio::FlatLister for out-of-tree
  services, so the same fix is applied. I did not add a test for it: covering
  it means a Service mock in that module, and I would rather follow your
  preference than guess — happy to add one, or to drop that half if you would
  rather fix it separately.

The ErrorKind::NotFound and PermissionDenied arms are untouched. Those
deliberately continue, skipping the directory, and that behaviour is
unchanged.

Are there any user-facing changes?

Yes, in that recursive listing stops losing entries. Concretely, a recursive
list that previously returned Ok with a directory missing now either
returns the complete listing (when the retry succeeds) or returns an error
(when it does not). Code that silently depended on the short result would
now see the error it should always have seen.

No API changes.

Breaking changes

None.

Testing

Three tests in core/core/src/layers/simulate.rs, driving the real
ServicerFlatLister against a mock service whose list walks a fixed tree
and can fail one directory once with a temporary error. No network, no
fixture server. The test drains the lister the way backon does — on a
temporary error, call next() again on the same lister.

running 7 tests
test layers::simulate::tests::simulate_layer_can_disable_read_with_suffix ... ok
test layers::simulate::tests::simulate_layer_exposes_read_with_suffix ... ok
test layers::simulate::tests::simulate_reader_uses_content_length_hint_for_suffix ... ok
test layers::simulate::tests::servicer_flat_lister_reports_a_directory_that_keeps_failing ... ok
test layers::simulate::tests::simulate_stat_dir_without_recursive_list ... ok
test layers::simulate::tests::servicer_flat_lister_retries_a_directory_whose_list_failed ... ok
test layers::simulate::tests::servicer_flat_lister_retries_a_directory_whose_first_entry_failed ... ok

test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 232 filtered out

The three new cases are the two ways the directory can be lost plus the
no-retry path:

Test	Asserts
..._retries_a_directory_whose_first_entry_failed	the fault surfaces once, a/ is requested a second time, both files are returned
..._retries_a_directory_whose_list_failed	the same, for a failure in list itself rather than the first next()
..._reports_a_directory_that_keeps_failing	a caller that does not retry still sees the error, so the restored next_dir cannot mask a real failure

Because "the tests pass" says little on its own, I deleted each restored line
in turn and re-ran, to check the tests actually hold the behaviour:

Mutation	Result
drop the restore in the list(...) error arm	caught by ..._whose_list_failed
drop the restore in the first-next() error arm	caught by ..._whose_first_entry_failed

Each mutation is the original code for that path, so this also confirms both
tests fail against main rather than only passing afterwards.

One fixture bug of mine is worth mentioning because it shaped a test: my
first attempt built the tree with "" as the root key and saw no entries and
no error at all. oio::Entry::with normalizes an empty path to "/", so
list was being called with "/". Both retry assertions now print the paths
that actually reached list, so the next mismatch of that kind says what
happened instead of 0 != 1.

Also run, on rust-toolchain.toml's stable (1.99.0): cargo clippy -p opendal-core --all-targets with no warnings, and cargo fmt -p opendal-core --check clean.

AI Usage Statement

• Harness: Claude Code (CLI).
• Model: Claude Opus 5 (1M context), claude-opus-5[1m].
• Effort: harness default; no effort setting exposed to me.
• Role: AI read the two listers and RetryWrapper::next to confirm the
  mechanism the issue hypothesised, wrote the fix and the three tests, and ran
  the test and mutation runs in a container. The reproduction in bug: recursive listing silently skips a directory after a temporary error with RetryLayer #8376 is the
  reporter's, not mine.

Assumptions and unknowns a reviewer should weigh:

• I did not run the issue's WebDAV fixture end to end. The mechanism is
  confirmed by reading RetryWrapper::next and by the mock-level tests, which
  reproduce the reported shape — Ok with entries missing and one request for
  the failed directory — but an independent run against the reporter's
  harness would be stronger evidence than anything I have.
• FlatLister is fixed by inspection and has no test, as described above.
• I assumed re-listing a directory from the start is safe because nothing has
  been yielded from its lister at either error point. That is true of these
  two paths as written; it is the one invariant worth a second pair of eyes.

[aayushbaluni]

CI note, so the two red checks don't cost a reviewer time: 365 green, 2 red,
and I don't believe either is this change. Evidence rather than assertion for
both, and one of them looks like something you'd want to know about
independently of this PR.

test (Bindings MoonBit CI) — fails in Install MoonBit, before anything
is compiled:

Downloading moonbit ...
curl: (22) The requested URL returned error: 403
error: Failed to download moonbit from
"https://cli.moonbitlang.com/binaries/0.10.6%2B80dc50f24/moonbit-linux-x86_64.tar.gz"

That workflow's own run history has it green through 2026-10-04 (including the
main push on 10-03) and red from today, so the pinned
MOONBIT_VERSION: 0.10.6+80dc50f24 appears to have stopped being served
upstream rather than anything changing here. It will presumably hit every PR
touching core/** until the pin moves — worth a look on its own.

stubs (Bindings Python CI) — fails in Check diff, on regenerated
bindings/python/docs/index.md, over two blank lines around a snippet:

@@ -174,10 +174,12 @@ Async API equivalent:
  import asyncio
+
 async def main():

This one predates the PR: the same workflow fails in the same job and the same
step on unrelated changes, e.g. dependabot run
37195194424
(JOB stubs: Check diff), and on the main push of 2026-10-04. This PR
touches no Python binding or docs file.

For completeness, I was initially wrong about how to check this: I compared
against #8371, #8372 and #8374, which all looked clean. They are not a valid
baseline — both workflows trigger on core/**, which those PRs do not touch,
so neither workflow ever ran for them. Comparing each workflow's own run
history is what actually answers it.

Happy to rebase once the MoonBit pin is sorted if you'd like a fully green run
before review.

[aayushbaluni]

Pushed c945a44, adding one test case. No change to the fix.

The three cases I opened with all fail a directory while only the root's lister
is on the stack. That left the shape I should have covered first untested: a
directory two levels down. When a/b/ fails, / and a/ are both open in
active_lister, so a lost next_dir is masked by the parent resuming rather
than by the listing ending — it pops back to /, moves on to c/, and returns
Ok without a/b/deep.txt. That is the same silent-shortfall this PR is
about, reached by a different route.

servicer_flat_lister_retries_a_nested_directory pins it, and I checked it
holds the behaviour rather than assuming: with the restore in the
first-next() arm removed it fails on exactly that assertion.

test ...servicer_flat_lister_retries_a_nested_directory ... FAILED
assertion `left == right` failed: the nested directory must be listed again,
not skipped by the parent

8 tests pass, cargo fmt --check clean, cargo clippy --all-targets with no
warnings.

Why I went looking: review on an unrelated PR of mine caught a defect that my
mutation testing had scored 5/5 on, because no fixture entered the affected
path. A mutation score only covers the paths the fixtures reach, so I came back
here to ask which paths my fixtures were not entering. This was the answer.

[aayushbaluni]

The red test job here is infrastructure, not this change — a re-run should clear it. It never reached the tests:

curl: (22) The requested URL returned error: 403
error: Failed to download moonbit from "https://cli.moonbitlang.com/binaries/0.10.6%2B80dc50f24/moonbit-linux-x86_64.tar.gz"
##[error]Process completed with exit code 1.

A 403 fetching the MoonBit toolchain from cli.moonbitlang.com, during setup. Breaking change declaration is green and every other job passes.

[aayushbaluni]

Rebased onto current main (9fdfac7). 375 checks green, 2 red, and neither is from this diff — which only touches core/src/raw/oio/list/flat_list.rs and the simulate layer's test.

Bindings Lua CI / test — the Lua binding does not build on the runner's Lua version:

opendal 0.1.0-1 depends on lua >= 5.1 (5.1-1 provided by VM)
error: the package 'opendal-lua' does not contain this feature: lua51
help: there are similarly named features: lua52, mlua

This is failing repo-wide, not here: the same workflow is red on main (27 Oct 13:58 and 18:30) and on hf-disable-config-load, codex/hf3fs-usrbio and nmh/add-azblob-list-with-deleted. The only recent green run is #8435 (fix(ci): pin lua bindings CI to ubuntu-24.04), which is the fix for exactly this. Nothing for this PR to do but wait for that to land.

Bindings Java CI / test (windows-latest, 17) — ExecutorShutdownTest.testJvmExitsAfterClosingOperator, both parameterised cases, each at 15.03 s and 15.07 s elapsed. That is the assertion's own timeout waiting for the JVM to exit, not a behavioural difference. Three things say it is flaky rather than mine:

• codex/hf3fs-usrbio has Java CI green at 06:22, red at 06:36 and green again at 03:43 on the same code
• the failing matrix cell moved between my runs — windows-latest, 21 on 23 Oct, windows-latest, 17 today; a deterministic break from a core change would hit the same cell
• the test is about JVM shutdown timing on Windows, and nothing in this diff spawns or holds a thread; the flat lister change only keeps a failed directory on the retry list

Happy to re-run or rebase again if either turns out to be real — but I did not want to leave a red PR without saying which of the two is a known repo-wide break and which is a timing flake.

[aayushbaluni]

Rebased onto main at 451bc73c, which is the first head on this branch to carry #8435. The diff is unchanged — 2 files, +326/−11.

Both reds on the previous head were outside this change, and I would rather show that than assert it.

Bindings Lua CI / test was broken repo-wide:

error: the package 'opendal-lua' does not contain this feature: lua51
help: there are similarly named features: lua52, mlua
Error: Build error: Failed building.

ubuntu-latest began providing Lua 5.1, which opendal-lua has no feature for. #8435 ("fix(ci): pin lua bindings CI to ubuntu-24.04") merged on 2026-10-28 at 10:50 UTC — about five hours after the run above — so this rebase is the first time the branch has that pin.

Bindings Java CI / test (windows-latest, 17) is a flake, and the matrix shows it rather than me arguing it. Same two commits, two heads:

head	JDK 17	JDK 21
8b270445	success	failure
9fdfac73	failure	success

The failing cell moved between JDKs while the diff stayed byte-identical. Both failures are the same assertion:

ExecutorShutdownTest.testJvmExitsAfterClosingOperator(String)[1] -- Time elapsed: 15.03 s <<< FAILURE!
Expecting actual:
  ""
to contain:
  "OPERATORS_CLOSED"

The child JVM's stdout came back empty after a 15 s wait, on a test in bindings/java that closes an Operator and checks the JVM exits. This PR touches core/core/src/raw/oio/list/flat_list.rs and core/core/src/layers/simulate.rs; nothing in the Java executor shutdown path reaches the recursive-listing code.

Happy to re-run or narrow anything if a reviewer wants a different cut of it.