[kinyoklion]

Summary

Applications and editors rewrite data files in place: they truncate the file and then write the new content. A reload that reads the file in between finds it empty. The file override source applied an emptied YAML file as a document with no entries, so it removed every override from that file until the next reload. With a rewrite that pauses 300 ms after truncating, a client served a -> default -> b: on every run in watching mode, and whenever a poll fell inside the pause in polling mode. An emptied JSON file failed to parse and kept the overrides, but each rewrite that a reload caught was logged as an error (Unable to load flags: Unexpected end of JSON input). The file data source never served an empty file, but it reported each such rewrite as an error through its error handler and the log. It did the same before it moved onto the shared file data engine.

The OVERRIDE spec's retry requirement exists to keep "a partially written or briefly invalid file from emptying the layer" (Requirement 1.3.10). This change makes both consumers of the shared file data engine that reload, FileDataSource and the file override source, keep their data through such a rewrite without reporting it.

An emptied file is held back until it has stayed empty

A file has no entries when its content is white space only (a byte order mark counts as white space), or when its parser reads it as no document: a YAML parser returns null for a file with only a comment or a --- marker, which a read can see of a YAML file that is being written. {} is a document, and is applied right away.

FileReloader now remembers, for each configured file, whether it contributed content to the data in effect, that is, to the last reload that was applied or skipped as unchanged; a missing file, or one with no entries, contributed none. For a file whose content is in effect and that reloads have found with no entries, it remembers since when the file has been so without being written, the file's state then (its modification time and size as the platform reports them, which the Node platform reads through any symbolic link, with the poller's file state helper), and when reloads began holding the file back. The clock is read after each file's own read, so that a slow read of another file does not shorten its delay.

A reload that finds such a file with no entries keeps the current data, logs at debug level (File ... is empty, possibly while it is being rewritten; reading it again before applying it), reports no failure, and arms a retry, whatever the format. The file is held back until it has stayed so, without being written, for exactly the retry delay (1 second):

• A change of the file's state means that it was written since a reload last found it empty, for example rewritten and truncated again before any reload read the new content, so its delay starts over. States are only compared with each other, never with the clock.
• A file is held back for at most five retry delays. This bounds a file that keeps being written while every reload finds it empty, and files that are rewritten in turn so that every reload finds one of them empty. The bound is the file's own: it starts at the first reload that holds the file back, is kept across reads with content and across failed reloads, and starts over when data is applied or when the file has stayed empty for the whole delay. So another file that keeps failing does not restart it, and one file reaching its bound does not cut short another file's ordinary rewrite.
• The hold's retry is armed for when the first held file's delay or bound ends, plus a tenth of the retry delay, so that a timer that fires a little early still finds the time over. A further change signal while the file stays empty and unwritten does not apply it.
• A reload that holds back one file still reads and parses the others, so that files emptied together start their delays in the same reload and are applied at the same retry. A file that cannot be read or parsed still fails the reload. Changes to the other files wait while a file is held back, because a reload applies all of the files or none of them.
• A file whose content is not in effect is never held back: a placeholder that has always been empty, a file whose empty state has already been applied, and, where a missing file contributes nothing (the override source), a file that was deleted and then created empty. Such a file never delays another file's change.
• The rule applies to every reload: a change signal, a poll, a change to another file while this one is mid-rewrite, and a retry left over from an earlier failure.
• Holding back is not a failure. It does not make the next success a recovery, so a rewrite with the same content applies nothing. The retry after a hold reports a failure the way the reload that held the file back would have: a failure after a change notification is news, and only a failure after a retry can be a repeat.
• With retries off (FileDataSource without autoUpdate, and the FDv2 file initializer, which both load once), nothing would read the file again, so an empty file is parsed right away.

A rewrite is covered when the time between truncating and writing is shorter than the retry delay. A writer that pauses longer gets an empty YAML override file applied, or an empty JSON file, or any empty FileDataSource file, reported as a load failure while the data in effect stays. A rewrite within the resolution of the file's modification time that keeps its size is not seen as a write, so its delay does not start over.

The commits show how the rule got here. The first read an emptied file again once, keyed on whether the last read found content; that still applied an empty file when no reload read the content between two truncations. The second and third hold a file back by its state, with a cap per file and a slack of a tenth in the delay check. The fourth applies the review findings: the cap per file let files rewritten in turn be held back without end, the slack let a reload apply a file that had been empty for less than the delay, a YAML read with only a comment was applied as no entries, and a reload that skipped unchanged content did not remember which file contributed it, so after a file moved to another configured path an in-place rewrite of it was applied empty. The fifth adds tests for a filesystem whose modification times are coarse. The sixth and seventh apply a second review: the clock read once per reload let a slow read of an earlier file shorten a later file's delay, a failed reload restarted the shared bound so that a file kept written was held without end while another file kept failing, and the shared bound let one file reaching it apply another file a fraction of a delay into an ordinary rewrite; it reads the clock after each file's read and bounds each file on its own. It is the rule of dotnet-core #394.

An empty JSON file says what is wrong

The parser for a file is chosen by its extension, so whether an empty file is valid is up to the format. An empty or white-space JSON file was already invalid: JSON.parse rejects it and the last good data stays. It now fails with the file has no content, which is not a JSON document (still a SyntaxError) instead of Unexpected end of JSON input. This applies to a file with no extension and to every extension other than .yml and .yaml, since those files are parsed as JSON. A YAML parser's null or undefined for an empty file is still a document with no entries in the override source.

Documentation

• FileOverrideSourceOptions says that a file can be rewritten in place, how to remove a file's overrides (delete it or write {}), that a file whose content is in effect and that is found empty, or with only a YAML comment or document marker, is held back for about a second, and for at most about five seconds while that file keeps being rewritten, that changes to the other files wait with it, and that after that an empty YAML file removes its overrides while an empty JSON file fails to load and the overrides in effect stay. Its yamlParser says what a null or undefined result means.
• FileDataSourceOptions.autoUpdate says that a file can be rewritten in place, that an emptied file is held back in the same way, and that a file that is still empty then fails to load, which is reported, while the previously loaded data stays in effect.
• Both recommend replacing a YAML file in one operation, for the first reason below.

Not changed

• YAML cut off between two entries. A read that overlaps a write can see the first part of a YAML file. If the cut falls between two entries, the part is valid YAML with fewer entries, and it is applied until the next reload. JSON cut off anywhere fails to parse. This predates this change.
• An empty YAML file in FileDataSource and the FDv2 file initializer. Their policy hands a YAML parser's null to the merge, which fails with Cannot read properties of null (reading 'flags'). An empty YAML data file therefore fails to load and the last good data stays, as before the shared engine. With this change, an emptied file, or one with only a comment, is held back before that failure is reported. Making it a document with no entries, as it is for override files, changes released behavior, so it is left for a separate change.
• The case of the extension. The YAML extension check is case-sensitive, so flags.YAML is parsed as JSON, as it always was.

Testing

• Parser: an empty, white-space or byte-order-mark-only JSON file is rejected with the new message, including a file with no extension or another extension. A YAML parser's null or undefined is returned as it is, and the override document parser makes it a document with no entries.
• Reloader (FileReloader.inPlaceRewrite.test.ts, mock filesystem with a modification time on every write, Jest's fake clock):
  • An emptied file (empty, white space, byte order mark) is held back, and the rewritten content is applied with no empty application in between; the debug line is logged.
  • A YAML file with only a comment, a --- marker, or both is held back; a file with {} is applied right away.
  • A file that stays empty is applied by the hold's retry, a tenth of a retry delay after its delay ends, and not before.
  • A load at 95 % of the retry delay still holds the file back, and a load at the delay applies it.
  • An emptied JSON file is not reported by the load that finds it empty; the retry reports it with the new message, and the data stays.
  • A further change signal does not apply a file that stayed empty, and the retry it arms comes when the delay that started at the first load ends.
  • Two quick rewrites: the write and the second truncation between two loads restart the delay, so a load after the first delay would have ended still holds the file back.
  • Several files: a change to another file while one file is mid-rewrite does not apply the emptied file.
  • Polling-like: the retry that finds the truncation of a later rewrite does not apply it.
  • The bound, with one file kept written so that every load finds it empty, and with two files rewritten in turn: they are applied at the retry when five retry delays have passed since the file was first held back.
  • Another file that fails at every other load does not restart the bound of a file kept written.
  • One file reaching its bound does not apply another file that is a quarter of a retry delay into an ordinary rewrite.
  • A slow read of an earlier file (900 ms) does not shorten a later file's delay.
  • A fresh bound or delay: for a file emptied again after its content was applied, for a JSON file that has been empty and failing for long and is then rewritten in place (no extra report), and for a file that is deleted and created empty again. With coarse modification times: a file emptied again after its content was read, applied or in a failed load, and a file deleted and created empty again; and a file emptied again after a read of its content, with its modification time set back to the first emptying's, while another file kept written keeps the load from applying.
  • The delay of the retry that a hold asks for: a retry delay and a tenth after the first hold; the rest of the delay after a later hold; the bound when that is sooner; the file whose delay ends first, configured first or second; the same for a hold by a retry; and a retry delay after a failure.
  • Content that moved from one configured file to another, with the contents unchanged, is held back when the file it moved to is rewritten in place.
  • An empty placeholder file is never held back and does not delay a change to another file; a file whose empty state is in effect is not held back again; with missing files skipped, a file deleted and then created empty is applied right away.
  • Two files rewritten together are each held back; two files emptied together that stay empty are applied at the same retry; a file that fails to parse fails the load even when another file is held back.
  • A retry left over from a failure that finds the file emptied holds it back instead of applying it empty; with retries off, an emptied file is applied right away.
  • Holding back is not a failure: a rewrite with the same content applies nothing.
  • A file whose state cannot be read is still held back, and its retry still comes.
  • Reporting after a hold: a failure after a hold that followed a change is news; after a hold by a retry, the next retry's failure is a repeat, and a change signal's failure is news.
  • A reload that completes after close does not hold anything back.
• End-to-end in server-node, with real in-place rewrites (truncate, pause, write; rewriteInPlace in __tests__/inPlaceWrites.ts) while the test samples evaluations:
  • A client with FileDataSource and autoUpdate serves a -> b and reports no error, for JSON and YAML, with a 300 ms pause; and never serves the default and reports no error through two rewrites 20 ms apart.
  • A client with a file override source serves a -> b and logs no load failure, for JSON and YAML, in watching mode (300 ms pause) and in polling mode at the default 1 second interval. In polling mode the rewrite starts 750 ms after a poll and pauses 600 ms, so the next poll finds the file empty.
  • The same client never serves the default and logs no load failure through two rewrites in a row: in watching mode 20 ms apart, and in polling mode with the second truncation 1900 ms after a poll, so that the next poll and the retry find the file empty again before any load read the first rewrite's content.
• On main, all 12 end-to-end tests fail: the YAML override file serves default in between, and the JSON override file and both data files log an error. 46 of the new unit and parser tests fail too. On the first commit of this branch, the 6 end-to-end tests with two rewrites in a row fail, and 27 unit tests; on the third commit, 20 unit tests; on the fifth commit (the run cap), exactly the 3 tests for the second review: another file failing, another file's bound, and the slow read.
• Mutants: 38 mutants, each reverting or changing one part of the change on its own with a clean rebuild. 37 fail at least one of the new tests: no hold; hold whatever the data in effect; the data in effect not remembered after an apply, after a skipped unchanged load, or for an empty or missing file; the bounds kept after an apply; the delay kept after a read with content or a missing file; the bound cleared by a read with content; no restart on a change of state; the bound kept after a file stayed empty for the whole delay; no bound; the bound restarted by each hold; a slack in the delay check; the bounds cleared by a failure; the hold's retry armed for a full delay, without the margin, ignoring the bound, or decided by the last or the first held file; YAML with no document not counted as empty; {} counted as no document; the override parser returning {} for no document; stop reading after a held-back file; a hold that hides another file's failure; holding back counted as a failure; no retry armed; no carry-over of the reason, a carry-over to any reload, or always the reason of a change; a file state lookup that throws; no check for close; either debug line removed; the default JSON.parse message for an empty JSON file; white space counted as content. The one that survives drops the explicit retryDelayMs <= 0 check from the hold, which is equivalent: with no retry delay the delay is zero long.
• Loops: the new unit and parser test files (82 tests) passed 15 runs in a row, and the two server-node test files with the end-to-end tests (20 tests) passed 12 runs in a row, on a loaded machine. The polling end-to-end tests take 3 to 4 seconds, so the override end-to-end tests have a 15 second timeout.
• @launchdarkly/js-server-sdk-common and @launchdarkly/node-server-sdk: full test suites, lint, format:check, build and build:doc pass locally.

_{Opened from termag}

---

Note

Overview
File reloads no longer apply or error on transient empty reads when a data or override file is truncated mid-rewrite. FileReloader remembers which paths contributed active content and holds back files that suddenly have no entries (blank bytes, YAML with only comments/markers, or parser null) until they stay unchanged for about one retry delay, capped at five delays while rewrites keep happening. Other configured files wait on the same reload; retries are scheduled without treating the hold as a load failure.

Parsing and empty-document semantics are tightened: hasContent and NO_ENTRIES distinguish “no document” from {}; empty JSON now fails with the file has no content, which is not a JSON document instead of a generic parse error; override YAML null/undefined maps to NO_ENTRIES rather than an immediate empty apply.

Docs for file data sources and file overrides describe in-place rewrite behavior and YAML atomic-replace guidance. Tests add a large FileReloader fake-clock suite, shared rewriteInPlace / sampleValues helpers, and Node SDK end-to-end coverage for FileDataSource and file overrides in watching and polling modes.

^{Reviewed by Cursor Bugbot for commit e8a950c. Bugbot is set up for automated code reviews on this repo. Configure here.}

[kinyoklion]

bugbot review

[github-actions]

@launchdarkly/js-sdk-common size report
This is the brotli compressed size of the ESM build.
Compressed size: 32220 bytes
Compressed size limit: 32500
Uncompressed size: 152548 bytes

[github-actions]

@launchdarkly/js-client-sdk size report
This is the brotli compressed size of the ESM build.
Compressed size: 34474 bytes
Compressed size limit: 35000
Uncompressed size: 122974 bytes

[github-actions]

@launchdarkly/js-client-sdk-common size report
This is the brotli compressed size of the ESM build.
Compressed size: 26066 bytes
Compressed size limit: 44000
Uncompressed size: 168326 bytes

[cursor]

Stale Bugbot comment from a previous run.

[kinyoklion]

bugbot review

[cursor]

Stale Bugbot comment from a previous run.

[kinyoklion]

bugbot review

[cursor]

Stale Bugbot comment from a previous run.

[kinyoklion]

bugbot review

[cursor]

Stale Bugbot comment from a previous run.

[kinyoklion]

bugbot review

[cursor]

Stale Bugbot comment from a previous run.

[kinyoklion]

bugbot review

[cursor]

Stale Bugbot comment from a previous run.

[kinyoklion]

bugbot review

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit e8a950c. Configure here.}