Recover safely from corrupt tracked watermark state

[lan17]

Priority

P3 — Low — the stale-publication safety property is already enforced; remaining work is focused operational documentation and edge-case coverage.

v0.14.0 grooming verification (2026-08-01)

The fail-closed watermark behavior and explicit-invalidation repair path remain intact on main@34bd022. The remaining work is limited to the repair runbook plus concurrent and persistent-malformed coverage; no new repair API, background subsystem, or automatic mutation is justified. This is a candidate for not planned closure if those out-of-contract corruption cases do not merit dedicated work, but no closure decision is recorded here.

Current behavior at v0.10.0

Malformed or overflowed tracked watermarks fail safely:

• Tracked reads treat them as misses.
• Tracked writes reject without replacing the cached value.
• DialCache returns the source fallback and suppresses tracked process-local publication.
• Explicit invalidateRemote() atomically replaces malformed or overflowed contents with a fresh Redis-time watermark.
• A valid persistent watermark remains persistent after invalidation.

Current evidence:

• Fail-closed read/write scripts:

  DialCache/src/internal/redis-scripts.ts

  Lines 57 to 115 in 84341f2

  	export const READ_TRACKED_CACHE_SCRIPT = [
  	PARSE_WATERMARK_LUA,
  	READ_FRAME_LUA,
  	String.raw`local raw_watermark = redis.call("GET", KEYS[2])
  	if not raw_watermark then
  	return false
  	end
  	local watermark = parse_watermark(raw_watermark)
  	if not watermark then
  	return false
  	end
  	local created_at = struct.unpack(">I8", string.sub(value, 2, 9))
  	if created_at <= watermark then
  	return false
  	end`,
  	RETURN_PAYLOAD_LUA,
  	].join("\n\n");
  	export const WRITE_CACHE_SCRIPT = [
  	CEIL_FINITE_NUMBER_LUA,
  	VALIDATE_WRITE_ARGUMENTS_LUA,
  	REDIS_TIME_LUA,
  	WRITE_FRAME_LUA,
  	"return 1",
  	].join("\n\n");
  	export const WRITE_TRACKED_CACHE_SCRIPT = [
  	PARSE_WATERMARK_LUA,
  	CEIL_FINITE_NUMBER_LUA,
  	VALIDATE_WRITE_ARGUMENTS_LUA,
  	REDIS_TIME_LUA,
  	String.raw`local raw_watermark = redis.call("GET", KEYS[2])
  	local watermark = 0
  	if raw_watermark then
  	watermark = parse_watermark(raw_watermark)
  	if not watermark then
  	return redis.error_reply("ERR invalid DialCache watermark")
  	end
  	end
  	if watermark >= now_ms then
  	return 0
  	end`,
  	WRITE_FRAME_LUA,
  	String.raw`local desired_ttl_ms = cache_ttl_ms + ${WATERMARK_TTL_MARGIN_MS}
  	if not raw_watermark then
  	redis.call("SET", KEYS[2], "0", "PX", desired_ttl_ms)
  	else
  	local current_ttl_ms = redis.call("PTTL", KEYS[2])
  	if current_ttl_ms == -2 then
  	redis.call("SET", KEYS[2], raw_watermark, "PX", desired_ttl_ms)
  	elseif current_ttl_ms ~= -1 and current_ttl_ms < desired_ttl_ms then
  	redis.call("PEXPIRE", KEYS[2], desired_ttl_ms)
  	end
  	end`,
  	"return 1",
  	].join("\n\n");

• Repairing invalidation script:

  DialCache/src/internal/redis-scripts.ts

  Lines 117 to 156 in 84341f2

  	export const INVALIDATE_CACHE_SCRIPT = [
  	PARSE_WATERMARK_LUA,
  	CEIL_FINITE_NUMBER_LUA,
  	String.raw`local future_buffer_ms = ceil_finite_number(ARGV[1])
  	if not future_buffer_ms or future_buffer_ms < 0 then
  	return redis.error_reply("ERR invalid DialCache future buffer")
  	end`,
  	REDIS_TIME_LUA,
  	String.raw`local proposed_watermark = now_ms + future_buffer_ms
  	local raw_watermark = redis.call("GET", KEYS[1])
  	local current_watermark = 0
  	if raw_watermark then
  	local parsed_watermark = parse_watermark(raw_watermark)
  	if parsed_watermark then
  	current_watermark = parsed_watermark
  	end
  	end
  	local watermark = math.ceil(math.max(current_watermark, proposed_watermark))
  	local current_ttl_ms = -2
  	if raw_watermark then
  	current_ttl_ms = redis.call("PTTL", KEYS[1])
  	end
  	local desired_ttl_ms = math.max(
  	future_buffer_ms + ${WATERMARK_TTL_MARGIN_MS},
  	watermark - now_ms + ${WATERMARK_TTL_MARGIN_MS}
  	)
  	if current_ttl_ms > desired_ttl_ms then
  	desired_ttl_ms = current_ttl_ms
  	end
  	local encoded_watermark = string.format("%.0f", watermark)
  	if current_ttl_ms == -1 then
  	redis.call("SET", KEYS[1], encoded_watermark)
  	else
  	redis.call("SET", KEYS[1], encoded_watermark, "PX", desired_ttl_ms)
  	end`,
  	"return 1",
  	].join("\n\n");

• Real-Redis malformed read/write coverage:

  DialCache/test/redis-real.integration.test.ts

  Lines 437 to 477 in 84341f2

  	it("fails open without caching malformed watermark state", async () => {
  	if (client === undefined || admin === undefined) {
  	throw new Error("Redis test clients did not start");
  	}
  	const scriptClient = client.adapter;
  	const logger = { debug: () => undefined, warn: () => undefined, error: () => undefined };
  	const dialcache = new DialCache({ namespace: "malformed", redis: { client: scriptClient, readTimeoutMs: 10_000 }, logger });
  	let calls = 0;
  	const getUser = dialcache.cached(async (id: string) => ({ id, calls: ++calls }), {
  	keyType: "user_id",
  	useCase: "MalformedWatermark",
  	cacheKey: (id) => id,
  	trackForInvalidation: true,
  	defaultConfig: remoteOnly,
  	});
  	await admin.set("{malformed:user_id:bad}#watermark", "0x10");
  	const first = await dialcache.enable(async () => await getUser("bad"));
  	const second = await dialcache.enable(async () => await getUser("bad"));
  	expect(first).toEqual({ id: "bad", calls: 1 });
  	expect(second).toEqual({ id: "bad", calls: 2 });
  	});
  	it("rejects malformed tracked watermark writes without overwriting the cached value", async () => {
  	if (client === undefined || admin === undefined) {
  	throw new Error("Redis test clients did not start");
  	}
  	const scriptClient = client.adapter;
  	const valueKey = "malformed-write:{item:malformed}:value";
  	const watermarkKey = "malformed-write:{item:malformed}:watermark";
  	expect(await client.raw.write(valueKey, 60_000, 0, "original")).toBe(1);
  	for (const malformed of ["not-a-watermark", "9".repeat(400)]) {
  	await admin.set(watermarkKey, malformed, { PX: 60_000 });
  	await expect(client.raw.writeTracked(valueKey, watermarkKey, 60_000, 0, "replacement")).rejects.toThrow(
  	"invalid DialCache watermark",
  	);
  	expect(await scriptClient.read({ valueKey })).toBe("original");
  	}
  	});

• Real-Redis repair/persistence coverage:

  DialCache/test/redis-real.integration.test.ts

  Lines 566 to 605 in 84341f2

  	it("creates missing and repairs malformed invalidation watermarks", async () => {
  	if (client === undefined || admin === undefined) {
  	throw new Error("Redis test clients did not start");
  	}
  	const scriptClient = client.adapter;
  	const missingKey = "invalidate-paths:{item:missing}:watermark";
  	const beforeMs = (await admin.time()).getTime();
  	await scriptClient.invalidate({ watermarkKey: missingKey, futureBufferMs: 100 });
  	const created = Number(await admin.get(missingKey));
  	expect(Number.isSafeInteger(created)).toBe(true);
  	expect(created).toBeGreaterThanOrEqual(beforeMs + 100);
  	expect(await admin.pTTL(missingKey)).toBeGreaterThan(60_000);
  	for (const [suffix, malformed] of [
  	["syntax", "not-a-watermark"],
  	["overflow", "9".repeat(400)],
  	] as const) {
  	const watermarkKey = `invalidate-paths:{item:${suffix}}:watermark`;
  	await admin.set(watermarkKey, malformed, { PX: 1_000 });
  	await scriptClient.invalidate({ watermarkKey, futureBufferMs: 0 });
  	expect(Number.isSafeInteger(Number(await admin.get(watermarkKey)))).toBe(true);
  	expect(await admin.pTTL(watermarkKey)).toBeGreaterThan(59_000);
  	}
  	});
  	it("keeps persistent invalidation watermarks persistent", async () => {
  	if (client === undefined || admin === undefined) {
  	throw new Error("Redis test clients did not start");
  	}
  	const scriptClient = client.adapter;
  	const watermarkKey = "invalidate-persistent:{item:persistent}:watermark";
  	await admin.set(watermarkKey, "1");
  	await scriptClient.invalidate({ watermarkKey, futureBufferMs: 0 });
  	expect(Number(await admin.get(watermarkKey))).toBeGreaterThan(1);
  	expect(await admin.pTTL(watermarkKey)).toBe(-1);
  	});

• refactor: derive watermark TTLs from active state #88 shortened normal marker retention to derived lifetimes and documented preservation/no-eviction requirements.

The remaining failure mode is availability/load, not stale value publication: a corrupt watermark forces repeated fallbacks and tracked-write errors until it expires or is explicitly invalidated.

Remaining scope

• Document the operational repair sequence: commit the source mutation, then call invalidateRemote(keyType, id, the application's normal safety buffer).
• Add concurrent repair coverage.
• Add a persistent-malformed repair case.
• State that applications should give DialCache exclusive ownership of its namespace/key format.

Decision required before implementation

Recommended simplicity-first direction: retain fail-closed reads/writes and use explicit invalidateRemote() as the sole repair path. Do not add automatic repair, a background subsystem, another public API, or a dedicated metric unless production evidence shows the existing bounded cache_write signal and logger are insufficient.

Acceptance criteria

• No corrupt watermark can make an older cached value valid.
• Ordinary reads/writes remain fail closed.
• Explicit invalidation repairs malformed and overflowed contents atomically under concurrency.
• Persistent malformed and valid-persistent behavior is documented and tested.
• Recovery instructions preserve the source-commit and futureBufferMs ordering contract.
• Close after the focused tests and documentation land.