diff --git a/.gitignore b/.gitignore index c116594..9eb4779 100644 --- a/.gitignore +++ b/.gitignore @@ -2,3 +2,4 @@ .vs .pack .claude/*.local.json +*.cka diff --git a/CLAUDE.md b/CLAUDE.md index 31fa598..a70cf56 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -13,10 +13,10 @@ When answering a question keep the following rules in mind: - Code examples are helpful, but they do not have to be complete or to compile. Use a comment like `// snip` to elude boilerplate code. ## Code -If you are asked to generate code,keep changes as small as possible. Focus on creating small, maintainable and easy to understand code. Work in small iterative steps and ensure the test in `Caprikit.Tests` cover the happy path of any code generated and follow the test guidelines found in `source\CapriKit.Tests\README.md`. When generating interop code for C or C++ libraries try to create stateless (functional style) code that mimicks the API of the native library but hides native pointers from the public C# API. Double check that any collection or other action by the garbage collector does not invalidate pointers. +If you are asked to generate code,keep changes as small as possible. Focus on creating small, maintainable and easy to understand code. Work in small iterative steps and ensure the test in `Caprikit.Tests` cover the happy path of any code generated and follow the test guidelines found in `source\CapriKit.Tests\README.md`. When generating interop code for C or C++ libraries try to create stateless (functional style) code that mimics the API of the native library but hides native pointers from the public C# API. Double check that any collection or other action by the garbage collector does not invalidate pointers. ## Documentation -When generating documentation focus on the target audiance: The code maintainers and library users. Write XML documentation to educate the library users on how to use a class, method, property or other component correctly and in which situations it should be used or not be used. Sparingly use in-line comments to explain gotcha's, non-obvious behavior or future points of improvemnts to code maintainers. Remember that documentation also needs to be maintained so focus on writing short to-the-point documentation that is easy for me to maintain. +When generating documentation focus on the target audience: The code maintainers and library users. Write XML documentation to educate the library users on how to use a class, method, property or other component correctly and in which situations it should be used or not be used. Sparingly use in-line comments to explain gotcha's, non-obvious behavior or future points of improvements to code maintainers. Remember that documentation also needs to be maintained so focus on writing short to-the-point documentation that is easy for me to maintain. ## Research In some cases I want to follow-up later on an interesting architecture, coding pattern or tooling suggestion. If I ask you to store or save such an idea for later, create a markdown document in the `research` folder where you explain the suggestion and summarize what we were working on when you suggested it. @@ -27,4 +27,4 @@ Assume you can only use Powershell to execute commands. The `dotnet` tool and la The `external` folder in the root of the repository contains git submodules that point to external repositories. You can depend on code in these external repositories but you can never make changes to them. ## Technologies -See `Directory.Packages.props` for an overview of NuGet packages used. \ No newline at end of file +See `Directory.Packages.props` for an overview of NuGet packages used. diff --git a/CapriKit.slnx b/CapriKit.slnx index 818c7eb..2851bfd 100644 --- a/CapriKit.slnx +++ b/CapriKit.slnx @@ -18,6 +18,7 @@ + @@ -32,6 +33,12 @@ + + + + + + diff --git a/Directory.Packages.props b/Directory.Packages.props index f0c18e1..5620115 100644 --- a/Directory.Packages.props +++ b/Directory.Packages.props @@ -12,12 +12,15 @@ + + + - \ No newline at end of file + diff --git a/PLANNING.md b/PLANNING.md new file mode 100644 index 0000000..9ef3b95 --- /dev/null +++ b/PLANNING.md @@ -0,0 +1,12 @@ +# Planning +_Ideas and future plans for the projects in this repository_ + +## Soon +- Finish integrating hot-reloading in the asset pipeline +- Test hot reloading in the live test project +- Clean up an asset-pipeline code and merge branch + +## Later +- Add logging support via `Microsoft.Extensions.Logging.Abstractions` to most projects +- Add dependency injection support via `Microsoft.Extensions.DependencyInjection.Abstractions` to most projects and use a LightInject extension/implementation in the demo project +- Add metrics support (if needed) via `System.Diagnostics`: `.Activity`, `.Meter`, `.Counter`, which are Open-Telemetry compatible. diff --git a/research/Polling AssetBundleLoader IsReady.md b/research/Polling AssetBundleLoader IsReady.md new file mode 100644 index 0000000..7696ee7 --- /dev/null +++ b/research/Polling AssetBundleLoader IsReady.md @@ -0,0 +1,238 @@ +# Polling AssetBundleLoader.IsReady + +_Written 2026-08-20, while going through `CapriKit.AssetPipeline` on the `feature/asset_pipeline` branch. We +had just finished the `AssetPool` tests and the fix for `PutOrLease` handing out a disposed asset when the +same instance is put twice. The question that came up next: `AssetBundleLoader.IsReady` is polled +every frame and walks all its handles, and bundles are expected to hold around 40 assets._ + +## What the current code costs + +```csharp +foreach (var handle in handles) +{ + if (!handle.IsResolved) { value = default; return false; } +} +``` + +The loop already stops at the **first** unresolved handle, so it never walks the whole bundle unless the +bundle is nearly done. What it repeats every frame is the prefix of handles that resolved earlier: a +pointer dereference plus a volatile bool read each, over a `List` of references that are +scattered across the heap. + +Worst case per poll is therefore ~40 dependent loads, so tens of nanoseconds (an estimate from the shape of +the loop, not a measurement). At 60 Hz with a handful of bundles in flight that is microseconds per second. + +**So none of the proposals below are a rescue of a hot loop.** Take one because it is simpler than what is there +now, or because it gives you the progress information the `// TODO` in `AssetBundleLoader.cs` asks for. + +## The invariant all three proposals lean on + +`AssetHandle.Resolve` asserts it is only called once and only ever flips `isResolved` from false to true. +A handle that is resolved stays resolved. "Resolved" is monotone, which is what makes it safe to remember +progress between polls without any notification from the handle side. + +## Proposal A: BitArray plus a resume index + +The idea as originally sketched: keep one bit per handle and resume from the first handle that was not +ready last time. Written out, and extended so the bits actually earn their keep by also counting progress: + +```csharp +private readonly BitArray resolved = new(handles.Count); +private int resolvedCount; +private int cursor; // every handle before this one is resolved + +public int Total => handles.Count; +public int Loaded => resolvedCount; + +public bool IsReady([NotNullWhen(true)] out TBundle? value) +{ + if (isReady) { value = result!; return true; } + + // Only look at handles that were not resolved yet, the bits let us skip the ones that were. + for (var i = cursor; i < handles.Count; i++) + { + if (resolved[i] || !handles[i].IsResolved) { continue; } + + resolved[i] = true; + resolvedCount++; + } + + // Everything up to the cursor is resolved, so the next poll can start there. + while (cursor < handles.Count && resolved[cursor]) { cursor++; } + + if (resolvedCount < handles.Count) { value = default; return false; } + + result = factory(new AssetHandleResolver(this)); + isReady = true; + value = result; + return true; +} +``` + +Cost per poll: one bit read per handle that resolved out of order, one volatile read per handle that is +still pending. Both shrink as the bundle loads. Total work over the bundle's life is O(n) plus one pass +over the shrinking tail per poll. + +Notes: +- A `bool[]` is the better container at this size: 40 bytes instead of 8, but no masking and no extra type. + `BitArray` only starts paying off in the thousands of handles. +- Without the counting you can drop the bits entirely, see proposal B for why. +- If you want progress but not the extra arrays, see proposal C. It shrinks the list instead of the scan + range, and it ended up dominating this proposal. + +## Proposal B: resume cursor only (recommended for a yes/no IsReady) + +If `IsReady` stays a yes/no question, one integer replaces the whole thing: + +```csharp +private int cursor; // first handle that was not resolved yet + +public bool IsReady([NotNullWhen(true)] out TBundle? value) +{ + if (isReady) { value = result!; return true; } + + // Handles never go back to unresolved, so everything before the cursor stays resolved and + // each poll only looks at the handles that were still pending during the previous poll. + while (cursor < handles.Count && handles[cursor].IsResolved) { cursor++; } + + if (cursor < handles.Count) { value = default; return false; } + + result = factory(new AssetHandleResolver(this)); + isReady = true; + value = result; + return true; +} +``` + +This is a two line change to the current implementation and it is *less* code than proposal A, not more. + +Cost per poll: one comparison plus one volatile read per handle that resolved since the previous poll. +Summed over the bundle's life that is n reads plus one comparison per poll, which is as good as it gets +without the handles pushing. + +**Why the bits drop out.** Once you also keep a cursor, the loop returns at the first unresolved handle, so +it never looks past it, so it never sets a bit past it. The bits are therefore always "set below the +cursor, clear at and above it", which is exactly what the cursor already says. The BitArray is redundant +unless something makes you keep scanning past the first unresolved handle, and only progress counting does +that. That is the whole difference between A and B. + +**Progress.** `cursor / (float)handles.Count` is a lower bound: monotone and never jumps backwards, so it +drives a progress bar fine, but it under-reports while an early asset is slow. An accurate `37 of 40` +needs proposal C. + +## Proposal C: shrink the pending list + +Instead of remembering where to resume, throw away what is done. The loader keeps its own list of handles +it is still waiting on and swap-removes each one as it resolves: + +```csharp +// A copy: the builder still owns the list it handed us. +private readonly List pending = [.. handles]; +private readonly int total = handles.Count; + +public int Total => total; +public int Loaded => total - pending.Count; +public IReadOnlyList Pending => pending; + +public bool IsReady([NotNullWhen(true)] out TBundle? value) +{ + if (isReady) { value = result!; return true; } + + // Backwards, so that swapping the last handle into the gap cannot skip a handle we still have to see. + for (var i = pending.Count - 1; i >= 0; i--) + { + if (!pending[i].IsResolved) { continue; } + + pending[i] = pending[^1]; + pending.RemoveAt(pending.Count - 1); + } + + if (pending.Count > 0) { value = default; return false; } + + result = factory(new AssetHandleResolver(this)); + isReady = true; + value = result; + return true; +} +``` + +Cost per poll: exactly one volatile read per handle that is still pending, and the scan shrinks as the +bundle loads. That is the price of an exact count. Proposal B can stop at the first unresolved handle +because it only has to answer yes or no, this one has to look at all of them to know how many arrived. + +Notes: +- **The copy is required.** `AssetBundleBuilder.Build` passes its own `Handles` list straight into the + constructor, so compacting it in place would corrupt a second bundle built from the same builder. + (`Build` already re-points every `handle.Owner`, so building twice is not really supported today, but + this proposal should not be the thing that breaks it.) +- Mentioning `handles` only in field initialisers means the primary constructor does not capture it, so + after the copy the loader no longer keeps the builder's list alive. +- It is the only variant that can say *what* it is waiting on: `pending` is exactly the set of assets that + have not arrived, which is what a loading screen or a debug overlay wants to show. +- The swap-remove destroys the order. Nothing reads it today (`AssetHandleResolver` goes through the handle, + not through this list). If that changes, walk forwards and compact with a write index instead, same cost. +- Keeping the resolved handles in a second list gives the same progress number for another allocation and + an `Add` per handle. Nothing in the pipeline consumes them, so I would leave that second list out until + something does. +## Which one + +| | A: bits + cursor | B: cursor | C: shrinking list | +|---|---|---|---| +| Progress | exact count | lower bound | exact count | +| Says what is still pending | no | no | yes | +| Extra state | `bool[]` + count + cursor | cursor | list copy | +| Allocates per bundle | `bool[n]` | nothing | list of n references | +| Reads per poll | one per pending handle, plus a bit per handle that resolved out of order | one while blocked, then one per newly resolved handle | one per pending handle | +| Lines vs. today | ~+12 | ~+2 (and one `foreach` removed) | ~+6 | + +Go with **B** while `IsReady` stays a yes/no question, and with **C** as soon as you want a real progress +number. B is by far the cheapest to poll: while it is blocked on one handle it reads that one handle and +returns, where A and C have to look at everything still pending to keep their count exact. + +**C ended up dominating A**, which is worth writing down since A is where this started. It has fewer moving +parts, it never rescans the resolved-but-out-of-order handles that A's bits exist to skip, and it hands you +the pending set for free. A's only edge is that it leaves the handle list alone, which buys about 320 bytes +per bundle. A stays in this document for the reasoning, not as a candidate. + +Optional add-on, composable with any of them: let `AssetManager` bump a `ResolveGeneration` counter whenever +`Update` resolves anything, and have the loader remember the generation it last saw. A poll on a frame +where nothing resolved at all then returns false after a single int comparison, for every bundle at once. +Costs the loader a reference to the manager, so only worth it if many bundles are in flight. + +## Considered and rejected: let the handles push + +The obvious O(1) answer is a countdown that `AssetHandle.Resolve` decrements through the `Owner` it already +carries: + +```csharp +internal void Resolve(object asset) +{ + value = asset; + isResolved = true; + Owner?.OnHandleResolved(); // remaining-- +} +``` + +This does not survive contact with the loading path, for two reasons: + +1. `AssetManager.Load` resolves straight from the cache while `AssetBundleBuilder.Load` is still running, + which is *before* `AssetBundleBuilder.Build` assigns `handle.Owner`. Every cache hit is a lost + decrement. `Build` would have to count the handles that are already resolved and seed the countdown. +2. That seeding then races: `CreateBundle`, `Load` and `Build` are documented as callable from any thread, + while `Resolve` runs on the main thread inside `Update`. A handle can resolve between `Build` reading + `handle.IsResolved` and `Build` writing `handle.Owner`, which either loses the decrement or counts it + twice. + +Closing that means assigning `Owner` and seeding the count under the `AssetManager.RequestLock`, so the +bundle builder starts depending on the manager's lock and the loading path grows a lock section, all to +save the ~40 bool reads per frame we started with. This is the complexity that was rightly suspected up +front, and the reason all three proposals above stay on the polling side. + +## Related TODOs in the same file + +- `// TODO: how can we put a sort of progress bar and progress information on this thing?` is exactly the fork above: + B if a lower bound that never jumps backwards is enough, C if you want to show a real count. +- `// TODO: Add a method to block and wait without eating all the CPU` is a different problem. None of the + proposals help: blocking needs something to wait on, and the only thing that could signal it is the main + thread inside `AssetManager.Update`, so it would have to be a `ManualResetEventSlim` (or a + `TaskCompletionSource`) that `Update` sets once the bundle's last handle resolved. Worth its own note. diff --git a/research/Recovering from a failed asset load.md b/research/Recovering from a failed asset load.md new file mode 100644 index 0000000..82003f7 --- /dev/null +++ b/research/Recovering from a failed asset load.md @@ -0,0 +1,263 @@ +# Recovering from a failed asset load + +_Written 2026-08-25, on the `feature/asset_pipeline` branch. We had just added +`AssetManagerTests.Load_RetriesAnAssetWhoseFirstLoadFailed`, which fails on purpose, to keep this bug from +being forgotten. This note is the plan for fixing it properly._ + +## What is actually broken + +Three separate defects, all in the same handful of lines. Only the first one is the wedge, but a fix that +does not address the other two just moves the problem. + +**1. The request is never cleared.** `AssetManager.Update` only removes the `Outstanding` entry on the +success path: + +```csharp +while (Incoming.TryRead(out var result)) +{ + var (id, trackAndTakeLease) = result; + var handles = Outstanding[id]; + // snip: resolve every handle + Outstanding.Remove(id); // never reached when the request failed +} +``` + +A failure travels through a *different* route: `LightweightChannel.Write(ExceptionDispatchInfo)` puts it in +a second queue, and `TryRead` drains that queue first and throws. So `Update` throws before it can clean +anything up, `Outstanding[id]` survives, and because `Load` hands every later request for that id to the +existing list instead of starting a new one, the asset can never be loaded again. + +**2. The error does not know which asset it belongs to.** The exception queue in `LightweightChannel` is +untyped and separate from the item queue, so by the time `Update` catches it there is no id to clean up +even if we wanted to. This is why the fix cannot be local to `Update` as it stands. + +**3. A failure takes down more than the asset.** `Update` throws from inside the lock, which skips +`Cache.DisposeReleased()` and `HotReloadManager.Update()` for that frame, and abandons any *successful* +loads still sitting in the queue behind the failure. + +## The half that is not a choice + +Whatever policy we pick, the cleanup has to happen **on the main thread, inside the same `RequestLock` +section that `Load` uses**, and the failure has to carry its id to get there. Route both outcomes through +the one queue: + +```csharp +// One item per finished request, successful or not, so Update sees both the same way. +internal readonly record struct LoadResult(AssetId Id, Func? Materialize, ExceptionDispatchInfo? Error); +``` + +`Load`'s failure handler stops using the channel's exception queue: + +```csharp +Task.Run(() => RequestAsset(id, settings)).FireAndForget( + ex => + { + LogFailed(Logger, id); + Incoming.Write(new LoadResult(id, null, ex)); // was: Incoming.Write(ex) + }); +``` + +and `Update` clears the entry for both outcomes: + +```csharp +lock (RequestLock) +{ + while (Incoming.TryRead(out var result)) + { + // Removing here is the whole fix: a handle either made it into this list before we took the lock, + // or it misses the list entirely and its Load starts a fresh request. + if (!Outstanding.Remove(result.Id, out var handles)) { continue; } + + if (result.Error is not null) { /* policy, see below */ continue; } + + foreach (var handle in handles) { handle.Resolve(result.Materialize!()); } + } +} +``` + +After this, `LightweightChannel.Write(ExceptionDispatchInfo)` is unused by the asset pipeline. Leave the +overload alone (it is a general-purpose primitive), but know that this path no longer relies on it. + +## The half that is a choice: what the app sees + +All three options below build on that same core. **A and B both give you the "proper exception that quits +the app if uncaught" behaviour** you asked for; they differ in *where* it is thrown. + +### Option A: `Update` throws + +```csharp +if (result.Error is not null) +{ + // The entry is already gone, so a later Load starts a clean request even if somebody catches this. + throw new AssetLoadException(result.Id, result.Error.SourceException); +} +``` + +Smallest possible diff on top of the core: no changes to `AssetHandle` or `AssetBundleLoader`. The handles +of the failed request are simply left unresolved forever. + +That last part is the catch. It is fine while the exception is uncaught and the process dies, but an app +that catches it is left with a bundle that never completes and no way to find out why — a silent hang. +Option A is therefore *only* correct as a fail-fast policy; catching it is unsupported. It also keeps +defect 3: the throw still skips `DisposeReleased` and the hot-reload update for that frame. + +### Option B: the handle carries the failure, `IsReady` throws (recommended) + +Give `AssetHandle` an error next to its value. Writing `error` before the `volatile` `isResolved` gives the +same release ordering the existing `value` write relies on: + +```csharp +private Exception? error; +internal Exception? Error => error; + +internal void Fail(Exception ex) +{ + Debug.Assert(isResolved == false); + error = ex; + isResolved = true; // volatile write publishes error too +} +``` + +`Update` fails the handles instead of throwing, so it stays a pure bookkeeping call: + +```csharp +if (result.Error is not null) +{ + foreach (var handle in handles) { handle.Fail(result.Error.SourceException); } + continue; +} +``` + +and `AssetBundleLoader.IsReady` treats a failed handle as arrived, then throws once everything is +in: + +```csharp +for (var i = Pending.Count - 1; i >= 0; i--) +{ + var handle = Pending[i]; + if (!handle.IsResolved) { continue; } + if (handle.Error is not null) { (failures ??= []).Add(new AssetLoadException(handle.Id, handle.Error)); } + // snip: existing swap-remove +} + +if (Pending.Count > 0) { value = default; return false; } + +// Everything arrived but some of it failed. Throwing here puts the error in front of the code that +// actually wanted the asset, rather than in the middle of the manager's bookkeeping. +if (failures is not null) { throw failures.Count == 1 ? failures[0] : new AggregateException(failures); } +``` + +`IsReady` is called from the game loop, so an uncaught `AssetLoadException` still takes the process down — +your requirement is met. What you gain over A is that `Update` never throws, so the cache collection and +the hot-reload pass always run, and the failure is attributed to a specific bundle and asset. + +Note that `IsReady` will throw again on every subsequent call, since `Pending` is empty but `isReady` was +never set. That is the right idempotent behaviour, but it means a caught exception throws once per frame. + +### Option C: the loader reports, nothing throws + +Same `AssetHandle.Fail` as B, but the loader exposes the failure instead of throwing: + +```csharp +public bool IsFaulted => Failures.Count > 0; +public IReadOnlyList Failures => failures ?? []; +``` + +`IsReady` returns false forever while faulted. This is what you want the day an app wants to substitute a +placeholder asset and carry on. It is the wrong *default*: an app that forgets to check `IsFaulted` sits on +a loading screen forever with nothing but a log line to show for it. + +Worth building **on top of** B rather than instead of it — B can grow `IsFaulted` and a non-throwing +`TryGetResult` later without changing what the throwing path does. + +## The load / hot-reload asymmetry + +Good news: the hot-reload half already behaves the way you want, and no option above changes it. +`HotReloadManager.FinishReload` catches everything, logs it, keeps the live asset's contents and returns +the lease in a `finally`; the faulted task's `.Exception` is read by `LogReloadFailed`, which marks it +observed so it never reaches `TaskScheduler.UnobservedTaskException`. Hot reload does not use `Incoming` at +all, so failures there cannot leak into the load path. + +What the fix should add is the *statement* of that split, because the same transcoder bug now produces a +hard crash in one path and a log line in the other, and that looks arbitrary until you say why: + +- **Load**: there is no valid asset to fall back on, so the app cannot sensibly continue. Fail fast. +- **Hot reload**: there is a perfectly good asset already live, and this is a development-time convenience. + Never take down the game over it. + +Concretely: document that contract on `IAssetTranscoder.Encode` and `.Decode`, and add a +regression test that a hot-reload failure does not throw out of `AssetManager.Update()` (today +`HotReloadManagerTests.Update_RebuildFails` only exercises `HotReloadManager` directly, never through the +manager). Under option A that test is load-bearing, because `Update` becomes a method that sometimes throws. + +## Side cleanup worth folding in + +`RequestAsset` calls `GetTranscoder()` on the thread pool, so a missing or mismatched +transcoder — a programmer error, not an asset failure — arrives through the same failure path as a corrupt +file. Hoisting that call into `Load` makes it throw synchronously, on the calling thread, with a clean +stack, before any `Outstanding` entry exists. Small and independent of the options above. + +## Open question: retrying a permanently broken asset + +Once the wedge is gone, every `Load` of a broken asset starts a fresh build. In practice `Load` is called +once per bundle rather than once per frame, so this is unlikely to become a rebuild storm — but nothing +stops it either. If it ever bites, the cheap answer is to remember failed ids with a timestamp and refuse to +retry within a few seconds, or to require an explicit `assetManager.Forget(id)`. Not worth building yet. + +## Which one + +| | A: `Update` throws | B: `IsReady` throws | C: loader reports | +|---|---|---|---| +| Wedge fixed | yes | yes | yes | +| Kills the app if uncaught | yes | yes | no | +| Error names the bundle and asset | id only | yes | yes | +| `Update` stays non-throwing | no | yes | yes | +| Cache collect + hot reload still run that frame | no | yes | yes | +| App can recover deliberately | no (silent hang) | no | yes | +| Touches | `Update` | `Update`, `AssetHandle`, `AssetBundleLoader` | B, plus loader API | + +Go with **B**. It gives you the fail-fast crash you asked for, but throws it where the app asked for the +asset instead of in the middle of the manager's bookkeeping, which keeps defect 3 fixed as well. A is worth +knowing about as the two-line version if you want the wedge gone today and the rest later. C is the natural +follow-up once something actually wants to survive a missing asset. + +## Test impact + +`Load_RetriesAnAssetWhoseFirstLoadFailed` goes green on `Attempts == 3` under all three options. Its last +assertion is the one that changes: + +- **A**: unchanged — the failed bundle's handles stay unresolved, so `IsReady` is still `false`. +- **B**: becomes `await Assert.That(() => failedLoader.IsReady(out _)).Throws();` +- **C**: `IsReady` stays `false` and `failedLoader.IsFaulted` is `true`. + +Two tests worth adding while in here: + +1. A hot-reload failure does not throw out of `AssetManager.Update()` (the asymmetry above). +2. Two concurrent `Load` calls for the same failing id both get failed, and neither is silently dropped — + this is the race that the naive fix gets wrong, see below. + +## Considered and rejected: clearing `Outstanding` from the thread pool + +The obvious two-line fix, and the one I probed with while checking that the new test can go green: + +```csharp +ex => +{ + LogFailed(Logger, id); + lock (RequestLock) { Outstanding.Remove(id); } + Incoming.Write(ex); +}); +``` + +It does turn the suite green, which is exactly why it is worth writing down as a trap. It races: + +1. Thread A calls `Load(id)`, creates `Outstanding[id] = [h1]` and starts the request. +2. The request fails; the continuation queues up on `RequestLock`. +3. Thread B calls `Load(id)`, wins the lock, sees the entry is still there and adds `h2` to it. B believes + its load is in flight. +4. The continuation takes the lock and removes the entry — dropping **both** `h1` and `h2`. + +`h2` now never resolves and no request was ever started for it, so the wedge has been traded for a silently +dropped handle, which is harder to notice. Doing the removal on the main thread inside `Update`, in the +same lock section that materializes successes, is what closes this: `h2` is either in the list we are about +to fail, or it missed the list and its own `Load` started a fresh request. There is no third case. diff --git a/source/CapriKit.AssetPipeline.DirectX11/CapriKit.AssetPipeline.DirectX11.csproj b/source/CapriKit.AssetPipeline.DirectX11/CapriKit.AssetPipeline.DirectX11.csproj new file mode 100644 index 0000000..c02ac0d --- /dev/null +++ b/source/CapriKit.AssetPipeline.DirectX11/CapriKit.AssetPipeline.DirectX11.csproj @@ -0,0 +1,11 @@ + + + $(TargetFramework)-windows + + + + + + + + diff --git a/source/CapriKit.AssetPipeline.DirectX11/Shaders/ShaderTranscoder.cs b/source/CapriKit.AssetPipeline.DirectX11/Shaders/ShaderTranscoder.cs new file mode 100644 index 0000000..9bfc580 --- /dev/null +++ b/source/CapriKit.AssetPipeline.DirectX11/Shaders/ShaderTranscoder.cs @@ -0,0 +1,26 @@ +using CapriKit.DirectX11.Resources.Shaders; +using CapriKit.IO.Streams; +using System.Buffers; + +namespace CapriKit.AssetPipeline.DirectX11.Shaders; + +internal static class ShaderTranscoder +{ + public static void WriteCommon(ShaderByteCode shader, IBufferWriter writer) + { + writer.Write(shader.EntryPoint); + writer.Write(shader.Name); + writer.Write(shader.Bytes.Length); + writer.Write(shader.Bytes); + } + + public static ShaderByteCode ReadCommon(ref SequenceReader reader) + { + var entryPoint = reader.ReadString(); + var name = reader.ReadString(); + var length = reader.ReadInt32(); + var bytes = reader.ReadBytes(length); + + return new ShaderByteCode(bytes, entryPoint, name); + } +} diff --git a/source/CapriKit.AssetPipeline.DirectX11/Shaders/VertexShaderTranscoder.cs b/source/CapriKit.AssetPipeline.DirectX11/Shaders/VertexShaderTranscoder.cs new file mode 100644 index 0000000..f1824d0 --- /dev/null +++ b/source/CapriKit.AssetPipeline.DirectX11/Shaders/VertexShaderTranscoder.cs @@ -0,0 +1,29 @@ +using CapriKit.DirectX11; +using CapriKit.DirectX11.Resources.Shaders; +using CapriKit.IO; +using System.Buffers; + +namespace CapriKit.AssetPipeline.DirectX11.Shaders; + +public sealed class VertexShaderTranscoder(Device device) + : NoSettingsTranscoder(Guid.Parse("{CA3CB37D-9880-4B61-AB09-EBC17E7533E6}"), 1) +{ + public override async Task Encode(AssetId id, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer) + { + var source = await fileSystem.ReadAllText(id.Path); + var includePath = id.Path.Directory; + var bytes = ShaderCompiler.CompileVertexShader(fileSystem, includePath, source, id.Key, id.ToString()); + ShaderTranscoder.WriteCommon(bytes.Common, writer); + } + + public override IVertexShader Decode(AssetId id, ref SequenceReader reader) + { + var common = ShaderTranscoder.ReadCommon(ref reader); + return ShaderCompiler.CreateVertexShader(new VertexShaderByteCode(common), device); + } + + public override void HotSwap(IVertexShader instance, IVertexShader newParts) + { + instance.HotSwap(newParts); + } +} diff --git a/source/CapriKit.AssetPipeline/Asset.cs b/source/CapriKit.AssetPipeline/Asset.cs new file mode 100644 index 0000000..c93cb80 --- /dev/null +++ b/source/CapriKit.AssetPipeline/Asset.cs @@ -0,0 +1,48 @@ +using CapriKit.IO; + +namespace CapriKit.AssetPipeline; + +/// +/// Unique asset identifier +/// +/// Virtual file path that points to the file the asset originates from. +/// Optional key to a sub-resources in Path. +public record AssetId(FilePath Path, string Key = "") +{ + public override string ToString() + { + if (string.IsNullOrEmpty(Key)) + { + return Path; + } + + return $"{Path}:{Key}"; + } +} + +/// +/// An asset, including its identifier and information on how it was built. +/// +internal record Asset(AssetId Id, TAsset Value, AssetBuildMetaData BuildMetaData) + where TAsset : class; + +/// +/// One finished load request handed back to the main thread: either a materializer that puts the asset in +/// the cache and takes a lease on it, or the failure that stopped the load. +/// +internal readonly record struct LoadResult(AssetId Id, Func? Materialize, AssetLoadException? Failure) +{ + public static LoadResult Success(AssetId id, Func materialize) => new(id, materialize, null); + + public static LoadResult Failed(AssetLoadException failure) => new(failure.Asset, null, failure); +} + +/// +/// Record of the exact transcoder, settings and files used to build the asset. +/// +internal record class AssetBuildMetaData(Guid TranscoderId, int TranscoderVersion, TSettings Settings, IReadOnlyList Dependencies); + +/// +/// A file used to build the asset and the date and time it was last changed +/// +internal sealed record Dependency(FilePath File, DateTime Version); diff --git a/source/CapriKit.AssetPipeline/AssetBundle.cs b/source/CapriKit.AssetPipeline/AssetBundle.cs new file mode 100644 index 0000000..d94ed12 --- /dev/null +++ b/source/CapriKit.AssetPipeline/AssetBundle.cs @@ -0,0 +1,203 @@ +using System.Diagnostics.CodeAnalysis; + +namespace CapriKit.AssetPipeline; + +/// +/// A set of assets that are loaded together and unloaded together. The bundle holds one lease per asset it +/// loaded for as long as it lives, disposing it hands every one of them back. Use to +/// describe the strongly typed value those assets add up to, the bundle itself owns their lifetime. +/// Threading: create, load and build from one thread, usually the primary one. +/// +public sealed class AssetBundle : IDisposable +{ + private readonly List HandleList = []; + private readonly AssetManager Manager; + + internal AssetBundle(AssetManager manager, string origin) + { + Manager = manager; + Origin = origin; + } + + /// + /// The number of assets in this bundle. + /// + public int Total => HandleList.Count; + + /// + /// The number of assets that have arrived, successfully or not, updated every time + /// is called. + /// + public int Loaded { get; private set; } + + /// + /// The last asset that had arrived the last time was + /// called, in the order the assets were requested. Meant to put a name on a loading screen, not to + /// report the exact order in which assets finished. + /// + public AssetId? LastCompletedItem { get; private set; } + + /// + /// Every handle in this bundle, failed ones included, kept for as long as the bundle lives. Unloading + /// counts handles rather than distinct assets because the pool hands out one lease per resolved handle: + /// an asset this bundle asked for twice holds two leases, and one that failed to load holds none. + /// + internal IReadOnlyList Handles => HandleList; + + internal bool IsActive { get; set; } = true; + + /// + /// The file and line that created this bundle, used to name it if it is never unloaded. + /// + internal string Origin { get; } + + /// + /// Starts building (if needed) and loading an asset, and adds it to this bundle. The handle it returns + /// is used in the lambda for to + /// describe how the assets in this bundle add up to one strongly typed value. Load everything the + /// bundle needs before building it, an asset loaded afterwards still belongs to the bundle but the + /// loaders that were already built cannot see it. + /// + public AssetHandle Load(AssetId id, TSettings settings) + where TAsset : class + { + var handle = Manager.Load(id, settings); + handle.Owner = this; + HandleList.Add(handle); + return handle; + } + + /// + public AssetHandle Load(AssetId id) + where TAsset : class + => Load(id, default); + + /// + /// Creates the loader that reports the progress of this bundle and that builds the strongly typed value + /// its assets add up to. The loader borrows the bundle, it does not own it: this bundle stays the thing + /// that has to be unloaded, which is why building twice is harmless. + /// + public AssetBundleLoader Build(Func factory) + where TBundle : notnull + => new(this, factory); + + /// + /// Unloads this bundle, see . + /// Unloading a bundle twice is safe, the second call does nothing. + /// + public void Dispose() => Manager.Unload(this); + + /// + /// Updates and and reports whether every asset in + /// the bundle arrived, successfully or not. Walks all handles rather than keeping a list of the pending + /// ones: a bundle holds a handful of assets, and one list is easier to reason about than two that have + /// to agree with each other. + /// + internal bool AllAssetsArrived() + { + var arrived = 0; + foreach (var handle in HandleList) + { + // An asset that failed has arrived too, it just arrived as an error instead of as a value. + if (handle.IsCompleted) + { + arrived++; + LastCompletedItem = handle.Id; + } + } + + Loaded = arrived; + return arrived == HandleList.Count; + } + + /// + /// Reports the assets in this bundle that could not be built or loaded, in the order they were + /// requested. Only call this once is true: until then an asset might + /// still take a lease, and reporting a failure before that would leave it unaccounted for. + /// + internal void ThrowOnFailedAssets() + { + List? failures = null; + foreach (var handle in HandleList) + { + if (handle.Error is not null) + { + (failures ??= []).Add(handle.Error); + } + } + + if (failures is null) { return; } + if (failures.Count == 1) { throw failures[0]; } + + throw new AggregateException(failures); + } +} + +/// +/// Tracks the loading progress of an and builds the strongly typed value its +/// assets add up to once they all arrived. Purely a view on the bundle: unloading goes through the bundle, +/// so a loader that is dropped costs nothing. +/// +public sealed class AssetBundleLoader + where TBundle : notnull +{ + private readonly AssetBundle Bundle; + private readonly Func Factory; + + private TBundle? result; + private bool isReady; + + internal AssetBundleLoader(AssetBundle bundle, Func factory) + { + Bundle = bundle; + Factory = factory; + } + + /// + public int Total => Bundle.Total; + + /// + public int Loaded => Bundle.Loaded; + + /// + public AssetId? LastCompletedItem => Bundle.LastCompletedItem; + + /// + /// Checks whether the bundle finished loading, and if so builds and returns it. The result is + /// cached so calling IsReady multiple times after loading finished is OK. + /// Throws a when an asset in this bundle could not be built or + /// loaded. Throws an if multiple assets failed to load. + /// Throws an once the bundle has been unloaded. + /// + public bool IsReady([NotNullWhen(true)] out TBundle? value) + { + // An unloaded bundle gave its leases back, so the assets it would hand out may already be disposed. + ObjectDisposedException.ThrowIf(!Bundle.IsActive, Bundle); + + if (isReady) { value = result!; return true; } + + if (!Bundle.AllAssetsArrived()) { value = default; return false; } + + // Everything arrived, so every lease this bundle will ever hold is settled and it is safe to report + // a failure. isReady stays false, so every later call throws again. + Bundle.ThrowOnFailedAssets(); + + result = value = Factory(new AssetHandleResolver(Bundle)); + isReady = true; + return true; + } + + /// + /// Busy waits until the bundle finishes loading, then builds and returns it. + /// Threading: primary thread only. + /// + public TBundle WaitUntilReady() + { + var wait = new SpinWait(); + while (true) + { + if (IsReady(out var bundle)) { return bundle; } + wait.SpinOnce(); + } + } +} diff --git a/source/CapriKit.AssetPipeline/AssetDecoder.cs b/source/CapriKit.AssetPipeline/AssetDecoder.cs new file mode 100644 index 0000000..b496ec9 --- /dev/null +++ b/source/CapriKit.AssetPipeline/AssetDecoder.cs @@ -0,0 +1,137 @@ +using CapriKit.IO; +using CapriKit.IO.Streams; +using System.Buffers; +using static CapriKit.AssetPipeline.AssetUtilities; + +namespace CapriKit.AssetPipeline; + +/// +/// Decodes the generic asset envelope and, using a specialized IAssetTranscoder, the asset itself +/// Threading: thread-safe +/// +internal static class AssetDecoder +{ + public static async Task> Decode(AssetId id, IAssetTranscoder decoder, IReadOnlyVirtualFileSystem fileSystem, Stream? inputStreamOverride = default) + where TAsset : class + { + var inputPath = ToEncodedFilePath(id); + if (!fileSystem.Exists(inputPath)) + { + throw new FileNotFoundException($"Could not find file: {inputPath} to load asset: {id}", id.Path); + } + + using var input = inputStreamOverride ?? fileSystem.OpenRead(inputPath); + var length = checked((int)input.Length); + var buffer = ArrayPool.Shared.Rent(length); + + try + { + await input.ReadExactlyAsync(buffer.AsMemory(0, length)); + var reader = SequenceReaders.Create(buffer, 0, length); + + var (encoderId, encoderVersion) = ReadHeader(ref reader); + ThrowOnDecoderMismatch(id, inputPath, encoderId, encoderVersion, decoder); + + var settings = ReadSettings(ref reader, decoder); + var asset = ReadPayload(ref reader, id, decoder, settings); + var dependencies = ReadDependencies(ref reader); + + var buildMetaData = new AssetBuildMetaData(encoderId, encoderVersion, settings, dependencies); + return new Asset(id, asset, buildMetaData); + } + finally + { + ArrayPool.Shared.Return(buffer); + } + } + + public static async Task?> TryDecodeBuildMetaData(AssetId id, IAssetTranscoder decoder, IReadOnlyVirtualFileSystem fileSystem) + where TAsset : class + { + try + { + var inputPath = ToEncodedFilePath(id); + if (!fileSystem.Exists(inputPath)) + { + throw new FileNotFoundException($"Could not find file: {inputPath} to load asset: {id}", id.Path); + } + + using var input = fileSystem.OpenRead(inputPath); + var length = checked((int)input.Length); + var buffer = ArrayPool.Shared.Rent(length); + + try + { + await input.ReadExactlyAsync(buffer.AsMemory(0, length)); + var reader = SequenceReaders.Create(buffer, 0, length); + + var (encoderId, encoderVersion) = ReadHeader(ref reader); + var settings = ReadSettings(ref reader, decoder); + SkipPayload(ref reader); + var dependencies = ReadDependencies(ref reader); + return new AssetBuildMetaData(encoderId, encoderVersion, settings, dependencies); + } + finally + { + ArrayPool.Shared.Return(buffer); + } + } + catch + { + return null; + } + } + + private static (Guid id, int version) ReadHeader(ref SequenceReader reader) + { + var id = reader.ReadGuid(); + var version = reader.ReadInt32(); + return (id, version); + } + + private static TSettings ReadSettings(ref SequenceReader reader, IAssetTranscoder decoder) + where TAsset : class + { + var settingsLength = reader.ReadInt32(); + var settingsReader = reader.SliceUnread(settingsLength); + return decoder.ReadSettings(ref settingsReader); + } + + private static TAsset ReadPayload(ref SequenceReader reader, AssetId id, IAssetTranscoder decoder, TSettings settings) + where TAsset : class + { + var payloadLength = reader.ReadInt32(); + var payloadReader = reader.SliceUnread(payloadLength); + return decoder.Decode(id, settings, ref payloadReader); + } + + private static void SkipPayload(ref SequenceReader reader) + { + var payloadLength = reader.ReadInt32(); + reader.Advance(payloadLength); + } + + private static List ReadDependencies(ref SequenceReader reader) + { + var count = reader.ReadInt32(); + var dependencies = new List(count); + for (var i = 0; i < count; i++) + { + var lastWriteTicks = reader.ReadInt64(); + var lastWrite = new DateTime(lastWriteTicks); + var filePathString = reader.ReadString(); + var filePath = new FilePath(filePathString); + dependencies.Add(new Dependency(filePath, lastWrite)); + } + return dependencies; + } + + private static void ThrowOnDecoderMismatch(AssetId id, FilePath file, Guid fileId, int fileVersion, IAssetTranscoder decoder) + { + if (fileId != decoder.Id || fileVersion != decoder.Version) + { + throw new InvalidDataException( + $"Decoder mismatch. Asset: {id} found in file: {file}, was encoded using {fileId}:v{fileVersion} but the current decoder is {decoder.Id}:v{decoder.Version}."); + } + } +} diff --git a/source/CapriKit.AssetPipeline/AssetEncoder.cs b/source/CapriKit.AssetPipeline/AssetEncoder.cs new file mode 100644 index 0000000..3d52b47 --- /dev/null +++ b/source/CapriKit.AssetPipeline/AssetEncoder.cs @@ -0,0 +1,79 @@ +using CapriKit.IO; +using CapriKit.IO.Streams; +using System.Buffers; +using System.IO.Pipelines; +using static CapriKit.AssetPipeline.AssetUtilities; + +namespace CapriKit.AssetPipeline; + +// File format: [encoder id][encoder version][settings length][settings][payload length][payload][dependency count][dependencies] + +/// +/// Encodes the generic asset envelope and, using a specialized IAssetTranscoder, the asset itself. +/// Threading: thread-safe +/// +internal static class AssetEncoder +{ + public static async Task Encode(AssetId id, IAssetTranscoder encoder, TSettings settings, IVirtualFileSystem fileSystem, Stream? outputStreamOverride = default) + where TAsset : class + { + ThrowOnFileNotFound(id.Path, fileSystem); + var outputPath = ToEncodedFilePath(id); + Stream? output = null; + try + { + output = outputStreamOverride ?? fileSystem.CreateReadWrite(outputPath); + var writer = PipeWriter.Create(output, new StreamPipeWriterOptions(leaveOpen: true)); + var spy = fileSystem.SpyOn(); + + WriteHeader(writer, encoder); + WriteSettings(writer, encoder, settings); + await WritePayload(writer, id, encoder, settings, spy); + WriteDependencies(writer, spy); + + await writer.FlushAsync(); + await writer.CompleteAsync(); + } + finally + { + // Only dispose of the stream if we created it. + if (outputStreamOverride is null) { output?.Dispose(); } + } + + } + + private static void WriteHeader(PipeWriter writer, IAssetTranscoder encoder) + { + writer.Write(encoder.Id); + writer.Write(encoder.Version); + } + + private static void WriteSettings(PipeWriter writer, IAssetTranscoder encoder, TSettings settings) + where TAsset : class + { + var buffer = new ArrayBufferWriter(); + encoder.WriteSettings(settings, buffer); + writer.Write(buffer.WrittenCount); + writer.Write(buffer.WrittenSpan); + } + + private static async Task WritePayload(PipeWriter writer, AssetId id, IAssetTranscoder encoder, TSettings settings, VirtualFileSystemSpy spy) + where TAsset : class + { + var payload = new ArrayBufferWriter(); + await encoder.Encode(id, settings, spy, payload); + writer.Write(payload.WrittenCount); + writer.Write(payload.WrittenSpan); + } + + private static void WriteDependencies(PipeWriter writer, VirtualFileSystemSpy spy) + { + writer.Write(spy.OpenedFiles.Count); + foreach (var dependency in spy.OpenedFiles) + { + var lastWrite = spy.LastWriteTime(dependency); + writer.Write(lastWrite.Ticks); + writer.Write(dependency); + } + } +} diff --git a/source/CapriKit.AssetPipeline/AssetHandle.cs b/source/CapriKit.AssetPipeline/AssetHandle.cs new file mode 100644 index 0000000..d1b640c --- /dev/null +++ b/source/CapriKit.AssetPipeline/AssetHandle.cs @@ -0,0 +1,72 @@ +using System.Diagnostics; + +namespace CapriKit.AssetPipeline; + +/// +/// Represents an asset that is in the progress of loading. +/// +public abstract class AssetHandle(AssetId id) +{ + public AssetId Id { get; } = id; + + private object? value; + private AssetLoadException? error; + private volatile bool isResolved; + + + internal AssetBundle? Owner { get; set; } + + /// True once the asset arrived, whether it loaded successfully or failed. + internal bool IsCompleted => isResolved; + + /// The failure that stopped this asset from loading, null while loading and after success. + internal AssetLoadException? Error => error; + + /// True once the asset arrived successfully, which is also when it holds a lease on the pool. + internal bool IsLoaded => IsCompleted && error is null; + + internal object? Value => value; + + internal void Resolve(object asset) + { + Debug.Assert(isResolved == false); + value = asset; + isResolved = true; + } + + /// + /// Hands this asset's failure to whoever is waiting for it. The volatile write to isResolved happens + /// last so a reader that sees the handle complete also sees the error, which is the same ordering + /// relies on for its value. + /// + internal void Fail(AssetLoadException exception) + { + Debug.Assert(isResolved == false); + error = exception; + isResolved = true; + } +} + +/// +public sealed class AssetHandle(AssetId id) : AssetHandle(id) { } + +/// +/// Helper class for resolving loaded assets from their asset handle. +/// +public sealed class AssetHandleResolver(AssetBundle owner) +{ + public TValue Get(AssetHandle promise) + { + if (promise.Owner != owner) + { + throw new InvalidOperationException($"Attempted to resolve a promise that was not owned by the bundle"); + } + + if (promise.Value is TValue value) + { + return value; + } + + throw new Exception($"Internal error: resolved value was not of type {typeof(TValue).Name} but {promise.Value?.GetType().Name ?? "null"}"); + } +} diff --git a/source/CapriKit.AssetPipeline/AssetLoadException.cs b/source/CapriKit.AssetPipeline/AssetLoadException.cs new file mode 100644 index 0000000..d8b4d8a --- /dev/null +++ b/source/CapriKit.AssetPipeline/AssetLoadException.cs @@ -0,0 +1,13 @@ +namespace CapriKit.AssetPipeline; + +/// +/// Thrown from when building or loading an asset failed. +/// +public sealed class AssetLoadException(AssetId asset, Exception innerException) + : Exception($"Failed to build or load asset: {asset}", innerException) +{ + /// + /// The asset that could not be built or loaded. + /// + public AssetId Asset { get; } = asset; +} diff --git a/source/CapriKit.AssetPipeline/AssetManager.cs b/source/CapriKit.AssetPipeline/AssetManager.cs new file mode 100644 index 0000000..310c3f0 --- /dev/null +++ b/source/CapriKit.AssetPipeline/AssetManager.cs @@ -0,0 +1,368 @@ +using CapriKit.Concurrency.Async; +using CapriKit.Concurrency.Primitives; +using CapriKit.IO; +using Microsoft.Extensions.Logging; +using System.Buffers; +using System.Collections.Concurrent; +using System.Runtime.CompilerServices; + +namespace CapriKit.AssetPipeline; + +/// +/// Manages the building, loading, caching, clean-up and hot-reloading of assets. +/// +public sealed partial class AssetManager : IDisposable +{ + private readonly ILogger Logger; + private readonly ScopedFileSystem FileSystem; + private readonly AssetPool Cache; + private readonly HotReloadManager HotReloadManager; + private readonly ConcurrentDictionary Transcoders; + + private readonly LightweightChannel Incoming; + private readonly Lock RequestLock; + private readonly Dictionary> Outstanding; + + // Every bundle that holds, or is going to hold, leases. Kept so that Dispose can name whoever forgot + // to unload instead of only reporting that some number of assets was left behind. + private readonly HashSet LiveBundles; + + public AssetManager(ILoggerFactory logger, ScopedFileSystem fileSystem) + { + Logger = logger.CreateLogger(); + FileSystem = fileSystem; + Cache = new(); + HotReloadManager = new(logger, Cache, FileSystem); + Transcoders = []; + Incoming = new(); + RequestLock = new(); + Outstanding = []; + LiveBundles = []; + } + + /// + /// Register a transcoder for the given asset type. Registering a transcoders for a type + /// that was already assigned a transcoder throws an exception. + /// Threading: thread-safe, multiple threads can register transcoders at the same time. + /// + public void RegisterTranscoder(IAssetTranscoder transcoder) + where TAsset : class + { + var key = typeof(TAsset); + var original = Transcoders.GetOrAdd(key, transcoder); + if (original != transcoder) + { + throw new Exception($"A transcoder for key: {key.FullName} was already registered"); + } + } + + /// + /// Creates a bundle to load assets into. The bundle owns everything loaded into it until it is + /// unloaded, so it is the thing to keep and to dispose. The call site is captured so that a bundle that + /// is never unloaded can point at the code that created it, callers should not pass those two arguments. + /// Threading: thread-safe. + /// + public AssetBundle CreateBundle([CallerFilePath] string file = "", [CallerLineNumber] int line = 0) + { + var bundle = new AssetBundle(this, $"{Path.GetFileName(file.AsSpan())}:{line}"); + lock (RequestLock) { LiveBundles.Add(bundle); } + + return bundle; + } + + /// + /// Starts loading an asset. The asset will either be loaded from the cache, from disk, or rebuild and then loaded. + /// The caller gets a handle to be used in an which can be resolved + /// to the actual asset when loading finishes using + /// Threading: thread-safe, can be called from any thread concurrently. This method guarantees that the same asset + /// is not loaded multiple times concurrently. + /// + internal AssetHandle Load(AssetId id, TSettings settings) + where TAsset : class + { + var transcoder = GetTranscoder(); + var handle = new AssetHandle(id); + + // At this time an asset is either already loaded, already requested or requested for the first time. + // The lock ensure that this does not change while we check what we should do with the request. + lock (RequestLock) + { + // Check if the asset was loaded before + if (Cache.TryLease(id, out var cachedAsset)) + { + LogLoadedFromCache(Logger, id); + handle.Resolve(cachedAsset); + return handle; + } + + // Check if the asset was already requested + if (Outstanding.TryGetValue(id, out var requestors)) + { + requestors.Add(handle); + } + else + { + // Request the asset + Outstanding[id] = [handle]; + Task.Run(() => RequestAsset(id, settings, transcoder)).FireAndForget( + ex => + { + LogFailed(Logger, id); + Incoming.Write(LoadResult.Failed(new AssetLoadException(id, ex.SourceException))); + }); + } + } + + return handle; + } + + /// + /// Performs the actual loading or building and loading of the asset. + /// Threading: The caller has to guarantee that this method does not run concurrently for the same asset-id. + /// + private async Task RequestAsset(AssetId id, TSettings settings, IAssetTranscoder transcoder) + where TAsset : class + { + // Check if the asset can be loaded from an up-to-date build + var build = await AssetDecoder.TryDecodeBuildMetaData(id, transcoder, FileSystem); + if (build != default && IsUpToDate(transcoder, settings, build, FileSystem)) + { + var upToDateAsset = await AssetDecoder.Decode(id, transcoder, FileSystem); + Incoming.Write(LoadResult.Success(id, () => TrackAndTakeLease(upToDateAsset, transcoder))); + LogLoadedFromFile(Logger, id); + } + else // If not, try to rebuild and load the asset + { + if (!FileSystem.Exists(id.Path)) + { + throw new FileNotFoundException("Could not find primary file to build asset from", id.Path); + } + + await AssetEncoder.Encode(id, transcoder, settings, FileSystem); + var freshAsset = await AssetDecoder.Decode(id, transcoder, FileSystem); + Incoming.Write(LoadResult.Success(id, () => TrackAndTakeLease(freshAsset, transcoder))); + LogBuildAndLoaded(Logger, id); + } + } + + /// + /// Unloads all assets in the bundle. A bundle that failed to load can be unloaded too: the assets in it + /// that did load are returned, and the ones that failed never took anything that needs returning. So can + /// a bundle that is still loading, the leases its assets take when they do arrive are returned right away. + /// Unloading the same bundle twice is safe, the second call does nothing. + /// Threading: Unload updates the internal state of the bundle using a lock so that it is safe + /// to unload the same bundle from multiple threads. + /// + internal void Unload(AssetBundle bundle) + { + try + { + RequestLock.Enter(); + if (bundle.IsActive) + { + // Give back exactly what was taken. The pool counts one lease per resolved handle, so an + // asset this bundle asked for twice has to be returned twice, and one that failed to load + // or never arrived has nothing to return. + foreach (var handle in bundle.Handles) + { + if (handle.IsLoaded) { Cache.Return(handle.Id); } + } + + LiveBundles.Remove(bundle); + } + } + finally + { + // Marks the handles that have not arrived yet as unwanted, Update returns their leases for us. + bundle.IsActive = false; + RequestLock.Exit(); + } + } + + /// + /// Materializes assets that have finished loading, removes unused items from the cache + /// and hot-reloads changed assets. + /// A failed load does not throw here, it is handed to the handles that were waiting for it and + /// surfaces from instead. + /// Threading: Should only be called from the primary thread. + /// + public void Update() + { + // Ensure that while we check which assets are done loading and up-to-date that administration + // a new request to `Load` of the same asset does not miss these update. + lock (RequestLock) + { + while (Incoming.TryRead(out var result)) + { + // Whether it loaded or failed the request is over, so it stops accepting handles. Doing that + // here, under the lock that Load uses, is what stops a handle joining a dead request: it + // either made this list, or it misses and its own Load starts a fresh request. + if (!Outstanding.Remove(result.Id, out var waiting)) { continue; } + + foreach (var handle in waiting) + { + if (result.Failure is not null) { handle.Fail(result.Failure); } + else + { + handle.Resolve(result.Materialize!()); + + // The bundle was unloaded while this asset was still on its way. Unload could not + // return the lease that resolving just took, because back then there was nothing to + // return yet, so give it back here instead. Materializing first and returning after + // is deliberate: it routes the asset through the pool, which is what disposes it. + if (handle.Owner is { IsActive: false }) { Cache.Return(handle.Id); } + } + } + } + } + + Cache.DisposeReleased(); + HotReloadManager.Update(); + } + + /// + /// Shuts the asset manager down. Bundles that were never unloaded are reported but deliberately not + /// unloaded for you: this only runs at shutdown, where there is nobody left to hand the assets to, so + /// quietly cleaning up after the caller would only hide the bug that the log and the exception report. + /// Threading: Should only be called from the primary thread. + /// + public void Dispose() + { + // Dispose the hot-reload manager first so that it can release any reference to + // assets it might still hold. + HotReloadManager.Dispose(); + + DrainIncoming(); + ReportAssetsThatWereNeverUnloaded(); + + Cache.Dispose(); + } + + /// + /// Settles the loads that finished after the last . Those assets were decoded but + /// never reached a handle, so nobody leases them and nothing would ever dispose them. Taking the lease + /// and immediately giving it back moves them through the pool, which does dispose them. + /// + private void DrainIncoming() + { + lock (RequestLock) + { + while (Incoming.TryRead(out var result)) + { + Outstanding.Remove(result.Id); + + // A failed load never built anything, so there is nothing to dispose of. + if (result.Failure is not null) { continue; } + + result.Materialize!(); + Cache.Return(result.Id); + } + } + } + + /// + /// Names every bundle that still holds leases, so that the leak the throws + /// about can be traced back to the code that caused it. + /// + private void ReportAssetsThatWereNeverUnloaded() + { + lock (RequestLock) + { + foreach (var bundle in LiveBundles) + { + // A bundle that never loaded anything is simply unused, not a leak. + if (bundle.Total > 0) + { + LogBundleNotUnloaded(Logger, bundle.Origin, bundle.Total); + } + } + } + } + + // Thread safe, only touches the file system and uses thread safe transcoder methods and properties. + private static bool IsUpToDate(IAssetTranscoder transcoder, TSettings settings, AssetBuildMetaData build, IReadOnlyVirtualFileSystem fileSystem) + where TAsset : class + { + // Transcoders differ + if (transcoder.Id != build.TranscoderId || transcoder.Version != build.TranscoderVersion) + { + return false; + } + + // Settings differ + var inUse = new ArrayBufferWriter(); + transcoder.WriteSettings(settings, inUse); + + var inFile = new ArrayBufferWriter(); + transcoder.WriteSettings(build.Settings, inFile); + + if (!inUse.WrittenSpan.SequenceEqual(inFile.WrittenSpan)) + { + return false; + } + + // Dependencies have changed + foreach (var (file, version) in build.Dependencies) + { + // Treat a file as up-to-date if does not exist + if (fileSystem.Exists(file)) + { + // Otherwise double check the file date matches + var lastWrite = fileSystem.LastWriteTime(file); + if (version != lastWrite) + { + return false; + } + } + } + + return true; + } + + /// + /// Used when assigning newly loaded assets to their handles. Puts the new asset into the cache, + /// tracks it and then takes a lease for the handle. + /// + private TAsset TrackAndTakeLease(Asset asset, IAssetTranscoder transcoder) + where TAsset : class + { + // Though the asset manager does not allow loading the same asset multiple times the cache contract does allow it + var actualObject = Cache.PutOrLease(asset.Id, asset.Value); + + var actualWrapper = new Asset(asset.Id, actualObject, asset.BuildMetaData); + HotReloadManager.Track(actualWrapper, transcoder); + return actualObject; + } + + // Thread safe because Transcoders is thread safe + private IAssetTranscoder GetTranscoder() where TAsset : class + { + var type = typeof(TAsset); + if (!Transcoders.TryGetValue(type, out var transcoder)) + { + throw new Exception($"Missing transcoder for asset type: {type.FullName}"); + } + + if (transcoder is not IAssetTranscoder specializedTranscoder) + { + throw new Exception($"Expected transcoder of type: {type.FullName} but got transcoder of type: {transcoder.GetType().FullName}"); + } + + return specializedTranscoder; + } + + [LoggerMessage(Level = LogLevel.Information, Message = "Loaded asset: {asset} from cache")] + private static partial void LogLoadedFromCache(ILogger logger, AssetId asset); + + [LoggerMessage(Level = LogLevel.Information, Message = "Loaded up-to-date asset: {asset} from file")] + private static partial void LogLoadedFromFile(ILogger logger, AssetId asset); + + [LoggerMessage(Level = LogLevel.Information, Message = "Build and loaded fresh asset: {asset}")] + private static partial void LogBuildAndLoaded(ILogger logger, AssetId asset); + + [LoggerMessage(Level = LogLevel.Error, Message = "Building or loading asset: {asset} failed.")] + private static partial void LogFailed(ILogger logger, AssetId asset); + + [LoggerMessage(Level = LogLevel.Error, Message = "The asset bundle created at {origin} was never unloaded, it still holds {assets} asset(s)")] + private static partial void LogBundleNotUnloaded(ILogger logger, string origin, int assets); +} diff --git a/source/CapriKit.AssetPipeline/AssetPool.cs b/source/CapriKit.AssetPipeline/AssetPool.cs new file mode 100644 index 0000000..44999e7 --- /dev/null +++ b/source/CapriKit.AssetPipeline/AssetPool.cs @@ -0,0 +1,192 @@ +using System.Diagnostics.CodeAnalysis; + +namespace CapriKit.AssetPipeline; + +/// +/// Pool of keyed, reference-counted deduplicated live assets. +/// Most methods are thread-safe and can be accessed concurrently. However, +/// cleaning-up unused asset using the must be done from the main thread. +/// +internal sealed partial class AssetPool : IDisposable +{ + private sealed class Entry(AssetId id, object asset, int refCount) + { + public AssetId Id { get; } = id; + public object Asset { get; } = asset; + public int RefCount { get; set; } = refCount; + } + + private readonly Lock Lock = new(); + private readonly Dictionary Entries = []; + private readonly Queue PendingDispose = []; + private bool isDisposed; + + /// + /// Stores the given asset and then leases it. If an asset for the same asset id is added multiple + /// times the first added asset wins. Assets added later just take a lease on the already + /// added one and the candidate is disposed of. + /// Threading: thread-safe, loading the same asset twice is wasteful but harmless, after calling this + /// method users must stop referencing and use the return value of this method instead. + /// + public TAsset PutOrLease(AssetId id, TAsset candidate) + where TAsset : class + { + lock (Lock) + { + ObjectDisposedException.ThrowIf(isDisposed, this); + + if (Entries.TryGetValue(id, out var entry)) + { + // If someone adds the same asset twice we ensure that + // the version already in the cache wins and update + // the ref count accordingly + if (!object.ReferenceEquals(candidate, entry.Asset)) + { + // If a different instance was added for the same id + // the candidate also needs to be disposed. + PendingDispose.Enqueue(new Entry(id, candidate, 0)); + } + + + var asset = Cast(entry, id); + entry.RefCount++; + return asset; + } + + Entries.Add(id, new Entry(id, candidate, 1)); + return candidate; + } + } + + /// + /// Attempts to retrieve the asset with the given id. + /// Threading: thread-safe. + /// + public bool TryLease(AssetId id, [NotNullWhen(true)] out TAsset? asset) + where TAsset : class + { + lock (Lock) + { + ObjectDisposedException.ThrowIf(isDisposed, this); + + if (Entries.TryGetValue(id, out var entry)) + { + asset = Cast(entry, id); + entry.RefCount++; + return true; + } + + asset = default; + return false; + } + } + + /// + /// Returns a leased asset. If every user returned their asset it becomes collectable. Which happens in + /// . After calling return the caller must no longer reference the asset instance. + /// Threading: thread-safe. + /// + public void Return(AssetId id) + { + lock (Lock) + { + ObjectDisposedException.ThrowIf(isDisposed, this); + + if (Entries.TryGetValue(id, out var entry)) + { + entry.RefCount--; + // Evict items immediately, but only dispose of them in Collect + if (entry.RefCount <= 0) + { + Entries.Remove(id); + PendingDispose.Enqueue(entry); + } + } + else + { + throw new InvalidOperationException($"Returned {id} which was not found in the cache."); + } + } + } + + /// + /// Disposes all assets that no longer have users. + /// Threading: this method must only be called from the primary thread. + /// + public void DisposeReleased() + { + List? toDispose = null; + + // Collecting usually is a no-op and runs on the most important thread,so avoid waiting on acquiring the lock. + if (Lock.TryEnter()) + { + try + { + if (isDisposed) { return; } + toDispose = DrainPendingDisposeQueue(); + } + finally + { + Lock.Exit(); + } + } + + DisposeDrainedItems(toDispose); + } + + public void Dispose() + { + int leaked; + List? toDispose; + + lock (Lock) + { + if (isDisposed) { return; } + isDisposed = true; + + leaked = Entries.Count; + Entries.Clear(); + + toDispose = DrainPendingDisposeQueue(); + } + + DisposeDrainedItems(toDispose); + + if (leaked > 0) + { + throw new Exception($"Cache will leak {leaked} entries that have not been returned before the cache was disposed."); + } + } + + // Must be called from inside a lock, returns null if there's nothing to dispose + private List? DrainPendingDisposeQueue() + { + List? toDispose = null; + while (PendingDispose.TryDequeue(out var entry)) + { + (toDispose ??= []).Add(entry); + } + return toDispose; + } + + // Must called from outside a lock. Dispose runs code outside of our control, might run long + // and might even interact with the asset cache (in which case running it inside the lock + // would cause deadlocks). + private static void DisposeDrainedItems(List? toDispose) + { + if (toDispose != null) + { + foreach (var entry in toDispose) + { + (entry.Asset as IDisposable)?.Dispose(); + } + } + } + + private static TAsset Cast(Entry entry, AssetId id) + where TAsset : class + { + return entry.Asset as TAsset + ?? throw new InvalidOperationException($"{id} is cached as {entry.Asset.GetType().Name} but was requested as {typeof(TAsset).Name}."); + } +} diff --git a/source/CapriKit.AssetPipeline/AssetUtilities.cs b/source/CapriKit.AssetPipeline/AssetUtilities.cs new file mode 100644 index 0000000..1060446 --- /dev/null +++ b/source/CapriKit.AssetPipeline/AssetUtilities.cs @@ -0,0 +1,25 @@ +using CapriKit.IO; + +namespace CapriKit.AssetPipeline; + +internal static class AssetUtilities +{ + public static FilePath ToEncodedFilePath(AssetId id) + { + if (string.IsNullOrEmpty(id.Key)) + { + return $"{id.Path}.cka"; + } + + var key = IOUtilities.EscapeFileName(id.Key); + return $"{id.Path}.{key}.cka"; + } + + public static void ThrowOnFileNotFound(FilePath path, IVirtualFileSystem fileSystem) + { + if (!fileSystem.Exists(path)) + { + throw new FileNotFoundException(null, path); + } + } +} diff --git a/source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj b/source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj new file mode 100644 index 0000000..3c2e557 --- /dev/null +++ b/source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj @@ -0,0 +1,10 @@ + + + + + + + + + + diff --git a/source/CapriKit.AssetPipeline/HotReloadManager.cs b/source/CapriKit.AssetPipeline/HotReloadManager.cs new file mode 100644 index 0000000..d63b692 --- /dev/null +++ b/source/CapriKit.AssetPipeline/HotReloadManager.cs @@ -0,0 +1,309 @@ +using CapriKit.IO; +using CapriKit.IO.Watchers; +using Microsoft.Extensions.Logging; +using System.Diagnostics; + +namespace CapriKit.AssetPipeline; + +/// +/// Rebuilds, reloads and hot-swaps tracked assets whenever one of the files they were built from changes. +/// Rebuilding and reloading happen on the thread pool, only the final hot-swap runs on the main thread +/// (see ). +/// A failed rebuild, reload or hot-swap only means that the asset keeps its current contents, it never +/// invalidates a live asset and never leaves a lease behind in the . +/// Threading: may be called from any thread, and +/// must be called from the main thread. +/// +internal sealed partial class HotReloadManager : IDisposable +{ + private readonly ILogger Logger; + private readonly AssetPool Cache; + private readonly ScopedFileSystem FileSystem; + private readonly IVirtualFileSystemWatcher Watcher; + private readonly FileSystemEventQueue FileChanges; + private readonly TimeSpan Debounce; + + // Guards the two collections that Track (any thread) and Update (main thread) share. + private readonly Lock Lock; + private readonly Dictionary Tracked; + private readonly Dictionary> Dependents; + + // Only touched by the main thread. + private readonly HashSet Stale; + private readonly Dictionary> InFlight; + private long lastChange; + + private bool isDisposed; + + /// + /// The optional debounce is how long to wait after the last relevant file change before rebuilding. + /// Editors write their buffer in several steps, without a pause we would rebuild the same asset once per + /// step and read half-written files. If not provided, it will be set to 500 milliseconds. + /// + public HotReloadManager(ILoggerFactory loggerFactory, AssetPool cache, ScopedFileSystem fileSystem, TimeSpan? debounce = null) + { + Logger = loggerFactory.CreateLogger(); + Cache = cache; + FileSystem = fileSystem; + Debounce = debounce ?? TimeSpan.FromSeconds(0.5); + + Lock = new(); + Tracked = []; + Dependents = []; + Stale = []; + InFlight = []; + + Watcher = FileSystem.Watch(); + FileChanges = new FileSystemEventQueue(Watcher); + } + + /// + /// Registers an asset so that it is rebuilt, reloaded and hot-swapped whenever one of the files it was + /// built from changes. Tracking the same asset more than once is a no-op. + /// Threading: thread-safe, may be called from any thread at any time. + /// + public void Track(Asset asset, IAssetTranscoder transcoder) + where TAsset : class + { + lock (Lock) + { + // After disposal we no longer listen for file changes, so tracking would only grow the maps. + if (isDisposed) { return; } + + // The asset manager materializes an asset once per outstanding handle, so the same asset arrives + // here several times. Everything we store comes from the build, so the first registration wins. + if (Tracked.ContainsKey(asset.Id)) { return; } + + var tracked = new TrackedAsset(asset, transcoder); + Tracked.Add(asset.Id, tracked); + RegisterDependencies(tracked); + } + } + + /// + /// Reacts to file changes, starts rebuilding the assets those files affect and hot-swaps the assets that + /// finished rebuilding. Every step is bounded work, the expensive rebuilding and reloading happens on the + /// thread pool so that the main thread only pays for the hot-swap itself. + /// Threading: must only be called from the main thread. + /// + public void Update() + { + if (isDisposed) { return; } + + MarkStaleAssets(); + + // Wait for the dust to settle so that a single save does not trigger a burst of rebuilds. + if (Stopwatch.GetElapsedTime(lastChange) >= Debounce) + { + StartReloads(); + } + + FinishReloads(); + } + + /// + /// Stops listening for file changes, abandons everything that has not started yet and finishes the + /// rebuilds that are already running so that their leases and freshly built data are handed back. + /// Threading: must only be called from the main thread. + /// + public void Dispose() + { + lock (Lock) + { + if (isDisposed) { return; } + isDisposed = true; + } + + Watcher.Stop(); + Stale.Clear(); + + // The running rebuilds hold a lease and own freshly built data that only the main thread can dispose + // of, so instead of abandoning them we wait and then finish them through the regular path. + try + { + Task.WaitAll([.. InFlight.Values]); + } + catch (AggregateException) + { + // Failures are reported and cleaned up per asset by FinishReloads + } + + FinishReloads(); + } + + /// + /// Marks every asset that depends on a changed file as stale. + /// + private void MarkStaleAssets() + { + lock (Lock) + { + while (FileChanges.TryDequeue(out var change)) + { + if (!Dependents.TryGetValue(change.File, out var dependents)) { continue; } + + // Only changes we actually care about restart the debounce window, otherwise unrelated + // writes (such as the asset pipeline writing its own build files) could postpone a rebuild. + lastChange = Stopwatch.GetTimestamp(); + + foreach (var id in dependents) + { + if (Stale.Add(id)) + { + LogAssetStale(Logger, change.File, id); + } + } + } + } + } + + /// + /// Starts rebuilding and reloading every stale asset on the thread pool. + /// + private void StartReloads() + { + if (Stale.Count == 0) { return; } + + lock (Lock) + { + foreach (var id in Stale.ToArray()) + { + // Let the running rebuild finish first. The asset stays stale so we pick it up again + // afterwards, which is exactly what we want because its files changed once more. + if (InFlight.ContainsKey(id)) { continue; } + + Stale.Remove(id); + + if (!Tracked.TryGetValue(id, out var tracked)) { continue; } + + if (tracked.TryStartReload(Cache, FileSystem, out var reload)) + { + InFlight.Add(id, reload); + LogReloadStarted(Logger, id); + } + else + { + // The cache is the authority on liveness: no entry means nobody uses this asset anymore. + UntrackAsset(tracked); + LogUntracked(Logger, id); + } + } + } + } + + /// + /// Hot-swaps every asset that finished rebuilding and returns the lease that its rebuild took. + /// + private void FinishReloads() + { + if (InFlight.Count == 0) { return; } + + foreach (var (id, reload) in InFlight.ToArray()) + { + if (!reload.IsCompleted) { continue; } + + InFlight.Remove(id); + FinishReload(id, reload); + } + } + + private void FinishReload(AssetId id, Task reload) + { + try + { + if (!reload.IsCompletedSuccessfully) + { + // Nothing was touched yet, so the asset simply keeps the contents it already had. + LogReloadFailed(Logger, id, reload.Exception!); + return; + } + + var reloaded = reload.Result; + + // Deliberately outside of the lock: the transcoder runs code we do not control here. + reloaded.HotSwap(); + + UpdateDependencies(id, reloaded.Dependencies); + LogHotSwapped(Logger, id); + } + catch (Exception ex) + { + // A transcoder that fails half-way leaves the asset in whatever state it made of it, all we can + // do is report it. The alternative, throwing, would take down the game over a development feature. + LogHotSwapFailed(Logger, id, ex); + } + finally + { + // Balances the lease that TryStartReload took, whether we managed to hot-swap or not. + Cache.Return(id); + } + } + + /// + /// Replaces the dependencies of an asset with the ones its latest build read. + /// + private void UpdateDependencies(AssetId id, IReadOnlyList dependencies) + { + lock (Lock) + { + if (Tracked.TryGetValue(id, out var tracked)) + { + UnregisterDependencies(tracked); + tracked.Dependencies = dependencies; + RegisterDependencies(tracked); + } + } + } + + // Threading: must be called while holding the lock + private void RegisterDependencies(TrackedAsset tracked) + { + foreach (var (file, _) in tracked.Dependencies) + { + if (!Dependents.TryGetValue(file, out var ids)) + { + ids = []; + Dependents.Add(file, ids); + } + + ids.Add(tracked.Id); + } + } + + // Threading: must be called while holding the lock + private void UnregisterDependencies(TrackedAsset tracked) + { + foreach (var (file, _) in tracked.Dependencies) + { + if (Dependents.TryGetValue(file, out var ids) && ids.Remove(tracked.Id) && ids.Count == 0) + { + Dependents.Remove(file); + } + } + } + + // Threading: must be called while holding the lock + private void UntrackAsset(TrackedAsset tracked) + { + UnregisterDependencies(tracked); + Tracked.Remove(tracked.Id); + } + + [LoggerMessage(Level = LogLevel.Information, Message = "Detected change in file: {file}, marking asset: {asset} as stale")] + private static partial void LogAssetStale(ILogger logger, FilePath file, AssetId asset); + + [LoggerMessage(Level = LogLevel.Information, Message = "Started rebuilding and reloading asset: {asset}")] + private static partial void LogReloadStarted(ILogger logger, AssetId asset); + + [LoggerMessage(Level = LogLevel.Error, Message = "Rebuilding or reloading asset: {asset} failed, it keeps its current contents")] + private static partial void LogReloadFailed(ILogger logger, AssetId asset, Exception exception); + + [LoggerMessage(Level = LogLevel.Information, Message = "Hot-swapped asset: {asset}")] + private static partial void LogHotSwapped(ILogger logger, AssetId asset); + + [LoggerMessage(Level = LogLevel.Error, Message = "Hot-swapping asset: {asset} failed")] + private static partial void LogHotSwapFailed(ILogger logger, AssetId asset, Exception exception); + + [LoggerMessage(Level = LogLevel.Information, Message = "Stopped tracking asset: {asset}, it is no longer in the cache")] + private static partial void LogUntracked(ILogger logger, AssetId asset); +} diff --git a/source/CapriKit.AssetPipeline/IAssetTranscoder.cs b/source/CapriKit.AssetPipeline/IAssetTranscoder.cs new file mode 100644 index 0000000..8c76b7d --- /dev/null +++ b/source/CapriKit.AssetPipeline/IAssetTranscoder.cs @@ -0,0 +1,59 @@ +using CapriKit.IO; +using System.Buffers; + +namespace CapriKit.AssetPipeline; + +/// +/// Interface for classes that build assets (such as texture, models and sound effects) and load them +/// when the program needs them. Implementers must take care that most access needs to be thread-safe. +/// +public interface IAssetTranscoder +{ + Guid Id { get; } + int Version { get; } +} + +/// +public interface IAssetTranscoder : IAssetTranscoder + where TAsset : class +{ + // Asynchronous since we expect the encoder to read external files + + /// + /// Loads the raw asset data from the file system and build/encodes it into a format optimized for loading. + /// Threading: thread-safe, encoding happens asynchronously and can happen on any thread. + /// + public Task Encode(AssetId id, TSettings settings, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer); + + /// + /// Decodes the file created the into an object by reading bytes from the given reader. + /// Threading: thread-safe, Though decoding itself is synchronous, it can happen as part of + /// a multi-threaded or async operation. + /// + public TAsset Decode(AssetId id, TSettings settings, ref SequenceReader reader); + + /// + /// Encodes the settings required to encode/decode the asset into the stream. + /// Threading: thread-safe, though this is by itself a synchronous action, + /// it can happen as part of a multi-threaded or async operation. + /// + public void WriteSettings(TSettings settings, IBufferWriter writer); + + + /// + /// Decodes the settings required to encode/decode the asset into the stream. + /// Threading: Though this is by itself synchronous, + /// it can happen as part of a multi-threaded or async operation. + /// + public TSettings ReadSettings(ref SequenceReader reader); + + /// + /// Moves the contents of into . + /// keeps its identity and stays the live object that callers + /// already hold references to. The transcoder is responsible for cleaning-up any + /// orphaned resources. After calling this method must no longer + /// be used or referenced. + /// Threading: must be called by the main thread. + /// + void HotSwap(TAsset instance, TAsset newParts); +} diff --git a/source/CapriKit.AssetPipeline/NoSettingsTranscoder.cs b/source/CapriKit.AssetPipeline/NoSettingsTranscoder.cs new file mode 100644 index 0000000..a86ec8c --- /dev/null +++ b/source/CapriKit.AssetPipeline/NoSettingsTranscoder.cs @@ -0,0 +1,32 @@ +using CapriKit.IO; +using System.Buffers; + +namespace CapriKit.AssetPipeline; + +public readonly struct NoSettings; + +/// +/// Abstract transcoder that removes some of the boilerplate code for implementations that do not have any settings. +/// +public abstract class NoSettingsTranscoder(Guid id, int version) : IAssetTranscoder + where TAsset : class +{ + public Guid Id { get; } = id; + public int Version { get; } = version; + + public abstract Task Encode(AssetId id, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer); + + public abstract TAsset Decode(AssetId id, ref SequenceReader reader); + + public abstract void HotSwap(TAsset instance, TAsset newParts); + + TAsset IAssetTranscoder.Decode(AssetId id, NoSettings settings, ref SequenceReader reader) + => Decode(id, ref reader); + + Task IAssetTranscoder.Encode(AssetId id, NoSettings settings, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer) + => Encode(id, fileSystem, writer); + + NoSettings IAssetTranscoder.ReadSettings(ref SequenceReader reader) => default; + + void IAssetTranscoder.WriteSettings(NoSettings settings, IBufferWriter writer) { } +} diff --git a/source/CapriKit.AssetPipeline/README.md b/source/CapriKit.AssetPipeline/README.md new file mode 100644 index 0000000..961e52c --- /dev/null +++ b/source/CapriKit.AssetPipeline/README.md @@ -0,0 +1,52 @@ +# CapriKit.AssetPipeline + +Reusable asset pipeline that provides multi-threaded building, loading and hot-reloading of assets. + +## Usage + +1. Create the asset manager and register a transcoder, an implementation of `IAssetTranscoder` that knows how to build and load one asset type: +``` +var assetManager = new AssetManager(LoggerFactory, ScopedFileSystem); +assetManager.RegisterTranscoder(new VertexShaderTranscoder(GraphicsDevice)); +``` + +2. Define the contents of a bundle: a plain type that reflects the strongly typed assets that you want to load. It needs no base class or interface, the `AssetBundle` that the pipeline hands you owns the lifetime of the assets in it. +``` +record UIBundle(PixelShader Shader, Texture2D Font); +``` + +3. Use the asset manager to create a bundle and start building (if needed) and loading assets into it. The bundle owns everything you load into it, so it is the thing to keep and to dispose. +``` +using var bundle = assetManager.CreateBundle(); +var shaderHandle = bundle.Load(new AssetId("./shader.hlsl")); +var fontHandle = bundle.Load(new AssetId("./"robo.png")); +``` +4. Use the obtained handles to describe how the contents of the bundle can be constructed once building and loading finishes. This gives you a loader, which only reports progress: dropping it costs nothing, unloading still goes through the bundle. +``` +var loader = bundle.Build(r => new UIBundle(r.get(shaderHandle), r.get(fontHandle)); +``` + +5. Do not forget to update the asset manager each frame so that it can manage the loading tasks and other bookkeeping. +``` +assetManager.Update(); +``` + +6. Check each frame if loading finished and obtain your strongly typed contents +``` +if (loader.isReady(out var ui)) +{ + // Yay! +} +``` + +7. If you no longer need the assets, or if the program exits, dispose the bundle so the assets can be disposed. Loading and unloading uses reference counting to ensure that an asset is only truly disposed of it nobody references it anymore. Unloading a bundle that is still loading is fine, the assets that are still on their way are returned as soon as they arrive. +``` +bundle.Dispose(); // or, the same thing: assetManager.Unload(bundle); +``` + +To ensure everyone correctly unloads their assets an exception will be thrown if the `AssetManager` (and underlying `AssetPool`) are disposed without first unloading all assets. The assets that were left behind are deliberately not unloaded for you, that would only hide the bug. To help you find it the asset manager logs an error naming every bundle that was never unloaded, and the file and line that created it. + +## Hot reloading +Hot reloading happens automatically if the files that were used to build the asset are present and change. + +## Implementation diff --git a/source/CapriKit.AssetPipeline/ServiceCollectionExtensions.cs b/source/CapriKit.AssetPipeline/ServiceCollectionExtensions.cs new file mode 100644 index 0000000..4f91264 --- /dev/null +++ b/source/CapriKit.AssetPipeline/ServiceCollectionExtensions.cs @@ -0,0 +1,19 @@ +using CapriKit.IO; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging; + +namespace CapriKit.AssetPipeline; + +// For Microsoft.Extensions.DependencyInjection.Abstractions +public static class ServiceCollectionExtensions +{ + public static IServiceCollection AddAssetPipeline(this IServiceCollection services, DirectoryPath assetDirectory) + { + return services.AddSingleton(sp => + { + var logFactory = sp.GetRequiredService(); + var fileSystem = new FileSystem().ScopedTo(assetDirectory); + return new AssetManager(logFactory, fileSystem); + }); + } +} diff --git a/source/CapriKit.AssetPipeline/TrackedAsset.cs b/source/CapriKit.AssetPipeline/TrackedAsset.cs new file mode 100644 index 0000000..134c725 --- /dev/null +++ b/source/CapriKit.AssetPipeline/TrackedAsset.cs @@ -0,0 +1,79 @@ +using CapriKit.IO; +using System.Diagnostics.CodeAnalysis; + +namespace CapriKit.AssetPipeline; + +/// +/// A freshly rebuilt asset that is waiting for the main thread to move it into the live instance. +/// +/// Files read while rebuilding, used to keep the file-to-asset map up-to-date. +/// +/// Moves the rebuilt data into the live instance. Threading: main thread only, see . +/// +internal sealed record ReloadedAsset(IReadOnlyList Dependencies, Action HotSwap); + +/// +/// Everything the needs to rebuild a single asset. The asset and settings +/// types are erased so that assets of every type can live in one collection. +/// +internal abstract class TrackedAsset(AssetId id, IReadOnlyList dependencies) +{ + public AssetId Id { get; } = id; + + /// + /// The files that the most recent successful build of this asset read. + /// Threading: only touched by the main thread while it holds the manager's lock. + /// + public IReadOnlyList Dependencies { get; set; } = dependencies; + + /// + /// Leases the live asset from the cache and then starts rebuilding it on the thread pool. Returns false if + /// the asset is no longer in the cache, in which case no lease was taken and nothing was started. + /// The lease is taken before the task starts so that the caller always knows whether a lease exists, the + /// caller owns that lease and must return it once has completed. + /// Threading: main thread only. + /// + public abstract bool TryStartReload(AssetPool cache, IVirtualFileSystem fileSystem, [NotNullWhen(true)] out Task? reload); +} + +/// +internal sealed class TrackedAsset : TrackedAsset + where TAsset : class +{ + private readonly TSettings Settings; + private readonly IAssetTranscoder Transcoder; + + public TrackedAsset(Asset asset, IAssetTranscoder transcoder) + : base(asset.Id, asset.BuildMetaData.Dependencies) + { + Settings = asset.BuildMetaData.Settings; + Transcoder = transcoder; + } + + public override bool TryStartReload(AssetPool cache, IVirtualFileSystem fileSystem, [NotNullWhen(true)] out Task? reload) + { + // The lease pins the live instance for the entire rebuild, so the background thread never has to + // wonder whether the object it is going to hot-swap into still exists. + if (!cache.TryLease(Id, out var live)) + { + reload = null; + return false; + } + + reload = Task.Run(() => Reload(live, fileSystem)); + return true; + } + + private async Task Reload(TAsset live, IVirtualFileSystem fileSystem) + { + // Build into memory rather than over the existing build on disk: other threads may be reading that + // file to load the very same asset and overwriting it underneath them would fail those loads. + using var stream = new MemoryStream(); + await AssetEncoder.Encode(Id, Transcoder, Settings, fileSystem, stream); + + stream.Seek(0, SeekOrigin.Begin); + var rebuilt = await AssetDecoder.Decode(Id, Transcoder, fileSystem, stream); + + return new ReloadedAsset(rebuilt.BuildMetaData.Dependencies, () => Transcoder.HotSwap(live, rebuilt.Value)); + } +} diff --git a/source/CapriKit.Concurrency/Async/FireAndForgetExtensions.cs b/source/CapriKit.Concurrency/Async/FireAndForgetExtensions.cs index a935aff..26a0f32 100644 --- a/source/CapriKit.Concurrency/Async/FireAndForgetExtensions.cs +++ b/source/CapriKit.Concurrency/Async/FireAndForgetExtensions.cs @@ -1,4 +1,5 @@ using System.Runtime.CompilerServices; +using System.Runtime.ExceptionServices; namespace CapriKit.Concurrency.Async; @@ -11,22 +12,16 @@ public static class FireAndForgetExtensions /// public static void FireAndForget( this Task task, - Action onException, - Action? onCompleted = null, - [CallerMemberName] string member = "", - [CallerFilePath] string file = "", - [CallerLineNumber] int line = 0) + Action onException, + Action? onCompleted = null) { - _ = AwaitAndCatch(task, onException, onCompleted, member, file, line); + _ = AwaitAndCatch(task, onException, onCompleted); } private static async Task AwaitAndCatch( Task task, - Action onException, - Action? onCompleted, - string member, - string file, - int line) + Action onException, + Action? onCompleted) { try { @@ -38,8 +33,9 @@ private static async Task AwaitAndCatch( } catch (Exception ex) { - // The original stack trace is preserved via the inner exception - onException(new FaFTaskException(ex, member, file, line)); + // Preserver the original stack trace and allow the handler to rethrow it. + var capture = ExceptionDispatchInfo.Capture(ex); + onException(capture); } finally { @@ -47,18 +43,3 @@ private static async Task AwaitAndCatch( } } } - -public sealed class FaFTaskException : Exception -{ - public FaFTaskException(Exception innerException, string member, string file, int line) - : base($"A fire-and-forget task started from '{member}' at {file}:{line} failed.", innerException) - { - Member = member; - File = file; - Line = line; - } - - public string Member { get; } - public string File { get; } - public int Line { get; } -} diff --git a/source/CapriKit.Concurrency/Async/Drain.cs b/source/CapriKit.Concurrency/Primitives/Drain.cs similarity index 89% rename from source/CapriKit.Concurrency/Async/Drain.cs rename to source/CapriKit.Concurrency/Primitives/Drain.cs index e7b2717..300e2b2 100644 --- a/source/CapriKit.Concurrency/Async/Drain.cs +++ b/source/CapriKit.Concurrency/Primitives/Drain.cs @@ -1,7 +1,7 @@ -using CapriKit.Concurrency.Primitives; +using CapriKit.Concurrency.Async; using System.Diagnostics.CodeAnalysis; -namespace CapriKit.Concurrency.Async; +namespace CapriKit.Concurrency.Primitives; /// /// Allows you to drain work as it completes diff --git a/source/CapriKit.Concurrency/Primitives/LightweightChannel.cs b/source/CapriKit.Concurrency/Primitives/LightweightChannel.cs index 71c56c1..6b98cc5 100644 --- a/source/CapriKit.Concurrency/Primitives/LightweightChannel.cs +++ b/source/CapriKit.Concurrency/Primitives/LightweightChannel.cs @@ -8,10 +8,8 @@ namespace CapriKit.Concurrency.Primitives; /// Lightweight variant of that allows ONE reader /// to receive items from MULTIPLE writers. Has the following semantics: /// - void Write(T value) always succeeds and supports multiple writers working in parallel -/// - void Write(Exception exception) always succeeds but only keeps the first exception -/// - bool TryRead(out T? value) is non blocking, it returns one item from the internal queue if available -/// if there are no items but an exception was written, it rethrows the exception. -/// +/// - void Write(ExceptionDispatchInfo exception) always succeeds and supports multiple writers working in parallel +/// - bool TryRead(out T? value) is non blocking, it first drains the exceptions in the queue (one per call) then the items. /// Note that there being no items doesn't mean the work is done. Users must keep track of the number of items /// they expect to see if the work is done, there is no Completed method or completion tracking to avoid /// problems like 'write after complete' that require extensive locking. @@ -19,12 +17,11 @@ namespace CapriKit.Concurrency.Primitives; public sealed class LightweightChannel where T : notnull { private readonly ConcurrentQueue Queue; - private volatile ExceptionDispatchInfo? Error; - + private readonly ConcurrentQueue Errors; public LightweightChannel() { Queue = []; - Error = null; + Errors = []; } public void Write(T value) @@ -32,20 +29,23 @@ public void Write(T value) Queue.Enqueue(value); } - public void Write(Exception exception) + public void Write(ExceptionDispatchInfo exception) { - var error = ExceptionDispatchInfo.Capture(exception); - Interlocked.CompareExchange(ref Error, error, null); + Errors.Enqueue(exception); } public bool TryRead([NotNullWhen(true)] out T? value) { + if (Errors.TryDequeue(out var error)) + { + error.Throw(); + } + if (Queue.TryDequeue(out value)) { return true; } - Error?.Throw(); return false; } } diff --git a/source/CapriKit.DirectX11/Resources/Shaders/ComputeShader.cs b/source/CapriKit.DirectX11/Resources/Shaders/ComputeShader.cs index e615f9b..581c23a 100644 --- a/source/CapriKit.DirectX11/Resources/Shaders/ComputeShader.cs +++ b/source/CapriKit.DirectX11/Resources/Shaders/ComputeShader.cs @@ -4,7 +4,22 @@ namespace CapriKit.DirectX11.Resources.Shaders; public interface IComputeShader : IDisposable { - internal ID3D11ComputeShader ID3D11ComputeShader { get; } + internal uint NumThreadsX { get; set; } + internal uint NumThreadsY { get; set; } + internal uint NumThreadsZ { get; set; } + internal ID3D11ComputeShader ID3D11ComputeShader { get; set; } + + + public void HotSwap(IComputeShader replacement) + { + NumThreadsX = replacement.NumThreadsX; + NumThreadsY = replacement.NumThreadsY; + NumThreadsZ = replacement.NumThreadsZ; + + var oldShader = ID3D11ComputeShader; + ID3D11ComputeShader = replacement.ID3D11ComputeShader; + oldShader.Dispose(); + } /// /// The shader kernel defines how big the groups for each dimension are. @@ -16,27 +31,49 @@ public interface IComputeShader : IDisposable internal sealed class ComputeShader : IComputeShader { - private readonly uint NumThreadsX; - private readonly uint NumThreadsY; - private readonly uint NumThreadsZ; - private readonly ID3D11ComputeShader Shader; + private uint numThreadsX; + private uint numThreadsY; + private uint numThreadsZ; + private ID3D11ComputeShader shader; internal ComputeShader(ID3D11ComputeShader shader, uint numThreadsX, uint numThreadsY, uint numThreadsZ) { - Shader = shader; - NumThreadsX = numThreadsX; - NumThreadsY = numThreadsY; - NumThreadsZ = numThreadsZ; + this.shader = shader; + this.numThreadsX = numThreadsX; + this.numThreadsY = numThreadsY; + this.numThreadsZ = numThreadsZ; + } + + ID3D11ComputeShader IComputeShader.ID3D11ComputeShader + { + get { return shader; } + set { shader = value; } } - ID3D11ComputeShader IComputeShader.ID3D11ComputeShader => Shader; + uint IComputeShader.NumThreadsX + { + get { return numThreadsX; } + set { numThreadsX = value; } + } + + uint IComputeShader.NumThreadsY + { + get { return numThreadsY; } + set { numThreadsY = value; } + } + + uint IComputeShader.NumThreadsZ + { + get { return numThreadsZ; } + set { numThreadsZ = value; } + } /// public (uint X, uint Y, uint Z) GetDispatchSize(uint dimX, uint dimY, uint dimZ) { - var x = GetDispatchSize(NumThreadsX, dimX); - var y = GetDispatchSize(NumThreadsY, dimY); - var z = GetDispatchSize(NumThreadsZ, dimZ); + var x = GetDispatchSize(numThreadsX, dimX); + var y = GetDispatchSize(numThreadsY, dimY); + var z = GetDispatchSize(numThreadsZ, dimZ); return (x, y, z); } @@ -48,6 +85,6 @@ private static uint GetDispatchSize(uint numThreads, uint dim) public void Dispose() { - Shader.Dispose(); + shader.Dispose(); } } diff --git a/source/CapriKit.DirectX11/Resources/Shaders/PixelShader.cs b/source/CapriKit.DirectX11/Resources/Shaders/PixelShader.cs index f0146e4..8eae3f1 100644 --- a/source/CapriKit.DirectX11/Resources/Shaders/PixelShader.cs +++ b/source/CapriKit.DirectX11/Resources/Shaders/PixelShader.cs @@ -4,20 +4,30 @@ namespace CapriKit.DirectX11.Resources.Shaders; public interface IPixelShader : IDisposable { - internal ID3D11PixelShader ID3D11PixelShader { get; } + internal ID3D11PixelShader ID3D11PixelShader { get; set; } + + public void HotSwap(IPixelShader replacement) + { + var oldShader = ID3D11PixelShader; + ID3D11PixelShader = replacement.ID3D11PixelShader; + oldShader.Dispose(); + } } internal sealed class PixelShader : IPixelShader { - private readonly ID3D11PixelShader Shader; + private ID3D11PixelShader Shader; internal PixelShader(ID3D11PixelShader shader) { Shader = shader; } - ID3D11PixelShader IPixelShader.ID3D11PixelShader => Shader; - + ID3D11PixelShader IPixelShader.ID3D11PixelShader + { + get { return Shader; } + set { Shader = value; } + } public void Dispose() { Shader.Dispose(); diff --git a/source/CapriKit.DirectX11/Resources/Shaders/ShaderCompiler.cs b/source/CapriKit.DirectX11/Resources/Shaders/ShaderCompiler.cs index a400951..5ea5b9f 100644 --- a/source/CapriKit.DirectX11/Resources/Shaders/ShaderCompiler.cs +++ b/source/CapriKit.DirectX11/Resources/Shaders/ShaderCompiler.cs @@ -5,13 +5,29 @@ namespace CapriKit.DirectX11.Resources.Shaders; -public abstract record ShaderByteCode(byte[] Bytes, string EntryPoint, string Name); +public record ShaderByteCode(byte[] Bytes, string EntryPoint, string Name); -public sealed record VertexShaderByteCode(byte[] Bytes, string EntryPoint, string Name) : ShaderByteCode(Bytes, EntryPoint, Name); +public sealed record VertexShaderByteCode(ShaderByteCode Common) +{ + public byte[] Bytes => Common.Bytes; + public string EntryPoint => Common.EntryPoint; + public string Name => Common.Name; +} -public sealed record PixelShaderByteCode(byte[] Bytes, string EntryPoint, string Name) : ShaderByteCode(Bytes, EntryPoint, Name); +public sealed record PixelShaderByteCode(ShaderByteCode Common) +{ + public byte[] Bytes => Common.Bytes; + public string EntryPoint => Common.EntryPoint; + public string Name => Common.Name; +} -public sealed record ComputeShaderByteCode(byte[] Bytes, uint NumThreadsX, uint NumThreadsY, uint NumThreadsZ, string EntryPoint, string Name) : ShaderByteCode(Bytes, EntryPoint, Name); + +public sealed record ComputeShaderByteCode(ShaderByteCode Common, uint NumThreadsX, uint NumThreadsY, uint NumThreadsZ) +{ + public byte[] Bytes => Common.Bytes; + public string EntryPoint => Common.EntryPoint; + public string Name => Common.Name; +} public static class ShaderCompiler { @@ -21,14 +37,14 @@ public static class ShaderCompiler public static IVertexShader CompileVertexShader(IReadOnlyVirtualFileSystem fileSystem, DirectoryPath includePath, Device device, string source, string entryPoint, string name) { - var byteCode = CompileVertexShader(fileSystem, includePath, source, entryPoint, name); - return CreateVertexShader(byteCode, device); + var bytes = CompileVertexShader(fileSystem, includePath, source, entryPoint, name); + return CreateVertexShader(bytes, device); } public static VertexShaderByteCode CompileVertexShader(IReadOnlyVirtualFileSystem fileSystem, DirectoryPath includePath, string source, string entryPoint, string name) { - var bytes = Compile(fileSystem, includePath, source, entryPoint, name, VERTEX_SHADER_PROFILE); - return new VertexShaderByteCode(bytes, entryPoint, name); + var byteCode = Compile(fileSystem, includePath, source, entryPoint, name, VERTEX_SHADER_PROFILE); + return new VertexShaderByteCode(byteCode); } public static IVertexShader CreateVertexShader(VertexShaderByteCode byteCode, Device device) @@ -47,7 +63,7 @@ public static IPixelShader CompilePixelShader(IReadOnlyVirtualFileSystem fileSys public static PixelShaderByteCode CompilePixelShader(IReadOnlyVirtualFileSystem fileSystem, DirectoryPath includePath, string source, string entryPoint, string name) { var bytes = Compile(fileSystem, includePath, source, entryPoint, name, PIXEL_SHADER_PROFILE); - return new PixelShaderByteCode(bytes, entryPoint, name); + return new PixelShaderByteCode(bytes); } public static IPixelShader CreatePixelShader(PixelShaderByteCode byteCode, Device device) { @@ -58,15 +74,15 @@ public static IPixelShader CreatePixelShader(PixelShaderByteCode byteCode, Devic public static IComputeShader CompileComputeShader(IReadOnlyVirtualFileSystem fileSystem, DirectoryPath includePath, Device device, string source, string entryPoint, string name) { - var byteCode = CompileComputeShader(fileSystem, includePath, source, entryPoint, name); - return CreateComputeShader(byteCode, device); + var common = CompileComputeShader(fileSystem, includePath, source, entryPoint, name); + return CreateComputeShader(common, device); } public static ComputeShaderByteCode CompileComputeShader(IReadOnlyVirtualFileSystem fileSystem, DirectoryPath includePath, string source, string entryPoint, string name) { - var bytes = Compile(fileSystem, includePath, source, entryPoint, name, COMPUTE_SHADER_PROFILE); - var (x, y, z) = QueryNumThreads(bytes, name); - return new ComputeShaderByteCode(bytes, x, y, z, entryPoint, name); + var common = Compile(fileSystem, includePath, source, entryPoint, name, COMPUTE_SHADER_PROFILE); + var (x, y, z) = QueryNumThreads(common.Bytes, name); + return new ComputeShaderByteCode(common, x, y, z); } public static IComputeShader CreateComputeShader(ComputeShaderByteCode byteCode, Device device) @@ -76,7 +92,7 @@ public static IComputeShader CreateComputeShader(ComputeShaderByteCode byteCode, return new ComputeShader(shader, byteCode.NumThreadsX, byteCode.NumThreadsY, byteCode.NumThreadsZ); } - private static byte[] Compile(IReadOnlyVirtualFileSystem fileSystem, DirectoryPath includePath, string source, string entryPoint, string name, string profile) + private static ShaderByteCode Compile(IReadOnlyVirtualFileSystem fileSystem, DirectoryPath includePath, string source, string entryPoint, string name, string profile) { using var includeResolver = new ShaderIncludeResolver(fileSystem, includePath); @@ -92,7 +108,7 @@ private static byte[] Compile(IReadOnlyVirtualFileSystem fileSystem, DirectoryPa var bytes = blob.AsBytes(); blob.Dispose(); - return bytes; + return new ShaderByteCode(bytes, entryPoint, name); } /// diff --git a/source/CapriKit.DirectX11/Resources/Shaders/ShaderIncludeResolver.cs b/source/CapriKit.DirectX11/Resources/Shaders/ShaderIncludeResolver.cs index a7e0e0e..b376350 100644 --- a/source/CapriKit.DirectX11/Resources/Shaders/ShaderIncludeResolver.cs +++ b/source/CapriKit.DirectX11/Resources/Shaders/ShaderIncludeResolver.cs @@ -12,11 +12,13 @@ private sealed class ShaderStream(FilePath source, byte[] buffer) : MemoryStream public FilePath Source { get; } = source; } - private readonly ReadOnlyScopedFileSystem FileSystem; + private readonly IReadOnlyVirtualFileSystem FileSystem; + private readonly DirectoryPath BasePath; - public ShaderIncludeResolver(IReadOnlyVirtualFileSystem fileSystem, string basePath) + public ShaderIncludeResolver(IReadOnlyVirtualFileSystem fileSystem, DirectoryPath basePath) { - FileSystem = new ReadOnlyScopedFileSystem(fileSystem, basePath); + FileSystem = fileSystem; + BasePath = fileSystem.GetAbsolutePath(basePath); } public Stream Open(IncludeType type, string fileName, Stream? parentStream) @@ -28,15 +30,16 @@ public Stream Open(IncludeType type, string fileName, Stream? parentStream) fileName = includer.Source.Directory.Append(fileName); } + var path = BasePath.Append(fileName); // The stream might be UTF-8 or UTF-16 even if the contents // are only ASCII characters. Read the full file and convert // it before passing it to DirectX. - using var fileStream = FileSystem.OpenRead(fileName); + using var fileStream = FileSystem.OpenRead(path); using var reader = new StreamReader(fileStream); var text = reader.ReadToEnd(); var bytes = Encoding.ASCII.GetBytes(text); - return new ShaderStream(fileName, bytes); + return new ShaderStream(path, bytes); } public void Close(Stream stream) diff --git a/source/CapriKit.DirectX11/Resources/Shaders/VertexShader.cs b/source/CapriKit.DirectX11/Resources/Shaders/VertexShader.cs index 39ccfb7..c43bf30 100644 --- a/source/CapriKit.DirectX11/Resources/Shaders/VertexShader.cs +++ b/source/CapriKit.DirectX11/Resources/Shaders/VertexShader.cs @@ -4,32 +4,52 @@ namespace CapriKit.DirectX11.Resources.Shaders; public interface IVertexShader : IDisposable { - internal ID3D11VertexShader ID3D11VertexShader { get; } + internal byte[] Blob { get; set; } + internal ID3D11VertexShader ID3D11VertexShader { get; set; } - IInputLayout CreateInputLayout(Device device, InputElementDescription[] elements); // TODO: How to do this without exposing types from Vortice? + IInputLayout CreateInputLayout(Device device, InputElementDescription[] elements); + + public void HotSwap(IVertexShader newParts) + { + Blob = newParts.Blob; + + var oldShader = ID3D11VertexShader; + ID3D11VertexShader = newParts.ID3D11VertexShader; + oldShader.Dispose(); + } } internal sealed class VertexShader : IVertexShader { - private readonly byte[] Blob; - private readonly ID3D11VertexShader Shader; + private byte[] blob; + private ID3D11VertexShader shader; internal VertexShader(byte[] blob, ID3D11VertexShader shader) { - Blob = blob; - Shader = shader; + this.blob = blob; + this.shader = shader; } - ID3D11VertexShader IVertexShader.ID3D11VertexShader => Shader; + byte[] IVertexShader.Blob + { + get { return blob; } + set { blob = value; } + } + + ID3D11VertexShader IVertexShader.ID3D11VertexShader + { + get { return shader; } + set { shader = value; } + } public IInputLayout CreateInputLayout(Device device, InputElementDescription[] elements) { - var inputLayout = device.ID3D11Device.CreateInputLayout(elements, Blob); + var inputLayout = device.ID3D11Device.CreateInputLayout(elements, blob); return new InputLayout(inputLayout); } public void Dispose() { - Shader.Dispose(); + shader.Dispose(); } } diff --git a/source/CapriKit.IO/CapriKit.IO.csproj b/source/CapriKit.IO/CapriKit.IO.csproj index c369e11..2ae79d6 100644 --- a/source/CapriKit.IO/CapriKit.IO.csproj +++ b/source/CapriKit.IO/CapriKit.IO.csproj @@ -1,3 +1 @@ - - - + diff --git a/source/CapriKit.IO/DirectoryPath.cs b/source/CapriKit.IO/DirectoryPath.cs index 0477c0f..aeee232 100644 --- a/source/CapriKit.IO/DirectoryPath.cs +++ b/source/CapriKit.IO/DirectoryPath.cs @@ -38,17 +38,6 @@ public DirectoryPath? Parent } } - public DirectoryPath ToAbsolute() - { - if (IsAbsolute) - { - return this; - } - - var full = System.IO.Path.GetFullPath(Path); - return new DirectoryPath(full); - } - public DirectoryPath ToAbsolute(DirectoryPath basePath) { if (IsAbsolute) diff --git a/source/CapriKit.IO/FilePath.cs b/source/CapriKit.IO/FilePath.cs index 162ea6f..ed25bcc 100644 --- a/source/CapriKit.IO/FilePath.cs +++ b/source/CapriKit.IO/FilePath.cs @@ -31,8 +31,6 @@ public FilePath(ReadOnlySpan path) public bool IsAbsolute => System.IO.Path.IsPathFullyQualified(Path); - public FilePath ToAbsolute() => new(System.IO.Path.GetFullPath(Path)); - public FilePath ToAbsolute(DirectoryPath basePath) { if (IsAbsolute) diff --git a/source/CapriKit.IO/FileSystem.cs b/source/CapriKit.IO/FileSystem.cs index 49ad138..2f7a189 100644 --- a/source/CapriKit.IO/FileSystem.cs +++ b/source/CapriKit.IO/FileSystem.cs @@ -1,3 +1,5 @@ +using CapriKit.IO.Watchers; + namespace CapriKit.IO; public class FileSystem : IVirtualFileSystem @@ -5,7 +7,14 @@ public class FileSystem : IVirtualFileSystem public Stream AppendWrite(FilePath file) { var absoluteFile = FindOrThrow(file); - return absoluteFile.Open(FileMode.Append, FileAccess.Write, FileShare.Read); + var options = new FileStreamOptions() + { + Mode = FileMode.Append, + Access = FileAccess.Write, + Share = FileShare.Read, + Options = FileOptions.Asynchronous + }; + return new FileStream(absoluteFile.FullName, options); } public Stream CreateReadWrite(FilePath file) @@ -16,8 +25,14 @@ public Stream CreateReadWrite(FilePath file) var directory = absoluteFile.DirectoryName ?? string.Empty; Directory.CreateDirectory(directory); } - - return absoluteFile.Open(FileMode.Create, FileAccess.ReadWrite, FileShare.Read); + var options = new FileStreamOptions() + { + Mode = FileMode.Create, + Access = FileAccess.ReadWrite, + Share = FileShare.Read, + Options = FileOptions.Asynchronous + }; + return new FileStream(absoluteFile.FullName, options); } public void Delete(FilePath file) @@ -41,7 +56,14 @@ public DateTime LastWriteTime(FilePath file) public Stream OpenRead(FilePath file) { var absoluteFile = FindOrThrow(file); - return absoluteFile.Open(FileMode.Open, FileAccess.Read, FileShare.Read); + var options = new FileStreamOptions() + { + Mode = FileMode.Open, + Access = FileAccess.Read, + Share = FileShare.Read, + Options = FileOptions.Asynchronous + }; + return new FileStream(absoluteFile.FullName, options); } public long SizeInBytes(FilePath file) @@ -65,9 +87,17 @@ public IReadOnlyList List(DirectoryPath directory) return filePaths; } - public FileSystemEventListener Watch(DirectoryPath directory, bool includeSubDirectories = true) + public IVirtualFileSystemWatcher Watch(DirectoryPath directory, bool includeSubDirectories = true) + { + return new FileSystemEventListener(directory, includeSubDirectories); + } + + public FilePath GetAbsolutePath(FilePath file) => file.ToAbsolute(Environment.CurrentDirectory); + + public DirectoryPath GetAbsolutePath(DirectoryPath directory) { - return new FileSystemEventListener(this, directory, includeSubDirectories); + if (directory.IsAbsolute) { return directory; } + return directory.ToAbsolute(Environment.CurrentDirectory); } private FileInfo FindOrThrow(FilePath file) @@ -81,20 +111,20 @@ private FileInfo FindOrThrow(FilePath file) throw new FileNotFoundException(null, file.ToString()); } - internal FilePath GetFilePath(string path) + internal static FilePath GetFilePath(string path) { return new FilePath(path); } - internal FileInfo GetFileInfo(FilePath file) + internal static FileInfo GetFileInfo(FilePath file) { - var absolutePath = file.IsAbsolute ? file : file.ToAbsolute(); + var absolutePath = file.IsAbsolute ? file : file.ToAbsolute(Environment.CurrentDirectory); return new FileInfo(absolutePath.ToString()); } - internal DirectoryInfo GetDirectoryInfo(DirectoryPath path) + internal static DirectoryInfo GetDirectoryInfo(DirectoryPath path) { - var absolutePath = path.IsAbsolute ? path : path.ToAbsolute(); + var absolutePath = path.IsAbsolute ? path : path.ToAbsolute(Environment.CurrentDirectory); return new DirectoryInfo(absolutePath.ToString()); } } diff --git a/source/CapriKit.IO/FileSystemEventListener.cs b/source/CapriKit.IO/FileSystemEventListener.cs deleted file mode 100644 index edc9f0f..0000000 --- a/source/CapriKit.IO/FileSystemEventListener.cs +++ /dev/null @@ -1,61 +0,0 @@ -namespace CapriKit.IO; - -public enum FileSystemChangeKind -{ - Created, - Changed, - Deleted, -} - -public delegate void FileSystemEventHandler(object sender, (FilePath target, FileSystemChangeKind reason) e); - -public sealed class FileSystemEventListener : IDisposable -{ - private readonly FileSystem FileSystem; - private readonly FileSystemWatcher Watcher; - - private event FileSystemEventHandler? onFileChanged; - - public FileSystemEventListener(FileSystem fileSystem, DirectoryPath directory, bool includeSubDirectories = true) - { - FileSystem = fileSystem; - Directory = directory; - - // Take the directory info since all file system types will point that to the true absolute path - var directoryInfo = FileSystem.GetDirectoryInfo(directory); - if (!directoryInfo.Exists) - { - throw new DirectoryNotFoundException($"Directory not found: {directoryInfo.FullName}"); - } - - Watcher = new FileSystemWatcher(directoryInfo.FullName) - { - NotifyFilter = NotifyFilters.LastWrite | NotifyFilters.FileName, - IncludeSubdirectories = includeSubDirectories, - EnableRaisingEvents = true, - }; - - Watcher.Created += (s, e) => onFileChanged?.Invoke(s, (FileSystem.GetFilePath(e.FullPath), FileSystemChangeKind.Created)); - Watcher.Changed += (s, e) => onFileChanged?.Invoke(s, (FileSystem.GetFilePath(e.FullPath), FileSystemChangeKind.Changed)); - Watcher.Deleted += (s, e) => onFileChanged?.Invoke(s, (FileSystem.GetFilePath(e.FullPath), FileSystemChangeKind.Deleted)); - } - - public event FileSystemEventHandler? OnFileChanged - { - add - { - onFileChanged += value; - } - remove - { - onFileChanged -= value; - } - } - - public DirectoryPath Directory { get; } - - public void Dispose() - { - Watcher.Dispose(); - } -} diff --git a/source/CapriKit.IO/IOUtilities.cs b/source/CapriKit.IO/IOUtilities.cs index e59d520..3656a1b 100644 --- a/source/CapriKit.IO/IOUtilities.cs +++ b/source/CapriKit.IO/IOUtilities.cs @@ -1,3 +1,6 @@ +using System.Buffers; +using System.Text; + namespace CapriKit.IO; public static class IOUtilities @@ -5,8 +8,8 @@ public static class IOUtilities private const char DirectorySeperator = '/'; private const char AltDirectorySeperator = '\\'; - private static readonly char[] InvalidFileNameChars = Path.GetInvalidFileNameChars(); - private static readonly char[] InvalidPathChars = Path.GetInvalidPathChars(); + private static readonly SearchValues InvalidFileNameChars = SearchValues.Create(Path.GetInvalidFileNameChars()); + private static readonly SearchValues InvalidPathChars = SearchValues.Create(Path.GetInvalidPathChars()); internal static StringComparison GetOSPathComparisonType() { @@ -77,6 +80,7 @@ public static ReadOnlySpan RemoveTrailingDirectorySeparator(ReadOnlySpan /// Determines if a file name is not just whitespace or contains any invalid character. /// + /// The list of invalid character (and thus the exact behavior of this method) is operating system dependent public static bool IsValidFileName(ReadOnlySpan name) { if (name.IsWhiteSpace()) @@ -84,13 +88,14 @@ public static bool IsValidFileName(ReadOnlySpan name) return false; } - return name.IndexOfAny(InvalidFileNameChars) < 0; + return !name.ContainsAny(InvalidFileNameChars); } /// /// Determines if a path to a file has any invalid characters. An empty /// path is invalid as it does not identify a single file. /// + /// The list of invalid character (and thus the exact behavior of this method) is operating system dependent public static bool IsValidFilePath(ReadOnlySpan path) { if (path.IsWhiteSpace()) @@ -108,9 +113,39 @@ public static bool IsValidFilePath(ReadOnlySpan path) /// Determines if a path contains any invalid characters, note that an empty path is valid /// as it could be a relative path to the current directory /// + /// The list of invalid character (and thus the exact behavior of this method) is operating system dependent public static bool IsValidPath(ReadOnlySpan path) { - return path.IndexOfAny(InvalidPathChars) < 0; + return !path.ContainsAny(InvalidPathChars); + } + + /// + /// Escapes a string so that it can be used as a valid file name + /// + /// The list of invalid character (and thus the exact behavior of this method) is operating system dependent + public static ReadOnlySpan EscapeFileName(ReadOnlySpan name) + { + if (IsValidFileName(name)) + { + return name; + } + + var builder = new StringBuilder(); + foreach (var c in name) + { + if (InvalidFileNameChars.Contains(c)) + { + var hexadecimal = ((int)c).ToString("X2"); + builder.Append('%'); + builder.Append(hexadecimal); + } + else + { + builder.Append(c); + } + } + + return builder.ToString(); } /// diff --git a/source/CapriKit.IO/IVirtualFileSystem.cs b/source/CapriKit.IO/IVirtualFileSystem.cs index f1270f6..8ffc761 100644 --- a/source/CapriKit.IO/IVirtualFileSystem.cs +++ b/source/CapriKit.IO/IVirtualFileSystem.cs @@ -1,3 +1,5 @@ +using CapriKit.IO.Watchers; + namespace CapriKit.IO; public interface IVirtualFileSystem : IReadOnlyVirtualFileSystem @@ -44,4 +46,21 @@ public interface IReadOnlyVirtualFileSystem /// Lists all files in the given directory. Throws an exception if the directory does not exist. /// IReadOnlyList List(DirectoryPath directory); + + /// + /// Watches for changes in the given subdirectory. + /// + IVirtualFileSystemWatcher Watch(DirectoryPath directory, bool includeSubDirectories = true); + + /// + /// Determines the absolute path of the given relative path, depending on the file system type + /// this can be resolved using the CWD or another method. + /// + DirectoryPath GetAbsolutePath(DirectoryPath directory); + + /// + /// Determines the absolute path of the given relative path, depending on the file system type + /// this can be resolved using the CWD or another method. + /// + FilePath GetAbsolutePath(FilePath filePath); } diff --git a/source/CapriKit.IO/InMemoryFileSystem.cs b/source/CapriKit.IO/InMemoryFileSystem.cs index 6dc1940..b3db41d 100644 --- a/source/CapriKit.IO/InMemoryFileSystem.cs +++ b/source/CapriKit.IO/InMemoryFileSystem.cs @@ -1,13 +1,17 @@ +using CapriKit.IO.Watchers; + namespace CapriKit.IO; public sealed class InMemoryFileSystem : IVirtualFileSystem { private record InMemoryFile(InMemoryFileStream Stream, DateTime LastWriteTime); private readonly Dictionary Disk; + private readonly List Watchers; public InMemoryFileSystem() { - Disk = new Dictionary(); + Disk = []; + Watchers = []; } public Stream AppendWrite(FilePath file) @@ -17,6 +21,7 @@ public Stream AppendWrite(FilePath file) var stream = inMemoryFile.Stream; stream.Position = stream.Length; + RaiseChange(file, FileSystemChangeKind.Changed); return stream; } @@ -27,17 +32,22 @@ public Stream CreateReadWrite(FilePath file) Disk[file] = inMemoryFile with { LastWriteTime = DateTime.Now }; inMemoryFile.Stream.SetLength(0); inMemoryFile.Stream.Position = 0; + RaiseChange(file, FileSystemChangeKind.Changed); return inMemoryFile.Stream; } var newStream = new InMemoryFileStream(); Disk.Add(file, new InMemoryFile(newStream, DateTime.Now)); + RaiseChange(file, FileSystemChangeKind.Created); return newStream; } public void Delete(FilePath file) { - Disk.Remove(file); + if (Disk.Remove(file)) + { + RaiseChange(file, FileSystemChangeKind.Deleted); + } } public bool Exists(FilePath file) @@ -78,6 +88,21 @@ public IReadOnlyList List(DirectoryPath directory) return files; } + public IVirtualFileSystemWatcher Watch(DirectoryPath directory, bool includeSubDirectories = true) + { + var watcher = new InMemoryFileSystemWatcher(this, directory, includeSubDirectories); + Watchers.Add(watcher); + return watcher; + } + + public FilePath GetAbsolutePath(FilePath file) => file.ToAbsolute(Environment.CurrentDirectory); + + public DirectoryPath GetAbsolutePath(DirectoryPath directory) + { + if (directory.IsAbsolute) { return directory; } + return directory.ToAbsolute(Environment.CurrentDirectory); + } + private InMemoryFile FindOrThrow(FilePath file) { if (Disk.TryGetValue(file, out var value)) @@ -87,22 +112,60 @@ private InMemoryFile FindOrThrow(FilePath file) throw new FileNotFoundException(null, file.ToString()); } + private void RaiseChange(FilePath file, FileSystemChangeKind kind) + { + var @event = new VirtualFileSystemEvent(file, kind); + foreach (var watcher in Watchers.ToArray()) + { + watcher.Notify(this, @event); + } + } + + private void RemoveWatcher(InMemoryFileSystemWatcher watcher) + { + Watchers.Remove(watcher); + } + private sealed class InMemoryFileStream : MemoryStream { + // Prevent disposing of the actual stream, just rewind it protected override void Dispose(bool disposing) { Position = 0; } + } + + private sealed class InMemoryFileSystemWatcher( + InMemoryFileSystem owner, DirectoryPath directory, bool includeSubDirectories) + : IVirtualFileSystemWatcher + { + private event VirtualFileSystemEventHandler? onFileChanged; - public override ValueTask DisposeAsync() + public event VirtualFileSystemEventHandler? OnFileChanged { - Position = 0; - return ValueTask.CompletedTask; + add => onFileChanged += value; + remove => onFileChanged -= value; } - public override void Close() + internal void Notify(object sender, VirtualFileSystemEvent @event) { - Position = 0; + if (Matches(@event.File)) + { + onFileChanged?.Invoke(sender, @event); + } + } + + private bool Matches(FilePath file) + { + var fileDirectory = file.Directory; + return includeSubDirectories + ? fileDirectory.StartsWith(directory) + : fileDirectory.Equals(directory); + } + + public void Stop() + { + owner.RemoveWatcher(this); } } } diff --git a/source/CapriKit.IO/ReadOnlyVirtualFileSystemSpy.cs b/source/CapriKit.IO/ReadOnlyVirtualFileSystemSpy.cs index 836062a..ec1bca9 100644 --- a/source/CapriKit.IO/ReadOnlyVirtualFileSystemSpy.cs +++ b/source/CapriKit.IO/ReadOnlyVirtualFileSystemSpy.cs @@ -1,3 +1,5 @@ +using CapriKit.IO.Watchers; + namespace CapriKit.IO; /// @@ -44,4 +46,13 @@ public long SizeInBytes(FilePath file) { return Actual.SizeInBytes(file); } + + public IVirtualFileSystemWatcher Watch(DirectoryPath directory, bool includeSubDirectories = true) + { + return Actual.Watch(directory, includeSubDirectories); + } + + public FilePath GetAbsolutePath(FilePath file) => Actual.GetAbsolutePath(file); + + public DirectoryPath GetAbsolutePath(DirectoryPath directory) => Actual.GetAbsolutePath(directory); } diff --git a/source/CapriKit.IO/ScopedFileSystem.cs b/source/CapriKit.IO/ScopedFileSystem.cs index 0e62093..2146964 100644 --- a/source/CapriKit.IO/ScopedFileSystem.cs +++ b/source/CapriKit.IO/ScopedFileSystem.cs @@ -1,3 +1,4 @@ +using CapriKit.IO.Watchers; using System.Diagnostics; namespace CapriKit.IO; @@ -38,36 +39,77 @@ public void Delete(FilePath file) /// that resolve to directories outside of the scope of this file system results /// in a ForbiddenPathException. /// -public class ReadOnlyScopedFileSystem(IReadOnlyVirtualFileSystem source, DirectoryPath basePath) : IReadOnlyVirtualFileSystem +public class ReadOnlyScopedFileSystem : IReadOnlyVirtualFileSystem { + private readonly IReadOnlyVirtualFileSystem Source; + + public ReadOnlyScopedFileSystem(IReadOnlyVirtualFileSystem source, DirectoryPath basePath) + { + Source = source; + if (basePath.IsAbsolute) + { + BasePath = basePath; + } + else + { + BasePath = source.GetAbsolutePath(basePath); + } + } + + + /// + /// Gets the absolute path of a file contained in this scoped file system. To be used with IO methods that are + /// not aware of the virtual file system. Throws if the file points to outside the scoped file system, + /// + public FilePath GetAbsolutePath(FilePath file) => GetFilePath(file); + + + /// + /// Gets the full path of a directory contained in this scoped file system. To be used with IO methods that are + /// not aware of the virtual file system. Throws if the directory points to outside the scoped file system, + /// + public DirectoryPath GetAbsolutePath(DirectoryPath directory) => GetDirectoryPath(directory); + + /// + /// The absolute path of the directory this file system is scoped to. + /// + public DirectoryPath BasePath { get; } + + public bool Exists(FilePath file) { - return source.Exists(GetFilePath(file)); + return Source.Exists(GetFilePath(file)); } public DateTime LastWriteTime(FilePath file) { - return source.LastWriteTime(GetFilePath(file)); + return Source.LastWriteTime(GetFilePath(file)); } public IReadOnlyList List(DirectoryPath directory) { - return source.List(GetDirectoryPath(directory)); + return Source.List(GetDirectoryPath(directory)); } public Stream OpenRead(FilePath file) { - return source.OpenRead(GetFilePath(file)); + return Source.OpenRead(GetFilePath(file)); } public long SizeInBytes(FilePath file) { - return source.SizeInBytes(GetFilePath(file)); + return Source.SizeInBytes(GetFilePath(file)); } protected DirectoryPath GetDirectoryPath(DirectoryPath path) { - var fullPath = path.GetPathRelativeTo(basePath); + if (path.IsAbsolute) + { + ThrowIfPathIsOutsideBasePath(path); + return path; + } + + var fullPath = BasePath.Append([path]); ThrowIfPathIsOutsideBasePath(fullPath); return fullPath; @@ -75,7 +117,13 @@ protected DirectoryPath GetDirectoryPath(DirectoryPath path) protected FilePath GetFilePath(FilePath path) { - var fullPath = path.GetPathRelativeTo(basePath); + if (path.IsAbsolute) + { + ThrowIfPathIsOutsideBasePath(path); + return path; + } + + var fullPath = BasePath.Append(path); ThrowIfPathIsOutsideBasePath(fullPath); return fullPath; @@ -84,18 +132,32 @@ protected FilePath GetFilePath(FilePath path) protected void ThrowIfPathIsOutsideBasePath(FilePath file) { Debug.Assert(file.IsAbsolute); - if (!file.StartsWith(basePath)) + if (!file.StartsWith(BasePath)) { - throw new ForbiddenPathException(file, basePath); + throw new ForbiddenPathException(file, BasePath); } } protected void ThrowIfPathIsOutsideBasePath(DirectoryPath path) { Debug.Assert(path.IsAbsolute); - if (!path.StartsWith(basePath)) + if (!path.StartsWith(BasePath)) { - throw new ForbiddenPathException(path, basePath); + throw new ForbiddenPathException(path, BasePath); } } + + public IVirtualFileSystemWatcher Watch(DirectoryPath directory, bool includeSubDirectories = true) + { + var fullPath = GetAbsolutePath(directory); + ThrowIfPathIsOutsideBasePath(fullPath); + var watchers = Source.Watch(fullPath, includeSubDirectories); + return new ScopedFileSystemEventListener(watchers, BasePath); + } + + public IVirtualFileSystemWatcher Watch(bool includeSubDirectories = true) + { + var watchers = Source.Watch(BasePath, includeSubDirectories); + return new ScopedFileSystemEventListener(watchers, BasePath); + } } diff --git a/source/CapriKit.IO/Streams/IBufferWriterExtensions.cs b/source/CapriKit.IO/Streams/IBufferWriterExtensions.cs new file mode 100644 index 0000000..da83f09 --- /dev/null +++ b/source/CapriKit.IO/Streams/IBufferWriterExtensions.cs @@ -0,0 +1,67 @@ +using System.Buffers; +using System.Buffers.Binary; +using System.Runtime.CompilerServices; +using System.Text; + +namespace CapriKit.IO.Streams; + +public static class BufferWriterExtensions +{ + /// + /// Writes a length prefixed string. The prefix is the byte count of the encoded string as a + /// 7 bit encoded integer, identical to + /// + /// The writer to write the string to + /// The string to write + /// Defaults to UTF8 + public static void Write(this IBufferWriter writer, string value, Encoding? encoding = null) + { + encoding = encoding ?? Encoding.UTF8; + var length = encoding.GetByteCount(value); + writer.Write7BitEncodedInt(length); + encoding.GetBytes(value, writer); + } + + /// + /// Writes an integer seven bits at a time, using the eighth bit to indicate that another byte + /// follows. The format is identical to + /// + public static void Write7BitEncodedInt(this IBufferWriter writer, int value) + { + const int MaxLengthInBytes = 5; // ceil(32 bits / 7 bits per byte) + + var span = writer.GetSpan(MaxLengthInBytes); + var written = 0; + + var uValue = (uint)value; + while (uValue > 0x7Fu) + { + span[written++] = (byte)(uValue | ~0x7Fu); + uValue >>= 7; + } + span[written++] = (byte)uValue; + + writer.Advance(written); + } + + public static void Write(this IBufferWriter writer, int value) + { + var span = writer.GetSpan(sizeof(int)); + BinaryPrimitives.WriteInt32LittleEndian(span, value); + writer.Advance(sizeof(int)); + } + + public static void Write(this IBufferWriter writer, long value) + { + var span = writer.GetSpan(sizeof(long)); + BinaryPrimitives.WriteInt64LittleEndian(span, value); + writer.Advance(sizeof(long)); + } + + public static void Write(this IBufferWriter writer, Guid guid) + { + var span = writer.GetSpan(Unsafe.SizeOf()); + guid.TryWriteBytes(span, false, out var written); + writer.Advance(written); + } +} diff --git a/source/CapriKit.IO/Streams/SequenceReaderExtensions.cs b/source/CapriKit.IO/Streams/SequenceReaderExtensions.cs new file mode 100644 index 0000000..c25b77e --- /dev/null +++ b/source/CapriKit.IO/Streams/SequenceReaderExtensions.cs @@ -0,0 +1,135 @@ +using System.Buffers; +using System.Runtime.CompilerServices; +using System.Text; + +namespace CapriKit.IO.Streams; + +public static class SequenceReaders +{ + public static SequenceReader Create(byte[] bytes, int start, int length) + { + var sequence = new ReadOnlySequence(bytes, start, length); + return new SequenceReader(sequence); + } +} + +public static class SequenceReaderExtensions +{ + /// + /// Creates a new reader to read a slice of the unread sequence. Advances the original reader. + /// Use this method when you want to delegate reading a part of the sequence to another method + /// without giving it access to the entire sequence. + /// + public static SequenceReader SliceUnread(ref this SequenceReader reader, int length) + { + if (!reader.TryReadExact(length, out var slice)) + { + throw new EndOfStreamException(); + } + + return new SequenceReader(slice); + } + + /// + /// Reads a length prefixed string written by + /// . + /// The format is identical to so both can be mixed + /// freely, as long as both sides use the same encoding + /// + /// The reader to read the string from + /// Defaults to UTF8 + public static string ReadString(this ref SequenceReader reader, Encoding? encoding = null) + { + encoding = encoding ?? Encoding.UTF8; + var length = reader.Read7BitEncodedInt(); + if (!reader.TryReadExact(length, out var sequence)) + { + throw new EndOfStreamException(); + } + + return encoding.GetString(in sequence); + } + + /// + /// Reads an integer written seven bits at a time by + /// + /// or + /// + public static int Read7BitEncodedInt(this ref SequenceReader reader) + { + const int MaxBytesWithoutOverflow = 4; + + var result = 0u; + for (var shift = 0; shift < MaxBytesWithoutOverflow * 7; shift += 7) + { + var current = ReadByte(ref reader); + result |= (current & 0x7Fu) << shift; + if (current <= 0x7Fu) + { + return (int)result; + } + } + + // The fifth byte can only hold the 4 remaining bits of a 32 bit integer + var last = ReadByte(ref reader); + if (last > 0b_1111u) + { + throw new FormatException("Invalid 7 bit encoded integer"); + } + + result |= (uint)last << (MaxBytesWithoutOverflow * 7); + return (int)result; + } + + public static int ReadInt32(this ref SequenceReader reader) + { + if (!reader.TryReadLittleEndian(out int value)) + { + throw new EndOfStreamException(); + } + + return value; + } + + public static long ReadInt64(this ref SequenceReader reader) + { + if (!reader.TryReadLittleEndian(out long value)) + { + throw new EndOfStreamException(); + } + + return value; + } + + public static Guid ReadGuid(this ref SequenceReader reader) + { + Span bytes = stackalloc byte[Unsafe.SizeOf()]; + if (!reader.TryCopyTo(bytes)) + { + throw new EndOfStreamException(); + } + + reader.Advance(bytes.Length); + return new Guid(bytes, bigEndian: false); + } + + public static byte[] ReadBytes(this ref SequenceReader reader, int length) + { + if (!reader.TryReadExact(length, out var sequence)) + { + throw new EndOfStreamException(); + } + + return sequence.ToArray(); + } + + public static byte ReadByte(ref SequenceReader reader) + { + if (!reader.TryRead(out var value)) + { + throw new EndOfStreamException(); + } + + return value; + } +} diff --git a/source/CapriKit.IO/VirtualFileSystemSpy.cs b/source/CapriKit.IO/VirtualFileSystemSpy.cs index a9ed7a0..18d840a 100644 --- a/source/CapriKit.IO/VirtualFileSystemSpy.cs +++ b/source/CapriKit.IO/VirtualFileSystemSpy.cs @@ -1,3 +1,5 @@ +using CapriKit.IO.Watchers; + namespace CapriKit.IO; /// @@ -64,4 +66,13 @@ public long SizeInBytes(FilePath file) { return Actual.SizeInBytes(file); } + + public IVirtualFileSystemWatcher Watch(DirectoryPath directory, bool includeSubDirectories = true) + { + return Actual.Watch(directory, includeSubDirectories); + } + + public FilePath GetAbsolutePath(FilePath file) => Actual.GetAbsolutePath(file); + + public DirectoryPath GetAbsolutePath(DirectoryPath directory) => Actual.GetAbsolutePath(directory); } diff --git a/source/CapriKit.IO/Watchers/FileSystemEventListener.cs b/source/CapriKit.IO/Watchers/FileSystemEventListener.cs new file mode 100644 index 0000000..a160f89 --- /dev/null +++ b/source/CapriKit.IO/Watchers/FileSystemEventListener.cs @@ -0,0 +1,56 @@ +namespace CapriKit.IO.Watchers; + +/// +/// Listens for file changes and notifies interested parties via an event +/// +public sealed class FileSystemEventListener : IVirtualFileSystemWatcher, IDisposable +{ + private readonly FileSystemWatcher Watcher; + + private event VirtualFileSystemEventHandler? onFileChanged; + + public FileSystemEventListener(DirectoryPath directory, bool includeSubDirectories = true) + { + Directory = directory; + var directoryInfo = FileSystem.GetDirectoryInfo(directory); + if (!directoryInfo.Exists) + { + throw new DirectoryNotFoundException($"Directory not found: {directoryInfo.FullName}"); + } + + Watcher = new FileSystemWatcher(directoryInfo.FullName) + { + NotifyFilter = NotifyFilters.LastWrite | NotifyFilters.FileName, + IncludeSubdirectories = includeSubDirectories, + EnableRaisingEvents = true, + }; + + Watcher.Created += (s, e) => onFileChanged?.Invoke(s, new VirtualFileSystemEvent(FileSystem.GetFilePath(e.FullPath), FileSystemChangeKind.Created)); + Watcher.Changed += (s, e) => onFileChanged?.Invoke(s, new VirtualFileSystemEvent(FileSystem.GetFilePath(e.FullPath), FileSystemChangeKind.Changed)); + Watcher.Deleted += (s, e) => onFileChanged?.Invoke(s, new VirtualFileSystemEvent(FileSystem.GetFilePath(e.FullPath), FileSystemChangeKind.Deleted)); + } + + public event VirtualFileSystemEventHandler? OnFileChanged + { + add + { + onFileChanged += value; + } + remove + { + onFileChanged -= value; + } + } + + public DirectoryPath Directory { get; } + + public void Stop() + { + Dispose(); + } + + public void Dispose() + { + Watcher.Dispose(); + } +} diff --git a/source/CapriKit.IO/Watchers/FileSystemEventQueue.cs b/source/CapriKit.IO/Watchers/FileSystemEventQueue.cs new file mode 100644 index 0000000..0f1e537 --- /dev/null +++ b/source/CapriKit.IO/Watchers/FileSystemEventQueue.cs @@ -0,0 +1,25 @@ +using System.Collections.Concurrent; +using System.Diagnostics.CodeAnalysis; + +namespace CapriKit.IO.Watchers; + +/// +/// Listens for file changes and puts events in a queue so an interested party can control when to handle them. +/// +public sealed class FileSystemEventQueue +{ + private readonly ConcurrentQueue Queue; + + public FileSystemEventQueue(IVirtualFileSystemWatcher watcher) + { + Queue = new ConcurrentQueue(); + watcher.OnFileChanged += (s, e) => Queue.Enqueue(e); + } + + public int Count => Queue.Count; + + public bool TryDequeue([NotNullWhen(true)] out VirtualFileSystemEvent? @event) + { + return Queue.TryDequeue(out @event); + } +} diff --git a/source/CapriKit.IO/Watchers/IVirtualFileSystemWatcher.cs b/source/CapriKit.IO/Watchers/IVirtualFileSystemWatcher.cs new file mode 100644 index 0000000..4602d81 --- /dev/null +++ b/source/CapriKit.IO/Watchers/IVirtualFileSystemWatcher.cs @@ -0,0 +1,21 @@ +namespace CapriKit.IO.Watchers; + +public enum FileSystemChangeKind +{ + Created, + Changed, + Deleted, +} + +/// The absolute path to the file affected +/// The kind of change the file underwent +public record VirtualFileSystemEvent(FilePath File, FileSystemChangeKind Kind); + +public delegate void VirtualFileSystemEventHandler(object sender, VirtualFileSystemEvent e); + +public interface IVirtualFileSystemWatcher +{ + public event VirtualFileSystemEventHandler? OnFileChanged; + + public void Stop(); +} diff --git a/source/CapriKit.IO/Watchers/ScopedFileSystemEventListener.cs b/source/CapriKit.IO/Watchers/ScopedFileSystemEventListener.cs new file mode 100644 index 0000000..61fbd46 --- /dev/null +++ b/source/CapriKit.IO/Watchers/ScopedFileSystemEventListener.cs @@ -0,0 +1,26 @@ +namespace CapriKit.IO.Watchers; + +/// +/// Wrapper for a general IVirtualFileSystemWatcher that ensures paths are relative to the basePath +/// +internal class ScopedFileSystemEventListener : IVirtualFileSystemWatcher +{ + private readonly IVirtualFileSystemWatcher Inner; + private readonly DirectoryPath BasePath; + + public ScopedFileSystemEventListener(IVirtualFileSystemWatcher inner, DirectoryPath basePath) + { + Inner = inner; + BasePath = basePath; + Inner.OnFileChanged += OnInner; + } + + public event VirtualFileSystemEventHandler? OnFileChanged; + + private void OnInner(object sender, VirtualFileSystemEvent e) + { + OnFileChanged?.Invoke(this, e with { File = e.File.GetPathRelativeTo(BasePath) }); + } + + public void Stop() { Inner.OnFileChanged -= OnInner; Inner.Stop(); } +} diff --git a/source/CapriKit.Tests.Tool/Program.cs b/source/CapriKit.Tests.Tool/Program.cs index a96ec03..1673707 100644 --- a/source/CapriKit.Tests.Tool/Program.cs +++ b/source/CapriKit.Tests.Tool/Program.cs @@ -1,4 +1,5 @@ using CapriKit.Concurrency.Async; +using CapriKit.Concurrency.Primitives; using CapriKit.DirectX11; using CapriKit.DirectX11.Debug; using CapriKit.IO; diff --git a/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs b/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs new file mode 100644 index 0000000..3c522a3 --- /dev/null +++ b/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs @@ -0,0 +1,412 @@ +using CapriKit.AssetPipeline; +using CapriKit.IO; +using CapriKit.IO.Streams; +using CapriKit.Tests.TestUtilities; +using Microsoft.Extensions.Logging.Abstractions; +using System.Buffers; +using System.Collections.Concurrent; + +namespace CapriKit.Tests.AssetPipeline; + +internal class AssetManagerTests +{ + private DirectoryPath? WorkingDirectory; + private readonly FilePath AssetFile = new("Hello.txt"); + private readonly FilePath HealthyFile = new("Goodbye.txt"); + private const string TranscoderText = "Hello World"; + + [Before(Test)] + public void Setup() + { + WorkingDirectory = FileSystemUtilities.CreateTemporaryDirectory(); + var fileSystem = new FileSystem().ScopedTo(WorkingDirectory); + using var stream = fileSystem.CreateReadWrite(AssetFile); + using var writer = new StreamWriter(stream); + writer.Write(TranscoderText); + } + + [After(Test)] + public void TearDown() + { + Directory.Delete(WorkingDirectory, true); + } + + [Test] + public async Task LoadAsset() + { + await Assert.That(WorkingDirectory).IsNotNull(); + + var logger = NullLoggerFactory.Instance; + var fileSystem = new FileSystem().ScopedTo(WorkingDirectory); + await fileSystem.WriteAllText(AssetFile, "Hello World"); + + var assetManager = new AssetManager(NullLoggerFactory.Instance, fileSystem); + + var transcoder = new TextTranscoder(); + assetManager.RegisterTranscoder(transcoder); + + var id = new AssetId(AssetFile); + + var bundle = assetManager.CreateBundle(); + var handle = bundle.Load(id, default); + var loader = bundle.Build(resolver => new TestBundle(resolver.Get(handle))); + + TestBundle? contents = null; + await Assert.That(() => + { + assetManager.Update(); + return loader.IsReady(out contents); + }) + .Eventually(v => v.IsTrue(), TimeSpan.FromSeconds(5)); + + await Assert.That(contents).IsNotNull(); + await Assert.That(contents.Asset.Text).IsEqualTo(TranscoderText); + + // Load again to verify loading the same thing twice gives us the cached value + var altBundle = assetManager.CreateBundle(); + var altHandle = altBundle.Load(id, default); + var altLoader = altBundle.Build(resolver => new TestBundle(resolver.Get(altHandle))); + + TestBundle? altContents = null; + await Assert.That(() => + { + assetManager.Update(); + return altLoader.IsReady(out altContents); + }) + .Eventually(v => v.IsTrue(), TimeSpan.FromSeconds(5)); + + await Assert.That(altContents).IsNotNull(); + await Assert.That(altContents.Asset).IsSameReferenceAs(contents.Asset); + + // Both bundles lease the one shared instance, so it only really goes away once both gave it back + bundle.Dispose(); + assetManager.Update(); + await Assert.That(contents.Asset.IsDisposed).IsFalse(); + + altBundle.Dispose(); + assetManager.Update(); + await Assert.That(contents.Asset.IsDisposed).IsTrue(); + + // Nothing is left over, so shutting down is quiet + await Assert.That(() => assetManager.Dispose()).ThrowsNothing(); + } + + /// + /// Unloading a bundle whose assets are still on their way used to leak them: the handles had taken no + /// lease yet so Unload had nothing to return, but the load still landed in a later Update and took one + /// that nobody would ever give back. Update now returns the lease of a handle whose bundle is gone. + /// + [Test] + public async Task Unload_ReturnsTheLeaseOfAnAssetThatWasStillLoading() + { + var fileSystem = new InMemoryFileSystem().ScopedTo("C:/Test"); + await fileSystem.WriteAllText(AssetFile, TranscoderText); + + var transcoder = new TrackingTextTranscoder(); + var assetManager = new AssetManager(NullLoggerFactory.Instance, fileSystem); + assetManager.RegisterTranscoder(transcoder); + + var bundle = assetManager.CreateBundle(); + var handle = bundle.Load(new AssetId(AssetFile)); + var loader = bundle.Build(resolver => new TestBundle(resolver.Get(handle))); + + // Act: unload before the first Update, so the asset is still loading and holds no lease yet. + // Disposing the bundle is the same thing as unloading it, and is how this is meant to be written. + bundle.Dispose(); + + // The load still finishes and still takes its lease, the manager has to hand that one straight back + await Assert.That(() => + { + assetManager.Update(); + return transcoder.Decoded.Count == 1 && transcoder.Decoded.All(asset => asset.IsDisposed); + }) + .Eventually(v => v.IsTrue(), TimeSpan.FromSeconds(5)); + + // The bundle no longer owns its assets, so its loader refuses to hand them out rather than + // serving disposed ones + await Assert.That(() => loader.IsReady(out _)).Throws(); + + await Assert.That(() => assetManager.Dispose()).ThrowsNothing(); + } + + /// + /// Everything that is loaded has to be unloaded before the game quits. The pool notices when that did + /// not happen but can only count the assets left behind, so the manager names the bundle they belong to + /// and the line that created it. It deliberately does not unload them: at shutdown that would only hide + /// the bug from whoever has to fix it. + /// + [Test] + public async Task Dispose_ReportsBundlesThatWereNeverUnloaded() + { + var fileSystem = new InMemoryFileSystem().ScopedTo("C:/Test"); + await fileSystem.WriteAllText(AssetFile, TranscoderText); + + var logger = new CapturingLoggerFactory(); + var assetManager = new AssetManager(logger, fileSystem); + assetManager.RegisterTranscoder(new TrackingTextTranscoder()); + + var bundle = assetManager.CreateBundle(); + var handle = bundle.Load(new AssetId(AssetFile)); + var loader = bundle.Build(resolver => new TestBundle(resolver.Get(handle))); + + await Assert.That(() => + { + assetManager.Update(); + return loader.IsReady(out _); + }) + .Eventually(v => v.IsTrue(), TimeSpan.FromSeconds(5)); + + // Act: quit without unloading anything + await Assert.That(() => assetManager.Dispose()).Throws(); + + var report = logger.Messages.SingleOrDefault(message => message.Contains("was never unloaded")); + await Assert.That(report).IsNotNull(); + await Assert.That(report!.Contains($"{nameof(AssetManagerTests)}.cs:")).IsTrue(); + } + + /// + /// A failed load used to wedge its asset id for the rest of the manager's life: the failure never + /// cleared the entry in Outstanding, and because Load hands every later request for that id to the + /// existing list instead of starting a new one, the asset could never be loaded again. Update now + /// forgets the failed request before it rethrows, so a later Load starts over. + /// + [Test] + public async Task Load_RetriesAnAssetWhoseFirstLoadFailed() + { + var fileSystem = new InMemoryFileSystem().ScopedTo("C:/Test"); + await fileSystem.WriteAllText(AssetFile, TranscoderText); + + var transcoder = new TranscoderThatCanFail { ShouldFail = true }; + + // Deliberately not a `using`: AssetPool.Dispose throws when leases are outstanding, and an exception + // from a dispose during unwinding replaces the assertion that actually failed. Disposing at the end + // keeps the leak check but lets a real failure report itself. + var assetManager = new AssetManager(NullLoggerFactory.Instance, fileSystem); + assetManager.RegisterTranscoder(transcoder); + + var id = new AssetId(AssetFile); + + // Act: the first load fails while building. Update stays quiet about it, the failure is handed to + // the bundle that was waiting and surfaces from IsReady. + var failedBundle = assetManager.CreateBundle(); + var failedHandle = failedBundle.Load(id); + var failedLoader = failedBundle.Build(resolver => new TestBundle(resolver.Get(failedHandle))); + + AssetLoadException? failure = null; + await Assert.That(() => + { + // Not guarded on purpose: Update throwing here fails this test, which is the point + assetManager.Update(); + + try { failedLoader.IsReady(out _); } + catch (AssetLoadException ex) { failure = ex; } + return failure is not null; + }) + .Eventually(v => v.IsTrue(), TimeSpan.FromSeconds(5)); + + // The exception names the asset that failed and keeps the transcoder's own exception as the cause + await Assert.That(failure!.Asset).IsEqualTo(id); + await Assert.That(failure!.InnerException).IsTypeOf(); + await Assert.That(transcoder.Attempts).IsEqualTo(1); + + // Act: take away the reason the build failed and ask for the very same asset again + transcoder.ShouldFail = false; + + var retryBundle = assetManager.CreateBundle(); + var retryHandle = retryBundle.Load(id); + var retryLoader = retryBundle.Build(resolver => new TestBundle(resolver.Get(retryHandle))); + + // Control: a second, untouched asset requested at the same moment, to show that a failure never + // stopped the manager as a whole and that the retry above is what actually changed. + await fileSystem.WriteAllText(HealthyFile, TranscoderText); + + var healthyBundle = assetManager.CreateBundle(); + var healthyHandle = healthyBundle.Load(new AssetId(HealthyFile)); + var healthyLoader = healthyBundle.Build(resolver => new TestBundle(resolver.Get(healthyHandle))); + + await Assert.That(() => + { + assetManager.Update(); + return healthyLoader.IsReady(out _); + }) + .Eventually(v => v.IsTrue(), TimeSpan.FromSeconds(5)); + + // Assert: the retry really rebuilt the asset, so the failure, the control and the retry each + // reached the transcoder exactly once + TestBundle? retried = null; + await Assert.That(() => + { + assetManager.Update(); + return retryLoader.IsReady(out retried); + }) + .Eventually(v => v.IsTrue(), TimeSpan.FromSeconds(5)); + + await Assert.That(transcoder.Attempts).IsEqualTo(3); + await Assert.That(retried!.Asset.Text).IsEqualTo(TranscoderText); + + // The first bundle keeps reporting its failure rather than quietly never completing + await Assert.That(() => failedLoader.IsReady(out _)).Throws(); + + // Every bundle can be unloaded, the failed one included: it returns nothing because its only handle + // never took a lease. Getting that wrong would return the lease that retryBundle holds on the same + // asset, so the clean dispose below is what proves the counting is right. + assetManager.Unload(failedBundle); + assetManager.Unload(retryBundle); + assetManager.Unload(healthyBundle); + await Assert.That(() => assetManager.Dispose()).ThrowsNothing(); + } + + /// + /// Forgetting to register a transcoder is a programmer error rather than a broken asset, so it has to + /// surface on the calling thread instead of arriving as a failed load a few frames later. Looking the + /// transcoder up before the cache is checked keeps that true whether or not the asset happens to be + /// cached already. + /// + [Test] + public async Task Load_ThrowsOnTheCallingThreadWhenNoTranscoderIsRegistered() + { + var fileSystem = new InMemoryFileSystem().ScopedTo("C:/Test"); + await fileSystem.WriteAllText(AssetFile, TranscoderText); + + using var assetManager = new AssetManager(NullLoggerFactory.Instance, fileSystem); + var bundle = assetManager.CreateBundle(); + + await Assert.That(() => bundle.Load(new AssetId(AssetFile))).Throws(); + } + + /// + /// Leases are counted per handle, not per distinct asset, so a bundle that asks for the same asset + /// twice holds two leases and has to give back two. Unloading used to work from the distinct asset ids + /// and gave back one, which leaked the asset for the rest of the program. + /// + [Test] + public async Task Unload_ReturnsOneLeasePerHandle() + { + var fileSystem = new InMemoryFileSystem().ScopedTo("C:/Test"); + await fileSystem.WriteAllText(AssetFile, TranscoderText); + + var assetManager = new AssetManager(NullLoggerFactory.Instance, fileSystem); + assetManager.RegisterTranscoder(new TextTranscoder()); + + var id = new AssetId(AssetFile); + + var bundle = assetManager.CreateBundle(); + var first = bundle.Load(id); + var second = bundle.Load(id); + var loader = bundle.Build(resolver => new TwiceBundle(resolver.Get(first), resolver.Get(second))); + + TwiceBundle? contents = null; + await Assert.That(() => + { + assetManager.Update(); + return loader.IsReady(out contents); + }) + .Eventually(v => v.IsTrue(), TimeSpan.FromSeconds(5)); + + // Both handles resolved to the single cached instance, but each of them took its own lease + await Assert.That(contents!.Second).IsSameReferenceAs(contents.First); + + assetManager.Unload(bundle); + await Assert.That(() => assetManager.Dispose()).ThrowsNothing(); + } + + /// + /// The counterpart of . A transcoder that throws + /// while loading is fatal, but the very same transcoder throwing during a hot-reload must not be: the + /// asset that is already live is still perfectly usable, and taking the game down over a development + /// feature would be worse than the stale contents. + /// + [Test] + public async Task Update_DoesNotThrowWhenAHotReloadFails() + { + var fileSystem = new InMemoryFileSystem().ScopedTo("C:/Test"); + await fileSystem.WriteAllText(AssetFile, TranscoderText); + + var transcoder = new TranscoderThatCanFail(); + var assetManager = new AssetManager(NullLoggerFactory.Instance, fileSystem); + assetManager.RegisterTranscoder(transcoder); + + var bundle = assetManager.CreateBundle(); + var handle = bundle.Load(new AssetId(AssetFile)); + var loader = bundle.Build(resolver => new TestBundle(resolver.Get(handle))); + + TestBundle? contents = null; + await Assert.That(() => + { + assetManager.Update(); + return loader.IsReady(out contents); + }) + .Eventually(v => v.IsTrue(), TimeSpan.FromSeconds(5)); + + // Act: make every rebuild fail, then change the file the asset was built from + transcoder.ShouldFail = true; + await fileSystem.WriteAllText(AssetFile, "Goodbye World"); + + // Pump past the hot-reload debounce until the rebuild has actually been attempted and failed + Exception? thrown = null; + await Assert.That(() => + { + try { assetManager.Update(); } + catch (Exception ex) { thrown = ex; } + return transcoder.FailedAttempts; + }) + .Eventually(v => v.IsGreaterThan(0), TimeSpan.FromSeconds(10)); + + // Assert: the failure stayed inside the hot-reload path and the asset kept its old contents + await Assert.That(thrown).IsNull(); + await Assert.That(contents!.Asset.Text).IsEqualTo(TranscoderText); + + assetManager.Unload(bundle); + assetManager.Dispose(); + } +} + +internal record TestBundle(TextAsset Asset); + +internal record TwiceBundle(TextAsset First, TextAsset Second); + +internal sealed class TextAsset(string text) : IDisposable +{ + public string Text { get; set; } = text; + + /// Lets tests see whether the pool really let go of this asset. + public bool IsDisposed { get; private set; } + + public void Dispose() => IsDisposed = true; +} + +/// +/// Hands out the assets it decoded so that a test can look at an asset the asset manager never gave it. +/// +internal sealed class TrackingTextTranscoder : TextTranscoder +{ + private readonly ConcurrentBag decoded = []; + + public IReadOnlyCollection Decoded => decoded; + + public override TextAsset Decode(AssetId id, ref SequenceReader reader) + { + var asset = base.Decode(id, ref reader); + decoded.Add(asset); + return asset; + } +} + +internal class TextTranscoder() : NoSettingsTranscoder(Guid.Parse("{6E4A1D0C-1F73-4C4E-9D2E-0B7F5C6A9E31}"), 1) +{ + public override TextAsset Decode(AssetId id, ref SequenceReader reader) + { + return new TextAsset(reader.ReadString()); + } + + public override async Task Encode(AssetId id, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer) + { + var text = await fileSystem.ReadAllText(id.Path); + writer.Write(text); + } + + public override void HotSwap(TextAsset instance, TextAsset newParts) + { + instance.Text = newParts.Text; + } +} diff --git a/source/CapriKit.Tests/AssetPipeline/AssetPoolTests.cs b/source/CapriKit.Tests/AssetPipeline/AssetPoolTests.cs new file mode 100644 index 0000000..0ef152f --- /dev/null +++ b/source/CapriKit.Tests/AssetPipeline/AssetPoolTests.cs @@ -0,0 +1,442 @@ +using CapriKit.AssetPipeline; + +namespace CapriKit.Tests.AssetPipeline; + +/// +/// The pool is where the loading threads, the hot-reloader and the main thread meet. A mistake in its +/// reference counting either disposes an asset that is still in use or leaks it, so these tests go beyond +/// the happy path and pin down the lease/return contract, the deferred disposal and the threading. +/// +internal class AssetPoolTests +{ + private static readonly AssetId Id = new("Hello.txt"); + private static readonly AssetId OtherId = new("Goodbye.txt"); + + [Test] + public async Task PutOrLease() + { + var pool = new AssetPool(); + var resource = new Resource(); + + var stored = pool.PutOrLease(Id, resource); + + await Assert.That(stored).IsSameReferenceAs(resource); + await Assert.That(resource.DisposeCount).IsEqualTo(0); + + // Putting an asset takes the first lease on it, returning that lease releases the asset again + pool.Return(Id); + pool.DisposeReleased(); + await Assert.That(resource.DisposeCount).IsEqualTo(1); + + pool.Dispose(); + } + + [Test] + public async Task PutOrLease_DisposesTheLoserAndLeasesTheWinner() + { + var pool = new AssetPool(); + var winner = new Resource(); + var loser = new Resource(); + + pool.PutOrLease(Id, winner); + var stored = pool.PutOrLease(Id, loser); + + await Assert.That(stored).IsSameReferenceAs(winner); + + // Disposal always waits for the main thread, never happens inside PutOrLease itself + await Assert.That(loser.DisposeCount).IsEqualTo(0); + pool.DisposeReleased(); + await Assert.That(loser.DisposeCount).IsEqualTo(1); + await Assert.That(winner.DisposeCount).IsEqualTo(0); + + // Both calls took a lease on the winner, so both have to be returned + pool.Return(Id); + pool.DisposeReleased(); + await Assert.That(winner.DisposeCount).IsEqualTo(0); + + pool.Return(Id); + pool.DisposeReleased(); + await Assert.That(winner.DisposeCount).IsEqualTo(1); + + pool.Dispose(); + } + + [Test] + public async Task PutOrLease_LeasesWithoutDisposingWhenTheSameInstanceIsPutTwice() + { + // The asset manager materializes an asset once per waiting handle but always from the same loaded + // instance, so putting the identical object twice is a normal path rather than a corner case. + var pool = new AssetPool(); + var resource = new Resource(); + + pool.PutOrLease(Id, resource); + var stored = pool.PutOrLease(Id, resource); + + await Assert.That(stored).IsSameReferenceAs(resource); + + // The instance is live and now leased twice, collecting must not touch it + pool.DisposeReleased(); + await Assert.That(resource.DisposeCount).IsEqualTo(0); + + pool.Return(Id); + pool.DisposeReleased(); + await Assert.That(resource.DisposeCount).IsEqualTo(0); + + pool.Return(Id); + pool.DisposeReleased(); + await Assert.That(resource.DisposeCount).IsEqualTo(1); + + pool.Dispose(); + } + + [Test] + public async Task PutOrLease_ThrowsWhenTheAssetIsStoredAsAnotherType() + { + var pool = new AssetPool(); + var stored = new Resource(); + var wrongType = new OtherResource(); + + pool.PutOrLease(Id, stored); + + await Assert.That(() => pool.PutOrLease(Id, wrongType)).Throws(); + + // The rejected candidate is still cleaned up, and the failed call did not take a lease + pool.DisposeReleased(); + await Assert.That(wrongType.DisposeCount).IsEqualTo(1); + await Assert.That(stored.DisposeCount).IsEqualTo(0); + + pool.Return(Id); + pool.DisposeReleased(); + await Assert.That(stored.DisposeCount).IsEqualTo(1); + + pool.Dispose(); + } + + [Test] + public async Task TryLease() + { + var pool = new AssetPool(); + var resource = new Resource(); + pool.PutOrLease(Id, resource); + + var found = pool.TryLease(Id, out var leased); + + await Assert.That(found).IsTrue(); + await Assert.That(leased).IsSameReferenceAs(resource); + + pool.Return(Id); + pool.Return(Id); + pool.DisposeReleased(); + pool.Dispose(); + } + + [Test] + public async Task TryLease_ReturnsFalseForAnUnknownAsset() + { + var pool = new AssetPool(); + + var found = pool.TryLease(Id, out var leased); + + await Assert.That(found).IsFalse(); + await Assert.That(leased).IsNull(); + + pool.Dispose(); + } + + [Test] + public async Task TryLease_ReturnsFalseForAnAssetThatIsWaitingToBeDisposed() + { + // Returning the last lease evicts the asset immediately and only defers the disposal itself, so a + // late lease has to miss instead of handing out an asset that is about to be disposed. + var pool = new AssetPool(); + var resource = new Resource(); + pool.PutOrLease(Id, resource); + pool.Return(Id); + + var found = pool.TryLease(Id, out var leased); + + await Assert.That(found).IsFalse(); + await Assert.That(leased).IsNull(); + + pool.DisposeReleased(); + await Assert.That(resource.DisposeCount).IsEqualTo(1); + + pool.Dispose(); + } + + [Test] + public async Task TryLease_ThrowsWhenTheAssetIsStoredAsAnotherType() + { + var pool = new AssetPool(); + var resource = new Resource(); + pool.PutOrLease(Id, resource); + + await Assert.That(() => pool.TryLease(Id, out _)).Throws(); + + // The failed lease must not have counted, so a single return still releases the asset + pool.Return(Id); + pool.DisposeReleased(); + await Assert.That(resource.DisposeCount).IsEqualTo(1); + + pool.Dispose(); + } + + [Test] + public async Task Return() + { + var pool = new AssetPool(); + var resource = new Resource(); + pool.PutOrLease(Id, resource); + pool.TryLease(Id, out _); + + pool.Return(Id); + + // One lease is left, so the asset stays alive and leaseable + pool.DisposeReleased(); + await Assert.That(resource.DisposeCount).IsEqualTo(0); + await Assert.That(pool.TryLease(Id, out _)).IsTrue(); + pool.Return(Id); + + pool.Return(Id); + + // The last return evicts the asset but leaves the disposal to the main thread + await Assert.That(resource.DisposeCount).IsEqualTo(0); + pool.DisposeReleased(); + await Assert.That(resource.DisposeCount).IsEqualTo(1); + + pool.Dispose(); + } + + [Test] + public async Task Return_ThrowsForAnAssetThatWasNeverStored() + { + var pool = new AssetPool(); + + await Assert.That(() => pool.Return(Id)).Throws(); + + pool.Dispose(); + } + + [Test] + public async Task Return_ThrowsWhenReturnedMoreOftenThanLeased() + { + var pool = new AssetPool(); + pool.PutOrLease(Id, new Resource()); + pool.Return(Id); + + // The entry is gone once the last lease came back, so an extra return is a bug in the caller + await Assert.That(() => pool.Return(Id)).Throws(); + + pool.DisposeReleased(); + pool.Dispose(); + } + + [Test] + public async Task DisposeReleased() + { + var pool = new AssetPool(); + var first = new Resource(); + var second = new Resource(); + pool.PutOrLease(Id, first); + pool.PutOrLease(OtherId, second); + pool.Return(Id); + pool.Return(OtherId); + + pool.DisposeReleased(); + + await Assert.That(first.DisposeCount).IsEqualTo(1); + await Assert.That(second.DisposeCount).IsEqualTo(1); + + // Collecting again must not dispose anything a second time + pool.DisposeReleased(); + await Assert.That(first.DisposeCount).IsEqualTo(1); + await Assert.That(second.DisposeCount).IsEqualTo(1); + + pool.Dispose(); + } + + [Test] + public async Task DisposeReleased_IgnoresAssetsThatAreNotDisposable() + { + var pool = new AssetPool(); + pool.PutOrLease(Id, new PlainResource()); + pool.Return(Id); + + pool.DisposeReleased(); + + await Assert.That(pool.TryLease(Id, out _)).IsFalse(); + + pool.Dispose(); + } + + [Test] + public async Task DisposeReleased_DoesNotHoldTheLockWhileDisposing() + { + // Assets dispose graphics resources and may well touch the pool themselves. Doing that while + // holding the lock would deadlock against every other thread that loads or returns an asset. + var pool = new AssetPool(); + pool.PutOrLease(OtherId, new Resource()); + + var otherThreadCouldUseThePool = false; + var reentrant = new CallbackResource(() => + { + var lease = Task.Run(() => pool.TryLease(OtherId, out _)); + otherThreadCouldUseThePool = lease.Wait(TimeSpan.FromSeconds(5)) && lease.Result; + }); + + pool.PutOrLease(Id, reentrant); + pool.Return(Id); + + pool.DisposeReleased(); + + await Assert.That(otherThreadCouldUseThePool).IsTrue(); + + pool.Return(OtherId); + pool.Return(OtherId); + pool.DisposeReleased(); + pool.Dispose(); + } + + [Test] + public async Task Dispose() + { + var pool = new AssetPool(); + var resource = new Resource(); + pool.PutOrLease(Id, resource); + pool.Return(Id); + + // Everything came back, so disposing is clean and still collects what was queued + pool.Dispose(); + + await Assert.That(resource.DisposeCount).IsEqualTo(1); + } + + [Test] + public async Task Dispose_ThrowsWhenAssetsWereNotReturned() + { + var pool = new AssetPool(); + var resource = new Resource(); + pool.PutOrLease(Id, resource); + + // The pool is the only place that can catch a bundle that was never unloaded + await Assert.That(() => pool.Dispose()).Throws(); + + // The leaked asset is reported, not disposed: somebody out there still holds a reference to it + await Assert.That(resource.DisposeCount).IsEqualTo(0); + } + + [Test] + public async Task Dispose_OnlyReportsLeaksOnce() + { + var pool = new AssetPool(); + pool.PutOrLease(Id, new Resource()); + + await Assert.That(() => pool.Dispose()).Throws(); + + // A pool in a using block is disposed a second time while the first exception unwinds + await Assert.That(() => pool.Dispose()).ThrowsNothing(); + } + + [Test] + public async Task Dispose_MakesThePoolUnusable() + { + var pool = new AssetPool(); + pool.Dispose(); + + await Assert.That(() => pool.PutOrLease(Id, new Resource())).Throws(); + await Assert.That(() => pool.TryLease(Id, out _)).Throws(); + await Assert.That(() => pool.Return(Id)).Throws(); + + // Collecting stays safe so that a game loop that is shutting down does not have to check + await Assert.That(() => pool.DisposeReleased()).ThrowsNothing(); + } + + [Test] + public async Task PutOrLease_IsThreadSafe() + { + // Several loading threads can finish the same asset at the same time. Exactly one instance may + // win, every caller must get a lease on that winner and every loser must be cleaned up. + const int threads = 16; + var pool = new AssetPool(); + var candidates = Enumerable.Range(0, threads).Select(_ => new Resource()).ToArray(); + + var results = await Task.WhenAll(candidates.Select(candidate => Task.Run(() => pool.PutOrLease(Id, candidate)))); + + var winner = results[0]; + await Assert.That(results.Distinct().Count()).IsEqualTo(1); + await Assert.That(candidates.Count(candidate => ReferenceEquals(candidate, winner))).IsEqualTo(1); + + pool.DisposeReleased(); + await Assert.That(winner.DisposeCount).IsEqualTo(0); + await Assert.That(candidates.Sum(candidate => candidate.DisposeCount)).IsEqualTo(threads - 1); + + // Every call took a lease, so the winner only dies after the last one comes back + for (var i = 0; i < threads; i++) + { + pool.Return(Id); + } + + pool.DisposeReleased(); + await Assert.That(winner.DisposeCount).IsEqualTo(1); + + pool.Dispose(); + } + + [Test] + public async Task TryLease_IsThreadSafe() + { + // The reference count is the only thing keeping an asset alive, so it has to survive many threads + // leasing and returning the same asset at the same time. + const int threads = 8; + const int iterations = 500; + + var pool = new AssetPool(); + var resource = new Resource(); + + // This lease keeps the asset alive for the entire test + pool.PutOrLease(Id, resource); + + await Task.WhenAll(Enumerable.Range(0, threads).Select(thread => Task.Run(() => + { + for (var i = 0; i < iterations; i++) + { + if (pool.TryLease(Id, out _)) + { + pool.Return(Id); + } + } + }))); + + // The asset was never evicted along the way, so only the initial lease is left + pool.DisposeReleased(); + await Assert.That(resource.DisposeCount).IsEqualTo(0); + + pool.Return(Id); + pool.DisposeReleased(); + await Assert.That(resource.DisposeCount).IsEqualTo(1); + + pool.Dispose(); + } + + /// Counts disposals so that tests can tell a missing, a late and a double dispose apart. + private class CountingResource : IDisposable + { + private int disposeCount; + + public int DisposeCount => Volatile.Read(ref disposeCount); + + public void Dispose() => Interlocked.Increment(ref disposeCount); + } + + private sealed class Resource : CountingResource; + + /// Unrelated to so that casting one to the other fails. + private sealed class OtherResource : CountingResource; + + private sealed class PlainResource; + + private sealed class CallbackResource(Action onDispose) : IDisposable + { + public void Dispose() => onDispose(); + } +} diff --git a/source/CapriKit.Tests/AssetPipeline/HotReloadManagerTests.cs b/source/CapriKit.Tests/AssetPipeline/HotReloadManagerTests.cs new file mode 100644 index 0000000..3f38da0 --- /dev/null +++ b/source/CapriKit.Tests/AssetPipeline/HotReloadManagerTests.cs @@ -0,0 +1,111 @@ +using CapriKit.AssetPipeline; +using CapriKit.IO; +using Microsoft.Extensions.Logging.Abstractions; +using System.Buffers; + +namespace CapriKit.Tests.AssetPipeline; + +internal class HotReloadManagerTests +{ + private static readonly FilePath AssetFile = new("Hello.txt"); + private static readonly TimeSpan NoDebounce = TimeSpan.Zero; + + [Test] + public async Task Update() + { + // Arrange: build, load and cache an asset the way the asset manager would + var fileSystem = new InMemoryFileSystem().ScopedTo("C:/Test"); + await fileSystem.WriteAllText(AssetFile, "Hello World"); + + var transcoder = new TextTranscoder(); + var id = new AssetId(AssetFile); + + await AssetEncoder.Encode(id, transcoder, default, fileSystem); + var asset = await AssetDecoder.Decode(id, transcoder, fileSystem); + + using var cache = new AssetPool(); + var live = cache.PutOrLease(id, asset.Value); + + using var sut = new HotReloadManager(NullLoggerFactory.Instance, cache, fileSystem, NoDebounce); + sut.Track(asset, transcoder); + + // Act: change the file the asset was built from + await fileSystem.WriteAllText(AssetFile, "Goodbye World"); + + await Assert.That(() => + { + sut.Update(); + return live.Text; + }) + .Eventually(v => v.IsEqualTo("Goodbye World"), TimeSpan.FromSeconds(5)); + + // Assert: the caller's instance was updated in place, and we left no lease behind + await Assert.That(live.Text).IsEqualTo("Goodbye World"); + cache.Return(id); + } + + [Test] + public async Task Update_RebuildFails() + { + // Arrange: build, load and cache an asset, then make every following rebuild fail + var fileSystem = new InMemoryFileSystem().ScopedTo("C:/Test"); + await fileSystem.WriteAllText(AssetFile, "Hello World"); + + var transcoder = new TranscoderThatCanFail(); + var id = new AssetId(AssetFile); + + await AssetEncoder.Encode(id, transcoder, default, fileSystem); + var asset = await AssetDecoder.Decode(id, transcoder, fileSystem); + + using var cache = new AssetPool(); + var live = cache.PutOrLease(id, asset.Value); + + var sut = new HotReloadManager(NullLoggerFactory.Instance, cache, fileSystem, NoDebounce); + sut.Track(asset, transcoder); + transcoder.ShouldFail = true; + + // Act: one update starts the rebuild, disposing waits for it and finishes it + await fileSystem.WriteAllText(AssetFile, "Goodbye World"); + sut.Update(); + sut.Dispose(); + + // Assert: the rebuild really was attempted, the live asset kept its contents, and returning the last + // lease empties the cache. If the manager leaked its lease the cache throws when it is disposed. + await Assert.That(transcoder.FailedAttempts).IsEqualTo(1); + await Assert.That(live.Text).IsEqualTo("Hello World"); + cache.Return(id); + } +} + + + +/// +/// Builds like a until is set. Encoding runs on the +/// thread pool while tests flip the switch and read the counters from the main thread, so all three cross +/// that boundary explicitly. +/// +internal sealed class TranscoderThatCanFail : TextTranscoder +{ + private volatile bool shouldFail; + private int attempts; + private int failedAttempts; + + public bool ShouldFail { get => shouldFail; set => shouldFail = value; } + + /// Every build the manager asked for, whether it succeeded or not. + public int Attempts => Volatile.Read(ref attempts); + + public int FailedAttempts => Volatile.Read(ref failedAttempts); + + public override Task Encode(AssetId id, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer) + { + Interlocked.Increment(ref attempts); + + if (ShouldFail) + { + Interlocked.Increment(ref failedAttempts); + throw new InvalidOperationException("Rebuilding this asset failed on purpose"); + } + return base.Encode(id, fileSystem, writer); + } +} diff --git a/source/CapriKit.Tests/CapriKit.Tests.csproj b/source/CapriKit.Tests/CapriKit.Tests.csproj index c907671..78a40ca 100644 --- a/source/CapriKit.Tests/CapriKit.Tests.csproj +++ b/source/CapriKit.Tests/CapriKit.Tests.csproj @@ -13,8 +13,10 @@ + + @@ -25,7 +27,4 @@ - - - diff --git a/source/CapriKit.Tests/Concurrency/Async/FireAndForgetExtensionsTests.cs b/source/CapriKit.Tests/Concurrency/Async/FireAndForgetExtensionsTests.cs index 85a1a0a..4c7dcd4 100644 --- a/source/CapriKit.Tests/Concurrency/Async/FireAndForgetExtensionsTests.cs +++ b/source/CapriKit.Tests/Concurrency/Async/FireAndForgetExtensionsTests.cs @@ -1,4 +1,5 @@ using CapriKit.Concurrency.Async; +using System.Runtime.ExceptionServices; namespace CapriKit.Tests.Concurrency.Async; @@ -31,7 +32,7 @@ public async Task FireAndForget_AlreadyCompletedTask() [Test] public async Task FireAndForget_CancelledTask() { - Exception? observed = null; + ExceptionDispatchInfo? observed = null; var completed = new TaskCompletionSource(); var cancelled = Task.FromCanceled(new CancellationToken(canceled: true)); @@ -44,14 +45,14 @@ public async Task FireAndForget_CancelledTask() [Test] public async Task FireAndForget_FaultedTask() { - Exception? observed = null; + ExceptionDispatchInfo? observed = null; var completed = new TaskCompletionSource(); var faulted = Task.FromException(new InvalidOperationException()); faulted.FireAndForget(ex => observed = ex, () => completed.SetResult()); await completed.Task.WaitAsync(Timeout); - await Assert.That(observed).IsTypeOf(); - await Assert.That(observed!.InnerException).IsTypeOf(); + await Assert.That(observed).IsNotNull(); + await Assert.That(observed.SourceException).IsTypeOf(); } } diff --git a/source/CapriKit.Tests/Concurrency/Primitives/LightweightChannelTests.cs b/source/CapriKit.Tests/Concurrency/Primitives/LightweightChannelTests.cs index 6b19fa3..d0e2a8f 100644 --- a/source/CapriKit.Tests/Concurrency/Primitives/LightweightChannelTests.cs +++ b/source/CapriKit.Tests/Concurrency/Primitives/LightweightChannelTests.cs @@ -1,4 +1,5 @@ using CapriKit.Concurrency.Primitives; +using System.Runtime.ExceptionServices; namespace CapriKit.Tests.Concurrency.Primitives; @@ -32,35 +33,16 @@ public async Task TryRead_ReturnsFalseWhenChannelIsEmpty() } [Test] - public async Task TryRead_RethrowsTheWrittenException() + public async Task TryRead_RethrowsTheWrittenExceptionBeforeReturningOtherItems() { var channel = new LightweightChannel(); - channel.Write(new InvalidOperationException()); + channel.Write(4); + channel.Write(ExceptionDispatchInfo.Capture(new InvalidOperationException())); await Assert.That(() => channel.TryRead(out _)).Throws(); - } - - [Test] - public async Task TryRead_DrainsQueuedItemsBeforeThrowingTheException() - { - var channel = new LightweightChannel(); - channel.Write(new InvalidOperationException()); - channel.Write(42); var read = channel.TryRead(out var value); - await Assert.That(read).IsTrue(); - await Assert.That(value).IsEqualTo(42); - await Assert.That(() => channel.TryRead(out _)).Throws(); - } - - [Test] - public async Task Write_KeepsOnlyTheFirstException() - { - var channel = new LightweightChannel(); - channel.Write(new InvalidOperationException()); - channel.Write(new FormatException()); - - await Assert.That(() => channel.TryRead(out _)).Throws(); + await Assert.That(value).IsEqualTo(4); } } diff --git a/source/CapriKit.Tests/DirectX11/Buffers/GenericBufferTests.cs b/source/CapriKit.Tests/DirectX11/Buffers/GenericBufferTests.cs index 4c3729d..82320c2 100644 --- a/source/CapriKit.Tests/DirectX11/Buffers/GenericBufferTests.cs +++ b/source/CapriKit.Tests/DirectX11/Buffers/GenericBufferTests.cs @@ -11,7 +11,7 @@ internal class GenericBufferTests { /// /// Tests StructuredBuffer, RWStructuredBuffer and StagingBuffer through a round-trip of data. - /// + /// [Test] public async Task Mix_Upload_Modify_Download_Staging() { @@ -69,7 +69,8 @@ public async Task Mix_Upload_Modify_Download_Staging() private static IComputeShader Create(Device device) { var fileSystem = new InMemoryFileSystem(); - return ShaderCompiler.CompileComputeShader(fileSystem, string.Empty, device, ShaderSource, "CS", "DeviceTest.cs"); + var includePath = new DirectoryPath(Path.GetTempPath()); + return ShaderCompiler.CompileComputeShader(fileSystem, includePath, device, ShaderSource, "CS", "DeviceTest.cs"); } private const string ShaderSource = """ diff --git a/source/CapriKit.Tests/IO/IOUtilitiesTests.cs b/source/CapriKit.Tests/IO/IOUtilitiesTests.cs new file mode 100644 index 0000000..7831269 --- /dev/null +++ b/source/CapriKit.Tests/IO/IOUtilitiesTests.cs @@ -0,0 +1,106 @@ +using CapriKit.IO; + +namespace CapriKit.Tests.IO; + +internal class IOUtilitiesTests +{ + [Test] + public async Task NormalizePathSeparators() + { + var normalized = IOUtilities.NormalizePathSeparators(@"C:\Windows\System32"); + await Assert.That(normalized.ToString()).IsEqualTo("C:/Windows/System32"); + } + + [Test] + public async Task NormalizeDotSegments() + { + var normalized = IOUtilities.NormalizeDotSegments(@"C:\Windows\System32\..\Fonts"); + await Assert.That(normalized.ToString()).IsEqualTo(@"C:\Windows\Fonts"); + } + + [Test] + public async Task Normalize() + { + var normalized = IOUtilities.Normalize(@"C:\Windows\System32\..\Fonts"); + await Assert.That(normalized.ToString()).IsEqualTo("C:/Windows/Fonts"); + } + + [Test] + public async Task AddTrailingDirectorySeparator() + { + var path = IOUtilities.AddTrailingDirectorySeparator("C:/Windows"); + await Assert.That(path.ToString()).IsEqualTo("C:/Windows/"); + } + + [Test] + public async Task RemoveTrailingDirectorySeparator() + { + var path = IOUtilities.RemoveTrailingDirectorySeparator("C:/Windows/"); + await Assert.That(path.ToString()).IsEqualTo("C:/Windows"); + } + + [Test] + public async Task IsValidFileName() + { + await Assert.That(IOUtilities.IsValidFileName("config.cfg")).IsTrue(); + } + + [Test] + public async Task IsValidFileName_InvalidCharacter() + { + await Assert.That(IOUtilities.IsValidFileName("config?.cfg")).IsFalse(); + } + + [Test] + public async Task IsValidFilePath() + { + await Assert.That(IOUtilities.IsValidFilePath("C:/Windows/config.cfg")).IsTrue(); + } + + [Test] + public async Task IsValidFilePath_InvalidCharacter() + { + await Assert.That(IOUtilities.IsValidFilePath("C:/Windows/config?.cfg")).IsFalse(); + } + + [Test] + public async Task IsValidPath() + { + await Assert.That(IOUtilities.IsValidPath("C:/Windows")).IsTrue(); + } + + [Test] + public async Task IsValidPath_InvalidCharacter() + { + await Assert.That(IOUtilities.IsValidPath("C:/Win|dows")).IsFalse(); + } + + [Test] + public async Task EscapeFileName() + { + var escaped = IOUtilities.EscapeFileName("shader.hlsl/VertexMain"); + await Assert.That(escaped.ToString()).IsEqualTo("shader.hlsl%2FVertexMain"); + } + + [Test] + public async Task SearchForDirectoryWithMarker() + { + var id = Path.GetRandomFileName(); + var rootPath = Path.Combine(Path.GetTempPath(), $"{nameof(IOUtilitiesTests)}.{id}"); + var root = new DirectoryPath(rootPath); + var nested = new DirectoryPath(Path.Combine(rootPath, "a", "b")); + try + { + Directory.CreateDirectory(nested); + File.WriteAllText(root.Append(new FilePath("marker.txt")), string.Empty); + + var found = IOUtilities.SearchForDirectoryWithMarker(nested, "marker.txt"); + + await Assert.That(found).IsEqualTo(root); + } + finally + { + new DirectoryInfo(root).Delete(true); + } + } +} diff --git a/source/CapriKit.Tests/IO/ScopedFileSystemTests.cs b/source/CapriKit.Tests/IO/ScopedFileSystemTests.cs index 4ad2a7e..72ebdf6 100644 --- a/source/CapriKit.Tests/IO/ScopedFileSystemTests.cs +++ b/source/CapriKit.Tests/IO/ScopedFileSystemTests.cs @@ -19,4 +19,18 @@ await Assert.That(() => sut.Exists(forbiddenPath); }).Throws(); } + + + [Test] + public async Task GetAbsolutePath() + { + var basePath = new DirectoryPath("C:/Temp"); + var fileSystem = new InMemoryFileSystem(); + var sut = new ScopedFileSystem(fileSystem, basePath); + + var relativePath = new FilePath("file.txt"); + var expected = new FilePath("C:/Temp/file.txt"); + + await Assert.That(sut.GetAbsolutePath(relativePath)).IsEqualTo(expected); + } } diff --git a/source/CapriKit.Tests/IO/Streams/IBufferWriterExtensionsTests.cs b/source/CapriKit.Tests/IO/Streams/IBufferWriterExtensionsTests.cs new file mode 100644 index 0000000..4a4f06b --- /dev/null +++ b/source/CapriKit.Tests/IO/Streams/IBufferWriterExtensionsTests.cs @@ -0,0 +1,76 @@ +using CapriKit.IO.Streams; +using System.Buffers; +using System.Buffers.Binary; +using System.Text; + +namespace CapriKit.Tests.IO.Streams; + +internal class IBufferWriterExtensionsTests +{ + [Test] + public async Task Write_Int32() + { + var writer = new ArrayBufferWriter(); + + writer.Write(-12345); + + var bytes = writer.WrittenMemory.ToArray(); + var value = BinaryPrimitives.ReadInt32LittleEndian(bytes); + + await Assert.That(bytes.Length).IsEqualTo(sizeof(int)); + await Assert.That(value).IsEqualTo(-12345); + } + + [Test] + public async Task Write_String() + { + var writer = new ArrayBufferWriter(); + var value = "héllo"; // 'é' encodes to two bytes, so the prefix must count bytes, not chars + + writer.Write(value); + + // The format promises to be identical to BinaryWriter/BinaryReader's + using var stream = new MemoryStream(writer.WrittenMemory.ToArray()); + using var reader = new BinaryReader(stream, Encoding.UTF8); + var text = reader.ReadString(); + + await Assert.That(text).IsEqualTo(value); + await Assert.That(stream.Position).IsEqualTo(stream.Length); + } + + [Test] + public async Task Write7BitEncodedInt() + { + int[] values = [0, 127, 128, 300, int.MaxValue, -1]; + var writer = new ArrayBufferWriter(); + foreach (var value in values) + { + writer.Write7BitEncodedInt(value); + } + + // The format promises to be identical to BinaryWriter/BinaryReader's + using var stream = new MemoryStream(writer.WrittenMemory.ToArray()); + using var reader = new BinaryReader(stream, Encoding.UTF8); + foreach (var value in values) + { + await Assert.That(reader.Read7BitEncodedInt()).IsEqualTo(value); + } + + await Assert.That(stream.Position).IsEqualTo(stream.Length); + } + + [Test] + public async Task Write_Guid() + { + var writer = new ArrayBufferWriter(); + var guid = Guid.NewGuid(); + + writer.Write(guid); + + var bytes = writer.WrittenMemory.ToArray(); + var roundTripped = new Guid(bytes, bigEndian: false); + + await Assert.That(bytes.Length).IsEqualTo(16); + await Assert.That(roundTripped).IsEqualTo(guid); + } +} diff --git a/source/CapriKit.Tests/IO/Streams/SequenceReaderExtensionsTests.cs b/source/CapriKit.Tests/IO/Streams/SequenceReaderExtensionsTests.cs new file mode 100644 index 0000000..21e3f45 --- /dev/null +++ b/source/CapriKit.Tests/IO/Streams/SequenceReaderExtensionsTests.cs @@ -0,0 +1,129 @@ +using CapriKit.IO.Streams; +using System.Buffers; +using System.Text; + +namespace CapriKit.Tests.IO.Streams; + +internal class SequenceReaderExtensionsTests +{ + // SequenceReader is a ref struct, so all reading happens before the first + // await and only plain locals cross into the assertions + + [Test] + public async Task SliceUnread() + { + var writer = new ArrayBufferWriter(); + writer.Write(111); + writer.Write(222); + + var reader = new SequenceReader(new ReadOnlySequence(writer.WrittenMemory)); + var slice = reader.SliceUnread(sizeof(int)); + var sliced = slice.ReadInt32(); + var sliceEnd = slice.End; + var remaining = reader.ReadInt32(); + var end = reader.End; + + await Assert.That(sliced).IsEqualTo(111); + await Assert.That(sliceEnd).IsTrue(); // the slice cannot see beyond its own section + await Assert.That(remaining).IsEqualTo(222); // the original reader skipped the sliced section + await Assert.That(end).IsTrue(); + } + + [Test] + public async Task ReadInt32() + { + var writer = new ArrayBufferWriter(); + writer.Write(-12345); + + var reader = new SequenceReader(new ReadOnlySequence(writer.WrittenMemory)); + var value = reader.ReadInt32(); + var end = reader.End; + + await Assert.That(value).IsEqualTo(-12345); + await Assert.That(end).IsTrue(); + } + + [Test] + public async Task ReadString() + { + var writer = new ArrayBufferWriter(); + writer.Write("héllo"); // 'é' encodes to two bytes, exercising the bytes-not-chars length prefix + + var reader = new SequenceReader(new ReadOnlySequence(writer.WrittenMemory)); + var value = reader.ReadString(); + var end = reader.End; + + await Assert.That(value).IsEqualTo("héllo"); + await Assert.That(end).IsTrue(); + } + + [Test] + public async Task ReadString_WrittenByBinaryWriter() + { + using var stream = new MemoryStream(); + using (var writer = new BinaryWriter(stream, Encoding.UTF8, leaveOpen: true)) + { + writer.Write("héllo"); + } + + var reader = new SequenceReader(new ReadOnlySequence(stream.ToArray())); + var value = reader.ReadString(); + var end = reader.End; + + await Assert.That(value).IsEqualTo("héllo"); + await Assert.That(end).IsTrue(); + } + + [Test] + public async Task Read7BitEncodedInt() + { + int[] values = [0, 127, 128, 300, int.MaxValue, -1]; + using var stream = new MemoryStream(); + using (var writer = new BinaryWriter(stream, Encoding.UTF8, leaveOpen: true)) + { + foreach (var value in values) + { + writer.Write7BitEncodedInt(value); + } + } + + var reader = new SequenceReader(new ReadOnlySequence(stream.ToArray())); + var results = new List(); + while (!reader.End) + { + results.Add(reader.Read7BitEncodedInt()); + } + + await Assert.That(results.SequenceEqual(values)).IsTrue(); + } + + [Test] + public async Task ReadGuid() + { + var guid = Guid.NewGuid(); + var writer = new ArrayBufferWriter(); + writer.Write(guid); + + var reader = new SequenceReader(new ReadOnlySequence(writer.WrittenMemory)); + var value = reader.ReadGuid(); + var end = reader.End; + + await Assert.That(value).IsEqualTo(guid); + await Assert.That(end).IsTrue(); + } + + [Test] + public async Task ReadBytes() + { + var bytes = new byte[] { 1, 2, 3, 4, 5 }; + var writer = new ArrayBufferWriter(); + writer.Write(bytes); + + var reader = new SequenceReader(new ReadOnlySequence(writer.WrittenMemory)); + var value = reader.ReadBytes(bytes.Length); + var end = reader.End; + + await Assert.That(value.SequenceEqual(bytes)).IsTrue(); + await Assert.That(end).IsTrue(); + } +} diff --git a/source/CapriKit.Tests/IO/FileSystemEventListenerTests.cs b/source/CapriKit.Tests/IO/Watchers/FileSystemEventListenerTests.cs similarity index 70% rename from source/CapriKit.Tests/IO/FileSystemEventListenerTests.cs rename to source/CapriKit.Tests/IO/Watchers/FileSystemEventListenerTests.cs index d991854..6503718 100644 --- a/source/CapriKit.Tests/IO/FileSystemEventListenerTests.cs +++ b/source/CapriKit.Tests/IO/Watchers/FileSystemEventListenerTests.cs @@ -1,6 +1,8 @@ using CapriKit.IO; +using CapriKit.IO.Watchers; +using CapriKit.Tests.TestUtilities; -namespace CapriKit.Tests.IO; +namespace CapriKit.Tests.IO.Watchers; internal class FileSystemEventListenerTests { @@ -9,10 +11,7 @@ internal class FileSystemEventListenerTests [Before(Test)] public void TestSetup() { - var id = Path.GetRandomFileName(); - var path = Path.Combine(Path.GetTempPath(), $"{nameof(FileSystemEventListenerTests)}.{id}"); - TempDirectory = new DirectoryPath(path); - Directory.CreateDirectory(TempDirectory); + TempDirectory = FileSystemUtilities.CreateTemporaryDirectory(); } [After(Test)] @@ -30,23 +29,22 @@ public async Task OnFileChanged() var changed = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); var deleted = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); - var fileSystem = new FileSystem(); var scopedFileSystem = new ScopedFileSystem(fileSystem, TempDirectory); - using var watcher = fileSystem.Watch(TempDirectory); + var watcher = scopedFileSystem.Watch(TempDirectory, false); watcher.OnFileChanged += (s, e) => { - if (e.reason == FileSystemChangeKind.Created && e.target.FileName.Equals(fileName, StringComparison.OrdinalIgnoreCase)) + if (e.Kind == FileSystemChangeKind.Created && e.File.FileName.Equals(fileName, StringComparison.OrdinalIgnoreCase)) { created.TrySetResult(); } - if (e.reason == FileSystemChangeKind.Changed && e.target.FileName.Equals(fileName, StringComparison.OrdinalIgnoreCase)) + if (e.Kind == FileSystemChangeKind.Changed && e.File.FileName.Equals(fileName, StringComparison.OrdinalIgnoreCase)) { changed.TrySetResult(); } - if (e.reason == FileSystemChangeKind.Deleted && e.target.FileName.Equals(fileName, StringComparison.OrdinalIgnoreCase)) + if (e.Kind == FileSystemChangeKind.Deleted && e.File.FileName.Equals(fileName, StringComparison.OrdinalIgnoreCase)) { deleted.TrySetResult(); } diff --git a/source/CapriKit.Tests/TestUtilities/CapturingLoggerFactory.cs b/source/CapriKit.Tests/TestUtilities/CapturingLoggerFactory.cs new file mode 100644 index 0000000..241fd1d --- /dev/null +++ b/source/CapriKit.Tests/TestUtilities/CapturingLoggerFactory.cs @@ -0,0 +1,35 @@ +using Microsoft.Extensions.Logging; + +namespace CapriKit.Tests.TestUtilities; + +/// +/// Logger factory that keeps every formatted message, for tests that assert on a diagnostic instead of on +/// a return value. Messages may be written from any thread, reading them is meant for the main thread. +/// +internal sealed class CapturingLoggerFactory : ILoggerFactory +{ + private readonly List messages = []; + + public IReadOnlyList Messages + { + get { lock (messages) { return [.. messages]; } } + } + + public ILogger CreateLogger(string categoryName) => new CapturingLogger(messages); + + public void AddProvider(ILoggerProvider provider) { } + + public void Dispose() { } + + private sealed class CapturingLogger(List messages) : ILogger + { + public IDisposable? BeginScope(TState state) where TState : notnull => null; + + public bool IsEnabled(LogLevel logLevel) => true; + + public void Log(LogLevel logLevel, EventId eventId, TState state, Exception? exception, Func formatter) + { + lock (messages) { messages.Add(formatter(state, exception)); } + } + } +} diff --git a/source/CapriKit.Tests/TestUtilities/FileSystemUtilities.cs b/source/CapriKit.Tests/TestUtilities/FileSystemUtilities.cs new file mode 100644 index 0000000..85db355 --- /dev/null +++ b/source/CapriKit.Tests/TestUtilities/FileSystemUtilities.cs @@ -0,0 +1,17 @@ +using CapriKit.IO; +using CapriKit.Tests.IO.Watchers; + +namespace CapriKit.Tests.TestUtilities; + +internal static class FileSystemUtilities +{ + public static DirectoryPath CreateTemporaryDirectory() + { + var id = Path.GetRandomFileName(); + var path = Path.Combine(Path.GetTempPath(), $"{nameof(FileSystemEventListenerTests)}.{id}"); + var directory = new DirectoryPath(path); + Directory.CreateDirectory(directory); + + return directory; + } +}